Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models
Authors: Nanxi Li
Xiang Wang
Yuanjie Chen
Haode Zhang
Hong Li
Yong-Lu Li
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal LLM
Intuitive Physics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper introduces two low-level tasks to test intuitive physics understanding and proposes Scene Dynamic Field, a method to integrate visual representation from physics simulators to MLLMs while showcasing generalization.

Abstract: While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in image and video understanding, their ability to comprehend the physical world has become an increasingly important research focus. Despite their improvements, current MLLMs struggle significantly with high-level physics reasoning. 
In this work, we investigate the first step of physical reasoning, i.e., **intuitive physics understanding**, revealing substantial limitations in understanding the dynamics of continuum objects. 
To isolate and evaluate this specific capability, we introduce two fundamental benchmark tasks: Next Frame Selection (NFS) and Temporal Coherence Verification (TCV). Our experiments demonstrate that even state-of-the-art MLLMs perform poorly on these foundational tasks. 
To address this limitation, we propose Scene Dynamic Field (SDF), a concise approach that leverages physics simulators within a multi-task fine-tuning framework. 
SDF substantially improves performance, achieving up to $20.7\%$ gains on fluid tasks while showing strong generalization to unseen physical domains. This work not only highlights a critical gap in current MLLMs but also presents a promising cost-efficient approach for developing more physically grounded MLLMs. Our code and data are available at https://github.com/andylinx/Scene-Dynamic-Field.

Published as a conference paper at ICLR 2026
BEYOND STATIC VISION: SCENE DYNAMIC FIELD UNLOCKS INTUITIVE PHYSICS UNDERSTANDING IN MULTI-MODAL LARGE LANGUAGE MODELS
Nanxi Li', Xiang Wang", Yuanjie Chen', Haode Zhang', Hong Li', Yong-Lu Li Shanghai Jiao Tong University 2 Tianjin University (andyc_03, yonglu.lij0s edu. en
SShnnhaii Innovation Institute
ABSTRACT
While Multimodal Large Language Models (MLLMs) have demonstrated im- pressive capabilities in image and video understanding, their ability to compre- hend the physical world has become an increasingly important research focus. Despite their improvements, current MILMs struggle significantly with high- level physics reasoning. In this work, we investigate the first step of physical reasoning, i.e., intuitive physics understanding. revealing substantial limita- tions in understanding the dynamics of continuum objects. To isolate and eval- uate this specific capability, we introduce two fundamental benchmark tasks: Next Frame Selection (NFS) and Temporal Coherence Verification (TCV). Our experiments demonstrate that even state-of-the-art MILMS perform poorly on these foundational tasks. To address this limitation, we propose Scene Dynamic Field (SDF), a concise approach that leverages physics simulators within multi- task fine-tuning framework. SDF substantially improves performance, achiev- ing up to 20.7% gains on fluid tasks while showing strong generalization to unseen physical domains. This work not only highlights a critical gap in cur- rent MLLMs but also presents a promising cost-efficient approach for devel- oping more physically grounded MLLMs. Our code and data available at https:://ithuu com/ andy i Saccen--Dnnaee ci c-Field
INTRODUCTION
Recently, Multimodal Large Language Models (MLLMs) have exhibited great success in image and video understanding Yue et al. (2024); Li et al. (2024)). However, MLLMs still face significant limitations in capturing the intuitive physical dynamics of real-world scenarios Zheng et al. (2024); Chow et al. (2025). These shortcomings stem from the training recipe and data property. The prevalent approach of treating videos as sequence of frames processed by image encoders and trained end-to-end fails to adequately capture the low-level dynamics essential for understanding physics |abss (2024). On the other hand, video encoders Wang al.) (2023): Zhha et al.) (2024a); Wang et al. (2024d); Bardes al.] are often trained in an unsupervised manner
fetion-focused datasets Kay et al. (2017); Goyal et al. 2017); Kuchne et al. (201I), which are effective for un- derstanding human-centered activities but lack the dynamics of continuum objects such liquids, cloth, and other deformable materials.
As shown in Figure 1 existing benchmarks Zheng et al. (2024): Chow et al. (2025) mostly evaluate the high-level physical reasoning capacities of multimodal large language models (MLLMs). These frameworks assess multiple capabilities through diverse tasks, including physical property question- answering (QA), predictive and counterfactual inference, and tasks are critical for advancing intuitive physics understanding MLLMs. they are related to not only physics but also vision, language, common sense, logic, etc.. This has led to an alarming performance gap revealed by empirical findings: MLLMs consistently fail across these benchmarks, often performing only marginally better than random guessing.
analysis. While such
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Existing Benchmarks
Next Frame Selection (NFS)
Q: (Promps] Which one of the following is the neat frame?
the stick were which orange fluid pass?
Temporal Coherence Verification (TCV)
Itapel
A. Gray stick Red stick O.Orange
(Prompt) Does
not consestnt in this video sequence?
Please acnover with
Low-Level Tasks
Physical Common
Hig
Sense
Dynamic
Scene Dynamic Field (SDF)
Perception
clect
Temporal Logical
Fluid NES Task
staoke
Spatial Inference
Reasoning
Transfer
Sprthtt Smoke
Scene Dynamic Field
Existing benchmarks entangle multiple capabilities, leading poor performance in SOTA MLLMs. To address this, we introduce two low-level tasks to assess intuitive physics understand- ing: Next Frame Selection and Temporal Coherence Verification. Our proposed Scene Dynamic Field (SDF) directly enhances MILMs dynamic understanding and shows strong generalization.
Figure 1:
Thus, similar to the concept of curriculum learning Bengio et al. 2009). we should also adopt "curriculum benchmarking" for MILM. In this work, disentangle the physical reasoning benchmarking itself and focus on its very first step, i.e., evaluating the most fundamental and prin- cipal physical understanding ability: intuitive physies understanding. The reason is that, without establishing whether MLLMs can accurately perceive physical motion and change over time, we cannot meaningfully address their higher-level reasoning deficiencies or develop targeted enhance- ment strategies. After grounding MLLMs in the fundamental perception of dynamics, we can then enhance their capacity for more complex, causal physical reasoning, as they can operate on a more faithful internal representation of the world's state changes.
natural ask How can we disentangle the problem to evaluate a model's intuitive physics un- derstanding capability effectively? and How can we further enhance this critical ability? To this end, we introduce two low-level tasks: Next Frame Selection (NFS) and Temporal Coherence Verification (TCV), which systematically assess intuitive physics understanding. Extensive exper- iments reveal that current MLLMs still fall short of achieving a satisfactory level of understanding. Even the best- persooerfonnnn model, Qwen2.5-V Team (2025). achieves only 30.0% accuracy multiple-choice NFS questions. To further investigate these limitations, we analyze the impact of model scaling and find that increasing scale alone cannot lead to significant improvements.
In light of the current reasoning capability of MILMs, we also explored the potential language- based reasoning. Despite observing improvement, there remains great potential, since leveraging only language to reason is far from enough to capture real-world physics. Recent progress in think- with-images yyu Su al. (2025) motivated us introduce additional visionary supervisory signals while leveraging existing MLLM capabilities.
To this end, we propose Scene Dynamic Field (SDF), a novel method designed enhance intu- itive physics understanding. SDF leverages physics simulators, which, despite their limitations in fine-grained detail, consistently capture dynamic trends that align with real-world physical phenom- ena. Through generating SDF data from simulators, we developed a carefully designed multi-task fine-tuning strategy that integrates Shain-of-Thought (CoT) reasoning, where SDF acts as a visual prompt to guide the model's 's understanding of intuitive physics. Empirical experiments demonstrate significant performance improvements across both benchmark tasks, with up to 20.7% gains on fluid tasks and strong generalization to unseen physical domains, validating the effectiveness of ap- proach. Moreover, as particle-base physics simulation engines continue to advance in speed and generation quality, our method provides scalable and cost-effective approach for distilling physical
2

---

Published as a conference paper at ICLR 2026
knowledge from these simulators. By abstracting synthetic data to optimal representational level, we enable MLLMs to better perceive underlying physical dynamics.
To summarize, contributions are mainly three-fold:
We introduce Next Frame Selection (NFS) and Temporal Coherence Verification (TCV), two complementary diagnostic tasks designed to disentangle and evaluate fundamental in- tuitive physics understanding abilities MLLMs, revealing their significant deficiencies.
We propose Scene Dynamic Field (SDF), an intermediate representation that bridges the gap between physics simulators and MLLMs, providing explicit dynamic cues that enhance MI.LMs' physical understanding.
Through extensive empirical analysis, we highlight the significant limitations intuitive physics understanding of current MLLMs, while demonstrating that our proposed SDF achieves strong generalization, paving the way for more physically grounded MLLMs.
RELATED WORK
Intuitive Physics Understanding. Existing benchmarks for physical reasoning Zheng et al. (2024);
Chow et al. (2025) primarily evaluate high-level physical reasoning through complex tasks that en- tangle multiple capabilities. ContPhy Zheng et al. (222)) integrates qualitative reasoning about di- verse physical properties with dynamic scenario prediction through video question answering, while PhysBench Chow et al. (2025) proposes a structured evaluation framework containing question- answering tasks across different cognitive levels. These frameworks assess multiple capabilities simultaneously, including vision, language, common sense, logical reasoning, and physical under- standing through diverse tasks such as realistic physical simulation Chow et al. (2025); Zheng et al.
(2024); Tung al. (2023). temporal sequence understanding Patel et (2022) Tung et al. (2023). and physical property recognition Johnson et al. (2016); Chen et al. (2022) Jassim et al. (2024).
This entanglement has led concerning empirical findings: studies examining MLLMs' capabilities in intuitive physics understanding reveal that performance approaches random chance levels on such tasks Ballout et al. (2025). Garrido et al. (2025) further confirmed these poor performance findings while revealing that V-JEPA Bardes et al. frameworks show more promise. However, the complexity of existing benchmarks makes it difficult to pinpoint whether models fail due to inadequate physical understanding or deficiencies in other cognitive abilities.
These existing benchmarks leave a fundamental gap: they do not isolate and evaluate low-level in- tuitive physical perception, which is the most basic prerequisite for physical reasoning. Without establishing whether MLLMs can accurately perceive physical motion and change over time, we cannot meaningfully address their higher-level reasoning deficiencies develop targeted enhance- ment strategies. Drawing inspiration from curriculum learning principles, we propose a systematic evaluation of this foundational capability through targeted, low-level tasks that isolate physical per- ception from other cognitive requirements.
Current enhancement approaches have attempted to mitigate these performance issues through var- ious strategies. Some work Sharma et al. (0255) incorporates external tools or components, while other physics-aware AI systems integrate Graph Neural Networks (GNNs) to predict particle dy- namics Kazemi et al. (2024) from state vectors or employ symbolic physics engines for explicit rea- soning. However, these methods are often designed for multi-step logical deduction rather than em- powering foundation models with fundamental perceptual capabilities. Moreover, language-bar reasoning improvements, while helpful, prove insufficient for capturing the complex temporal and spatial relationships inherent in physical phenomena.
In contrast, we propose enhancing MILMSs by leveraging physics simulators to provide direct visual cues about physical motion and change, addressing the perceptual limitations at their source rather than relying solely on language- mediated reasoning or external computational modules.
Multi-modal Large Language Models (MLLMs). Recently, there has been remarkable progress systems like Qwen VL series Wang et al. (2024b); Team (025),
in MLLMs, with state-of-
Intern VL series Chen et al. 2023); Zhu et al. (2025). and GLM-V serires Hong et al. 2025 achiev- ing impressive performance on traditional vision- languaee benchmarks such as image captioning, visual question answering, and multi-modal reasoning tasks Wang al. 2024e).
3

---

Published as a conference paper at ICLR 2026
Despite these remarkable achievements, current MLLMs exhibit significant limitations in under- standing physical phenomena and dynamics, particularly
centric scenarios such as fluid dynamics, material deformation, and object interactions governed by physical laws Zheng al. (2024); :Chow et al. 2025). This stems from their training focus on high-level semantic understand- ing rather than fundamental physical principles. Recent advances have introduced explicit reasoning capabilities through reinforcement learning, with models like GLM-4.1 V. Hong et al. (2025) achiev- ing success in mathematics and structured reasoning tasks. However, without an accurate perception of physical motion and temporal dynamics, these reasoning mechanisms cannot compensate for the lack of intuitive physies understanding. This creates critical bottleneck: while current MLLMs excel complex logical reasoning about abstract concepts, they struggle with basic physical un- derstanding that humans develop intuitively, highlighting the need for systematic evaluation and targeted enhancement of low-level physical perception capabilities.
BENCHMARK CONSTRUCTION
3.1 MOTIVATION
Contemporary video understanding benchmarks Li et al. (2024c); Wang et al.] (2024c); Yu et al. (219)) have made significant strides in evaluating high-level event comprehension, successfully capturing human actions [i et al. (2023a) 2019), object interactions, and temporal relationships in videos Li et al. (202dd). However, they often overlook the crucial aspect of low-level dynamic understanding of MLLMs, particularly the intricate physical behaviors and temporal evolution of objects and materials Li al. (2023b). This limitation is especially apparent when dealing with continuum objects, whose behavior is governed by complex physical principles rather than discrete state changes. Among various continuum phenomena, fluid dynamics presents an ideal testbed due to its ubiquity in everyday scenarios and its rich, continuous dynamic patterns. Therefore, choose fluid dynamics our primary focus to comprehensively evaluate current foundation models' capability in intuitive dynamics.
To disentangle the problem of physical reasoning, we propose focused benchmark that specifi- cally evaluates models' capability in intuitive physical understanding, allowing us to disentangle and better comprehend this fundamental intelligence.
3.2 EVALUATION SYSTEM
Let J = (1i denote sequence of I'' input frames. The strategy proceeds as follows:
Interval Sampling with Stride. Partition F into non-overlapping intervals {I.} via a temporal
stride Each interval Z [ ftantt...... Stau) spans subsequence of frames, where consec-
utive intervals satisfy ((+1) -1((1) =
Distractor Candidate Generation. For each interval I,, extract distractor candidate set D; by excluding frames within a temporal buffer of size around the interval, then
D. = {fe & [max([1 tstar 6), min(tens + 8, 8,T)1}.
(I)
ensuring D, excludes frames that are temporally close to I,
Similarity-Ban -Bssed Pruning. Let fa denote the ground truth frame associated with Z,, Using similarity metric sim(...), filter out candidates overly aligned with fgi
Dj = {fe D, sim( <T}.
(2)
where r is threshold ensuring only semantically distinct distractors retained. The final evalua- tion set for I, combines fg with D/. challenging models to isolate the true successor amid plausible alternatives.
We employ two complementary protocols to assess temporal and dynamic understanding capabili- ties:
4

---

Published as a conference paper at ICLR 2026
Next Frame Selection (NFS). For interval I, with ground truth fgi First sampling 3 distractors d1:3 ~ D and then computing the selection accuracy ACCNES
(Pmodel (fgal4a > max Pmodal ( (d)lZ)).
(3)
NFS Ace (%)
TCV Acc (%)
Category
Model
#Param
Stride 2
Stride
Stride 2
Stride 4
Intern VL2.5
8B
17.53
20.19
52.95
52.31
Intern VL3
8B
19.33
18.33
53.00
54.94
Open-source
GLM-4.1V
24.25
25.43
51.40
54.01
mPLUG-OW13
7B
27.77
29.37
52.19
53.20
Qwen2-VL VL
24.00
26.80
54.00
53.24
Qwen2.5-VL
7B
32.73
30.00
57.80
56.63
GPT-4o
33.19
39.79
70.10
69.91
Closed-source
Gemini-2.5-Flash
31.20
31.37
67.00
70.06
Table 1: Performance of models on NFS and TCV tasks. NFS Ace: Accuracy on next frame selec- tion (4-choice MCQ). TCV Ace: Accuracy on temporal coherence verification (Yes/No).
Temporal Coherence Verification (TCV). Given sequence S and corrupted version Š with random distractor insertion, evaluate binary detection accuracy Acercv
Σ
T(arg max Proodel (.15) TIS S)). Y/N
(4)
M
This systematic approach mitigates evaluation bias by enforcing diversity in distractor frames while preserving temporal coherence.
3.3
DATA PREPARATION
We have adopted fluid video data from multiple sources, including Contphy Zheng al. (2024) and PhysBench Chow et al. (2025). To enhance the diversity and real-world applicability our dataset, supplement these synthetic simulations with real-world videos collected through web mining. Specifically, we employ LLMs to generate structured search queries combining action verbs with fluid types (e.g.. "pour honey").
To ensure data quality, we segment the collected videos into 5-second clips and implement a ro- bust filtering mechanism. This mechanism utilizes ensemble of MLLMs, including Qwen VL- Max Waag et al. (2024b). LLaVA-One (024), to perform consensus-based filtering, eliminating clips that do not contain relevant fluid in- teractions. Then, we use the above-deseribed process to curate the benchmark. Specifically, we use SigLIP Zhai al. (2023) embeddings to compute frame-level representations and utilize cosine similarity as our sim(-, metric. To ensure benchmark quality, we incorporate a manual verification phase where human annotators filter out ambiguous or low-quality samples. The final curated test set comprises 4,000 samples derived from about 1,000 unique videos.
Li et al. (2024a). and InternVLa (att20ppeGGLaa Team
3.4 ANALYSIS AND INVESTIGATION
We evaluate our benchmark with state-of-the-art MILMs, including Intern VI.2.5 OpenCVLLat Team (2024), Intern Zhu et al. (2025), mPL.U-OOI33 Ye et al. (2024), Qwen2-VL. Wang et al. (2024b), Qwen2.5-Vi VL Team (2025) and GLM4. Hong et al. (2025). These models have demon- strated superior performance in various vision-language tasks and feature different model architec- tures and pre-trane tre-training strategies. All models are evaluated without any task-specific fine-tuning to assess their zero-shot capabilities. For both the Next Frame Selection (NFS) and Temporal Coher- ence Verification (TCV) tasks, we conduct experiments with sequences of 5 input frames, utilizing temporal strides of 5 2 and 8 4 frames. A detailed ablation is presented in Section B
5

---

Published as a conference paper at ICLR 2026
Model
#Param
NFS
TCV
Model
NFS
TCV
3B
24.4
51.9
Qwen2.5-VL + +ooT
30.00
56.63
Qwen2-VL
7B
26.8
53.2
31.12 (+1.12)
64.20 (+7.57)
72B
28.1
52.1
GLM-4.1V
25.43
54.01
2B
21.3
52.9
+ +Thinking
37.20 (+11.77)
79.07 (+25.06)
4B
22.2
51.6
Intern VL2.5
8B
20.2
52.7
Gemini-2.5-Flash
31.37
70.06
26B
20.6
55.1
+ +Thinking
37.33 (+5.96)
72. 16 (+2.10)
Table 2: Model scaling analysis (left) and the effect of CoT prompting and thinking (right).
The experimental results in Table T reveal striking deficiencies in current models' intuitive physics understanding capabilities. Open-source models demonstrate particularly poor performance, with most achieving NFS accuracies below 30% (compared to a 25% random baseline for 4-choice se- lection). Even the best-performing open-source model, Qwen2.5- VL,, reaches only 32.73% on NFS. While closed-source models like GPT-4o and Gemini-2.5-PP performance remains far from satisfactory, with NFS accuracies peaking 39.79%. For the TCV task, open-source models cluster around 52-58% accuracy, only slightly above the 50% random baseline for binary classification. These results collectively indicate that current MLLMS fail to develop effective dynamic representations for intuitive physics understanding.
show limited improvements, their
Scaling Performance. Most current foundation models evaluated in our benchmark have parameters around 7-8B, showing limited performance on physical dynamics understanding. To investigate whether this limitation be addressed through model scaling, we conduct a comprehensive scaling analysis using Intern models and Qwen2-VI. models on the NFS task.
As shown in Table 2 although larger model sizes generally correlate with better performance, the observed improvements remain relatively incremental. For instance, Qwen2-VL demonstrates pro- gressive gains NFS, rising from 24.40% to 28.1 as parameters scale from 3B 72B. However, MiterVLL.55 exhibits less consistent scaling behavior, with its NFS accuracy actually declining from 25.33% 20.60% over the 2B to 26B parameter range. This divergence underscores the limitations of parameter scaling alone for achieving robust comprehension of physical dynamics, suggesting that targeted training methodologies may be necessary to complement pure model size expansion.
CoT Prompting and Thinking Mode. As shown in the right panel of Table 2 Chain-of- Thoughh prompting and thinking modes demonstrate notable improvements over baseline performance. GLM-4. benefits most dramatically from thinking mode, achieving gains of 11.77 on NFS and 25.06 on TCV, while CoT provides modest improvements for Qwen2.5-VL. (1.12 on NFS, 7.57 on TCV). These results indicate that explicit reasoning processes enhance intuitive physics under- standing, with thinking modes generally outperforming standard CoT approaches across both tasks.
However, while these language-baset reasoning improvements promising, they reveal funda- mental limitations in addressing the core challenge of physical dynamics understanding. The gains, though significant, still leave models far from satisfactory performance levels, suggesting that pure linguistic reasoning is insufficient for capturing the complex temporal and spatial relationships in- herent in physical phenomena. This observation motivates our approach: rather than relying solely on language- Language-mediated mediated reasoning, we propose to enhance models' perceptual capabilities through explicit visual representations of physical dynamics, leading us develop the Scene Dynamic Field (SDF) method described in the following section.
SCENE DYNAMIC FIELD
Our evaluation and comprehensive analysis reveal a significant limitation in current MLLMs: they struggle to effectively understand and reason about physical dynamics. This limitation persists across model scales and architectures, suggesting fundamental gap in their ability to process tem- poral dynamics. While language-basee based reasoning improvements show promise. they are ultimately insufficient for capturing the complex spatio-temporal relationships inherent in physical phenomena.

---

Published as a conference paper at ICLR 2026
This observation motivates approach to enhance the models' perceptual capabilities through explicit visual representations. To this end, we introduce the Scene Dynamic Field (SDF), a lightweight, representation-levd bridge that injects physical knowledge without requiring costly architectural overhauls. SDF leverages physics simulators to generate visual prompts that represent motion. Although these simulators may lack fine-grained detail, they consistently capture dynamic trends that align with real-world phenomena. By abstracting these dynamics, SDF provides a valu- able foundation for learning, allowing for broad compatibility with existing MLLMs while directly addressing the perceptual gap we identified.
SDF implements an explicit mechanism for modeling temporal evolution and physical interactions through visual prompts derived from physics engines such as Unity Unity Technologies (2023) and Blender Community (2018), thereby complementing MILLMs' inherent strengths high-level rea- soning while addressing their deficiencies in low-level physical understanding.
4.1 CONSTRUCT SDF FROM SIMULATOR
We utilized the Flip Fluids Fluids addon, which performed well in Blender Community (2018). To generate simulated videos that contribute meaningfully, constructed scenes commonly encountered video-related tasks, such embodied manipulation and VQA.
0.5
008
Settings. We generated a series of videos featuring various liquid-related actions, such as pouring, stirring. and comparicoo demonstrations. To improve the model's gener- alization across diverse scenarios, we system- atically varied multiple factors, including the physical properties of the liquids (e.g., initial velocity, viscosity, and color), the visual char- acteristics of the containers, the background en- vironments, and the camera perspectives. For details, please refer to the Appendix(D)
o.
prjjetees valccb
Figure 2: Illustration our Scene Dynamic Field (SDF).
Visual Prompting Strategy. As shown in Fig- ure enhance the utility of the Scene Dy- namic Field for MIL.Ms, we devised visual prompting strategy as follows.
Consider system of particles with velocity vectors v, E R For camera positioned at RR,
the projected velocity magnitude Upmjji for each particle is computed as
Uproj. = ||vel| cos 0; (v; ft),
(5)
Pt where |[c-p.]l the unit vector pointing from the particle's position pa to the camera. The density of the blue channel Dg is then modeled as a line integral:
|v
DB(C) = 1+alcc p.I dR,
(6)
where 6 scales velocity to color intensity, governs spatial attenuation, and f2 represents the observ- able domain. Particles with larger Uprjji values contribute disproportionately to the blue channel due to the ||v term, creating depth perception through spectral segregation. This formulation effec- tively maps dynamics into the camera's 's reference frame to a perceptually calibrated blue gradient.
4.2 MULTI-TASK FINE-TUNING STRATEGY
Our analysis reveals fundamental gaps in intuitive physics understanding that cannot be resolved through standard video pretraining. We propose a targeted adaptation strategy leveraging physics simulator outputs, structured follows:
Task Dynamic Perception. As depicted in Figure 3
the MLLM is tasked with analyzing an RGB video sequence alongside candidate images to identify the SDF representation, where regions

---

Published as a conference paper at ICLR 2026
Dynamic Perception
SDF-Guided Chain-of-Thought
ViGa
Given of video frames
PSDF
Sceue Dynemic Field visualize the flow of motion by assigning different velocity maguitudes, where with
higher velocities are regresented deeper blue shades.
Psnp
Scene Dynamic Field visualie
VRGB
Given sequenooe
VSDF Which one of the following is the SDF of the les, giveniven Frame?
CoT
Let's think stop:
Step 1: malyze fluid dynamics the provided frames.
Original Low-Level Tasks
Step The Scene Dynamic Field of the last
frame should be following: </
NFS Which one of Lhe following the nexl liame?
Step3: Based on these dynamic, the next frome the same dynemic pattem as established in the previous Irames.
<mage>> <image> <imgge D <image>
TCV Does frame look unnelurd graol consistent?
Question
Which one of the following is the next trame?
Figure 3: Our multitask framework integrates low-level tasks, a dynamic perception task, and an SDF-guided CoT reasoning task.
of higher velocity magnitude are encoded with increased blue chromatic intensity. The input consists of two components: an RGB video VRGB = |o |KGB] depicting dynamic interactions, and N candidate images ..,,...,
Among these candidates, one image corresponds to the ground- truth SDF (/SDF) generated through velocity-to-coke mapping based on the formulation Section while the remaining N images serve as distractors.
Following the approach in Section 3.2, we used a similar method to select distractors. These care- fully chosen examples balance challenge and clarity, helping the model better understand movement and flow in dynamic scenes.
Task 2: 3DF-Guided
Thiought.- We develop three- stree Chain-of- Thhought (CoT) rea- soning framework enhanced by SDF visual prompts to generate a physics grriiide NES task.
For input frames FF
{fe 1T insert SDF frames at strategic positions visual prompt: Fost , JRGB (SDD).
[SRGB (RGB, PRGB
In the context of the NFS, the dynamic field of the final frame plays critical role in the CoT process. As shown in Figure a meticulously designed three-step CoT framework was developed to enhance dynamic reasoning and predictive capabilities. Initially, MLLM is tasked with analyzing the fluid dynamics present in the provided sequence of frames. Subsequently. it integrates the given scene dynamic field corresponding to the last frame. Finally, the model is required select the subse- quent frame based on the provided visual prompts and reasoned thought processes. This structured approach aims to optimize the model's ability to interpret and predict dynamic scenarios effectively.
Together with Task1, Task2, and the original task (NFS/TCV), we propose to further leverage the model's s inherent reasoning capabilities through self-distillation the thinking procedure. en- hance performance, we incorporate responses from stronger models (e.g., Gemini-2.5-Par to guide effective utilization our proposed SDF as visual prompts in the reasoning process. However, as demonstrated by recent work Chen et al. (2025). reasoning processes from expert models are not always optimal solutions. Self-distillation approaches can also promising [Zhang al. (2025), they exhibit smaller distribution shifts during model training. Therefore, combine expert- generated data with self-distileed data at a ratio of : 10. We refer to the Appendix for ablation studies 6 on this choice and the specific prompts employed.
4.3 EXPERIMENT
Settings. We evaluate four distinct experimental settings NFS and TCV benchmarks. The Zero-Shot setting evaluates MLLMS without any task-specific tuning to establish baseline perfor- mance capabilities. The Finetune setting applies supervised fine-tuning the base model on NFS
8

---

Published as a conference paper at ICLR 2026
Performance Fluid Benchmark
Qwen2-VL Transfer Parformance
GLM4.1V Transfer Performance
100
60
60-
Shot
TDY 11)
50-
1GL MEVI
50F-Ours
50F
NFS Score (16)
TCV Score (%)
40
40-
-0-
-80
Score (%0
Score (%)
24.8
25.4
30
30
20-
-70
20-
20-
Finetune SDF-Ours 50
Shot
Cloth
Plasticime
Cloth
SmokePlavticine
Figure 4: Performance of our SDF method across various evaluation scenarios. (A) shows results on the Fluid dataset for both NFS and TCV tasks. (B) and (C) present transfer results to cloth, smoke, and other particle-based objects Qwen2-VI. and GLM4.1 VV, respectively.
and TCV tasks separately, using the same number of training instances as our SDF method to ensure a fair comparison. The CoT (Reasoning Only) setting employs Chain-of-Thought prompting Wwu eet al. (2023) without any additional training. leveraging its demonstrated effectiveness in enhanc- ing intuitive physics understanding Jiang et al. (2025). We develop lask-specific CoT prompts for both NFS and TCV tasks (see Appendix for details B). Finally, SDF-Ours represents proposed approach. which fine-tunes models using our multi-task framework incorporating Dynamic Percep- tion, SDF-guided COT, and the original NES/TCV tasks. Both Finetune and SDF-Ours employ full-parameter supervised fine-tuning via the SWIFT Zhao et al. (2024b) framework, with learning rate of le 5 across epochs. To show the sim-to-real performance, we test on the previously proposed NFS and TCV datasets, which contain real-world datasets. All experiments are conducted over independent runs on 4 A100 40G GPUs, and results are reported with confidence intervals.
To test the hypothesis that our SDF method facilitates a more fundamental learning of physical dynamics rather than domain-spectie memorization, we conducted exploratory transfer experiments on other continuum phenomena, including cloth, sand, and smoke, and other particle-basee objects. We conducted transfer experiments on Qwen2-V BL and GLM4.1V under identical settings, with both the fine-tuned and SDF configurations trained exclusively on the Fluid dataset. These experiments aim to assess the adaptability and robustness of SDF when applied to diverse physical simulation domains. For more details, please refer to the Appendix 8
Figure f demonstrates the effectiveness of our proposed SDF across various evaluation scenarios. As shown in (A), SDF-Ours achieves substantial improvements on the Fluid benchmark for both NFS and TCV tasks. For the NFS task, SDF-Ours shows a significant performance gain of 14.40% on Qwen2-VL and 20.7% on GLM4.IV compared to Zero-Shot, More remarkably, as illustrated in (B) and (C), while standard Finetune performs nearly identically to Zero-Shot when transferred to other domains (e.g., 23.6 23.64% vs. 22.42% for cloth Qwen2-Vi.), SDF-Ours maintains noticeable improvements across all transfer domains. These results strongly suggest that our approach facili- tates genuine learning of physical dynamics rather than mere domain-specific pattern recognition, enabling effective generalization to diverse physical phenomena beyond the training domain. We provide additional case studies and failure case analysis G in the Appendix for further discussion.
CONCLUSION
In this work, we introduced two fundamental tasks for evaluating physical dynamics understanding in multimodal large language models: Next Frame Selection and Temporal Coherence Verifica- tion. Through comprehensive experiments, we revealed critical limitations in current MLLMs when it comes to understanding intuitive physics. To address this, we proposed Scene Dynamic Field (SDF). which effectively integrates knowledge from physical simulators into MLLMs. By abstract- ing physical representations into visual reasoning cues, SDF enables models to better perceive and understand physical dynamics. Our results demonstrate decent improvements across both bench- mark tasks, with the method also exhibiting strong transfer capabilities to unseen scenarios.

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under Grant No. U25A20442, Shanghai Municipal Science and Technology Major Project No. 2025SH2DZX025114, National Natural Science Foundation of China under Grant No. 62306175.
REFERENCES
Mohamad Ballout, Serwan Jassim, and Elsa M. Bruni Bruni. Pixels to principles: Probing intuitive physics understanding in multimodal language models. ArXiv, abs/2507.16572, 2025. URL
attps://apl. semant wrrpaanttccchhssss wwt/Corpus ID: 28:0155555
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. URL
08471
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-JEPA: Latent video prediction for visual representation learning. 2024. URL htttps://openreiee...coo
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learn-
ing. In International Conference on Machine Learning, 2009. URL https://api.
remanticschoo
Pegg/CorpusID::0
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Ci- hang Xie. Sft rl? an early investigation into training r1-like reasoning large vision-language models. ArXiv, abs/2504. 11468, 2025. URL ott. org/
CorpusID:27782428.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 2rXiv:2312.142.00 14238, 2023.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos, 2022. URL httptt/s: orq// org. abs/ / /20//22552222 01089
Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Bench- marking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411 2025.
Blender Online Community. Blender - a 3D modelling and rendering rendering Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URLpttpp://// hetttp://ww.bbr. http:///wwwbbeee.r...m
Flip Fluids.
Flip fluids: fluid simulation addon for blender. URL httts::/fflipfluit
com
Jensen Gao, Bidipta Sarkar, F. Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and
Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462-12469, 2023. URL
https: wappp.ssmatttccccceee.. ar. .r...r...oo..................... www.:orrggooouooeeee
Corpus ID: :D: 21665595599
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael G. Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos. ArXiv, ahs/2502.11831 2025. URL. https:// api semanticscho ar. waagg//oopustti : 276408877
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter N. Yianilos, Moritz Mueller-Freitag.
10

---

Published as a conference paper at ICLR 2026
Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something some- thing" video database for learning and evaluating visual common sense. 2017 IEEE Interna- tional Conference Computer Vision (ICCV), pp. 5843-5851, 2017. URL https://aag
semant 1 cscho ar. wwwp..rg//ooe Corpus ID: 834612
GLM-V Team Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng. Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang. Yan Wang. Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang. Boyan Shi, Changyu Pang. Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jia-Zhou Xu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang. Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingzhi Zhang, Qinkai Zheng, Shengchao Yang, Shilong Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqing Tu, Shengbiao Meng. Tianshu Zhang. Tian- Yuan Luo, Tianxiang Hao, Wenkai Li, Wei Jia, Xinpeng Lyu, Xuancheng Huang, Yanling Wang. Ya-Qi Xue, Yanfeng Wang, Yifan An, Yifan Du, Shi, Yiheng Huang. Yilin Niu, Yuan Wang. Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuxuan Zhang, Zhanxiao Du, Zhenyu Hou, Zhao Xue, Zhengxiao Du, Zihan Wang. Peng Zhang, De-Huan Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4.1v-thinde Towards versatile multimodal reasoning with scalable reinforcement learning. 2025. BtRLsttps:/// //api..///....// semant wesccolcccr.....ccc www.pooorrrr/ooooItDD:D 280049141
Serwan Jassim, Mario Holubar, Annika Richter, Comelius Wolff, Xenia Ohmer, and Elia Bruni Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. arXiv preprint arXiv:2311.0904 2024.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, and Hongsheng Li. Mme-cot: Bench- marking ofuin-of-thougle in large multimodal models for reasoning quality, robustness, and effi- ciency. 2025. URL https: / /ap.. semant icscholar. org/ oopp/ooopps
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. hRtp:tt:s////xxii wttp://wwwwwwwp/b/////baaa/ wrg. 26660/
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. URL ptttps://arxive
abs/1705. 06950]
Amirkhosro Kazemi, Isaac Josh Abecassis, and Amir A. Amini Pi-gnn: Physics-infommn graph neural network for super-resolution of 4d flow mri. 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2024. URL BtL.http::/ /tpp.ppp..../ www.ntmmattttio
:aat::::/ap..semnntccssstte www.smmmnniccchhlaa........ org/
CorpusID:27193493
Hilde Kuchne, Hueihan Jhuang, Estibaliz Garrote, Tomaso A. Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. 2011 International Conference on Computer Vision, pp. 2556-2563, 2011. URL
206769852
Twelve Labs. Twlv-i: Analysis and insights from holistic evaluation on video foundation models. 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.000 03326, 2024a.
Hong Li, Nanxi Li, Yuanjie Chen, Jianbin Zhu, Qinlu Guo, Cewu Lu, and Yong- Li. The labyrinth of links: Navigating the associative maze of multi-modas llms. ArXiv, abs/2410.01417, 2024b. URL https:/ httpp/aap..////// www.sssemannsshhaa
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi multi-modal video under- standing benchmark, 2024c. URR.nttss.//arxiv.orr////o

---

Published as a conference paper at ICLR 2026
Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3585-
3594, 2019.
Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, Zuoyu Qiu, Liang Xu, Yue Xu, Hao-Shu Fang, and Cewu Lu. Hake: A knowledge engine foundation for human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8494 8797-444--506, 2023a.
Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, Yuan Yao, Siqi Liu, and Cewu Lu. Beyond object recognition: new benchmark towards object concept learning. International Conference on Computer Vision (ICCV), 2023b.
Yong-Lu Li, Xiaoqian Wu, Xinpeng Liu, Yiming Dou, Yikun Ji, Junyi Zhang, Yixing Li, Jingru Tan, Xudong Lu, and Cewu Lu. From isolated islands pangea: Unifying semantic space for human action understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024d.
Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang. Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning. ArXiv, abs/2406. 1300, 2024e. URL
https: /tpps///apl www.seeaaniicccccc htcposss/poi..eeennncccccccoaai. wttp:ooooo/oooooCooooooo 2775559555
Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. 2025. URL
https://apt iesemmantcschlhaat
Cttpg//rrussaae ID: 282992334
Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, 2020. URL
https: / semant ics ics.hhlas wttp/oorpps:::22555552.332
Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Y. Zou, Xin Eric Wang. Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucina- tion in multimodal reasoning models. ArXiv, 2ttp/2555.2152... 2025. URL /nttps://api.
semanti iesaaniccccllar.. www.chool...r///ooopoooo.o wtg/Corpus 2D:
OpenGVLab Team. Internv12: Better than the best expanding performance boundaries of open- source multimodal models with the progressive scaling strategy, jul 2024. URL /htpps://
internvl. inteeewww..ggiaoo....ooo////00-0 -nt---nn----eeee-----------
Online blog post.
Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Cripp- vqa: Couterrfactuall about implicit physical properties via video question answering, 2022. URL htlphttss////xxi. org brg/abs/ 22 03779
Giulia Polverini and Bor Gregorcic. Multimodal large language models and physics visual tasks: comparative analysis of performance and costs. European Journal of Physics, 2025, URL
https://apt www.nticschoooooooooe.o.o www.aaarrrrgooooruuee Corpusrrpussppcoo 280295946
Aditya Sharma, Linh Nguyen, Ananya Gupta, Chengyu Wang, Chiamaka Adebayo, and Jakub Kowalski. Inducing causal world models in 1lms for zero-shot physical reason-
ing. ArXiv, 0bs/2507.11885. 2025. URL https: bttps::/api..stt /ttts://ppp..eemnntcccchho......
CorpusID:2803220
Qwen Team. Qwen2.5-vl, January 2025.
URL https:://pennmm.ithhuee hccbllg./////////
qwen2. 5-v17
Hsiao-Ya Yu Tung, Mingyu Ding, Zhenfang Chen, Daniel Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel LK Yamins, Judith E Fan, and Kevin A. Smith. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties, 2023. URL https:
Unity Technologies. Unity, 2023. URL httpp::/unity.coom
Game development platform.
12

---

Published as a conference paper at ICLR 2026
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Uttama Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models. In North American Chapter of the As- sociation for Computational Linguistics, 2024a. URL https://appp semant cscholar. org/CorpusTD::0 1D:755559222
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He. Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Pp- 14549 14560, June 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv: 12191, 2024b.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024c.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvide02: Scaling video foundation models for multimodal video understanding. arXiv e-prints, pp. arXiv-2403, 2024d.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang. Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXiv, abs/2406.01574 2024e. URL https:
sapi..enantticscoolarrr.
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C. Gee, and Yixin Nie. The role 2023.
of chain-of-thought in complex vasion-language reasoning task. ArXiv,
wttp : 265212798
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 15:13191,,,958, 2022. URL https: //tps:////pp..ee sem.ttiittttsccooo.......
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models, 2024. URL htt.:httts://rettt http:///wwww..ggaa////////////// wttg/bs////00/ /408 04840
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qu: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Zhao yu Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, and Yi R. Fung. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers. ArXiv, has/2506.23918 2025. URL https:://///. /aaps:///ap.... semantisccholarr org/ Corpus
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- treiriinnig. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
13

---

Published as a conference paper at ICLR 2026
Yong Zhang, Bingyuan Zhang, Zhitao Li, Ming Li, Ning Cheng. Minchuan Chen, Tao Wei, Jun Ma, Shaojun Wang, and Jing Xiao. Self-enhanced reasoning training: Activating latent reasoning in small models for enhanced reasoning distillation. In IEEE International Conference Acous- fics, Speech, and Signal Processing, 2025. URL htltttts:/// /tpppppppp..p.. semant inntiicssiinn wwwccccc....ooooo orrg
CorpusTD:: (orpusTD:27642172
Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung. Florian Schroff, Ming Yang, David Ross, Huisheng Wang. Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. Videoprism: A foundational visual encoder for video understanding. ArXiv, semanticscholantte
abs/2402. 13217, 2024a. 267760035
URL https:
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning, 2024b. URL utthttss: pecttss//arren /ttppis.///rvv.../o/ crg.e.rgggaaaa.. /2008. 05517
Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang. Qin Zhi Eddie Lim, Joshua Tenen- baum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. arXiv preprint arXiv:2402.06/ 19, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Cong He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Ying Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Lijun Wu, Kai Zhang, Hui Deng, Jiaye Ge, Kaiming Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. ArXiv, abs/2504.10779 10479, 2025. URL
https: // /app.. semant semant ar. otg/Corrps
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A STATEMENT OF LLM USAGE
In the preparation of this manuscript, large language models were used as editorial tools. The LLM assisted in the review of the text to identify potential issues with grammar, spelling, and punctuation. was also used to find out areas with potential logical inconsistencies or flaws the presented arguments. All suggestions were reviewed and approved by the human authors, who maintained full control over the final content and voice. The authors take responsibility for the paper's contents.
B ABLATION STUDY
Number of Input Frames. To investigate the impact of input frame count on model performance, we conducted experiments using Qwen2.5-V1 VI. and InternV1.2.5. Surprisingly. our results demon- strate that the number of input frames does not significantly affect the model's performance beyond certain threshold. As shown in Table 3 the performance metrics remain relatively stable across different frame counts, with only marginal improvements when increasing the number frames. Based on this finding, adopt input frames as default configuration for subsequent experi- ments, as it provides an optimal balance between computational efficiency and model performance.
Stride. The selection of temporal stride is pivotal hyperparameter in our low-level benchmark, directly modulates the temporal resolution of frame sampling. As empirically validated in Figure 5 extreme stride values yield suboptimal evaluation regimes. A stride of 8 creates artificially simplistic task, where models exploit trivial frame-wise similarity cues rather than genuine motion understanding. Conversely, strides exceeding 8 5 introduce excessive temporal sparsity, causing performance degradation that misrepresents a model's true low-level intuitive physics understanding capabilities due to information loss between sampled frames. To reconeile these competing objec- tives, we establish 5 {2.4} } as the optimal configuration for our benchmark. This intermediate range ensures sufficient temporal resolution to capture nuanced motion details while maintaining tractable fram- variation for robust dynamic understanding evaluation.
Number of Frames
3
4
VL
29.45
30.00
30.28
29.93
Intern
16.80
20.19
20.27
20.18
Table 3: Performance analysis of Qwen2.5-VL and Intern2.5VL across different frame counts, Re- sults show minimal variation in key metrics, indicating that the model maintains robust performance regardless of input frame quantity.
RECUNTT
Figure 5: Stride ablation study on the NFS benchmark performance for Qwen2.5-VL and In- tern VL.2.5.
Prompt Design
As shown in Table we designed five prompts with different lengths to ablate the performance on the NES task.
Results in Table 5 demonstrate that different prompts lead to minor performance variations, where prompt complexity exhibits non-linear relationship with model performance. Interestingly, Prompt
15

---

Published as a conference paper at ICLR 2026
Index
Prompt Text
Focus on physical dynamics.
Focusing on the physical dynamics within the sequence of video frames is crucial for under- standing and solving the problem presented be- low.
Please carefully pay attention the physical dynamics, including kinematic properties, dy- namic properties, mechanical system properties, fluid dynamics properties.
Understanding the kinematic properties, dy- namic properties, mechanical system properties, fluid dynamics properties in the video frame se- quence very helpful to tackle the problem be- low.
5
Comprehensively analyze the physical dynam- ics of the system, with detailed attention to:
1. Kinematic Properties: Displacement, ve- locity, acceleration, and their relationships for predicting motion within the system. 2. Dy- namic Properties: Forces, momentum, and en- ergy conservation. 3. Mcchanical System Prop- erties:
Friction, clasticity, and damping effects. 4. Fluid Dynamies Properties: Pressure, viscos- ity, Reynolds number, and flow characteristics. Additionally, ensure both theoretical principles (such governing equations) and practical ap- plications (e.g., motion prediction or flow be- havior) are covered.
Table The prompts used in the ablation study.
Prompt Index Qwen2.5-VI. Intern VL2.5
2
3
4
5
28.13
29.10
30.27
29.40
27.44
20.05
20.75
20.15
19.87
19.00
Table 5: NFS Task performance of Qwen2.5-VL and Intern2.5VL across different prompts. Prompt Index herein corresponds one-to-one with the index in the Table above.
3, which provides explicit guidance specific physical properties consider without overwhelm- ing detail, yields the best performance for Qwen2.5-VL. (30.27%) In contrast, more verbose prompts (e.g., Prompt 5) or overly simplistic ones (e.g., Prompt 1) show reduced effectiveness. This pattern suggests that MLLMs benefit from focused prompting that directs attention to relevant physical aspects without excessive elaboration. The consistent performance ranking across prompts between models indicates that prompt engineering alone cannot compensate for fundamental model limitations in intuitive physies understanding. This finding reinforces the need for our proposed SDF approach, which addresses these limitations at more fundamental representational level.
Expert and Self-distileed Data Mixture Ratio To better understand the optimal balance between expert data from Gemini-2.55PPp and self-distilled data, we conduct an ablation study examining model performance across different mixture ratios. We evaluate Qwen2-VL and GLM4.1V as our target models, performing full-parameter supervised fine-tuning with total of 3,000 data points. presetss the NFS accuracy under different data mixture ratios. We employ consistent training configurations across all experiments, training for 3 epochs with learning rate of 1 x 10 on A (40GB) GPUs. The results demonstrate that the optimal balance is achieved at the 10 ratio, excessive expert data interfere with the model's inherent reasoning processes, while relying
16

---

Published as a conference paper at ICLR 2026
Expert:Self Data Ratio
Model
1:0
1:1
1:5
1:10
1:15
0:1
(Expert)
(Self)
Qwen2-VL
32.91
38.10
39.10
41.18
39.20
31.00
GL
39.91
40.70
45,90
46.11
40.20
41.00
Table 6: NFS accuracy (%) across different expert-to-sele sel------se---isileedddeedd data mixture ratios.
solely self-distilled data fails adequately leverage the guidance provided by our proposed SDF framework. Thus, we adopt the 1 10 ratio as our default setting in all the main experiments.
Different Representations.
Last Frame
Depth Estimation
Optical Flow
SDF (ours)
Figure A demonstration different representations.
Firstly, we argue that reconstructing motion representations from videos such as optical flow is unnecessary for our setting. Our generated simulator provides precise velocity for each particle every time step. This velocity is a natural source for the Scene Dynamic Field and yields a clean and physically grounded visual prompt for training. In contrast, reconstruction from real world videos noisy due to occlusion, texture aliasing, camera compression and illumination changes, which weakens the learning signal for intuitive physics.
However, to ablate on the effectiveness of different visual representations, we still conducted an additional comparison across four prompt conditions using the same question answering protocol. To be precise, we randomly select subset of NFS and TCV tasks and generates its corresponding representations inserted the visual prompt into question as shown in Figure 6
w/o Visual Prompt: Direct QA without any visual prompt.
2. w/ Optical Flow: Adding the reconstructed optical flow Xu et al. QA.
2022 in the prompt for
3. w/ Depth: Adding depth estimation from Depth Anything Lin al. 2025 of the last frame in the prompt for QA.
4. w/ SDF (Ours): Adding our proposed Scene Dynamic Field as a visual prompt for QA.
These results in Table indicate that while optical flow provides some helpful motion cues for the Temporal Coherence Verification (TCV) task, is less effective on Next Frame Selection (NFS), where reconstruction noise likely limits its discriminative value. In contrast, SDF achieves the best
17

---

Published as a conference paper at ICLR 2026
Table Ablation study comparing the effectiveness of different visual prompts on NFS and TCV.
Visual Prompt
NFS Ace (%)
TCV Acc (%)
w/o Visual Prompt
29.1
57.5
w/ Optical Flow
27.0
65.1
w/ Depth
32.4
60.1
w/ SDF (Ours)
41.2
68.9
performance across both tasks, demonstrating that clean velocity data derived from the simulator yields more reliable and generalizable visual prompt.
C DATA PREPARATION PIPELINE
Generate key words
Retrive from web
LLMs
Videos from Web
InternVL
Clip
Filter
Refine
Qwen-VL
LLaVA-OneVision
5 seconds
Human annotators
Multi-modal LLMs
Existing Dataset
Figure 7: A demonstration of our data preparation pipeline.
We have curated our fluid video dataset by aggregating synthetic simulation data from established repositories such as Contphy Zheng et al. (2024) and PhysBench Chow et al. (2025), complemented by an extensive collection of weal-world scenario videos featuring liquid-related actions through web mining. During generate retrieval keywords. The keywords are mainly composed of two parts: actions (e.g.. "pour", "stir", "shake") and categories of liquids (e.g., "oil", "milk", "juice"), with each action paired with liquid category to form search query(e.g.. "pour oil"", "pour juice", "stir milk"). To ensure data quality, we segmented the collected video data into 5-second clips and employed
We have architected and deployed a data preparation pipeline, as shown in Figure 7
implementation, employ Large Language Models(LLMs)
ensemble of multimodal L.Mss, including Qwen- VL-Max Wang et al. 20244)), LLaVA-One Vision Li et al. (2024a), and Intern OpenGVVLa Team (2024), to filter those clips that do not contain liquid interactions. For data refinement, continue to develop filtering interface for human annotators, as shown in Figure g which further filters the videos with noticeable perspective shifts obvious changes in playback speed from the human perspective, ensuring that the content remains consistent and coherent. For potential ethical concerns, this user interface incorporates dedicated options for annotators report issues related to privacy or other ethical considerations.
D SCENE DYNAMIC FIELD SETTINGS
To improve generalization, we construct scenes that encompass a variety of liquid-related actions, systematically varying the physical properties of the liquid, the visual attributes of the containers, the
18

---

Published as a conference paper at ICLR 2026
Video Filtering Tool (Remaining: 2)
Ethics Report
Descrabe
Report
Reject
Pass
Guidelines:
There should be no noticeable perspective shifts.
obvious changes in playback speed.
content must remain consistent and coherent.
Figure 8: The human evaluation interface in data refinement is designed to filter low-quality data and flag potential ethical issues.
Background
View
Blank
Color
Figure 9: demonstration of various environment backgrounds, liquid color, viscosity, and camera views settings.
background environments, and the camera perspectives. During implementation, we selected three representative viscosity values within an appropriate range, corresponding to liquids with low vis- cosity (e.g., water), moderate viscosity (e.g., oil), and high viscosity (e.g., honey). Additionally, we introduced variations in liquid color to ensure diverse visual representation by altering the materials of the environments blank, indoor, and outdoor along with five distinct camera perspectives to enhance the robustness of model across varying physical contexts. This achieved by modifying the environment HDRs and adding cameras from different angles in Blender. A demonstration these different settings is shown in Figure 9
in FLIPMeshes. Furthermore, we incorporated three representative background
19

---

Published as a conference paper at ICLR 2026
E
CASE STUDY
Input
Given sequence of video frames [frame list]. Which one of the following four images is more likely to be the next frame? Your answer should be in the format of Conclusion: [lmage0 or Imagel or Image2 or Image3].
Frame
Frame
Imaged
Imagel
mage?
Image1
Qwen2-VL
Conclusion: Image
Qwen2-VL SDF (ours)
To determine which image most likely to be the next frame, let's analyze the sequence of actions in the frames:
Frame to Frame 4: Milk is poured with steady velocity (consistent flow rate) and a downward direction into the coffee cup. In Frame 4, the milk-pouring action still ongoing (the stream of milk is visible). Evaluate the options:
Image0: Shows the pouring action has stopped, with a stable liquid level. This does not match the ongoing action Frame 4.
Imagel: Shows the pouring action still ongoing, with the liquid level rising. Retains the steady pouring velocity and downward direction of milk into the cup, consistent with the continuous action in Frame 4.
Image2: Shows the pouring action has stopped, with stable liquid level. This does not match the ongoing action.
Image3: Shows the pouring action has stopped, with stable liquid level. This does not match the ongoing action.
Since the pouring action is still in progress in Frame 4, the next frame should continue this action, which aligns with Imagel.
Conclusion: Image1.
Figure 10: Case study Qwen2-VL VL for zero-shot and SDF setting.
20

---

Published as a conference paper at ICLR 2026
Input
Given sequence of video frames [frame list]. Which one of the following four images is more likely to be the next frame? Your answer should be in the format of Conclusion: [lmage0 or Imagel or Image2 or Image3].
Frame
Fremee
Frame
Frame
imagel
Imoge3
GLM4.1V CoT
To determine the next frame, analyze sequence frames (Frame 0-4) for patterns object movement and positioning. Compare these patterns with the four images (lmageO-Imagee). The brown shape, green line, pink/purple lines, and bottom shapes in Frame 4 show consistent trajectory that matches Image3's elements. Other images deviate from this progression.
Conclusion: Image3.
GLM4.1V + SDF (ours)
To determine which image is most likely to be the next frame, let's analyze the sequence of actions in the framesi Frame 0 to Frame 4: The sequence shows colored elements flowing with consistent velocity and direction. The fluid dynamics demonstrate laminar flow patterns with the colored material maintaining its trajectory and showing predictable behavior based fluid mechanics principles. The elements follow parabolic trajectory under the influence of gravity momentum conservation. In Frame the flow still in progress, with the maintaining its integrity and following the established path. Evaluate options:
fluid flow with same velocity direction established Frames 0-4, The fluid dynamics remain consistent with Bernoulli's principle and conservation momentum, showing the expected continuation of the laminar flow observed in the previous frames. The pattern of colored lines and blocks continues logically from Frame 4, maintaining the physical properties demonstrated throughout the sequence.
Image0: Shows the continuation
Image1: Shows disruption in the fluid flow, with turbulence change in direction that doesn't align with the laminar flow pattern established the previous frames. This would require external force not evident the sequence.
Image2: Shows the fluid flow stopped reached equilibriu, which would contradict the angoing
abserved Frame and violate conservation momentum.
Image3: Shows reversal or significant alteration in fluid dynamics wouldn't naturally occur based the physical principles demonstrated in the sequence, The trajectory and behavior of the colored elements change way that isn't supported by the physics established Frames 0-4,
Based on the principles of fluid dynamics, including conservation of momentum, Bernoulli's principle, and the continuity equation, the fluid motion observed in Frames 0-4 would mast logically continue in the manner shown in Image0. The velocity field and pressure distribution support this continuation without introducing external forces or violating physical laws. Conclusion: ImageO.
Figure 11: Case study on GLM4. IV for CoT and SDF setting.
As demonstrated in Figure 10 our proposed SDF method successfully enhances the Qwen2-Ve VL model's ability reason about the spatial and physical dynamics of continuum objects. The method enables the model to generate physically grounded interpretations, such as recognizing that milk maintains steady pouring velocity and follows downward trajectory into the cup. This exem-
21

---

Published as a conference paper at ICLR 2026
plifies how SDF effectively inject physical visual perception and reasoning capabilities into the model. Furthermore, Figure II illustrates a more challenging scenario where our SDF-trained G tackles a complex problem from Contphy Zheng al. (024) involving multiple simulta- neous fluid flows. Our trained model demonstrates the ability to compare and reason about velocity and momentum characteristics arrive the correct answer. When directly prompt the model to think step-by-step, the CoT may be influenced by superficial visual features such as color or appearance rather than relying purely on deeper physical knowledge.
F IMPLEMENTATION DETAILS
F.1 COT PROMPTS
Chain-of- Thooghh prompting can enhance the reasoning capabilities of Multi-modal Large Lan- guage Models (MILMs) in tasks such Visual Question Answering (VQA) et al. (2023). By explicitly guiding the model through a series of intermediate reasoning steps, CoT enables a deeper understanding and more accurate responses, particularly when complex visual or multimodal infor- mation is involved. To demonstrate this, we have designed distinct CoT prompts tailored to both the NFS (Next Frame Selection) and TCV (Temporal Coherence Verification) tasks, optimizing the models' ability to reason through intricate queries and generate more informed answers.
CoT prompt for NFS task:
Let's ' think step by step. To determine which of the four images is most likely be the next frame in the sequence, following chain of thought: Sequence Analysis: Look at the sequence of video frames from Frame-O Frame-4. Assess the general movement, objects, or changes between these frames. Identify any objects or actions that are progressively changing. such as motion, appearance, or position. Motion and Trends: Identify patterns or trends within the sequence: whether any objects are moving in a consistent direction or interacting with other objects. This can help to predict the direction or type of changes to expect in the next frame. Consistency with Previous Frames: Compare the four candidate images (Image0, Imagel, Image2, Image3) with the sequence. Which one follows the motion, appearance, or transitions observed the previous frames? Does one of the candidate images exhibit the next logical state movement? Discarding Outliers: If any of the candidate images significantly diverges from the observed trends in the sequence (e.g., in terms of position, motion, object changes), it can be ruled as less likely. Final Prediction: Based on the trends and the consistency of the images with the observed dynamics, choose the most plausible candidate image. Now, apply these steps to the sequence of frames and the candidate images, and determine which one most likely to be the next frame.
approach the task by analyzing the sequence of frames and considering the
Cloth
Sand
Smoke
Plasticine
The demo of the data we collected and used in transfer experiments. In each field of the image, the upper and lower frames are sampled in the same 3-second clip, and the lower frame is later than the upper frame.
Figure 12:
22

---

Published as a conference paper at ICLR 2026
CoT prompt for TCV task:
Let's s think step by step. Inspect the video frames one by one for obvious irregularities; compare object movement and appearance for consistency across frames; check for any unusual transitions, lighting, or distortions; assess background and environmental consistency; if any frame diverges in way that violates natural progression, answer "yes", otherwise answer "no".
Annotation prompt for self-annotating process:
You are an expert in visual reasoning. Your task is generale a step-by-step "thought process" that logically explains how to arrive at the correct answer for a given question. You will be provided with the question and its ground truth answer. Your output should only be the reasoning steps. Do not repeat the question; just generate the thinking process. [START OF TASKJ 1. QUESTION: {question) 2. GROUND TRUTH ANSWER: {gr choice) You should end with "Conclusion: {gr choice)".
Annotation prompt for 2emini-2.5-pro:
You are expert in visual reasoning. Your task is to generate step-by-step "thought process" that logically explains how to arrive the correct answer for a given question. You will provided with the question and its ground truth answer. Your oulput should only be the reasoning steps. Do not repeat the question; just generate the thinking process. [START OF TASK] QUESTION: Given sequence video frames: [Video Frames] Scene Dynamic Field (SDF) visualizes the flow of motion by assigning colors to different velocity magnitudes, where regions with higher velocities represented in deeper blue shades. Which of the following the SDF of the last given frame? ? [Choices] Your response should be one of the following: A, B, C, D. GROUND TRUTH ANSWER: {gl choice)
G FAILURE CASES ANALYSIS
QwenZ-VL failure
Owen2-VL success
0.5
0.5
6.4
0.4
Aotention
Actention
0.1
D.1
0.0
0.0
Problem
Mideo Frimes
Candidate
Outzut
Problem
Miee fromes
Candidate
Outzut
QwerZ-VL SDF failure
Qwen2-VL SDF success
0.5
0.5
Attention
Attention
0.2
0.1
0.1
Problem
Mideo fromes
Condidate
Outout
Prblemm
Mideo fromes
Cansidate
Outzut
0.4 Atterblon weight (deeper higher)
Attention weight visualization for Qwen2 VL on the NFS task. Left panels show suc- cessful cases, right panels show failed cases. Top row: zero-shot setting; bottom row: our proposed SDF setting. Darker colors indicate higher attention weights.
Figure 13:
Different from high-level physical reasoning tasks, which can be further disentangled into several sub-steps, fundamental physical understanding tasks represent low-level perception capabilities that are often harder to probe. Therefore, we quantify the attention weights for model inputs without explicitly requiring output reasoning processes. As found by previous work Li et al. (2024b); Wang let al.] (2024a), MLLMs are weak at long-context reasoning, especially when the input consists of multiple images. This similar phenomenon is also observed in experiments.
23

---

Published as a conference paper at ICLR 2026
As shown in Figure analyze the model's 's focus distribution. The attention weights the output instruction and candidate frames part significantly higher than those of the previous video frames in the zero-shot setting. This indicates that the model tends to focus on the most recent frames, potentially overlooking important temporal dynamics present in earlier frames. This bias towards recent information lead to suboptimal performance in tasks that require a comprehensive understanding of the entire sequence.
1
average the attention weights across all output tokens and all layers to
We can observe higher attention weights on earlier video frames in successful samples compared failed ones. The failed cases often focus more on choices and instruction parts, failing attend to the equally important video frames. This suggests that the model may be overly influenced by the provided choices and instructions, rather than fully engaging with the visual content where con- tinuum dynamics (e.g.. fluid motion) are most apparent. Importantly, our proposed SDF method demonstrates a clear improvement in attention distribution.
illustrated in the bottom row of Fig- ure 13 SDF, to some extent, draws more attention to the equally important video frames where continuum phenomena such as fluid dynamics can be observed. This enhanced attention allocation enables the model to better capture the temporal evolution of physical processes and make more informed predictions, ultimately leading to improved performance in selecting the correct choice.
Examining the explicit reasoning processes reveals additional insights into failure mechanisms. We observe that visual hallucination becomes more prevalent longer reasoning sequences, consistent with findings from prior hallucination research Liu al. (2225). Our analysis shows that extended reasoning leads to approximately 31 % reduced attention allocation to visual tokens, thereby dimin- ishing perceptual capabilities. This attention degradation particularly pronounced in later stages of the reasoning process, where models increasingly rely on internally generated content rather than grounding their responses in the provided visual information.
In all, these failure cases show the fundamental challenges MLLMs encounter when integrating and reasoning over temporal visual sequences. The observed patterns highlight the critical need for enhanced attention mechanisms and reasoning frameworks that can more effectively leverage the complete contextual information within visual sequences, particularly for tasks involving complex physical dynamics. These findings suggest that future research should pay more attention to de- veloping more supervision mechanisms for visual perception capabilities throughout the reasoning process. Additionally, incorporating physically grounded training data will be essential for cultivat- ing more physics-aw aware MLLMs Polverini & Gregorcic (2025), ultimately enabling their broader application across diverse physical reasoning domains Gao al. (2023)
H ENCODER EVALUATION
80
(xed(x)
TCV Accuracy (%)
60
Similarity
Tested Video Encoder
40
ViT
ViT
Random
20
Qwen2.5-V
Stride
Stride
Frame
Frame t+++1
Distractors
0
InternVideo2
VideoMAEV2
V-JPEA
VIT
t+1, "",
(a)
(b)
Figure 14: (a) Architectural illustration of the V-JEPA-like encoder test. (b) Performance evaluation of various video encoders on TCV tasks.
24

---

Published as a conference paper at ICLR 2026
Given the relatively poor performance of current foundation models physical dynamics under- standing, a natural question arises: can we obtain better performance by effectively leveraging the encoders? To investigate this possibility, we designed experiment inspired by the V-JEPA Bardes iet al. 2024) architecture to directly test the representational power of various encoders.
We employ three-layer Transformer- baassorrmer-bsedd prediction network (Pred(x)) that takes input the rep- resentation generated by frozen video encoder. This encoder processes a sequence of consecutive video frames specifically, frames [+n produce compact, contextualized embedding.
The output of the prediction network is then compared via similarity function (e.g., cosine simi- larity) against fixed reference representation derived from a pre-trained Vision Transformer (VīT). This reference ViT encodes a single target frame (frame (+++l) representing the immediate future frame
Additionally, the same similarity metric is computed between the predicted representation and em- beddings of "distractor" frames (also encoded by the same frozen ViT), which serve as negative examples during training. This contrastive setup encourages the prediction network to learn repre- sentations that are semantically aligned with the true future frame while being distinct from irrele- vant distractors.
The entire system trained end-to-end to maximize the similarity between predicted and target representations, while minimizing similarity with distractors effectively learning to anticipate visual content a self-supervised manner.
Upon evaluation on simple subset of TCV tasks in Figure 14 it was observed that directly lever- aging the visual representation alone was insufficient for capturing the complexities of dynamic behavior, evidenced by significant performance gap compared to the Qwen2.5-VL model. It motivates further exploration into harnessing the inherent capabilities of MLLMs to enhance the representational power needed for intuitive physics understanding.
This finding led us to focus on the data-centric perspective. The recent improvements in models like Qwen3-VL, which was trained more embodied and spatial task data, lend strong support to this view. It's performance improved on our intuitive physical understanding tasks with the help of these spatial and physical data. It indicates that the underlying architectures of MLLMs are capable of learning physical reasoning, but they must be exposed to the right kind of data. We believe that intuitive physical understanding is foundational capability. Therefore, attempting to learn implicitly from high level, end to end task data (such as embodied action) would be akin to putting the cart before the horse. A more effective strategy is to first equip MILMS with this fundamental capability using targeted data.
Therefore, our work adopts data centric approach precisely address this gap. By generating structured data focused on physical dynamics (SDF), we aim to directly enhance the MLLM's ca- pacity for intuitive physical reasoning, providing solid foundation for wide range of real world applications.
I TRANSFER EXPERIMENT DETAILS
Domain Cloth Sand Smoke Plasticine
Object
Action fluttering pouring, falling ejecting, spreading pouring, cutting, stretching
cloth, skirt, shirt, ribbon sand, grain, lime, seeds steam, smoke plasticine, kinetic sand
Table 8: The data composition for transfer experiments.
During experiments within the liquid domain, we found that MLLMs can accurately predict the deformation of liquids by capturing the viscosity of fluid particles, cohesive forces, and the laws of momentum transfer. This suggests that MLLMs may have developed the capability to model the "dynamics of continuous objects formed by particle aggregates." " Fluids share deep-seated sim- ilarities in physical essence with materials such as cloth, sand, smoke, and plasticine. Traditional
25

---

Published as a conference paper at ICLR 2026
methods require customized physics engines to model these materials, whereas the generalization ability of MLLM may break through domain barriers and achieve unified representation. Therefore, adopted the same paradigm (shown in Section C used for liquid dataset collection, shown in Figure T2 to gather data from four domains: cloth, sand, smoke, and plasticine. Table g shows the composition of the data.
We acknowledge that our study primarily centers on fluid dynamics as a representative case for continuum physics. And our exploratory transfer experiments show promising generalization to other particle-based materials. We do not exhaust the full spectrum of intuitive physics, which also includes principles governing rigid body collisions, thermodynamics. or optics, since fluid has always been proposed to be a difficult ZZeng et al. (2024) but useful material to start with (e.g,, robotics manipulation tasks Lin et al.] (2020)). Future work should develop more comprehensive benchmarks that span wider array of physical phenomena to fully assess and cultivate a holistic physical understanding in MLLMs.
26


Reviews:
[{"review_id": "El4N02cPEa", "reviewer": "Reviewer_tuv4", "summary": "This paper try to addresses the gap in MLLMs understanding of intuitive physics, particularly for continuum objects like fluids. The authors first introduce two low-level benchmark tasks, next frame selection and temporal coherence verification, to demonstrate that current MLLMs perform poorly at perceiving physical dynamics. To solve this, they propose SDF, an intermediate representation generated by physics simulators that visually encodes motion (e.g., velocity as color intensity). Through a multi-task fine-tuning strategy, their SDF-enhanced model achieves substantial gains on fluid tasks and shows strong generalization to unseen physical domains like cloth and sand.", "questions": "See in the weekness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the paper correctly focuses on a simpler, core problem, which is just perceiving physical motion, separating it from complex, high-level reasoning.\n\n- Using a visual map (SDF) from a physics simulator to train the model works. This helps the model learn the idea of dynamics, letting it generalize from fluids to unseen materials like cloth or sand.\n\n- The authors built a strong benchmark by mixing simulated data with real-world videos, and they ran thorough experiments to validate their approach.\n\n- The paper is well presented with clear logic.", "weaknesses": "- The SDF representation is very basic, encoding just the projected velocity magnitude into one color channel. It's questionable if this simple map captures enough information for complex interactions, or if it needs to include more data like 3D vector direction or using optical flow (which avoids physics simulation). A discussion comparing the performance, advantages, and disadvantages of these different representations would greatly strengthen the analysis.\n\n- The pipeline is complex. It requires a full-parameter fine-tuning process, data from simulators, and knowledge distilled from expert models, which is computationally expensive and hard to scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7lO4v3XW1q", "reviewer": "Reviewer_deha", "summary": "The paper focuses on the task of fluid and physical scene understanding. It makes two primary contributions:\n\n1. The authors build a comprehensive benchmark for physical reasoning from videos, introducing two key tasks — Next Frame Selection (NFS) and Temporal Coherence Verification (TCV). The dataset is constructed from multiple sources, including ContPhy, PhysBench, and real-world video clips, to cover both synthetic and natural dynamics.\n\n2. The paper proposes the Scene Dynamic Field, a framework that leverages physical simulation principles and integrates Chain-of-Thought (CoT) reasoning with explicit physics modeling. This approach enables multimodal models to reason about scene dynamics beyond static appearance, bridging the gap between perception and physical understanding.", "questions": "1.  Data Modality and Encoder Usage:\nWhat is the exact data format of the SDF samples used for training? From Figure 3, the SDF appears to be an RGB-like image generated through velocity-to-color mapping. Could the authors clarify whether these SDF images are indeed three-channel RGB inputs, and if they are processed by the same image encoder as the regular video frames during training?\n\n2. Ablation on the SDF Step in CoT Fine-Tuning. In the SDF-guided Chain-of-Thought fine-tuning process, the framework first predicts the SDF representation and then predicts the next frame based on that SDF. What would happen if we retain the same training pipeline but skip the explicit SDF step, i.e., directly fine-tune the model with simplified CoT instructions to predict the RGB frame without generating the SDF?\nThis comparison would help clarify how much the explicit SDF stage contributes to physical reasoning, especially since the SDF image appears visually similar to the original frame except for color-coded fluid regions (as shown in Figure 3). Such an ablation could reveal whether the SDF acts mainly as a visual prompt or as a truly distinct physical representation.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important and challenging problem — physical and fluid scene understanding — which is a key step toward enabling models to reason about real-world dynamics.\n\nThe proposed explicit physical reasoning task is interesting and well-motivated, encouraging models to go beyond visual perception and engage in physically grounded understanding.\n\nThe paper is generally well-written and structured, with a clear motivation and logical flow from problem definition to methodology and experiments.", "weaknesses": "1. Limited physical understanding of only fluid dynamics. The proposed framework primarily focuses on liquid and fluid dynamics, which narrows the generality of the method. Extending the benchmark and tasks to cover a broader range of physical interactions — such as rigid-body motion, collision, or elastic deformation — would strengthen the overall contribution.\n\n2. Few missing references to physical reasoning VQA. The paper could benefit from citing and discussing more recent physical understanding and VQA-related research, such as Comphy (https://arxiv.org/abs/2205.01089) and DynSuperCLEVR (https://arxiv.org/abs/2406.00622), which address the VQA for physical reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jqfHq2fY4", "reviewer": "Reviewer_Vi2b", "summary": "The paper proposes two low level tests for intuitive physics in MLLMs, Next Frame Selection and Temporal Coherence Verification, focused mainly on fluids. It introduces Scene Dynamic Field, a simulator derived motion visualization used as a visual prompt within a multi task fine tuning scheme, and reports sizable gains on the proposed tests with some transfer to cloth, sand, smoke, and plasticine.", "questions": "- How does SDF compare to simple optical flow overlays, grayscale motion magnitude, or event frame stacks when training with the same budget?\n\n- Are results robust when distractors are generated with a feature space disjoint from any model under test?\n\n- Can you evaluate on external physics benchmarks without re curating the data to validate generality?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Clear problem decomposition toward low level dynamics rather than high level QA\n- Simple intermediate representation that is easy to plug into existing MLLMs\n- Ablations on stride, prompts, model scale, and expert vs self distilled data\n- Some transfer beyond fluids and an attention analysis that supports the claim that SDF shifts focus to earlier frames", "weaknesses": "- The weakest point is that there is no comparison to strong motion baselines. SDF is a velocity magnitude style visual prompt, which is conceptually close to optical flow magnitude, flow stacks, dynamic images, or even simple frame differencing. Without head to head baselines under the same training data and budget, the gains could come from adding any explicit motion cue rather than from SDF itself. This leaves the central claim unproven.\n\n- Benchmarks are author designed and multiple choice, so improvements may reflect distractor design rather than genuine physics understanding\n\n- Absolute accuracy remains low, so practical impact is unclear\n\n- Limited evaluation beyond fluids for rigid body scenes or causal reasoning tasks, so the title and claims feel broader than what is shown\n- Distractor pruning uses SigLIP embeddings which are related to encoders used by evaluated models, creating a risk of bias\n\nMy recommendation is reject. The idea is interesting and the empirical gains are clear on the authors benchmark, but the evaluation misses strong motion baselines, relies on potentially biased distractor construction, and the absolute performance and scope do not yet support the paper’s broad claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLs3U9qGC9", "reviewer": "Reviewer_HQBD", "summary": "This paper addresses a critical limitation in MLLMs: their poor understanding of intuitive physics, particularly for continuum objects like fluids. The authors introduce two diagnostic tasks—Next Frame Selection (NFS) and Temporal Coherence Verification (TCV)—to systematically evaluate low-level physical perception. They reveal that even state-of-the-art MLLMs perform poorly on these tasks, often near random baselines. To bridge this gap, the authors propose Scene Dynamic Field (SDF), a method that leverages physics simulators to generate visual prompts representing motion dynamics. Through multi-task fine-tuning, SDF significantly improves model performance and generalizes well to unseen physical domains like cloth and smoke.", "questions": "See weaknesses. I am willing to discuss with the authors", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is interesting for addressing the problem of poor low-level physical perception by proposing practical physics simulators for visual prompting to improve performance.\n\n2. The paper is well-organized and accessible, with clear explanations of both the problem and the solution.", "weaknesses": "1. Limited Scope of Transfer Experiments: While the transfer experiments across continuum domains (cloth, sand, smoke, plasticine) are promising, the scope remains limited. The paper would be significantly strengthened by evaluating the method's generalization to other fundamental physical phenomena, such as rigid-body dynamics, collisions, or optical effects. \n\n2. Experimental Design and Baselines: To provide a more comprehensive performance comparison, we suggest expanding Table 1 to include additional baseline models. Specifically, it would be informative to compare against larger models with more parameters.\n\n3. Dependence on Synthetic Data and Real-World Applicability: The SDF method's reliance on synthetic data from simulators is a potential limitation, as such data may not fully capture the complexity and noise of real-world physical systems. The paper could be improved by discussing the feasibility and potential challenges of applying this method to real-world physics problems. For instance, how would the SDF approach perform with real sensor data that is often incomplete or noisy?\n\n4. Analysis of MLLM Limitations: The paper identifies that MLLMs struggle with low-level dynamics but does not deeply investigate the root cause. A more thorough analysis is needed to determine whether these failures stem from architectural limitations (e.g., an inductive bias towards high-level semantics) or a bias in the pre-training data (e.g., a lack of low-level physical reasoning examples). Uncovering this would provide valuable insight for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)