Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos
Authors: Meng Cao
Haoran Tang
Haoze Zhao
Mingfei Han
Ruyang Liu
Qiang Sun
Xiaojun Chang
Ian Reid
Xiaodan Liang
Venue: Accepted by TMLR ()

Abstract: Understanding the physical world, including object dynamics, material properties, and causal interactions, remains a core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general reasoning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. In this paper, we propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding. We introduce PhysGame, an instruction-tuning dataset containing 140,057 glitch-centric question–answer pairs across five physical domains and sixteen fine-grained categories. To ensure data accuracy, we design a meta-information–guided prompting strategy that utilizes gameplay metadata such as titles and descriptions to guide high-quality QA generation. Complementing PhysGame, we construct GameBench, an expert-annotated benchmark with 880 glitch-identified gameplay videos designed to evaluate physical reasoning capabilities. Extensive experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real-world physical reasoning performance of Qwen2.5-VL by 2.5% on PhysBench, and Game2General transferability, yielding a 1.9% gain on the MVBench benchmark. Moreover, PhysGame-tuned models achieve a 3.7% absolute improvement on GameBench, demonstrating enhanced robustness in detecting physical implausibilities. These results indicate that learning from gameplay anomalies offers a scalable and effective pathway toward advancing physical world understanding in multimodal intelligence.

Published in Transactions on Machine Learning Research (January /2026
Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos
Meng Cao Haoran Tang?; Haoze Zhao'; Mingfei Han', Ruyang Liu?, Qiang Sun Xiaojun Chang', Ian Reid", Xiaodan Liangl.5t
'Mohamed bin Zayed University of Artificial Intelligence PPeking University
University of Waterloo *University of Toronto SSun Yat-sen University
Authors contributed equally to this rescarch. 'Corresponding author.
Reviewed on OpenReview:
Gameplay Videos
Real-wortd Physical Understanding
Game2Real Transferability
PhysBench
Question: What violates physical commonsense this video?
Question: What optical processes are involved in the formation of the phenomenon in the video?
Qwen2-VL-778
M/ PhyoGame
General Video Understanding
Game2General Transferability
MVBBech
Video-MME
S6.1
642
Question: How did the photographer who took pictures of the three tourists in the video take the photo?
Qwen22VL-78
PhysGame
Figure 1: We propose PhysGame, an instruction-tumi dataset constructed from gameplay videos with Suitch-induces question-answer pairs. Our method demonstrates consistent performance improvements on downstream benchmarks for both real-world physical understanding (Game2Real) and general vidco under- standing (Game2General) tasks.
Abstract
Understanding the physical world, including object dynamics, material properties, and causal interactions, remains core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general rea- soning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. this paper, we propose a novel paradigm that leverages glitches game- play videos, referring to visual anomalics that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding, We introduce PhysGame, an instruction-tuni tonniniionntuinggggg. dataset containing 140,057 glitch-centric question centiion-nnseer pairs across five physical domains and sixteen fine-gra fne-grained categories. To ensure data accuracy, we de- prompting strategy that utilizes gameplay metadata such as litles and descriptions to guide high-quality QA generation. Complementing PhysGane, we gameplay
sign mnta--nformattin-nnnn
construct GameBench, an expert-annotate benchmark with 880 vidcos designed to evaluate physical reasoning capabilitics. Extensive experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real-world physical reasoning performance of Qwen2.5- 5.55-VV 5-VT. by 2.5% PhysBench, and Game2General

---

Published in Transactions on Machine Learning Research (January (Jamary/2026)
Question: is there any notable glitch in the video that affects the gameplay? Answer: While navigating the course, the car door oddly hanging open, which does seem affect its speed or performance. This could considered a minor glitch, such damage should typically impact aerodynamics handling realistic scenario. There major collision effect noticeable when cars collide, suggesting physics might fully replicate real-world damage
Question:
How would you describe the glitch in the video?
Option:
A: Blocked tree, the car's speed instantly surged from zero to over 200 mph. While the player's
the bridge, casts shadow.
left lane the tree-lined road, there are flickering red light patches. The clouds in the sky exhibit an unusual green
Figure 2: Sampled examples in the proposed PhysGame instructing dataset (left) and the GameBench evaluation benchmark (right).
transferability, yielding 1.9% gain on the MVBench benchmark. Moreover, PhysGame- tuned models achieve 3.7% absolute improvement GameBench, demonstrating enhanced robustness in detecting physical implausibilities. These results indicate that learning from gameplay anomalics offers scalable and effective pathway toward advancing physical world understanding in multimodal intelligence.
1 Introduction
"In all chaos there is cosmos, in all disorder secret order." Carl Jung, Archetypes and the Collective Unconscious
Physical world understanding McCloskey et al. (1983): Mclnik et al. (2024); Duan et al. 2022 represents a fundamental challenge in artificial intelligene, encompassing the understanding of physical properties, object interactions, and temporal dynamics. Even before language acquisition, children begin to grasp fundamental principles of the physical world by observing the environment around them Hespos Spelke (2004b. While recent multi-modal large language models (MLLMs) AAchiam et al. (2023): ; Reid et al. (2024) Brown al. g Touvron al. (2023) have achieved remarkable advancements across various domains,
their capabilities in physical world understanding remain substantially inferior to human-level intelligence
Wang et al. (2024c); Zheng et al. (2024); Chow et. al.] (2025); Zhao et al. (2025); Xiang et. al. (2025).
For example, on the physical understanding benchmark PhysBBnch [Chow et al. (025)) the state-of-the-att MLLM GPT-4o OpenAT (2024) achieves merely 49.49% average accuracy, falling far short of the human-level
performance (95.87%).
The majority of existing physical world understanding datasets can be categorized into two types: 1) Real- world video data [Cho et al. (2025) Zhao et al. (2025); Xiang et al. (2025): While capable of capturing complex backgrounds and heterogeneous compositional elements for physical reasoning, such data typically requires high annotation costs involving careful curation of specific video types and time-consuming question- answering (QA) annotation processes; 2) mimulator-generated synthetic data Yi et al. (2020): ; Zheng et al.
(2024): Chen et al.] (2022): : Bear et al. (2021)) These datasets employ physical simulators (e.g.. Blender Conlan (2077) and Unity Haas] (2014)) to create controllable scenarios for physical reasoning. However, they often exhibit uniform backgrounds and simplistic visual primitives (c.g., cube, cylinder), lcading to a substantial domain gap between synthetic represenlations and complex real-world physical scenarios Wang et al. (2024c). Based on the preceding analysis, contend that neither data type is well-suited as source for generating large-scale instruction-tuning datasets aimed at physical world understanding.
In this paper, we propose an alternative perspective by learning from glitches in gameplay videos Chen el al. (2021]; ; Ling et al. (2020) Nantes et al. (2008), i.e., inconsistencies between pre-defined physical rules and observed visual manifestations caused by rendering errors, physics engine limitations, or unexpected object interactions. For example, Figure Cauureeillsttaaa illustratee an implausible scenario in which a motorcycle unrealistically propels into the air, violating basic physical laws. Our motivation is inspired by the philosophical
2

---

Published in Transactions on Machine Learning Research (annuary/2200)
notion of Order from Chaos, which emphasizes that understanding often emerges more clearly by identifying and characterizing deviations from expected regularities. In other words, explicitly modeling violations of physical rules enables a sharper grasp of the principles that normally govern the physical world. To this end, we introduce PhysGame, an instruction-tumingg datasct constructed by harvesting gameplay videos with glitches to advance physical world understanding. PhysGame contains 140,057 question-anie ansserr pairs targeting glitch-related content across five key physical domains (f.e., mechanics, optics, material properties, Meermodynamics and electromagnetism) and sixteen fine-grained categories (e.g., gravity and velocity). To prompting strategy, which incorporates video-associated metadata (e.g., content-summariiing video titles) when prompting lead- ing MLLMs such as GPT-4o OpenAl (2024). Furthermore, we introduce GameBench, complementary evaluation benchmark comprising 880 glitch-identifie gameplay videos, each accompanied by expert-curated multiple-choice questions that explicitly probe glitch characteristics. The sampled examples of PhysGame instruction dataset and GaneBench evaluation benchmark are presented in Figure X Compared to existing approaches, our proposed PhysGame captures dynamic and diverse scenes that resemble real-world physical complexity, while remaining highly scalable due to the abundance of online data and reduced annotation cost through glitch identification. In addition, incorporating previously unseen glitches from the PhysGame dataset during post-training further enhances generalization and remedies the deficiencies of current MLLMs in glitch identification within gameplay videos. For example, built upon Qwen2-VT. Wang et al. (2024a) ) post-training with the PhysGame dataset yields an absolute improvement of 6.3% in average accuracy on our constructed GameBench benchmark (ef. Table 3).
ensure the accuracy of the generated QA pairs, propose a meta-inforrmaiio-gauinn
In terms of the aforementioned advantages, demonstrate that. PhysGame consistently enhances down- stream benchmark performance across both physical understanding and general video understanding tasks. Specifically, two transferability characteristics of PhysGame are highlighted: 1) Game2Real Transferabil- ity: Despite being constructed from simulated gameplay videos, PhysGame finetuning significantly improves real-world physical reasoning. As shown in Figure 1 Qwen2-VL Bai et al. (2025) achieves 2.5% absolute im- provement (46.6% us. 49.1%) on PhysBench benchmark Chow et al. (2025) after finetuning on the proposed PhysGame dataset; 2) Game2General:: Physical knowledge acquired from gameplay videos can generalize to broader and general-purpose video understanding tasks. For example, the PlysGame--nhaaned Qwen2.5- VL Bai et al. (2025) shows 1.3% performance gain (58.0% us. 59.3%) Video-MME Fu et al. (2024) benchnarkk This dual transferability paradigm demonstrates that gameplay-derived training bridges both comulation-to-ralliin and physical-to-genatn general understanding, offering new pathways for scalable and effective multi-modal intelligence development.
In sunmaary, our contributions are in three-folds:
We pioneer the use of gameplay videos as a novel training medium for physical world understanding, which overcomes the simplified settings in synthetic data while avoiding the high annotation costs of real-world video curation.
We introduce PhysGame and GameBench, establishing a suite of scalable datasets for physics-oriented instruction tuning and evaluation, respectively.
Experimental results validate the Game2Real and Game2General transferabilities of the proposed Phys- Game datasct, bridging the simulated physics data and real-world physical understanding tasks as well as general video understanding lasks.
2 Related Work
Physical World Understanding. Recent advances in physical understanding have driven interdisciplinary
research spanning psychology Bobrow (1984): Hespos et al. (2016); McCloskey et al. (1983). language rea- soning Bisk et al. (2020): Forbes et al. (2019). visual rcasoning Lerer et al. (0116); Yi ct al. (2020). vidco
generation Meng el al. 2024); Bansal et al. (2024), robotics Agrawal el al. (0016) Byravan el al. (0117).
etc. The primary datasets for physical understanding Battaglia et al. (2016) : Chang et al. (2016) employ symbolic representation by encoding physical knowledge through textual descriptors (c.g., object attributes
and interaction rules). However, such representations fundamentally lack the capacity to capture complex
visual in information, resulting in compromised perception. The recent datasets can be categorized into two
3

---

Published in Transactions on Machine Learning Research (January /2026
Table 1: (a) Comparisons of physical understanding datasets; (b) Comparisons with existing gameplay video benchmarks in terms of whether they are basod, whether they follow an instruction- following format, and support multi-modal evaluations.
Photo- realistic
Scalable to Colloct
Instruct Tuning
Video- Based
Instruct
Multi- modality
Dataset
Dataset
Tuning
Phys-ennh Taesrri Besemer 2024
X
GameBunny
Tocstri Beaereer 2024
MMVU ef Lisetzi
Tamiri et.ol
Taesiri
CLEVER Ticeiri
126
GameBugDerrap
Tuesir
b022b
v
Comply
Cheu el 10
GlitchBeuch
Taesin el
202
V
PhysGame (Ours)
V
PhysGame (Ours)
(a)
(b)
principal types: 1) Real-world observational datasets capturing natural physical phenomena Chow et al. (2025); Zhao et al. 2025). This kind of dataset requires labor-intensive video curation, inherently limiting scalability in physical scenario coverage; 2) Physics-simulated datasets generated through differentiable en- gines Yi et al. 2020 Zheng el al. (2024): Chen et. al. (2022); Bear el al. (202)). While offering controllable
physical environments, they are constrained to simplified geometric primitives (e.g., rigid cuboids/ cuboids/spheres /sphere)) and homogencous backgrounds, leading to significant sim-to-real discrepancy visual perception. Build- ing on this analysis, we propose a novel approach to facilitate physical world understanding by analyzing physics glitches in gameplay videos. This method establishes scalable framework with closer alignment. to real-world physical scenarios, effectively bypassing the limitations of existing datasets.
Gameplay Video Understanding. Digital games Hu et al. (024a) Xu et al. (2024b) are considered pivotal in pursuing artificial general intelligence, as they act as controllable real-world simulators and create complex problem-solving contexts. Therefore, gameplay videos are typically employed as benchmarks for evaluating the capabilities of vision-language models from the perspectives of environment perception HHng
et al. (2023) Akoury et al. (2023), context reasoning Liu al. (2023); Wang et al. (2023), decision-mann
Chen et al. [2023): Qian et al. [2023 etc. The majority of existing games can be classified into two categories:
1) Competilion games Ma et al. [2023) ; Shao et al.] (2024) ; Hu et al. (2024b); ; Feng et al. (2024); Toshniwal
jtt al. (2022); Li et al. (202)): Gupta (2023): Huang et al. (022]]; Guo et al. (2023); Zhang et al. (024cc) in
which players compete against one another, with the objective of outperforming others to achieve victory.
Notable examples inchude StarCraft II Ma et al.
(2023); Shao et al. (2024). Pokémon Battles Hu et al.
(2024b), Chess Feng el. al. (2024); Toshniwal et al. (2022) Li et al. (2023) and Poker Gupta (2023); Huang
et al. (2024); Guo et al. (2023); Zhang al. (2024c). Ma et al. Ma et al. (2023) introduce Textttarrraff II,
natural language-basse based interface that equips LLMs with the functionality to play StarCraft II, fostering
more effective reasoning and decision-making capabilities; 2) Cooperation games
Carroll et al. 2019 Gong (2021b) Puig et al. (2018)
et al. (2023):
; Wu et al. (2021): Puig et al. (2021)
: Chen et al. (2024a); Gan et al.
Gan et al. 2021a) are structured around collaboration, requiring players to work together to achieve shared
goals. These games emphasize teamwork, communication, and joint problem-solving, where players must
coordinate efforts to succeed and reach mutual accomplishments. In Overcooked-AT Carroll et. al. (2019)
the preparation of an onion soup requires two agents collaborate by loading three onions into a cooking pot, thereby initiating a cooking process that spans 20-time steps.
The most relevant line of research to ours is the topic of game bug detection (Taesiri et al. 2022a 2024a; 2022b Tacsiri & Bezemer 2024 Tacsiri et al.) 2024b). Existing methods, however, either focus the dassification/ retrieval lasks Taesiri el al. (2022a) or limited in the static image domain Taesiri & Bezemer (024h; Taesiri et al. (2024a] (cf. Table lb). The prior work GameBugDescriptions Taesiri et al. (022b) employs LLMs to detoct bugs in game vidcos with the reliance on pre-extracted event-wisc textual descriptions and lacks support for multi evaluations. In contrast, our PhysGame addresses all these limitations and supports multi-modal instruction tuning in videos.
4

---

Published in Transactions on Machine Learning Research (January /2026)
Magnessen
Granity
Blctrictty
Blencity
Statistics of PhysGame
Value
Meal
Priction
Total number of QA pairs
140,057
condection
Unique Videos
38,957
Electro.
Question Length (avg/max)
9.81 / 113
Pusse change
Velocity
Answer Length (avg/ max)
53.49 / 143
Thermo.
Mechankes
Video Length (Scconds, avg/max)
23.33
2611.17
Body gesture
Accelenation
Statistics of GameBench
Value
V
Material
Objest shape
pruperties
Optics
Reflection
Total number of QA pairs
880
Unique Videos
880
Rigulty
dt natonisin uatoniiiin
Ahooptinn
Rottaction
Question Length (avg/max)
9.06
17
Answer Length favg/max)
14.46 / 38
Video Length (Seconds, avg/max)
25.82 / 239.57
(a)
(b)
Figure 3: (a) The taxonomy and (b) key statistics of the PhysGame and GameBench dataset.
3 Dataset Construction
3.1 PhysGame
Building on the intuitive cognition of physical understanding, PhysGame introduces comprchensive tax- onomy for lask categorization including five primary domains, i.e., mechanics, optics, material properties, thermodynamics, and electromagnetimm, and sixteen fine-grained categories (ef- Figure 3a
This category deals with forces and torques as well as their effects on motion, which provides the foundational principles interpret and analyze the motion of objects in videos. Typical cases include gravity, elasticity, friction, velocity and acceleration over time.
Mechanics:
Optics: It focuses on the behavior and properties of light as well as its interactions with matter. Tt includes reflection, refraction, and absorption & transmission.
Material properties: It refers to the inherent material characteristics including color, rigidity, object shape, and human body gesture.
Thermodynamics: This category involves the study of heat, temperature, and energy transfer between systems. Typical cases include heat conduction and phase change.
Electromagnetism: It encompasses phenomena related to the electric and magnetic fields and their inter- actions with matter.
The videos of PhysGame dataset are curated from the GamePhysics subreddi a community where users share gameplay videos exhibiting various unusual events and glitches across dif- ferent games, accompanied by the users' discussions (metta-nforratiinn). This plat offers abundant glitch-containing gameplay footage alongside meta-information that broadly describes glitch characteristics, enabling the scalability of our proposed PhysGame dataset.
Video Collection.
inttruction-folloiing Generation. We follow the self-instructoon paradigm Li et al. 2024a to construct PhysGame by prompting GPT-40 OpenAl (2024). In terms of the instruction generation, we aspire for the questions to be as diverse as possible. We set up three question types, varying from direct to indirect assessment the glitches in videos: i) explicitly inquiring about glitches in the video, ii) probing anomalies present the vidco, iii) or merely straightforward questions regarding the vidco content. We provide three examples corresponding to the mentioned three scenarios: What is the description of the glitch observed
www.reddit. cctp:r//aaammeyyiiccco
5

---

Published in Transactions on Machine Learning Research (January /2026
Question: What the description of the glitch observed in the video?
Glitch content: No realistic wave propagation
Title:
[GTA MV] hope they bring back the water physics.
In the videa, when the player pilots helicopter to land on the water surface, the spinning rotor blades create realistic disturbance, pushing the water away and forming clear circular pattern beneath the aircraft. ..... At specific moments, the surface the water lacks believable wave propagation, especially when interacting with nearby objects terrain. These inconsistencies slightly detract from realism
Answer w/ title
The glitch involves a helicopter interacting with water in an unrealistic and physically implausible way. After descending and making contact with the water's surface, instead of behaving like damaged crashing aircraft-sinking, breaking apart, stopping- begins to bounce erratically and spin uncontrollably. The helicopter skims and ricochets across the water if it rubber ball, completely ignoring expected aerodynamic drag water resistance...
Answer w/o title
Figure 4: The generated responses PhysGame dataset with or without meta-inforrmation hints (i.e., titles). The title indicates a glitch related water physics. With the title hints, the generated responses accurately identify that the surface of the water lacks realistic wave propagation; in contrast, the response contains hallucinated content without the title hints.
Table 2: Quality validation for the proposed PhysGame dataset.
Meta Data Ace
QA Ace w/ Meta
QA Ace w/o Meta
86%
91%
64%
in the video! ii) Are there any abnormalities present in the videos? iii Please provide a description of the video content.
As for the response generation, the preliminary experiments suggest that the intuitive prompting method leads to remarkable errors. To alleviate this, we propose a meta-information guided prompting strategy. Specifically, found that the meta-information (e.g., title) associated with each video offers insight into the fundamental content. Therefore, we propose to incorporate video- wise meta-information in the prompt, resulting in more accurate instruction-follownng generation. As shown in Figure 4 the meta-information helps GPT-4o to detect the "water physics" glitch, while its absence leads to the degradation of physical commonsense understanding. The key statistics of the constructed PhysGame are presented in Table 3b
Quality Control. Given the scale of our instruction dataset, per-sample manual verification proves infea- sible. We implement systematic quality validation framework by sampling 2,000 instances and conducting two-lier manual verification: 1) meladala relevance to video semantics, and 2) QA alignment with observed physical anomalies. The experiments in Table show 86% metadata accuracy and 91% QA accuracy, demon- strating the overall reliability of our PhysGame. In addition, the comparison between QA accuracy with and without metadata (91% us. 64%) confirms that incorporating meta-information improves the quality of generated instruction data.
3.2 GameBench
Vidco Collection and Filter. The vidco collection procedure of GameBench follows the same process as that in PhysGame. To prevent data leakage we diligently exclude any videos included in PhysGame. We conduct manual checks based on the following two criteria: 1) Duplicate check: The Reddit discussion forum may feature multiple references to the same vidco, resulting in duplicate downloading. We manually check to confirm that each video in GameBench is distinct; 2) Content check: The pool of downloaded videos may incorporate non-game elements, which we rigorously filter out of our GameBench benchmark.

---

Published in Transactions on Machine Learning Research (January
Annotation Scheme. Based on the collected gameplay videos, create the question-answer pairs in four-way multiple-choice format to facilitate convenient evaluation. Specifically, the correct options describe the video-specific glitches that contravene physical commonsense principles. It is important to note that some vidcos may cxhibit multiple glitches. We therefore instruct expert annotators to review the entire video to ensure all the appearing glitches are included in the correct answer.
To enhance the plausibility of the distractor options, we have provided expert annotators with three guiding principles: 1) Instead of imagining arbitrary glitches, the glitch in the distractor options should be highly correlated to the individuals actions observed in the videos. For example in Figure 2 (right), the distractor option B includes carr' and "shadow" that are gemuincly present in the vidco. This annotation principle forces MLLMs to comprehend the glitchy content, rather than merely selecting answers by identifying contained objects or actions: 2) The four choice options should be of similar length, which helps prevent any preference biases MLLMs; 3) To mitigate choice bias, the distribution of the correct option among the four choices should be equitable, (i.e 25% likelihood for cach option).
Quality Control. To guarantee the quality of our dataset, we conduct two-fold quality control process: 1) LLM-assisted inspection: We exclude question-answering pairs that can be correctly answered by GPT-40 OpenAl (002)) solely based on the question and options without the need to view the video. By statistics, we limit the accuracy of GPT-4o in the question-only scenario to less than 25%; 2) Human inspection: pairs undergo rigorous cross- crosssinspection by different human annotators. For the correct options, the inspectors must assess whether they comprehensively and accurately describe all instances of physical commonsense violations present. For the distractor options, the inspectors are required to evaluate whether they are sufficiently deceptive, specifically by including objects actions depicted in the video. Through the rigorous construction and review process, we present the GameBBnch benchmark which is of high quality and well-balanced. The specific statistics are available in Table 3b
All the initially annotated
4 Experiments
4.1 Experimental Settings
Baseline and Implementation Details. To evaluate the efficiency of the PhysGame dataset, we per- formed supervised fino-tuning on three MLLMs: Qwen2-VI.-7B Wang et al. (2024a). Qwen2.5-VIL-7B Bai et al. 2025), and InternVL2.5-8B Chen et al. (202ce). To prevent the model from becoming overly biased toward the gameplay video domain during fine-tuning, we additionally incorporated 20K samples from the general video instruction tuning dataset LLa VA- Vo-nound Zhang et al. (2024b), forming a total training set of 160K samples. The Qwen series models Wang et al. (2024a); Bai et al. (2025) were trained for one epoch with a batch size of 128 and a learning rate of 20-7, whereas InternVT.2.5-88 Chen et al. (2024c) employed higher learning rate of 1e-6 with an equivalent batch size. All models processed 32-frame video sequences as input. Experiments were conducted on 8 NVIDIA A100 GPUs.
Evaluation Benchmarks. To demonstrate the broad applicability of our PhysGame dataset, we conducte experiments across both physical world understanding and general video understanding tasks:
Physical World Underslanding: benchmarks: gameplay- based dataset (i.e., self-constructed GameBench) and real-world datasets (i.e., PhysBench Chow et al. (2025) and MMVU Zhao et al. (2025)).
Depending on the video data source, we evaluate on two categories of
General Video Understanding: We further validate the broad value of PhysGame through evaluations on established general video understanding benchmarks including MVBench Li et al. (002b)), VidcoMME Fu et al. (2024) and LongViieoBench Wu et al. (2024).
4.2 Physical Understanding
Evaluation Results on GameBench. For comparisons, we benchmark the proposed GameBench on
both proprictary MLLMs (i.e., Clandc3.5-Sonnct |Anthropic (2024), Sauudc3.--SoneetVV.. Anthropic (2024).
Gemini-L.5-pro. Team et al. (2024). Gemini-1.55 1.-pniii..--ro--lasll Team et al. (2024). GPT-4V Achiam et al. (2023)
GPT-40-0806 ApenAT 2024). GPT-40-mini-0718 OpenA (2024) and Qwen-VT.mmas Bai et al. (2223) and
7

---

Published in Transactions on Machine Learning Research (January 22026)
Table 3: Evaluation (%) results on the proposed GameBench benchmark including the domains of mechanics, optics, material properties, thermodynamics, and eloctromagnetism.
Models
AVG
Mechanics
Optics
Material
Thermo
Electro
Claude3.5-Sonnee Anthropic
(2024
54.3
53.7
49.6
54.5
71.4
55.3
Sauude3.--Soneeee Authropic 2024
47.6
47.0
42.1
51.9
53.1
46.8
Gemini-1.8 1.5-pro Team et al. 22024
55.2
55.0
49.6
58.7
63.3
51.1
Gemini-1 Comini--15-pro-- 15-pro-flash Team et al.
2024
48.5
50.4
41.4
50.3
49.0
42.6
GPT-4V Achiam et al. 2023
45.9
45.9
43.6
43.9
53.1
53.2
GPT-4o OpenAT
2024
56.1
55.4
48.9
60.3
67.3
55.3
GPT-4o-mini OpenAl
2024
40.3
40.7
39.1
41.3
36.7
40.4
Qwen-Vl--Max
50.9
46.8
52.6
55.0
69.4
51.1
LLaVA-Next- Video Liu al.
2024b
32.2
31.8
25.6
37.0
42.9
23.4
Video-LLaVe Lin et al.
20233
29.0
29.2
28.6
31.2
30.6
17.0
LLaVA-OneVision
47.7
46.3
48.1
49.2
59.2
42.6
InternV1.2 Chen et al.
2024c
33.4
34.2
37.6
29.1
30.6
34.0
VideoChat2
34.3
33.1
31.6
37.0
44.9
31.9
ST-LLM
32.8
30.3
30.8
38.6
49.0
23.4
Chat-UniVi Jin et al.
2024
29.5
30.7
30.1
30.7
20.4
21.3
PPLLaVA
38.4
37.7
38.3
36.0
51.0
42.6
Qwen2- VL-7B Wang et al.
2024a
37.5
35.5
34.6
37.0
34.7
34.0
w// PhysGame
43.8
45.5
47.4
48.7
40.8
36.2
+6.3
+10.0
+12.8
+11.7
+6.1
+2.2
Qwen2.5- VL-7BB
Bai et al.
2025
44.4
46.5
40.6
46.0
34.7
38.3
w// PhysGame
48.1
51.5
42.1
48.1
34.7
44.7
A
+3.7
+5.0
+1.5
+2.1
0.0
+6.4
I.2ernV1.2.5-8B w/ PhysGame
Chen et al.
2024c
38.6
35.9
33.1
43.4
59.2
40.4
48.0
44.4
43.6
56.6
67.3
40.4
A
+9.4
+8.5
+10.5
+13.2
+8.1
+0.0
LA-VA- OneVision Li et al. (2024a), Intern L2 Chen et al. (2024d). VideoChat2 Li et al. (2024b). ST-LLN Liu et al. et al. (2024d). We follow Vidco-MME
open-source models including LLa VAL Next- Video Liu et al. 2024b), Video-LLa VA Lin et al. 2023).
Fu et al. (204)) to utilize the official frame configurations provided for each MLLM. We employ the selection accuracy as the evaluation metric for our curated multi-choice questions.
2024c Chat-Uni' Jin et al. (2024] and PPLLa
The evaluation results the GameBench benchmark presented in Table 3 Key observations include: Current methods struggle on GumeBench: Among proprietary models, GPT-4o OpenAT (2244) and Gemini- 1.5-pro Team et al. (2024) achieve the best performance with average accuracy scores of only 56.1% and 55.2%, respectively. As a reminder, the GPT-4o score of 56.1% is achieved on GameBench after filtering out questions that were found to be answerable by text alone (using GPT-4o itself during quality control), which further underscores the challenging nature of the benchmark. This indicates persistent challenges in understanding physical commonsense within complex, dynamically evolving scenarios; 2) Instruction tuning on PhysGame benefits: Baseline models (Qwen2-VL-7B Waag et al. (2024a). Qwen2.5-VL-77 Bai et al. (2025), InternVL2.5-881 Chen et al. (202cc)) show substantial improvements after PhysGame- based instruction tuning. For instance, the Gamo-tnntd Qwen2- VL achieves an 6.3% higher average accuracy on GameBench compared to its baseline version, demonstrating the dataset's value for gameplay-based physical reasoning; 3) Inconsistent improvements: An interesting observation is that InternVL2.5-5 exhibits the most significant improvement in average accuracy after fine-tuning on PhysGame, achicving a gain of 9.4%, with consistent performance increases across all subcategories. In contrast, the Qwen-series models Wang et al. (2024a); Bai et al. (2025) show relatively limited improvement, likely because their pretrained
8

---

Published in Transactions on Machine Learning Research (January 20066
Table 4: Evaluation (%) Reae2Real transferability on PhysBench and MMVU, "w/ PhysGame"
denotes finetuning bascline models on the proposed PhysGame dataset.
PhysBench
MMVU
Models
Average
Property
Relationships
Scene
Dynamics
Val
GPT-4V Achiam et al.
2023
41.3
49.6
45.8
26.3
42.2
GP'T-40 OpenA1
2024
49.5
56.9
64.8
30.2
47.0
67.4
CPT-Moo-mini OpenAT
(2021
43.2
53.5
14.2
30.6
42.9
61.6
Gemini-1.5-flash Team et al.
2024
46.1
57.4
32.2
34.3
40.9
58.8
Gemini-L.5-pro Team al.
2024
57.3
36.5
41.6
Llava-Next-Videed Liu et
2024b
35.4
38.3
30.8
31.0
37.2
28.6
Phi-3V Abdin et al. 22024
38.4
43.7
37.9
34.9
36.9
VILA-1.5-8B Lin et al.
2024
32.9
33.4
29.9
30.9
35.9
VILA-1.5-13B
Lin et al. 20024
37.2
40.5
40.2
32.0
36.1
Video-LLaVA Lin et al. 2023
37.0
36.8
36.1
33.7
10.5
Univ-Univi-7B Jin al.
20244
22.2
19.3
21.0
18.9
28.5
Chat-Univi-131 Jin al.
(2024
10.4
4.3
11.5
15.7
11.5
Claude--.5-sonneet Anthropic
[2024)
38.1
46.5
11.1
27.9
37.6
65.2
1 Chen et al.
2024d
47.5
53.1
70.1
37.0
44.8
Qwen2-VL-7B Wang et al.
2024a
46.6
54.3
60.4
39.4
47.0
42.1
w/ PhysGame
49.1
57.5
63.2
41.8
49.1
(vs. Qwen-2VVL-BI
+2.5
+3.2
+2.8
+2.4
+2.1
+6.5
Qw Bai al.
2025
45.9
54.6
63.2
37.0
46.9
48.8
w/ PhyssGame
47.6
57.7
64.1
38.3
48.2
50.5
A (us. Qwen-2.5-VI-7B)
+1.7
+-3.1
+0.9
+1.3
+1.3
+1.7
InternVL2.5-8B Chen el al.
2024c
43.9
55.9
48.7
29.4
41.2
41.1
w/ PhysGame
46.0
57.9
66.6
30.7
51.2
49.6
(18. InternVI.2.5-8BB
+2.1
+2.0
+17.9
+1.3
+10.0
+8.5
vision-language alignment and reasoning capabilities are already highly optimized through large-scale multi- modal instruction tuning, leaving limited room for further enhancement from task-specific physical reasoning data.
Game2Real Transferability. In addition to gameplay and simulator datasets, our PhysGame dataset demonstrates the GeallReal transferability on real-world physical understanding benchmarks including PhysBench Chow et al. 2202)) and MMVU Zhao et al. 20255) PhysBench proposed to evaluate MLLMs' physical world understanding capabilities, which integrates QAs regarding property, relationships, scene, and dynamics. MMVU assesses knowledge-intensvee reasoning on specialized-domanl vidcos. We conduct experiments its validation set.
The Game2Real transfer performance is shown in Table 4 Across four subcategories of PhysBench and MMVU, fine-tuning models on PhysGame (denoted by "w/ PhysGamc" in Table 4 consistently improves their performance over the baselines, dernonstrating the effectiveness of our proposed PhysGame in enhancing physical understanding and generalization. For example, QJwe22VV--72 and InternVL2.5-88 see significant absolute gains of +2.5% and +2.1% respectively on average after fino-tuning, while 1.ve22.5-VI-7B achieves the minimal improvement of +1.7%. These results confirm the strong Game2Real transferability facilitated by the PhysGame dataset.
4.3 General Video Understanding Results
We investigate the transfcrability of physics knowledge learned from PhysGame general video under- standing (Game2General transforability). Our experiments span MV-Bench Li et al. (2024b). Video-MME Fu et. al. 20044) and Long Bencc Wu et al. (2024), where MV-Bench focuses multifaceted video

---

Published in Transactions on Machine Learning Research (January 22026)
Table 5: Evaluation (%) of Game2General transferability on MVBench, Video-MME and
Long Vongeideoennch. "w/o subs' denotes results without subtitles. "w/ PhysGame' denotes finctuning bascline models on the proposed PhysGame dataset.
Video- MME (w/o subs)
Long VideoBench
Models
MVBench
AVG
Short
Medium
Long
Val
GPT-4V Achiam al.
22033
43.5
59.9
70.5
55.8
53.5
Vidoo-LLaVA, Lin al.
2023
45.3
38.0
36.2
39.1
LLaVA-1.5 Liu et
2024a
36.0
40.3
PLLaVA Xu et 202-aa
46.6
40.2
Qwen-VL-Max Bai et
(2023
55.8
49.2
ShareGPPTViddro-5B Chen
2024b
51.2
48.3
36.3
35.0
39.7
InteraVI-Chau-V1.. Chen et
2024d
45.6
50.7
60.2
51.2
VideoChat2 Li al. 2024b
37.0
39.3
LongLLaVA- Wang et
2024bb
49.1
43.7
52.4
42.2
36.4
Video-CCAM--D)
al.
2024
64.6
61.9
49.2
NVILA Liu al.
2024e
75.7
62.2
Apollo Zohar et
[2024
61.3
58.5
LongVA-7BB Zhang al.
(2024a
52.6
61.1
50.4
46.2
LLaVA- Video 7B Zhang
2024d
58.6
63.3
58.2
Qwen2-VT. 2-B Wang et al.
(002aa
64.5
58.0
70.2
55.8
48.1
55.3
w/ PhysGame
66.4
59.3
70.7
57.6
49.7
56.1
A (us. Qwen-2VL--B Wang et al.
2024a
+1.9
+1.3
+0.5
+1.8
+1.6
+0.8
Q Qwen2.5-V VL-7B Bai et al. 2025
66.0
64.7
76.2
63.3
54.6
56.0
w/ PhysGame
67.3
65.8
76.6
65.7
55.2
58.3
(1s. Quven-2.5 Bai al.
22025
+1.3
+1.1
+0.4
+2.4
+0.6
+2.3
InternVL2..555 Chen et al. (2021c
72.0
64.0
75.8
62.8
53.4
57.2
w/ PhysGame
72.8
64.2
76.8
62.6
53.3
59.5
A (18. InternVL2.5-8B
Chen et al. 2021c)
+0.8
+0.2
+1.0
-0.2
-0.1
+2.3
derstanding capabilitios short vidcos while Vidco-MME and LongVideoBench emphasize long-form vidco comprehension. All evaluations follow the official protocols.
As shown in Table 5 finetuning models on PhysGame consistently boosts their performance across all bench- marks. For example, Qwen-2-VL-7B achieves 1.9% improvement on MVBench and sces gains across all three temporal scopes of Video-MME, with notable 1.8% gain medium-length videos. Qwen-2.5-VI-7BB also benefits from PhysGame, attaining a 2.3% accuracy boost in LongVideoBench. Similarly, InternVT.2.5- 8B improves from 72.0% to 72.8% on MVBench, marking a new state-of-theeart result. We also observe a slight performance drop on the medium and long video splits of the Video-MME dataset after fine-tuning on PhysGame, which may be attributed to the fact that most videos PhysGame are short, leading InternVL2.5-22 to develop bias toward short-duration videos. Overall, these findings confirm that physies- oriented pretraining via PhysGame not only improves physical reasoning but also enhances the generalization of hideo-languee models to diverse and longer-duration video tasks.
4.4 Ablation Studies
Data-scaling effects. To quantify the data-scaling behavior of our PhysGame datasct, we conduct addi- tional experiments using Qwen2.5-VL-7B fine-Luned with different amounts of PhysGame data (OK, 40K, 80K, 140K) to analyze the scaling trends. As noted in Section 4.1 for real-world and general-video tasks (PhysBench, MMVU, MVBench, Video-MME, LongVidcoBonch), we mix PhysGame with 20K LLaVA- Hound samples maintain domain balance. The results in Figure demonstrate that the performance consistently improves more PhysGame data is used. The gains monotonic and do not plateau even
10

---

Published in Transactions on Machine Learning Research (January /2026
GameBench
PhysBench
48
50.5
Accuracy
Accuracy
47.0
Accuracy
55.0
46
49.5
49.0
Data Size (K)
Data Size [K)
Data Size (1)
MVBench
Video-MME
LongVideoBench
67.2
58.0
67.0
Accuracy
65.6
Accuracy
57.5
66.8
65.4
57.0
66.6
65.2
56.5
0
25
50
75
100
125
0
25
50
75
100
125
25
50
75
100
125
Data Size (K)
Data Size (K)
Data Size (K)
Figure 5: Data-scaling effects of our PhysGame dataset.
Table Ablation study on meta-data and training data mixture based on Qwen2.5- VL.
Method
PhysBench
MMVU
MVBench
Video-MME LongVideoBench
Baseline
45.9
48.8
66.0
64.7
56.0
Full Mode
47.6
50.5
67.3
65.8
58.3
w/o meta-data
42.9
46.7
65.2
63.5
55.2
w/o data mixture
45.1
49.2
66.7
65.1
57.4
140K samples, indicating that the dataset's scaling potential has not been saturated. This trend validates the scalability of PhysGame and suggests that further expansion could yield additional improvements.
Ablations datasct mixture. In Section 4.1 constructed the overall training set by combining 110K with 20K data from LLa VAa-o-od Zhang et. al. (2024b). We ablate on this dataset mix- ture strategy by fine-tuning using only PhysGame (140K) dataset without any LLaVA-Hound data. The experimental results in Table D the data mixing strategy plays a clear rolc, particularly in enhancing gen- eralization to real-world benchmarks (e.g., +2.5% on PhysBench compared to using only PhysGame).
Ablations on training dataset composition. We conduct fixed-budgee budget and reed-controlled resampling to ablate the training dataset configuration. Specifically, fix the total training budget to 160K samples and systematically vary the mixing ratio between PhysGame (PG) and LLaVA- (LH) OK/160K, 40K/120K, 80K /8KK, 140K 140K/20K] /2K).. A fixed random seed ensures the same LH subset is used across runs. eliminating sampling randomness.
As the results in Table 7 indicate, using only VL-unnd does not yield performance gains. For instance, with Qwen2.5-VVL VL. only using the general video instruction tuming dataset V.L.VA-Houn maintains the same results the zero-shot evaluation (45.9%). Using only PhysGame leads to performance drop (45.1% vs. 45.9%), likely because post-training solely PhysGame induces bias towards gameplay videos. Conse- quently, selected mixing ratio of 140K samples from PhysGame and 20K samples from LLa to achieve optimal performance. One interesting finding is that the performance of the pure PhysGame configuration (1 (140K/0K) is slightly lower than that of the pure general-dala configuration (OK/160K). This may be primarily due to two factors. First, the difference in total training samples (140K vs. 160K) gives the gencral-data setting a scale advantage in learning diverse visual representations. Sccond, training solely on PhysGame, which is visually centered on gameplay scenes, can induce a mild domain bias, slightly hindering generalization to the broader visual distributions found in real-world benchmarks like PhysBench.
11

---

Published in Transactions on Machine Learning Research (January 2002)
Table Ablation study on the configuration PhysGame (PG) and LLa LLLaVA-Hou Hound (LH) datasets.
Configuration of PG/LH
0K/160K
80K/80K
140K/20K
140K/0K
AVG PhysBench 45.9%
46.9%
47.3%
47.6%
45.1%
Question: What main reason the phenomenon vidco?
A. Temperature
B. water
C. dynamite
D. human activity
E. gravity
Qwen2.5-VI.-7B:
PhysGame:
Question:
In the mechunism depicted, the system has
degree(s) of freedom.
Qwen2.5-VL-7B:
w/ PhysSGame
(b)
Figure 6: Comparisons between the baseline Qwen2.5-VVL-BB and its PhysGame fine-tuned counterpart on MMVU benchmark.
Ablations of the meta-infornation-guieeee prompting. In Section 3.1 we conduct a manual quality validation Lo remonstrate that. meta-information enhances the quality generated QA pairs. Here we present quantitative experiments using Qwen2.5-VL trained on PhysGame with with/without meta-information guidance. Our experimental results in Table G demonstrate that integrating meta-information leads to 4.7% absolute improvement (42.9% us. 47.6%) on PhysBench, which further highlights the necessity of using meta-information guidance during the prompt generation process.
Visualizations. Figure G presents qualitative comparisons between the baseline Qwen2.5-VL-7B and its PhysGame fine-tuned counterpart on representative cases from the MMVU benchmark. In Figure 6a the baseline model incorrectly attributes the observed landslide phenomenon toward surface-level visual cues (e.g., explosion-like motion and dust dispersion). In contrast, the PhysGame- enhanced model correctly identifies gravity the underlying cause, demonstrating infer causal relationships grounded in physical principles. In Figure 6b where the task involves determining the degree of freedom of a linkage mechanism, the bascline model outputs two, failing to account for the kinematic coist imposed by joint connections. The PhysGame model, however, accurately infers one, reflecting a stronger grasp motion constraints and mechanical structure reasoning. Together, these examples illustrate that PhysGame effectively bridges the gap between visual perception and intuitive physics reasoning, enabling the model generate more physically consistent interpretations of both natural and synthetic scenes.
dynamite, indicating a bias
improved ability to
12

---

Published in Transactions on Machine Learning Research (Jamary/2026)
St-onnet
GPT-4o
Gemini-1.5-
10%
7%
4%
7%
6%
2%
5%
7%
81%
79%
80%
Perception Error
Lack Knowledge
Refusal to Answer
Reasoning Error
Figure 7: Error lype analysis.
Error type analysis. To investigate the bottlenecks of current MLLMs in physical world understanding, we analyze the top-performing models on the proposed GameBench benchmark: Gemini 1.5 Pro TTam al.)
2024h, GPT-4o OpenAT 2024). and Claude 3.5 Sonnet, Anthropic (2024). We manually examine the error
distribution across 100 selected cases, categorizing errors into four types:
Perception error: MLLMs fail to correctly recognize entities, characters, or actions in the video.
Lack of knowledge: MLLMs correctly perceive the video content but make incorrect judgments about the physical glitch.
MLLMs identifies both the video content and the glitch but fails to detect reasoning flaws in the provided answer options (e.g., causal errors).
Rensoning error:
Refusal to answer: MLLMs declines to respond due to limited capabilities.
Figure presents comparative analysis of error distributions across three leading MLLMs including Claude 3.5-Sonnet, GPT-4o, and Gemini 1.5-Pro. Across all models, the dominant error type is "Lack of Knowl- odge', accounting for approximately 79-81% of total crrors. This trend indicates that factual incompletenoss or outdated knowledge remains the principal bottleneck current large-scale multimodal reasoning systems, rather than perception or reasoning capabilitiess Among the secondary categories, Reasoning Errors con- tribute between 7-12%, with GPT-4o showing the highest proportion (12%). Perception Errors are relatively minor (4 7%) across all models, reflecting the general maturity of visual grounding components. Refusal to Answer occurs least. frequently, with Claude 3.5-Sonnet exhibiting the most conservative behavior (2%), while GPT-4o and Gemini 1.5-Pro show slightly higher refusal rates (5 (67%,, possibly due to stricter safety or uncertainty calibration mechanisms. Overall, the results highlight a consistent pattorn: while perception and reasoning are gradually improving, the knowledge base's coverage and retrieval precision remain the dominant limitations shaping multimodal model reliability.
5 Conclusion
In this work, we propose PhysGame, scalable and photo-realistic instruction-tuning dataset constructed from gameplay videos to advance physical world understanding of MLLMs. By leveraging glitch-inducee annotations, PhysGame enables systematic discovery of physical inconsistencies across diverse scenarios involving mechanics, optics, material propertics, thermodynamics, and electromagnctism. Compared to existing real-world or synthetic datasets, PhysGame achieves a better trade-off between realism and scal- ability. Extensive experiments demonstrate that PhysGame not only improves real-world physical reason- ing (Gamc2Real), but also enhances general-purpose vidco understanding (Gamc2General). We position PhysGame a scalable and effective paradigm for equipping MLLMs with robust physical reasoning and generalizable understanding abilities.
13

---

Published in Transactions on Machine Learning Research (Januar/
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 1rX1v:2404.14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXi::2333.00 087744 2023.
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of inluitive physics. Advances in nearal information processing systems, 29, 2016.
Nader Akoury Qian Yang, and Mohit Tyyer. A framework for exploring player perceptions of Imm-generated di- alogue in commercial video games. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2295-2311, 2023.
Authropic. Claude 3.5 sonnet, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large lision-language model with versatile abilities. arXiv preprint arXiv:2088. 12966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, at al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 19923, 2025.
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsensee for video generation. Xiv preprint arii::2406. 03520, 2024.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
Daniel Bear, Elias Wang, Damian Mrowca, Felix Jedidja Binder, Hsiao- Yu Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin A Smith, Fan- Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. In Thirly-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 2020.
Daniel G Bobrow. Qualitative reasoning about physical systems: an introduction. Artificial intelligence, 24 (1-3):1-5, 1984.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Arunkumar Byravan, Felix Leeb, Franziska Meier, and Dieter Fox. Se3-pose-nels: Structured deep dynamics models for visuomotor planning and control. arXiv preprint arXio:1710.00489, 2017.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca. Dragan. On
the utility of learning about humans for human-ai coordination. Advances in newral information processing systems, 32, 2019.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object- based approach to learning physical dynamics. arXiv preprint arXiv: 162.00011,, 2016.
14

---

Published in Transactions on Machine Learning Research (annuary/2200
Jiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: Self-organizing agents in open-ended envi- ronments. In ICLR 2024 Workshop on Large Language Model (T..M) Agents, 2024a.
Ke Chen, Yufei Li, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Wei Yang. Glib: towards automated test oracle for graphically-rich applications. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp- 1093- 1104, 2021.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. SharegptAvidoo: Improving vidco understanding and generation with better captions. Advances in Neural Informalion Processing Systems, 37:19472 19495, 2024b.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi- Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Represenlations, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arxiv preprint arXiv:2412. 05271, 2024c.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint (xxx:2404411822) 2024d.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. ar preprint arXiv:2205.00 01089,, 2022.
Zhenfang Chen, Shilong Dong, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Compositional physical reasoning of objects and events from videos. arXiv preprint arXiv:2408.0 02688, 2024e.
Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding, arXiv preprint 3rXiv:2501.16411 2025.
Chris Conlan. The blender python api. Precision 3D Modeling and Add, 2017.
Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pp. 5444 5452. International Joint Conferences on Artificial Intelligence Organization, 2022.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2488. 14023, 2024.
Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Newral Information Processing Systems, 36, 2024.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical common- sense? arXiv preprint arXio:1988.0255 2019.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in vidco analysis. arXiv preprint arXiv:2405.210% 2024.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Fre- itas, Jonas Kubilius, Abhishok Bhandwaldar, Nick Haber, et al. Throedworld: A platform for interactive multi-modal physical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Dalasets and Benchmarks Track (Round 1), 2021a.
15

---

Published in Transactions on Machine Learning Research (January
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, et al. The threedworld transport challenge: A visually guided lask-and-motidn planning benchmark for physically realistic embodied ai. arXiv preprint arXiv:2103. 14025, 2021b.
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint
arXiv:2309.080 09991,, 2023.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. auspicion-agent: Playing imperfoct information games with thcory of mind aware gpt-4. arxiv preprint wxXi:2300017277 2023.
Akshat Gupta. Are chatgpt and gpt-4 good poker players?-a pre-flop analysis. arXiv:2308. 12466, 2023.
arXiv preprint
John K Haas. A history of the unity game engine. 2014.
Susan J Hespos and Elizabeth S Spelke. Conceptual precursors to language. Nature, 430(6998):4533 -4666466633666 2004.
Susan J Hespos, Alissa I. Ferry, Erin M Anderson, Emily N Hollenbeck, and Lance J Rips. Five-month-old infants have general knowledge of how nonsolid substances behave and interact. Psychological science, 27 (2):244 256, 2016.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arxiv preprint arXiv:2308. 00352, 2023.
Sihao Hu, Tiansheng Huang, Fatih Ilhan, Sclim Tekin, Gaowen Liu, Ramana Kompella, and Ling Liu. A survey on large language model-based game agents. arXiv preprint ar Xiv:24404 02039, 2024a.
Sihao Hu, Tiansheng Huang, and Ling Liu. Pok) 'ellmon: A human-parity agent for pokk 'emon battles with large language models. Xin preprint 2ix:2402.00111 2024b.
Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. Pokergpt: An end- to-end wight solver for multi-player texas hold'en em via large language model. arXiv preprint NnXiv:240106679) 2024.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CE /CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. Tn International conference on machine learning, pp. 430 438. PMLR, 2016.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint ar arXi::2408 03326, 2024a.
Kennoth Li, Aspen Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhno Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprchensive multi-modal vidco understanding benchmark. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recogrilion, pp. 22195 22206, 2024b.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXio preprint arXim:331 10122, 2023.
16

---

Published in Transactions on Machine Learning Research (January /202)
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVR Conference on Compuler Vision and Pattern Recognition, pp. 26689 26699, 2024.
Carlos Ling, Konrad Tollmar, and Linus Gisslén. Using deep convolutional neural networks detect rendered glitches in video games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pp. 66 66-73, 2020.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 26296 26306, 2024a.
Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL sttps://llava-vl.ggee .it..git.u...... io/ b1og/2224-01130-11aa lllaaalavaeet...
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. Lim Amm-oowered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXim:2312.1522, 2023.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-lim: Large language models are effective temporal learners. arxic preprint orXiv:2404.00308 2024c.
Ruyang Liu, Haoran Tang, Haibo Liu, Yixiao Ge, Ying Shan, Chen Li, and Jiankun Yang. Ppllava: Varied video sequence understanding with prompt guidance. arxie preprinl orXiv:24 Niw:2211.00321 11.02327, 2024d.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXio preprint 1rXiv:2412.04444 (tl::2412.04444 2024e.
Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play starcraft it: Benchmarks and chain of summarization approach. arXiv preprint arXio:2312. 11865, 2023.
Michael McCloskey, Allyson Washburn, and Linda Felch. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(4):636, 1983.
Andrew Melnik, Robin Schiewer, Moritz Lange, Andrei Toan Muresanu, Animesh Garg, Helge Ritter, et al. Benchmarks for physical reasoning ai. Transactions on Machine Learning Research, 2024.
Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfong Lu, Kaipong Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation. orXiv preprint arXiv:24 10. 10.05333, 2024.
Alfredo Nantes, Ross Brown, and Frederic Maire. A framework for the semi-automatic testing of video games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 4, pp. 197 202, 2008.
OpenAl. Hello gpt-4o. OpenAl Blog, 2024. URL http:://pppaaaatcoommmm/e//aaala
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494-8502, 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
Chen Qian, Xin Cong, Cheng Yang. Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. Xio preprint ar 20X7:23307.7724 2023.
17

---

Published in Transactions on Machine Learning Research (January
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Xiao Shao, Weifu Jiang, Fei Zuo, and Mengqing Liu. Swarmbrain: Embodied agent for real-time strategy game starcraft via large language models. arxiv preprinl urXiv:2401.1772 17744, 2024.
Mohammad Reza Taesiri and Cor-Paul Bezemer. Videogamebunny: Towards vision assistants for video
games. arXiv preprint arXiv:2407115295, 2024.
Mohammad Reza Taesiri, Finlay Macklon, and Cor-Paul Bezemer. Clip meets gamephysics: Towards bug identification in gameplay videos using zero-shot transfer learning. In Proceedings of the 19th International Conference on Mining Software Repositories, pp. 270-281, 2022a.
Mohammad Reza Tacsiri, Finlay Macklon, Yihe Wang. Hengshuo Shen, and Cor-Paul Bezemer. Large language models are pretty good zero-shot video game bug detectors. arXiv preprint orXiv:2210.0900 02500, 2022b.
Mohammad Reza Taesiri, Tianjun Feng, Cor-Paul Bezemer, and Anh Nguyen. Glitchbench: Can large multimodal models detect video game glitches? In Proceedings of the TEEE/CVP Conference on Compuler Vision and Pattern Recognition, pp. 22444-2222 22445,, 2024a.
Mohammad Reza Taesiri, Finlay Macklon, Sarra Habchi, and Cor-Paul Bezemer. Searching bug instances in gameplay video repositories. IEEE Transactions on Games, 2024b.
Gemini Team, Petko Georgiev, Ving Ian Lci, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint orXiv:2403.05500 2024.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as a testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelligence volume 36, pp. 11385-11393, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0928 2023.
Pong Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXic preprint orXiv:2409.1111 12191 2024a.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXio:2310.0133 01320,, 2023.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409. 009.089,,,0 2024b.
Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Composi- tional 4d dynamic scenes understanding with physics priors for vidco question answering. arXiv preprint arXiv:2406.006 2024c.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint xxXiv:240.15754, 2024.
Sarah Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman- Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cog- mitáve Science, 13(2):414-433 2021.
18

---

Published in Transactions on Machine Learning Research (January
Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, et al. Seephys: Does seeing help thinking? benchmarknnnn vision-based physics reasoning. arx preprint arXiv:2505. 1ato:2995.199,,, 2025.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter- free llava extension from images to videos for vidco dense captioning. arXiv preprint 2Xx::2444.1999, 2024a.
Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, and Börje F Karlsson. A survey on game playing agents and large models: Methods, applications, and challenges. arXiv preprint arXiv:2403.1022 2024b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Temenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arxiv preprint arXiv:2406.1 16852, 2024a.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. aiv preprint #xXx:2444.01255 2024b.
Wenqi Zhang, Ke Tang, Hai Wu,, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint XrXix:2402. 17774, 2024c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Vidco instruction tuning with synthetic data. arXin preprint or Xiv: 10.02713, 2024d.
Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint ar 2fw:2501122380 2025.
Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B Tenenbaum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. In International Conference on Machine Learning, pp. 61526 61558. PMIR, 2024.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruel Licheng Yu, Xiaofang Wang, Felix Juefei-Xu. Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint Xrv:v:2412. 10360, 2024.
19


Reviews:
[{"review_id": "fUgQtJEEpA", "reviewer": "Reviewer_64BW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors argue that identifying violations of physical laws is a scalable way to teach physical principles. They introduce PhysGame, an instruction-tuning dataset with over 140k QA pairs pulled from glitchy gameplay videos collected from Reddit, along with GameBench, a manually curated benchmark for evaluation. The authors use a meta-information-guided prompting strategy to generate high-quality training data. Experiments show that fine-tuning with PhysGame improves performance on gameplay benchmarks and also transfers effectively to real-world physical understanding and general video understanding tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors have conducted thorough comparison and ablation experiments to back up their claims: learning from \"glitches\" improves physical understanding, PhysGame boosts general video understanding, and metadata is key for generating high-quality data.", "audience_yn": "Yes", "audience_explanation": "The ideas and validation around learning from glitches could further expand the scale of the dataset. The proposed PhysGame and GameBench will be valuable resources for researchers dedicated to physical world understanding tasks.", "requested_changes": "- This relies a lot on the quality of Reddit post titles. While the authors validated a sample, Reddit titles can sometimes be sarcastic, unrelated, or use slang that doesn't really describe the physical phenomenon accurately. If the metadata is noisy, the teacher model (GPT-4o) might generate hallucinated explanations. While the paper briefly touches on this, it could go deeper into how these potential issues are handled, particularly in terms of robustness. There’s also no real discussion about how potential biases in the data are addressed or how they impact the dataset’s overall reliability. More transparency here would be helpful.\n- The paper mentions filtering out non-game elements, but it’s unclear how the pipeline distinguishes between intentional game mechanics that break physics (which shouldn’t be \"corrected\") and unintentional rendering glitches. It would be useful if the authors clarified the line between a “glitch” and “game logic” (for example, double-jumping or magic spells that are part of the game’s design).\n- For GameBench, the distractor options are designed to include objects that actually appear in the video, which prevents the model from just matching patterns. It would be helpful if the authors explained whether these distractors were generated automatically or written manually. If they were auto-generated, how did the authors ensure they were \"deceptive\" enough to challenge the model?", "broader_impact_concerns": "There is no explicit Broader Impact Statement present in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aLfIhBJjgl", "reviewer": "Reviewer_xhzp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes learning physical-world understanding from glitchy gameplay videos. It introduces PhysGame, a large instruction-tuning dataset of glitch-centric QA pairs, built via a meta-information–guided prompting scheme that leverages video titles/descriptions for higher-quality QA generation. The authors also release GameBench, an expert-annotated benchmark of gameplay clips with multiple-choice questions targeting physical implausibilities. Experiments show consistent transfer: finetuning on PhysGame improves real-world physical reasoning  and general video understanding. These results suggest that gameplay anomalies offer a scalable supervision signal for multimodal physical understanding.\n\n\nStrengths\n\n1. Clear motivation and generally well-written exposition; the ''order from chaos'' idea is intuitive and compelling.\n\n2. Solid empirical evidence: consistent gains on real-world physics (PhysBench), general video benchmarks (MVBench/Video-MME), and the new GameBench. \n\n3. Dataset design is scalable; construction details (taxonomy, metadata-guided prompting) are clearly documented. \n\nWeaknesses\n\n1. Cross-model analysis could be deepened (e.g., broader architectures/scaling trends or more systematic with/without PhysGame comparisons across tasks). \n\n2. The ablation section’s setup and takeaways are not fully transparent; clearer descriptions of what’s varied (e.g., metadata usage, glitch types, filtering) and why would improve interpretability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper shows that strong video understanding models struggle to achieve high accuracy on the proposed GameBench, indicating meaningful headroom and validating the benchmark’s challenge.\n \nAfter fine-tuning on the PhysGame dataset, multiple model families exhibit performance improvements on GameBench, supporting the claim that PhysGame imparts useful physical reasoning rather than overfitting to a single architecture.\n\n Fine-tuned models also achieve higher accuracy on two real-world physics datasets, demonstrating transferability from gameplay videos to natural videos and directly supporting the core claim about real-world physical understanding.\n\nImprovements on standard video-understanding benchmarks further suggest that PhysGame enhances models' general video comprehension, not only physics-specific judgments.\n\nOverall, the evidence is coherent across benchmarks, models, and domains, making the submission’s claims convincing and well supported.", "audience_yn": "Yes", "audience_explanation": "This work introduces an expert-annotated benchmark and demonstrates transfer to real-world physics and general video understanding. These elements make the findings relevant to researchers in video understanding, multimodal learning, and dataset design.", "requested_changes": "1. Broaden cross-model comparisons. Report with/without PhysGame across more model families and sizes (e.g., additional open-source baselines and parameter scales) to clarify architecture- and scale-robustness of the gains.\n\n2. Quantify data-scaling effects. Provide a clear scaling curve (e.g., 0->40k->80k->140k PhysGame samples) on GameBench, PhysBench, and MVBench to test whether accuracy continues to improve with more PhysGame data. (Some ablations exist; please extend them to cover GameBench and general-video tasks explicitly.) \n\n3. Clarify the ablation setup around LLaVA-Hound resampling. Make explicit whether resampling LLaVA-Hound contributes to the PhysBench gains. Include fixed-budget controls, seed-controlled resampling, and an ablation that isolates meta-information prompting vs. data-source mixing so readers can attribute improvements unambiguously.", "broader_impact_concerns": "The paper lacks a Broader Impact statement and leaves key ethics points unclear. The dataset is built from Reddit gameplay videos without explicit disclosure of creator consent, licensing compliance, takedown procedures. Likely demographic/genre skews may introduce bias, with no distributional stats or disparate-impact analysis. Safety in high-risk uses (e.g., autonomous driving) and misuse risks (harmful content, deepfakes) are not addressed. Environmental costs of training and annotation subjectivity (guidelines, agreement, compensation) are also unreported.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5qR5u9L0Cw", "reviewer": "Reviewer_sbHr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel paradigm for enhancing the physical world understanding of multi-modal large language models (MLLMs), centered on the idea of \"Order from Chaos.\" The authors posit that by learning to identify \"glitches\" (bugs or anomalies) in gameplay videos that violate physical laws, models can gain a deeper understanding of real-world physical rules.\n\nThe primary contributions are two-fold:\n\n- **PhysGame Dataset**: A large-scale instruction-tuning dataset containing 140,057 glitch-centric question-answer pairs. This dataset covers five physical domains (e.g., mechanics, optics) and was built using a novel \"meta-information-guided prompting strategy\" that leverages metadata like video titles to guide high-quality QA generation .\n\n- **GameBench Benchmark**: An expert-annotated evaluation benchmark with 880 glitch-identified gameplay videos. It is designed to evaluate a model's ability to detect physically implausible events and was constructed with rigorous quality control, including LLM-assisted inspection and human cross-checking .\n\nKey Strengths:\n\n- **Novel and Scalable Approach**: Using glitch videos as a data source is a highly novel perspective. It cleverly bypasses the limitations of real-world data (high annotation cost) and traditional synthetic data (lacking realism and diversity). This approach is highly scalable due to the abundance of online gameplay footage.\n\n- **Convincing Transferability**: The paper's strongest arguments are its \"Game2Real\" and \"Game2General\" transferability. Experiments compellingly demonstrate that models trained on game glitches show significant improvement in understanding real-world physics (e.g., on PhysBench) and general video understanding (e.g., on MVBench) .\n\n- **Rigorous Dataset Construction and Evaluation**: The authors not only propose a training set but also a companion evaluation benchmark (GameBench). The construction method for PhysGame (meta-information-guided) is proven effective (QA accuracy boosted from 64% to 91% ). GameBench's construction is also well-considered, especially its distractor design , making it a challenging benchmark.\n\nKey Weaknesses:\n\n- **Dependence on General-Purpose Data**: The ablation study (Table 6) shows that training only on PhysGame (140K/0K) actually performs worse (45.1%) than the baseline using only general video data (0K/160K) (45.9%). This indicates a need for general data to prevent overfitting to the gameplay domain, slightly weakening the strength of glitch data as a standalone source and highlighting the importance of the mixing ratio.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the paper's central claims are supported by sufficient and convincing evidence:\n\n- **\"Game2Real\" Transferability Claim**: This is the boldest claim. The authors support it by experimenting on two benchmarks based on real-world videos (PhysBench and MMVU) . As shown in Table 4, models fine-tuned on PhysGame (e.g., Qwen2-VL-7B) achieved a 2.5% absolute gain on PhysBench (from 46.6% to 49.1%) and a 6.5% gain on MMVU (from 42.1% to 48.6%). InternVL2.5-8B also saw a significant 8.5% gain on MMVU.\n\n- **\"Game2General\" Transferability Claim**: This claim is also well-supported by data in Table 5. Models show consistent performance boosts across multiple general video understanding benchmarks (MVBench, Video-MME, LongVideoBench) . For instance, Qwen-2-VL-7B improved by 1.9% on MVBench.\n\n- **Data Construction Method Effectiveness**: The authors claim their \"meta-information-guided prompting\" is crucial for high-quality data. This is doubly verified: (a) Manual quality validation showed QA accuracy at 91% with meta-info vs. 64% without; (b) A quantitative ablation showed a 4.7% absolute performance gain on PhysBench from models trained with meta-info.\n\n- **Value of the New Benchmark (GameBench)**: The authors claim GameBench is a challenging benchmark for physical reasoning. The results in Table 3 confirm this, as even SOTA proprietary models (like GPT-4o and Gemini 1.5-pro) only achieve 56.1% and 55.2% accuracy, respectively. This performance, far from human-level, proves the benchmark's utility and difficulty.", "audience_yn": "Yes", "audience_explanation": "Yes, a broad audience at TMLR would be interested in this paper for several reasons:\n\n- **Multimodal and Video Understanding Community**: The paper addresses a core challenge in AI—physical world understanding. This is highly relevant to researchers in MLLMs, video analysis, and commonsense reasoning.\n\n- **Data-Centric AI Community**: The paper presents a novel, low-cost, and highly scalable method for dataset construction. The insight of using \"glitches\" and metadata to auto-generate high-quality instruction-tuning data is extremely valuable for researchers focused on data curation and management.\n\n- **Sim-to-Real Community**: The \"Game2Real\" transferability finding is particularly compelling. Bridging the sim-to-real domain gap is a long-standing problem. This paper offers a new path: instead of striving for perfect simulation, learn from the simulation's flaws. This opens new lines of thought for domain adaptation and generalization research.", "requested_changes": "The paper is of high quality and well-argued. The following suggestions are intended to strengthen the work further and are not critical for acceptance:\n\n- **(Strengthening) Deeper analysis of the data mixture ratio**: As mentioned, the ablation in Table 6 is very interesting . The finding that the 140K/0K (pure PhysGame) configuration performs poorly (45.1%) is attributed to domain bias. The paper would be stronger with more analysis on this \"catastrophic forgetting\" or domain bias phenomenon. For instance, why are the gains from 80K/80K (47.3%) to 140K/20K (47.6%)  so marginal? A deeper discussion on the interplay between the PhysGame data and general-purpose data (LLaVA-Hound) would be valuable.\n\n- **(Strengthening) Qualitative example of a \"Lack of Knowledge\" error**: The error analysis in Figure 6 is excellent, identifying \"Lack of Knowledge\" as the primary error type (79-81%) for SOTA models. It would be helpful to provide one qualitative example of what this error looks like. For instance, does the model correctly perceive the visual phenomenon (e.g., \"car flies into the air\") but fail to identify the physical principle being violated (e.g., \"gravity\"), or does it misattribute it to another, incorrect physical principle? This would give readers a more concrete understanding of this dominant error category.\n\n- **(Minor Strengthening) Clarify GPT-4o's role in evaluation**: The authors mention using GPT-4o for data generation and as a filter in GameBench's quality control (to remove questions answerable by text alone) . In Table 3, GPT-4o is also an evaluated model, scoring 56.1%. It would be worth adding a single sentence in the Section 4.2 discussion of this result, reminding the reader that this 56.1% score is achieved after the samples GPT-4o found \"easy\" were already filtered out. This further highlights the challenging nature of the GameBench benchmark. (While this is mentioned in Sec 3.2, a reminder in the results section would be impactful.)", "broader_impact_concerns": "I have no major ethical or broader impact concerns regarding this work.\n\n- The research aims to advance the fundamental understanding of physical commonsense in AI, a goal of significant scientific merit.\n\n- The dataset is sourced from a public Reddit forum (GamePhysics subreddit), which is standard practice in MLLM research and does not involve sensitive personally identifiable information.\n\n- The technology does not have an obvious dual-use case or direct negative societal application.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)