Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering
Authors: Xingrui Wang
Wufei Ma
Angtian Wang
Shuo Chen
Adam Kortylewski
Alan Yuille
Venue: ICLR 2025 Poster (2025)
Keywords: Video question answering
Compositional reasoning
Physical scene understanding
3D scene understanding
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce DynSuperCLEVR, a video question answering dataset focused on the dynamic properties of 3D objects. We propose NS-4DPhysics, which use 4D world states with a 3D generative model and uses neural symbolic reasoning to answer questions.

Abstract: For vision-language models (VLMs), understanding the dynamic properties of objects and their interactions in 3D scenes from videos is crucial for effective reasoning about high-level temporal and action semantics. Although humans are adept at understanding these properties by constructing 3D and temporal (4D) representations of the world, current video understanding models struggle to extract these dynamic semantics, arguably because these models use cross-frame reasoning without underlying knowledge of the 3D/4D scenes.
In this work, we introduce **DynSuperCLEVR**, the first video question answering dataset that focuses on language understanding of the dynamic properties of 3D objects. We concentrate on three physical concepts—*velocity*, *acceleration*, and *collisions*—within 4D scenes. We further generate three types of questions, including factual queries, future predictions, and counterfactual reasoning that involve different aspects of reasoning on these 4D dynamic properties.
To further demonstrate the importance of explicit scene representations in answering these 4D dynamics questions, we propose **NS-4DPhysics**, a **N**eural-**S**ymbolic VideoQA model integrating **Physics** prior for **4D** dynamic properties with explicit scene representation of videos. 
Instead of answering the questions directly from the video text input, our method first estimates the 4D world states with a 3D generative model powered by a physical prior, and then uses neural symbolic reasoning to answer the questions based on the 4D world states.
Our evaluation on all three types of questions in DynSuperCLEVR shows that previous video question answering models and large multimodal models struggle with questions about 4D dynamics, while our NS-4DPhysics significantly outperforms previous state-of-the-art models.

Published as a conference paper at ICLR 2025
COMPOSITIONAL 4D DYNAMIC SCENES UNDERSTANDING WITH PHYSICS PRIORS FOR VIDEO QUESTION ANSWERING
Xingrui Wang', Wufei Ma', Angtian Wang', Shuo Chen²
Shuo
Adam Kortylewski Alan Yuille'
1 Johns Hopkins University 2 Tsinghua University
3 Max Planck Institute for Informatics UUnivesitty of Freiburg
ABSTRACT
For vision-language models (VLMs), understanding the dynamic properties of objects and their interactions in 3D scenes from videos is crucial for effective reasoning about high-level temporal and action semantics. Although humans are adept understanding these properties by constructing 3D and temporal (4D) representations of the world, current video understanding models struggle to ex- tract these dynamic semanties, arguably because these models use cross-frame reasoning without underlying knowledge of the 3D/4D scenes. In this work, we introduce DynSuperCLEVR, the first video question answering dataset that fo- cuses on language understanding of the dynamic properties of 3D objects. We concentrate on three physical concepts - velocity, acceleration, and collisions within 4D scenes. We further generate three types of questions, including factual queries, future predictions, and counterfactual reasoning that involve different aspects of reasoning about these 4D dynamic properties. To further demonstrate the importance of explicit scene representations in answering these 4D dynamics questions, we propose NS-4DPhysics, a Neural-Symbolic VideoQA model inte- grating Physies prior for 4D dynamic properties with explicit scene representation of videos. Instead of answering the questions directly from the video text input, our method first estimates the 4D world states with 3D generative model powered by physical priors, and then uses neural symbolic reasoning to answer the questions based on the 4D world states. Our evaluation on all three types of questions in DynSuperCLEVE shows that previous video question answering models and large multimodal models struggle with questions about 4D dynamics, while our NS- 4DPhysics significantly outperforms previous state-of-the-ar models, Our code will be available at http://hhhbbbbooom//iiii////////
INTRODUCTION
Visual question answering (VQA) is comprehensive task to assess how well machine learning models can identify objects, understand their relationships, and perform reasoning with multimodal input. When it comes to video question answering (VideoQA), models must not only capture the static features in frames but also understand temporal dynamics, such as object movements and interactions over time, especially in 3D space. Despite the recent progress in multimodal foundation models, understanding these dynamic features remains challenging (Ma et al. 2024).
series of studies in cognitive science (Hamrick et al.) 2016 UUlman et al. 2018) has found that humans excel understanding the dynamics and interactions in the physical world. This enables humans to understand the spatial and temporal relationships between objects, predict future object states and interactions, and ultimately perform complex tasks such planning and manipulation in the 3D world. For video question answering, it is also important to incorporate these dynamic features integral part of video understanding.
However, existing video question answering datasets have primarily focused on high-level temporal semantics, such human activities (Fabian Caba Heilbron & Niebles 2015 Goyal al. 2017).

---

Published as a conference paper at ICLR 2025
Input Videos
Questions
Answers
Framee
Frome-100
Question How fast the SUV moving compared to the yellow bus?
Slow
(a)
Factual Question
What the color the which collides with the SUV?
Green
Trann
Predictive Question
(b)
Will the red bus collide with the ycllow object?
100
Counterfactual Question
(c)
Would the bus collide with the minivan if it were standing still from boginning?
No
Figure 1: We propose DynSuperCLEVE to study the 4D dynamics properties of objects and their collisions. We also design three types of questions. Factual questions and counterfactual questions will take the whole 120 frames as imput, while the predictive questions will take the first 30 frames.
due a lack of 3D/4D annotations. Such evaluations are often susceptible to biases and shortcuts in natural videos, so models may perform quite well without truly understanding key dynamic properties (Ma al. 2024). Another line of work exploits synthetic environments and studies how models understand dynamic and physical properties, given 3D ground truth obtained from simulators (Yi et al. 2019 et al. 2022 Bear et al. 2021 Tung et al. 2023 Ates al. 2022
Patel et al. 2022 Girdhar & Ramanan 2020: Zheng al.) 2024). However, these datasets often consider only simplified settings, either lacking diverse physical properties in simulations (Ates
et al. 2022 Yi et al. 2019) or lacking language integration (Bear et al. [02T Tung al. 2023).
Moreover, these datasets employ rudimentary rendering with clear backgrounds and toy-like 3D) assets, creating significant domain gap between synthetic videos and real-world natural videos. While many conceptual findings have been derived from experiments on these datasets, these results lack practicality for understanding complex physical events in the real world. Without realistic rendering and natural language support. these datasets are also not suitable for analyzing the 4D dynamics understanding of recent large multimodal models.
To quantitatively study 4D dynamics understanding and to analyze the limitations of current video un- derstanding models, we introduce DynSuperCLEVE the first video question answering (VideoQA) dataset that focuses on the language understanding of the dynamic 4D properties of objects. Specifi- cally, our dataset studies three crucial dynamic properties of 3D objects velocities, accelerations, and collisions, and designs natural language questions for three types of reasoning, i.e., factual, future prediction, and counterfactual. Notably, by introducing acceleration, we achieve a more complex physical scene compared to the previous data generator (Greff et al.) (022). To address the synthetic-to-re ro-rell domain gap in previous video benchmarks (Yi al. 2019; Tung et al., 2023) and enable zero-shot evaluation of large multimodal models our videos by introducing diverse appearances of objects and backgrounds with new textures.
dataset, we improve the realism of
To demonstrate the importance of explicit scene representation in answering these 4D dynamics questions, we present NS-4DPhysies, neural-symbolic model that reasons about these dynamics by first estimating explicit 4D scene representations. Our NS 4DPhysisss consists two key modules: a 4D scene parsing module followed by a symbolic reasoning module. The 4D scene parsing module learns a robust 3D generative model (Ma et al., 2002) Wang et al. 20235) of the scene and captures the dynamics of objects with a physical prior. With 3D generative representation of the scene, our scene parsing module is robust to partial occlusion and generalizes well to unseen 3D poses and object appearances during training, which are very common natural videos.
We test a wide range of state-of-the-art VideoQA models on our DynSuperCLEVR dataset, including neural symbolic models (Yi et al.) 2019 Wang et al. 2024), standard end-to-end models (Perez et al.) 2018), large multimodal models (Xu et al. 2024 Lin et al. 2023). and the proprietary model GPT-4o. Results show that these models struggle to the 4D scenes due to a lack of explicit knowledge of the 3D world or the 3D dynamics of objects. Meanwhile, our NS-4DPhssics outperforms previous state-of-the-ard models by wide margin across
challenging questions about dynamic properties of
2

---

Published as a conference paper at ICLR 2025
different dynamic properties and types of reasoning questions. This demonstrates the importance of developing an explicit 4D dynamics representation for multimodal agents to understand physical events presented in the videos and to foresee upcoming events.
Our contributions are as follows: (1) We present DynSuperCLEVR, the first VideoQA dataset that focuses on language understanding of the dynamic properties of objects (velocity, acceleration) and multi-object interactions (collisions). (2) We propose NS-4DPhysics, a neural-symbolic model that first reconstructs the 4D scene with dynamic 3D generative model with physical priors as the perception module and then reasons about dynamics over the explicit 4D scene representation. (3) Extensive results on different settings of DynSuperCLEVR reveal key pitfalls of state-of-the- art VideoQA models, including those powered by LMMs and industrial training data. We further demonstrate that NS-4DPhysics outperforms other VideoQA baselines by a wide margin, showing the advantages of learning explictt 4D dynamic representations.
RELATED WORK
Video question answering. Video question answering (VideoQA) is a challenging task because models must not only detect and identify objects from static images but also track and infer objects' changes and interactions over a sequence of frames. A number of VideoQA datasets annotate question-answer pairs on natural videos (Xue et al., 2017:] Yang et al., 2022 Majumdar et al. 2024] Wu et al. 2024). However, these datasets are not suitable for studying dynamic physical properties, as natural videos contain limited object interactions. Another line of work focuses on physical reasoning
in simulated environments (Yi et al., 2019; Chen et al.) 2022 Ding et al. [2021 Tung et al. 2023)).
However, these datasets are either built in restricted settings Yi et al.] al.] 2019 Chen et al. (022)) or lack real dynamics and forces common in the real world (Tung et al., (023). In order to further challenge and explore the limitations of current video-text models on dynamics reasoning, we develop DynSuperCLEVR with improved realism for both video quality and physics simulation.
Video-text models. With the availability of web-scale image-text or video-text paired datasets (Schuhmann et al. 2022, Bain et al., 202T). recent video-text models (Wang al.) [002]] Lin et al.) | Xu et al.) 2024) have adopted heavy multi-modal pretraining and achieved improved results on a wide range of video understanding tasks. However, these models often exploit biases and shortcuts for dynamic reasoning. failing to capture the physical intrinsics that drive object movements and interactions. Our NS-4DPhysics incorporates a 4D dynamic scene representation in our scene parser, allowing compositional reasoning of various physical events in the video sequence.
Physical scene understanding. Understanding the physical events and inferring the future state requires model to apply physical principles to explain the observed events or simulate the future outcomes. Previous studies explored physics engines for simulation and learning (Kadambi et al.) 2023| Battaglia et al., (013), or integrating differentiable physies engine deep learning models (Wu et al. 2015 2017 Kyriazis& & Argyros 2013). Other works learn physical properties from compositional scene (Hamrick et al. 2016)] Ullman et al. 2018 Ding et al. 2021 Chen et al. 2022 Zheng et al., 2024). However, previous methods were limited to simple scenes and we extend our scope to more realistic scenes with real-world objects and complex simulations, enabling physical scene reasoning with our dynamic 3D compositional reasoning.
3D generative models. Our model is built on top of previous 3D generative models for image classification (Jesslen et al. 2023), 6D pose estimation (a al., 2022), and 3D-aware visual question answering (Wang et al. (223a). In image understanding, these generative models learn compositional representation
3D objects with feature activations on vertices, conducting the scene understanding tasks with silissss----sshssiiissssss As found in Wang et al. (2023a). the advantage of these methods is the robustness in recognizing 3D objects from scenes, compared with the discrimi- native baseline such Faster R-CNN (Ren et al. 2016). We extend the previous 3D compositional models to the video understanding task, which learns 4D dynamic scene representation from video. Such representation allows us reconstruct world states at each timestep, analyze the trajectories of objects, and reason about the physical events.
3 DATASET
To investigate 4D dynamic properties in VideoQA tasks, we introduce the DynSuperCLEVR dataset. This synthetic dataset provides fully annotated 4D scene structures for videos. along with the reasoning steps to answer corresponding questions. We concentrate on the 4D dynamic properties of objects - vel l occit ty and accel aeceleratiin eration in 3D scenes and their comparison. We also introduce
3

---

Published as a conference paper at ICLR 2025
Objects
Question: fisst moving compared
Statio
yellow bus?
Shape
Color
Answer: Slow
Realistic material for with diverse color.
Dynamies
3D position
seleer Hooe
3D pose
Velecity
+Stnne]owr)
valociity
Wuadlty
Accelenation
Filter(yxsiow
Fasterraan)
Velecity
Collisions
Frune I0
Obj. A
Ohjj
(b)
Environment map dome shape as buckground.
(c) Strueture annotations objects and events.
Example of program exccution for questions answering on dymanical properties
An illustration of the construction of DynSuperCLEVR. (a) We use the same 3D object meshes from SuperCLEVR but generate more realistic textures for different colors. (b) The back- ground is created by mapping real image environment map onto dome shape. (c) Our video data is fully annotated with 4D dynamic scene structure, containing static and dynamic properties for objects and collision components. (d) For each question, we design new operation programs for 4D) dynamic properties, which can be executed on the scene structure to answer the questions.
Figure 2:
the colli wilision sion event in 3D space, which more challenging than in the 2D ground plane. The collision demonstrates how the dynamic properties will affect the interactions among objects.
As an advanced video reasoning task, we design questions across three levels - (1) factual ques- tions, which query properties and events directly from the visible frames, (2) predictive questions, which require forecasting future collisions based on the current 4D dynamics and scenes, and (3) counterfactual questions, which challenge models to imagine how changes in the 4D dynamics would alter the outcomes.
In this section, first introduce the three key components for designing the video data in our proposed dataset: (1) the design of more realistic textures for objects and backgrounds (.1. (2) the 4D dynamical scene annotations involving 4D dynamic properties for objects (Sec (3.2p and their collision events, and (3) the rendering of the video and physical simulation process (.3) Then, in Sec 3.4 we describe our three types of questions different reasoning levels and the reasoning program for question answering.
3.1 3D SCENE CONSTRUCTION
3D Objects. WWe use the same 3D objects from SuperCLEVR et al. 2023), which include five vehicle categories (car, plane, bicycle, motorbike, bus) with 21 sub-types, but we regenerate the textures for each mesh model improve realism. For each color label (gray, red, brown, yellow, green, cyan, blue, and purple), we create 6 variants and change the face colors of the object meshes. Based on the part annotation in SuperCLEVV, only the primary body receives color edits, while others are updated with natural elements like black wheels, transparent windows, red tail lights on cars (see Fig. 2). More examples of all textured 3D objects are provided in the Appendix
Backgrounds. Compared with the previous synthetic dataset (Yi et al. 2019 Li et al. 2023), change the background to highly realistic images and lights. As shown in Fig. 2 (b), we use real HDRI images"] which contain real captured images with environment maps in 509 different scenes. We map the image as material onto a dome shape as the background to create 3D view.
3.2 4D DYNAMICS SCENE ANNOTATIONS
After defining the shape and color of objects as static attributes, we assign them dynamic properties, allowing them to move and interact within 3D physical world. Each object is defined by its 3D position, pose, velocity, and acceleration, as described below (see Fig. c)).
3D Position and pose: The 3D position (zr, y, 2) records the exact coordinates of the object each timestep, specifying its location within the scene. The pose (a, y) describes the object's orientation and rotation in 3D space, defining how the object aligned relative to its surroundings.
https://pel com/hdris
4

---

Published as a conference paper at ICLR 2025
Velocity. The velocity (Ug, uy, Uz) represents the change in the object's s 3D position time, capturing both the speed and direction of motion. As one of the properties included in the question answering. we define three velocity states: static (0 m/s), s slow (< 3 m/s), and fast (2 3 m/s) for language description. Given that the exact velocity is often difficult to determine from video alone, we ensure that each scene includes at least object moving within each velocity state provide clear distinctions.
Acceleration. The acceleration (Aa, ay. az) represents the rate of change of velocity over time in 3D space. Similar to the label for velocity, we define two types of motion for acceleration. Along the x-axis, objects can either accelerate due an internal engine force or decelerate due to friction. In the z direction, the objects can either maintain a constant height (zero accelcration) or experience downward acceleration due to gravity. our dataset, we simplify into a binary label: accelerating or not, and fl Float ng or not.
Collision Events. Due to perceptual illusions, reasoning about collisions in 3D space more challenging than in a 2D plane. Each collision involves two objects, and we record the frame ID when the collision occurs. The collision between objects a direct result of their 4D dynamic properties. In our dataset, we capture not only the collision events in given frame but also future collisions from simulation and counterfactual scenarios, where changes in the 4D dynamic properties at the beginning lead to question types, which will be introduced in Sec. 3.4
sequence of collision events. This forms the foundation for the three
3.3 PHYSICAL SIMULATION AND VIDEO GENERATION
The video data rendered using a combination of the physics engine PyBullet and the Blender renderer. Building the generation pipeline of Kubric (Greff et al. 2022), we integrate support for the new acceleration feature. At each timestep, we capture the 3D position, pose, velocity, and acceleration of all objects, along with all collisions between objects, described above.
Each scene has a corresponding counterfactual version. We re-simulate the scene, randomly selecting one object's 4D dynamic property (velocity acceleration) modify its initial value. We then record the new property values and the resulting sequence of collision events caused by the changes.
3.4 QUESTIONS GENERATION FOR 4D DYNAMICS PROPERTIES
Following previous video question answering datasets Yi al. 2019; Chen al. 2022). we develop question templates and generate three types of questions for factual, predictive, and counterfactual reasoning in our DynSuperCLEVR (see Fig. TT. Each question is paired with a corresponding program execution for reasoning (see Fig. d)). Below, describe the design of the three question types, while the full list of programs and more examples of program execution can be found in the
Appendix C
Factual Questions. Factual questions focus on the direct understanding of static and dynamic properties of objects, their comparisons, and collision events within 3D space in given frames. For newly proposed 4D dynamic properties, we define questions that query the velocity and acceleration states of objects or compare their velocities 3D space at specific moments. These moments are either defined by the start of frame or the point when collision event occurs. The questions about collisions involve predicting whether two objects have collided or identifying which objects are involved in collision. We introduce new programs query sitrryyelloctty compare compare_velocity velocityy and www._ooooinnddirrddeeoe for velocity; query_ query_accelerativgg and query_floating for acceleration.
Predictive Questions. Predictive questions require forecasting future collision events that are not present in the video. our DynSuperCIEEVR, it is crucial for the model first understand the positions, velocities, and accelerations of objects in 3D space from the video, and then apply reasoning to predict their future trajectories.
Counterfactual Questions. Counterfactual questions challenge the model to reason about hypo- thetical scenarios by changing the initial 4D dynamic properties of objects. This requires the model to have strong understanding of how changes in dynamic features (velocity acceleration) can affect the world states and cause collisions, and to apply these to re-simulate the potential outcomes contrary to the video input.
5

---

Published as a conference paper at ICLR 2025
Input Video
1.30-MMMs sere
hound Mesh
4D Dynamies Scene Representation
Estimations
predictive Simulation
Compari
Scene Likeltood
State T
or
CNN
Image feature
Counter-
Physical Prior
factual
State T
Physies
State
Engine
Question parue
Q: How fast the SUV moving compared to the yellow bus?
Question
Answer: Slow
Program ceccution
Figure 3: Our NS-4DPhysics has three main components. I: A 3D neural mesh scene parser, which combines the rendering likelihood with a physics prior parse the video into a 4D dynamic scene representation. II: The future states or counterfactual states can be simulated with the reconstruction result by the physics engine. III: A question parser that processes questions into reasoning programs and then executes the program over the predicted scene representation to answer the questions.
3.5 DATASET STATISTICS
We generate 1,000 video clips for training, 100 for validation, and 100 for testing. Each video is 2,000 ms long with frame rate of 60, resulting in 120 frames per scene. In total, we create 7,850 factual questions. 2,750 predictive questions, and 989 counterfactual questions. Factual questions are open-ended and can be answered with few words, while predictive and counterfactual questions are structured as true or false, requiring the model to determine if events will oceur.
MODEL
In this section, we introduce NS NS-4DPhysics, neural symbolic VideoQA model for solving questions about object dynamies 3D space. As illustrated in Fig. the model combines 3D-aware scene parser with probabilistic physical prior module to convert video sequences into an explicit 4D scene representation. This explicit representation enables interpretable reasoning, allowing the model to predict future or counterfactual world states through physics simulation. For question answering, employ question parser first convert the questions into executable programs, which are then used to answer the questions step by step based on the predicted 4D scene representation.
In the following sections, introduce the 4D symbolic scene representation in Section 4.1 the scene parser with the physical prior in Section 4.2 future and counterfactual simulation in Section and the language model and reasoning program in Section 4.4
4.1 4D SYMBOLIC SCENE REPRESENTATION
The 4D dynamics scene representation is the world states used to reconstruct the video scene annotation shown in Fig. p.. For each object O, we define its static attributes as shape s² and color c². The dynamic properties at each timestep consist of 3D position T,, rotation (pose) R, velocity its and acceleration at, where each is 3D vector. For an input video, each image frame timestep t is denoted as Z, The 4D symbolic scene representation S is defined as a sequence of object states number of objects the scene.
S. (0), Of ON over time, where
4.2 DYNAMIC SCENE PARSER WITH PHYSICS PRIOR
The scene parser takes the videos input and estimates the 4D dynamic world state S at time step This involves predicting the 6D poses of objects in frames, predicting their temporal properties (e.g., velocity and acceleration), and reasoning about collision events. In this work, we adopt 3D generative model that learns a feature representation of the scene and predicts 3D world states from
analysis-by byssytthssis perspective. We extend the work of Ma et al. (2022); Jesslen et al. (2023) to natural videos by employing a sequence decoding approach and adding physical prior in our 4D scene representation for the model to understand real-world physical events.

---

In our NS-4DPhysics, a is the 3D position TІ and rotation Rg of each object for time step 4, which estimated independently for each video frame (i.e., at the image level). The training phase learns Published as a conference paper at ICLR 2025
3D-NMMs Scene Parser 3D neural mesh models (3D-NMMs) (Ma et al. 2022 Jesslen et al. 2023 learn a generative model to produce the 3D feature representation of the objects and parse the scene with seallyiis-b-syytheest
For each object shape, we train object mesh M, {u, R' /==1 with neural texture T. (/ E } i==1. where is the object category (shape), NN is the number of vertices, and c is the feature dimension. 3D-NMM can generate the 3D-aware scene representation by rendering the neural mesh model O. (M..2 T.) in the given 6D pose a: Fa R(O,, a:) R x W xc, with soft rasterization (Liu et al., 2019). By comparing the rasterization of 3D features and the 2D image feature, poses a, or estimate a and 8 during inference. We formulate this render-and-con optimization of the likelihood model:
learn the neural textures with the ground truth category and 6D process as an
p(F Oz Cgg B)= II p(f 1 a.) II P(f B) SEBG
(1)
iEFG
where FG and BG are the set of foreground and background locations the 2D feature map and fa is the feature vector of F' at location i. Here, the foreground and background likelihoods are modeled as Gaussian distributions.
the CNN extractor for feature F, the neural mesh Os, and the background feature B jointly (see Appendix D.I for details). During inference, by maximizing the likelihood function in Eq. G each frame, we can obtain the Tj and RG for the predicted objects.
The dynamical 3D world states predicted from the image frame do not consider the consistency in the time dimension. As our algorithm predicts S, for each time step sequentially by maximizing the likelihood function in Eq. 6 one strategy to initialize St with the predicted S,- in gradient descent. However, this only fits the case when the objects are moving uninterruptedly. In complex dynamical 3D scenes, the trajectory of objects can change dramatically due to their interactions, such as collisions. These require comprehensive prior knowledge of the physical world.
Physical Prior
It's nontrivial to directly model the physical functions. Many previous studies incorporate physical engines within neural networks, but only apply to simple shapes like cubes, as the complex shapes of 3D objects and the interaction process are hard to model. On the other hand, computational physical engines (Coumans & Bail 2016) excel at modeling physical functions for any known shapes but are not differentiable and are challenging to integrate into the de-rendering process during inference.
In method, modify discriminative physics engine PE(-) into probabilistic model by assuming (Re, T:) PE(Ai,I,,P-- + after obtaining Rt and T- Here, we take (000,=²) o²)). Inspired by the idea from Bayes-Kalman filtering (Salzmann & Urtasun] (001)) we integrate physics prior of (Re, Ti) into the likelihood function as a correction term after Eq. Formally, the prior q(e,, Te. R can be computed as:
where μ = PE(R, f.-), and C= 1//
Finally, the rotation RG and translation T. can be estimated by maximizing the joint likelihood of the rendering likelihood in Eq. T and the physical prior likelihood in Eq.
R f. arg max p(F Oyy Tú, RG, B) R.T.
R fT-1).
Dynamics Attributes. After estimating the world states (R,, TT), we compute the dynamics attributes Velocity and Accelerat inttion ion by calculating the differences between consecutive translations and velocities over time. To reduce noise, a moving average filter with a window size of 5 is applied.
Static Attributes. Shape is determined by selecting the mesh model category with the highest likelihood (Wang et al., 2024). For Color prediction, we crop the object's region from the RGB image and train an additional CNN classifier for color recognition.
Collisions. Collisions are obtained from the physics engine PE(-) at the time of estimating the physical prior. For each collision, record the time and objects involved.
7

---

Published as a conference paper at ICLR 2025
4.3 FUTURE AND COUNTERFACTUAL SIMULATION
From the explicit 4D scene representation, we can easily simulate the future or counterfactual states of the objects. For either case, we apply the physics engine and assign the predicted translation, rotation, velocity, or counterfactual condition to all objects as initial conditions and simulate. We visualize the re-simulation results in Sec. 5.4
4.4 QUESTIONS PARSING AND PROGRAM EXECUTION
After obtaining the 4D scene representation from the scene parser, we can parse the question into reasoning programs, which are then executed on the scene representation to predict the answer. The question parser follows previous work (Yi et al. 2019; Chen et al. (022), where an LSTM sequence-to-sequeene model trained parse the question into its corresponding program.
EXPERIMENTS
5.1 EXPERIMENT SETUP
Baseline Models. We select the representation models from the following three categories as baseline models on DynSuperCLEVR. (1) Simple classification-base methods: CNN+LSTM and FiLM. We extract frame-level features and average them over the time dimension. We encode questions with the last hidden state from an LSTM and then concatenate the two features to predict answers. (2) Neural symbolic models: NS-DR and PO3D-VQA, which represent the model with explicit 2D/3D scene representation for question answering, compared with our 4D representation. (3) Video large language models (Video-LL.Ms). including a fine-tuned Intern Video, and Video-LLaVA, PLLaVA. GPT-4o for zero-shot evaluation with in-context learning. Please refer the appendix for detailed implementations.
NS-4DPhysics Implementation. The dynamic scene parser and CNN classifier are trained on the images, using 120k images for training. The dynamic scene parser uses ResNeXT as the feature extractor and is trained for 50 epochs with batch size of 4 4 GPUs. The attributes classifier uses ResNet50 and is trained with the cropped images from the ground truth bounding box. During inference, we the variance of the physical prior
5.2 VIDEO QUESTION TION ANSWERING RESULTS
We first compare the NS-4DPhysics with baseline models for the VideoQA task of DynSuperCLEVR. The results are shown in Tab. [ For GPT-4o, we also prompt GPT-4o to first describe the video, then reason step-by-step before answering questions (GPT-4o+rcasoningg)
Table I: Performance on the DynSuperCLEVR testing split for each question type: factual, predictive, and counterfactual. Factual questions are further divided into sub-types: Velocity, Acceleration, and Collision, with "All" representing overall accuracy. The average is taken as the overall accuracy across the three question types. indicates GPT-assisted zero-shot evaluation. The highest performance among all baseline models is indicated by underlined values.
Average
Factual
Predictive
Counterfactual
All
Vel.
Acc.
Col.
CNN+LSTM
48.03
40.63
41.71
56.79
25.37
56.04
47.42
FiLM (Perez al. 2018
50.18
44.07
48.58
53.09
26.87
$4.94
51.54
NS-DR (Yiet 2019
51.44
51.44
55.63
46.34
46.86
PO3D-VQX Wang
2024
62,93
61.22
62.21
73.17
51.20
65.33
62.24
InterVVidoo Wang al.
2022
52.62
51.07
59.29
49.08
36.06
54.74
59.18
Video-I.LaVA Lin et
2023
38.09
37.04
37.62
52.76
23:56
38.78
40.88
PLLaVA Xu al 2024
59.24
54.61
55.00
63.80
46.63
67.52
73.47
GPT.46"
51.59
50.82
51.19
57.67
44.71
54.38
50,00
GPT-4o reasoning
56.06
55.50
58.81
57.67
47.12
56.93
58.16
NS-4DPhysies
82.64
87.70
88.66
83.73
88.46
85.71
74.51
Comparison with Blasieication-Baseed Methods. CNN+LSTM only reaches 40.63%, 56.04%. and 47.42% for factual, predictive, and counterfactual questions, while FiLM achieves 44.07%, 54.94%, and 51.54%. The performance of these models is notably lower than that of NS-4DPhysics, which achieves 87. 87.70%, 85.71% and 74.51% for factual questions, respectively. As classificattion based methods generally rely extracting features through CNNs, demonstrate significant advancements in handling dynamic content with explicit scene representation.
8

---

Published as a conference paper at ICLR 2025
Comparison with the Neural-Symbolic Methods. The comparison between the Neural-Symbolle models shows significant performance improvement as the dimension of scene representation in- creases. PO3D-VQA constructs explicit 3D scene representation, leading to better estimation of the objects' positions and poses, which is beneficial for inferring the dynamics and interactions compared with the 2D presentation in NS-DR (9.78%) for factual questions. By pushing the dimension into 4I) and considering the physical prior knowledge, our NS-4DPhysics achieves 82.64% overall accuracy, which is 19.71% % higher than PO3D-VQA.
Comparison with Video-LLMs. Although Video-LLMs demonstrate strong video understanding and generalization abilities, they underperform on DynSuperCIEVR. In zero-shot settings, Video- LLaVA achieves an overall accuracy of 38.09%, while PLLaVA performs better at 59.24%, excelling in predictive and counterfactual reasoning among the baselines. Even with fine-tuning, Intern Video reaches only 52.62 22,, lagging behind our NS-4DPhysics, These results indicate that existing video foundation models struggle reason effectively complex dynamic scenes.
5.3 ANALYSIS
We further analyze the effectiveness two key components of NS-4DPhysics: the physics prior and the symbolic reasoning model. (1) Without the physics prior, the 4D world states are estimated using only the rendering likelihood, with time consistency maintained by initializing S, from the previous frame, S,-1. The result in Table shows
shows that adding the physics prior to the 4D scene representation significantly improves VideoQA performance. We also visualize the predicted world states of both models in Sec. [5.4 with the failure cases. (2) We also compare a variant of GPT-4o, where, language model, processes the predicted scene structure from NS-4DPhysies as text input instead of raw images. However, the LLM still underperforms in 4D dynamic reasoning, highlighting the advantages of our symbolic reasoning model in capturing dynamic object properties.
Table 2: Analysis of the physics prior module and symbolic reasoning (SR) in our model. We compare variants without the physics prior and augment LLM with explicit 4D scene representation.
Factual
Average
Predictive
Counterfactual
All
Vel.
Acc.
Col.
4D Representation SR (Ours)
82.64
87.70
88.66
83.73
88.46
85.71
74.51
w/o Physics Prior SR
75.97
79.68
81.40
81.30
74.88
78.83
69.39
Representation + GPT-4o
61.39
65.49
66.67
54.60
71.63
57.14
51.09
Video GPT-4o
56.06
55.50
58.81
57.67
47.12
56.93
58.16
5.4 QUALITATIVE RESULTS
We visualize how our model achieves more accurate results in reconstructing scenes and simulating future scenes for predictive questions, assisted by the physies prior. Fig. 4 4sshws an example of factual question. Without the physics prior (row c). the model fails to track the correct position of the minivan and provides the wrong answer regarding the collision. Fig. illustrates predictive
Question:
Inpur
green minivan with the eyan thing in the given frunes?
Answer: Yes
Ours:
())
Prediction:
wio plysics
wio phystes:
X
Prodiction:
Figure Qualitative examples of factual questions. (a) shows the input video; (b) Our NS-4DPhysies provides a better estimation for the motion with the physical prior. (c) The error in position predicted by the baseline w/o physics in the red box leads to a mistake in the answer.
question. Our model predicts more accurate positions for all the objects (b1), and the simulation of future frames in (b2) shows no collisions. However, without the physics prior, the error in 3D position prediction (especially for the school bus) causes all the vehicles to collide in future frames.

---

Published as a conference paper at ICLR 2025
Inpet Frames
Ground Truth Future Frimes
Will rod SUV collide with the school bus nex??
Jnpu
Answrr:
Ours:
(b1)
Ours
(b2)
Predictions No
wio plasics
physies:
(e1)
(o7)
Prediction: Yes X
Reconstruction
Figure 5: Qualitative examples of predictive questions. (al) The first 30 frames are given to models as input video, and (a2) the following frames are hidden as ground truth future states; (b1) our NS-4DPhysics has better estimation of the poses of objects, and (b2) provides plausible simulation by re-simulation. (cl) The red box shows the error in pose estimation of the bus when lacking a physics prior, which (c2) causes the red SUV to collide with the school bus in the future.
Analysis of Video-LLM. To analyze the reasoning failures in Video-LLM, we examine two failure cases PLLa VA and GPT-4o. (a) Misunderstanding the location and collisions in 3D space: Both PLLaVA and GPT-4o incorrectly predict that the truck collides with the fighter jet. In reality, the truck only collides with the purple articulated bus, while the plane is near the truck but does not make contact. This error results from relying 2D projection, which fails to capture the correct 3D spatial relationships. (b) Failure to predict future 4D dynamics world states: Both models identify the objects correctly but fail to understand the objects' motion from the given frames. Although no collision occurs in the current frames, the bus moving toward the airplane and will collide in the somethig the models are unable to predict.
Question: What shape thing which collides the truck in the given frames?
Question: Will the double hus collide with the nirliner next?
modd
The ebject that collides lunck gives shaped plane.
PLIaVA:
airplanc, bicycle and surfice and there is no indication of any collisoon betwee them.
GPT-4:: Rased as the images provited, the object that collides with
double currens ftames, it is mollear whether the bus moving towands the airiner not. Yes
appears
positioned near
lus the shape a fibhier jet ciore
Ours: Ameulated
(a) Misunderstand the location and collisions in 3D space
(b) Fail to predict future 4D dynamics world states
Figure 6: Two failure cases of VA and GPT-4o. (a) Both models incorrectly predict truck colliding with fighter jet due to misunderstanding 3D spatial relationships, likely caused by reliance on 2D projection; (b) While the models correctly identify objects, they fail to predict future collisions due to a lack of understanding of the 4D dynamics of objects from the given frames.
6 CONCLUSION AND DISCUSSION
In this work, explore 3D dynamic properties in video question answering. We introduce DynSu- perCLEVR to study the 4D dynamics of scenes in VideoQA tasks with improved realistic textures. Our findings show that existing video-language models, even those with large-scale pretraining, struggle to capture critical 4D dynamic properties necessary for temporal and predictive reasoning. To address this, we develop NS-4DPhysics, neural- -ymbolly model that estimates an explicit 4D scene representation and answers questions through program execution. Experimental results on DynSuperCLEVR demonstrate that our approach significantly outperforms previous state-of-the-an methods, highlighting its effectiveness in inferring dynamic properties and predicting future states. In ongoing work. as Kaushik et al. (2024) has demonstrated the potential of 3D NMMs for domain adaptation, are studying how NS-4DPhysics can be extended work with real images. Limitations. As a synthetic VideoQA dataset, our DynSuperCLEVR currently focuses only on rigid objects with linear velocity and acceleration. More complex objects, such as articulated and non-rigid objects with complex trajectories, are not yet considered.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT This work is supported by ONR with award N00014-23-1-2641, from ARL Army Research Lab- oratory with award
2812. Adam Kortylewski acknowledges support via his Emmy Noether Research Group funded by the German Reserach Foundation (DFG) under Grant No. 468670075. We appreciate the anonymous reviewers for their valuable feedback. We also thank Zhuowan Li and Qing Liu for their insightful discussions during the early stages of this project.
REFERENCES
Tayfun Ates, M. Ateşoğlu, Çagatay Yigit, llker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret. CRAFT: A benchmark for causal reasoning about forces and inTeractions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguisties: ACI. 2022, pp. 2602-2627, Dublin, Ireland, May 2022. Association for Computational Linguistics, doi: 10. 1/23 findinng--nnal.2255. URL https: https:://aclantooloryy wwtp:yyy..ggg.ooooo/o/// Findings acl. 205
Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. Coke: Contrastive learning for robust keypoint detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 65-74, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110445):1322-11332. 2013.
Daniel M Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao- Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Yun Sun, al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.0820 2021.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. arXiv preprint arXiv:2205.01083 2022.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances In Neural Information Processing Systems, 34:887-899, 2021.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for Compositional Actions and
TEmporal Reasoning. In ICLR, 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji. Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi Kubric: a scalable dataset generator. 2022.
11

---

Published as a conference paper at ICLR 2025
Jessica B Hamrick, Peter W Battaglia, Thomas L Griffiths, and Joshua Tenenbaum. Inferring mass in complex scenes by mental simulation. Cognition, 157:61-76, 2016.
Artur Jesslen, Guofeng Zhang, Angtian Wang, Alan Yuille, and Adam Kortylewski. Robust 3d-aware object classification via discriminative render-and -ooomprree. arXiv preprint arXiv:2305. 2023.
Achuta Kadambi, Celso de Melo, Cho-Jui Hsieh, Mani Srivastava, and Stefano Soatto. Incorporating physics into data-driven computer vision. Nature Machine Intelligence, 5(6:552--888 2023.
Prakhar Kaushik, Aayush Mishra, Adam Kortylewski, and Alan Yuille. Source-free and image- only unsupervised domain adaptation for category level object pose estimation. arXiv preprint arXiv:2401.10844 2024.
Nikolaos Kyriazis and Antonis Argyros. Physically plausible 3d scene tracking: The single actor hypothesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9-199916622 2013.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963-14973 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012, 2023.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7708-7717, 2019.
Wufci Ma. Angtian Wang, Alan Yuille, and Adam Kortylewski. Robust category-level 6d pose estimation with coarse-to-fie rendering of neural features. In European Conference on Computer Vision, pp. 492-508. Springer, 2022.
Wufei Ma, Kai Li. Zhongshi Jiang. Moustafa Meshry, Qihao Liu, Huiyu Wang, Christian Häne, and Alan Yuille. Rethinking video-text understanding: Retrieval from counterfactually augmented data. arXiv preprint arXiv:2407. 13094, 2024.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. OpenEQA: Embodied Question Answering in the Era of Foundation Models. In CVPR, 2024.
Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Cripp-vqa: Counterfactual reasoning about implicit physical properties via video question answering. arXiv preprint arXiv:2211.03779, 2022.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6):1137-1144 2016.
Mathieu Salzmann and Raquel Urtasun. Physically-based motion models for 3d tracking: A convex formulation. In 20/ International Conference on Computer Vision, pp. 2064-2077 2011, doi: 10.1 10.1100// 1.6.01111.11100 1.612648..
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open lcrge-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25222 255944, 2022.
12

---

Published as a conference paper at ICLR 2025
Fish Tung, Mingyu Ding, Zhenfang Chen, Daniel M. Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith Fan, and Kevin A. Smith. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties. arXiv, 2023.
Tomer D Ullman, Andreas Stuhlmüller, Noah D Goodman, and Joshua B Tenenbaum. Learning physical parameters from dynamic scenes. Cogmitive psychology. 104:57-82, 2018.
Angtian Wang, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. arXiv preprint arXiv:2101.12378, 2021.
Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan Yuille. 3d-aware visual question answering about parts, poses and occlusions. In Thirrty-evvnnth Conference Neural Information Processing Systems, 2023a. URL htttps://openrevieww.ce hww.:/rutttto/urrm/mm/// i AeterrumiddAAMM..... d=
Xingrui Wang, Wufei Ma. Zhuowan Li, Adam Kortylewski, and Alan L Yuille. 3d- aware visual question answering about parts, poses and occlusions. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 58717-58735. Curran Associates, Inc., nww. ccppuiii..ccpppppppeeeeee files/ /1les//ppprr2223/ Conference pdf.
2023b. URL
fi
Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan L Yuille. 3d-aware visual question answering about parts, poses and occlusions. Advances in Neural Information Processing Systems, 36, 2024.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models generative and discriminative learning. arXiv preprint arXiv:2212.03 2022.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. arXiv preprint arXiv:2405.0977 2024.
Jiajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galilco: Perceiving
physical object properties by integrating physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
Jajjun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. Advances in neural information procesing systems, 30, 2017.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024.
Hongyang Xue, Zhou Zhao, and Deng Cai. Unifying the video and question attentions for open-ended video question answering. IEEE Transactions on Image Processing, 26(1)))665--566, 2017. doi: 10.1 10.11099/1P.201..27445
Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Learning to answer visual questions from web videos. IEEE TPAMI, 2022.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 1rXiv:19/0.01442. 2019.
Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua Tenenbaum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. arXiv preprint 46X 19, 2024.
13


Reviews:
[{"review_id": "C11ZWQLsKS", "reviewer": "Reviewer_jmWM", "summary": "The paper introduces DynSuperCLEVR, a video question answering (VideoQA) dataset that emphasizes understanding dynamic 3D object properties within 4D (3D + time) scenes. \nAdditionally, the authors present NS-4DPhysics, a model that combines neural-symbolic reasoning with physics-based priors to analyze these dynamic properties. The model first constructs an explicit 4D scene representation using a 3D generative model, followed by neural-symbolic reasoning to answer questions. \nExperimental results demonstrate that NS-4DPhysics surpasses existing VideoQA models across various question types (factual, predictive, and counterfactual), underscoring its effectiveness in reasoning about object dynamics in complex, synthetic environments.", "questions": "1. Have the authors considered extending the dataset to include articulated or deformable objects? If so, what challenges or limitations do they anticipate with this extension?\n\n2. Could the authors provide an efficiency analysis of the proposed model, including resource usage and runtime under typical conditions?\n\n3. What modifications to the NS-4DPhysics framework would make it more efficient for real-time performance or deployment in resource-limited environments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Novel Dataset:** DynSuperCLEVR is a novel dataset that focuses on 4D dynamics, addressing a critical gap in existing VideoQA datasets which typically overlook explicit physics-based scene understanding.\n\n**2. Innovative Model Design:** The NS-4DPhysics model combines 3D generative modeling with physics-informed priors, represents an innovative approach to handling dynamic 4D scene reasoning.\n\n**3. Comprehensive Benchmarking:** Extensive evaluations against baseline models, including video large language models (Video-LLMs) and other symbolic frameworks, highlight the superior performance of NS-4DPhysics in capturing 4D dynamics.\n\n**4. Future and Counterfactual Simulations:** By leveraging physics-based priors, the model excels at simulating both future and hypothetical states, demonstrating practical value and broad application potential.", "weaknesses": "**1. Synthetic Data Limitations:** While the dataset is suitable for testing dynamic properties, its synthetic nature may limit generalizability to real-world applications. Despite the authors’ efforts to improve aspects like background realism (L201), models trained exclusively on synthetic data often struggle to handle real-world noise and variability.\n\n**2. Computational Complexity:** The NS-4DPhysics model is computationally demanding due to its reliance on 3D generative modeling and physics-based priors, presenting challenges for scalability and use in resource-constrained environments.\n\n**3. Limited Object Diversity:** The dataset is limited to a narrow range of rigid objects, which may not adequately represent the complexity of real-world scenes that often include deformable or articulated objects.\n\n**4. Evaluation of Real-World Applicability:** The paper lacks an analysis of the model’s performance on real-world video data, which is essential for evaluating its practical applicability outside synthetic benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cn2Us8Nfb8", "reviewer": "Reviewer_oBU1", "summary": "This paper makes two main contributions: (1) introducing DynSuperCLEVR, a novel video question answering dataset that focuses on understanding 4D dynamics (velocity, acceleration, collisions) of objects in 3D scenes, and (2) proposing NS-4DPhysics, a neural-symbolic model that integrates physics priors with 3D scene understanding for dynamics reasoning. Through extensive experiments, their model significantly outperforms existing approaches, including large multimodal models, demonstrating current limitations in physical reasoning capabilities of video-language models.", "questions": "- It would be great if the author analyze the performance of proposed model on more complex dynamic scenarios such as non-rigid object deformation or fluid dynamics\n- How do the authors plan to address the challenges in real datasets such as motion blur, camera shake, and varying lighting conditions?\n- How do architectural choices  or important hyperparameters such as different CNN backbones or the choice of physics engine parameters impact on the performance of proposed model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's main objective of addressing multimodal 4D dynamics understanding is well-motivated.\n- The authors provide comprehensive evaluation results across three types of reasoning tasks (factual, predictive, and counterfactual), demonstrating the model's capabilities in different scenarios.\n- The proposed physics-aware neural-symbolic architecture presents an innovative approach", "weaknesses": "- The dataset only considers rigid objects with linear velocity and acceleration. Real-world scenarios often involve more complex dynamics like non-rigid deformation, rotation-based motion, and fluid dynamics.\n- The dataset uses synthetic rendering which may not capture real-world challenges like motion blur, camera shake, varying lighting conditions, and partial occlusions.\n- The ablation studies are limited. While the paper shows the importance of physics priors, there could be more detailed analysis of other architectural choices and hyperparameters, like the impact of different CNN backbones or the choice of physics engine parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BVT8HdXdGM", "reviewer": "Reviewer_gPQm", "summary": "The paper addresses the dynamic properties of 3D objects in videos in the task of video question answering. It first proposes a new dataset called DynSuperCLEVR that composes multiple transportation objects into a scene and generates videos of these objects moving. The considered properties are speed, acceleration, and collision. Three types of questions are designed to test VLM's ability to understand the 3D dynamics of objects in these videos. A neural symbolic method, NS-4DPhysics, is proposed to address the importance of explicit 4D representation.", "questions": "- How did you input the video frames to GPT-4o?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Innovative Dataset: DynSuperCLEVR fills a gap in VideoQA with a focus on 4D dynamics, including velocity, acceleration, and collision, enhancing video-based physics reasoning. The scene is programmed in a way that the ground truth information of object speeds and collision events can be documented and transformed into question-answer pairs. \n- Effective Model Design: NS-4DPhysics uses a physics-informed 4D scene representation and neural-symbolic reasoning, excelling in complex VideoQA tasks over baseline models.\n- Comprehensive experiments demonstrate NS-4DPhysics’s superior performance in factual, predictive, and counterfactual reasoning.", "weaknesses": "- The proposed dataset only spans a narrow domain of scenes and objects and may not generalize well to open-domain scenarios. The CLEVR-like setting makes things look nice and clean; for example, the objects have uniform colors, noise-free textures, rigid objects, and much fewer high-frequency details compared to realistic videos. Method comparison on the dataset may not reflect the true ability of the method in real-world videos.\n- The reliance on physics priors might reduce flexibility in scenarios where these priors don’t apply as expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tPirFQKiF2", "reviewer": "Reviewer_kjMd", "summary": "This work focuses on the understanding of the dynamic properties of 3D objects in videos. It uses a simulator to control physical concepts (velocity, acceleration, collisions, etc.)  to generate videos, and uses pre-defined programs to generate annotations. Based on the proposed dataset, this work introduces a physics prior based VQA model. The experiment shows its effectiveness on the proposed dataset.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Overall, the paper is easy-to-follow.\n2. The motivation of the dataset is clear.\n3. Experiment verifies the effectiveness on the proposed dataset.", "weaknesses": "1. The content of the penultimate paragraph is confusing. Firstly, the motivation of the model is unclear. Why an explicit scene representation should be introduced to answer 4D dynamics questions? Secondly, what exactly is the\"explicit scene representation\"? Thirdly, what is the relation between the \"scene parsing module\" and \"3D generative model\", and how do the \"3D generative representation\" and \"symbolic reasoning module\" work together?\n2. The significance of the proposed dataset is doubtful. In the reviewer's opinion, the scenarios in the proposed dataset may be too limited and fall far short of covering all real-world situations.\n3. The effectiveness of the proposed model is also doubtful. The modules are specifically designed based on the actually unknown priors, i.e., the dataset generation and annotation processes. The authors should conduct experiments on spatiotemporal questions of other datasets, including both synthetic and real datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)