On the Learning Mechanisms in Physical Reasoning
Authors: Shiqian Li
Kewen Wu
Chi Zhang
Yixin Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Intuitive physics
physical reasoning
dynamics prediction

Abstract: Is dynamics prediction indispensable for physical reasoning? If so, what kind of roles do the dynamics prediction modules play during the physical reasoning process? Most studies focus on designing dynamics prediction networks and treating physical reasoning as a downstream task without investigating the questions above, taking for granted that the designed dynamics prediction would undoubtedly help the reasoning process. In this work, we take a closer look at this assumption, exploring this fundamental hypothesis by comparing two learning mechanisms: Learning from Dynamics (LfD) and Learning from Intuition (LfI). In the first experiment, we directly examine and compare these two mechanisms. Results show a surprising finding: Simple LfI is better than or on par with state-of-the-art LfD. This observation leads to the second experiment with Ground-truth Dynamics (GD), the ideal case of LfD wherein dynamics are obtained directly from a simulator. Results show that dynamics, if directly given instead of approximated, would achieve much higher performance than LfI alone on physical reasoning; this essentially serves as the performance upper bound. Yet practically, LfD mechanism can only predict Approximate Dynamics (AD) using dynamics learning modules that mimic the physical laws, making the following downstream physical reasoning modules degenerate into the LfI paradigm; see the third experiment. We note that this issue is hard to mitigate, as dynamics prediction errors inevitably accumulate in the long horizon. Finally, in the fourth experiment, we note that LfI, the extremely simpler strategy when done right, is more effective in learning to solve physical reasoning problems. Taken together, the results on the challenging benchmark of PHYRE show that LfI is, if not better, as good as LfD with bells and whistles for dynamics prediction. However, the potential improvement from LfD, though challenging, remains lucrative.

On the Learning Mechanisms in Physical Reasoning
Shiqian Li Kewen Wu5.3,5 Chi Zhang Yixin Zhu 1,224
School of Intelligence Science and Technology, Peking University
2 Institute for Artificial Intelligence, Peking University
Department Automation, Tsinghua University
Department of Computer Science, University of California, Los Angeles Beijing Institute for General Artificial Intelligence (BIGAI)
Project Website
wtt.:/...tthhhhiooo.......oooo Page
Abstract
Is dynamics prediction indispensable for physical reasoning? If what kind of roles do the dynamics prediction modules play during the physical reasoning process? Most studies focus designing dynamics prediction networks and treat- ing physical reasoning as a downstream task without investigating the questions above, taking for granted that the designed dynamics prediction would undoubtedly help the reasoning process. In this work, we take closer look at this assumption, exploring this fundamental hypothesis by comparing two learning mechanisms: Learning from Dynamics (LfD) and Learning from Intuition (Lff). In the first experiment, we directly examine and compare these two mechanisms. Results show surprising finding: Simple LfI is better than or on par with state-of-the-an LfD. This observation leads to the second experiment with Ground-truth Dynam- ics (GD), the ideal case of LfD wherein dynamics are obtained directly from a simulator. Results show that dynamics, directly given instead of approximated, would achieve much higher performance than Iff alone on physical reasoning: this essentially serves as the performance upper bound. Yet practically, LfD mechanism only predict Approximate Dynamics (AD) using dynamics learning modules that mimic the physical laws, making the following downstream physical reasoning modules degenerate into the LfI paradigm; see the third experiment. We note that
this issue is hard to mitigate, as dynamics prediction errors inevitably accumulate the long horizon. Finally, in the fourth experiment, we note that LfI, the extremely simpler strategy when done right, is more effective in learning to solve physical reasoning problems. Taken together, the results the challenging benchmark of PHYRE [3] show that Lff is, if not better, as good as LfD with bells and whistles for dynamics prediction. However, the potential improvement from LfD, though challenging, remains lucrative.
Introduction
Humans possess distinctive ability of understanding physical concepts and performing complex physical reasoning. The literature on humans learning mechanisms for solving physical reasoning problems can be categorized into schools of thought [43]: (i) physical intuition at a glance without much thinking, such as judging whether stacked block tower will collapse [6], and (ii) more
indicates equal contribution. X indicates corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Solved
LFI
Classifier
Unsolved
Degenerate
Initial scene
Simply inilial scene without any dynamic information prediction
AD
Solved
Predictor
Classifier
Unsolved
Idealize
Generate dynamic sequences from predictor
LFD
GD
Solved
Simulator
Classifier
Unsolved
Oblain dynamic sequences from simulator
Figure I: Comparison of the two learning mechanisms. Learning from Intuition (Lfl) learns from intuition by directly using classifier (first row in yellow). Based on the source of dynamics, we divide Learning from Dynamics (I.fD) into learning with Approximate Dynamics (AD) (second row in green) and Ground-truth Dynamics (GD) (third row blue). Specifically, learning with GD leverages ground-truth dynamics from the simulator, whereas learning with AD predicts how the ohjects' positions and poses unfold via dynamics predictor. In theory, learning with AD should reach the same performance as learning with GD dynamics prediction were perfect: ie.. learning with GD can be regarded as the ideal case of learning with AD. However. practice, learning with AD usually degenerates into LfI due very inaccurate prediction.
extensive unfolding of states under the assumed physical dynamics when facing complex physical tasks [7]. Such a disparity between System- gestem-l-and-3e motivates us to think over the learning mechanisms for physical reasoning in machines:
problem-solving strategies [36]
When performing physical reasoning, is it better for machines to learn from intu- ition by simply analyzing the static physical structure, or to learn from dynamics by predicting future states?
Recent physical reasoning benchmarks for machine learning mostly physics engines focusing evaluating the task-solving abilities of models. For example, PHYRE [3] and Virtual Tools [i] contain physical scenes with long-term dynamics and complex physics interactions. These environments have various tasks with explicit goals, such as making the green object touch the blue object by placing red ball into the initial scene. An artificial agent is tasked predict the final outcome, e.g.. whether the placed red ball will successfully solve the given task.
Existing moobien-solving methods approach such physical reasoning problems by designing various future prediction modules [23, 53). These modules are devised under the assumption that human brains inherently possess a simplified physics engine (called intuitive physics engine) [18], akin to computer game simulator, capable of predicting objects' future states and changes.
Although the intuitive theory claims that humans can predict physical outcomes rapidly, it does not directly guide at the computational level more than the hypothesis that we might have physics- engine-like mechanism in our brain [18, 43]. Critically, although humans can predict dynamics under the intuitive theory, dynamics prediction might not necessary for all types of physical reasoning tasks. This hypothesis is largely left untouched, especially at the computational level. Just noted by Lerer al. [46], directly learning by intuition without dynamics prediction sufficient for various physical reasoning tasks. Of note, this hypothesis does not contradict the intuitive theory but rather provides a new perspective at the computational level.
In this paper, we conduct series of experiments to answer the above questions empirically, To the best of knowledge, ours is the first systematically compare the LiI and LID paradigms. In the first experiment, we verify the simple approach of Lfl by training classifier [15] to predict whether an action would lead success problem-solvee Surprisingly. in the preliminary study, such a model already reaches the state-of-ther (SOTA) performance and even outperforms existing LfD methods unseen scenarios, indicating better generalization. Inspired by this counter-intuitive result, we conduct more experiments In the second experiment, we first set out to investigate whether L.fD could work better than Lff in theory by measuring the performance of video-based classifier [8] using the ground-trum dynamics,
the two learning mechanisms; see Fig. 1 for an illustration.

---

setting an upper bound for the route of dynamics prediction. In the third experiment, we replace the ground-truth truth dynamics with predicted dynamics from an advanced future prediction model [62] to see how LfD performs in practice. The results from these two experiments reveal that precise dynamics significantly improve model performance, but the predicted approximations fail to work as expected: Approximation brings disturbing biases and causes the performance to degenerate into LfI or even worse in certain tasks. In the fourth experiment, through a series of experiments on various LfI models, we conclude that LfI could be simpler and more practical paradigm in physical reasoning. However, making breakthroughs in physical prediction is still promising though challenging. We hope that our discussions shed light on future studies on physical reasoning.
2 Related Work
Intuitive physics and physical reasoning Since Battaglia et al. [6], the computational aspects of
intuitive physics have attracted research attention [43, 75]; intuitive physics and stability has since been further incorporated in complex object [77, 64, 49, 17, 50, 72] and scene [74, 65, 73, 76, 47, 30, 9, 63], and task [26, 27, 32] understanding tasks. The progress enables machines to learn to judge (i) which object is heavier after observing two objects collide [21, 54, 59], (ii) whether stacked block tower will fall [6, 24, 46), (iii) whether water in two different containers will pour at the same angle if titled [41, 57] or liquid in general [5], and (iv) behaviors of dynamics with various materials [42]. However, this line of work primarily focuses on physical tasks without long-term dynamics, either by knowledge- based approaches [6, 54, 70] or learning- based approaches [24, 46].
More complex physical reasoning problems [1, 3, 69], including those involving question answering [10, 11, 14, 29, 71], have also been studied. In partieular, Allen et al. [1] propose use knowledge- based simulation; Xu et al. [68] adopt a Bayesian symbolic method; Battaglia et al. [7], Girdhar et al. [23], and Qi et al. [53] recruit the graph-based interaction network. However, none of these methods fully justify the necessity of dynamics prediction. In this work, we challenge this fundamental assumption and point out simpler and efficacious but overlooked solution.
PHYRE and relevant environments Bakhtin et al. [3] introduce the novel physical reasoning task of PHYRE, wherein an agent is tasked with finding action in an initial scene to reach the goal under physical laws. Current methods for solving PHYRE include reinforcement learning (e.g., DQN [3]), forward prediction neural networks with pixel-based representation [23], and object-based representation [23, 53]. Notably, Girdhar et al. [23] adopt different kinds of forward prediction architectures to perform PHYRE tasks but fail to obtain significant performance improvement, whereas Qi et al. [53] design convolutional interaction network to learn long-term dynamics, achieving SOTA performance by leveraging ground-trutt information about object states in the physical scenes.
In fact, the physical reasoning task of PHYRE can be regarded as an image classification task on judging whether initial action would lead to a successful outcome, or video classification task by considering the dynamics after the initial action is performed. The success of the former sat on deep convolutional neural networks [28, 40] and has now shifted to Transformer- The change from convolutional architectures to attentional models also inspires recent advances in video classification. Models such as TimeSformer [8] and ViVIT [2] in this domain also expand into fields such as action recognition [22] and group activity recognition [20].
models [4, 15, 51].
Unlike PHYRE which directly focuses on physical reasoning in simplified virtual environment, some benchmarks include physical reasoning in their environments more implicitly and take physics as an aid to finish tasks in real life, such as the ones in autonomous driving [16] and embodied AI [39, 67, 19, 55, 48]. Robotic controller based on physics engines [25, 45, 38, 12, 60], navigation tasks on 3D physical scenes [66], and more broadly task and motion planner [61, 37, 33, 35, 34] may
also need physical understanding modules in the system.
Dynamics prediction Predicting dynamics into the future is one of the most extensively studied top- ies in the vision community. One modern approach is to extract image representation and incorporate an RNN predictor [58, 62] or cycle GAN-based approach [44]] However, these approaches cannot extract robust representation from the pixels and incur accumulated errors in long-term prediction. To tackle this problem, Janner et al. [31] and Qi et al. [53] focus on object-centric representation; these are task-speeific solutions with various inductive biases (e.g., spatial information, the number of objects), and the performance drops when dealing with multiple objects with occlusions [52].
3

---

3 The Two Learning Mechanisms
In this section, we define the learning mechanisms for solving physical reasoning problems. Henceforth, we denote all objects' states at time as X, Given an initial background image of a physical setup and random distribution of actions A, the model needs to learn distribution of the final outcome P where Xo {A, J. and y denotes the possible outcome.
Mechanism 1. Learning from Intuition (Lff) In LfI, the outcome is learned directly from the initial images and actions using a task-solution model f:
Poy|Xo) - [(xo:
where denotes the parameters of the task-solution model f(-). call this mechanism Lfl because f(+) can be viewed as intuitive map from the initial conditions to the outcome.
Mechanism Learning from Dynamics (LID) The nature of physies is inherently dynamic. As such, in LfD, y is no longer directly learned from the initial scenes, instead, this approach first learns the underlying dynamics D {Xr]t O, ,...., within time window T using a dynamics prediction module g(-). and then predicts the outcome from the predicted dynamics. Formally, the forward process is described as below:
Xo) = f(D;)), where D (Xo:
where Φ represents the parameters of the dynamics prediction model 9(-). Usually, g(-) is implemented as either an auto-regressive module based on pixel presentation or a graph-based interaction network from object-based representation. In this work, we consider two optimization schedules for g()): parallel optimization during joint learning of f(-) or serial optimization by learning and fixing g(-) first; please refer to Algs. and for more details.
Algorithm 1: Parallel optimization of LfD
Variables:
is the initial background image. A is the action. f(-) and g(.) are the task-solution model and the dynamics prediction model, respectively. and y denote predicted dynamics and outcome, and Dgt Mur the ground truth ones. dynamics loss and cross-enter coos--y loss are denoted La(D, 2at) and Vyt) respectively. and are hyperparameters balance the losses.
1: repeat
2: Predict the dynamics D from I and A using g(-);
3: Predict the outcome from D) using f(-):
4:
Compute the total loss Leorat (D, y, Dat Vee) aLa(D. (D, Dge) + +L,
5:
Optimize [(() and g(-) simultaneously using the gradient of the total loss Litoaa (D, (D., Dyts Myt).
6: until max iteration
Algorithm 2: Serial optimization of LfD
Variables:
I is the initial background image. A is the action. f(-) and g(.) are the task-solution model and the dynamics prediction model, respectively. 1) and denote predicted dynamics and outcome, and Dgt and Mgt the ground-truth ones.
1: repeat
Predict dynamics D from / and using g(-):
Compute the dynamics loss L.(DD Dat):
4: Optimize g(-) using the gradient of dynamics loss LL Dee);
5:
until iteration
6:
Freeze g(-):
7:
repeat
8:
Predict the dynamics D from I and A using the pre-trained 9(:);
9:
Predict the final outcome y from D using f());
10:
Compute the cross-entropy loss L.(y, );
11: Optimize f(.) using the gradient the cross entropy loss Le Doc);
until max iteration
While the dynamics prediction step and the outcome prediction step can be integrated together, it is worth noting that in LfD, additional architecture changes and supervisory signals are necessary to learn the underlying dynamics, without which the paradigm degenerates into LfI. Ideally, a physics engine or simulator plays the role of future prediction. However, inversely learning the physical laws [56, 13] is very demanding due to the intrinsic challenges in long-term prediction.

---

4 Experimental Setup
In this section, we briefly introduce the challenging physical reasoning benchmark of PHYRE-B and the setup of our experiments. Additional training details are in the supplementary material.
Environment PHYRE-B is goal-driven benchmark consisting of 25 different task templates of physical puzzles that can be solved by placing red ball (hence the "B"). Each template has 100 similar tasks, which enables two different evaluation settings. (i) Within-template setting: train on 80% of tasks of each template and test on the rest 20% of each template. (ii) Cross-tempp toosstte setting: train on all tasks of 20 from the 25 templates and test on the rest five previously unseen templates.
The performance is evaluated by AUCCESS [3], weighted sum of the accuracy rate related to the number of attempts. To encourage as few attempts as possible to solve a puzzle, the weights are calculated as: wi = log(k + 1) - log(k), where & E f1,2, 3.0.100} represents the range of attempts. Formally, AUCCESS E. 18 where 83 is the success rate with attempts.
This physical reasoning benchmark is deemed challenging primarily due to the following three reasons: (i) The environment involves variety of objects of different shapes and mass distribution, such as balls, bars, standing sticks, and jars. (ii) The solution set takes up only a small part of the whole action space, making randomly sampled actions hardly ever work out. (iii) Before reaching the goal, a series of complex physical interactions would happen, involving falling, rotation, collision, and friction. To the best of our knowledge, success has been claimed on the environment.
Experiment 1: LfD vs. LfI the first experiment, we compare the SOTA LID model on PHYRE and a transformer-basee classifier for LfI. Specifically, we pick the RPIN model [53] to represent LiD and the ViT model [15] for LfI. The RPIN model leverages a convolutional interaction network based on object-centric representation and predicts the object states (bounding boxes and masks) into the future. When solving a PHYRE task, the model first predicts 10-time steps into the future for an action and then recruits MLP-based task-solution model to predict the outcome. In comparison, ViT is a transformer- based classifier initially designed for image classification. The model patches an image, decomposes it into non-overlapping regions, and performs series of self-attention. The image-level representation is finally extracted by appending a CLS token.
Experiment 2: LID under GD The second experiment serves diagnostic test for the efficacy of dynamics physical reasoning tasks. To this end, we directly extract simulator results and feed the image sequence into video classifier model, setting upper bound for the LfD paradigm by replacing dynamics prediction with the ground truth. In this experiment, we adopt the TimeSformer model [8] due to its superior performance. The model patches the image sequence and replaces traditional convolution architectures with interleaving spatial attention and temporal attention.
Experiment 3: LfD under AD We dive deep into the ineffectiveness of LfD methods like RPIN by fixing the task-solution model with TimeSrormer but replacing the input with trajectories predicted by learned module. In particular, we train a PredRNN [62] to learn the dynamics. PredRNN is an advanced pixel-based video predictor extending the inner-layer transition function of memory states in LSTMs to zigzag memory flow that propagates in both bottom-up and top-down directions. A curriculum set up during learning for better long-term prediction [62].
Experiment 4: More on Lil In the final set of experiments, we verify the performance of different Lil methods. In addition to ViT, we consider BEiT [4] and Swin Transformer [51], which have been proven beneficial in image classification tasks.
Comparison of Learning from Intuition and Learning from Dynamies
In this section, we start our discussion with preliminary experiment and consequently explore whether dynamics contribute to better judgments in physical reasoning tasks both in ideal conditions and in reality by comparing the performance of LfI and LID.
5.1 LID vs. LII: A surprising finding
To tackle the demanding physical puzzles in PHYRE-B, previous studies use pixel-based or object- based future prediction modules to boost mroblem-solvigg performance [23, 53]. The SOTA LID
5

---

method RPIN uses convolutional interaction network to predict long-term dynamics. Inspired by previous studies on intuitive physics, we out to see whether a decision can be made directly by classifier of ViT. Counter-intuitively while the supervision from the dynamics should ideally help a model in LfD look into the future, the performance of the L.f model reaches better than or on par with SOTA as shown in Tab. 1. The AUCCESS of ViT in the cross-template setting even significantly outperforms RPIN's, demonstrating its high generalization ability when solving unseen physical puzzles. Driven by this surprising and intriguing finding, we conduct additional follow-up experiments to determine if the dynamics prediction component helps physical reasoning.
Table 1: The performance of RPIN and VIT in solving PHYRE-B B puzzles. We report the AUCCESS in both
within and cross settings. RPIN uses objects' information as prior knowledge and extra supervisory signals for dynamics prediction. whereas VIT is trained only supervised by the final outcomes with initial scenes as input.
Model
Mechanism
Input
Supervision
Within
Cross
RPIN
LID
Initial scenes, bboxs
Bboxs, masks, outcomes Outcomes
85.49
50.86
VIT
LfI
Initial scenes
84. 84.16=00.30
1..31
5.2 LfD under GD: Is ground-truth dynamics better than intuition?
The surprising discovery motivates why the LfD model has similar or even worse performance than LfI. As first step, question the merit of dynamics: Do dynamics help solve these physical puzzles? To answer this question, we supply the model with the ground-truth
ask
100
96.64
94.18
86.75
73.53
73.97
73.81
AUCCESS
8
60
52.39
46.02
and pro- vide the information to our task-solution model of Tfomerrr,er, due to the video-classificttion like nature. We vary the number of time frames supplied to the model: We consider the input of lengths 1, 2, 4, and 8 extracted directly from the simulator with a time interval of I second. is worth noting that using GD, we assume an ideal dynamics prediction model g() that accurately predicts the future.
20
0
within
cross
Protocal
Figure 2: Performance of TimeSformer PHYRE with different ground-truth input lengths. We com- pare the AUCCESS within-template and cross- template evaluation settings.
The performance of LiD with GD is shown in Fig. 2. While the model underperforms ViT with fewer than 3 frames, which is out of the scope of our work, we see that AUCCESS is significantly boosted with four or more frames. Specifically, with 8 input frames, AUCCESS can be improved by 23.11 in the within setting and 16 the cross setting, reaching about 95% in both settings. This observation answers the above question: Dynamics do help problem-solve in physical reasoning. Such an observation aligns with our intuition that dynamics can tell more about the future states, thus making learning decisions easier for the task-solution model. With sufficient dynamics information, the problem could be "solved" to a large extent.
However, if dynamics do help, why do current LID methods fail to outperform LfI as shown in Sec. 5.1, even considerably lower than the simple baseline generalization? it because of inaccurate dynamics prediction or because of the outdated design in task-solution models? We answer this question in the following set of experiments.
5.3
LID under AD: How do approximate dynamics perform?
solilinm--ooig.....o we explore the reason why AD-th the dynamics prediction in practice--ceee ddoes not take effect as expected. Specifically, we fix the task-solution model π(.) with TimeSformer [8] as in Sec. 5.2 and instantiate the dynamics prediction model 9(-) with an advanced video prediction model PredRNN [62]. We train the LiD pipeline using both of the two optimization schedules mentioned in Sec. 3 to investigate which schedule can better help in learning useful dynamics information.
Given the fact that GD does help

---

The PredRNN first takes one initial frame as the input and outputs three predicted future frames. Next, the TimeSformer takes all the frames as input, including the first initial frame and the three predicted frames, and outputs the final outcome. For parallel optimization, we train an end-to-end framework by integrating the output from PredRNN with the input of TimeSformer. The model parameters é and 0 are updated simultaneously by backpropagating both the dynamics-learning loss and the final cross-entropy loss. For serial optimization, the PredRNN is trained first independently using ground-truth future dynamics as supervision and kept fixed, after which the TimeSformer is optimized using the output from the pre-trained PredRNN, supervised by the ground-truth decisions. For both optimization schedules, the PredRNN only takes raw RGB images as the input without extra information, such as bounding boxes or masks of each object, and directly generates the predicted future frames without further rendering.
We report the AUCCESS of both optimization sched- ules on PHYRE-B in Tab. 2. These results show that independent of the optimization schedule used, AD falls far behind from GD and performs equally even worse than LfI, indicating that approximate dy- namics do little help for the task-solution model in making better judgments. Specifically, by comparing the performance of parallel optimization to TimeS- former with a single frame, we observe a tendency of LiD pipeline's 's performance degeneration to LfI, end- ing up with similar AUCCESS. Moreover, Sec. 5.3 shows that for both within and cross settings, the test dynamics loss in parallel optimization is much higher than the corresponding one in serial optimization, which suggests that the representation learned by LID pipeline in parallel optimization has difficulty in capturing accurate dynamics and is severely impacted by the cross-entropy loss, as is evident in the visualization results in Fig. 3. Although serial optimization gives the dynamics prediction model ample opportunities to learn better pixel-wise approximations of future frames, it still fails to make better decisions than parallel optimization. We hypothesize that in serial optimization, the task-solution model eventually uses the noisy predicted dynamics, which hinders the performance due to inaccuracy; in parallel optimization, the model can avoid this problem by paying less attention to learning future dynamics and focusing more on directly learning the outcome in a manner similar to LfI. Taken together, these experimental results pinpoint the reasons for the inefficacy of current LID methods with AD: (i) It is the poor dynamics learned, rather than the task-solution model, that causes the performance degradation. Inaccuracy dynamics prevent accurate reasoning. (ii) Existing LfD methods with AD degenerate into LfI in the best case.
Table 2: The performance of AD following two optimization schedules in LID. We use the same task-solution model TimeSformer for all the exper- iments. NF denotes the number of input frames used by the task-solution model. We also list the results using GD for comparison.
Prediction
NF
Within
Cross
PredRNN (parallel)
4
75.22
46,42
PredRNN (serial)
4
64.90
44.33
/
1
73.53
46.02
Simulator
4
86.75
73.81
By comparing the models above, we conclude that dynam- ies prediction, in theory, does help solve physical puzzles by providing more temporal information, while struggling to show the full strengths in reality due to the challenges in handling accumulated uncertainty in a long horizon. The results in serial and parallel optimization also reveal the fact that dynamics prediction plays an important role regardless of the task-solution model, but inaccurate dy- namics will eventually harm downstream reasoning; in the best case, LfD degenerates into Lfl.
Opt
Loss
Within
Cross
Parallel
entropy dynamics
0,0638
0,5726
0.0039
0.0049
Serial
entropy dynamics
0.1285
0.6554
0.0003
0.0021
5.4 More on LfI: How does Lfl perform?
Observing the notable success of ViT in the physical reasoning task, we consider testing additional visual classification models to verify the effectiveness of the Lfl paradigm. Among a myriad of models, we pick VIT, Swin Transformer, and BEWT for experiments due their demonstrated superiority in image classification. Of note, TimeSformer with a single input frame can also be considered as a model the Lil paradigm. All of the models take only the first frame as the input. For equal comparison, we train all of the Lff models with an equal number of actions per task as in RPIN. Nevertheless, the Lfl models take no extra prior object information as the input and use only final outcome labels as supervision signals, significantly simplifying the learning process.
7

---

T
T=2
T-
T
T-2
T-3
T
T-2
T=3
Ground Truth
Within
Within Sedial
Within Paralle
T
T=2
T
T=
T-
T
T
T-2
T=3
Closs Ground Truihh
Cross Scatal
Parallll
Cross
Figure 3: Predicted dynamics from PredRNN in LfD's two optimization schedules. In parallel optimization, PredRNN hardly learns any dynamics, whereas the dynamics learned in serial optimization However, serial optimization still fails perform better than parallel optimization shown Tab. indicating that improved but still noisy dynamics don't 't lead better problem-solving. We refer readers supplementary material for more results.
accurate.
The results of these Lfl models are presented in Tab. 3. The performance of Lfl models in the within- template setting is competitive with the SOTA RPIN. addition, all of the Lfl models outperform SOTA in the cross-template setting in AUCCESS; ViT even outperforms by almost 6 points on average. By visualizing the distribution heat maps the possible solution learned by VIT and comparing them with the ground truth in within-template and cross-template settings (see Fig. 4), we observe that the ViT model not only achieves high performance but also accurately recovers the underlying solution distributions. We hypothesize that intuitive models, though trained without dynamics information, can still extract high-level spatial knowledge and physical commonsense that generalizes well in unseen scenarios by looking at the data only; for instance, the most suitable red ball position should be close to moving objects to exert influence. However, we believe that performance of LfI still has significant room for improvement since the inductive biases and structure designs for physical reasoning are not explicitly considered.
Table 3: Performance of different Lff models in solving PHYRE-B problems. For ease of comparison, we also list results from the previous SOTA. We report the AUCCESS in welhin-template and cross-template settings. Input VIT, Swin Transformer, and BEiT only the first frame. For Dec [Joint] and RPIN, directly their reported AUCCESS for comparison.
Model
Mechanism
Object Info
Supervision
Within
Cross
VIT
Lff
False
Outcome
04.16:00.30
56.31
Swin
Lff
False
Outcome
84.71-10.33
54.92-42.30
BETT
Lil
False
Outcome
83.59-10.09
54.07+1.88
Dec [Joint]
LfD under AD LfD under AD
False
Dynamics & Outcome 79.73
52.64
RPIN
True
Dynamics & Outcome
85.49
50.86

---

Within Ground Truth
Within Prediction
Cross Ground Truth
Cross Prediction
Figure 4: The ground-truth Xo) distribution heat maps and the ones predicted by VIT in PHYRE-B, The maps are generated using the first 10,000 actions from the simulation cache offered by PHYRE-B. We only map the 2D positions of placed red balls without showing with a likelihood above 0.8 are used for visualization. A warmer area denotes higher likelihood of success. VIT learns an accurate solution set as the ground truth. We refer the readers to the supplementary material for the maps of Swin and BEIT.
radii. For prediction heat maps, only actions
Besides the promising performance, Lfl models also demonstrate the following advantages:
They are design-efficient without complex hand-crafted tweaks for dynamics prediction modules, which in some cases only introduce harmful distraction noisy representation; see Sec. 5.3.
They only take initial scenes as the input and require no extra task- prior knowledge about the objects as used in object-centric dynamics prediction.
They can easily rre-trained on other computer vision tasks (e.g., image classification on Ima- geNet), incorporating general domain aggnoiic representation to avoid task-specific overfitting.
summary, we view Lff as a simpler and more effective paradigm physical reasoning.
Conclusion and Discussion
We introduce the concepts of two learning mechanisms in physical reasoning, i.e., Learning from Intuition (Lff)) and Learning from Dynamics (Lff)). While it generally helieved that learning the dynamics of physics could help downstream reasoning, a beginning trial of RPIN and ViT on PHYRE- challenges this fundamental assumption: A ViT model effectively learns to perform physical reasoning without any additional supervision from the ground-truth dynamics, object-centric or not. and surprising discovery motivates us to ask whether dynamics play an essential role in physical reasoning. We proceed to answer this question by using Ground-trufe Dynamics (GD) from PHYRE's simulator and feeding the sequence to a TimeSformer model. Experimental results show that accurate dynamics can boost problem-solving performance. Approximate Dynamics (AD) from dynamics predictors perform unfavorably in physical reasoning. We note that the task-solution model not the one to blame. addition, despite increasingly accurate dynamics prediction the years, notice that noisy dynamics prediction still has
This
further explore why

---

a negative impact on the overall performance in reasoning; during parallel optimization, the LfD paradigm collapses into LfI. We hypothesize that in the long run, uncertainty in dynamics prediction unavoidably accumulates, leading to the inferior final performance. Finally, we dig deeper into the Lff paradigm and check the performance of various classification models in PHYRE. The experimental results show that these models achieve much better cross-template generalization while remaining competitive in within-template generalization. It turns out LfI can still perform well even if accurate dynamics are hard to predict, providing route for a simpler, more natural, and less task-specific framework for physical reasoning. However, the LID route, though challenging, remains a lucrative approach to this problem if dynamies could be predicted in significantly more accurate manner.
Why do dynamics- based models struggle make accurate predictions? Analyzing the ex- perimental results, we try to provide preliminary explanations on why current dynamics prediction models struggle. We summarize the following three possible reasons:
1. The dynamics prediction itself is challenging, especially in unseen scenarios. For one thing, prediction into the far future is intrinsically difficult as unforesceable events might steer the dynamics in another direction. For another, the errors will accumulate from earlier frames, leading to exploding uncertainty and noise in the future. Unfortunately, current dynamics prediction methods cannot produce robust model to predict accurate dynamics in physical scenes.
2. Pixel-based dynamic representation has more information than object-based representation, while object-based representation is more concise, Arguably, pixel-based representation potentially incorporates all necessary information, such as shapes of objects, potential points of collision, and angular velocity. However, such a representation extremely noisy, hence extracting useful information embedded is difficult. In comparison, object- representation is by design concise and follows the general principles in the laws of physics. Nevertheless, object-centric methods lose essential clues in the scenes especially when it comes to collision and its aftereffects. The fact that there is not yet a feature representation method that summarizes all necessary information for physical modeling further complicates physical reasoning.
3. Task solutin model design might play role, though not significant. The self-attention mechanism has constantly proven rewarding in a variety of tasks. The strong performance of Lff could also benefit from this architectural design.
Limitations and future work ending journey. However, there are still many aspects left to be studied in the future.
Building intelligence with physical reasoning ability is a never-
In contrast to the topic's significance is the lack of rich environments for experiments. Therefore, we only conduct experiments on PHYRE to support our insights. In the future, we hope to develop a suite of physical reasoning tasks and further investigate the insights presented in this work. In addition, we would also like to incorporate the latest advancements in the suite to see if Lil is a more feasible approach to physical reasoning.
In the experiments, we use original LfI models in our experiments without extra design for physical reasoning tasks. We hypothesize that better perceptual modules and more useful spatial analyzing modules related to physical reasoning could further improve the performance of Lil. In particular, the most challenging physical dynamics in PHYRE involve collision which is a non-smooth and angular movement that does not lie in the traditional linear Euclidean space. We hope to incorporate insights for physical modeling into LfI models for further refinement.
Analyzing the experiments in Sec. 5, an additional question to ask is how accurate the dynamics prediction needs to be for it to be beneficial. We did not figure out measure for this problem in this work. Carefully masking or introducing noise to specific image areas and testing physics predictors at different levels may lead to answer.
While difficult, the dynamics prediction route remains appealing as indicated by its upper bound and the status quo. We are curious how dynamics will benefit other reasoning processes, such as counterfactual and hypothetical reasoning. We argue the dynamics prediction route is still worth additional efforts and look forward to advances made in this field.
Societal impacts
No foreseeable negative societal impacts in this work.
Acknowledgment We would like to thank Miss Chen Zhen at BIGAI for making the nice figures and the anonymous reviewers for their constructive comments.
10

---

References
[1] Allen, K. R., Smith, K. A., and Tenenbaum, J. B. (2020). Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences (PNAS), 117(47):29302-2933 2, 3
[2] Arnab, A., Dehghani, M., Heigold, G., Sun, C.. Lučié, M.. and Schmid, C. (2021). Vivit: A video vision transformer. International Conference on Computer Vision (ICCV). 3
[3] Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L... and Girshick, R. (2019), Phyre: new benchmark for physical reasoning. In Advances in Neural Information Processing Systems (NIPS). 2, 5,5
[4] Bao, H. Dong, L., Piao, S., and Wei, F. (2022). Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR). 3.5
[5] Bates, C., Battaglia, P. W., Yildirim, I., and Tenenhaum, J. B. (2015). Humans predict liquid dynamics using probabilistic simulation. In Annual Meeting of the Cognitive Science Sociery (CogScí). 3
[6] Battaglia, P. W., Hamrick, J. B., and Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences (PNAS), (0045):18327-18333 1,3
[7] Battaglia, W. P., Pascanu, R., Lai, M., Rezende, J. D., and Kavukcuoglu, K. (2016). Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems (NIPS). 2,3
[8] Bertasius, G., Wang. H., and Torresani, L. (2021). Is space-time attention all you need for video understand- ing? In International Conference Machine Learning (ICML). 3, 6
[9] Chen, Y., Huang. S., Yuan, T., Zhu, Y.. Qi, S., and Zhu, S.-C. (2019). Holistic++ seene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In International Conference Computer Vision (ICCV). 3
[10] Chen, Z., Mao, J., Wu, J., Wong, K. K.-Y., Tenenbaum, B. J., and Gan, C. (2021a). Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference on Learning Representations (ICLR). 3
[11] Chen, Z., Yi, K., Li, Y., Ding, M., Torralba, A., Tenenbaum, J. B., and Gan, C. (2021b). Comphy: Compositional physical reasoning of objects and events from vidcos. In International Conference on Learning Representations (ICLR).
[12] Coumans, E. (2015). Bullet physics simulation. In ACM SIGGRAPH 2015 Courses. 3
[13] Dai, B. and Seljak, U. (2021). Learning effective physical laws for generating cosmological hydro- dynamics with lagrangian deep learning. Proceedings of the National Academy of Sciences (PNAS), 118(16)e2020324111
[14] Ding, M., Chen, Z., Du, T., Luo, P., Tenenbaum, J., and Gan, C. (2021). Dynamic visual reasoning by learning differentiable physics models from vidco and language. In Advances in Neural Information Processing Systems (NIPS). 3
[15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold. G.. Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR). 2,3,5
[16] Dosovitskiy, A., Ros, G.. Codevilla, F., Lopez, A., and Koltun, V. (2017). Carla: An open urban driving simulator. In Conference on robot learning.
[17] Edmonds, M., Gao, F,, Liu, H., Xie, X., Qi, S., Rothrock, B., Zhu, Y., Wu, Y. Lu, H., and Zhu, S.-C. (2019). tale of two explanations: Enhancing human trust by explaining robot behavior. Science Robotics, 4(37).
[18] Fischer, J., Mikhael, J. G., Tenenbaum, J. B., and Kanwisher, N. (2016). Functional neuroanatomy of intuitive physical inference. Proceedings of the National Academy of Sciences (PNAS).
[19] Gao, X.. Gong, R., Shu, T.. Xie, X., Wang. S., and Zhu. S.-C. (2019). Vrkitchen: an interactive 3d virtual environment for task-oriented learning, arXiv preprint 1rXiv:1903.05757
[20] Gavrilyuk, K., Sanford, R., Javan, M., and Snoek, C. G. (2020). Actor-transformet for group activity recognition. In Conference Computer Vision and Pattern Recognition (CVPR).
11

---

[21] Gilden, D. L. and Proffitt, D. R. (1994). Heuristic judgment of mass ratio in two-body collisions. Perception & Psychophysics, 56(6):708-720. 3
[22] Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A. (2019). Video action transformer network. In Conference on Computer Vision and Pattern Recognition (CVPR).
[23] Girdhar, R., Gustafson, L.,, Adoock, A., and van der Maaten, (2020). Forward prediction for physical reasoning, arviv preprint 2rv::20006. 10734. 2, 3,5
[24] Groth, O., Fuchs, F., Posner, I., and Vedaldi, A. (2018). Shapestacks: Learning vision-based physical intuition for generalised object stacking. In European Conference on Computer Vision (ECCV). 3
[25] Grzeszczuk, R., Terzopoulos, D., and Hinton, G. (1998). Neuroamimator: Fast neural network emulation and control of physics-based models. ACM Transactions on Graphics (TOG), pages 9-20. 3
[26] Han, M., Zhang, Z., Jiao, Z., Xie. X., Zhu, Y., Zhu, S.-C., and Liu, H. (2021). Reconstructing interactive 3d scene by panoptic mapping and cad model alignments. In International Conference Robotics and Automation (.CRA).
[27] Han, M., Zhang, Z.,, Jiao, Z., Xie, X.. Zhu, Y.. Zhu, S.-.., and Liu, H. (2022). Scene reconstruction with functional objects for robot autonomy. International Journal of Computer Vision (ICCV).
[28] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition (CVPR).
[29] Hong, Y., Yi, L., Tenenbaum, J., Torralba, A., and Gan, C. (2021). Ptr: benchmark for part-based conceptual, relational, and physical reasoning. In Advances in Neural Information Processing Systems (NIPS).
[30] Huang, S., Qi, S., Zhu, Y., Xiao, Y., Xu, Y., and 7hu, S.-C. (2018). Holistic 3d scene parsing and reconstruction from single rgb image. In European Conference on Computer Vision (ECCV).
[31] Janner, M., Levine, S., Freeman, W. T., Tenenbaum, J. B., Finn, C., and Wu, (2019). Reasoning about physical interactions with object-oriented prediction and planning. International Conference on Learning Representations (ICLR). 3
[32] Jia, B., Lei, T., Zhu, S.-C., and Huang. S. (2022). Egotaskqu: Understanding human tasks in egocentric vidcos. In Advances in Neural Information Processing Systems (NIPS). 3
[33] Jiao, Z., Niu, Y., Zhang, Z., Zhu, S.-C., Zhu, Y., and Liu, H. (2022). Planning sequential tasks on contact graph. In International Conference on Intelligent Robots and Systems (IROS). 3
[34] Jiao, Z., Zeyu, Z.. Jiang, X., Han, D.. Zhu, S.-C., Zhu, Y., and Liu, H. (2021a). Consolidating kinematic models to promote coordinated mobile manipulations. In International Conference on Intelligent Robots and Systems (IROS).
[35] Jiao, Z. Zeyu, Z., Wang, W., Han, D., Zhu, S.-.., Zhu, Y., and Liu, (2021b). Efficient task planning for mobile manipulation: a virtual kinematic chain perspective. In International Conference on Intelligent Robots and Systems (IROS). 3
[36] Kahneman, D. (2011). Thinking, fast and slow. Macmillan. 2
[37] Karpas, E. and Magazzeni, D. (2020). Automated planning for robotics. Annual Review of Control, Robotics, and Autonomous Systems, 3:417 439..439.
[38] Koenig, N. and Howard. A. (2004). Design and use paradigms for gazebo, open-source multi-robot simulator. In International Conference on Intelligent Robots and Systems (IROS). 3
[39] Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A.,, Gordon, D., Zhu. Y., Gupta, A.. and Farhadi, A. (2017). Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 171.0:47/2.
[40] Krizhevsky, A., Sutskever, L., and Hinton, E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS). 3
[41] Kubricht, J.. Jiang, C., Zhu, Y.. Zhu, S.-C., Terzopoulos, D., and Lu, H. (2016). Probabilistic simulation predicts human performance on viscous fluid flud--ouuring problem. In Annual Meeting of the Cognitive Science Society (CogSci). 3
12

---

[42] Kubricht, J., Zhu. Y., Jiang, C., Terzopoulos, D., Zhu, S.-C., and Lu,, H. (2017a). Consistent probabilistic simulation underlying human judgment in substance dynamics. In Annual Meeting the Cognitive Science Society (CogSci).
[43] Kubricht, J. R., Holyoak, K. J., and Lu, (2017b). Intuitive physics: Current research and controversies. Trends in Cognitive Sciences, 21(10)749-759. 2,
[44] Kwon, Y.-H. and Park, M.-G. (2019). Predicting future frames using retrospective cycle gan. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
[45] Lee, S.-H. and Terzopoulos, D. (2006). Heads up! biomechanical modeling and neuromuscular control of the neck, ACM Transactions on Graphics (TOG), pages 1188 1198. 3
[46] Lever, A., Gross, S., and Fergus, R. (2016). Learning physical intuition of block towers by example. In International Conference on Machine Learning (ICML). 2,
[47] Li, C., Liang, W., Quigley, C., Zhao, Y., and Yu, L.-F. (2017), Earthquake safety training through virtual drills. IEEE Transactions on Visualization and Computer Graph (TVCG), 23(4):1275-1284, 3
[48] Li, C., Xia, F., Martin, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G.. Jain, T., al. (2021). igibson 2.0: Object-centric simulation for robot learning of everyday household tasks, arXiv preprint arXiv:2108.0322 008.0327..
[49] Liu, H. Zhang, C., Zhu, Y., Jiang, C., and Zhu, S.-C. (2019). Mirroring without overimitation: Learning functionally equivalent manipulation actions. In AAAI Conference on Artificial Intelligence (AAAI). 3
[50] Liu, T., Liu, Z., Jiao, Z., Zhu, Y., and Zhu, S.-C. (2021a). Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robotics and Automation Letters (RA-L), 7(1):400-44
[51] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021b), Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV). 3.5
[52] Oprea, S., Martinez-
P., Garcia-Garcia, A., Castro Castro-Vargas, J. A., Orts-Escolano, S., Garcia- Rodriguez, J., and Argynos, A. (2020). A review deep learning techniques for video prediction. Transactions on Pattern Analysis and Machine Intelligence (TPAMI).
[53] Qi, H., Wang, X., Pathak, D., Ma, Y., and Malik, J. (2021). Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations (ICLR). 2, 3, 5
[54] Sanbom, A. N., Mansinghka, V. K., and Griffiths, T. L. (2013). Reconciling intuitive physics and newtonian mechanies for colliding objects. Psychological Review, 120(2):411. 3
[55] Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J.. Liu. J., Koltun, V., Malik, J., et al. (2019). Habitat: A platform for embodied ai research. In International Conference on Computer Vision (
[56] Schmidt, M. and Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923)81-85.-
[57] Schwartz, D. L.. and Black, T. (1999). Inferences through imagined actions: Knowing by simulated doing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(1);116. 3
[58] Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-. (2015). Convolutional Istm network: A machine learning approach for precipitation nowcasting, In Advances in Neural Information Processing Systems (NIPS). 3
[59] Todd, J. T. and Warren Jr. W. H. (1982). Visual perception of relative mass in dynamic events. Perception. 11(3):22-335..
[60] Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS).
[61] Toussaint, M., Allen, K., Smith, K. A., and Tenenbaum, J. B. (2018). Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems (RSS). 3
13

---

[62] Wang, Y., Wu, H., Zhang, J. Gao, Z., Wang, J., Yu, P., and Long, M. (2022a). Predrnn: A recurrent neural network for spatiotemporal predictive learning. Transactions on Puttern Analysis and Machine Intelligence (TPAMI). 5,
[63] Wang. Z., Chen, Y., Liu, T. Zhu, Y., Liang, W., and Huang. S. (2022b). Humanise: Language--ouddiooeed human motion generation 3d scenes. Advances Neural Information Processing Systems (NIPS).
[64] Wu, J., Lim, J. J., Zhang. H., Tenenbaum, J. B., and Freeman, W. T. (2016). Physics 101: Learning physical object properties from unlabeled videos. In British Machine Vision Conference (BMVC).
[65] Wu, Yildirim, L. Lim, J. J., Freeman, B., and Tenenbaum, (2015). Galileo: Perceiving physical object properties by integrating physies engine with deep learning. Advances in Neural Information Processing Syxtems (NIPS).
[66] Wu, Y. Wu, Y., Gkioxari, G., and Tian. (2018). Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv: 3
[67] Xie, X., Liu, H., Zhang, Z. Qiu, Y., Gao, F., Qi. S., Zhu, Y., and Zhu, S.-C. (2019). Vrgym: A virtual lestbed for physical and interactive ai. In Proceedings of the ACM Turing Celebration Conference-China. pages 1 6. 3
[68] Xu, K., Srivastava, A., Gutfreund. D., Sosa, E, Ullman, T., Tenenbaum, J., and Sutton, C. (2021). bayesian-symboln approach to reasoning and learning in intuitive physics. In Advances in Neural Information Processing Systems (NIPS). 3
[69] Xue, C., Pinto, V., Gamage, C., Nikonova. E.. Zhang, P., and Renz, J. (2021). Phy-q: A benchmark for physical reasoning. arXiv preprint arXiv:2108.13694 13696.
[70] Ye, T., Qi, S., Kubricht, J., Zhu, Y., Lu, H. and Zhu, S.-C. (2017). The martian: Examining human physical judgments across virtual gravity fields. IEEE Transactions Visualization and Computer Graph (TVCG), 23(4):1399 1408.
[71] Yi, K., Gan. C., Li, Y. Kohli, P., Wa, J.. Torralba, A., and Tenenbaum, B. (2020). Clevrer: Collision events for video representation and reasoning. International Conference on Learning Representations (ICLR).
[72] Zhang, Z., Jiao, Z.. Wang, W., Zhu Y., Zhu, S.-C., and Liu, H. (2022). Understanding physical effects for effective tool-use. RA-L, 7(4):9469 9476.
[73] Zheng, B., Zhao, Y., Yu, J., Ikeuchi, K., and Zhu, S.-C. (2015). Scene understanding by reasoning stability and safety. International Journal of Computer Vision (ICCV), 112(2):221 3
[74] Zheng, B., Zhao, Y., Yu, J. C., Ikeuchi K., Zhu, S.-C. (2013). Beyond point clouds: Scene understanding by reasoning geometry and physics. In Conference on Computer Vision and Pattern Recognition (CVPR).
[75] Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, N., et (2020). Dark, beyond deep: A paradigm shift to cognitive with humanlike common sense. Engineering, 6(3):310-33 100:310--45. 3
[76] Zhu, Y., Jiang, C., Zhao, Y.. Terzopoulos, D., and Zhu, S.-C (2016). Inferring forces and learning human utilities from vidcos. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
[77] Zhu, Y.. Zhao, Y., and Zhu, S.-C. (2015). Understanding tools: Trik-oriented object modeling, learning and recognition. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
14


Reviews:
[{"review_id": "nDBAZuPLzH_", "reviewer": "Reviewer_h3js", "summary": "The paper tries to understand the utility of learning dynamics in solving physical reasoning tasks. To do this, the authors use the PHYRE benchmark and test a dynamics-based model (trained to predict future observations) and an intuition-based model (trained to predict the answer directly). Through different experiments, the authors find that 1) Intuition-based models outperform dynamics-based models. 2) Dynamics does help when it is accurate 3)  Approximate modelling of dynamics hurts performance 4) Directly predicting the outcomes with different tweaks/heuristics can be very effective.", "questions": "- The central question directly seems to stem from a model-free vs model-based approach. How would the authors relate their work to relevant work in this field?\n- What is the ideal benchmark that the authors would want to test the relevant questions?", "limitations": "The authors do a good job with discussing the limitations of the paper. Others are discussed in the weakness section above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written and is easy to understand.\n- The motivations, models and questions are well placed in the current literature trying to understand how we could model the physics in the world\n- The experiments are well motivated and the successive questions follow well from the previous ones.\n- The authors do a good job of testing different models and training regimes throughout the experiments presented in the paper.\n\nWeakness\n\n- The main problem that makes the claims presented in the paper weak is the use of a single benchmark to test hypotheses. This makes it difficult to assess whether the observed effects are an artefact of the benchmark or are more consistent across domains.\n- Intuition-based models might be better at the relatively narrow Phyre benchmark (where the goal is to place a ball), and dynamics-based models could be better at answering a broader set of questions. For example, how would the performance of intuition-based models differ when solving tasks that require counterfactual reasoning or hypotheticals? Experiments /discussions on this are missing from the paper.\n- Relatedly, dynamics seems to be most useful on out of distribution or systematic generalization. The cross-task split answers this question partially as GD models are able to achieve a cross-task performance similar to within-task. But this split might still not be general enough. For example, one can imagine that a dynamics-based model would find it easier to transfer knowledge when the task is changed slightly, like changing the location of another object in the scene instead of the ball.\n- A direct extension of 5.3 is to use Ground truth Dynamics with noise. This would give a more complete answer of how good the dynamics model needs to be to help solve a task.\n- Minor: I would have liked to see more of a discussion of relevant cognitive science literature on how humans use simulation of dynamics vs intuition for physical reasoning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gB06ggiYv2U", "reviewer": "Reviewer_8EL5", "summary": "The paper systematically studies the impact of different learning mechanisms on physical reasoning problems through four controlled experiments. The results suggest that (i) reasoning based on a static scene without dynamics predictions can perform as good as that based on an approximate dynamic model subject to errors, (ii) reasoning based on highly accurate dynamics models (Ground-truth\nDynamics) can yield even stronger results.", "questions": "It’s noted in Discussion that `object-centric methods lose essential clues in the scenes especially when it comes to collision and its aftereffects`. Why does a dynamic object-centric model lose essential clues?", "limitations": "Yes, it’s well discussed. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper challenges the common belief on the role of dynamics prediction for physical reasoning through a rich set of empirical experiments. The experiments are very well motivated and designed. Many recent models like ViT, Swin, BEiT, etc. are included in comparison. The results provide strong evidence about the pros and cons of different learning mechanisms on the PHYRE benchmark.\n* The paper is very well written. In particular, the detailed discussions on open questions, limitations and future work are inspiring. \n\nWeaknesses\n* The final conclusions from this paper seem highly in line with common expectations: good static models might be better than mediocre dynamic models but worse than accurate dynamic models for physical reasoning. I’m not sure if the paper really brings any new insights other than empirical evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JI4IcBxLhK", "reviewer": "Reviewer_TKqA", "summary": "The paper compares Learning from Dynamics (LfD) and Learning from Intuition (LfI) in the context of physical reasoning. Through four sets of experiments, it concludes that the state-of-the-art LfI method is at least as good as LfD while being much simpler. It also sheds light on potential improvements in LfD methods.", "questions": "Does the paper provide a position on the preference between LfI and LfD? The experimental results seem to suggest LfI is simpler and generalizes better. But it's also mentioned in the paper that \"the potential improvement from LfD, though challenging, remains lucrative\".\n\n", "limitations": "Lack of rich experimental environments in the experiments (upfront). There're existing environments that would provide a more comprehensive study, such as [1-4]. This could also explain why better image classification models used in Sec 5.4 did not yield better AUCCESS.\n\n\n\n\n\n[1] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Proceedings of the Annual Conference on Robot Learning, pages 1–16, 2017.\n[2] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,J. Malik, D. Parikh, and D. Batra. Habitat: A platform for embodied ai research. arXiv preprint arXiv:1904.01201, 2019.\n[3] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the International Conference on Intelligent Robots and Systems, 2012.\n[4] Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian. Building generalizable agents with a realistic and rich\n3D environment. In arXiv:1801.02209, 2018.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\nA thorough systematic comparison between LfI and LfD: It challenges the common assumptions in improving dynamics prediction and designed four experiments subsequently. It provided detailed analysis for each experiment, with insightful observations such as challenges in accurate dynamics prediction.\n\nThe code is provided and is also well documented.\n\n### Weakness\nExperimental results were not repeated (such as to calculate the mean and variance of the accuracy rate) thus its statistical significance can be challenged.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3e0gYVLzKkh", "reviewer": "Reviewer_SWzY", "summary": "The paper examines two types of approaches for physical reasoning, namely learning from intuition (LfI) and learning from dynamics (LfD). LfI directly predicts the outcome without trying to learn or predict the trajectory of the process. Surprisingly enough, the paper shows that existing state-of-the-art LfD methods are only as good as LfI. The reason is that LfD methods are very inaccurate for predicting long-term dynamics. These claims are supported by thorough experiments.", "questions": "I wonder if the LfD baselines considered are a bit weak. For example, [1] provides a strong method in learning the dynamics of physics scenes. Is LfI really as good as LfD?\n\n[1] Janner, Michael, et al. \"Reasoning about physical interactions with object-oriented prediction and planning.\" arXiv preprint arXiv:1812.10972 (2018).\n", "limitations": "Limitations discussed thoroughly by the author.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\nThe paper clearly states the research questions and answers them with thorough experiments. Many of the experiments clearly ablate the components and provide convincing conclusions.\n\nThe paper is well written with minimal error.\n\n$\\textbf{Weaknesses}$\n\nThe baselines from LfD seem to be a bit weak and maybe the authors should perform more experiments on stronger LfD baselines.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept