Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper compares Learning from Dynamics (LfD) and Learning from Intuition (LfI) in the context of physical reasoning. Through four sets of experiments, it concludes that the state-of-the-art LfI method is at least as good as LfD while being much simpler. It also sheds light on potential improvements in LfD methods.
strengths_and_weaknesses: ### Strength
A thorough systematic comparison between LfI and LfD: It challenges the common assumptions in improving dynamics prediction and designed four experiments subsequently. It provided detailed analysis for each experiment, with insightful observations such as challenges in accurate dynamics prediction.

The code is provided and is also well documented.

### Weakness
Experimental results were not repeated (such as to calculate the mean and variance of the accuracy rate) thus its statistical significance can be challenged.


questions: Does the paper provide a position on the preference between LfI and LfD? The experimental results seem to suggest LfI is simpler and generalizes better. But it's also mentioned in the paper that "the potential improvement from LfD, though challenging, remains lucrative".


limitations: Lack of rich experimental environments in the experiments (upfront). There're existing environments that would provide a more comprehensive study, such as [1-4]. This could also explain why better image classification models used in Sec 5.4 did not yield better AUCCESS.





[1] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Proceedings of the Annual Conference on Robot Learning, pages 1–16, 2017.
[2] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,J. Malik, D. Parikh, and D. Batra. Habitat: A platform for embodied ai research. arXiv preprint arXiv:1904.01201, 2019.
[3] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the International Conference on Intelligent Robots and Systems, 2012.
[4] Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian. Building generalizable agents with a realistic and rich
3D environment. In arXiv:1801.02209, 2018.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tries to understand the utility of learning dynamics in solving physical reasoning tasks. To do this, the authors use the PHYRE benchmark and test a dynamics-based model (trained to predict future observations) and an intuition-based model (trained to predict the answer directly). Through different experiments, the authors find that 1) Intuition-based models outperform dynamics-based models. 2) Dynamics does help when it is accurate 3)  Approximate modelling of dynamics hurts performance 4) Directly predicting the outcomes with different tweaks/heuristics can be very effective.
strengths_and_weaknesses: Strengths

- The paper is clearly written and is easy to understand.
- The motivations, models and questions are well placed in the current literature trying to understand how we could model the physics in the world
- The experiments are well motivated and the successive questions follow well from the previous ones.
- The authors do a good job of testing different models and training regimes throughout the experiments presented in the paper.

Weakness

- The main problem that makes the claims presented in the paper weak is the use of a single benchmark to test hypotheses. This makes it difficult to assess whether the observed effects are an artefact of the benchmark or are more consistent across domains.
- Intuition-based models might be better at the relatively narrow Phyre benchmark (where the goal is to place a ball), and dynamics-based models could be better at answering a broader set of questions. For example, how would the performance of intuition-based models differ when solving tasks that require counterfactual reasoning or hypotheticals? Experiments /discussions on this are missing from the paper.
- Relatedly, dynamics seems to be most useful on out of distribution or systematic generalization. The cross-task split answers this question partially as GD models are able to achieve a cross-task performance similar to within-task. But this split might still not be general enough. For example, one can imagine that a dynamics-based model would find it easier to transfer knowledge when the task is changed slightly, like changing the location of another object in the scene instead of the ball.
- A direct extension of 5.3 is to use Ground truth Dynamics with noise. This would give a more complete answer of how good the dynamics model needs to be to help solve a task.
- Minor: I would have liked to see more of a discussion of relevant cognitive science literature on how humans use simulation of dynamics vs intuition for physical reasoning
questions: - The central question directly seems to stem from a model-free vs model-based approach. How would the authors relate their work to relevant work in this field?
- What is the ideal benchmark that the authors would want to test the relevant questions?
limitations: The authors do a good job with discussing the limitations of the paper. Others are discussed in the weakness section above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper systematically studies the impact of different learning mechanisms on physical reasoning problems through four controlled experiments. The results suggest that (i) reasoning based on a static scene without dynamics predictions can perform as good as that based on an approximate dynamic model subject to errors, (ii) reasoning based on highly accurate dynamics models (Ground-truth
Dynamics) can yield even stronger results.
strengths_and_weaknesses: Strengths
* The paper challenges the common belief on the role of dynamics prediction for physical reasoning through a rich set of empirical experiments. The experiments are very well motivated and designed. Many recent models like ViT, Swin, BEiT, etc. are included in comparison. The results provide strong evidence about the pros and cons of different learning mechanisms on the PHYRE benchmark.
* The paper is very well written. In particular, the detailed discussions on open questions, limitations and future work are inspiring. 

Weaknesses
* The final conclusions from this paper seem highly in line with common expectations: good static models might be better than mediocre dynamic models but worse than accurate dynamic models for physical reasoning. I’m not sure if the paper really brings any new insights other than empirical evidence.
questions: It’s noted in Discussion that `object-centric methods lose essential clues in the scenes especially when it comes to collision and its aftereffects`. Why does a dynamic object-centric model lose essential clues?
limitations: Yes, it’s well discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper examines two types of approaches for physical reasoning, namely learning from intuition (LfI) and learning from dynamics (LfD). LfI directly predicts the outcome without trying to learn or predict the trajectory of the process. Surprisingly enough, the paper shows that existing state-of-the-art LfD methods are only as good as LfI. The reason is that LfD methods are very inaccurate for predicting long-term dynamics. These claims are supported by thorough experiments.
strengths_and_weaknesses: $\textbf{Strengths}$

The paper clearly states the research questions and answers them with thorough experiments. Many of the experiments clearly ablate the components and provide convincing conclusions.

The paper is well written with minimal error.

$\textbf{Weaknesses}$

The baselines from LfD seem to be a bit weak and maybe the authors should perform more experiments on stronger LfD baselines.


questions: I wonder if the LfD baselines considered are a bit weak. For example, [1] provides a strong method in learning the dynamics of physics scenes. Is LfI really as good as LfD?

[1] Janner, Michael, et al. "Reasoning about physical interactions with object-oriented prediction and planning." arXiv preprint arXiv:1812.10972 (2018).

limitations: Limitations discussed thoroughly by the author.
review_title: null

Meta-review: metareview: This paper investigates a simple and important question: does learning to predict physical dynamics help an agent perform better physical reasoning? While most prior work automatically treats this as a given, the paper provides interesting findings that intuition-based learning is better than dynamics-based learning, especially when the dynamics model is approximate. All the reviewers appreciated the clear writing and thorough experiments performed. I believe this will be an insightful and impactful paper.
confidence: Certain
Decision: Accept