Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: A neural-symbolic Bayesian framework for learning intuitive physical models, by making abduction from a few observations provided by the environment and domain knowledge given by grammar.
main_review: This paper proposes a neural-symbolic Bayesian framework for learning intuitive physical models, by making abduction from a few observations provided by the environment and domain knowledge given by grammar. The framework is consistent with the general methodology, i.e., *sample(hypothesis)-search-observation loop* of learning-by-explanation paradigm. The solutions, E-M algorithm, and S-R are also canonical methods for combining symbolic knowledge with statistical observation, where the former serves as the hypothesis-generator operation and the latter selects a most plausible hypothesis as the interpretation of the observation. In general, the authors formulate a significant problem and provided the community with a human-like computational approach. 
limitations_and_societal_impact: * **Novelty**: Although exploiting the E-M algorithm and S-R method for solving reasoning-by-observation problems like the ones presented in this work is valid, this method has already been an established and typical framework for explanatory learning that tries to learn a (logical and statistical) model from raw observation and ungrounded primitives. Hence, I think the proposed method is not a principle contribution for novelty. A domain-specific PCFG is a canonical workstream in the neural-symbolic community, either. The novelty of this work comes in applying the learning-by-explanation framework to induce physical rules with data efficiency. This is a principle contributor to the community, yet it goes not so far as the authors claimed;

* **Scalability**: The authors exploit heuristics like checking the consistency of measurement for pruning the hypothesis space, which serves as metaknowledge. This is a good trial but make the method much too ad-hoc. It may be better if formulated in a formal logic way, making the symbols generalizable. Either, the given grammar is way overstrong, making both the framework hard to extrapolate and the task trivial. It would be better if the framework makes abstraction over a small set of primitives;

* **Validity**: Since physical reasoning is not entirely a novel task for the community, it is necessary to carefully benchmark methods for the task. However, the authors only compared with four end-to-end learning methods, without consideration of neural-symbolic counterparts like DeepProblog and Abductive Learning. Since neural methods can hardly encode domain knowledge, the comparison is naturally unfair. It may make the results more solid if the authors compare their methods with neural-symbolic counterparts using identical domain knowledge.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents the Bayesian-symbolic framework (BSP), a novel framework that combines symbolic reasoning and learning to discover intuitive physics. BSP uses learning to infer object properties, and uses symbolic expression to recover physical laws. The models are learned in an EM manner, where BSP alternates between inferring the object properties with the current force, and inferring the forces given the object property distributions. The dynamics are assumed to be Newtonian. The experiments show that BSP outperforms selected baselines, and achieve relatively good results compared to humans on the SYNTH benchmark and PHYS101 dataset.

main_review: === Strengths ===

+ The paper is well written and the presentation is strong.

+ The evaluations on physical forces in section 5.1 are comprehensive, as it covers both gravity, collision, and friction.

+ The general idea of combining symbolic reasoning and statistical inference is nice. It also bypasses the non-differentiability issue in collisions, often encountered by pure neural network methods.

+ The use of prior knowledge such as dimensional analysis to prune the symbolic search is neat, as these are also tricks that people commonly use in physics.

=== Weaknesses ===

- This is my opinion only, but the EM optimization method seems slightly naive here. How often would such a search get stuck in the local optimum or saddle points, for example, due to an wrongly searched expression?

- I am really curious to know whether BSP can discover certain physical constants. For example, the Kepler constant K of Earth, or the gravitational constant G? This is my opinion only, but these constants are more exciting compared to the properties such as masses or the spring constants analyzed in section 5.

- I am slightly concerned about the performance of BSP in table 3, as it does not outperform humans. I would imagine humans are not good at such tasks, and would expect better results from BSP, especially considering the task is more of a classification problem rather than inferring the exact physical quantity. Could the authors explain this more in detail?

limitations_and_societal_impact: Yes, limitations and impact are discussed in section 6.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is proposing a new Bayesian symbolic framework that can assist in being able to perform types of physical reasoning over items and environments. The motivation in the paper is that humans are able to understand the physical properties of a system given very few data samples therefore we should be able to develop machine learning systems that can determine the underlying physical properties of the environment with similar data efficiency. The idea in the paper is to use a Bayesian inference method in order to be able to determine a list of parameters and how to combined different functions to describe the physics of a particular environment.

Pros
- The method does propose an interesting grammar that can be used to more efficiently explore the search space of possible physical functions to describe an environment
- The method also shows some promising performance across a number of different environments for understanding the underlying physical properties and only a few data samples.
- Learning the physical functions as a combination of terms is also a promising direction for this research. It could be interesting to explore this further and apply the method to systems where the physics may not be well known to discover the physical properties of new physical systems. 

Cons
- The assumptions for the model are still not perfectly clear and well contextualized with respect to prior work. This is referring to where and how the symbolic representation for the items in the environment comes from. It's not clear if prior methods also have access to this or similar information.
- It would also be good to better outline the limitation on the types of physical systems this model can learn to predict over. The equations that are supported seem to indicate the most Newtonian/classical physical systems should be possible but it is not clear what are the limitations of the grammar, along with the types of functions that can be represented.
- Well it is mentioned in section 6, it's not clear how the perceptual video data used in the real experiments or The ULLMAN experiment is processed into symbolic values that can be used by BSP. Is this mechanism easy to automate such that it's not a significant limitation of applying the method in the paper?


main_review: Additional comments on the work:
- It says the environment is model is a top-down generative model in which things are assumed to interact with each other, does this mean that the method requires the top-down model beforehand?
- Figure one in the paper is challenging to understand. Is each column designed to represent the different papers that are described in the caption? If this is true then this figure is displaying a lot of information but it is difficult to understand the connections between these different papers and what is the significance of the transitions between them. It is important to discuss this well in the related work section to help contextualize how this work fits into prior methods and how it is different.
- In the introduction, it's outlined that the paper introduces a grammar for Newtonian physics that also greatly helps the overall learning problem. It would be helpful to be able to describe this grammar more in the introduction so the reader can already get an intuition for why this grammar is helpful.
- From the description given in section 3.1 the model is given a vector of intrinsic physical properties for each of the items in the environment. This particular information can greatly assist in being able to learn the overall problem and being able to estimate interacting bodies between them. It would be good to know if previous methods use a similar structure in this paper or if the primary contribution is being able to take advantage of this type of structure where other methods have not had access to this graph of intrinsic properties.
- While the first feature of the grammar described in the paper makes sense such that units are preserved between operations. The second point of making sure to use absolute positions feels like something that is well known in the machine learning community as it reduces the complexity of the problem in order to learn behaviors that are more egocentric and less absolute.
- In section 4.1 the paper reminds the reader that the average person can perform physical learning tasks similar to the ones we use with fewer than five scenes. Yet it seems that the environments described in the paper are new and have not been evaluated on people to understand how many scenes are needed for a person to be able to accurately understand and perform predictions in those physical simulations. It is important to better validate these environments to make sure that humans can easily understand the physics in them given just a few scenes. This will also provide information with respect to the motivation that people can understand these systems very quickly. On the other hand there may be environments that already exist in the community that can be used to perform these types of experiments. For example, the environments used in the Battaglia at all, 2016 paper.
- A number of the referenced papers are citing ArXiv versions of the papers when the papers are already published. The proper published papers should be cited in the paper not the ArXiv versions. For example, Interaction networks for learning about objects, relations and physics.
- The ablations in section 4.2 are interesting and also provide good information as to the helpfulness of the grammar that's designed in the paper. However part of the motivation for creating the grammar was to make the learning problem more tractable. Therefore it would also be helpful to include an additional comparison over the same ablations but when more time is given to the training algorithm in order to learn the problem. In this case, the method proposed in the paper should be able to learn quickly with fewer iterations of learning in order to be able to discover the proper physical settings while not using the full grammar may be able to learn with the same performance but will require a significant amount of time that is undesirable.
- For the analysis and section 5.2 how does bsp compare to the prior neural-based methods for being able to do predictions in those real environments? It's not mentioned whether or not those real environments are impossible to solve using the neural-based methods. Some additional context here in comparison to prior work would be extremely helpful to the reader and help the reader understand how the method performs outside of the SYNTH environments that were created for this paper.
- Well it is mentioned in section 6, it's not clear how the perceptual video data used in the real experiments or The ULLMAN experiment is processed into symbolic values that can be used by BSP. Is this mechanism easy to automate such that it's not a significant limitation of applying the method in the paper?

----- Post Author Response ----
I have updated my score thanks to the authors' responses and the supplementary material.
limitations_and_societal_impact: Well it is mentioned in section 6, it's not clear how the perceptual video data used in the real experiments or The ULLMAN experiment is processed into symbolic values that can be used by BSP. Is this mechanism easy to automate such that it's not a significant limitation of applying the method in the paper?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose the Bayesian symbolic (BSP) framework for inferring both the symbolic force laws and the values of latent properties of entities.
They contribute a grammar of Newtonian physics over which they perform inference with the BSP framework.
Empirical results show higher sample efficiency than neural approaches, and performance that is between humans and chance on the ULLMAN dataset.
main_review: Originality: on the one hand, the method can be seem as novel compared to the approaches in Figure 1. On the other hand, the methodology seems to be straightforward variant of [0] to intuitive physics domains.

Clarity: the writing and presentation is clear.

Quality: 

While it is true that neural approaches are more sample inefficient, Figure 4 is also an unfair comparison. As the authors acknowledge in section 6, they assume that BSP has access to a grammar of Newtonian physics that neural approaches do not have. Indeed, neural approaches must both learn the grammar and how the grammar applies to a particular dataset, while BSP only needs to learn about the latter. Therefore, the results in Figure 4 do not seem to be very informative of possible advantages BSP has over neural baselines. 

What might make for a better empirical case for BSP is if the authors could dig deeper into Figure 1 and identify explicit axes along which to measure the advantages and disadvantages of BSP vs other baselines in transfer to a new domain *after* the model has been trained. This could help address the problem with the unequal starting point between BSP and neural approaches. For example, 
- suppose we first learn in a multi-task fashion on BOUNCE and MAT, and then transfer to a domain with balls bouncing around over a mat? How would this compare with IN [1] or NPE [2]?
- the authors state that [tracking algorithm + BSP] learns Hooke's law, after having trained on this environment, what would be the sample efficiency of [tracking algorithm + BSP] be on a spring environment with a different spring constant? And how would this compare with a neural approach that has object perception built in, such as [3]?

Can BSP generalize to completely different kinds of domains from that which it has been trained, such as generalizing to collisions of cars in a simulated freeway from training on the BOUNCING environment? Presumably that would be what I'd expect the strength of symbolic regression to be, it would strengthen the paper if this hypothesis were true, or alternatively if the authors could provide an explanation of why this hypothesis is false.

Significance: 

I am assuming that the authors propose BSP and the Newtonian grammar as a hypothesis for a computational model of intuitive physics in humans.
In this case it would have been enlightening to see how BSP compares to the computational model in Ullman et al. (2018) -- would the authors be able to provide this comparison, and comment on the tradeoffs between the two approaches.
Without a proper comparison with an alternative hypothesis (e.g. Ullman et al. 2018) as a baseline, rather than simply the random chance baseline, it is not clear how to judge the merits of BSP in modeling humans.

Questions:

How does the wall clock time of inference compare between humans, BSP, and a neural method like [1] or [2]?

I was not able to find the appendix, but I think it would be very valuable to see to understand the proposed method.

Summary:
Overall, while the proposed idea is intriguing, the empirical results seem to fall slightly short of clearly elucidating the merits of the method. The comparison to neural baselines would likely need to be in a transfer setting in order to be a fair comparison, and the human experiments require a discussion with an additional baseline (e.g. Ullman et al. 2018) on the axes along which one or the other method is a better explanation for human intuitive physics induction.

[0] Ellis, Kevin, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. "Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning." arXiv preprint arXiv:2006.08381 (2020).
[1] Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D., & Kavukcuoglu, K. (2016). Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222.
[2] Chang, M. B., Ullman, T., Torralba, A., & Tenenbaum, J. B. (2016). A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341.
[3] Veerapaneni, R., Co-Reyes, J. D., Chang, M., Janner, M., Finn, C., Wu, J., ... & Levine, S. (2020, May). Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning (pp. 1439-1456). PMLR.


limitations_and_societal_impact: Yes, they were addressed.
review_title: null

Meta-review: The paper presents a neural-symbolic Bayesian framework to learning physical models in a similar way as human, by making abduction from a few observations provided by the environment and domain knowledge given by grammar. It aims to address an important problem. The proposed solution is reasonable with promising results.

A few concerns have been raised: (1) evaluation: stronger baselines should be included and some results are not as good as human; (2) how the grammar was learned - ad hoc versus principled ways; (3) how general the models can be - only for some specific physical systems versus general systems. The authors were able to address these concerns to a certain extent. After discussion, it was agreed that while the paper still has some limitations, the work is interesting enough to be presented to the NeurIPS community. 
Decision: Accept (Poster)