Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper addresses the problem of sample-efficient inference for symbolic physical rules. In the literature, there exists neural-network based models for learning a physical engine which have good predictive accuracy but poor sample efficiency, as well as symbolic models which are highly sensitive to deviations from their fixed physics engine. To be able to overcome issues as such, authors propose a generative model along with a symbolic regression framework, in which forces are produced from a probabilistic context free grammar that is designed to mimic simple Newtonian physics. This particular grammar is parameterized by a few latent variables related to unobserved properties of the physical environment, such as mass and charge. Finally, they develop an Expectation-Maximization algorithm, in order for estimating these latent variables as well as inferring the underlying physical laws of the system.

The methodology described in the paper enables prior physical knowledge to be incorporated into the statistical machine learning models in the form of a probabilistic context free grammar, which in return makes inference via EM applicable. The paper is well written in general, i.e. the main idea of the paper is easy to grasp and all the related technical concepts are explained in a very simple way. Mathematical notation is clear and consistent. Proposed methodology is novel and sound. Claims of the authors are supported by the experimental results on simulated datasets, and there seems to be no fallacy in empirical evaluations. Nevertheless, experimental section can be extended by large-scale applications or real world datasets. In other words, learning the underlying physical rules from a data set which is collected through real-world sensors would provide further evidence to the impact of such a framework. In addition, I think it’s necessary to include more results from symbolic and neural-network based learned models as a reference. Although relation to prior work is clearly addressed in the paper, authors compare their results to only OGN. 

In my opinion, learning symbolic rules for physical reasoning and prediction in a sample-efficient manner will certainly be of interest to the ICLR community. I believe that the proposed methodology is clearly worth exploring, and the presented experimental results are promising. On the other hand, it is no surprise that a carefully constructed grammar for physics is sample-efficient. Authors chose to restrict the grammar with a small set of production rules, in order to narrow down the search space for deriving force terms. However, this can be also seen as a compromise from the generality of the model, since such a predefined grammar might be too restrictive and it might necessitate adding new rules for each new physical phenomena. For instance, in the fields related to optics or thermodynamics, to be able to model phenomena such as refraction-diffraction of light or heat transfer, one might need to modify the grammar manually. 

Another drawback of the methodology could be enforcing position and velocity vectors to reside in the Cartesian coordinate system. Such a choice limits the expressiveness of the model, for instance allowing a general coordinate system could make it easier to model harmonic movement via angular coordinates. In short, my major concern is that enforcing too many constraints inspired by the physics rules we already know introduces a bias, which in return affects the generalization of the model. 

My final remark is about the EM algorithm in the paper. I think that the details about the EM should be written more clearly and thoroughly. I recommend writing the generative model explicitly which could improve the clarity of the paper in general, as well as allowing different inference algorithms to be applicable.
review_title: The proposed grammar is claimed to be generic, however its generality is still limited

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a fully Bayesian approach to symbolic intuitive physics that, by combining symbolic learning of physical force laws and statistical learning of unobserved properties of objects. The paper proposes an EM-based method where in E-step object properties distribution are sampled using the current best estimated force laws and in M-step symbolic regression is used to update those laws.

The paper is clearly-written with clear explanation of the proposed EM-style method. However, one of the main claim that the method "enjoys the sample efficiency of symbolic methods with the ac- curacy and generalization of data-driven learned approaches" is not well-supported in the experiments. While in Sec. 4.1 it shows the proposed method is more data efficient than the neural baseline, it's not clear how the method generalized to complete different scenarios. Ideally the symbolic physical laws are universal thus can be applied to all physical interactions.

Some other questions / comments:
- One of the contribution states "Through empirical evaluations, we demonstrate that the BSP approach reaches human-like sample efficiency, often just requiring 1 to 5 observations to learn the exact force laws – usually more than 10x fewer than that of the closest neural alternatives." This (10x more data-efficient) is hard to tell from figure 5 as it doesn't show when the neural baseline reaches the same performance.
- The error bars in figure 5 is somewhat not very indicative of the stability of the method and some of them for the neural baseline are extremely large. It states that  the values are "out of five experiments with different shuffling of the training set". How many different shuffling is there?
- Sec 3.3.1 "In practice, as the cross-entropy method itself is sensitive to random initializations, in order to ro- bustify the M-step, we repeat it for r runs and pick the best optimizer." Is 'm' supposed to be r in Algorithm 2? 
- For figure 6, if it converges after iteration 3, can we show the expression tree for F3 instead F5? How much do they differ? 
review_title: A computational symbolic framework for intuitive physics

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a fully Bayesian approach to learn an intuitive physics model by combining symbolic regression and statistical learning (MCMC). 

Pros:

+ Personally, I like this paper as it approaches the learning of the intuitive physics model in the right direction, highlighted in figure 1. It is a nice combination of symbolic and learning-based approach.

Cons:

- The related work section is too brief to see the main difference between the present work and prior work. The current related work section only focuses on the machine learning-based intuitive physics model but does not cover symbolic regression in general. Using symbolic regression has a long history, especially in material science, soft robotics, and machine learning in general. Prior work has demonstrated that SR can indeed learn the physical law in a much more complex setting [1-2].

- For researchers who are opposed to the idea of intuitive physics, they would ask where the prior knowledge comes from. Some cognitive research has shown that they are innate for humans and some animals. But this rule cannot be applied for a machine: If we give them all the rules, why not just directly use the physics engine? In the experiments, the results demonstrated here are too simple so that most of the naive physics-based simulator can produce similar results at a probably faster speed. The question is, what the benefit of learning? One may argue that if I know some properties individually, I might be able to transfer this knowledge to unseen scenarios. The authors do demonstrate some capability in 3.3.2 and 4.2, but it seems to come naturally with SR based on the given prior, not something new or surprise. Such capabilities have been demonstrated in [1-2].

- I would like to see if the authors would be able to demonstrate the learned physics can be generalized to scenarios beyond the training datasets. For instance, can the learned model from MAT dataset be transferred to simple scenarios created by bullet-like engine with a similar friction-based interaction, but not identical? The learned physical knowledge should be general enough. Even not as the perfectly correct Newtonian physics, it should be able to generalize to unseen scenarios.

[1] Distilling free-form natural laws from experimental data, Science 2009
[2] AI Feynman: A physics-inspired method for symbolic regression, Science Advance 2020
review_title: A nice Bayesian approach to learn an intuitive physics model, but the relations with prior work and more comprehensive results are missing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an Bayesian-symbolic physics (BSP), an intuitive physics model that jointly infers symbolic force laws and object properties (mass, friction coefficient). The inductive bias is force summation, F=ma, and a grammar of force laws to express object interactions. The inference is done via an EM method that alternates between object property estimation (E-step) and force law induction (M-step), using techniques like symbolic regression and Hamiltonian Monte Carlo (HMC). Some preliminary experiments are shown for the method's effectiveness and data efficiency. 

**Strength**: 
The paper fills an important missing position in the spectrum of intuitive physics models, as Figure 1 argues. The force law grammar, to my knowledge, is something novel in this area, and represents a reasonable inductive bias that balances expressivity and physical plausibility (with two further physical constraints: dimensional analysis and reference invariance). The grammar also helps improve data efficiency. From the inference side, the proposed EM approach is also reasonable. Symbolic regression for force law inference can be interesting for future research.

**Weakness**:
Experiments seem a bit weak for now, and I have some technical concerns about the inference. 

**Questions/suggestions**:
1. Learning with unobserved properties (Figure 6) is the key experiment setting according to the paper's selling point, but is obviously lacking. It'd be great to see more scenarios (than graviton) with more diverse settings (e.g. object mass, initial position), some quantitive numbers, and comparison with some baseline (if possible). 
2. For some qualitative sense, would also be nice to show true trajectories vs. predicted trajectories in the main paper, or if the learned symbolic law force is correct (for Figure 5). Would be interesting to see if any symbolic laws are predicted wrong and how they look like.
3. For object property inference, why use Monte Carlo methods over variational/gradient-based methods? Also, is it possible to use MCMC for formula inference? Some explanations or experiments would solidify design choices for the inference part.
4. Is there an ambiguity when force constants and object properties are jointly inferred? How tricky is designing priors for these? (I see some tricks are used in the paper to handle small constants, for example)
5. The paper currently only compares with one side of the spectrum, i.e. more neural approaches. Comparing BSP with more symbolic approaches like (Smith et al., 2019), the strength is supposed to be the ability to handle non-standard object interactions outside fixed physics engine but within the force grammar. Is it possible to generate some random force laws and show BSP still works while baselines from BOTH sides of the spectrum may fail? It could be the most powerful experiment to support this position in the spectrum.

In general I like the ideas and hope to see more experiments and justifications for design choices.


review_title: interesting position in the spectrum of intuitive physics models

Meta-review: This paper proposes a method for learning physics combining symbolic computation and learning in an interesting way, targeting sample efficiency. At the initial evaluation, it was on the fence but leaning towards acceptance, with 3 slightly positive and one slightly negative review. 

The strengths lie in the combination between symbolic reasoning and statistical ML with a formulation around the classical EM framework. On the other hand, an important issue of the paper is its quite simplistic evaluation on now very easy problems and benchmarks. While benchmarks tend to be simple in the field of learning physics, current work does address more difficult problems than the problems tackled in this paper.

Another issue discussed was the simple trade-off in injecting hand-crafted inductive bias into a system leading to increased sample efficiency, which was perceived as unsurprising by some reviewers. While this is common in ML, and even strongly more so in learning physics from data synthetically generated with known physical laws, it was perceived to be particularly unsurprising in this paper where the benchmarks are indeed very simple and the laws directly encoded.

The AC discussed this paper with the PCs, and it was judged that the weaknesses in evaluation, in particular the simplicity of the tasks, cannot compensate for the interesting hybrid symbolic/ML formulation, and decided to reject the paper.
Decision: Reject