Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper discusses the importance of forward prediction in physical reasoning, and particularly in the PHYRE benchmark: a dataset of physical tasks where the agent is asked to place a ball of a chosen radius in a 2d environment. The authors build a classifier that, given an initial state, predicts the probability of success in a given task. `Given the classifier, one can solve PHYRE tasks by sampling actions uniformly at random, and using actions that achieve good scores under the classifier. The classifier consists of an encoder, dynamics model, decoder, and a final classifier. The authors then evaluate two types of dynamics models (interaction nets and transformers) that act on two different types of representations (either images or object states).
The paper concludes that forward prediction is, in fact, beneficial to solving PHYRE tasks, with the best model being a conv-net operating in the pixel space achieving the a SOTA on PHYRE.

The introduced method delivers a new SOTA and some of the reported results are interesting. The evaluation is done on only one dataset, but it is nonetheless quite extensive. I find the paper interesting, but I have mixed feelings about it for I find some of its aspects deeply unsatisfying: 
1. The paper considers only a single fully-observable 2D dataset and many findings cannot be generalized beyond this domain. 
It is therefore unclear if the paper's conclusion would transfer to other domains, partially-observable settings, or even datasets. Specifically, I am concerned about the fact that in this paper's experiments the pixel-based model does better than the object-based one. This is surprising, as authors correctly note, but I am concerned about the justification: "it is easier to determine whether a task is solved in a pixel-based representation than in an object-based one" as stated in the 2nd paragraph of the intro. This would not be the case in any partially-observed environment, or even fully-observed 3D where some parts of the objects are not visible due to self-occlusion.

2.  A large body of related is omitted from the paper and not even mentioned.
When the authors write say "models that operate on object representation" in the abstract, I immediately think about models that infer object representations end-to-end like AIR (Eslami et. al.), MONET (Burgess et. al.), GENESIS (Engelcke et. al) and similar but equipped with transition models (e.g. SQAIR (https://arxiv.org/abs/1806.01794), SILOT (https://arxiv.org/abs/1911.09033) or RNEM (https://arxiv.org/abs/1802.10353)) that seem to fit the paper's setting ideally. However, these models are not considered in this work, and are not even mentioned in the paper. I consider this a severe oversight. This is made even worse by the fact that the best model of the paper is the one trained end-to-end, which is impossible with the considered object-based models, but would be possible with the models I mention above. Please at least discuss these models in the related works and justify why you decided to not use them. Ideally, at least some of these models would be included as baselines.

3.  The paper lacks clarity in places and some of the modelling choices are non-obvious and lack justification.
The best example of the lack of clarity is the following. All models considered in the paper supposedly can be described in terms of encoders, dynamics models, state decoders and task-solution models as shown in Figure 2. The paper, however, fails to explicitly mention what encoders and decoders are used for the considered dynamics models.
As for the modelling choices, a simple example is that of "Objects in PHYRE can have seven different colors; hence, the input of the network
consists of seven channels" in Sec 4.1. Why is the image represented like that? Is that something that comes with the dataset or was it done in a preprocessing step? It is not natural to represent images this way, though I understand why this is the case here.
Another example is that of inputs to dynamics models and the task-solution models: they all seem to have access to the whole history of inputs and/or latent states. If the task-solution model sees all the data, why does it even need a dynamics model? 

4. Solving PHYRE tasks is done by sampling actions uniformly at random and evaluating their success probability under the model, and the authors choose to try K=1000 such actions. While not wrong in principle, it is conceptually unsatisfying that such a high number of random actions is used. Why not train a policy that would learn to sample appropriate actions for a given task?

 As it stands, I think that this paper requires a little bit more work before it can be accepted. I would be happy to change my score if these issues are addressed.

Other remarks:
- in the 2nd sentence of the intro there is an object change from "humans" to "we" that is a bit confusing.
- in the abstract you also say that "[...] these improvements are contingent on the training tasks being similar to the test tasks", which is fair, but is hardly surprising and is an issue with deep learning at large: models generally fail to generalize out of distribution. If so, why report in in the abstract?
- also in the abstract you say that "Surprisingly, we observe that forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance". https://arxiv.org/abs/1802.03006 makes a similar observation in the context of RL, and it might be a good idea to cite it. I think this finding was reported in a number of papers now and is not "surprising" anymore.

review_title: Some findings are interesting but the scope of the paper is very narrow.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This work investigates how a forward prediction model helps the physical reasoning task. The authors propose two variants of forward prediction model, i.e., object-based and pixel-based. The authors also design a classification model, taking predicted results as inputs, to evaluate the efficiency of the prediction model. Task-related metric, i.e., FPA, is proposed to assess the performance. An interesting conclusion, indicated by the authors, is that an accurate predictor does not necessary help the success of physical reasoning.

Pros:

+ Quality: This paper is well written. The authors introduce the motivation behind this work, i.e., to investigate the usefulness of the forward prediction model in the physical reasoning task. The method part is clearly presented with sufficient details, including the model architecture and input/output description. The experiment part is well organized in the format of corresponding concerns related to this work.

+ Clarity: I have carefully checked the manuscript and supplementary material. The whole paper is overall easy to understand. It is enjoyable to read this work. The majority of the details needed to be clearly presented are considered, e.g., the experimental settings, illustration of the newly introduced metric, the description of the figure and table.

+ Significance: The conclusions claimed by the authors are all supported by convincing evidence. The performance boosting on physical reasoning of complex scene proves the usefulness of the proposed method. Meanwhile, authors also point out that the generalization issue, i.e., generalizing to other template, is still challenging. The authors also show that the necessity of developing an accurate prediction model is still open to discussion, which is supported by convincing evidence.

Cons:

- Originality: My major concern lies in the novelty. In my point of view, this work is more like an analysis project. This most significant part is the design of the full pipeline connecting the prediction model and the downstream task, i.e., physical reasoning. The detailed architecture of prediction model and training scheme generally follows the basic configuration of prediction task. However, considering that the main focus of this paper is not pursuing a better performance of prediction model, I think this part is not so important.
review_title: Interesting work with surprising conclusion and supported by convincing experiments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: === Summary

This paper investigates the performance of several state-of-the-art forward-prediction models in the complex physical-reasoning tasks of the PHYRE benchmark. The authors have provided thorough evaluations of the models by ablating on different ways of representing the state (object-based or pixel-based), forms of model class (Interaction network, Transformers, Spatial transformer networks, etc.), and specific evaluations settings (within-template or cross-template), from which they made several interesting observations. For example, forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance. Their best-performing model also sets a new state-of-the-art on the PHYRE benchmark.


=== Strengths

This paper targets a very challenging task, the PHYRE benchmark, where the simulation results can be very sensitive to small changes in the action (or the world), and their best-performing model from this paper outperforms the prior state-of-the-art methods on this benchmark.

This paper provides a thorough evaluation of the forward-prediction models by considering different state representation and model class, resulting in some interesting observations about the relationship between the modeling choices and the physical-reasoning performance.

The videos in the supplemental materials are very illustrative and provide good qualitative comparisons between the methods.


=== Weaknesses

My primary concern of this paper is the novelty is a bit limited. This paper does not propose any new method, but mainly focus on comparing several existing forward-prediction approaches by assessing their ability to perform physical reasoning on the PHYRE benchmark. Although the best model achieves a new state-of-the-art, I would not consider it to be particularly novel.

The proposed method seems to be very specific to the PHYRE benchmark, which, although challenging but only contains open-loop tasks with rigid objects of simple shapes in 2d space. It is hard to know whether this paper's observations are still valid in more diversified and complicated environments. For example, this paper suggests that "pixel-based models are more helpful in physical reasoning" than object-based models, which may not be true if we apply the methods in three-dimensional environments where pixel-based models could suffer from occlusions and a poorer estimation of the 3d location and geometry of an object.

Even if the forward model is reasonably accurate, the control problem can still be very challenging. Given that the results are very sensitive to small variations in the initialization, the task-solution model and the search strategy proposed in this paper may require extensive samples to find a suitable solution to circle around bad local minimums. What is the distribution of the sampling space? How does the performance change with respect to the number of samples? How long does the process take?

When training the forward-prediction model, using a fully-connected graph to model the contacting events between the objects can sometimes lead to unsatisfying results. This is because neural networks are essentially a continuous function, whereas contacts are inherently discontinuous. As shown in the videos accompanying this paper, IN sometimes misses or smoothes the contacts, which is also the reason why some other works choose to add edges between constituent components only when they are close enough [1,2]. The object-based model may have a better performance if the graph is built dynamically.

When constructing the training set, the authors "sample task-action pairs in a balanced way: half of the samples solve the task and the other half do not." How did the authors specify the sampling space? How likely will a random sample solve the task, given the sensitivity of the simulation results to the input variations?

In Figure 5, are there any intuitive explanations of why the red curve first decreases and then increases?


[1] A Compositional Object-Based Approach to Learning Physical Dynamics, ICLR 2017

[2] Learning to simulate complex physics with graph networks, ICML 2020


=== Post rebuttal

Having read the rebuttal and the reviews from other reviewers, my rating remains the same (5: Marginally below acceptance threshold). Many of the reviewers share similar concerns:

(1) The novelty is a bit limited as the paper did not introduce any novel technique approaches. While not every paper needs to propose a new method, a more in-depth analysis of the benchmarked approaches may be needed to provide insights into how existing methods fail and how we can improve them.

(2) The scope of this paper is a bit narrow where the authors only evaluated the methods on PHYRE that is fully-observable and only contains open-loop tasks with rigid objects of simple shapes in 2D space.

(3) The conclusion that the pixel-based model does better than the object-based counterparts may be a bit controversial. The observation is very specific to the methods and environments the authors were using and may not hold when generalizing to more complicated partially-observable or 3D scenarios.
review_title: Thorough evaluation, but the novelty is a bit limited, and the observations may be very specific to the testing environments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Rebuttal Update #####
I thanks the authors for their responses to my questions. They were very helpful, and I think the work, when explored further, would be a great submission to a future conference. However I do share sentiments with other reviewers about following set of issues.

(1) The novelty is a bit limited as the paper did not introduce any novel technique approaches. While not every paper needs to propose a new method, a more in-depth analysis of the benchmarked approaches may be needed to provide insights into how existing methods fail and how we can improve them.

(2) The scope of this paper is a bit narrow where the authors only evaluated the methods on PHYRE that is fully-observable and only contains open-loop tasks with rigid objects of simple shapes in 2D space.


######
Strengths:
The paper is pretty well written.
I enjoyed the analysis in the paper and thought the experiments were relatively thorough.

Weaknesses
The overall approach seems to be rather incremental,  with many past papers on MPC control on some type of learned dynamics model, some with reward functions and others with value functions. For example see [1]. Dynamics learning also uses standard architectural components.

Why is it that better pixel prediction accuracy in object models actually lead to lower AUCCESS? Is it because it is harder to learn the goal model?

Could authors provide more intuition on the templates in which an agent does poorly? Is it similar to the tasks that humans typically perform poorly?

It may also be more interesting to learn more stochastic dynamics model, and average rollouts over a large number of samples from the dynamics model.

The performance gains over a DQN agent appear to be relatively minor

Along this line, it might be more interesting to explore this method in combinations with some type of value function over future states.

I think it would beneficial for the community if source code for the submission was provided, as it still seems there are many free parameters that seem difficult to describe in the paper.

Minor:
[2] might a somewhat related reference that might be good to add.

[1] Chelsea Finn et. al. Deep Visual Foresight for Planning Robot Motion
[2] Yilun Du, Karthik Narasimhan. Task-Agnostic Dynamics Priors for Deep Reinforcement Learning. ICML 2019
might also be worth citing about learning physical-reasoning.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: == Update ==

After reading the rebuttal I have left my score unchanged. I appreciate the clarifications, but am very concerned about the result that the pixel-based models perform worse than the identitiy function in the FPA metric. When a model fails a sanity check like that, I believe the causes and consequences need to be thoroughly investigated. In its current form, the paper does not provide that.

== Original Review ==

The paper evaluates different methods for forward prediction on the PHYRE benchmark for physical
reasoning. Using either ground-truth simulator state or pixel-based observations as input, both
object-based and pixel-based methods are trained to predict future observations. By combining the
physics models with a classifier predicting task success, search agents for the PHYRE benchmarks are
constructed. The results show that the success rate of agents based on the forward prediction models
is slightly higher than that of an agent without forward prediction, but still far lower than that
of one with access to the oracle simulator. Incorporating forward prediction was found to be most
helpful on a subset of tasks with comparatively high complexity and object count, but was of little
help in simpler tasks. Overall, the reported improvement over a previously proposed DQN agent is
narrow, but not significant. The pixelwise accuracy of forward prediction was not found to be
correlated with task performance of the resulting agent.

Strenghts:
 * The paper is one of the first to offer quantitivate results on the PHYRE benchmark, and the first
   (to my knowledge) to do so for forward prediction methods.
 * The selection of methods is sensible and their way they were applied to the benchmark is clearly
   described. 
 * The paper is well written, and largely easy to follow.

Weaknesses:
 * The paper does not introduce novel methods or techniques, but evaluates existing ones.  As such,
   it offers little technical novelty.
 * Qualitativly, based on the visualizations referenced in Appendix A, none of the examined methods
   generate convincing, physically plausible rollouts. All exhibit glitches, such as objects
   overlapping, passing through each other, or changing in size. The joint deconvolutional model in
   particular, which performed best quantitatively, generates severly distorted shapes.
   These shortcomings are not discussed or analyzed in depth, even though they potentially explain
   much of the empirical results.
 * It is not clear to me why joint training is described as a unique capability of the
   deconvolutional model, e.g. by stating that it is its "key advantage". There is no conceptual
   reason why joint training should not be possible for the other models, and Figure 10 in the
   appendix seems to indicate that it is even helpful. Given the spurious rollouts mentioned above,
   it seems plausible that the performance gain of the joint model may have largely come from its
   greater model capacity, which may be utilized to improve the accuracy of the task-solution model
   while bypassing forward prediction. 

While I think that not every paper needs to introduce new techniques, and that papers providing
analysis or negative results can be valuable contributions, in this case, I find it difficult to
derive actionable insights from the presented study. It shows that physics prediction on an
environment like PHYRE is still difficult, but offers few pointers as to what is going wrong or how
it might be improved. As a results, I view it as slightly below the threshold in its current state.

Questions:
 * How does the forward-prediction accuracy (FPA) (Fig. 5) compare to naive baselines, e.g. the identity
   function? Given that not all scenes even exhibit movement, it is hard to judge how strong the
   reported values really are.
 * It is stated that FPA is only computed for the pixels corresponding to dynamic objects. Does this
   refer to dynamic pixels in the ground-truth, the model's prediction, or both?
 * How well does the architecture of the joint model perform when it is purely trained using the
   task-solution loss, without the prediction loss? If it is no better than the no-forward model,
   then that would disprove my concern regarding model capacity mentioned above.

  
review_title: A quantitative evaluation with unclear takeaways

Meta-review: This paper evaluates several methods for physical prediction on the PHYRE benchmark, finding that while object-based methods (e.g. IN, Transformer) perform better in terms of predictive accuracy, pixel-based methods (e.g. STN, Deconv) perform better in terms of downstream task performance. The justification is that it is easier for the agent to evaluate good actions using an image-based representation rather than an object-based representation.

Pros:
- Important attempt to catalogue the current state of the field of physical reasoning
- Improved baselines on PHYRE

Cons:
- As pointed out by R5, there is a failure to evaluate any hybrid pixel-relational methods, such as OP3, R-NEM, C-SWM, etc. Given that the paper's main contribution is its assessment of the current state of the field (in the authors' own words: "providing a realistic picture of the current state of the field"), this seems like a major oversight to me.
- As pointed out by several reviewers, the analysis itself is somewhat limited. I don't see it as a problem that the paper does not propose any new methods, but in that case it needs to present a more thorough picture of why certain methods work better in some cases. For example, I share R1's concern that the Dec model performs worse than the identity function. Can you provide more detailed analysis demonstrating why the latent space is more useful? Can you demonstrate in what cases the object-based classifiers struggle, and why? Incorporating more careful hypotheses and ablations I think would help a lot in turning this into a much stronger paper.

I don't think it's a problem that the paper relies solely on 2D, fully-observed environments (many other papers on physical reasoning do this, so I think it's a reasonable choice), and I don't think it's a problem that the paper does not propose a new method. But I do find myself agreeing with the reviewers that the evaluations done within this context are insufficient. In the rebuttal, the authors emphasize the various conclusions stemming from the results (regarding the effect of model error, the extent of generalization, what "accuracy" means), but these conclusions are not that surprising (model error is a well-known problem in MBRL, deep models are notorious for their failure to achieve strong generalization, and the limitations of pixel accuracy have spawned whole research areas, such as contrastive and adversarial approaches). Again, I don't think the lack of surprising conclusions is itself an issue. But, the fact that the paper does not really make an attempt to explain any nuances or details regarding the conclusions makes it hard to draw a clear contribution from the paper; in that sense, I don't feel the paper really provides "clear guidance" as is argued in the rebuttal.

I do think this paper is very close to being acceptable, and could make a great submission to a future conference if the authors can spend a bit more time on (1) the baselines (i.e., incorporating hybrid models, and ensuring all methods pass basic gut checks) and (2) supporting their conclusions with more detailed analyses.
Decision: Reject