Title: Learning to Perform Physics Experiments via Deep Reinforcement Learning
Authors: Misha Denil
Pulkit Agrawal
Tejas D Kulkarni
Tom Erez
Peter Battaglia
Nando de Freitas
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations.  We also compare our learned experimentation policies to randomized baselines and show that the learned policies lead to better predictions.

TL;DR: We train agents to conduct experiments in interactive simulated physical environments.

Review discussion:
I'm a bit confused as to why the agent is not able to obtain a perfect score in the 'which is heavier' experiment, when it is using the z coordinate features of each of the 4 blocks. It seems like the best strategy is simply to move each of the objects once and observe their maximum z-value, and report the smallest displacement as the heaviest block; I would have thought this would be learnable with the A3C architecture (even for low values of Beta). Is there stochasticity in the agent's force or in the z displacement? If not, once this rule is learned the game plays out similarly to a deterministic bandit, which is easily solvable. Is there something I'm missing, or am I underestimating the difficulty of the task?
Thanks!
EDIT:
After re-reading the section, it seems like simply moving a block and then waiting to observe its trajectory is harder than I guessed since there is no 'no-op' action (i.e. the agent must move a block at every time step). Could you provide your justification for requiring the agent to move a block at each time step? To me, having a 'no-op' action seems more natural as young humans (which you mention in order to motivate your paper) will often move things in their environment and observe the effects.
We have trained agents that include no-op actions in the "direct" action space (although they are not discussed in the paper)  We have also trained agents that use a completely different "point and throw" action space where there are (effectively) no-op actions available.  In both cases this does not affect the ability of agents to solve the task.
There are a few confounding factors that I think contribute to the no-op actions not being helpful:
There is always time pressure from the discount factor and waiting for a block to settle after poking it takes several frames.
The fact that small distinctions in mass are relevant is not explicitly made available to the learning algorithm, so it needs to learn to make these distinctions.  This can manifest as a capacity issue (learning to make more fine distinctions requires more representational power) or a learning issue (discovering that fine distinctions are relevant is hard).
It is true that in principle this task is perfectly solvable by measuring small differences in displacement of the blocks after poking them, but in practice these differences can be quite small in episodes where the mass gap is small.
It will be great if the authors can clarify the technical contributions of the paper. Is the main contribution of the paper on applying "LSTMs with convolutional layers using Asynchronous Advantage Actor Critic" for the problem of physical property inference?
Also, having a figure visualizing exact inputs and outputs of LSTMs and how reinforcement signals influence them might help the understanding.
The contributions of this paper are to show
How to construct environments that ask questions about latent physical properties by constructing environments with latent structure
That RL agents can perform well on these tasks without explicit access to this latent structure
How to analyze the behavior of these agents by relating their behavior to behaviors we would expect from solutions that exploit the latent structure
That RL agents exhibit these properties
We can add more pictures of two environments to the paper to visualize inputs, and we can also make some videos available that show trained agents behaving in the environments.
1. [Related work] I would appreciate if you could comment on the following :
1.1 I think that there is a great opportunity to develop the connection of the "cost of information" with ideas from Bounded Rationality and the "expected value of control" (Botvinick).  There is also a great deal of related work on the value of information in economics and OR (Howard, Delage, etc).
1.2 The partition between question/answer and its role in representation learning is an important concept of Sutton's "Experience-oriented" perspective on AI. This view underlies early work on PSRs, TD networks and more recently GVFs.
2. [Bandit formulation] Analyzing your problem under the lens of Bandit theory could be relevant, but I think that the equivalence to a "latent bandit" that you put forward in section 4.1 seems incorrect. The way that the problem is posed from the very beginning is under the RL setting, thus assuming an MDP. The use of deep learning is to make state, and we're therefore not in the bandit setting.
3. [Discount factor] The way that you use the discount factor is really much like a regularizer for the "cost of information". Since the discounted setting in MDPs is equivalent to a finite horizon MDP with random termination, I can see why this can encourage the agent to "answer as quickly as possible".
However, I have the feeling that this termination event should be explicit, and not implicit through the discount factor.
In other words: the agent should have direct control over this termination decision. In your setting, it seems that the cost of information is purely "extrinsic". What do you think is the right balance between intrinsic vs extrinsic ? Is it purely environmental or self-motivated ?
There are many approaches to analyzing how agents do and should make tradeoffs between acting in ignorance and the cost of acquiring more information, as you note.  In this paper the tradeoff that needs to be made is deliberately very simple.  The agent can either pay a fixed cost for gathering more information or can commit to a choice given their current knowledge.
However, much of the power of these approaches comes from giving a framework for balancing between several options with different costs.
For example, when the cost of actions are fixed the "expected value of control" framework reduces to simply linearly shifting the reward by the (fixed) cost of action.
I think a more detailed development of the connection of cost of information to these other frameworks is best left for future work where we require agents to make more sophisticated information tradeoffs, where the tools they offer will have more power.
Thank you for pointing out the connection to Sutton's work. I agree this connection is one we should make explicit in the paper.  In Sutton's language our Section 2 might be better titled "Encouraging Agents to Ask Questions", and what we call the "implicit question posed by the environment" would be better called "the question the environment encourages the agent to ask".
We will add some discussion about how our setup is related to Sutton's framework to the final version of the paper.
The equivalence of which is heavier to a bandit problem is at the "question" level  (in our sense, not in Sutton's sense), rather than the behavioral level.  The underlying problem is a bandit in the sense that the return (mass) of each arm (block) does not change as a function of actions taken by the agent.  The instantiation of this problem in a physical environment (the physical lens) means that the agent needs to deal with the environment as an MDP, but the underlying structure of the problem to be solved is simpler.
The agent controls the length of the episode by choosing when to produce a label.  Taking a labeling action always immediately ends the episode.
If we interpret the discount factor as a probability of termination then it effectively controls the cost of information by forcing the agent to behave as if the episode might terminate prematurely at any step even though this event cannot actually occur.  I agree that the semantics of this seem sort of strange, but stochastic termination is not the only way to interpret the discount factor.  We can also interpret the discount factor deterministically as explicitly de-valuing rewards that take longer to obtain, and the semantics of this interpretation are less strange in our setting.
The cost of information must be extrinsic.  It can be imposed explicitly (as we do) or implicitly (through metabolic costs, time sensitive decisions, etc), but the only way for information to have a cost is for the cost to be imposed through properties of the environment.
This paper presents interesting experimental findings that state-of-the-art deep reinforcement learning methods enable agent learning of latent (physical) properties in its environment. The paper formulates the problem of an agent labeling environmental properties after interacting with the environment based on its actions, and applies the deep reinforcement learning model to evaluate whether such learning is possible. The approach jointly learns the convolutional layers for pixel-based perception and its later layers for learning actions based on reinforcement signals.
We have a mixed opinion about this paper. The paper is written clearly and presents interesting experimental findings. It introduces and formulates a problem potentially important for many robotics applications. Simultaneously, the paper suffers from lacking algorithmic contributions and missing (some of) crucial experiments to confirm its true benefits.
Pros:
+ This paper introduces a new problem of learning latent properties in the agent's environment.
+ The paper presents a framework to appropriately combine existing tools to address the formulated problem.
+ The paper tries reinforcement learning with image inputs and fist-like actuator actions. This will lead to its direct application to robots.
Cons:
- Lacking algorithmic contribution: this paper applies existing tools/methods to solve the problem rather than developing something new or extending them. The approach essentially is training LSTMs with convolutional layers using the previous Asynchronous Advantage Actor Critic.
- In the Towers experiment, the results of probably the most important setting, "Fist Pixels", are missing. This setting receiving pixel inputs and using the Fist actuator in a continuous space is the setting closest to real-world robots, and thus is very important to confirm whether the proposed approach will be directly applicable to real-world robots.
However, Figure 5 is missing the results with this setting. Is there any reason behind this?
- The paper lacks its comparison to any baseline methods. Without explicit baselines, it is difficult to see what the agent is really learning and what aspect of the proposed approach is benefitting the task.
For instance, in the Towers task, how would an agent randomly pushing/hitting the tower (using 'Fist') a number of times and then passively observing its consequence to produce a label perform compared to this approach?
That is, how would an approach with a fixed action policy (but with everything else) perform compared to the full deep reinforcement learning version?
Thank you for your balanced feedback.
In addition to the specific issues addressed below, please also see our general reply which addresses the importance and positioning of this paper in spite of not not offering an algorithmic contribution.
To address your concern over the lack of the Fist Pixels setting for the Towers environment, we have conducted the experiment you requested. We have added the corresponding plots to the paper which show that the agents are also able to learn in this setting.  We have also modified the "Waiting for information" experiment to use the Fist Pixels agent (replacing the Fist Features agent we used for this experiment in the original version of the paper). Thank you for this suggestion to improve the paper.
Your suggestion to compare against baselines that use a fixed (randomized) policy is a good one.  We have added experiments on both environments comparing the performance of our agents using learned vs randomized interaction policies (see the updated paper for full details).  These experiments show that when using the learned interaction policies agents are more accurate and often take less time to produce correct outputs as compared to randomized interactions.
The paper has been appropriately revised to address my concerns. With the new experiments, I believe the paper is in a good shape. Will update the rating accordingly.
This paper purports to investigate the ability of RL agents to perform ‘physics experiments’ in an environment, to infer physical properties about the objects in that environment. The problem is very well motivated;
indeed, inferring the physical properties of objects is a crucial skill for intelligent agents, and there has been relatively little work in this direction, particularly in deep RL. The paper is also well-written.
As there are no architectural or theoretical contributions of the paper (and none are claimed), the main novelty comes in the task application – using a recurrent A3C model for two tasks that simulate an agent interacting with an environment to infer physical properties of objects. More specifically, two tasks are considered – moving blocks to determine their mass, and poking towers such that they fall to determine the number of rigid bodies they are composed of. These of course represent a very limited cross-section of the prerequisite abilities for an agent to understand physics.
This in itself is not a bad thing, but since there is no comparison of different (simpler) RL agents on the tasks, it is difficult to determine if the tasks selected are challenging. As mentioned in the pre-review question, the ‘Which is Heavier’ task seems quite easy due to the actuator set-up, and the fact that the model simply must learn to take the difference between successive block positions (which are directly encoded as features in most experiments).
Thus, it is not particularly surprising that the RL agent can solve the proposed tasks.
The main claim beyond solving two proposed tasks related to physics simulation is that “the agents learn different strategies for these tasks that balance the cost of gathering information against the cost of making mistakes”. The ‘cost of gathering information’ is implemented by multiplying the reward with a value of gamma < 1. This is somewhat interesting behaviour, but is hardly surprising given the problem setup.
One item the authors highlight is that their approach of learning about physical object properties through interaction is different from many previous approaches, which use visual cues.
However, the authors also note that this in itself is not novel, and has been explored in other work (e.g. Agrawal et al. (2016)). I think it’s crucial for the authors to discuss these approaches in more detail (potentially along with removing some other, less relevant information from the related work section), and specifically highlight why the proposed tasks in this paper are interesting compared to, for example, learning to move objects towards certain end positions by poking them.
To discern the level of contribution of the paper, one must ask the following questions:
1)	how much do these two tasks contribute (above previous work) to the goal of having agents learn the properties of objects by interaction; and
2)	how much do the results of the RL agent on these tasks contribute to our understanding of agents that interact with their environment to learn physical properties of objects?
It is difficult to know exactly, but due to the concerns outlined above, I am not convinced that the answers to (1) or (2) are “to a significant extent”.
In particular, for (1), since the proposed agent is able to essentially solve both tasks, it is not clear that the tasks can be used to benchmark more advanced agents (e.g. it can’t be used as a set of bAbI-like tasks).
Another possible concern, as pointed out by Reviewer 3, is that the description of the model is extremely concise. It would be nice to have, for example, a diagram illustrating the inputs and outputs to the model at each time step, to ease replication.
Overall, it is important to make progress towards agents that can learn to discover physical properties of their environment, and the paper contributes in this direction.
However, the technical contributions of this paper are rather limited – thus, it is not clear to what extent the paper pushes forward research in this direction beyond previous work that is mentioned. It would be nice, for example, to have some discussion about the future of agents that learn physics from interaction (speculation on more difficult versions of the tasks in this paper), and how the proposed approach fits into that picture.
EDIT: score updated, see comments below
I should clarify - I would be willing to modify the review if the authors address the following issues:
1) More thorough comparison to the approaches taken in other papers for agents learning physical properties of their environment (e.g. Agrawal et al. (2016) and others). Why is the approach in this paper measuring something different (in an interesting way) than previous approaches? The current related work section has significant breadth, but needs more depth in this area.
2) Similarly, the authors argue that the 'Which is Heavier' task becomes more challenging as the mass gap decreases. If the authors want to argue that this is an interesting axis of difficulty, they should explain why in more detail.
For example, it is unlikely that humans would be able to tell two blocks apart if their mass gap was extremely small -- why is this necessary for artificial agents? Shouldn't we focus on one of the many other areas of physics understanding where humans outperform RL agents? More discussion of the general direction of physics-learning agents (and how this paper fits in) would greatly benefit the paper.
3) Comparison of some simpler RL baselines on the tasks considered.
Thank you for your careful feedback.
I think there is some confusion in comparing this paper to Agrawal et al. (2016), since the approach taken here is very different.  In that paper the model being learned is a model of the environment dynamics from vision.  Their model must learn to map static visual observations to dynamic properties of the objects by relying on correlations between appearance and material properties.  In this work we have carefully constructed our environments to eliminate these correlations between static appearance and physical properties, and because of this the model of Agrawal et al. (2016) would be unable to complete our tasks.
The relationship between appearance and physical properties is quite rich, but relying on vision alone gives an incomplete physical story.
For example, the approach of Agrawal et al. (2016) would have trouble distinguishing between a full thermos and an empty thermos, or between a bag full of rocks and a bag full of tennis balls.
There are two different questions at play here:
1. Given the appearance of an object, how will it move when I poke it?
2. If I want to know how heavy an object is (or if it comes apart, etc), how should I poke it?
Both of these are questions are important. The work of Agrawal et al. (2016) targets the first question, whereas this work is focused on the second.
The purpose of analyzing the which is heavier experiments as a function of the mass gap is to show that the solutions found by the agents behave similarly to what we would expect from solving the underlying structured problem directly.
Indeed it does not seem important to have agents make fine distinctions between extremely similar masses, which is why we did not focus on ensuring that our agents obtain perfect solutions to the more difficult settings for this task.
However, probing this axis of the problem allows us to better understand the solutions we obtain.
In particular, the fact that episode length is correlated with instance level difficulty shows that our agents learn an adaptive strategy for gathering information, that behaves as though the agents are reasoning using an implicitly learned model.  The fact that this effect persists at the instance level suggests the agents have learned more than simply to identify the generating distribution of problems.
We did not compare to other RL baselines because the choice of RL algorithm is not central to the paper. We chose to use A3C because we have an implementation of this algorithm that is robust and scalable, but we have no reason to think that there is anything unique about A3C that would preclude other algorithms from obtaining similar results, and introducing a comparison between different learning algorithms here would only distract from the message of the paper.
Thank you for the detailed response, and for the high-level comment describing the importance of the paper.
Regarding Agarwal et al. (2016): indeed, I assumed the authors were familiar with the differences in these works, considering the fact that Agarwal is the second author on this paper. My opinion was more that, on my first read of the paper, it did not seem that this would be clear to readers with only slight familiarity with previous works. I think it would be worthwhile to mention a key difference (that you ensure the agent can't infer the physics from simply visual observations) earlier in the paper, perhaps in the introduction.
Regarding paper motivation: I quite like the top-level comment that describes the importance of the paper. I completely agree that the usual paradigm (come up with a slightly changed algorithm, show slightly better results) is often not the most productive way forward for science. I think papers in the vein of "laying foundations for exploring an important direction" are less often published, but certainly no less valuable. This paper is along those lines;
however, unlike many other papers that 'lay foundations', it is purely experimental. This is not necessarily a bad thing, but I think the paper has to be presented in the right way in order to have maximum impact.
In its current form, the paper (understandably) tries to walk the line between the usual format of papers in terms of layout, and the fairly unusual content (no algorithmic, model or theoretical contributions). Given that the authors view this content as perhaps 'more conducive to useful science' than traditional papers, I would encourage the authors to also do this for the tone of the paper. I found the top-level comment motivated the paper better than the introduction, so why not add some of these arguments into the main paper?
Meta-comments like "the point is not the algorithm/model we are using" are rarely put into papers, but are often invaluable in terms of the reader's understanding. I think if the authors are more ambitious in this respect (given of course that you don't want to completely re-write the paper), I think the paper could benefit from this (although I understand why this wasn't originally done, since some reviewers may object to it).
Most importantly, one of the claims of the authors in the introduction is that:
"We believe this work establishes an important foundation
for future research in grounded experimentation, intuitive physics and transfer of skills across tasks."
In my opinion, a crucial element of 'foundation' papers is that they provide clear directions for future research (a framework for 'building the house', following the metaphor). I think this should be more prominently featured in the paper. What should other researchers focus on if they are trying to build agents that can understand physics intuitively (building off this work)? The only part of the paper that can be interpreted in this light is the (very thorough) related work section.
But I think it would be invaluable to highlight some of these works in particular as worthy of investigation (in the very near future), perhaps in a 'discussion' section (I realize the paper is long already).
I have updated my score in light of the author's comments, and in anticipation of some of these comments being added to the main paper.
This paper investigates the question of gathering information (answering question)
through direct interaction with the environment. In that sense, it is closely
related to "active learning" in supervised learning, or to the fundamental
problem of exploration-exploitation in RL. The authors consider a specific
instance of this problem in a physics domain and learn
information-seeking policies using recent deep RL methods.
The paper is mostly empirical and explores the effect of changing the
cost of information (via the discount factor) on the structure of the learned
policies. It also shows that general-purpose deep policy gradient methods are
sufficient powerful to learn such tasks. The proposed environment is, to my knowledge,
novel as well the task formulation in section 2. (And it would be very valuable to the
the community if the environment would be open-sourced)
The expression "latent structure/dynamics" is used throughout the text and the connection
with bandits is mentioned in section 4. It therefore seems that authors aspire
for more generality with their approach but the paper doesn't quite fully ground
the proposed approach formally in any existing framework nor does it provide a
new one completely.
For example: how does your approach formalize the concept of "questions" and "answers" ?
What makes a question "difficult" ? How do you quantify "difficulty" ?
How do you define the "cost of information"? What are its units (bits, scalar reward), its semantics ?
Do you you have an MDP or a POMDP ? What kind of MDP do you consider ?
How do you define your discounted MDP ? What is the state and action spaces ?
Some important problem structure under the "interaction/labeling/reward"
paragraph of section 2 would be worth expressing directly in your definition
of the MDP: labeling actions can only occur during the "labeling phase" and that the transition
and reward functions have a specific structure (positive/negative, lead to absorbing state).
The notion of "phase" could perhaps be implemented by considering an augmented state space : $\tilde s = (s, phase)$
Thank you for your suggestions to improve the precision of the paper.
In particular, our intent when describing the the Which is Heavier environment as a "latent bandit" seems to have caused quite a bit of confusion and this is connected to our imprecise formulation of our "question answering" framework which you identified earlier.  We will attempt to make these ideas more precise as we continue to update the paper.
This paper addresses the question of how to utilize physical interactions to answer questions about physical outcomes. This question falls into a popular stream in ML community -- understanding physics. The paper moved a step further and worked on experimental setups where there is no prior about the physical properties/rules and it uses a deep reinforcement learning (DRL) technique to address the problem. My overall opinion about this paper is: an interesting attempt and idea, yet without a clear contribution.
The experimental setups are quite interesting. The goal is to figure out which blocks are heavier or which blocks are glued together -- only by pushing and pulling objects around without any prior. The paper also shows reasonable performances on each task with detailed scenarios.
While these experiments and results are interesting, the contribution is unclear. My main question is: does this result bring us any new insight?
While the scenarios are interesting and focused on physical experiments, this is not any more different (potentially easier) than learning from playing games (e.g. Atari).
In other words, are the tasks really different from other typical popular DRL tasks? To this end, I would have been more excited if authors showed some more new insights or experiments on learned representations and etc. Currently, the paper only discusses the factual outcome.
For example, it describes the experimental setup and how much performances an agent could achieve. The authors could probably dissect the learned representations further, or discuss how the experimental results are linked to the human behavior or physical properties/laws.
I am very in-between for my overall rating. I think the paper could have a deeper analysis. I however recommend the acceptance because of its merit of the idea.
The followings are some detailed questions (not directly impacting my overall rating):
(1) Page 2 "we assume that the agent has no prior knowledge about the physical properties of objects, or the laws of physics, and hence must interact with the objects in order to learn to answer questions about these properties.": why does one "must" interact with objects in order to learn about the properties? Can't we also learn through observation?
(2) Figure 1right is missing a Y-axis label.
(3) Page 3: A relating to bandit is interesting, but the formal approach is all based on DRL.
(4) Page 5 "which makes distinguishing between the two heaviest blocks very difficult": I am a bit confused why having a small mass gap makes the task harder (unless it's really close to 0). Shouldn't a machine be possible to distinguish even a pixel difference of speed? If not, isn't this just because of the network architecture?
(5) Page 5 "Since the agents exhibit similar performance using pixels and features we conduct the remaining experiments in this section using feature observations, since these agents are substantially faster to train.": How about at least showing a correlation of performances at the instance level (rather than average performances)?
Even so, I think this is a bit of big conclusion.
(6) Throughout the papers, I felt that many conclusions (e.g. difficulty and etc) are based on a particularly chosen training distribution.
For example, how does an agent really know when the instance is any more difficult? Doesn't this really depend on the empirically learned distribution of training samples (i.e. P(m_3 | m_1, m_2), where m_i indicates masses of object 1, 2, and 3)?
In other words, does what's hard/easy matter much unless this is more thoroughly tested over various types of distributions?
(7) Any baseline approach?
Thank you for your feedback.
With regards to the desire for more analysis of the learned representations, we have added some experiments comparing the learned policies to a randomized baselines.  These experiments show that when using the learned interaction policies agents are more accurate and often take less time to produce correct outputs as compared to randomized interactions.
We would also draw your attention, particularly for the which is heavier environment, to the experiments where we examine episode length as a function of difficulty.  These are intended to show that the learned representations include information both about the prior distribution of masses (the population level experiment) as well as information the agent's state of knowledge about the mass gap for individual instances.  We assess this by measuring the behavior of the agent, but the behavior is wholly determined by the representations.
> "why does one "must" interact with objects in order to learn about the properties? Can't we also learn through observation?"
This is intended as a descriptive statement about our particular environments, not as a principle of how learning must happen in a general setting.  Our environments are designed so that mere passive observation is not sufficient to perform better than chance, and therefore agents "must" interact with the environment in order to learn.  In real world settings appearance will often provide strong clues as to physical properties, but for this paper we have deliberately removed such clues from our environments.
> I am a bit confused why having a small mass gap makes the task harder (unless it's really close to 0). Shouldn't a machine be possible to distinguish even a pixel difference of speed? If not, isn't this just because of the network architecture?
The mass gap is frequently quite small in the more difficult settings (see Figure 1 (right)), when this is not the case the task is reliably solved perfectly. It is possible a different architecture or more training would allow the agents to achieve perfect performance on the more difficult settings as well;
however, as we argue in our reply to Reviewer 6, achieving perfect performance on this task is not really the point, and the value of doing so is questionable.
> "For example, how does an agent really know when the instance is any more difficult? Doesn't this really depend on the empirically learned distribution of training samples (i.e. P(m_3 | m_1, m_2), where m_i indicates masses of object 1, 2, and 3)?"
The agent is not told explicitly about the difficulty of any instance. It must come to know which instances are difficult through interaction, and the purpose of the experiments on the which is heavier environment are intended to show that the agent does in fact come to know this.  The reason for including both population and instance level experiments is to show that what is learned goes beyond simply the conditional distributions of masses: in the instance level experiments the conditional distributions you mention are fixed and we still see behavior that adapts to the difficulty of the individual instances.
> Any baseline approach?
We have added baseline comparisons for both environments.  Please see the updated version of the paper.
We would like to highlight to all reviewers that the experiments have been updated substantially since the previous version of the paper.
In particular, we now include the missing results for Fist Pixels as requested by Reviewer 3, as well as additional experiments comparing our models to random baseline policies in both environments.
In the process of making these updates we have have made several minor changes to the environments themselves. These changes have made the tower environment harder to solve, but do not alter any conclusions drawn from the experiments.
The machine learning field has profited from hundreds of yearly publications that propose a new algorithm, which outperforms baseline algorithms and has a theoretical proof for some idealized form of the algorithm. This paper, however, is different. It does not propose a new algorithm. It incidentally uses an RL algorithm, but many other RL algorithms could have been used just as well. This paper is not about new models or algorithms.
So what is this paper about? It is an initial research step toward understanding objects and intuitive reasoning in physical worlds. Why is this important? Our best AI agents currently fail on simple control tasks and simple games, such as Montezuma’s Revenge, because when they look at a screen that has a ladder, a key and a skull they don’t immediately know that keys open doors, that skulls are probably hazardous and best avoided, that ladders allow us to defy gravity, etc. The understanding of physics, relations and objects enables children to solve seemingly simple problems that our best existing AI agents do not come close to begin to solve.
Endowing our agents with knowledge of objects would help enormously with planning, reasoning and exploration.
Yet, this is far from trivial.
First, what is an object? It turns out this is not an easy question to answer and has baffled psychologists and philosophers. To simplify matters, in this paper we restrict our attention to rigid bodies. Staring at a thing in one’s hands is not enough to understand what it is. Vision or open loop perception in general is not enough. Trying to pull the thing apart to see whether it is a single entity might help. (Children do indeed love to tear things like paper apart to understand them.) Touching the thing to see what happens also helps --- perhaps it lights up and starts beeping.
Further interaction might reveal that it enables us to remotely talk to someone else, to tweet, etc. Much of the knowledge gained is the result of interaction, that is perception by action.
This paper is also about designing tasks to understand how agents acquire intuitive reasoning and experimentation strategies in physical worlds. An abundant body of evidence in psychology --- see eg the works of Gerd Gigerenzer https://scholar.google.co.uk/citations?user=iw7cepUAAAAJ --- shows that people use intuitive rules to reason about physics, eg a baseball player catching a ball. How do agents acquire such intuition? In this paper, we show for the first time how this is possible with RL and the right type of task.
For example, in the proposed Which is Heavier task we show that our agents learn a valuable intuitive reasoning ability. If the agent probes three blocks blocks and notices that one of them remains much closer to the ground than the other two it doesn’t even bother to try the forth block. It simply points at the block that didn’t lift and chooses to terminate. Through experience, the agent has learned a valuable rule of thumb that enables it to solve the task efficiently, terminating at an optimal stopping time. Our experiments show clearly that our agents have learned to act in this reasonable manner.
This paper shows that model free RL methods can learn how to gather information about physical properties of objects, even when this information is not available to a passive observer, and use this information to make decisions.
Previous approaches to this problem have relied on either explicit knowledge of the underlying structure of the environment (eg hard-wired physical laws) or on exploiting correlations between material appearance and physical properties, see eg the many convnets for intuitive physics research, Galileo, physics 101, learning to poke by poking, etc. (Incidentally, please note that the main author of the work on learning to poke by poking is also author here.)
One of the contributions of this paper is to show that our agents can still learn about properties of objects, even when the connection between material appearance and physical properties is broken. This setting allows us to show that our agents are not merely learning that blocks are heavy; they are learning how to check if blocks are heavy.
None of the previous approaches give a complete account of how agents could come to understand the physical properties of the world around them.  Specifying a model manually is difficult to scale, generalize and to ground in perception.  Making predictions from only visual properties will fail to distinguish between objects that look similar, and certainly it will be insufficient to answer whether a closed thermos is full or empty.  Our approach is a novel contribution to the field of learning intuitive physics that addresses an important gap that had not been filled by previous approaches.

Meta-review:
role: PC
data: comment: The following statement best summarizes the contribution: "This paper shows that model free RL methods can learn how to gather information about physical properties of objects, even when this information is not available to a passive observer, and use this information to make decisions." So this is not a paper about new theory or algorithms, but rather about solving the problem of acquiring knowledge about the physics of the world around us, which is important for many problems and helps explain human performance in many tasks. There are still some concerns about the depth-of-analysis of the paper, but on balance, it is seen as an unconventional but interesting paper. As per AnonReviewer6, the final version could still aim to better address "What should other researchers focus on if they are trying to build agents that can understand physics intuitively (building off this work)?"
 -- Area chair

Decision: Accept (Poster)