Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to leverage an ensemble of GNN-based forward-models to estimate an agent's uncertainty with interacting with some part of an object-centric state-space. This uncertainty is used as an intrinsic reward signal during task-agnostic environment exploration with a model (dubbed "Free Play"). Given a fully observable MDP, the authors show that this kind of "Free Play" enables qualitatively object-centric exploration along with the ability to perform zero-shot generalization to reward functions defined over object attributes.
strengths_and_weaknesses: Strengths:
* Very clear method, results, and introduction
* High potential for impact in psychology and robotics
* While the results are somewhat expected, no one has combined disagreement with GNN based methods and the results are convincing


Weaknesses:
* The method has some hand-crafting in parsing out the agent from the objects, though this seems like a fine first-step.

questions: Experiments:
- Is is possible to run MLP + iCEM but instead of concatenating the object-representations, you simply sum them, or do something that exploits structure but is not a GNN? This would help isolate the benefits of using a GNN.

Questions:
- During evaluation, you give the task reward function to a planner in order to select the appropriate action. Your model however, only learns to predict the next state, given an action. It doesn't learn to predict the environment reward---in fact, the agent never sees environment reward. How does the planner know how to leverage the task reward to construct a plan towards that reward? Does the planner have knowledge about all ground-truth states, along with the reward R(s,a) for all ground-truth states?
- It seems that an MLP leads to the same relative amount of exploration with different objects as the GNN but takes more time to do so? Why do you think it leads to similar results? 
- Figure 3: It seems structured world models lead to object interactions *more quickly*? With infinite time, how do you think of GNN & MLP would have the same asymptotic behavior?
- What are the numbers in Table 3? Success rate?

Baselines:
- Plan2Explore seems like a good baseline since it seems similar in principal but doesn't offer combinatorial optimization. Why is this not a baseline? Could you expand on the differences to this baseline?


limitations: Limitations are adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper an agent that learns a world model through exploratory play, which it then deploys to solve downstream tasks via a planner. The world model is an ensemble of GNNs, where nodes represent objects, which introduces an object-centric inductive bias. Effective playful exploration is achieved by using disagreement among the ensemble as a proxy for uncertainty, which allows the agent to select actions whose outcomes it cannot yet reliably predict. The authors show that this method works better than baselines, in the sense that it quickly starts engaging in “interesting” interactions (with objects). They do this in two simulated environments with dynamics - a planar world with a variety objects that the agent can push around, and a robot-arm environment where the agent can stack, flip, and throw objects. The authors show that the resulting agent achieves zero-shot generalisation on downstream tasks, to numbers of objects unseen at training time.
strengths_and_weaknesses: The paper makes progress on an important problem in the RL field, namely learning with intrinsic motivation. As the authors argue, a learned world model that is reward-neutral is a key ingredient for better generalisation and transfer learning. The benefits of learning an object-centric representation in this context are well supported by the authors’ work. The experimental methodology seems sound, and the results are persuasive. The paper is well written and clear and, as far as I can determine, technically sound. Overall I think this is a very good paper that is well above the acceptance threshold for NeurIPS.

I am not sufficiently familiar with the literature on intrinsic motivation to assess the related work section, or to judge the originality of the work with great confidence (hence my “low confidence” rating). However, the following paper got my attention a few years back, which seems very related, and isn’t cited:

Nick Haber, Damian Mrowca, Li Fei-Fei, Daniel L. K. Yamins. Emergence of Structured Behaviors from Curiosity-Based Intrinsic Motivation. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018. (https://arxiv.org/abs/1802.07461)

I notice that a subset of the authors of that paper have a more recent ICML paper that also looks relevant, and isn't in the bibliography: https://arxiv.org/abs/2007.07853


questions: See below
limitations: I believe the ensemble disagreement method that the authors use is susceptible to the white noise problem, wherein the agent is motivated to fixate on dynamics that are random and therefore cannot be learned. Perhaps the authors could comment on this.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a curiosity-driven method for exploration via structured world models. It uses GNN ensemble disagreement for computing intrinsic motivation signals in RL, which achieves sample-efficient and interaction-rich exploration in multi-object manipulation environments. Further, the trained world models combined with planning-based methods can achieve zero-shot generalization to many object manipulation tasks. Experiments shows the data-efficiency, the zero-shot generalization ability and manipulation accuracy of the proposed method.


strengths_and_weaknesses: Strengths:
1.	This paper achieves quite a high sample-efficiency compared with other methods using the curiosity-driven method.
2.	This paper uses GNN with high combinatorial generalization abilities, which makes the method generalizes to environment with novel number of objects.
3.	The world models trained using the proposed method can easily generalize to many downstream manipulation tasks without specific trainings.


Weakness:
1.	The world model takes proprioceptive states of objects, instead of the image observation. This may be a hinder to the transfer to the real-world applications, in that: (1) we can not easily get the accurate proprioceptive states of objects; (2) objects with diverse geometries (e.g., real-world cabinets) do not have simple proprioceptive states, and therefore need image observations as input, otherwise the application range will be limited; (3) training using images has a much harder setting than proprioceptive states, which may potentially require the adjustment of the method proposed in this paper. Therefore, in my opinion, it is really important to train the proposed method based on image observations, instead of proprioceptive states.
2.	Though resulting in good data-efficiency, using an ensemble of M GNNs makes it heavy to train the model. Training with designed intrinsic rewards (e.g., the number of agent-object / object-object interactions, the state prediction accuracy, etc) may make the training simpler with only one GNN, while keeping the curiosity-driven good data-efficiency.

questions: 1.	Could the author show in detail how to train the proposed method using image observations instead of proprioceptive states as the input, and further transfer the proposed method in real-world applications?
2.	Could the author show the heaviness (e.g., the training time of the ensemble of M GNNs compared the MLP baseline) of the proposed method? Also, I wonder if it is possible to use some designed intrinsic rewards instead of the ensemble of M GNNs to train a world model with good data-efficiency?

limitations: The most significant limitation of this paper is the limited application in the real-world setting as the proprioceptive states instead of image observations used in the paper acts as an important constraint.
Besides, the heaviness in using an ensemble of GNNs may be a limitation. And this limitation may be even serious when the setting becomes more complex, which requires the ensemble to contain more GNNs with more model parameters.
review_title: null

Meta-review: metareview: Reviewer zPKR summarizes the paper well: "This paper an agent that learns a world model through exploratory play, which it then deploys to solve downstream tasks via a planner. The world model is an ensemble of GNNs, where nodes represent objects, which introduces an object-centric inductive bias. Effective playful exploration is achieved by using disagreement among the ensemble as a proxy for uncertainty, which allows the agent to select actions whose outcomes it cannot yet reliably predict. The authors show that this method works better than baselines, in the sense that it quickly starts engaging in “interesting” interactions (with objects)."

All reviewers unanimously vote to accept the paper. The limitations of the work are well summarized in the comment: "The most significant limitation of this paper is the limited application in the real-world setting as the proprioceptive states instead of image observations used in the paper acts as an important constraint. Besides, the heaviness in using an ensemble of GNNs may be a limitation." 

The benefits of the method outweigh the limitations. I hope authors work on extending their work to operate on sensory observations in the future. 
confidence: Certain
Decision: Accept