Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduced the Entity-Factored Markov Decision Process (EFMDP) for modeling the entity-based compositional structure in controlling tasks. The authors studied several structured policy architectures that can utilize the factorized discrete entities on a suite of manipulation tasks. Experimental results showed that structured policy architectures have faster learning speed in general when compared to MLP policy. Robust extrapolation and OOD generalization at the skill-level can also be observed.
strength_and_weaknesses: ### Strength
- The experiments are well designed, and the results support the author's statement well.

### Weakness
- Major:
  - As the authors mentioned also, the current model takes vector state as input. However, a model that can handle visual input is more desired. A related work, SMORL[1], has shown promising results given visual input. Similar results can also be obtained from there.


-------------After rebuttal-----------------------

Thanks for the author's responses. I decide to increase my score to 5. 
1)I still think the proposed work shares some common motivations with SMORL, such as the reduced size of effective state space. The authors did investigate more architecture than the SMORL work. However, the experiment's only finding is that utilizing the factorized entity-based structure improves the performance, which does not have much novelty. Similar findings have been demonstrated in prior work like SMORL. 2)Though given the ground-truth entity states and goal state as input, the proposed work is solving a much simpler task than SMORL, I'm convinced that the OOD experiments are meaningful, and I'd like to increase my score to 5. However, I am leaning toward rejection based on the current experimental results.


[1]: Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with object-centric representations.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The results can be reproduced with reasonable effort.

summary_of_the_review: This paper introduced the Entity-Factored Markov Decision Process (EFMDP) for modeling the entity-based compositional structure in controlling tasks. Experiments showed that structured policy outperforms the MLP policy. However, similar results have been obtained from related work already.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for compositional structure modeling in control tasks, that is a modification of the typical Markov Decision Process (MDP). The framework, called Entity-Factored Markov Decision Process provides several insights into developing robot control policy architectures for compositional generalization. Results on simulated robot manipulation tasks show extrapolation to different number of test entities, and stitching of skills in novel ways. 
strength_and_weaknesses: Strengths

- The paper provides a formal study of understanding compositional tasks, which can be useful for several papers that target this domain, for developing future algorithms 

- Understanding deep RL policy architectures is generally not very well-studied, and this paper proposes a formal study of this, which can be useful for to several works in RL. 

- The experimental evaluations are very detailed and show the emergence of properties like extrapolation to out-of-domain targets, and stitching of skills for compositional generalization. 

Weaknesses

- The limitations of the proposed framework are unclear. It will be good to be precise, clear, and transparent about exactly which king of control problems can be cast in this framework, and exactly when will the benefits be observed empirically, compared to casting the problem in the usual MDP framework, 

- No external baselines are compared against. In my understadning, a one-on-one comaprison with *any* RL algorithm that is goal-conditioned is possible, and should be performed so that the it is clear how the algorithms developed in this framework comapre to existing goal-conditioned RL algorithms under the MDP framework. 
clarity,_quality,_novelty_and_reproducibility: The paper is largely clear and easy to follow. The proposed framework is novel to the best of my understanding. Experiment details are provided for reproducibility. 
summary_of_the_review: I think the paper has interesting insights and opens up possibilities for future works to build upon, and I am leaning towards acceptance. Please look at the details of strengths/weaknesses above. 

------ AFTER AUTHOR RESPONSES ----

The authors have updated the limitations section of the paper to include more details, so the current paper is better scoped in terms of its contributions. My second point regarding comparisons to baselines is not adequately addressed and would likely require significant additional work: I was referring to comparisons with other goal-conditioned RL approaches (not just RL + HER) for example approaches described in this paper [A]. Currently, the baselines are not "external" goal-conditioned RL approaches in the sense that they do not correspond directly to a prior paper. 

In addition, after going through the other reviews, some concerns regarding proper ablations to tease out the benefits in performance are unaddressed, as overall performance is not indicative of whether confounding factors were at play. 

Based on these, I am not able to strongly recommend accept, but would keep my rating of weak accept with lower confidence as I think the paper is interesting for future work to build on

[A] Liu, Minghuan, Menghui Zhu, and Weinan Zhang. "Goal-conditioned reinforcement learning: Problems and solutions." arXiv preprint arXiv:2201.08299 (2022).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a framework called Entity-Factorized Markov Decision Process (EFMDP), where the task is factorized across different entities, and the state and goal configurations of the entities are permutation-invariant. Based on this framework, the paper studies various permutation-invariant architecture designs, including Deep Sets, Graph Convolution, and Self Attention. Experiments on object pushing, switching, and stacking demonstrate that the method with permutation-invariant architectures learns faster than MLPs when there are multiple objects in the scene, and can do zero-shot extrapolation to an unseen number of objects.
strength_and_weaknesses: **Strength:**

- The paper presents a neat and intuitive idea to factorize the state and goal configurations across entities, which is beneficial to generalization and scaling.
- Experiments are extensive, where multiple permutation-invariant architectures are studied in various different settings, e.g., zero-shot extrapolation and zero-shot stitching. The results demonstrate the effectiveness of the proposed method and empirically show how different architectures compare for these tasks.
- The paper is well structured with clear problem definitions, and how different architectures fit in the big picture. The method is well-motivated and easy to follow.

**Weakness:**

- The method is only evaluated on robot manipulation, it would be nice to also evaluate on other settings such as multi-agent systems, and strategic game playing, which would make the experiments stronger.
- While the paper shows different permutation-invariant architectures such as Deep Sets, Graph Convolution, and Self Attention can fit into EFMDP, the framework itself doesn’t provide any insights or guidelines as to which one is better in certain settings, which makes the theoretical analysis less useful.
clarity,_quality,_novelty_and_reproducibility: - As mentioned, the paper is well-written and easy to follow. The writing quality is high.
- As far as I know, the paper has good novelty in terms of its EFMDP framework, but I’m not certain if similar ideas exist in multi-agent RL literature, maybe other reviewers could also verify this.
- The paper should be easy to produce since it provides detailed experiment settings, and the authors have also promised to release the code upon publication.
summary_of_the_review: Overall, I think this is a solid work with good novelty and sufficient experiments. I believe the proposed EFMDP setting could be useful for many other problems as well. Therefore, I vote to accept this paper.


--- After rebuttal ---

I'm happy with the authors' response and decide to keep my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an architecture that leverages the symmetry of entity-centric goal specification. The author extends the MDP framework to modeling the symmetric structure of the goal specification. The framework motivates using architectures that are agnostic to permutation of the inputs like Deep Set, Self-Attention, and Graph convolution. Experiments are conducted in two robotic tasks, showing the advantages of the proposed architectures over the MLP architecture. 


strength_and_weaknesses: Strengths:
* The paper is clearly written. 
* The problem studied is interesting and important.
* Results quantitatively show the effectiveness of the proposed method. 

Weaknesses:
* The proposed architectures are not novel (according to the authors, the application of these architectures on the experimented robotic tasks is novel; I don't work on this space so I am not qualified to verify this claim). 
* The authors show mostly evaluation metrics without analyzing the behavior of the learned policies. How do the learned policies solve the entity tasks? Sequentially or simultaneously? Visualizing the change in the attention over the goals may offer such insights. 
* I am not sure why the "oracle" is called an "oracle" rather than a rule-based baseline. The results would be interpreted much differently if the "oracle" were considered as a baseline. 
* I am not convinced why the approach would work better than learning a policy separately for each entity. The authors mention that entity-entity relationships may not be considered in such an approach, but I can easily enforce some constraints to the policies through rewards or demonstrations (e.g. don't push other blocks while moving a block). The proposed method may resort to learning the "oracle" policy where it only attends to one goal at a time. Analyzing the behavior of the learned policy is important to show that the method learns a non-trivial policies. The current numerical results show that most of the time the method is on par with the "oracle" baseline, which hints that it may have learned trivial policies.  
* The proposed architecture can outperform MLP because it simply has more learning capacity. The authors should show the number of parameters of the models in the comparisons. 
* When training the MLP, do you permute the input entity subgoals in each episode or do you always feed the subgoals in the same order? Doesn't permuting the subgoal order make the MLP somewhat invariant to permutation?
* The problem studied, although interesting, is of limited practicality. In the real life, the decomposition into entity goal is usually inferred rather than given explicitly to the policy as in this work (e.g. "switch all light off" rather than "switch light 1 off, switch light 2 off, ..."). I understand that the authors are studying a sub-problem of this more difficult setting, but I just want to point out a limitation of the current task formulation. 
* The conclusion says "These policy architectures decompose goal-conditioned tasks into their constituent entities and subgoals" but this seems like a false claim, as the goal specification already consists of entity subgoals stitched together. Figure 3 illustrating the architectures also does not show any goal decomposition.  Am I missing something?
clarity,_quality,_novelty_and_reproducibility: Please see my strengths and weakness for clarity, quality, and novelty. The training setting is not clearly specified so reproducibility is limited. 

summary_of_the_review: The paper applies recent state-of-the-art architectures to a special class of robotic tasks. The author provides a theoretically-motivated justification but I don't find it providing significant value to the community, because the authors do not provide analyses to support the justification. I am leaning towards rejecting the paper.  

=====After Rebuttal======

I have read the responses from the authors and decided to keep my current score. My main ground for rejection is that the proposed framework lacks technical depth and it does not lead to novel methods, surprising findings, or useful insights. The framework justifies using architectures that are permutation-invariant for problems that require that property, but such a choice seems to be quite natural even without considering the framework. In addition, it is unclear from the experimental results whether the data provided to the agent enforces permutation invariance and the learned policies are verified to have actually learned that property. Showing improved performance on various task conditions is not sufficient; there could be various other properties of the experimented architectures that affect their performance. More ablation studies need to be conducted to identify the cause of the improvements and to demonstrate that the actual cause coincides with the anticipated cause.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of learning goal-conditioned policies in settings where the number of entities (objects) and goal compositions vary. The paper proposes the Entity-Factorized Markov Decision Process (EFMDP) framework, which takes advantage of the symmetric nature of entity-centric goal specifications. Using this framework, the paper explores different  architectures that are invariant to the permutation of the inputs (e.g., Self Attention, Graph Convolution, and Deep Sets) and are able to exploit the entity factorization. Experiments on simulated robot manipulation tasks demonstrate that the framework is capable of zero-shot extrapolation to different numbers of entities, and that the permutation-invariant architectures outperform standard MLP baselines in terms of learning efficiency.

The paper received four reviews that exhibit a notable amount of variability in their overall assessment of the paper. That said, the reviewers generally agree that the paper is well written, that it addresses an interesting problem, and that the experimental results provide a empirical demonstration of the effectiveness of the EFMDP framework.

The reviewers disagreed regarding the novelty of the proposed framework and the significance of the contributions. Two of the reviewers (KcU5 and Et3S) comment that the use of object-centric representations as a means of achieving structured policies is not new. The theoretical focus is on the role of permutation invariance and while the authors claim that the experiments demonstrate why permutation invariant policies are effective for multi-entity control tasks, some reviewers initially commented that this is not supported by the experiments, which are instead about generalization to novel compositions and largely empirical, resulting in a mismatch between the theoretical discussion and the experimental evaluation.

In light of the disparity in the reviews, the AC held a virtual meeting with three of the reviewers (Et3S, dhRF, and HR7M; Reviewer KcU5 was traveling and not available to meet, but did follow up with the AC and reviewers), during which the strengths and weaknesses of the paper were discussed at-length. While the reviewers were not able to come to a full consensus, they generally agreed that the paper provides more empirical evidence compared to recent work (notably SMORL), but that beyond this empirical evidence in terms of success rate, the analysis of the different architectures and insight into their performance is lacking. A revised version of the paper that better aligned theory with the experimental evaluation, including more detailed analysis and insights into the results would provide a valuable contribution.
justification_for_why_not_higher_score: In their response, the authors mention that the contribution of the paper is in showing **why** permutation-invariant policies work well for multi-entity control tasks, however the reviewers find that the results are largely empirical, with relatively little analysis or insight into the results.
Decision: Reject