Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper presents an approach for object-centric representation learning for planning to accomplish complex tasks. The learned representations are at an abstract level, resulting in desirable knowledge transfer capabilities between tasks. The learned action knowledge is represented using PDDL. Experiments have been conducted using blocks world and Minecraft domains. Results show that the agent was able to learn useful operators (actions) and that learned actions can be applied to different tasks. 

The object-centric idea is highlighted in the paper, though the work is more about learning symbols for abstraction, which is not new. The issue is that many of the "learning abstraction" works have been demonstrated in much more complex domains. For instance, the work of Konidaris et al 2018 (cited in the paper) has enabled robots to learn action preconditions and effects in the real world. In comparison, this work does not go beyond toy problems. 

The developed approach is more like an integration of a few existing methods, and the connections among the pieces are rather weak (see Figure 1). For instance, once the actions are learned, it looks like the agent faces a planning problem, and the planner does not have a way to go back to improve its learned representation. Since each component introduce its own errors, there's the cumulative error that can potentially make the whole system rather unstable. Some co-learning functionalities will be good for future work. 

The experiment section is relatively weak. For instance, the blocks world domain was mostly used as a demonstration platform. There were a couple of examples presented, while there were no statistical results discussed in the paper. Figure 6 is on the Minecraft domain, where the baseline (such as no transfer) is very weak. The results are not convincing to support the claims on transferability and learning efficiency. 

-------

The reviewer appreciates very much, and has read the response letter, which is mostly about clarity though it's not the main concern of the reviewer. 
review_title: The paper has a few issues

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an approach to learn a Probabilistic PDDL representation for tasks and generalize to new tasks with very small number of additional samples. It does that by first learning a compact and lifted representation across different training tasks that get the gist of multiple different objects, then converts it into symbolic representation (in PDDL) for future planning. 

I think the problem itself is super interesting and important: how to combine neural based and traditional approach remains a key challenge for AI community. And this paper seems to make a good attempt towards addressing it. 

However, I found this paper is hard to follow without any detailed descriptions of the actual algorithms.  The description is mainly high-level, many symbols/terms are mentioned but not defined, and there is no clue how to implement the idea. There seems to be 5 steps (Sec. 3.1-3.3 and Fig. 1). But it is not clear at all how the entire system work. Here I only list a few:

1. what are the features defined in Sec. 4.1? 
2. Are the options o learned or pre-defined? 
3. How is the clustering /precondition classifier/feature selection in Sec 3.1 done? 
4. How are the profiles in Def. 1 computed? 
5. What is symbol_10 in Fig. 2? How is it learned? 
6. How does the algorithm replace three propositions AOnTable, BOnTable and COnTable with a single predicate InHand? This is the “lifting operation” mentioned in the paper but I didn’t find any algorithmic description. 

While the author defers “the exact implementation and domain details to the appendix”, I don’t find many details unfortunately, except for the learned/generated results.  The algorithm block in appendix F doesn't help much, with many undefined subroutines and symbols. 

Given the difficulty in understanding the details, I feel that the paper is a bit premature for top-tier conference. 	

======

After reading the author's comments I still keep the score. 

After rebuttal/revision, the paper still has a lot of steps, many of them are human designed and are not well-defined. For example, in the author response, they said "their effect distributions look similar, and so are merged into one class type", what is the criterion to merge them together? In the revised paper, what is the procedure "COMPUTEMASK" defined?  In line 38 of the algorithm proposed in Appendix F in the revised paper, what does the sign of "approximate equal" mean?  Overall, this makes it hard to reproduce and it is not clear whether the proposed approaches can be applied to other problems than the specific tasks mentioned in the paper. 
review_title: The paper is too high-level and hard to digest. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: - No Baseline Comparisons: Paper proposed a way to generate object level representation that can be used across the tasks with same objects. Authors claim that this should reduce the number of environment interactions required to solve the new task. However, there is no baseline comparison being done to figure out how sample efficient it is.

- Paper assumed that tasks are solvable from set of provided options, will the method work of the task is out of provided options space ?




review_title: Paper proposes a way to create object centric representation that can transferred across task with same objects

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a planning approach that generalises across objects. Their approach groups objects into object types; if two objects have the same outcomes in planning then they are considered to be the same object. This is certainly a very interesting idea however I have some concerns about how this approach would work in more complex domains and how well the approach would generalise to imperfect object individuation.

Quality:

The partitioning of initial states I_o  (for each option) is achieved by “clustering state transition samples based on terminating states”. Since the state is defined by a set {f_a, f_1, …., f_n} I have a few questions: (1) How do you ensure that the clustering is invariant to the order of the objects, f_i? (2) The clustering must depend heavily on how the state is represented?

When learning the precondition model, if each object is treated independently, did the authors consider how this might be generalised to take into account interactions between objects? Similarly, when learning the effect model? 

In this work the authors assume that the objects have already been individuated. While this is an important first step, it’s not clear how the approach would generalise if the object were imperfectly individuated. For example if only half of the door was visible in a particular frame? How would this approach scale to more complex observations?
 
Were there any fail cases when partitioning options based on their terminal state? Particularly in the Minecraft setting?

It would be good to show examples of which object types were learned  and if there were any fail cases.

Figure 6a is a very nice result — is it for BoxWorld of Minecraft? (Similar question for 6b).


Clarity:
The authors could make their contributions more clear in the introduction.

From section 2 Is not clear what {f_a, f_1, …, f_n} are in the authors’ implementation.  The authors say that they are pixels, but go on to say that their “state space representation assumes that individual objects have already been factored into their constituent low-level attributes”. The first might suggest that f_i are image patches of objects while the latter suggests that they are vectors representing object attributes, it would help to clarify.

The authors use of the term representation is not well defined. From “a sound and complete abstract representation must necessarily be able to estimate the set of initiating and terminating states for each option” the authors’ definition of the term representation is not clear.

At the end of the “Partitioned Options” section, it would be good to clarify which approach will be used in this paper.

Figure 1 is not very clear. 

The feature selection process for determining which objects are are relevant to the precondition is only explained in the Appendix and may not be general. It is also not clear why this is needed? Why can the classifier not learn for itself which objects are useful and which to ignore? Is the implementation invariant to the ordering of the objects {f_i} for multiple i's?

When learning the precondition model do you treat each object independently and predict Pre(o) for each object? It would be good to clarify this early on.

It would help if in the introduction and experiments sections the authors were more clear about what they want to measure to show that their approach is beneficial in task transfer.



Originality and Significance:

The authors’ novel contribution is to propose clustering objects based on their effect, allowing transfer between tasks. This is an interesting way to group objects and could enable future research towards using objects in RL, for example by relaxing assumptions that the objects have already been individuated. 


Typos:

In the introduction, PPDDL, is not defined.

“initial and terminating states I_o and \beta_o” -- \beta_o is the probability of a terminating state, not the state itself.

In section 3.1 and Figure 1, steps 2: It’s not clear that you *generate* a positional forward model.

“ to compute whether the skill can be executed there” —> do the authors mean option rather than skill here?

Figure 4 is referenced before Figure 2.

Check the caption on Figure 4.
review_title: Interesting idea; a few concerns.

Meta-review: 
The paper proposes a general framework for learn object-centric abstractions represented using PPDDL (a probabilistic planning language).  The work assumes that objects and their attributes / features are identified.  The key contribution of the paper appears to be proposing to group individual objects into object types based on whether objects have the same outcome in planning. Using the learned object types, it would then be possible to transfer learned operators from one task to another.  The framework is demonstrated on block world (blocks are stacked on top of on another) and minecraft.  


Review Summary: Initially, the submission received negative to borderline reviews with R4 being the most positive (score 6), and R1, R2, R3 being more negative (scores 4, 3, 5).  After discussion, R4 lowered their score to 4 and indicated that they felt the work was not ready for acceptance at ICLR.  Overall, there was limited discussion by the reviewers.  Reviewers (R2,R4) found the direction of the work promising and interesting.  After the author response, reviewers indicated that the revision and author response clarified some points, but believe that the work is not yet ready for acceptance, as 1) there is a significant amount of hand-crafting required and 2) parts of the approach is still not clearly specified.  

Clarity: As some reviewers note, the description of the framework is at a very high level, making it difficult to follow with missing details on specific details of how the object types are groups.  The specific aspect of the work that is novel is also not clearly stated, thus making it difficult to judge the originality and significance of the work.  After revision (the authors added brief paraphrase to the introduction to clarify the contribution, and additions to the appendices providing more details on how the difference steps work for the Minecraft scenario as well as failure cases), the manuscript is improved but the overall manuscript is still difficult to follow.

Pros:
- Interesting and important problem (combining probabilistic/neural approaches with symbolic approaches) that is timely and deserves attention
- The idea of clustering objects based on their effect is interesting (R4).  
- The framework proposed by the paper is interesting and potentially useful direction and can stimulate followup work 

Cons:
- The paper is difficult to follow with symbols/terms that are not clearly defined and missing details. (R1) The specific contributions of the work, wrt prior work, is also not clearly stated.
- The novelty/contribution of the work on top of existing work (Konidaris et al 2018, Ugur & Piater 2015, etc) is not that clear (R2)
- The experimental setup is weak with limited comparisons and no statistical results. Overall, reviewers felt the results are not convincing enough to support claims on transferability and learning efficiency.
- Lack of baselines comparisons (R3).  In the rebuttal, the authors argue that there is no appropriate baselines.  
- The set of steps that is involved is fairly complex (R1), with many important details provided in the appendix
- There are concerns about the generalization of the approach as many of the steps are handcrafted (R1, R4).  In the provided scenarios, many of the steps, including the set of provided options, and representation of objects, are manually designed.  

Recommendation:
The AC agrees with the reviewers that the paper is not ready for acceptance to ICLR.  It is the AC's opinion that the work addressing a very interesting problem and 
would potentially be of interest to the community.  However, the exposition of the paper needs to be improved so that 1) the contribution of the work over prior work (Konidaris et al 2018, Ugur & Piater 2015, etc) is clearer 2) the assumptions and details of the proposed method is also clearer and easier to follow.  The authors are encouraged to improve their work and resubmit to an appropriate venue.  
Decision: Reject