Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers task planning in relational domains using relational RL based on learned Q functions.
One challenge in this setup is representing the space of all possible states compactly.
To address this, the authors propose to define the Q function as a linear combination of lifted features. This allows those features to be queried for different MDPs, even with different numbers of objects.
In order to evaluate the Q function, however, these features must be fully grounded. This is achieved by introducing a context, which is a reduced set of objects from which the features can be grounded.
The work largely builds upon iFDD+ [1] with the difference of how the features are defined.

strengths_and_weaknesses: Strength:
- The idea of representing lifted features instead of grounded ones is compelling

Weaknesses: 
- However, it is unclear to me if the idea of lifted features really works well when only considering task planning domains without geometric conditioning on the features. Can it be shown that the lifted approximation really is a valid assumption? In the case of Loc + goal context, how many free variables are left?
- For a pure task planning scenario, the number of variables does not seem to be particularly high. I would assume that if the number of objects is larger, then the lifted approximation becomes less and less reliable.
summary_of_recommendation: As such, the paper is sound, but the method does not solve a particularly challenging or urgent topic in robotics, therefore, the potential impact is limited.
issues: - The memory limit seems rather arbitrary and outdated. 4gb of memory is not much in today's standards.
- Notation: Q(s,a) is a value, not a function. The function is called Q. Similarly for \phi(s,a).
- Section 4.1 could be written more clearly.
- I don't particularly like that the formal method is explained using examples only, especially in sec 4.3
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is focused on generalization over different planning problems. Their core idea is to exploit the first-order structure of the robot-environment interaction to provide an approximation of the Q-function that can generalize to large-scale problems and adapt to different environments. This approach is useful in situations where a model is not available and where the number of interactions with the environment is restricted, which is the case for real world scenarios. 
Their approach can deal with different objects, different number of objects, different initial conditions and final goals, without having to learn a different policy for every situation. 
In practice, they use model-free relational RL with online feature discovery using another algorithm from literature, iFDD+. 
They develop their method in the context of Relational MDP (RMDP), that allows them to learn a generalized policy $\pi$ able to solve different planning problems (any of the ground MDPs of the RMDP). The policy is induced by a linear approximation of the Q-function, defined over features discovered online by iFDD+. 
strengths_and_weaknesses: ### Strengths
The authors focus on a problem that is not very popular in the community, yet quite relevant. They provide a simple way to derive a generalized policy, based on a first-order approximation making the formulation independent of state, action and object space. This is important for transfer learning, as showed in the experiments.


### Weaknesses
Nevertheless, the authors do not present any baseline related to existing literature work on transfer learning. In addition to this, only one type of domain/task is considered in the main paper, the service robot domain illustrated in figure 1. 
summary_of_recommendation: The idea of the paper is to use domain knowledge to do contextual grounding and then use this to train a linear Q-function. The features are expressed as a set of conjunctive state predicates, discovered with the use of an off the-shelf algorithm (iFDD+).
The main reason motivating my recommendation us that there is not a single baseline comparison with related work on transfer learning, nor a comparison with any other Q-learning method from scratch. Like this the reader cannot understand how it compares with previous work or how faster it is compared to a setting without transfer, for example vs a policy learned with any SOTA model-free RL.
issues: - I find the writing style to be a bit raw and overloaded with notation (e.g. in Example 2 and 3). I appreciate that you provided said examples to make the reader understand how free variables are grounded. But maybe they could be complemented by another set of figures/a small scheme, also considering that there are only 2 figures in the paper.
- The reader does not get an idea of how your work compares with related work, baselines are missing
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose utilizing the first-order structure of some robotic domains to learn abstracted object oriented policies. The first-order structure refers to the fact that in some domains, multiple objects can belong to the same object type, and learning generalized policies over object and action types can lead to fast transfer on a new domain with same object types but different instantiations of the objects. The authors propose the use of an incremental feature learning algorithm (from prior work) along with contextual ground schemes (proposed by the authors).

The authors demonstrate the benefit of first-order learning by training in small service robot domain and successfully demonstrating transfer to a larger domain
strengths_and_weaknesses: **Strengths:**

- The authors successfully demonstrated the utility of using a first-order abstraction to MDPs while learning the important conjunctive features from experience as well. 

- The discussion in Section 4.4 about where the first order method is most applicable and useful is quite important, and I would like to see more of it in RL papers.

**Comments:**

- It seems that the grounding problem for the free variables could be a crucial bottleneck to successfully applying this approach broadly. The contextual grounding heuristics seem domain specific

- It would be interesting to see if the imposed relational structure shows any benefit in terms of faster learning curves for a non-relational MDP (i.e. without exploiting the first-order structure at all)

- As a non-expert in relational MDPs I found it hard to pinpoint the novel contribution of the paper. I believe the paper makes some important novel contributions, but the authors should provide an explicit description of their contributions vis-a-vis prior art in the introduction



summary_of_recommendation: I believe that the model-free problem formulation is novel and technically sound. I quite like the idea of using incrementally constructed logical features to decision-making. I would recommend acceptance
issues: - There seems to be a lot more prior research on RL for relational MDPs [1],[2] and I found the related work section quite sparse and superficial. I would request the authors to develop the text there to indicate what is known and what is the key research gap that your work addresses.

[1] - Diuk, Carlos, Andre Cohen, and Michael L. Littman. "An object-oriented representation for efficient reinforcement learning." Proceedings of the 25th international conference on Machine learning. 2008.

[2] - Kim, Beomjoon, and Luke Shimanuki. "Learning value functions with relational state representations for guiding task-and-motion planning." Conference on Robot Learning. PMLR, 2020.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper focuses on how to generalize and transfer planning knowledge across a set of related tasks and domains. To this end, this paper presents a method for learning how to approximate q-values over first-order features (lifted features) for Relational Markov Decision Processes (RMDPs) and the ability to leverage contextual knowledge to reduce approximation errors in a specific task. By learning a generalized policy, it can be re-used and adapted for multiple planning problems to enable more efficient robot learning and planning. The method uses a feature discovery algorithm to identify first-order features so as to approximate first-order approximations of the Q-values for the RMDP, which enable the approximations to then be transferred to specific task instances. Feature discovery can also be used to discover more fine-grain features for a task so as to further reduce the approximation error based only on the first-order features. 
strengths_and_weaknesses: Strengths:
- This paper presents a novel approach to generalizing task planning by discovering first-order features that provide a feature space to learn lifted q-values and transfer the learned knowledge to specific task instances for robot planning.
- The preliminary section is succinct and straight-forward and easy to understand.
- The paper presents experiments for a service robot domain that is interesting and challenging since it involves waypoint locomotion, human interventions, and high-level object manipulation actions.
- The results of this paper demonstrate that the proposed method of learning over first-order features provides useful generalization across domains and reduces computational cost and memory usage.

Weaknesses:
- The technical section (section 4) is very challenging to read, and understanding the mechanics of the method is difficult due to the writing. The introduction of first-order features (section 4.1) and first-order approximations is not immediately clear, and the example provided (Example 1) is also challenging to read. The reviewer suggests starting with the example and focus on clearly presenting the domain and associated RMDP, rather than presenting all of details on first-order features first then describing the example 1. Because of this, it also challenging to follow the online feature discovery process (Section 4.2) and the contextual grounding of free variables (section 4.3). (The authors have addressed this weakness in the revision)

summary_of_recommendation: This paper presents a novel and interesting approach to learning generalized policies for transfer learning across planning domains, which is an important problem for robot learning. This paper also presents experimental results and analysis that support the claims that the new approach improves computational cost when learning in new domains. However, the clarity of the writing for the technical sections is not easy to follow and is unclear, making it challenging to fully assess the validity and impact of this paper.
issues: The authors should aim to make their technical writing more clear and grounded in the example, to provide the reader with a better understanding of how the first-order features are discovered, used for the first-order q-value approximations, and grounded for the instantiated planning problem.
review_title: null

Meta-review: To enable generalization across different instances of a problem, the authors propose to learn a Q-function (and by extension a policy) defined over a more abstract state-action space that is oblivious to most of the specifics of the different entities composing the state save for their type (type such as 'person', 'object', 'location'...). The main technical contribution of the paper is how to resolve the ambiguity of grounding the Q-function to a specific problem instance if there are several entities of the same type. 

Pros:
- The reviewers find the idea compelling, the setting of relational RL original and relevant for transfer and planning.
- The consensus among the reviewers is that the experiments successfully demonstrate the ability of the proposed algorithm to generalize across different instances. 

Cons: 
- There are concerns regarding the generality of the proposed contextual grounding method, which is the main contribution of the paper. R-r (Reviewer rhBR) is unclear how a lifted planner could work without geometric information, and deplores that the grounding is only described informally through examples. R-u similarly thinks that the grounding appears heuristic and domain specific. 
- There are issues with the clarity of the presentation (R-r, R-j, R-d), especially the technical sections.
- R-d notes that comparisons to baselines are missing and that it is in general hard to appreciate the work in the context of the existing literature. R-u finds the related work section superficial and proposes some additional references to enrich it and better highlight the contributions of the paper.

The general consensus among reviewers is leaning towards rejection, but the authors are highly encouraged to engage with the reviewers and revise their manuscript accordingly. Among other things, addressing concerns regarding the generality of the method by improving the clarity of the technical section and better highlighting the contributions would greatly improve the quality and potential impact of the paper. 

I thank the authors for engaging with the reviewers and clarifying their submission. The reviewers are more positive about this work after the rebuttal period. The submission is original and improving generalization of RL through added abstraction is a commendable direction. I thus recommend the paper for presentation.
Decision: Accept (Poster)