Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to automatically learn a PDDL-like abstractions for high-level problem solving based on the different states (either symbolic representation, or high-dimensional embeddings) of the objects in the environment. Such abstractions could support a PDDL solver to find a solution for the task where the abstractions were learned, and the authors claim such abstractions could be shared among different tasks as long as object types remain the same.
main_review: The main contribution of this paper is that the author devises a learning framework to abstract object types, pre and post-conditions by grouping object/symbol instances that are option-equivalent or effect-equivalent. Although the proposed method, to some extent, alleviates human efforts to define the planning domain manually, it has two main drawbacks that cannot solve the fundamental problem of the human-defined planning domain. 

1. The author claims the learned object-centric abstraction can be transferred to a new task with the same object type. However, it makes no sense if we cannot define how same is w.r.t. the same object type in the new task. For example, we learned the abstraction from picking an apple on the table, and now, given a new task, we want to pick an apple on a tree. These two apples are the same in semantics, but they are different in the planning domain as they have different preconditions for the `pick` action. Such object-centric abstractions are still more of task-specific representation, even for the same objects in semantics. As such, very likely, they have different constraints (preconditions) that limit the actions across different tasks.

2. As mentioned in the paper, we could use Problem-Specific Instantiation tricks to solve the problem of the same-type object but have different preconditions in a different scenario, i.e., adding some task-specific constraints for the object types in different scenarios. However, such a process is exactly the same as how we manually maneuver the planning domain to adapt to a different scenario, and it does not solve the fundamental limitation of the human-defined planning domain: Learned symbolic predicates of specific object types are not general enough to across different task. However, the proposed object-centric abstraction has the same issue. It only groups the similar conditions it observed, but does not actually abstract the preconditions of why an action could be performed. Like the aforementioned picking apple example, if we place the apple on $N$ different heights, the proposed method will lead to $N$ different apple object types as they have N different preconditions. If we place the apple even higher--a condition out of the $N$ previously seen cases, the proposed method still cannot solve this problem as it does not abstract the `reachable` concept for the picking action.  
summary_of_the_review: The authors proposed a learning framework to learn object-centric abstractions for high-level planning, yet it does not solve the fundamental limitation of the manually designed planning domain, and it is more of an autonomous labeling tool that partially alleviates human efforts.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for learning symbolic, object-centric abstractions from object-factored environment observations for long-term planning tasks. It extends the symbolic representation learning framework by Konidaris et al. (2018) by factoring the state into objects, learning object type abstractions, and “lifting” the model to operate on these object-centric, typed abstractions. The method is successfully demonstrated on three environments of complexity ranging from simple per-object discrete feature vectors to a Minecraft long-horizon planning task from (object-factored) pixel observations.
main_review: This paper is overall well-written, clearly presented and generally of high quality. The problem of learning sample-efficient abstractions for long-term planning is an important and difficult problem, and taking an object-centric approach to address this problem is very timely and of high significance. The proposed method is largely based on a framework introduced in prior work (Konidaris et al., 2018), but introduces sufficient novelty in terms of integrating object-centricity in an elegant way into this framework. The experimental evaluation is carried out on sufficiently complex environments to demonstrate that the method can solve tasks that are not trivial.

My main concern with this paper is with regards to its sample complexity claims and limited experimental comparison. The paper claims (in the abstract) that the proposed method results in being able to obtain a successful agent for long-term planning „with considerably fewer environment interactions“. Given this central statement, I would expect some form of quantitative evaluation against a baseline that demonstrates this strong reduction in environment interactions (which unfortunately is not provided). Potentially it is trivial to see that this method uses fewer environment interactions than prior works (for example the works by Kaiser et al. (2020) and Hafner et al. (2021), as discussed in the paper) — this would not be a fair comparison, however, as the presented method assumes that environment observations are pre-factored and pre-processed into object components (e.g. provided as abstract per-object feature vector or as cropped images around individual objects). Extracting object information in this way without direct supervision is a highly non-trivial task, which might partially explain the sample inefficiency of earlier methods (Kaiser et al., 2020 and Hafner et al., 2021). This present paper sidesteps this issue by providing ground truth information. I would recommend either 1) significantly reducing the prominence of the sample efficiency claims in the paper or 2) providing a fair experimental comparison against a baseline that has access to the same factored, pre-processed observations and abstract options (instead of low-level actions) as the presented technique.

summary_of_the_review: Despite the above-mentioned limitations, I think that this paper carries sufficient novelty and is an interesting, significant extension of prior work in this area, which opens up interesting follow-up questions and opportunities for future work. Overall, I recommend “weak accept”.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for learning an object-centric symbolic representation of an environment that allows for planning. It extends the framework introduced in Konidaris et al. (2018), which learns a symbolic representation of an environment that can be expressed using a PDDL for planning. Importantly, the method presented in Konidaris et al. (2018) does not make assumptions about structure in the environment and how different states may relate to one another. As a consequence, the learned representations are highly tasks-specific and transfer/generalization to new tasks and across states is not possible. 

In this paper it is assumed that the world consists of objects and that similar objects are common across tasks. This enables object-centric representations (and the associated model) to be reused across tasks, and organizing objects according to 'object types' that behave similarly when acted upon. This is expected to improve learning efficiency and generalization/transfer to new tasks. 

The method presented here incorporates techniques for merging objects into object types, and integrating problem-specific information to ground these representations into specific tasks. The working of these techniques is validated experimentally, and it is shown how greater sample efficiency can be obtained when generalizing to new tasks.
main_review: The focus of this paper is on learning representations of an environment that allow for planning, which is an important problem. The main driver for the presented improvements is the assumption that the world can be organized according to objects that are relevant for many different tasks. Recently, there has been an increasing interest in incorporating such real-world priors in representation learning methods, which makes the overall focus highly relevant. The paper is also well written and the ideas are presented clearly. Further, the paper does a good job at crediting prior work wherever appropriate.

The main improvements in this paper compared to prior work of Konidaris et al. (2018) are

1. A method for estimating object types by evaluating their effect distributions when applying (subgoal) options. 

2. A method for integrating task-specific information in the learned representations, i.e. grounding them for particular problem instantiations.

I think (1) is quite clever since it allows for a grouping into object types that focuses on their functionality, which transcendent a grouping based on other features such as appearance. On the other hand, it is not quite clear to me why a grouping based on effect distributions alone is preferred over one that also considers the pre-condition? The assumption that an option only affects the object it acts upon is quite strong and in my view somewhat unrealistic, yet it wasn't quite clear to me how this relates to this choice either.

Regarding (2), I find that the current presentation falls somewhat short. For example, it is unclear to me how it is determined what objects/operators require grounding (is this prior knowledge that is assumed given. Eg. for the workbench or door in the experiments). It is also not clear to me how the clustering proceeds for X. In particular, is X still partitioned according to objects similar to S? And how is the result from clustering X connected to clustering S, i.e. how are the clusters mapped onto each other and how is it ensured that a similar split is obtained? I am particularly interested in understanding how much domain knowledge is injected in (2). It may very well be that I misunderstood something here, so further clarification is much appreciated.

The approach is evaluated on three different environments. On Blocks World it is demonstrated how the 'correct' environment representation is learned and the benefits from merging into object types is clearly demonstrated.

On the Craft environment from Andreas et al. (2017) it is shown how the approach scales to situations where there are many different objects, and further how grounding can be achieved for operators that require identities for a specific problem instantiation. Here the identities are derived from the option masks, and it is shown how a grounded operator is learned. However, similar to above it is unclear to me what are the steps needed to achieve this result, i.e. how is the need for having identities identified (or is it assumed given?), how is it determined which objects require ids (or is it assumed given?), and how is the id connected to the object. What I liked about this experiment is how not the 'ground truth' types are recovered but rather a suboptimal grouping based on the data collected so far. It also suggests that it may be worthwhile to relearn the object types after having interacted with the environment for a number of steps, which is something that could be commented on.

On the MineCraft environment from Johnson et al. (2016) it is shown how this approach can be combined when learning feature representations of objects from their pixel representation rather than using pre-specified features. Here each object's appearance is given by a high-dimensional image although it is still assumed that objects are pre-segmented. Here, the learned operators must be grounded to consider a particular door (and I have similar questions as above). By considering 5 procedurally generated tasks it is also demonstrated how object types allow for transfer compared to prior work of Konidaris et al. (2018), which clearly demonstrates the benefits of this approach.

As a final comment, I would have liked to see some more discussion of the limitations of this approach. The appendix includes several failure cases, which are interesting, but the framework presented here makes several strong assumptions that are worth commenting on: (1) the frame assumption, which appears crucial for learning effect distributions, (2) that options only affect the object they act upon, (3) that the world comes pre-segmented according to objects. When also inferring the objects themselves (i.e. the segregation process) the merging into object types may affect one another, since by adopting a different notion of an object their effect distribution is likely to change.

summary_of_the_review: This is a well-written paper that introduces several techniques into an existing framework for learning symbolic representations for planning. The proposed contributions are interesting and the improvement over prior work is clearly demonstrated. 

While the final approach itself is still limited, in the sense that it relies on several strong assumptions, it advances what symbolic approaches are currently capable of. Further, by highlighting the trade-off between creating object types and problem-specific grounding it contributes an interesting problem that may not yet be on everyone's radar.

While I would like to see this paper accepted I also think it can still be improved. The presented approach is quite complex, and although the authors do a good job overall, the presentation for the grounding is lacking and it is not made sufficiently clear how this is achieved and what knowledge is assumed to arrive at the presented results. I also suggest that the authors consider creating a figure to accompany figure 1 that specifically visualizes step 1 (and step 5). In order to make the paper accessible to a wider audience, an introduction to PPDDL notation would be helpful to include.
review_title: null

Meta-review: This paper extends the symbolic representation learning work of Konidaris et al. (2018) to be object-centric, and generalize in this respect. All the reviewers agreed that this is an interesting problem, and that the approach is novel.
Two reviewers gave positive evaluations (6,8), and one reviewer gave a mildly negative review (5), where the main critique is that the method still requires some human effort in designing the planning domain.
While completely alleviating human efforts is definitely a good goal to pursue, I believe that it's too high a bar to ask for in the setting of limited data, and I believe that there are many real world problems where requiring some human effort is not too limiting. 
Therefore I recommend acceptance. 
Please take all reviewer comments into account when preparing the final version.
Decision: Accept (Poster)