Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a framework for learning environment dynamics models more efficiently by leveraging small amount of human guidance. Specifically, the paper points out that for common robotics tasks, transition models defined in terms of object attributes and relationships are inherently sparse and local (e.g., an action only affects a small subset of the state space), and exploiting such sparsity and locality can improve the data efficiency of model learning. The paper proposes a framework that allows human users to specify such structures with a PDDL-like planning domain language, and the framework converts such specification to a differentiable computational graph that can be optimized using environment interaction data. The method is evaluated on a BabyAI toy environment and a simulated robotic manipulation environment. The paper shows that the resulting sparse model tends to be more sample-efficient and generalizes better to unseen tasks compared to a “dense” object-level transition model.
strengths_and_weaknesses: Overall I enjoyed reading the paper — the idea of leveraging a small amount of human priors to build a sparse transition model for online planning is conceptually simple and is well-executed in the paper. I especially like the manner that human prior is introduced. Specifying the action effect sketch using a PDDL-like language and letting the neural network fill in the low-level details is both elegant and practical.

However, I do have some comments that are mainly focused on the generality of the proposed method, especially when one wishes to apply it to more complex manipulation tasks.
The method assumes a finite number of atomic primitive skills. However, it is unclear to me how this can be applied to more practical settings where skills have parameters, e.g., grasping angles and placing targets. One might argue that the neural network can in addition take in the continuous parameters as input to model the effects of continuously-parameterized skills. But the fact that the skills are not atomic would likely make the locality & sparsity assumption invalid. For example, a grasp may fail, and the end effector may accidentally bump into other objects. In addition, it seems like the paper assumes fully observed & deterministic transition, which is not practical. How would the model learning and the online planning process handle stochastic transition? I hope the authors could address these limitations in the next revisions of the paper.

My second set of comments concern the experiment evaluation, especially the choice of baselines.
- First of all, DT seems to be not the best baseline to compare against, as it tends not to perform well with sparse reward, and it’s a model-free method. How about comparing to more powerful & structured transition models such as PlaNet / Dreamer (Hafner et al., 2019)? They also assume observation-space input and sparse reward, which could make a fair comparison.
- Second, how does PDSketch PDSketch compare to unsupervised methods that try to discover transition sparsity such as Xia et al., 2018? It would be interesting to see whether an unsupervised method can discover similar sparsity patterns as human supervision and how data efficiency of PDSketch scales with the number of training samples.
- Thirdly, PDSketch also shares some similarities with Regression Neural Networks (Xu et al., 2019), which is not mentioned in the related works. RPN shares the same input & output as PDSketch (object-centric encoding input & first-order logic goal). RPN also assumes that humans provide prior guidance on predicate dependencies, which is similar to the idea of sketching action definitions. I would like to see conceptual and possibly empirical comparison between PDSketch and RPN.
- Finally, can a network learn sparsity by itself? How does the effect of human-specified sparse transition rules scale with the amount of data? Table 4 only compares the sample efficiency between PDSketch and model-free method. This calls for an experiment on how PD-Base or similar model-based approaches scales with the amount of training data.

questions: Please refer to my main comments ("weaknesses") for questions & comments. They mainly concern limitations of the method and baseline comparison.
limitations: Like mentioned in the main comment, the method makes assumptions about the primitive skills as well as the nature of environment dynamics. I hope the authors can elaborate on these assumptions and how they can be relaxed in the next revision of the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a symbolic language for adding structure to world models in robotics and for defining task-independent heuristics. The language describes predicates, which explain the state of the environment, and actions, which change the state using pre-conditions and effects. Based on the descriptions, neural networks are instantiated to ground the language.

strengths_and_weaknesses: I find it difficult to review this paper because many pages are spent describing the PDSketch language and important model and experiment details are delegated to the appendix. For example, the use of a VQ-VAE for discretization is mentioned in the third-to-last sentence of the methods section. Moreover, the robot experiment, which should be the highlight of this paper, is described in only one paragraph. The scientific contribution might be sufficient for NeurIPS, but the paper needs significant work.

Strengths:
* The paper proposes a general way of automatically adding structure to world models for robotics.
* The proposed method has a fall-back mechanism onto a generic world model, so that the designer does not have to specify every detail of the environment.

Weaknesses:
* There is no discussion of the difficulties of getting training data for the proposed model. It is not clear to me if ground-truth predicates are used to train the predicate classifiers. 
* The results show that the model can fit training tasks from few demonstrations, but the evaluation of generalization to novel tasks is limited. Figure 9 shows examples of four goal states, but there are no quantitative results.

# Detailed comments

– Starting on line 34, an argument is made against “classical hand-engineered approaches”. There are no citations and the argument is not revisited in the rest of the paper. I think it would be useful to devote a paragraph to task and motion planning, and how the proposed approach overcomes its limitations.

– [1] is a recent paper that appears to be quite related to this work (e.g. they also put emphasis on sparse transition models). While it is cited in this paper, it is not explicitly discussed. I would like to see a short discussion of the similarities and differences.

– PDS-Abs and PDS-Rob are both described in one sentence. This makes the BabyAI experiment difficult to understand without going into the appendix.

– In the program shown in Figure 5, I think it should be “wetness::assign ?o1 (??h (wetness ?o2))” because we are checking if o2 is a container with paint?

– Line 58: space after comma.

– Reference for [1] has the year 2010 instead of 2019.

– The robotic environment seems to be derived from RAVENS (https://github.com/google-research/ravens) / Transporter Nets. This is not stated explicitly.

– There are a few pre-2018 works that could be cited for graph neural networks. E.g. [2].

# References

[1] Victoria Xia, Zi Wang, and Leslie Pack Kaelbling. Learning Sparse Relational Transition Models. In ICLR, 2019.

[2] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729–734 vol. 2, 2005.

questions: 1. What training data is required by the system? How difficult is it to collect this data on a physical robot?
2. I believe it is assumed that each symbolic action has a pre-defined controller. Could you comment on the difficulties of coming up with the symbolic actions and their controllers in your robotics domain?
limitations: "Limitations we hope to address include the lack of hierarchy and the inability of the method to discover novel factorizations." – To me, the biggest limitations of this method concern gathering training data with annotated symbols and coming up with a PDSketch description of a domain. I am not saying that these limitations warrant a rejection, but they must be discussed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes PDSketch to generate the object and its features as high-level constructs of the transition model. Compared to earlier tools like PDDL, PDSketch allows vector values in computation and supports learning of unspecified functions from data. PDSketch is a collection of neural networks to convert encode the state, evaluate the goal and build a transition model. The experiments show that PDSketch adapts to novel first order logic goal objectives for navigation in BabyAI and Painting Factory. 
strengths_and_weaknesses: The objective of PDSketch is to efficiently model the “local” changes induced by an agent’s actions and “sparse” changes that occur in object properties for generalization in unseen scenarios. The paper describes how predicates, goal evaluation and transition models can be represented as computation graph and learned. Overall, the paper is well-motivated and detailed. 

There are assumptions that on object properties that are task relevant, defining its predicates, transition models. This may increase significantly as the different category of objects, their properties and their interdependence increases. Although the authors motivate the use of factored representation of transition model and eval function, this implies that all the objects and their properties are known at test time. The tasks considered are quasi-static, and it is not clear if the proposed approach can be somehow extended to handle dynamics, noisy sensing and actuation of any physical robot. While the proposed approach is shown to perform better than BC or DT and its variants, some stronger baselines can include recent variants of PDDL like [1] and as cited in related work.

[1] Learning symbolic operators for task and motion planning, Tom Silver*, Rohan Chitnis*, Josh Tenenbaum, Leslie Kaelbling, Tomas Lozano-Perez, IROS, 2021
questions: - What is #Obj. Gen. in Table 1?
- PDS-Rob uses predefined transition model, so it seems that it is only learning object properties. Can you clarify how feasible it is obtain the transition model without simulator in real tasks?
- What is the notion of timestep in change of properties? For example, the wetness property changes within the next timestep or based on location change?
- How to compare the capability of PDSketch over existing planning languages in terms of modeling general-purpose TAMP domains?
limitations: The paper briefly mentions the limitations as not modeling object hierarchy and unable to detect novel objects and properties. Some discussion on potential societal impact can be included as how much modeling time or iterations are needed for human designer to specify the object attributes, or how a system built with PDSketch is easy/hard to update with task specification as compared to more unstructured learning approaches like BC or DT.
review_title: null

Meta-review: metareview: This paper present a new domain definition language aimed at defining high-level structures of transition models. The benefit of using this language is the possibility of injecting human priors into the models thus achieving better learning.

The idea of the paper is novel and bold. 

The reviewers highlighted different aspects of how the paper could be improved in terms of clarity and structure (potentially move some of the experiments from the appendix), but generally agreed on the value of its contribution.

I encourage the authors to incorporate all the feedback received.
confidence: Less certain
Decision: Accept