Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper casts rearrangement planning as a sequence modelling problem, and applies the transformer architecture to solve it. The user prompts the model with a demonstration, e.g. placing a few plates into a dishwasher, from which the model infers the user’s preferences for arranging the scene. Given a new set of objects, the model can then predict a sequence of actions, including opening drawers and moving objects, to complete a rearrangement which is consistent with the preferences in the prompt. Target poses are selected from a set of discrete options. Experiments show that this particular transformer architecture is well-suited to modelling these rearrangement sequences, generalises well in several ways, and the method is successfully validated with a real robot demo.
strengths_and_weaknesses: Strengths:
1. The technical quality and execution of the paper is impressive, in adapting this powerful transformer architecture for the rearrangement setting and addressing a challenging rearrangement scenario. There are many useful insights and techniques in the paper which the community will benefit from seeing used. For example, the dot-product technique for selecting an instance used on Line 132.
1. The authors have thought carefully about how they envisage the user interacting with this system (e.g. through preference prompts), which makes it a promising direction for deployment to robots working with users.
1. Unifying actions for moving objects and opening/closing containers is done elegantly, and is a useful capability for rearrangement methods to expand from tabletop environments to more complex scenes.
1. The real-world demo is useful, because it gives insights on which parts of the method make it well-suited for deployment on the real robot and which components are challenging to transfer.
1. Investigating the effect of the number of objects, especially in the out-of-distribution setting, is interesting. There is insightful analysis about failure cases, e.g. closing the dishwasher early, which is useful for future work to build on this. Other useful ablations are also performed.

Limitations:
1. A major contribution of this paper seems to be encoding the preferences of a user for the order in which objects are placed. However, the importance of this capability needs to be motivated more persuasively. If the end arrangement is the same, why does the user care about the order in which objects were moved there? Without this motivation, the impact of this contribution is not very clear. One way to frame this might be: human demonstrations contain not only preferences over end states, but clues about how to achieve those end states (e.g. open the dishwasher first). That is why encoding the order of actions is also useful.
1. Discretization of the solution space is a limitation of this method in its current form. If I understand correctly, the target pose for an object is picked from a set of pre-defined options. In simulation, these options can indeed be generated automatically. But how can these options be generated for real scenes? Does an engineer have to define a set of possible target poses for each object (the appendix suggests this)? Does a model of the environment have to be created and loaded into simulation? This also makes it difficult to apply this method to rearrangement problems without the grid-like structure of a dishwasher, for example lining up mugs on a shelf, since the number of options would need to be large to achieve precise alignment. Have the authors explored training this method to regress poses instead, i.e. make continuous pose predictions?
1. In the current version of the method, the object features focus on the bounding box and could be made more semantically expressive. I.e., can the method tell a fork and knife apart if they have the same bounding box? Does it know that they are both cutlery, and so they share semantic features and so they should be placed similarly? To be fair, this is addressed in the limitations section.
1. There are some limitations with the experimental procedure, and places where it could be described more clearly. “Packing efficiency” is used as a metric, which makes sense for dishwasher loading, but the use of this metric makes it difficult to intuitively understand how good a method is, and how good it would be on other tasks, compared to a metric such as distance error from an acceptable position for an object. Additionally, this packing efficiency is set to zero if the policy violates a preference (Line 187), which I assume includes arranging the drawers in a different order. It might be good to have an experiment which uses a metric solely measuring how good the end state is, since in most scenarios it is unclear why the user would care about anything else, so long as the end state is the same. There are some useful details about the experiment setup that are in the appendix but could go in the main paper to make it self-contained as a description of the experiments. Specifically: full examples/descriptions of what constitutes a preference in these experiments. Additionally, a more self-contained description of how the GNN-IL baseline method works would help.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This is a technically strong paper which addresses a challenging rearrangement problem: predicting how to load a dishwasher well. To improve it further, there are a few limitations to address which would further motivate its contributions and clarify experimental details. If these limitations can be overcome, this work will likely have a major impact on the field of object rearrangement and will be useful for the robot manipulation community at large.
- - -
EDIT: thank you to the authors for their hard work in completing the rebuttal. Taking the paper and discussion into consideration, I have decided to keep my recommendation of Weak Accept. The main reasons for this are as follows:
* The authors have argued more persuasively for the importance of temporal preferences. This is particularly evident in the cooking context. To improve this work further, it would be interesting to see experiments that show this method applied to additional tasks such as this, rather than focusing on the single task of loading a dishwasher, where preferences may more easily be inferred from the end state.
* I agree with the claim in the rebuttal which says that discrete prediction is easier that continuous prediction **if** you have a set of target poses to choose from which are all feasible. It is clear how this makes the task easier for the model. The limitation is that densely sampling these target poses is challenging for real-world scenes. One way forward, as the authors suggest, is a pre-defined dictionary of suitable poses based on the specific model of the dishwasher that the user has in their home. However, this does make the method less easy to apply and limits the tasks for which it can be used. That is why this is a limitation of the method. However, loading items into dishwashers, drying racks, ovens, etc is still useful for a robot, and so the paper is sufficiently impactful. I encourage the authors to briefly mention this hurdle to real-world deployment in the limitations section of the final main paper, since they do agree with this and mention it in the appendix. The rebuttal mentions that this was included in Section 6, but the limitations section still seems to be Section 5 in the updated paper, and does not seem to mention this point, if I am reading it correctly.
* Thank you to the authors for their clarifications on the experimental setup, and the detailed responses in general. Much appreciated!

Overall, the paper is technically well-executed and addresses a useful task in home robotics, so it will make a valuable addition to CoRL 2022.
issues: 1. I think it needs to be made clearer early in the paper why the robot cannot simply copy the prompt actions. It is because the prompt set of objects are different to those in the situation (as explained in the video). For example, in the real world, the user might demonstrate preferences with a couple of plates, and the robot should extend this preference to the whole object set with many more plates. If the only difference between the prompt and the situation were the initial poses of the objects, then simply copying the prompt actions might be fine.
1. The related work discussion can be made more precise to the specific problem addressed by the paper. For example, instead of discussing the general benefits of machine learning approaches on Line 19, this space can be used to discuss the related work in more detail, and explain in more detail where this method extends on it (e.g. here is where we extend beyond [23]).
1. It would be useful for the reader if the paper could discuss the relationship between graph attention networks and transformers, since the former are used in prior work discussed in this paper [23,24]. When applied to this use-case, the two architectures share many similarities (computing attention between all objects in the scene). So, if prior work has already applied a similar attention mechanism, then where does the novelty come from in applying transformers? Perhaps transformers are a better fit than GATs when rearrangement planning is viewed as a sequence modelling problem?
1. This is a minor thing, but some of the formal notation can be clarified to be more consistent. Indexing starts from 0 in some places and from 1 in others when referring to the same variables. E.g. see indexing of x in lines 56 and 61. It does not affect the clarity of the overall paper too much, however.
1. Line 256: possible typo. Should this be “not overfitting” or “not underfitting”? I think not underfitting. Because if the model is overfitting, then it would still benefit from the additional training data. Whereas if it were underfitting, it would not benefit because it is struggling to fit the data it already has, if I understand this argument correctly.
1. The limitations section provides a good overview of future work, but could benefit from also having a frank discussion about inherent limitations due to the design choices in the method, for example the need to have a pre-defined set of poses to choose from.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a transformer-based approach to incorporating human preferences for high-level task planning. The authors focus on simulated and real-world dishwasher loading tasks, in which humans have preferences about where items should be loaded and the order in which loading occurs. Preferences are communicated to the robot through a single demonstration. The proposed approach demonstrates an ability to understand those preferences from that demonstration. The proposed approach improves on an imitation-learned baseline (using a GNN) trained by the authors in situations where the number of objects matches that seen during training and is competitive otherwise.
strengths_and_weaknesses: The paper has weakness stemming from two main areas: the motivation behind the chosen experimental scenario and clarity regarding the technical approach. [I will also admit that the clarity of the paper could also use some improvement; it is possible that I have misunderstood some key aspects of the paper, and have incorporated clarification questions in my discussion below so that the authors can help me better understand if I am incorrect.]

First, I have concerns about the central metric being used for evaluation (or the clarity with which it is presented): the Packing Efficiency. The definition of the Packing Efficiency must be clearer, as it is difficult right now to understand how it is computed yet is key for understanding the performance of the proposed approach. Do a_hat and b_hat incorporate the preferences, or is the equation on lines 185--186 computed first using the items placed in the dishwasher and *then* the result is set to zero if the preferences are not met? Does this mean if *any* preferences are violated, the PE is zero. If so, this would help to explain why the performance of the random agent is so low, since it would require getting quite lucky to achieve a result in which all the objects are placed correctly. If not, it is not clear why the performance of the random agent would be so poor, since if the dishes were randomly placed, should not the performance of the system be roughly 50% accurate? Either way, some clarity on the random baseline and the precise meaning of the PE metric would greatly help the paper.

The experimental setup is not particularly well justified in the text. Critically, the agent's tasks include opening and closing the dishwasher and sliding racks in and out. The authors mention in a few places in the text that planning failures may occur when an agent attempts an action to put an item in the dishwasher yet the dishwasher is closed. There are two issues with this.

First, the rate of failures associated with constraints borne of these manipulation constraints is not reported in the text, making it difficult to understand the impact they have on the performance. The authors should either better justify their decision to incorporate these actions/constraints or provide statistics along with their existing results that identify the rate of failures associated with physical constraints. The purpose of the proposed approach is to evaluate the ability of the agent to identify and incorporate preferences, yet it seems that many of the demonstrations are limited instead because the system is additionally asked to predict physical constraints.

Second, expecting the agent to explicitly enumerate those actions feels like an artificial limitation; the agent is already running a non-learned deterministic planner to plot the trajectories to move the dishes into the dishwasher, so it is unclear why that planner would also not be able to ensure that those constraints are first met. On line 173, the authors point out that these manipulation constraints are why the problem setting is more challenging than a simple prediction task. However, if the motion planner takes care of those constraints simply predicting where objects should be placed in the dishwasher may be able to achieve high performance on the PE metric (though not the Inverse Edit Distance metric). In the text, the authors must better justify the nature of this problem setting or clarify why the proposed methodology is the way to best evaluate this system, since it is important to motivate the need for their approach to be used.

Finally, it is not clear why the order in which objects are loaded into the dishwasher is a particularly salient concern, and so the authors should clarify why dish loading preferences need to include order of operations in addition to placement region. This is the smaller of my high-level concerns, since I am willing to accept that there are household scenarios in which the sequence of operations matters. I am not convinced that loading dishes into the dishwasher is one of those tasks. However, the results showing that it is *possible* using the proposed approach are still valuable, and I think do add to the contributions of the paper. Rather than performing new experiments, which I do not think necessary for this purpose, the authors should discuss a situation in which order of operations is a preference that robots should consider when executing household tasks (even if that scenario does not appear in the paper).

Smaller questions and comments:
- Why is a bounding box used as an input for the category label rather than a one-hot encoding of the object? This seems prone to error and feels like an artificial limitation. While additional experiments are unlikely to occur, the authors should better justify this decision.
- The equation for PE seems to have an error; if the robot's performance is perfect, each of the terms inside the argument should be 1 (totaling to 2) and then summed over the individual demonstrations.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper was somewhat hard to follow in places, as some details (especially the definition of the central metric under scrutiny) were difficult to follow or missing. In addition, the motivation for various aspects of the paper were not very clear; both the problem setting and the experiments were lacking, raising questions about the design decisions behind both.
issues: The authors should focus on clarifying both the technical details of their approach as well as better motivating the problem setting and experiments presented in the paper. In addition, experiments or results that show the impact of expecting the agent to predict actions to open/close the dishwasher and doors will greatly help to understand the efficacy of the proposed approach on learning preferences without the need to also learn manipulation constraints.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper uses a household task "placing kitchen objects in the dishwasher" to test a newly proposed preference-guided task planner model using a transformer. The authors applied plenty of techniques from the natural language processes in the current work to encode the environment input and embed human preferences, which is very interesting and brings good inspiration to the robotics community. The work is both theoretical and technical and the experiment result has been well discussed. 
strengths_and_weaknesses: Pros:
- The motivation of long-term motion sequence planning is well explained. The reason to use a transformer as a planner is clear and convincing.
- The conception, model, and math symbols are structured and explained in a very clear way. It is very straightforward to capture the workflow and ideas from the paper, even for people without much experience in transformer and sequence planning, like me. 
- The paper is well written. I enjoy reading it without finding any explicit typos or mistakes.
- The authors have offered a comprehensive prior work investigation, including sequential manipulation, transformers as sequence modeling, and preferences and prompt training, which benefits the readers to work in a similar field.
- The authors used the task "putting kitchen bowls and plates in a dishwasher" to prove their model's advantages. In my opinion, such a setup is a good choice and is complex enough to prove the advantage of the model. 
- This paper has offered a proper ablation study to prove all the attributes (pose, time, category) are necessary to achieve a good result.
- Interestingly, the authors used some knowledge from the NLP field in the current robotic task to support the good performance of the transformer model, which breaks the barrier between the two fields and offers some inspiration to the researcher in the robotics field who do not have much experience in the NLP.
- Due to the page number limitation, some of the details regarding the experiment settings, type of preferences, the reason for using drawer in reality, camera perception, etc. are missing in the main manuscript but offered in the appendix, which has answered some of my questions. The demo video is also nicely made and brings a lot of useful information.

Cons:
- It will be great if the authors can extend figure 3 and illustrate the content from lines 122 to 134 in it as well. This will help understand the mechanism details. 
- The author presented in the prior work section, that there is a lot of related work in this field. But unfortunately, I do not see a clear reason to use GNN-based preference learning as a baseline.
- In contrast to the proposed model, however, I don't know how the GNN baseline works, especially because the baseline is a combination of two different papers on GNN. I would suggest the authors add some illustrations in the appendix to show the workflow on page 6 of the baseline as well, as to improve the confidence of the readers.


Neutral:
- A nitpicky question regarding the preferences, but I expect an answer from the authors. Sometimes we as users do not know our preferences. Badly, we are also lazy to perform any demonstration to show our preferences. So is it possible to have some pre-programmed preferences mode, e.g., "place bowls first on the higher rack, then put plates on the lower rack"? Because the preference embedding is an intermediate variable in the current model, it is hard to translate it to "mode 1, mode2, ..." and ask the user to choose.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend this paper as l enjoy reading it, although I am neither a researcher in human preference learning nor attention mechanism. This paper focuses on a very practical problem that needs a high-level sequence learning conditioning human preferences. The model is well explained and the experiment is clearly presented and discussed. I can capture most of the ideas of this paper as well as recognize the main contribution of this paper. Although there are a few unclear things that I have pointed out in the weakness section, I believe the authors are capable to improve the paper in the revision session to strengthen my confidence even further. However, if the other reviewers with more knowledge in Attention or Preference learning pointed out some critical issues of the current paper, I will also consider their arguments. 
issues: - I would ask the authors to improve fig.3 to add more technical details in it, see the weakness section.
- I would ask the authors to add some reasons for why choosing GNN as a baseline and an illustration of how the baseline has been used in the current setups.
- Figure 5, caption, typo: "if worse"
- There are a few small errors in the appendix, like missing hyperlinks.
- Figure 5b, why does the GNN-IL perform better in unseen preferences and unseen objects?
- My understanding of human demonstrations is a long sequence of which object to take and where to place it, which is the high-level action choices, and it is also the main focus of the current paper. But, how can we get the low-level control commands, and deal with the issues such as collision avoidance?
- In line 225, the performance on 3-4 objects is poorer and the authors claimed it is because it is unseen. But my question is that if e.g. 5 objects have already been placed in the dishwasher from the setup with 8 objects initialized at the beginning, it reached a similar situation having 3 objects as the initial state. So what are the differences between the two cases?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a transformer-based approach for learning preference-conditioned policies in object-centric manipulation environments (specifically, loading a dishwasher). Preferences are expressed through single demonstrations. Learning is done via behavioral cloning on a given set of preference-labeled demonstrations. Experiments in simulation and on a real robot compare the proposed approach with graph neural network baselines and consider generalization to unseen numbers of objects and unseen preferences.
strengths_and_weaknesses: ### Strengths

* I really like the “dynamically appearing objects” setting in the experiment and appreciate the in-depth analysis in the appendix.
* I really appreciate the detailed experimental analysis, including the results in Figure 6, Table 1, and in the appendix. The experimental design overall is great.
* The writing overall is very clear and easy to understand.
* The related work seems thorough.
* The dishwasher application is nice and clear, and seems like a challenging application.


### Weaknesses

* My biggest concern is the representation of preferences that is given to the GNN baselines. L198 states that the representation is categorical and argues that this is privileged information. For the in-distribution experiments, this is maybe okay, but there is still considerable information loss in using categories versus full demonstrations. But for the out-of-distribution experiments, do I understand correctly that the GNN will simply receive a different categorical value than it saw during training? There would be no hope for generalization if this is the case, and this would be an unfair comparison to TTP.
* It is a little disappointing that generalization to out-of-distribution object numbers is poor, but I appreciate the analysis in Figure 6a on this point.
* The work is a little more incremental than novel, but the all-around quality and thoroughness still makes for a solid contribution.
* There is some redundancy in Section 2 that could be removed. For example, in the State-Action Representation subsection, there is discussion of the policy. But then the policy is discussed again throughout the rest of the section. Section 2.2 is also somewhat redundant with the end of Section 2.1. Overall, more strict adherence to the topics of the subsections would help clarity.
* If I understand correctly, the demos used in the main results are generated as follows: K demos are generated for loading the dishwasher in order A, B, C…; then K more demos are generated for loading the dishwasher in order C, A, B, …; etc. This procedure seems pretty far removed from a real LfD setup. In reality, I would expect that humans loading dishwashers may have some preferences, but are also largely ambivalent about order. Furthermore, any particular human would probably be inconsistent if you asked them to load the dishwasher multiple times. It would be nice to see results for demos collected from real users, or at least demos collected in a noisier and less structured way.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This work is limited in novelty, but otherwise very well done. I think the in-depth experimental analysis and the details of the policy architecture will comprise nice additions to the literature.
issues:    * I think that use of the word “planning” in the title and name of the approach is misleading. Planning usually has the connotation of imagining possible futures with a transition model and reasoning toward a goal. The approach in this work uses a reactive policy and does not use transition models or goals.
   * The word “category” is also potentially misleading, because if I understand correctly, the category is just the bounding box of the object. A better term might be “shape” or “size” or “geometry”. I like the additional analysis and discussion in Figure 8b, though.
   * The definition of the encoded state on L85 suggests that the instances are ordered into a list before passing through the network. Is the model invariant to this order? It should be, right? If so, I would just suggest changing the list to a set in the notation.


### Minor
   * L16: “task constraints through symbolic task description” -> descriptions
   * The x_i start at 0 on L61 but started at 1 on L56
   * I found the term “situation” confusing in Section 2.1. For example, in figure 3, “Situation” is input to the instance encoder on the bottom. But isn’t that just the current state? In other places, it seems like situation is meant to refer to both the state and the preference. Also the “situation decoder” and “policy” are maybe synonymous?
   * L112: “The left half is prompt encoder” → the prompt encoder
L113: “acting on given state” → “the given state”
   * L124: “as well as, placement” → “as well as placement”
   * Figure 4 caption missing a period
   * L172: “unseen number of object” → “numbers”
   * Figure 5 caption: “if worse” → “is worse”
   * It would be good to make the y axis limits identical in Figure 6
   * I didn’t understand what this means: “The policy was successful once with 100% success rate, and once with 75%”
   * There are broken figure and section references in the appendix
review_title: null

Meta-review: metareview: Below is a summary of the strengths and weaknesses of the paper, according to the reviewers.

Strengths:

- The method proposed is well motivated and has several interesting technical contributions.
- The example application of loading a dishwasher helps to show intuition behind the method, and shows the practicality of the method.
- There is a real-world demo.
- The ablation studies are particularly useful.

Weaknesses:

- It is not clear why the order in which items are placed, is an important concept to encode in a user's preferences.
- Discretisation of the solution space limits the practicality of the method in real-world settings.
- There are concerns in the motivation for, and the clarity in the description of, the "packing efficiency" metric.
- There are concerns about the experimental setup regarding failures associated with physical constraints.

In the rebuttal, please address the above weaknesses, as well as the other concerns and questions raised by the reviewers.

---------

Update after the rebuttal:

On average, the reviewers increased their recommendation following the rebuttal, with all four reviewers now recommending acceptance. This is a technically strong paper tackling a challenging problem, with a useful real-world demo addressing a classic dishwasher loading task, which many in the community will be able to relate to. However, reviewers found the motivation for some aspects of the method to be a little unclear, so these should be clarified in the final paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)