Transformers Are Adaptable Task Planners
Authors: Vidhi Jain
Yixin Lin
Eric Undersander
Yonatan Bisk
Akshara Rai
Venue: CoRL 2022 Poster (2022)
Keywords: Task Planning
Prompt
Preferences
Object-centric Representation

Abstract: Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user's preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.

Transformers are adaptable task planners
Vidhi Jain Yixin Lin', Eric Undersander', Yonatan Bisk?, Akshara Rai Meta, Carnegie Mellon University
Abstract: Every home is different, and every person likes things done in their own way. Therefore, home robots of the future need to both reason about the sequential nature day-to-day tasks and generalize to user's preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level reasoning from demonstrations by leveraging object attribute- trained on multiple preferences in a simulated dishwasher loading task and shows generalization to unseen preferences using single demonstration prompt. Further, we demonstrate real-world dish rearrangement using TTP with Franka robotic arm. See videos and code at this website.
representations. TTP pre-
Keywords: Task Planning, Prompt, Preferences, Object-centric Representation
Introduction
Consider robot tasked with loading dishwasher. Such robot has to account for task constraints (e.g. only an open dishwasher rack can be loaded), and dynamic environments (c.g. more dishes may arrive once the robot starts loading the dishwasher). Dishwasher loading such a canonical task with user-specific preferences, like a user may place mugs on the top rack and plates on the bottom or load dirtier bowls before casier-to-clean cups, encoding their preference. Additionally, they pull out rack before loading it, inherently encoding structural constraint. This also holds in other household tasks, like cooking. For example, person may start cooking potatoes before frying onions, potatoes often take longer to cook, while another person first caramelizes the onions before adding in potatoes for flavor. In a kitchen, user might have a preferred cabinet for plates, and another for cups. Learning temporal and spatial constraints and preferences from demonstrations requires policies with temporal context that can consider the sequence of actions demonstrated. Classical task planning [1] deals with such constraints through symbolic task description, but such descriptions are difficult design and modify for new preferences in complex tasks. Building easily adaptable hong-horizon task plans, under constraints and uncertainty, is open problem in robotics.
Transformers are well-suited to this problem, as they have been shown to learn long-range relation- ships [2], although not in temporal robotic tasks. Recent work [3] has shown Transformers [4] can learn temporally-consise
representations, and generalize to new scenarios [5, 6, 7J. Our central question is: Can a Transformer learn task structure, adapt to user preferences, and achieve complex long-horizon tasks using no symbolic task representations? We propose Transformer Task Planner (TTP) an adaptation of classic transformer architecture that includes temporal, pose, and cate- gory embeddings to learn object-oriented relationships over space and time. By pre-training TTP on multiple preferences, TTP learns to infer unseen preferences from single demonstration and adhere to it with a variable number of objects and dynamic environments.
Our main contributions are introduce transformers as promising architecture for learning task plans from demonstrations using object-centric embeddings (Sec. 2), (ii) show that preference- conditioned pre-training generalizes at test time to new, unseen preferences and outperforms com- petitive baselines (Sec. 3.1), and (iii) transfer TTP to rearrangement problem in the real world, where a Franka arm places dishes in two drawers, using a single demonstration (Fig. and Sec. 3.2).
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

4
5
6
Prompt
Top rack
Bottom rack
5
TTP makes decisions about "when to open drawers" and "what objects to pick" for robot arm, such that it follows the preferred order and location, as indicated in a prompt demonstration. Top:
Figure 1:
Prompt summarized as an illustration. Bottom: Robot loads 4 dishes (1-4) opens top drawer and places blue bowl and cup, then (5-8) opens bottom drawer to place plate and pink bowl. Note that the prompt can have different number of objects than in the situation where the robot acts. TTP learns to infer and apply preferences simulation and transfers zero-shot to the real world.
2 Transformer Task Planner (TTP)
We introduce TTP, transformer-based policy architecture for learning sequential manipulation tasks. We assume low-level generalized pick-place primitive actions that apply to both objects like plates and bowls, as well as the dishwasher door and racks. TTP learns a high-level policy for pick- place in accordance with the task structure and preference shown in demonstrations. The following sections are descrihed with dishwasher-loading as an example, but our setup is applicable to most long-horizon manipulation tasks with generalized pick-place.
State-Action Representations We consider a high-level policy that interacts with the environment discrete time steps. At every timestep t, we receive observation o, from the environment which passes through perception pipeline to produce set of rigid-body 'pick" instances {rr)e=> corresponding to the n objects currently visible. The pick state Spick is described as the set of instances that are visible in o, Sppick {.1, ,In]. Once a 'pick' object is chosen, we decide where to place it. We pre-compile a list of discrete placement poses corresponding to viable place locations for each object category, created by randomly placing objects in the dishwasher and measuring the final pose they land in. All possible placement locations for the picked object category, whether free or occupied, are used create action a, in terms of what to pick and where to place it. Specifically, policy æ outputs the pick action, which is object in Ot, i.e. (SSiikk) - Ztarget- The place state is Splace {g))/=1- The same policy æ then chooses where to place the object, π m{Splace) Note that the input for predicting place includes both objects and place instances since the objects determine whether a place instance free to place or not. On the other hand, the 'pick' action only looks the current object instances to determine the next best object to pick. demonstration is (Sr-1,ar-1) (Sr)). See B.
of 'place" instances {gg V = We describe
Spick U Starget (Fig. 2)-
state-action sequence ((Sg, ao); (S1, a1);
Instance Encoder We describe both 'pick' and 'place' instances in terms of their attributes such as (p, c, r, where p is the pose the object, is category, is the timestep while recording o and is boolean indicator, for instance, type- 'pick" or 'place' (Fig. 2a). The pose is embedded as pp. using positional encoding scheme similar to NeRF [8] to encode the 3D positional coordinates and 4D quaternion rotation. To encode the category, use the dimensions of the 3D bounding box

---

Pick latent
Place latent
Encoder
Timestep
Eavi
Transformer
Instance encudings eypu
Category
Set Rigid
Place instances
Pose
Instance Encoder
Txxt
x x xf apick gi 02 of detace
(a) Scene to Instance embeddings
(b) Instances to prediction
of instances. Each instance is comprised of attributes, i.e. pose, category, timestep. and whether it is an object or place instance. (Right) Instance attributes (gray) passed to the encoder, which returns instance embeddings for pickable objects (red), placeable locations (green), and <ACT> embeddings (blue). The transformer outputs a chosen pick (red) and place (green) instance embedding.
Figure 2: (Left) Architecture overview of how scene is converted
of the object to build a continuous space object types and process this through an MLP Φ.. For each discrete 1D timestep. we model Φ as a learnable embedding in lookup table, similar the positional encodings in BERT [9]. The concatenated d-dimensional embedding for an instance at timestep t is represented f25) @e | T C.. See E. for instance encoding design. The encoded state at time / is represented as: Seree = {eo, e1, e.,...e. ...) We drop ()enc for brevity.
2.1 Prompt-Situation Transformer
We use Transformers [4], a deep neural network that operates sequential data, to learn high-level pick- place policy The input to the encoder is a d- dimensional token' per instance e; E [1, in the state S. In addition to instances, we introduce spe- cial <ACT> tokens?, zero vector for all attributes, to demarcate the end of one state and start of the next. These tokens help maintain the temporal struc- ture; all instances between two <ACT> tokens in a sequence represent one observed state. A demon- stration without actions
Prompt Encoder
Optck
Aplace
Situation Decoder
Slot Attention Jan
Transformer Decoder
Transformer Encader fu
fst
Instance Encoder fa < k-..> Instance Encoder fx
<SCT>> [S==o <ACT>
Tyromp!
<ACT>, < Strr]. To learn a common policy for multiple preferences, we propose a prompt- situation architecture, where "prompt" is a demon- stration with some underlying preference, and "sit- uation" a different environment scenario where the policy acts. The policy cannot simply copy the prompt actions because the prompt object set is dif- ferent from the situation. Instead, it needs to infer the underlying preference from the prompt and then apply it in the given situation.
Sp-p S5-2 S
Figure Prompt-Situation Architecture. Left-half: prompt encoder & that takes as in- put a "prompt" (demonstration) and outputs learned preference embedding y. Right- half: the situation decoder ग,, which is conditioned on preference embedding from prompt encoder and is acting on the state from given "sitattion" (environment).
In Fig. the prompt encoder is the left and the situation decoder or policy æ acting on the given state of the situation is the right. The prompt encoder falot fte fe consists of an instance encoder fe, transformer encoder fte, and a slot-attention layer fstot [10]. Slot attention
TTrminology horrowed from natural language processing where tokens are words; here, they are instances. Similar to < CLS tokens used for sentence classification.

---

is an information bottleneck, which learns semantically meaningful representations of the prompt as a fixed and reduced dimensional embedding. v takes the demonstration Tprompt as input and returns ? = Tprumpp) The situation decoder π : fed o Se receives as input N instance tokens from the current scene S, consisting of objects, as well as, placement instances (Fig. 2b). consists transformer decoder [4] with self-attention layers on the N input tokens, followed by a cross- attention layer with preference embedding " from the prompt encoder. We select the output of the situation decoder at the <ACT> token and calculate dot-product similarity with the N input tokens eq- The token with the maximum dot- dot-product is chosen as the predicted instance: Xpred = ie (1,N)) ($<ACT>> e.). is trained with cross-entropy to maximize the similarity of output latent with the expert's 's chosen instance embedding as target. To summarize, the prompt encoder receives a prompt demonstration as input and outputs a learned representation of the preference. These output prompt tokens are input to situation decoder и, which also receives the current state as input. The decoder predicts the action chosen by the expert in the given situation, adhering to the preference indicated in the prompt.
2.2 Multi-Preference Task Learning
We adopt a prompt-situation architecture for multi-preference learning. This design (1) enables multi pulti-preference training by disambiguating preferences, (2) learns task-level rules shared between preferences (e.g. dishwasher should be open before placing objects), (3) can generalize to unseen preferences at test time, without fine-tuning. Given a 'prompt' demonstration of preference m, our policy semantically imitates it in a different 'situation' (i.e. a different initialization of the scene). To this end, we learn a representation of the prompt ym which conditions π to imitate the expert.
(pppopp))) + 7
T (Sstuntionn (r()) - a," (drpeed Spred
(2)
We train neural networks v and π together minimize the total prediction loss on all preferences using the multi-preference training dataset D. The overall objective is:
min 120 M.- D... (S,a)Dm
LcE(a, ((S)()
For every preference m in the dataset, we sample a demonstration from D. and use it without actions as a prompt. We also sample a state-action pair (5, from any demonstration C E Dm- At test time, we record one prompt demo from a seen or unseen preference to output action: a = (ST@((T)o)))) All policy weights are kept fixed during testing, and generalization to new preferences zero-shot using the learned preference representation y. Unlike [11], 7 captures not just the final state, but a temporal representation of the whole demonstration. Building temporal representation is crucial to encode demonstration preferences like the order of loading racks and objects. Even though the final state is the same for two preferences that only differ in which rack is loaded first, our approach is able to distinguish between them using the temporal information in Tpropp-- To the best of our knowledge, our approach is the first to temporally encode preferences inferred from a demonstration in learned task planners.
3 Experiments
We present the "Replica Synthetic Apartment 0 "itchen"3 (see figure 4, appendix and video), an artist-authored interactive recreation of the kitchen of the "Apartment 0" space from the Replica dataset [12]. We use selected objects from the ReplicaCAD [13] dataset, including seven types of dishes, and procedurally generate dishwasher loading demonstrations, See B.
"Replica Synthetic Apartment 0 Kitchen" was created with the consent of and compensation to artists and will be shared under Creative Commons license for non-commercial use with attribution (CC-BY-NC).

---

Figure 4: Dishwasher Loading demonstration in AI Habitat Kitchen Arrange Simulator. Objects dynamically appear on the countertop (ii-iv), and need to be placed in the dishwasher. If the dish- washer racks are full, they land in sink (v).
We define a preference in terms of expert demonstration 'prop- erties', like which rack is loaded first with what objects? Table 1 describes the preferences of dishwasher loading in terms of three properties first loaded tray, objects in the top and bot- tom tray. Preferences 1 & 2 vary in the order of which rack loaded first, while 2 & 3 both load the bottom rack first with similar categories on top and bottom but with different orderings for these categories. Our dataset consists of 12 pref- erences (7 train, 5 held-out test) with 100 sessions per prefer- ence. In session,
Table I: Three example pref- erences for dishwasher loading. Rack order and their respective contents (ordered by preference).
First?
Top
Bottom
1. big plates
1. cups
2. small plates
Top
2. glasses
3. trays
3. small bowl
4. big bowls
{3. .... 10} instances are loaded in each rack. The training data consists of sessions with 6 or 7 objects allowed per rack. The held-out test set contains 5 unseen pref- erences for {3, 4, 5, 8, 9, 10} objects per rack. See B.2. Ad- ditionally, to simulate a dynamic environment, we randomly initialize new objects mid-session on the kitchen counter. This simulates situations where the policy does know every ob- ject to be loaded at the start of the session and has to learn to be reactive to new information. See B.3.
1. big plates
cups
2. small plates
Bottom
2. glasses
3. trays
3. small bowl
4. big bowl
1. big bowls
1. small plate
2. trays
Bottom
2. glasses
3. big plates
3. cups
4. small bowl
To measure success, we rollout the policy from an initial state and compare the resultant trajectory with an expert demonstration. Rollouts require repeated decisions in the environment, without any resets. A mistake made early on in a rollout can result in poor performance, even if the prediction accuracy is high. For example, if a policy mistakenly fails open a dishwasher rack, the perfor- mance will be poor, despite good prediction accuracy in later steps. For in-distribution evaluation, 10 sessions are held-out per preference. For out-of-distributiio evaluation, we create sessions with unseen preferences and unseen numbers of objects. Specifically, we measure:
Spatial Preference Adherence (SPA) SPA measures how well is the final state packed per user's spatial preference. Let à; be the number of objects placed in the ith receptacle adhering the pref- erence by the learned policy, and a; be the number for expert, then SPA \ - mar(e..a.) SPA measures the packing efficiency of the final state of the dishwasher after policy rollout, while respecting the spatial preference of top bottom rack per category. Note that if the policy fol- lows the wrong spatial preference (places objects in the wrong rack), then SPA is low, if the dishwasher is full.
Temporal Preference Adherence (TPA): TPA measures how much the policy deviates from the order actions shown in an expert demonstration. Let ř, 7 be the sequence of picked instances by the learned policy and expert respectively, then TPA = Edit Distanorss))))))) TPA accounts for the maz([F]]FF)
order in which objects are picked, irrespective of where they are placed, and measures the deviation from the temporal actions of the user. TPA is high if the policy follows both sequential constraints of task (open dishwasher before loading), and the order-related preferences (bowls before cups).
If the expert preference was to load the top rack first and the learned policy loads the bottom first, SPA would be perfect at 1.0, but TPA would be low. Both metrics are low if the policy repeatedly violates the task's 's physical constraints or does not make progress towards the task. See C.4.
5

---

Tompare Adhorcncc
0.6
Spatial Adhorence
Rai Ramdoom
Rendom
0.0
0.0
0.4
0.4
0.0
0.0
Seen prof
Seen pref
Unseen pref
Unseen pref
Seen pref
Seen pref
Uhssen pref
Unosen praf
(a) Spatial Preference Adherence (SPA)
Temporal Preference Adherence (TPA)
Figure 5: Comparisons of TTP, GNN-IL/RL and Random policy in simulation across two metries: SPA and TPA. TTP shows good performance at the task of dishwasher loading on seen and unseen preferences, and outperforms the GNN and random baselines. However TTP's generalization to unseen objects worse, but still close to GNN-IL (which was trained on all preferences), and better than GNN-RL RRN and random.
0.6
0.5
# 0.4
0.4
0.3
0.2
0.2
0.0
0.0
0.0
10
5
10
100
(a) Performance decays for OOD (b) Performance improves with the (c) Performance improves with the
number objects, i. 3-5 & 8-10
unique sessions in training
of unique preferences training
Figure 6: TTP experiments (a) Out-of-distribution generalization to #objects. (b-c) ablations.
3.1 Simulated Dishwasher Loading
We compare our approach against Graph Neural Network (GNN) based preference learning from [14, 11] at this task of dishwasher loading. Neither of these works is directly suitable for our task, so we combine [14, 11] to create stronger baseline. We use ground-truth preference labels, repre- sented categorical distribution, and add them the input features of the GNN, similar to [11]. We use the output of the GNN make sequential predictions, as in [14]. Thus, by combining the two works, we create GNN baseline that can act according to preference in dishwasher-loading scenario. The GNN policy is trained on all preferences, including the 'unseen' preferences. For unseen preferences, this privileged information that our approach does not have access to. We train the GNN policy using imitation learning (II.), and reinforcement learning (RI.), following [14]. GNN-IL is trained from the same set of demonstrations as TTP using behavior cloning (see Fig. 12a). For GNN-RL,
Proximal Policy Optimization [15] from [16]. GNN-RL lears from scratch by directly interacting with the dishwasher environment and obtaining dense reward. For more details on baselines, including an altemnte variant of GNN-IL, see Appendix C.
GNN-IL does not reach the same performance as TTP for in-distribution tasks (SPA of 0.34 for GNN-IL vs 0.62 for TTP). Note that unseen preferences are also in-distribution for GNN-IL since provide ground-truth preference labels to the GNN. Hence, there isn't drop in performance for unseen preferences for GNN, unlike TTP. For detailed failure analysis, Appendix D. Despite having no privileged information, TTP outperforms GNN-IL. in unseen preferences and performs comparably on unseen #objects. Due to the significantly long time horizons per session (more than 30 steps), GNN-RL fails to learn a meaningful policy even after large budget 32,000 environ- ment interactions and a dense reward (SPA 0.017 0.002 using GNN-RL). Lastly, we find that the random policy RP is not able to solve the task at all due the large state and action space. TTP is able solve dishwasher loading using unseen preference prompts well (SPA 0.54). In con- trast, classical task planners like [1] need to be adapted per preference. This experiment shows that Transformers make adaptable task planners, using our proposed prompt-situation architecture.

---

However, TTP's performance on unseen #objects deteriorates (SPA of 0.62 for seen versus 0.34 on unseen #objects), and we look more closely at that next.
Generalization unseen objects: Fig.6a examines SPA on distriiiitiiittonn sessions with lesser i.e. 3-5 more i.e. 8-10 objects per rack. The training consists of demonstrations with 6 or 7 objects per rack. The policy performs well on 5-7 objects, but poorer as go further away from the training distribution. Poorer SPA is expected for larger number of objects, as the action space of the policy increases, and the policy is more likely to pick the wrong object type for given preference. Poorer performance on 3-4 objects is caused by the policy closing the dishwasher early, as it has never seen this state during training. Training with richer datasets, and adding more randomization in the form of object masking might improve -ff-----isttitttionn performance of TTP.
3.1.1 Ablations
We study the sensitivity of our approach to training hyperparame- ters. First, we vary the number of training sessions per preference and study generalization unseen scenarios of the same prefer- ence. Figure 6b shows that the performance of TTP improves the number of demonstrations increases, indicating that our model might benefit from further training samples. Next, we vary the num- ber of training preferences and evaluate generalization performance to unseen preferences. Figure shows that the benefits of adding additional preferences beyond are minor, and similar performance observed when training from 4-7 preferences. This is an interest- ing result since one would assume that more preferences improve generalization to unseen preferences. But for the kinds of preferences considered in our problem, 4-7 preference types are enough for generalization.
Table 2: Attribute Ablations
Pose
Cat
Time
SPA
X
0.0
X
X
V
0.0
X
0.027
X
0.142
X
0,411
X
V
0.419
0.517
0.606
Finally, we analyze which instance attributes are the most important for learning in an object-centric sequential decision-making task. We mask different instance attributes remove sources of infor- mation from the instance tokens. In Table all components of our instance tokens play significant role (0.606 with all, versus the next highest of 0.517). The most important attribute is the pose of the objects (without the pose, top SPA is 0.142), followed by the category. The timestep is the least important, but the best SPA comes from combining all three. More ablations are in Appendix E.
3.2 Real-world resr-rearrangement
As a proof-of-concept, we demonstrate zero-shot transfer of our policy trained simulation to robotic hardware, by assuming low-level controllers. We use Franka Panda with a Robotiq 2F-85 gripper, controlled using the Polymetis control framework [17]. For perception, we use three Intel Realsense D435 RGBD cameras [18] whose outputs combined using Open3D [19] and seg- mented [20] to generate 3D point clouds of visible objects, used to calculate object-centric attributes used by TTP. For low-level pick, we use grasp candidate generator [21] applied to chosen ob- ject's point cloud and use it to grasp the target object. 'Place' is approximated as 'drop' action in a pre-defined location. More details are in Appendix A.
Our hardware setup mirrors our simulation, with different categories of dishware (bowls, cups, plates) table, and a "dishwasher" (cabinet with two drawers). The objective is to select an object to pick and place it into a drawer (rack) (see Fig. 1). We use policy trained in simulation and apply it to a scene with four objects (2 bowls, 1 cup. 1 plate) using the hardware pipeline de- scribed above. We start by collecting a prompt human demonstration which is manually segmented into pick and place motions, and the environment state recorded at each timestep. The learned pol- icy, conditioned on the prompt, is applied to two variations of the scene, and the predicted actions executed. The hardware success defined in terms of pick only, shown in Fig. 1, since we do not reason about 'place' on hardware. The policy successful places all 4 objects in the correct order in the drawer once and 3 out of 4 objects in the second attempt. The hardware failure case was
7

---

caused by perception error; a bowl was classified as cup. This demonstrates that TTP can be trained in simulation and applied directly to hardware, and the policy is robust to minor hardware errors, like noise in object location. It is also reactive to failure, such as if a bowl grasp fails, it just repeats the grasping action. However, it is sensitive to perception errors like failing to detect objects, or mis-classifying them. See Appendix F. In the future, TTP should be evaluated in more diverse real-world settings and its sensitivity to the different hardware components measure more carefully.
4 Prior Work
Learning for Sequential manipulation aims to overcome the challenges in designing and scaling classical Task and Motion Planning approaches. Object-centric pick-place representations using off- the-shelf perception methods are common in learning policies [22, 23, 24, 25, 26, 27]. specifically for sequential manipulation [1, 283. TransporterNets [29] primarily deal with table-top Pick&Place generalized to new object locations and do not consider in 3D sequential reasoning problems. SOR- Net [30] assumes access to a task planner to reason about sequential reasoning instead of learned policy. Such task planner is often tedious to design for complex tasks, and difficult to modify per preference. While [14] presents a learned sequential reasoning approach using GNNs, and [11] shows generalization of GNNs to preferences, neither is directly suitable for our task, we com- bine [14, 11] to create stronger baseline. Recent works have repurposed Transformers for other sequence modeling tasks [31, 32, 33, 34, 35, 36]. Prompt DT [37] considers single model to en- code the prompt and the successive sequence of state. This limits the size of the prompt and requires special tokens to demarcate between prompt and current state. We use the encoder-decoder setup and share the parameters for instance encoding. PlaTe [32] proposes planning from videos, while [34, 35, 36] model sequential decision-making task. We consider long-horizon tasks with partially observable state features and user-specific preferences.
Preferences and prompt training There are several ways of encoding preferences in recent works. [11] propose VAE to learn user preferences for a spatial arrangement based on just the final state, while our approach models temporal preference from demonstrations. Preference-hasee RI. learns rewards based on human preferences [38, 39, 40, 41], but do not generalize to unseen preferences. For complex long-horizon tasks, [42] shows that modeling human preferences enables faster learn- ing than RL. We show generalization to unseen preference by learning to infer them from the demonstration. Large language and vision models have shown generalization through prompting [43, 44). Prompting has been used to guide a model to quickly switch between multiple task objec- tives [45, 46, 47]. Recent language models learn representations that can be easily transferred to new tasks in a few-shot setting [48, 49, 5, 43]. Our approach similarly utilizes prompts for preference generalization in sequential decision-maker robotic tasks.
Conclusions and Limitations
We present Transformer Task Planner (TTP): a high-level, sequential, preference-basee policy from single demonstration using prompt-situation architecture. We introduced simulated dishwasher loading task with demonstrations that exhibit different preferences. TTP can solve this complex, long-horizon task in simulation and transfers to the real world. This is a first step towards learning to adapt to different users' preferences in terms of both ordering and spatial location.
While our task is complex by virtue of its strict sequential nature, it is incomplete as we assume doors and drawers can be easily opened and perception perfect. In complex real settings, the policy needs to learn how to recover from its low-level execution mistakes. Moreover, preferences can have further nuanced differences based on visual or textural patterns objects, for which the instance encoder needs to be modified to encode such attributes. Another important question to address is how more complex motion plans interact with or hinder the learning objective, due to different human and robot affordances. Finally, prompts are only presented via single demonstration, while language might be a more natural interface for users to instruct the robot.
8

---

Acknowledgments
We thank Tingfan Wu and Austin Wang for their support in the hardware setup, as well Amy Zhang, Chris Paxton, Ben Newman, Adriana Romero Soriano, Vikash Kumar, and Dhruv Batra for the helpful initial discussions. We also thank all anonymous reviewers for CoRL'22 and Rishi Veerapaneni for high-quality feedback and suggestions.
References
[1] C. R. Garrett, R. Chitnis, Holladay, B. Kim, T. Silver, L. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. arXiv preprint arXiv:2010.01083, 2020.
[2] D. S. Chaplot, D. Pathak, and J. Malik. Differentiable spatial planning using transformers. In International Conference on Machine Learning, pages 1484-1495. PMLR, 2021.
[3] R. Kaplan, C. Sauer, and Sosa. Beating atari with natural language guided reinforcement learning. arXiv preprint 3rXiv:1704.05533 2017.
[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.
N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances in neurul information processing systems, 33: 33:1877-1901, 2020.
[6] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:21 2021.
[7] X. Liu, K. Ji, Fu, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 2021.
[8] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European conference on computer vision, pages 405-421. Springer, 2020.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[10] Locatello, D. Weissenborn, Unterthiner, A. Mahendran, G. Heigold, Uszkoreit,
A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neu- ral Information Processing Systems, 33:11525-11538, 2020.
[11] I. Kapelyukh and E. Johns. My house, my rules: Learning tidying preferences with graph neural networks. In Conference on Robot Learning (CoRL), 2021.
[12] Straub, Whelan, L. Ma, Y. Chen, Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, Yon, Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv: arXiiv1906.000 057977 2019.
[13] A. Szot, A. Clegg, E. Undersander, Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, Gokaslan, Vondrus, S. Dharur, Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.

---

[14] Y. Lin, A. S. Wang, E. Undersander, and A. Rai. Efficient and interpretable robot manipulation with graph neural networks. IEEE Robotics and Automation Letters, 2022.
[15] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: orXiv:/707.06444 2017.
[16] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
[17] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and R. Meier. facebookresearch, github. http//arrropllymmts...//s 2021.
Polymetis.
https://
[18] L. Keselman, Iselin Woodfill, A. Grumnet-Jepsen, and Bhowmik. Intel realsense stereo- scopic depth cameras. In Proceedings the IEEE conference on computer vision and pattern recognition workshops, pages 1-10, 2017.
[19] Q.-Y. Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint 1801:1800.09000 2018.
[20] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. arXiv preprint arXiv:2007.15157, 2020.
[21] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-Ibillion: A large-scale benchmark for general object grasping. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 11444 11453, 2020.
[22] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2642-26422222 2452- 2019.
[23] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Sule-supe 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference Roboties and Automation (ICRA), pages 3665-3671. IEEE, 2020.
[24] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and Xiao. Multi-view self- supervised deep learning for 6d pose estimation in the amazon picking challenge. 2017 IEEE international conference on robotics and automation (ICRA), pages 1386-1383. IEEE, 2017.
[25] M. Zhu, K. G. Derpanis, Y. Yang. S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. In 20/4 IEEE International Conference on Robotics and Automation (ICRA), pages 3936-3943, 2014.
[26] Y. Yoon, G. N. DeSouza, and A. C. Kak. Real-time tracking and pose estimation for industrial objects using geometric features. In 2003 IEFE International conference on robotics and automation (cat. no. 03CH37422), volume 3, pages 3473-3478. IEEE, 2003.
[27] Jain, Agarwal, S. Patil, and K. Sycara. Learning embeddings that capture spatial semantics for indoor navigation. arXiv preprini arXiv:2 2021.
[28) C. Paxton, N. Ratliff, C. Eppner, and D. Fox. Representing robot task plans as robust logical- dynamical systems. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5588-5595. IBBE, 2019.
[29] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020.
10

---

[30] W. Yuan, C. Paxton, K. Desingh, and D. Fox. SORNet: Spatial object-centric representations
for sequential manipulation. 5th Annual Conference on Robot Learning, 2021. URL https: // //openreview.. www.:/ee/ffruu/i dd
[31] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for
language-guus gorliggeggudeddddde semantic rearrangement of novel objects. ArXiv, abs/2110.10188) 2021.
[32] Sun, D.-A. Huang, B. Lu, Y.H. Liu, B. Zhou, and A. Garg. Plate: -gouuddd planning with transformers procedural tasks. IEEE Robotics and Automation Letters, 7(2): 4924 4930, 2022.
[33] V. Jain, R. Jena, H. Li, T. Gupta, D. Hughes, M. Lewis, and K. Sycara. Predicting human strategies in simulated search and rescue task. arXiv preprint wrXiv:2011.076:5 2020.
[34] L. Chen, K. Lu, A. Rajeswaran, Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34, 2021.
[35] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[36] A. L. Putterman, K. Lu, I. Mordatch, and Abbeel, Pretraining for language conditioned im- itation with transformers, 2022. URL https: / htt////ppprreeeeeeeeeeeee httpe///oorumnddddPddeeee-ee
[37] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, B. J. Tenenbaum, and C. Gan. Prompting decision transformer for few-shot policy generalization. In Thirty-ninth International Conference on Machine Learning, 2022.
[38] X.
Wang, K. Lee, K. Hakhamaneshi, P. Abbeel, and M. Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning, pages 1259-12221 1268. PMLR, 2022.
[39] K. Lee, L. Smith, A. Dragan, and P. Abbeel. B-pref: Benchmarking preference- based rein- forcement learning, arXiv preprint arXiv:2111.03026. 2021.
[40] X. Liang, K. Shu, K. Lee, and P. Abbeel. Reward uncertainty for exploration in preference- based reinforcement learning. arXiv preprint 22ti::2200.120411 2022.
[41] W. Knox, S. Hatgis-Kessell, Booth, S. Niekum, P. Stone, and A. Allievi. Models of human preference for learning reward functions. arXiv preprint arXiv:2206.022311 2022.
[42] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[43] Y. Chen, Y. Liu, I.. Dong, S. Wang, C. Zhu, M. Zeng, and Y. Zhang. Adaprompt: Adaptive model training for prompt-based nlp, arXiv preprint arXiv:2202.0485 04824, 2022.
[44] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[45] C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910.10683, 2020.
[46] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to se-euueeet--seuuncee pre-training for natural language gen- eration, translation, and comprehension. In ACL, 2020.

---

[47] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICMI,, 2019.
[48] T. Schick and H. Schütze. It's just size that matters: Small language models are also few-shot learners, arXiv preprint 2xXv:2000.00718,, 2020.
[49] Liu, W. Yuan, J. Fu, Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107133886 2021.
[50] M. Fiala. Artag, a fiducial marker system using digital techniques. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume 2, pages 590-596, IEEE, 2005.
[51] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[52] M. Du. O. Y. Lee, S. Nair, and C. Finn. Play it by car: Learning skills amidst occlusion through audio-visual imitation learning. 2022. doi: 1044850/AXXI..220.145558 URL https: // wexiv.org//bar.
14850.
[53] Anderson, A. Chang, D. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:180..06757. 2018.
[54] Carpentier, Valenza, N. Mansard, al. Pinocchio: fast forward and inverse dynamics for poly-articulated systems, https://stack-of http://stackkoo-takksssssop ofw.oofftttttttiiiisee fasks.iitubb.oo/pnoochii. 2015-2021.
12

---

A Hardware Experiments
A.1 Real-world prompt demonstration
Here we describe how we collected and processed a visual, human demonstration in the real-world treat a prompt for the trained TTP policy (Fig. 7). Essentially, we collect demonstration pointeloud sequences and manually segment them into different pick-place segments, followed by extracting object states. At each high-level step, we measure the state using three RealSense RGBD cameras[18], which are calibrated to the robot frame of reference using ARTags [50]. The camera output, extrinsics, and intrinsics are combined using Open3l [19] to generate a combined point- cloud. This pointcloud segmented and clustered to give objects' pose and category using the algorithm from [20] and DBScan. For each object point cloud cluster, we identify the object pose based on the mean of the point cloud. For category information we use median RGB value of the pointcloud, and map it to apriori known set of objects. In the future this be replaced by more advanced techniques like MaskRCNN [51]. Placement poses are approximated as fixed, known location, as the place action on hardware is a fixed 'drop' position and orientation. The per step state of the objects is used to create the input prompt tokens used to condition the policy rollout in the real-world, as described in Section 3.2.
Figure 7: Human demonstration of real-world rearrangement of household dishes.
A.2 Hardware policy rollout
We zero-shot transfer our policy æ trained in simulation to robotic hardware, by assuming low- level controllers. We use Franka Panda equipped with a Robotiq 2F-85 gripper, controlled using the Polymetis control framework [17]. Our hardware setup mirrors our simulation, with different categories of dishware (bowls, cups, plates) The objective is to select object to pick and place it into drawer (rack) (see Fig. 7).
table, a "dishwasher" (cabinet with two drawers).
Once we collect the human prompt demonstration tokens, we can use them to condition the learned policy π from simulation. Converting the hardware state to tokens input to π follows the same pipeline as the ones used for collecting human demonstrations. At each step, the scene is captured using 3 Realsense cameras, and the combined pointcound segmented and clustered get object poses and categories. This information along with the timestep is used generate instance tokens as described in Section 2 for all objects visible to the cameras. For visible already placed objects, the place pose is approximated as a fixed location. The policy ग, conditioned on the human demo, reasons about the state of the environment, and chooses which object to pick. Next, we use a grasp generator from [21] that operates on point clouds to generate candidate grasp locations on the chosen object. We filter out grasp locations that are kinematically not reachable by the robot, as well as grasp locations located on points that intersect with other objects the scene. Next, we select the top 5 most confident grasps, estimated by the grasp generator, and choose the most tow-down grasp. We design an pre-grasp approach pose for the robot which is the same final orientation as
13

---

Motion Planning
Grasping
Prompt- Situation
Instance Segmentation
Policy
Figure 8: Pipeline for Real Hardware Experiments
the grasp. located higher on the grasping plane. The robot moves to the approach pose following a minimum-jerk trajectory, and then follows a straight line path along the approach axes to grasp the object. Once grasped. the object is moved the pre-defined place pose and dropped in drawer. The primitives for opening and closing the drawers are manually designed on hardware.
The learned policy, conditioned on prompt demonstrations, is applied to two variations of the same scene, and the predicted pick actions are executed. Fig.9 shows the captured image from one of the three cameras, the merged point cloud and the chosen object to pick and selected grasp for the same. The policy was successful once with 100% success rate, and once with 75%, shown in Fig.1. The failure case was caused due perception error bowl was classified as a plate. This demonstrates that our approach (TTP) can be trained in simulation and applied directly hardware. The policy is robust to minor bardware errors like failed grasp; it just measures the new state of the environment and chooses the next object to grasp. For example, if the robot fails to grasp bowl, and slightly shifts the bowl, the cameras measure the new pose of the bowl, which is sent to the policy. However, TTP relies on accurate perception of the state. If an object is incorrectly classified, the policy might choose to pick the wrong object, deviating from the demonstration preference. In the future, we would like further evaluate our approach on more diverse real-world settings and measure its sensitivity the different hardware components, informing future choices for learning robust policies.
A.3 Transforming hardware simulation data distribution
The policy trained in simulation applies zero-shot to real-world scenarios, but requires a coordinate transform. Fig. 10 shows the coordinate frame of reference in simulation and real world setting. Since our instance embedding uses the poses of ob- jects, it is dependant ing data collected in. Since hardware and simulation significantly different, this coordinate frame is not the same between sim and real. We build transformation that converts hardware measured poses to the simulation frame reference, which is then used create the instance tokens. This ensures that there is no tim-to-real gap in object positions, reducing the challenges involved in applying such simulation trained policy to hardware. In this section describe how convert the real world coordinates to simulation frame coordinates for running the trained TTP policy on a Franka arm.
the coordinate frame that the train-
Figure Coordinate Frame of reference in simulation (left) and real world setting (right). Red is x-axis, green is y-axis and blue is z-axis.
We use the semantic work area in simulation and hardware to transform the hardware position coor- dinates simulation position coordinates. We measure the extremes of the real workspace by man- ually moving the robot to record positions and orientations that define the extents of the workspace for the table. The extents of the drawers measured by placing ARTag markers. We build
14

---

RGB
Point Cloud
Grasps
images
Figure 9: Point cloud and grasps for different objects during policy rollout.
real-to-sim transformations using the extents for counter, top rack and bottom rack: Let X e B contain homogeneous 2 coordinates of a work area, along its column, as follows:
(1)
(2)
X
{(1)
2(2)
(1)
(1)
x(2)
...]
(4)
As the required transformation from real to simulation involves scaling and translation only,
have 4 unknowns, namely, [azy Glys Ptranss Ztrans]. Here ag 0₂ scaling factors and Etrans: Strans translation offset for and axis respectively. To solve Xsim AX
Itrans
need find the transformation matrix A -= à
0
02 Strans
15

---

Figure 11: Human demonstration with point and click in simulation
Xaa a.X kw
Rewriting the system of linear equations,
(6)
ssm
2 Jau
0
(1)
hw
w
sim
2 Jew
(2)
0
a
(2)
0
simm
(8) Let the above equation be expressed as Ysim a where Ysim E R22x1, Znw E 2N and aT E R4x11 Assuming we have sufficient number pairs corresponding points simulation and real world, we can solve for a by least squares a = (Ki Znw) -1 Yoim The height Ueim is chosen from a look-up table based on ylow- Once we compute the transformation A, we store it for later to process arbitrary coordinates from real sim, as shown below.
def get_ iost_simulationcom how.riiiitoocooooittetteeeete (xyz_nv List [float], A: mp. array) -> List:
xz_hw [xyz_hw[ie xyz_hw [2]]
X_hw get_homogge grtt_hmoggnous_coorreee
X_sim_home homo np matmul (A, X_hw)
y_sim process_ height (xyz_hw[1))
homo [2]] X_sim [x_ iX_sim_homo [0] / $__sim_tee homo [2], y_sim, X_sim homo [1] / K_sim, simm
return X_sim
The objects used in simulation training are different from hardware objects, even though they belong the same categories. For example, while both sim and real have small plate, the sizes of these plates different. We can estimate the size of the objects based actual bounding box from the segmentation pipeline. However, it is significantly object mismatch. So, we map each detected object to the nearest matching object in simulation and use the simulation size as the input to the policy. This is non-ideal, as the placing might differ for sim versus real objects. In the future, we would like to train with rich variations object bounding box size in simulation so that the policy can generalize to unseen object shapes in the real world.
from the training data, due to
B Simulation Setup
B.1
Dataset
"Replica Synthetic Apartment 0 Kitchen" consists of a fully-interactive dishwasher with door and two sliding racks, an adjacent counter with sink, and a "stage" with walls, floors, and ceiling. We selected objects from the ReplicaCAD [13] dataset seven types of dishes (cups, glasses, trays, small bowls, big bowls, small plates, big plates). Fig. 11 shows human demonstration recorded in simulation by pointing and clicking on the desired object to pick and place.
16

---

We initialize every scene with an empty dishwasher and random objects placed on the counter. Next, we generate dishwasher loading demonstrations, adhering to a given preference, using an expert-designed data generation script. Expert actions include opening/closing dishwashers/racks and picking/placing objects feasible locations or the sink there are no feasible locations left. Ex- perts differ their preferences and might choose different object arrangements the dishwasher. A demonstration is sttteacction sequence S is the set of object and place instances and are the pick-place actions chosen by expert at time i. At every time step, we record the state of the objects, the pick instance chosen by the expert, place instances for the corresponding category, and the place instance chosen by the expert. Expert actions are assumed to belong to the set of visible instances in S. However, different experts can exhibit different preferences over a;- For example, one expert might choose to load bowls first in the top rack, while another expert might load bowls last in the bottom rack. In the training dataset, assume labels for which preference each demonstration belongs, based on the expert used for collecting the demonstration. Given K demonstrations per preference m M,, have dataset for preference m: Dn.. = (TI.... TK)- The complete training dataset consists of demonstrations from all preferences: D U Dm- During training, we learn a policy that can reproduce all demonstrations in our dataset. This is challenging, since the actions taken by different experts different for the same input, and the policy needs to disambiguate the preference. At test time, we generalize the policy to both unseen scenes and unseen preferences.
{(So; 0o), (S1, a1),... (Sr-1. ar-1), (Sr)). Here
B.2 Expert Preferences
In Section 3, we define preference, that set of "properties' of the demonstration trajectory, like which rack is loaded first with what objects? Individual task preferences differ in the sequence of expert actions, but collectively, preferences share the underlying task semantics. For example, the user always opens the dishwasher rack before loading it for all preferences. By jointly learning overall preferences, our policy can benefit from cross-preference data to learn task structure, and sparse per per-preference demonstration data to learn to follow the desired preference in the current situation.
There combinatorially many preferences possible, depending how many objects training set. Given 7 categories of dishes and two choices in which rack to load first, the hypothesis space of possible preferences is 2 x 7!. Our dataset consists of 12 preferences (7 train, 5 held-out tests) with 100 sessions per preference. In a session, E {3, ...y 10} instances are loaded in each rack. The training data consists of sessions with 6 or 7 objects allowed per rack. The held-out test set contains 5 unseen preferences and settings for (3, 4, 5, 8, 9, 10) objects per rack. Thus, there are 2 x 7" unique possible session demonstrations, i.c. different prompts and situations.
in the
B.3 Dynamically appearing objects
To add additional complexity to our simulation environment, we simulate setting with dynamically appearing objects later the episode. During each session, the scene is initialized with p% of max- imum objects allowed. The policy/expert starts filling a dishwasher using these initialized objects. After all the initial objects are loaded and both racks are closed, new objects are initialized one per timestep the policy. The goal simulate an environment where the policy does not have perfect knowledge of the scene and needs to reactively reason about new information. The policy reasons on both object configurations in the racks, and the new object type to decide whether to 'open rack and place the utensil' or 'drop the object in the sink'.
C Simulation Experiment Details: Baselines, Training and Metrics
GATs and Transformers are closely related network architectures. In theory, a GAT architecture over a fully-connected graph with an additional time-based feature would be equivalent our Transformer-basee model. We are certainly not the first to apply attention based architectures to planning problems, but unlike [14, 11], consider sequential planning over multiple timesteps,
17

---

which is critical to understanding human preferences [52]. Another difference from GNN for se- quential planning is that we use cross-attention to condition the policy on the prompt, instead of adding preferences as an input feature. We use a slot attention architecture to learn a preference representation, and re-use instance encoders across prompt and situation. While these architectural choices are possible in GNNs, they are more intuitive and faster implement Transformers.
C.1 Baselines
Random We sample the action, that is, pick or place instance from the set visible instances in the current state/observation with equal probability. Similarly, the place pose sampled randomly from all possible poses for the chosen category. The performance of random policy reduces expo- nentially with the length of the action sequence (typically around 20-30 steps long), and number of visible objects in the scene (between 6 to 20). SPA is low if the policy either (I) fails to adhere the physical constraints, or (2) violates the spatial preference. Even if we ignore the physical constraint violations and consider the task of selecting the preferred rack for each object, the probability of a SPA of 1.0 for a random policy would be around (1/2)Noojeell since it has to choose either top or bottom rack per object.
GNN-IL We use GNN with attention-based message passing. The input consists of 12 dimen- sional attributes (ID-timestep, 3D-category bounding box extents, 7D-pose, ID-is object bool) and 12 dimensional one-hot encoding for the preference This forms the imput node features, with each node representing a visible object in the environment. All the nodes are fully connected with equal edge weight of 1. The preference vector u is concatenated with each node x,;= of the state from the situation, resulting in 24 dimensional vector representation per node, which is in turn fed into the GNN decoder as used in [14]. Refer figure 12a for the overall architecture. The policy is trained using supervised learning to minimize the cross-entropy loss over the expert demonstration (choose the same object and place pose as selected by the expert of given preference).
We train the GNN decoder with Proximal Policy Optimization (PPO) [15]. Each node represented similarly 12 instance attributes and ground-truth one-hot encoded preference (12- dimensional), like in GNN-IL. Reward function for the RL policy is defined terms of preference. The policy gets a reward of every time it predicts the instance pick that has the category according to the preference order and whether is placed on the preferred rack.
GNN-RL
GNN-VAE-IL Note that GNN-IL gets privileged information about the preference encoding and from the prompt demonstrations. So, we train preference encoding based policy using VAE loss per [11] and call this variant GNN-VAES VAE-IL. The prompt is passed through the GNN encoder, global add pool and the feed-forward layers to predict mean μr and log variance of preference latent as per [1]]. This is used to sample from preference embedding space to concatenate with the visible object instances in the situation. The concatenated preference to instance features is passed into the state-action policy modelled as GNN decoder in [14] to predict pick and place instances. GNN-VAE-IL trained minimize the VAE reconstruction loss over the actions taken by an expert of given preference in a situation.
C.2 Architectural details
TTP We use 2-layer 2-head Transformer network for encoder and decoder. The input dimension
of instance embedding 256 and the hidden layer dimension is 512. 100 slots and iterations for Slot Attention. Instance attributes encoded follows: Category 3 64, Pose 7 the
128, Timestep + 32, Is object marker? + 32. For preference encoding, slot attention layer at the head of Transformer encoder with 100 slots and 3 slot iterations.
GNN Decoder We three layer GATCony, each followed by MLP for the GNN decoder with 128-dimensiona hidden layer as per [14]. At the output, each node feature is reduced to a logit. We
18

---

Qpick
aplace
NeatNet Encoder!?
apick
aplace
GNN Decoder(3)
GNN Decoder(8)
User Preference Encoder f.
GNN Decoder fed
GNN Decoder fid
GNN encoder fe
One-hot 24/ situation
2 S situation
preference
T prompp
{ul 1 fullgs}}
0 . St-
{ {
(b) GNN-VAE-IL
(a) GNN-IL
Figure 12: Model workflow for GNN-IL and GNN-VAE-IL inspired from [14, 11]. The one-hot encoding per preference is passed as input to GNN-IL or GNN-GT-IL (there is no unseen preference for GNN-GT-IL). GNN-VAE-IL infers the preference from prompt through reconstruction and KL loss on preference embedding space.
select the pick and place instance using the input 'is object marker', and consider the softmax from each respective set.
GNN Encoder We use the GNN encoder in [11] with 128 hidden dim GATCony layer followed by elu activation and 256 dimensional MIP. This is passed preference encoder, which is 2-layer MLP with leaky ReLU and that outputs 100-dimensional mean and log variance corresponding to the assumed Gaussian distribution of the inferred preference encoding. During training, the pref- erence encoding is sampled from this distribution. While evaluation, we use the mean as the best representation of preference encoding.
C.3 Training
We use batch-size of 64 sequences. Within each batch, we use pad the inputs with 0 upto the max sequence length. Our optimizer of choice is SGD with momentum 0.9, weight decay 0.0001 and dampening 0.1. The initial learning rate is 0.01, with exponential decay of 0.9995 per 10 gradient updates. We used early stopping with patience 100. Fig. 13 shows the accuracy achieved for picked category on uniformly sampled input states from the training data, unlike rollout where the next state depends on the previous action taken. We show runs grouped by batch sizes 64, 128 and 256.
C.4 Metries
In Section 3, presented Spatial Preference Adherence (SPA) and Temporal Preference Adher- ence (TPA) metries collected policy rollout. Here discuss additional metrics about training progress and rollouts for GNN-VAE-II.
Category-token Accuracy indicates how well the policy can mimic the expert's action, given the current state. We monitor training progress by matching the predicted instance to the tar- get chosen in the demonstration (Fig. 13). We see that TTP able to predict the same category object to pick perfectly (accuracy close 1.0). However, this is simpler set- ting than sequential decision-making. During the rollout, any error in state could ate a setting that is out-of-distribution for the policy. Thus, category token accuracy sets
19

---

0.6
GNN-LL
TIP
0.0
CINNA
TTP
GAN--UnE-IL
GNTA-WRE--
0.5
emmprra Acverence
0.5
Spatal Adherencc
0.4
0.4
0.3
0.2
0.2
0.1
0.1
0.0
0.0
Seen pref
Undeen pret
Seen pref
Lnseen pref Seen #fob)
Seen 6co
Soon #0D)
Seon Vooj
(a) Spatial Preference Adherence (SPA)
(b) Temporal Preference Adherence (TPA)
Figure 14: Comparisons of TTP, GNN-IL, and GNN-VAE-IL simulation across two metrics: SPA and TPA. TTP shows good performance at the task of dishwasher loading on seen and unseen pref- erences and outperforms the GNN-IL and GNN-VAE-IL baselines. Note that GNN-IL has ground- truth training data on unseen preferences.
an upper bound for rollout performance, that is, while having high category token accu- racy is necessary, it is not sufficient for high packing efficiency and inverse edit distance.
test/Acc/category.cu tokeee token
Policy Rollouts for Evaluation We evaluate trained policies on rollouts in the simulation, more complex setting than the accuracy of prediction. Rollouts require repeated decisions in the environment, without any resets. A mistake made early on in a rollout session can be catastrophic, and result in poor performance, even if the prediction accuracy is high. For ex- ample, if a policy mistakenly fails to open dishwasher rack, rollout performance will be poor, despite good prediction accuracy.
Showing
content. batch_size:
conntent.batch.... batch_size: 256
content. batch_size: 128
0.8
0.6
0.4
Note that the simulated dishwasher loading is a challenging task. The performance of the random agent is bad because we evaluate the SPA at the end of an episodic rollout. The per- formance of random policy reduces exponentially with the length of the action sequence (typically around 20-30 steps long), and the number of visible objects in the scene (between 6 to 20). SPA is low if the policy either (1) fails adhere to the physical constraints, (2) violates the spatial preference. Even if we ignore the physical constraint violations and consider the task of selecting the preferred rack for each object, the probability of a SPA of 1.0 for random policy would be around (1a Nobjects since it has to choose either top or bottom rack per object.
0.2
Stop
Figure 13: Category level accu- racy grouped by batch size for prompt-situation training.
We compare TTP, GNN-IL, and GNN-VAE-IL performs better than GNN-VAL VAE-IL in terms of SPA but both are similar in terms of TPA. Intuitively, GNN-VAE-IL solves harder learning problem of inferring preference from the trajectory, unlike GNN-IL which uses a privileged ground-truth label for the unseen preferences.
and unseen preferences in Fig. 14. GNN-IL
Temporal efficiency Just like SPL [53] for navigation agents, we define the efficiency of tem- poral tasks in policy rollout, order to study how efficient the agent was at achieving the task. For episode E [1,...]] let the agent take p; number of high-level interactions to execute the task, and the demonstration consists of l interactions for the initial state. We scale the spa- tial preference adherence SPA, of the policy by the ratio of steps taken by the expert versus the learned policy. Temporal efficiency is defined between 0 to 1, and higher is better. This
20

---

5
Task Obj Place
Task-Rack
Pref-Loc
Pref-Order
Repeat Obj
Repeat Rack
Time Exceed
Sim Err
Figure 16: Failure analysis for TTP. X-axis indicates enlists different types concerning physical constraints (Task-Obj Place: object place pose is infeasible, Bask-Rack: not placing when in top when its open, not placing in bottom when top is closed and bottom is open, placing object when both racks are closed), preference constraints (Pref-Loc: location violation in terms of incorrect rack, Pref-Order: order violation of preference) and No-ops (Repeat Obj: picking an already placed object, Reposition: Adjust the placement of an object within the same rack, Repeat Rack: picking a dishwasher part repeatedly; leads termination, Time Exceed: leads to termination). Y-axis is the еπrо count rollout. Most errors are due to failing in the location- aspect and order- most rollouts, indicating
aspect of the preference. The repeat-* pattern triggers the termination that failure scenario usually obeys physical constraints but does not make progress towards the task, thereby resulting in lower SPA and TPA.
value will equal to or lower than the packing efficiency. This especially penalizes policies that present 'looping' behavior, such as repeatedly opening/closing dishwasher racks, over poli- cies that reach low SPA in shorter episodes (for example, by placing most objects in the sink). Fig. 15 shows the temporal efficiency over our main held-out test settings (as shown in Fig. 5).
0.6
D Failure Analysis
Tanpprral emouncy
0.4
We present the failure analysis in terms of error counts over 63 rollouts. Figure 16 shows that TTP learns physical con- straints of the problem perfectly, and never takes an action that violates them. Instead, the failure cases stem from ei- ther (1) not following user preferences, or (2) repeating certain actions which don't make progress towards the task, like repeatedly picking and placing the same object. In contrast, our baselines like continuous pose prediction suffer from violating task constraints, like attempting place object in already occupied location. Fig. 17 provides distribution per error type shown here, with leftmost bar near zero indicating that that error never occurs, and the right extreme denotes that that error occurs in high frequency in some rollouts. Both (1) and (2) stem out of the problem of sequential decision making. An open problem is to learn policies that can adapt from interaction and correct for such errors.
0.2
Seen
Unieei #rB1
Sieen eobi
Unieen
Figure 15: TE metric for held-out test settings.
E Additional Ablation Experiments
Towards the end of the section 3.1, presented ablation experiments the number of demon- strations per preference used for training, and the number of unique preferences used. In this section,
21

---

column Task-Obj Place
column Task-Rack
60
50
40
30
20
10
0
column Pref-Loc
column Pref-Order
60
50
40
30
20
10
0
column Repeat Obj
column = Repeat Rack
60
50
40
30
20
10
0
column Time Exceed
column Sim Err
60
50
40
30
20
10
0
5
10
value
value
Histogram per error type to estimate the frequency (Fig 16 shows only mean and std). X-axis shows the value of the error count. Y-axis shows the number of times that value occurs in all of the rollouts. Most rollouts have 0 or very few preference based errors. Error count show peak near 0 and gradually reduces of towards higher numbers. This is an extended analysis of the error counts in figure 16 showing the failure profiles for the different error types.
Figure 17:
22

---

we present additional ablation experiments over (E.1) the design of instance encodings in TTP, (E.2) the effect of increasing the temporal context of TTP on performance, the comparison between the choice of (E.3) discrete category tokens vs continuous box extent representation in TTP, and (E.4) continuous placement pose vs discrete alternative used in TTP.
E.1 Design of Instance Encoding
Why object attributes are important? The pose describes where an object is. For this task, it depends on whether the object is on the counter or in the dishwasher.
The category is important to follow the order-based preference and to group the placement instances. big bowl may have very limited placement instances in dishwasher rack compared a plate or small cup.
The timestamp is important for encoding the prompt it allows the neural network to reason over prior instances without explicit tracking. Tracking an object across frames is difficult, especially with partially observed scenarios (such as an bowl in the top rack is not visible the top rack is closed).
The marker is used to denote whether it is pick place instance. Marker acts as flag and allows us to use the same architecture input to process a variable number of both pick and place instance embeddings.
Dest/Acc/categorytool token
test//Acccateegry_too _token
test/Acc/category eokee token
Tompora CenporalEncoderfae
Dunny
PoseEncoderHLF DunnyInput Poseincodrrtouriie
Tereporal Exbeddidn
wew.ggorynooeFoouurr SerHLP
JumnyInput
CategorsEncoerrtLP
0.9
0.9
0.8
0.8
0.7
0.6
0.6
0.5
0.5
Stop
Step
0.4
2k
10k
(a) Temporal encoding
(b) Category encoding
(c) Pose encoding
Figure 18: [Left-to-Right Right] Comparing different design choices of attribute encoders in terms of category token accuracy on held-out test prompt-situation session pairs.
How much does temporal encoding design matter? Fig. 18a shows that learning an embedding per timestep or expanding it as Fourier transformed vector sufficient size achieves high success. On the other hand, having no timestep input shows slightly lower performance. Timestep helps in encoding the order of the prompt states. The notion of timestep is also incorporated by autoregres- sive masking in both the encoder and the decoder.
How much does category encoding design matter? In our work, we represent category as the extent of an object's bounding box. An alternative would be to denote the category as discrete of categorical labels. Intuitively, bounding box extents capture shape similarity between objects and their placement implicitly, which discrete category labels do not. Fig. 18b shows that Fourier transform of the bounding box achieves better performance than discrete labels, which exceeds the performance with no category input.
How much does pose encoding design matter? We encode pose as a 7-dim vector that includes 3d position and 4d quaternion. Fig. 18c shows that the Fourier transform of the pose encoding performs better than feeding the dim through MLP. Fourier transform of the pose performs better
23

---

De:
Cw:
cN:
Pick latent
Place latent
0.6
Transformer Encoder
0.4
Instance Encoder
0.2
x1
X2
Xi
Opick
xi x2 ...x dpick of 92 ...9] aptace
Step
20k
30%
50k
(a) Model workflow for context History k
(b) Plot with context history of 0,1,3,7
Figure 19: Incorporating Context History instead just current observation. Left: Modified TTP for processing with previous Context History k. Right: Plot showing category level accuracy for the held-out test sessions for single preference training with context windows. While larger context window size learns faster, the asymptotic performance for all context windows converges in our setting.
because such a vector encodes the fine and coarse nuances appropriately, which otherwise either require careful scaling or can be lost during SGD training.
E.2 Markoy assumption on the current state partial visibility scenarios
Dynamic settings, as used in our simulation, can be partially observable. For example, when the rack is closed, the policy doesn't know whether it full or from just the current state. If new object arrives, the policy needs to decide between opening the rack if there is space, or dropping the object in the sink if the rack is full. In such partially observed settings, the current state may may not contain all the information needed to reason about the next action. However, given information from states in previous timesteps, the policy can decide what action to take (whether to open the rack or directly place the object in the sink). With this in mind, we train a single preference policy for picking only with different context history. As shown in Fig. 19a, the context window of size processes the current state as well l predecessor states, that is, in total l: + 1 states.
Let context history & refer to the number of previous states included in the input. Then the input is sequence of previous h states' instances (including the current state), shown in Fig. 19a.
Fig. 19b shows that TTP gets > 90% category-level prediction accuracy in validation for context windows. While larger context windows result faster learning at the beginning of the training, the asymptotic performance of all contexts is the same. This suggests that the dataset is largely visible and single context window captures the required information. In the future, would like to experiment with more complex settings like mobile robots, which might require longer context.
E.3 Discrete Category Tokens
Category embeddings should be more descriptive than just bounding box and include some lan- guage labels like knife versus fork. In our current work, we assumed that similar-sized objects have similar occupancy footprints, and this was enough to reason about feasible placing locations rack. This was conducive to learning the physical constraints of objects in the scene but ignores other relationships between objects. Figure 20 shows TTP-DCat that indicates the performance with the discrete category is much lower and requires hyperparameter tuning for the embedding size. In future work, would like to explore additional multi-modal object features, including language descriptions along with the geometry information.

---

0.6
Acherence
0.4
0.4
Tamporal
0.
0.3
Conti
DCat
TTP
(a) Spatial Preference Adherence (SPA)
(b) Temporal Preference Adherence (TPA)
Figure 20: Comparisons of TTP, Continuous Place Pose Prediction (TTP-ComIP), and Discrete Cat- egory Token Input (TTP-DCat), in simulation across metrics: SPA and TPA (on Y-axis), preferences and objects during training. Continuous Pose prediction achieves much lower SPA due to infeasible object place pose prediction and thereby violates the location-aspect of the preference. As the TPA is based on the pick order sequence, TPA is high indicating that the policy learns the order-aspect of the preference. Discrete category prediction achieves low SPA and TPA due few as categories mapped to the embedding of size 64; similar to original TTP with continuous category representation in terms of the bounding box.
Task-Obj Place
Rack
Pref-Loc
Pref-Order
Repeat Obj
Reposition
Repeat Rack
Model
Discrete place pose (Ours)
Exceed
Continuous place pose
2
6
8
10
12
Counts per Rollout
Figure 21: Failure case analysis for continuous place pose prediction. The X-axis indicates the count per rollout. The Y-axis enlists different error types concerning physical constraints (Task-Obj Place: object place pose infeasible. Task-Rack: not placing when in the top when its open, not placing in the bottom when the top is closed and the bottom is open, placing object when both racks closed), preference constraints (Pref-Loc, Pref-Obj) and No-ops (Repeat Obj: picking an already placed object, Reposition: Adjust the placement of an object within the same rack, Repeat Rack: picking dishwasher part repeatedly; leads to termination, Time Exceed: leads to termination).
25

---

E,4 Continuous Placement Pose
In TTP, we consider two solution spaces: decision over "what to pick', and 'where to place". The pick solution space is naturally discretized by the objects of interest visible in the environment (dishwasher, racks, dishes, etc.), and does not limit real-world transfer.
Note that while we consider discrete objects, their features like location are continuous. An object segmentation pipeline, as used in our real-world experiments, can provide discrete set of objects to interact with, and their continuous features. On the other hand, our choice of discretized 'where to place' solution space can be seen as a hurdle to real-world transfer. However, in our experiments, dense discrete poses generalized to unseen scenarios better than continuous prediction. Figure 20 shows that discrete placement pose instances achieve higher SPA and TPA than continuous place pose prediction. The TPA looks comparable the original TTP but the SPA is much lower. This is because the TTP-ContP policy Iearns to pick objects in the correct order like TTP but suffers in placement pose prediction. Figure 21 shows the count of error types over 9 rollouts for each of the 7 preferences (=total 63 rollouts). Here lower error count is better. Error for continuous place pose policy occurs mostly in the prediction of infeasible place pose for the object.
The geometry of dishwasher well-suited for discrete positions, as small prediction errors can make target place position infeasible. For example, a plate can only fit in a particular pose, and small orientation errors make placing infeasible. In the real world, one could imagine creating dense place pose dictionary based on the model of the dishwasher in the user's ss home. However, we agree that for other receptacles, like shelf, this discretization is not necessary, and continuous prediction approach like our baseline might be better suited.
Finally, we note that while exclusive. Hence, several discrete poses overlap, and if one pose is occupied by object, there might be several others that are also occupied by the same object. TTP has to reason about the size of an object and distances between place poses to decide which pose is empty to place the next object, while maximizing packing efficiency. Hence, our policy able to reason about 3D space occupancy, even if working with discretized poses.
discrete place poses, they are densely sampled, and not mutually
F Limitations and Future scope
In Section 5, we briefly discussed the limitations and risks. Here we enlist more details and highlight future directions.
Pick grasping depends on accurate segmentation and edge detection Grasping policy depends on the quality of segmentation and edge detection of the selected object. Due to noise in calibration, shadows, and reflections, there are errors in detecting the correct edge to successfully grasp the object. For example, it hard to grasp plate in a real setting. The plate very close the ground and the depth cameras cannot detect clean edge for grasping. As the depth edges are not often not accurate due to noisy point clouds, the collision prediction with the candidate grasps and the object is not very robust and conservatively results in no feasible grasps. Therefore, in our work, we place the plate on an elevated stand for easy grasping. Grasping success also depends on the size and kind of gripper used.
Placement in real setting For placement, the orientation of the final pose is often different from the initial pose and may require re-grasping. The placement pose at the final settlement is different from the robot's end-effector pose while releasing the object from its grasp. Similar to picking, placement accuracy will largely depend on the appropriate size and shape of the gripper used. Due to these reasons, placement in the real world is an open challenging problem and we hope to address this in future work.
Hardware pipeline issues due to calibration The resulting point cloud has noisy depth edges due to two reasons. First, incorrect depth estimation due to camera hardware, lighting conditions,
26

---

shadows, and reflections. Second, any small movements among cameras affect calibration. If we have a noisy point cloud, it is more likely to have errors in subsequent segmentation and edge detection for grasp policy. Having sufficient coverage of the workspace with cameras is important to mitigate issues due to occlusions and incomplete point clouds.
Undesired motion due to IK and grasping We use the Pinnochio library [54] for planning robot joints given the end-effector this is fast and easy to But it assumes collision-free plans, which is difficult with relaxed IK limits to enable robot arm is reachable to the base of the drawers. Also, the robot faces difficulty in grasping certain objects that are closer to the base and occluded in the camera images.
We use GraspNet [21] for obtaining analytical candidate grasps on the segmented point cloud for dish. The best candidate grasp may not be always semantically appropriate, for example, the best candidate grasps
cup wine glass closer to the pinched at the rim (due to clearer edge detection) but many humans prefer to hold the curved cylinder stem. Incorporating preferences in low-level grasping is an open direction for future work.
Incomplete information prompt The prompt session may not contain all the information to execute the situation. For example, in a prompt session, there might be no large plates seen, which incomplete/ambiguuus information for the policy. This can be mitigated by ensuring complete information in the prompt demo or having multiple prompts in slightly different initialization.
27


Reviews:
[{"review_id": "gQq1BSizYnda", "reviewer": "Reviewer_3v7A", "summary": "This paper uses a household task \"placing kitchen objects in the dishwasher\" to test a newly proposed preference-guided task planner model using a transformer. The authors applied plenty of techniques from the natural language processes in the current work to encode the environment input and embed human preferences, which is very interesting and brings good inspiration to the robotics community. The work is both theoretical and technical and the experiment result has been well discussed. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The motivation of long-term motion sequence planning is well explained. The reason to use a transformer as a planner is clear and convincing.\n- The conception, model, and math symbols are structured and explained in a very clear way. It is very straightforward to capture the workflow and ideas from the paper, even for people without much experience in transformer and sequence planning, like me. \n- The paper is well written. I enjoy reading it without finding any explicit typos or mistakes.\n- The authors have offered a comprehensive prior work investigation, including sequential manipulation, transformers as sequence modeling, and preferences and prompt training, which benefits the readers to work in a similar field.\n- The authors used the task \"putting kitchen bowls and plates in a dishwasher\" to prove their model's advantages. In my opinion, such a setup is a good choice and is complex enough to prove the advantage of the model. \n- This paper has offered a proper ablation study to prove all the attributes (pose, time, category) are necessary to achieve a good result.\n- Interestingly, the authors used some knowledge from the NLP field in the current robotic task to support the good performance of the transformer model, which breaks the barrier between the two fields and offers some inspiration to the researcher in the robotics field who do not have much experience in the NLP.\n- Due to the page number limitation, some of the details regarding the experiment settings, type of preferences, the reason for using drawer in reality, camera perception, etc. are missing in the main manuscript but offered in the appendix, which has answered some of my questions. The demo video is also nicely made and brings a lot of useful information.\n\nCons:\n- It will be great if the authors can extend figure 3 and illustrate the content from lines 122 to 134 in it as well. This will help understand the mechanism details. \n- The author presented in the prior work section, that there is a lot of related work in this field. But unfortunately, I do not see a clear reason to use GNN-based preference learning as a baseline.\n- In contrast to the proposed model, however, I don't know how the GNN baseline works, especially because the baseline is a combination of two different papers on GNN. I would suggest the authors add some illustrations in the appendix to show the workflow on page 6 of the baseline as well, as to improve the confidence of the readers.\n\n\nNeutral:\n- A nitpicky question regarding the preferences, but I expect an answer from the authors. Sometimes we as users do not know our preferences. Badly, we are also lazy to perform any demonstration to show our preferences. So is it possible to have some pre-programmed preferences mode, e.g., \"place bowls first on the higher rack, then put plates on the lower rack\"? Because the preference embedding is an intermediate variable in the current model, it is hard to translate it to \"mode 1, mode2, ...\" and ask the user to choose.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend this paper as l enjoy reading it, although I am neither a researcher in human preference learning nor attention mechanism. This paper focuses on a very practical problem that needs a high-level sequence learning conditioning human preferences. The model is well explained and the experiment is clearly presented and discussed. I can capture most of the ideas of this paper as well as recognize the main contribution of this paper. Although there are a few unclear things that I have pointed out in the weakness section, I believe the authors are capable to improve the paper in the revision session to strengthen my confidence even further. However, if the other reviewers with more knowledge in Attention or Preference learning pointed out some critical issues of the current paper, I will also consider their arguments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- I would ask the authors to improve fig.3 to add more technical details in it, see the weakness section.\n- I would ask the authors to add some reasons for why choosing GNN as a baseline and an illustration of how the baseline has been used in the current setups.\n- Figure 5, caption, typo: \"if worse\"\n- There are a few small errors in the appendix, like missing hyperlinks.\n- Figure 5b, why does the GNN-IL perform better in unseen preferences and unseen objects?\n- My understanding of human demonstrations is a long sequence of which object to take and where to place it, which is the high-level action choices, and it is also the main focus of the current paper. But, how can we get the low-level control commands, and deal with the issues such as collision avoidance?\n- In line 225, the performance on 3-4 objects is poorer and the authors claimed it is because it is unseen. But my question is that if e.g. 5 objects have already been placed in the dishwasher from the setup with 8 objects initialized at the beginning, it reached a similar situation having 3 objects as the initial state. So what are the differences between the two cases?"}}, {"review_id": "dOoBlpszivk", "reviewer": "Reviewer_999X", "summary": "This paper casts rearrangement planning as a sequence modelling problem, and applies the transformer architecture to solve it. The user prompts the model with a demonstration, e.g. placing a few plates into a dishwasher, from which the model infers the user’s preferences for arranging the scene. Given a new set of objects, the model can then predict a sequence of actions, including opening drawers and moving objects, to complete a rearrangement which is consistent with the preferences in the prompt. Target poses are selected from a set of discrete options. Experiments show that this particular transformer architecture is well-suited to modelling these rearrangement sequences, generalises well in several ways, and the method is successfully validated with a real robot demo.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The technical quality and execution of the paper is impressive, in adapting this powerful transformer architecture for the rearrangement setting and addressing a challenging rearrangement scenario. There are many useful insights and techniques in the paper which the community will benefit from seeing used. For example, the dot-product technique for selecting an instance used on Line 132.\n1. The authors have thought carefully about how they envisage the user interacting with this system (e.g. through preference prompts), which makes it a promising direction for deployment to robots working with users.\n1. Unifying actions for moving objects and opening/closing containers is done elegantly, and is a useful capability for rearrangement methods to expand from tabletop environments to more complex scenes.\n1. The real-world demo is useful, because it gives insights on which parts of the method make it well-suited for deployment on the real robot and which components are challenging to transfer.\n1. Investigating the effect of the number of objects, especially in the out-of-distribution setting, is interesting. There is insightful analysis about failure cases, e.g. closing the dishwasher early, which is useful for future work to build on this. Other useful ablations are also performed.\n\nLimitations:\n1. A major contribution of this paper seems to be encoding the preferences of a user for the order in which objects are placed. However, the importance of this capability needs to be motivated more persuasively. If the end arrangement is the same, why does the user care about the order in which objects were moved there? Without this motivation, the impact of this contribution is not very clear. One way to frame this might be: human demonstrations contain not only preferences over end states, but clues about how to achieve those end states (e.g. open the dishwasher first). That is why encoding the order of actions is also useful.\n1. Discretization of the solution space is a limitation of this method in its current form. If I understand correctly, the target pose for an object is picked from a set of pre-defined options. In simulation, these options can indeed be generated automatically. But how can these options be generated for real scenes? Does an engineer have to define a set of possible target poses for each object (the appendix suggests this)? Does a model of the environment have to be created and loaded into simulation? This also makes it difficult to apply this method to rearrangement problems without the grid-like structure of a dishwasher, for example lining up mugs on a shelf, since the number of options would need to be large to achieve precise alignment. Have the authors explored training this method to regress poses instead, i.e. make continuous pose predictions?\n1. In the current version of the method, the object features focus on the bounding box and could be made more semantically expressive. I.e., can the method tell a fork and knife apart if they have the same bounding box? Does it know that they are both cutlery, and so they share semantic features and so they should be placed similarly? To be fair, this is addressed in the limitations section.\n1. There are some limitations with the experimental procedure, and places where it could be described more clearly. “Packing efficiency” is used as a metric, which makes sense for dishwasher loading, but the use of this metric makes it difficult to intuitively understand how good a method is, and how good it would be on other tasks, compared to a metric such as distance error from an acceptable position for an object. Additionally, this packing efficiency is set to zero if the policy violates a preference (Line 187), which I assume includes arranging the drawers in a different order. It might be good to have an experiment which uses a metric solely measuring how good the end state is, since in most scenarios it is unclear why the user would care about anything else, so long as the end state is the same. There are some useful details about the experiment setup that are in the appendix but could go in the main paper to make it self-contained as a description of the experiments. Specifically: full examples/descriptions of what constitutes a preference in these experiments. Additionally, a more self-contained description of how the GNN-IL baseline method works would help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a technically strong paper which addresses a challenging rearrangement problem: predicting how to load a dishwasher well. To improve it further, there are a few limitations to address which would further motivate its contributions and clarify experimental details. If these limitations can be overcome, this work will likely have a major impact on the field of object rearrangement and will be useful for the robot manipulation community at large.\n- - -\nEDIT: thank you to the authors for their hard work in completing the rebuttal. Taking the paper and discussion into consideration, I have decided to keep my recommendation of Weak Accept. The main reasons for this are as follows:\n* The authors have argued more persuasively for the importance of temporal preferences. This is particularly evident in the cooking context. To improve this work further, it would be interesting to see experiments that show this method applied to additional tasks such as this, rather than focusing on the single task of loading a dishwasher, where preferences may more easily be inferred from the end state.\n* I agree with the claim in the rebuttal which says that discrete prediction is easier that continuous prediction **if** you have a set of target poses to choose from which are all feasible. It is clear how this makes the task easier for the model. The limitation is that densely sampling these target poses is challenging for real-world scenes. One way forward, as the authors suggest, is a pre-defined dictionary of suitable poses based on the specific model of the dishwasher that the user has in their home. However, this does make the method less easy to apply and limits the tasks for which it can be used. That is why this is a limitation of the method. However, loading items into dishwashers, drying racks, ovens, etc is still useful for a robot, and so the paper is sufficiently impactful. I encourage the authors to briefly mention this hurdle to real-world deployment in the limitations section of the final main paper, since they do agree with this and mention it in the appendix. The rebuttal mentions that this was included in Section 6, but the limitations section still seems to be Section 5 in the updated paper, and does not seem to mention this point, if I am reading it correctly.\n* Thank you to the authors for their clarifications on the experimental setup, and the detailed responses in general. Much appreciated!\n\nOverall, the paper is technically well-executed and addresses a useful task in home robotics, so it will make a valuable addition to CoRL 2022.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "1. I think it needs to be made clearer early in the paper why the robot cannot simply copy the prompt actions. It is because the prompt set of objects are different to those in the situation (as explained in the video). For example, in the real world, the user might demonstrate preferences with a couple of plates, and the robot should extend this preference to the whole object set with many more plates. If the only difference between the prompt and the situation were the initial poses of the objects, then simply copying the prompt actions might be fine.\n1. The related work discussion can be made more precise to the specific problem addressed by the paper. For example, instead of discussing the general benefits of machine learning approaches on Line 19, this space can be used to discuss the related work in more detail, and explain in more detail where this method extends on it (e.g. here is where we extend beyond [23]).\n1. It would be useful for the reader if the paper could discuss the relationship between graph attention networks and transformers, since the former are used in prior work discussed in this paper [23,24]. When applied to this use-case, the two architectures share many similarities (computing attention between all objects in the scene). So, if prior work has already applied a similar attention mechanism, then where does the novelty come from in applying transformers? Perhaps transformers are a better fit than GATs when rearrangement planning is viewed as a sequence modelling problem?\n1. This is a minor thing, but some of the formal notation can be clarified to be more consistent. Indexing starts from 0 in some places and from 1 in others when referring to the same variables. E.g. see indexing of x in lines 56 and 61. It does not affect the clarity of the overall paper too much, however.\n1. Line 256: possible typo. Should this be “not overfitting” or “not underfitting”? I think not underfitting. Because if the model is overfitting, then it would still benefit from the additional training data. Whereas if it were underfitting, it would not benefit because it is struggling to fit the data it already has, if I understand this argument correctly.\n1. The limitations section provides a good overview of future work, but could benefit from also having a frank discussion about inherent limitations due to the design choices in the method, for example the need to have a pre-defined set of poses to choose from."}}, {"review_id": "XN_pBUK5V--", "reviewer": "Reviewer_jkUB", "summary": "This paper presents a transformer-based approach to incorporating human preferences for high-level task planning. The authors focus on simulated and real-world dishwasher loading tasks, in which humans have preferences about where items should be loaded and the order in which loading occurs. Preferences are communicated to the robot through a single demonstration. The proposed approach demonstrates an ability to understand those preferences from that demonstration. The proposed approach improves on an imitation-learned baseline (using a GNN) trained by the authors in situations where the number of objects matches that seen during training and is competitive otherwise.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has weakness stemming from two main areas: the motivation behind the chosen experimental scenario and clarity regarding the technical approach. [I will also admit that the clarity of the paper could also use some improvement; it is possible that I have misunderstood some key aspects of the paper, and have incorporated clarification questions in my discussion below so that the authors can help me better understand if I am incorrect.]\n\nFirst, I have concerns about the central metric being used for evaluation (or the clarity with which it is presented): the Packing Efficiency. The definition of the Packing Efficiency must be clearer, as it is difficult right now to understand how it is computed yet is key for understanding the performance of the proposed approach. Do a_hat and b_hat incorporate the preferences, or is the equation on lines 185--186 computed first using the items placed in the dishwasher and *then* the result is set to zero if the preferences are not met? Does this mean if *any* preferences are violated, the PE is zero. If so, this would help to explain why the performance of the random agent is so low, since it would require getting quite lucky to achieve a result in which all the objects are placed correctly. If not, it is not clear why the performance of the random agent would be so poor, since if the dishes were randomly placed, should not the performance of the system be roughly 50% accurate? Either way, some clarity on the random baseline and the precise meaning of the PE metric would greatly help the paper.\n\nThe experimental setup is not particularly well justified in the text. Critically, the agent's tasks include opening and closing the dishwasher and sliding racks in and out. The authors mention in a few places in the text that planning failures may occur when an agent attempts an action to put an item in the dishwasher yet the dishwasher is closed. There are two issues with this.\n\nFirst, the rate of failures associated with constraints borne of these manipulation constraints is not reported in the text, making it difficult to understand the impact they have on the performance. The authors should either better justify their decision to incorporate these actions/constraints or provide statistics along with their existing results that identify the rate of failures associated with physical constraints. The purpose of the proposed approach is to evaluate the ability of the agent to identify and incorporate preferences, yet it seems that many of the demonstrations are limited instead because the system is additionally asked to predict physical constraints.\n\nSecond, expecting the agent to explicitly enumerate those actions feels like an artificial limitation; the agent is already running a non-learned deterministic planner to plot the trajectories to move the dishes into the dishwasher, so it is unclear why that planner would also not be able to ensure that those constraints are first met. On line 173, the authors point out that these manipulation constraints are why the problem setting is more challenging than a simple prediction task. However, if the motion planner takes care of those constraints simply predicting where objects should be placed in the dishwasher may be able to achieve high performance on the PE metric (though not the Inverse Edit Distance metric). In the text, the authors must better justify the nature of this problem setting or clarify why the proposed methodology is the way to best evaluate this system, since it is important to motivate the need for their approach to be used.\n\nFinally, it is not clear why the order in which objects are loaded into the dishwasher is a particularly salient concern, and so the authors should clarify why dish loading preferences need to include order of operations in addition to placement region. This is the smaller of my high-level concerns, since I am willing to accept that there are household scenarios in which the sequence of operations matters. I am not convinced that loading dishes into the dishwasher is one of those tasks. However, the results showing that it is *possible* using the proposed approach are still valuable, and I think do add to the contributions of the paper. Rather than performing new experiments, which I do not think necessary for this purpose, the authors should discuss a situation in which order of operations is a preference that robots should consider when executing household tasks (even if that scenario does not appear in the paper).\n\nSmaller questions and comments:\n- Why is a bounding box used as an input for the category label rather than a one-hot encoding of the object? This seems prone to error and feels like an artificial limitation. While additional experiments are unlikely to occur, the authors should better justify this decision.\n- The equation for PE seems to have an error; if the robot's performance is perfect, each of the terms inside the argument should be 1 (totaling to 2) and then summed over the individual demonstrations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper was somewhat hard to follow in places, as some details (especially the definition of the central metric under scrutiny) were difficult to follow or missing. In addition, the motivation for various aspects of the paper were not very clear; both the problem setting and the experiments were lacking, raising questions about the design decisions behind both.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "The authors should focus on clarifying both the technical details of their approach as well as better motivating the problem setting and experiments presented in the paper. In addition, experiments or results that show the impact of expecting the agent to predict actions to open/close the dishwasher and doors will greatly help to understand the efficacy of the proposed approach on learning preferences without the need to also learn manipulation constraints."}}, {"review_id": "IRlu8onj3mi", "reviewer": "Reviewer_qr8n", "summary": "This work proposes a transformer-based approach for learning preference-conditioned policies in object-centric manipulation environments (specifically, loading a dishwasher). Preferences are expressed through single demonstrations. Learning is done via behavioral cloning on a given set of preference-labeled demonstrations. Experiments in simulation and on a real robot compare the proposed approach with graph neural network baselines and consider generalization to unseen numbers of objects and unseen preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* I really like the “dynamically appearing objects” setting in the experiment and appreciate the in-depth analysis in the appendix.\n* I really appreciate the detailed experimental analysis, including the results in Figure 6, Table 1, and in the appendix. The experimental design overall is great.\n* The writing overall is very clear and easy to understand.\n* The related work seems thorough.\n* The dishwasher application is nice and clear, and seems like a challenging application.\n\n\n### Weaknesses\n\n* My biggest concern is the representation of preferences that is given to the GNN baselines. L198 states that the representation is categorical and argues that this is privileged information. For the in-distribution experiments, this is maybe okay, but there is still considerable information loss in using categories versus full demonstrations. But for the out-of-distribution experiments, do I understand correctly that the GNN will simply receive a different categorical value than it saw during training? There would be no hope for generalization if this is the case, and this would be an unfair comparison to TTP.\n* It is a little disappointing that generalization to out-of-distribution object numbers is poor, but I appreciate the analysis in Figure 6a on this point.\n* The work is a little more incremental than novel, but the all-around quality and thoroughness still makes for a solid contribution.\n* There is some redundancy in Section 2 that could be removed. For example, in the State-Action Representation subsection, there is discussion of the policy. But then the policy is discussed again throughout the rest of the section. Section 2.2 is also somewhat redundant with the end of Section 2.1. Overall, more strict adherence to the topics of the subsections would help clarity.\n* If I understand correctly, the demos used in the main results are generated as follows: K demos are generated for loading the dishwasher in order A, B, C…; then K more demos are generated for loading the dishwasher in order C, A, B, …; etc. This procedure seems pretty far removed from a real LfD setup. In reality, I would expect that humans loading dishwashers may have some preferences, but are also largely ambivalent about order. Furthermore, any particular human would probably be inconsistent if you asked them to load the dishwasher multiple times. It would be nice to see results for demos collected from real users, or at least demos collected in a noisier and less structured way.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work is limited in novelty, but otherwise very well done. I think the in-depth experimental analysis and the details of the policy architecture will comprise nice additions to the literature.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "   * I think that use of the word “planning” in the title and name of the approach is misleading. Planning usually has the connotation of imagining possible futures with a transition model and reasoning toward a goal. The approach in this work uses a reactive policy and does not use transition models or goals.\n   * The word “category” is also potentially misleading, because if I understand correctly, the category is just the bounding box of the object. A better term might be “shape” or “size” or “geometry”. I like the additional analysis and discussion in Figure 8b, though.\n   * The definition of the encoded state on L85 suggests that the instances are ordered into a list before passing through the network. Is the model invariant to this order? It should be, right? If so, I would just suggest changing the list to a set in the notation.\n\n\n### Minor\n   * L16: “task constraints through symbolic task description” -> descriptions\n   * The x_i start at 0 on L61 but started at 1 on L56\n   * I found the term “situation” confusing in Section 2.1. For example, in figure 3, “Situation” is input to the instance encoder on the bottom. But isn’t that just the current state? In other places, it seems like situation is meant to refer to both the state and the preference. Also the “situation decoder” and “policy” are maybe synonymous?\n   * L112: “The left half is prompt encoder” → the prompt encoder\nL113: “acting on given state” → “the given state”\n   * L124: “as well as, placement” → “as well as placement”\n   * Figure 4 caption missing a period\n   * L172: “unseen number of object” → “numbers”\n   * Figure 5 caption: “if worse” → “is worse”\n   * It would be good to make the y axis limits identical in Figure 6\n   * I didn’t understand what this means: “The policy was successful once with 100% success rate, and once with 75%”\n   * There are broken figure and section references in the appendix\n"}}]

Author rebuttal:

Decision: Accept (poster)