SLAP: Spatial-Language Attention Policies
Authors: Priyam Parashar
Vidhi Jain
Xiaohan Zhang
Jay Vakil
Sam Powers
Yonatan Bisk
Chris Paxton
Venue: CoRL 2023 Poster (2023)
Keywords: learning from demonstration
language-based robotics

Abstract: Despite great strides in language-guided manipulation, existing work has been constrained to table-top settings. Table-tops allow for perfect and consistent camera angles, properties are that do not hold in mobile manipulation. Task plans that involve moving around the environment must be robust to egocentric views and changes in the plane and angle of grasp. A further challenge is ensuring this is all true while still being able to learn skills efficiently from limited data. We propose Spatial-Language Attention Policies (SLAP) as a solution. SLAP uses three-dimensional tokens as the input representation to train a single multi-task, language-conditioned action prediction policy. Our method shows an 80% success rate in the real world across eight tasks with a single model, and a 47.5% success rate when unseen clutter and unseen object configurations are introduced, even with only a handful of examples per task. This represents an improvement of 30% over prior work (20% given unseen distractors and configurations). We see a 4x improvement over baseline in mobile manipulation setting. In addition, we show how SLAPs robustness allows us to execute Task Plans from open-vocabulary instructions using a large language model for multi-step mobile manipulation. For videos, see the website: https://robotslap.github.io

SLAP: Spatial-Language Attention Policies
Priyam Parashar'
Vidhi Jain"
Xiaohan Zhang 1.3
Jay Vakil
Sam Powers!:2
Yonatan Bisk 1.2
Chris Paxton
Meta
Carnegie Mellon
SSUNY Binghamton
Abstract: Despite great strides in language-guided manipulation, existing work has been constrained to table-top settings. Table-tops allow for perfect and consis- tent camera angles, properties are that do not hold mobile manipulation. Task plans that involve moving around the environment must be robust to egocentric views and changes the plane and angle of grasp. further challenge is ensur- ing this is all while still being able learn skills efficiently from limited data. We propose
Language Attention Policies (SLAP) solution. SLAP uses tiree-dimensional tokens as the input representation to train single multi-task, dnnguage-conditionns action prediction policy. Our method shows an 80% success rate in the real world across eight tasks with single model, and 47.5% success rate when unseen clutter and unseen object configurations are introduced, even with only a handful of examples per task. This represents an improvement of 30% over prior work (20% given unseen distractors and configurations). We see 4x improvement over baseline in mobile manipulation setting. In addition. show how SLAPs robustness allows to execute Task Plans from open- pocabbocay in- structions using a large language model for multi-step mobile manipulation. For videos, see the website: https: robots aap gi thtb.uu.... io
Introduction
Transformers have demonstrated impressive results on natural language processing tasks by be- ing able to contextualize large numbers of tokens over long sequences, and even show substantial promise for robotics in a variety of manipulation tasks [1, 2, 3]. However, when it comes to using transformers for mobile robots performing long-horizon tasks, we face the challenge representing spatial information in a useful way. other words, we need a fundamental unit of representation an equivalent of "word" or "token" that can handle spatial awareness in a way that is independent of the robot's exact embodiment. We argue this essential for enabling robots to perform manipula- tion tasks in diverse human environments, where they need to be able to generalize to new positions, handle changes in the visual appearance of objects and be robust irrelevant clutter. In this work, Attention Policies (SLAP), that use a point-cloud based tokenization which can scale to number of viewpoints, and has a number of advantages over prior work.
we propose Spatial-
SLAP tokenizes the world into a varying-length stream of multiresolution spatial embeddings, which capture local context based on PointNet++ [4] features. Unlike ViT-style [1], object-centric [5, 3), or static 3D grid features [2], our PointNet++-based [4] tokens capture free-form relations between observed points in space. This means that we can combine multiple camera views from moving camera when making decisions and still process arbitrary-length sequences.
Our approach leverages a powerful skill representation we refer to as "attention- robot poli- cies" [6. 7, 8, 2, operating on an input-space combining language with spatial information. Unlike other methods that directly predict robot motor controls [10, 1]. these techniques predict goal poses in Cartesian space and integrate them with motion planner [6, 8, 2] conditional low-level pol- icy [9] to execute goal-driven motion. This approach requires less data but still has limitations, such as making assumptions about the size and position of the camera in the input scene and long training
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Bring me my bottle
Take the lemon out of the drawer
Figure I: We propose SLAP, which allows us learn skills for mobile manipulators accomplish multi-step tasks given natural language goals. Our system works by training a Lunguage-conditioned interaction prediction module, which will determine which areas of scene should be interacted with, in addition to an action policy which operates on predicted interaction points. This allows us to scale to more complex scenes, while predicting continuous actions.
times [7, 6, 2]. However, these methods fall into a different trap: they make strong assumptions about how big the input scene is [2], where the camera [7, 6]. and generally take a very long time train [7, 6]. meaning that they could not be used quickly teach policies new environment.
SLAP uses hybrid policy architecture. The interaction prediction module determines which parts of the tokenized environment the robot focuses on, and relative action module predicts parameters of continuous motion with respect to the interaction features in the world. SLAP generalizes better to unseen positions and orientations, as well as distractors, while being unrestricted by workspace size and camera placement assumption, using fewer demonstrations and training in roughly day.
We evaluate SLAP two robot platforms. First, Franka Panda comparison to the current state-of-the-art, PerAct, [2], where we demonstrate better performance with 80% success rate on 8 static real-world tasks on held-out scene configurations and a 47.5% success rate tested with out-of-distribution objects. Second, unlike prior work, we move beyond the stationary camera views and robot arms a table-top setting, and demonstrate SLAP on the Hello Robot Stretch RE-2 mobile manipulator with an ego-centric camera and 6-DoF end-effector configuration. In this setting, also include task planning successfully execute natural language task instructions with ten demonstrations over five learned skills and three heuristic skills (Fig. 1). When SLAP is compared to the PerAct baseline for four skills in controlled setting, we see 4x improvement in the task success rate (Table 3).
perform direct skills
Related Work
Attention-Based Policies. Attention-basse policies have been widely studied in prior research and have been found to have superior data efficiency, generalization, and the ability to solve previously unsolvable problems [11, 9, 12, 2, 13]. However, these approaches often rely on strong assump- tions about the robot's workspace, such modeling the entire workspace as a 2D image [12, 6. 8] 3D voxel cube with predetermined scene bounds [2, 9]. This restricts their applicability and may lead issues related to camera positioning, workspace location, and discretization size. Ad-

---

Demonstrations
Roll
Inout
Approach
Interact
Retreat
diswerr
"Close
"Pick uo lonen
1) Predict Interaction Point
2) Predict Actions
from beskes""
Green ling command,
Given lccatoon
observations
"Pick the bottle
Figure 2: Spatial Language Attention Policies (SLAP) learn language-conditinng skills from few demonstrations a wide variety of cluttered scenes. SLAP has two components: an "interaction prediction" module which localizes relevant features in scene, and an "action prediction" module which uses local context to predict an executable action.
ditionally, these works can be seen, at least partly, as applications of object detection systems like Detic [14] or 3DETR [15], but they lack the manipulation component.
Compared to previous works, some recent studies focus unstructured point clouds [11, 16). These approaches demonstrate improved data efficiency and performance compared to traditional behav- ior cloning. For instance, Where2Act [11] and VAT-Mart [16] predict interaction trajectories, while UMPNet [17] supports closed-loop 6DoF trajectories. They share a common framework: a general- izable method predict the interaction location and then predict local motion for the robot.
Training Quickly with Atfention-Based Based Policies. CLIPort [7] and PerAct [2] are attention-based policies similar to Transporter Nets [6]. While fitting our definition of attention -ased policies, they confine their workspace, use a rigid grid-like structure and treat action prediction as dis- crete classification. While still a limited workspace, SPOT [12], demonstrated the usefulness of 2D attention-based policies for fast RL training, including sim-to-real transfer, and Zeng et al. [6] have shown these policies are valuable for certain real-world tabletop tasks like kitting.
Manipulation of Unknown Objects. Manipulation of unknown objects includes segmentation [18, 19], grasping [20, 21], placement [22], and multi-objec rearrangement either from a goal image [23, 24] or from language instructions [25, 26]. These approaches rely, generally, on first segmenting relevant objects out, and then predicting how to grasp them and where to move them using separate purpose-bis built models, including for complex task and motion planning [27].
Language and Robotics. Language is natural and powerful way specify goals for multi-task robot systems. Several recent works [10, 1, 28] use large-language model for task planning to com- bine sequential low-level skills and assume to learn the low-level skills with IL or RL. To realistically handle language task diversity, we need to learn these skills quickly. SLAP is more sample-effiiien than prior II. or RL approaches. PaLM-EE [3], textual and multi-modal tokens are interleaved as inputs the Transformer for handling language and multimodal data generate high-level plans for robotics tasks. Our approach is spatial extension of this strategy.
Language for Low-Level Skills. A number of works have shown how to learn low-level language- conditioned skills, e.g. [7, 2, 1, 29]. Like our work, Mees et al. [29) predicts 6DoF end effector goal positions end-to-end and sequences them with large language models. They predict 2D affordance heatmap and depth from RGB; We do not predict depth, but specifically look robustness and generalization, where theirs is trained from play data in mostly-fixed scenes. Shridhar et al. [2] predict a 3D voxelized grid and show strong real-word performance with relatively few examples, but don't 't look at out-o--domain generalization and are limited coarse voxelization of the world.
3

---

3 Approach
Most manipulation tasks necessarily involve interacting with environment objects [11]. We define an 'atomic skill' as a task that can be specified by an interaction point, and sequence of relative offsets from this interaction point. For example, pick (" mug') ) is an atomic skill as can be defined in terms of an interaction point on the 'mug' and subsequent relative waypoints for approach, grasp, and lift actions. Similarly, open (' drawer') is an atomic skill for which the interaction point is on the drawer handle, and relative waypoints from it can be defined for approach, grasp, and pull. While these examples illustrate the concept, SLAP can handle prediction of variable number of waypoints per skill.
We train two-phase language-condittoned policy Txx;1), which takes visual observation and language command I as inputs and predicts interaction point Pr, well set of relative motions, which offsets from this point, instead of absolute gooddinatesHowwwwe any realistic task given to a home robot by user typically involves more than one atomic skill. Our system breaks down a high-level natural language task description (T) into sequence of atomic skill descriptions {y}} } and uses them to condition the atomic skill motion policies. Our full paradigm as follows:
T + flo,... (n - {my(Z,,,4) 4)
Vj E n, π = (T,, TR), where:
* xr(3j,4)) - pr m 4 pr) (a)
(3D interaction point)
(sequence of actions)
The interaction point Pı is predicted by Interaction Prediction Module лτ, and the continuous component of the action by Relative Action Module The Interaction Prediction Module xy predicts where the robot should attend to; it is specific location in the world, where the robot will be interacting with the object as part of its skill, as shown in Fig. 2. RR predicts relative action sequence with respect to this contact point in the Cartesian space. These actions are then provided as input to a low-level controller to execute the trajectory. These models are trained using labeled expert demonstrations; complete overview of the training process is shown Fig. 4. Overall, the system outputs sequence of end-effector actions
3.1 Scene Representation
The input observation z is structured point-cloud (PCD) in the robot's base-frame, constructed by combining the inputs from a sequence of pre-defined scanning actions. This point cloud is then preprocessed by voxelizing at a Imm resolution to remove duplicate points from overlapping camera views. The pointcloud is then used as input into both æı and TR-
For πı, perform second voxelization, this time 5mm resolution. This creates the down- sampled set of points P, such that the interaction point Pi P. This means πτ has consistently high-dimensional input and action space for robot looking at its environment with a set of N aggregated observations, this can be 5000-8000 input "tokens" representing the scene.
While SLAP discretizes the world similar to prior work [30, 31, 2], our approach ensures fine reso- lution even in larger scenes. We couple this with set-based learning formulation which allows us to attend to fine details in data-efficient manner.
3.2 Interaction Prediction Module
We use our insight about tasks being shaped around an interaction point to make learning more robust and more efficient: instead of predicting the agent's motion directly, we formulate our ny to solely focus on predicting a specific point P1 E P. representing single 5mm voxel that is referred to as the "interaction point". This formulation is akin to learning object affordance and can be thought of similar to prior work like Transporter Nets in 2D [6]. We hypothesize that predicting attention directly on visual features, even for manipulation actions, will make SLAP

---

Additional goadth,
Nx1
PerceiveriO
Top 5%
PCB with cuplicanes removed
(x)
Nx64
Nx64
77x128
Interaction
Unioomll cnceized PCD)
Language: Close the drawer
Languagge
Frosrio
Figure 3: An overview of the architecture of the interaction prediction module. The point cloud is downsampled remove duplicates and encoded using two modified set-abstraction layers. The SA layers generate a local spatial embedding which is concatenated with proprioceptive features in our case, the current gripper state. Both spatial and language features are concatenated and input into PerceiverIO transformer backbone. We then predict an interaction score per spatial feature and the argmax is chosen as the interaction site for command I.
more general. We use a PerceiverlO [32] backbone to process the data, based on prior work on language-conditione real-world policies [2].
We first pass our imput point cloud through two modified set abstraction layers [4] which result in sub-sampled point-cloud with each point's feature capturing the local spatial structure around it at two different resolutions. This encourages the classifier to pay attention local structures rather than specific point that may not be visible in real-world settings. We concatenate the CLIP [33] tokenized natural language command with the encoded point cloud to create an input sequence.
Each point i P the point cloud is assigned a score with respect to task which results in the interaction point for that task, vf = argmax(S(( = 1/ P, Di)), where Dí the set of expert s.y.
demonstrations provided for task The IPM architecture overview provided in Fig. 3. Note we also use binary semantic features from Detic in the Stretch experiment for training SLAP as an additional feature channel apart from the color-channels.
Modified Set Abstraction Layer. The default SA layer as introduced by Qi et al. [4] uses farthest point sampling (FPS) to determine which locations feature vectors are created. FPS ensures that subsampled point-cloud is a good representation of a given scene, without any guarantees about the granularity. However, it's very sensitive to the number points selected in most PoimtNet++-base policies, a fixed number of points are chosen using FPS [4]. However, SLAP must adapt to scenes of varying sizes, possibly with multiple views, and still not miss small details.
We propose an alternative PointNet++ set abstraction layer, which computes embeddings based on the original and an evenly downsampled version of the point-cloud, P. This results in a denser spatial embedding by considering a subset of all points within a certain radius of each-point in the downsampled point-cloud. This downsampled set of points guarantees we can attend even small features, and allows us to predict an interaction point pı from the PointNet++ aggregated features.
3.3 Relative Action Module
The relative action module relies on the interaction point predicted by the classifier and operates on a cropped point cloud, ZR. around this point to predict the actions associated with this sequence. As in the interaction prediction module, the model uses a cascade of modified set abstraction layers as the backbone to compute multi-resolltion encoding feature over the cropped point cloud. We train three multi-head regressors (described further below) over these features to predict the actions for the overall task. Specifically, RR has three heads, one for each component of the relative action space: gripper activation 9, position offset 8p, and orientation Our LSTM-based architecture (details in B.2) can predict skills with variable number of actions (3,4 in our experiments).
5

---

Imarattin paints leylomes. oteanee point douds:
"Sbe diewer"
Collection
iupt Scone
Inteacter Prediction
Action Preiition
Open top duser
Training Testing
Opon
Vick
Figure 4: Illustration of the complete process for training SLAP. Demonstrations are collected and used train the Interaction Prediction module and the Action Prediction Module separately.
Also note that the cropped input point-cloud is not perfectly centered at the ground truth interaction point Pı, but rather with some noise added: p = pi (0( This done to force the action predictor to be robust to sub-optimal interaction point predictions by the interaction predictor module during real-world roll-outs. Thus, for each part of the action sequence, the keyframe position is calculated as: p pı + 6p. When acting, the robot will move to (p. q) via motion planner, and then will send a command to the gripper to set its state to g.
3.4 Training SLAP
To collect data, expert operator guides the robot through trajectory, pressing button to record keyframes representing crucial parts of task. At each keyframe, we record the associated expert action à = (óp, 9. 9). We assume that low-level controllers exist in our case, we use Polymetis [34] for the Franka arm and Hello Robot's controllers for Stretch. Example tasks are shown in Fig. 5.
Interaction Prediction Module. We train क, with cross-entropy loss, predicting the interaction point pr from the downsampled set of coarse voxels P. We additionally apply what we call locality loss (Lioc), per prior work [35]. Conceptually, are from the contact point, both to encourage learning relevant features as well as to aid in ignoring distractors. To achieve this, we define the locality loss as: Lioe = Σ softmax ( kl|2, where f& is the output of the transformer for point I: E P. The softmar turns f& into attention over the points, meaning that Lioe can be interpreted as a weighted average the square distances. Points further from Pı are therefore encouraged to have lower classification scores. Combining our w two losses, we obtain I, CF(P, pr) [PT Liocs where is scaling constant that implicitly defines how much spread to allow in our points.
want to penalize points the further they
the weighted of three different losses. We train (p, 9. =n(2n) with an 1.2 loss over the óp, quaternion distance metric for the loss based on prior work [36] and binary cross-entropy loss for gripper action classification (Sec. A.3).
Relative Action Module. To train nR.
3.5 Task Planning
Consider natural language instruction from user such as 'put away the bottle on table'. We decompose it to sequence of atomic skills as: goto (' bottle') ), pick.up (' bottle') ) goto (' ttale'' ), and place on ('table') ). We procedurally generate natural language and code templates for task families. Refer AC. We use LLaMA [37] models for in-context learn- ing [38, 39] and adapter fine-tuning [40] learn the mapping between natural language task instruc- tions the corresponding sequence of atomic skills.
pettps://gitee www/he httpa/////eooooooboo/ttttteeeee.

---

Soan Objects
Unseen Objects
Figure 5: Examples of tasks executed on a Franka arm through our trained model in a clean setting. We trained numerous tasks (left) and tested both seen and unseen objects (right).
Experiments
We report the success rate of our model for 8 real-world manipulation tasks in Table 2, and compare it against prior baselines trained using the same labeling scheme. Overall, we see an improvement of 1. 1.6z over best comparative baseline, PerAct [2]. Our robotics code was implemented using the HomeRobot framework [41]². We test each model under two different conditions: Seen setting assumption; i.e. those with seen distractor objects and objects placed roughly in the same range of positions and orientations as in the training data in any relative arrangement (inlcuding unseen). Second, we test under unseen setting assumptions; i.e. those with unseen distractor objects and the implicated object placed significantly out of the range of positions and orientations already seen. We run tests per scene setting per skill per model and report the percentage success numbers in Table We compare our model against Perceiver-Actor (PerAct) [2]. We train each model for the same number of training steps and choose the SLAP model based on the best validation loss. For PerAct, we use the last checkpoint, per their testing practices [2].
We also run a per-skill evaluation of SLAP and PerAct on Stretch under the unseen setting assump- tion (see Table. 3). This was accomplished by adding unseen distractor objects to the scene and moving the robot base position within reachable distance of the object. We needed to increase the workspace bounds of PerAct (1.5m cube from 1m) capture the larger observation area for the mobile manipulator, to keep it consistent with SLAP. Note that demonstrations were collected different robot than the one policies were deployed on.
4.1 Longitudinal Task Execution on Stretch
GT
Inferred
We trained a multi-task model for the Stretch robot for five skills using 10 demonstrations each. This model was deployed in an end-to-end system which operates over code-list generated by task-planner (as in Sec. 3.5). We five prompts end-to-end with four eight skills each, using ground truth plans we verify the viability of generating these task plans in $4.2. These experiments are done in the umseen setting with the robot starting anywhere with respect to the objects.
Heuristic
66.0
48.5
Learned
80.0
53.2
Total
68.5
58.5
Table 1: End-to-end per- formance. Learned skills outperform heuristics cept when Detic fails.
For fair evaluation in the low-data regime, we add some structure by specifying an orthogonal viewing direction for objects. Once the agent finds the object of interest, it fires an initial prediction using SLAP to find most promising interaction point. This prediction happens under any dynamic viewing angle of the object (we assume the robot can see the object). This dynamic prediction and pre- preeprogrammed viewing angle is used to approach the object at an orthogonal viewing angle where the model fires an actionable prediction for the full skill execution (details see SC.3). We observe the learned skills suffering significantly when inferred plans are used create language prompts for SLAP. This is due to discrepancies between object descriptions that the LLM generates and Detic's object detection capabilities. SLAP does not get the necessary semantic masks thus its predictions suffer. On the other hand, when semantic features
Phtpps: // httpa//.ww./eeeoooooeeeeee///eee

---

Seen
Unseen
Skill Name
PerAct
SLAP
PerAct
SLAP
Skill Name
PerAct
SLAP
Open bottom drawer
00%
80%
00%
60%
Open Drawer
0%
60%
Open top drawer
60%
80 %
40%
40%
Close Drawer
40%
100%
Close drawer
100%
100 %
40%
40%
Take bottle
0%
80%
Pick lemon from basket
60%
%0%
10%
40%
Pour into bowl 40%
80%
Pick bottle
60%
60
60%
40%
Table 3: SLAP
mobile
Place into the drawer
60%
80 %
40%
60% %
manipulator using a multi-task model across 4 skills, over tries, With semantic predictions added our feature space, we see the model perform better on unseen scenes with new distrac- tors and unseen relative position of the robot with respect to the scene
Place into the basket
40%
100 %
10%
60 %
Place into the bowl
40%
60 %
00%
40 %
Average Success Rate Improvement
50%
80%
27.5%
47.5%
1.6x
1.7x
Table 2: SLAP and PerAct [2] performance on real world Franka manipulation tasks. We evaluate both seen scenes (seen object positions and distractors), but in different arrangements, and unseen scenes with previously-unsee object positions and distractors. SLAP is notably better overall in both conditions.
from Detic are available IPM performance significantly improves even with unseen distractors (80%
against 47.5% in Table 2). We still see failures when relative position is significantly perturbed. Please see $E.1 for ablation analysis our design against PerAct.
4.2 Task planning with in-context learning and fine-tuning LLaMa
Task Corr.
Lat.
Previous work has shown the strength of language mod-
LLaMa
Verb
Noun Acc.
(sec.)
els as zero-shot planners [42], a result strengthened
IC
7B
83
81
76
61
16.4
by improved techniques for "in-context learning" or prompting [43]. To verify that models can produce task plans with the skills we defined, we experiment with both in-context learning (IC) [44] of LLaMA [37] and adapter fine-tuning (FT) [40]. Table 4 presents the mod- els' verb, noun, and combined accuracies. Task Correctness is binary score if the entire prediction was correct, and finally, latency is measured in seconds on single A6000 with 16 GB RAM. 3 High Task Correctness from a small model verifies the compatibility of our skills with LLM task planning.
IC
30B
81
81
76
62
27.6
FT 7B
100
98
99
91
19.5
Table 4: Fine-tuning (FT) outperforms in- context learning (IC) for same latency.
5 Limitations
SLAP has high variance in out-o--distribution situations, resulting in complete failure π fails to correctly identify the context. For æn, multimodal noisy data still poses issues; replacing nn with a policy which can better handle this data, e.g. Diffusion Policies [46]. Overall system has multiple points-of-failure due to heuristic policies, unaligned language and vision models; end-to- end trainable architectures and cross-modal alignment could help.
Conclusion
We propose a new method for learning visual-language policies for decision making in complex environments. SLAP is a novel architecture which combines the structure of point-cloud based input with semantics from language and accompanying demonstrations predict continuous end- effector actions for manipulation tasks. We demonstrate SLAP on two hardware platforms, including an end-to-end evaluation on a mobile manipulator, something not present in prior work. SLAP also outperforms previous state-of-tt of-the-art, PerAct, on both mounted and mobile robot setups.
Adaptor fine-tuning increases the model size by ~6%, which accounts for the additional latency compared IC. We use standard inference libraries results comparable, but not optimized for runtime [45].

---

7 Acknowledgements
We thank Sriram Yenamandra for his help working with the Stretch codebase and running real-robot navigation and grasping. More generally, we thank the HomeRobot team for software support [41].
References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068// 2022.
[2] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209.05451, 2022.
[3] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint
arXiv:2303.03378, 2023.
[4] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space. Advances neural information processing systems, 30, 2017.
[5] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Somet: Spatial object-centric representations for sequential manipulation. In Conference on Robot Learning, pages 148-157. PMLR, 2022.
[6] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin,
D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning, pages 726-747, PMIRG, 2021.
[7] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[8] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arXiv:2210.05711, 2022.
[9] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Roboties and Automation Letters, 7(2):1612-1619, 2022.
[10] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint (xXi:220040019) 2022.
[11] K. Mo, L. J. Guibas, M. Mukadam, Gupta, and S. Tulsiani. WhereZact: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813-6823, 2021.
[12] A. Hundt, B. Killeen, N. Greene, H. Wu, Kwon, C. Paxton, and G. D. Hager. ""Good robot!": Efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE Robotics and Automation Letters, 5(4):6724-673) 6731, 2020.
[13] P.-L. Guhur, S. Chen, R. Garcia, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. arXiv preprint arXiv:2209.04899, 2022.
[14] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
[15] I. Misra, R. Girdhar, and A. Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906- 2917, 2021.

---

[16] R. Wu, Y. Zhao, K. Mo. Z. Guo. Y. Wang. T. Wu, Q. Fan, X. Chen, L. Guibas, and H. Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv: 10:: 14440, 2021.
[17] Z. Xu, Z. He, and Song. Umpnet: Universal manipulation policy network for articulated objects. arXiv preprint 1Xiv:2109.05666, 2021.
[18] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments, IEEE Transactions Robotics, 37(5):1343-1399 2021.
[19] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. In Conference on Robot Learning, pages 461-470. PMLR, 2021.
[20] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation clutter. In 2020 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 6232-6238. IEEE, 2020.
[21] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspne Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11488-3344,, IEEE, 2021.
[22] C. Paxton, C. Xie, T. Hermans, and D. Fox. Predicting stable configurations for semantic placement of novel objects. In Conference on Robot Learning, pages 806-815. PMLR, 2022.
[23] A. H. Qureshi, A. Mousavian, C. Paxton, M. C. Yip, and Fox. Nerp: Neural rearrangement planning for unknown objects. arXiv preprint arXiv:2106.01352, 2021.
[24] A. Goyal, A. Mousavian, C. Paxton, Y.W. Chao, B. Okom, J. Deng, and D. Fox. Ifor: Iter- ative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14787-14797, 2022.
[25] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for snaguage-guidedd semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 5322-6322 IEEE, 2022.
[26] W. Liu, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. arXiv preprint arXiv:2211.0466 2022.
[27] A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano- Pérez, and C. R. Garrett. Long-horizon manip- ulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[28] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions feasible plans, arXiv preprint 2rXiv:2303.12/55: 2023.
[29] O. Mees, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. arXiv preprint arXiv:22 2022.
[30] V. Blukis, Paxton, D. Fox, A. Garg, and Y. Artzi A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning, pages 706-717. PMLR, 2022.
[31] S. Y. Min, D. S. Chaplot, P. Ravikumar, Y. Bisk, and R. Salakhutdinov. Film: Following instructions language with modular methods. arXiv preprint arXiv:21 2021.
[32] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:1107.14795 2021.
10

---

[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 7748-8763. PMLR, 2021.
[34] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and E. Meier. Polymetis. https:// Facebookresearchh github. ggithub.i//ffaali.....ee wwl.faarrrooolyyttteeoo 2021.
[35] S. Powers, A. Gupta, and C. Paxton. Evaluating continual learning home robot, 2023.
[36] C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Foxl. Prospection: Interpretable plans from language by predicting the future. In 2019 International Conference on Robotics and Automation (ICRA), pages 6942-6948. IEEE, 2019.
[37] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
[38] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan,
R. Child, A. Ramesh, D. M. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler,
M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, McCandlish, A. Radford, I. Sutskever, and Amodei. Language models are few-shot learners. ArXiv, 2020.
[39] E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023.
[40] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021.
[41] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner, et al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306. 11565, 2023.
[42] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. arXiv preprint arXi::2201.072077 2022.
[43] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-Context Retrieval- Augmmeened Language Models, Jan. 2023. abs/2300 00083, 3. arXiv:2302.00083 [es].
[44] J. Wei. J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou, and T. Ma. Larger language models do in-context learning differently, Mar. 2023. URL http: arxiv. org. abs / 2303. 03846. arXiv:2303.03846 [cs].
[45] J. Fermandez, J. Kahn, C. Na, Y. Bisk, and E. Strubell. The Framework Tax: Disparities Between Inference Efficiency in Research and Deployment. ArXiv, 2023. URL https: / //arxiv. wrtp:/s///////2222// 06117.
[46] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[47] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A com- pact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 3150-3157. IEEE, 2022.
[48] B. Akgun, M. Cakmak, J. W. Yoo, and A. Thomaz. Trajectories and keyframes for kines- thetic teaching: A numan--obot interaction perspective. In Proceedings of the seventh annual ACM/ΣЕЕ international conference Human-Robor Interaction, pages 391-398, 2012.
11

---

[49] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv preprint arXiv:2210.13702, 2022.
[50] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprini arXiv: / 703.093 12, 2017.
[51] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022.
[52] B. Bolte, A. Wang, Yang, M. Mukadam, M. Kalakrishnan, and Paxton, Usa-net: Unified semantic and affordance representations for robot memory. arXiv preprint arXiv:2304. 12164, 2023.
12

---

Appendix
Table of Contents
A Training
13
A.1 Data Collection and Annotation
13
A.2 Data Processing
14
A.3 Action Prediction Losses
14
A.4 Skill Weighting
15
A.5 Training for PerAct and SLAP
15
Relative Action Module
MLP Implementation
LSTM-based Implementation
High-level Task Planning and Execution
Dataset
C.2 Models
................................
C.3 System Architecture and Plan Exccutionr
D Experimental Setup for Skills
18
D.1 In vs. Out Of Distribution
18
D.2 Skill Definitions and Success Conditions
18
D.3 Language Annotations
22
D.4 Out of distribution Results from SLAP
22
D.5 Motion Planning Failures
23
E
Additional Analysis
23
E.1 Ablations
23
E.2 Visualizing the Learned Attention
23
E.3 Language Generalization
25
F Additional Related Work
25
A
Training
Below is expanded information on our training, algorithm, and data processing to improve repro-
ducibility.
A.1 Data Collection and Annotation
When collecting an episode with the Franka arm, we first sean the scene with pre-defined list of scanning positions to collect aggregated x. In our case, make assumption what how many these are, how large the resulting input point cloud is. With the Hello Robot Stretch [47], we collect data based on exactly where the robot is looking.
Then, we collect demonstration data using kinesthetic teaching for the Franka arm (demonstrator physically moves the robot) and via controller teleoperation for the Stretch robot. The demonstrator moves the arm through the trajectory associated with each task, explicitly recording the keyframes [48] associated with action execution. These represent the salient moments within task the bottlenecks in the tasks' state space, which can be connected by low-level controller.
13

---

A.2 Data Processing
We execute each individual skill open-loop based on an initial observation. We use data augmenta- tion to make sure even with relatively few examples, we still see good generalization performance.
Data Augmentation. Prior work in RGB-D perception for robotic manipulation (e.g. [18, 491) has extensively used variety of data augmentation tricks to improve real-world performance. In this work, we use three different data augmentation techniques to randomize the input scene x used to train (z l):
Elliptical dropout: Random ellipses are dropped out from the depth channel to emulate occlusions and random noise, as per prior work [50, 18]. Number of ellipses are sampled from Poisson distribution with mean of 10.
Multiplicative Noise: Again per prior work [50, 18, 22], we add multiplicative noise from gamma process to the depth channel.
Additive Noise: Gaussian process noise is added to the points in the point-cloud. Param- eters for the Gaussian distribution are sampled uniformly from given ranges. This is to emulate the natural frame-to-frame point-cloud noise that occurs in the real-world.
Rotational Randomization: Similar to prior work [2, 22, 25], we rotate our entire scene around the z-axis within a range of +: degrees to help force the model to learn rotational invariance.
Random cropping: with 0.75, we randomly crop to a radius around ýr + ô, where is a random translation sampled from a Gaussian distribution. The radius to crop is randomly sampled in (1,2) 2) meters.
Data Augmentation for Te- We crop the relational input zn C around the ground-truth pı, using fixed radius 0. We implement an additional augmentation for learning our action model. Since ps is chosen from the discretized set of downsampled points P, we might in principle be limited to this granularity of response. Instead, we randomly shift both py and the positional action 6p by some miiformyy-saplle offset dr R2, with up to 0.025m of noise. This lets nH adapt to interaction prediction errors of up to several centimeters.
A.3 Action Prediction Losses
Following [36] for the orientation, compute the angle between two quaternions @ as: = cos -1 (2 ?)
We can remove the cosine component and use it as a squared distance metric between 0 and 1. We then compute the position and orientation loss
LR 1p116p 6pll + 11 (Q.4))
(2)
where A and X, are weights on the positional and orientation components of the loss, set to 1 and 1e respectively.
Predicting gripper action is a classification problem trained with a cross-entropy loss. For input we use the task's language description embedding and proprioceptive information about the robot as input, i.e. = (1, 9acts 9w., ts) where Jast. is if gripper is closed and 0 otherwise, Ine is the distance between fingers of the gripper and ts is the time-step. The gripper action loss is then:
Ly Ag (E)..9)
where Ag is the weight on cross-entropy loss set to 0.0001. The batch-size is set at for this implementation.
We train πy and गं separately for n 85 epochs. At each epoch, we compare validation per- formance to the current best if validation did not improve, we reset performance the last best model.
14

---

8p
SA Layer
MLP RetU
MLP
BDDF keytrame (for training)
Language Progrio Embeddings
Figure 6: Regression model architecture with separate heads for each output. The point-cloud is cropped around the interaction point with some perturbation and passed to cascade of set ab- straction layers. Encoded spatial features are then concatenated with language and proprioception embeddings to predict position offiett of action from interaction point, absolute orientation and grip- per action boolean.
A.4 Skill Weighting
In Stretch experiments, we used a wide range of skills with different error tolerances and corre- sponding variances. As a result, we needed to use two different sets of weights for learning position, orientation and gripper targets. A more forgiving weight-set for noisier tasks like pouring and han- dover, and a tighter weight-set for task with more consistent trajectories like opening and closing the drawer. These weights were empirically determined but can be further optimized via hyper- parameter tuning methods.
A.5 Training for PerAct and SLAP
In all our experiments we ensure PerAct and SLAP are trained on the same data volume. Data volume is defined total number of augmented samples per collected sample. Note this results in different number of training steps per model. This is due the LSTM-based model updating once per trajectory while PerAct updates once per sample in trajectory.
B Relative Action Module
In our work, the Relative Action Module nn assumed to be some local policy which predicts end-effector poses. In our case, we implement two different versions of this policy, one which was used on the static Franka manipulator and one which was implemented on the Stretch. In both cases:
The policy predicts end effector pose relative the predicted interaction point from m
The policy is conditioned on a local crop around this interaction point.
B.1
MLP Implementation
Fig. gives an overview of the MLP version of the regression model. The model takes in the cropped point cloud (augmented during training as discussed Sec. A.2. We saw that injecting random noise to the interaction point during training allowed the policy to, at test time, recover from failures (because predicted interaction point near the correct area, instead of at the correct position).
B.2 LSTM-based Implementation
We observed the MLP architecture suffer when positional distribution of actions varied widely with respect the interaction point position across tasks due to the multi multi-modality this introduced. Thus we modified the above model by adding an LSTM to condition each task and action with a hidden-
15

---

Poitblct+
+8p
Language embedding
Regression Head
LSTM
MLP
8
Increments
Time one hot embedding
Proprtocoption emibedding
Figure 7: LSTM-based regression model architecture based on the regression head and PointNet++ embeddings introduced in Fig. 6. LSTM-based architecture shows higher stability in learning action distribution with wider distribution due to the conditioning effect.
state. This model exhibited better performance in learning wider action distribution, based on our initial experiments with the outlined Stretch tasks.
C
High-level Task Planning and Execution
C.1
Dataset
We procedurally generated a dataset consisting of more than 500k tuples of language instructions and the corresponding sequence of atomic skills. The data is created for 16 task families and can be extended further in the future. For each task family, 10% samples are held-out for evaluation. The distribution samples across task families shown in 5.
For each task family, define corresponding template containing the sequence of atomic skills. This means that the sequence of atomic skill "verbs" is the same among the samples of task family. Each sample within a task family differs in terms of language instructions and object(s) of interaction.
To populate these templates and generate the data, we create list of more than 150 movable objects kitchen objects, surfaces like table. kitchen counter and articulated objects like drawer, cabinets. For pour skill, we create list of "spillable" items such cup of coffee, or bowl of jelly beans. Similarly, for wipe skill, we have a list of items to wipe with such as sponge, or brush.
C.2 Models
Table 4 shows that in-context learning with 5 examples from the same task family achieves close to 76% accuracy. There is no training involved with in-context learning, so it can't overfit. For finetuning, our evaluation consists of a held-out dataset with unseen variations in the phrasing of the language instruction and/or object(s) of interaction. The goal of the fine-tuned model to demonstrate that it is possible to achieve improved task planning performance with lower latency than in-context learning with larger models. We do not evaluate the generalization of the fine-tuned model with unseen task families in this work. The remaining results in this work the fine-tuned model for task planning.
16

---

Table 5: Data Distribution of procedurally generated samples across task families.
Task Family
Total number of samples
Bring X From Y Surface To Pour In Z Then Place On W Surface
35640
Bring X From Y Articulated To Wipe Z
612
Move X From Y Surface To Z Surface
16092
Move X From Y Surface To Z. Articulated
301400
Take X From Human To Z Articulated
19300
Bring X From Y Surface To Human Bring X From Y Surface To Wipe Z Take X From Human To Pour In Z.
5933
3168
2184
Take From Human To Z Surface
8050
Take X From Human To Wipe Z.
1458
Move X From Y Articulated To Z. Articulated
97923
Bring X From Y Articulated To Human
13617
Bring X From Surface To Pour Z
3960
Bring X From Y Articulated To Pour In Z. Move X From Y Articulated To Z Surface
8712
50060
Take X From Human To Pour In Z And Place On Y Surface
19656
587765
C.3 System Architecture and Plan Executionr
We use SLAP with three heuristic policies: SEARCHFOROOJEEC (Tzeareh). PICKUPORJECT (Teick) and PLACEON (Tplace). Taearch uses Detic, frontier-based exploration policy and lan- guage query to explore the map until object described by the language query found in the view of the robot. This is also one of the primary points of failure when LLM is integrated into the pipeline. LLM expects Detic be able handle any freeform query of the object (for example, "detect open drawer" is typical output from the LLM however always fails Detic has no notion of an open drawer).
pick is heuristic picking policy which always grabs given object from the top given its mask from Detic and depth from camera. This is generally robust policy but fails when contextual, task- oriented grasps are required in task. For example consider a bottle placed within cabinet, the only pick policy which will solve this scene is one where gripper grasps bottle laterally and not from the top. place places whatever object is in robot's gripper on the surface previously detected by Detic. Another point of failure, since up-close Detic is not able to detect objects like table, counter surfaces, etc.
SLAP is used after the object successfully detected by Detic, given language query, and is in the robot's field of view. Note the robot at this point can be any unconstrained orientation and position with respect to the object, the only condition sufficed here is that the object is within sight. We use SLAP's interaction prediction module to estimate the affordance over this object with Detic's mask as an additional feature, and predict an interaction point on the object. The robot then uses hand- engineered standoff orientations to move to a head-on position with respect to the object, where we use the full SLAP system to predict the action trajectory.
The standoff orientations used that SLAP can be tested fairly within reasonable bounds of the state distribution it was trained for. Training data for all our skills is recorded from very narrow range of robot positioning with respect to the objects (see Appendix D. This means the action prediction module's s sense action orientation is not robust to huge rotational variations over object's position around the robot's egocentric frame. Interaction prediction module on the other hand is very robust as does not need consider directionality, just the local structure of object and related affordance. See Fig. for details on how regions are assigned specific objects, and related stand-off orientations provided to the robot so that it is always facing the object head-on after estimating the interaction point. Given predicted interaction point ps, pre-determined standoff
17

---

Table
Counter
Chair
Table
Human for handover
Bowl and bottle
Drawer
Figure 8: Diagram showing where immovable objects were placed in the environment (note that chair can move anywhere within the blue region). The three colored regions signify the placement assignment for different artifacts involved in the tasks. The circular symbol signifies the robot's pre-determined orientation where the beak represents where the robot will be facing. Position for robot's placement is determined based on predicted p;
orientation vector VEC V and pre-determined 3D standoff distance vector dist diendtond fs the robot moves an orientation VeCstandof and position:
p08next Pi disl standoff (-1 * vCCstandoff)
The orientation vector is of the form (1,0, 10,1,0) in this evaluation.
D Experimental Setup for Skills
Here we refer to atomic skills learned by SLAP as simple tasks "tasks". This allows to discuss corresponding "actions" that are defined in terms of the relative offset from the interaction point.
D.1 In vs. Out Of Distribution
We used number of objects for manipulation experiments, which included both in- and out- of-distribution objects (see Fig. 9 and Fig. 10). One goal of SLAP is to show that our methods generalize much better than others to different types of scenes and different levels of clutter.
We also randomized the objects with seen and unseen clutter around them, as well as placed them unseen environments. Fig. 12 and Fig. 13 show the extent of variation captured in test scenes against training scenes for Stretch experiments. Fig. 11 shows range of scenes from Franka evaluations.
D.2 Skill Definitions and Success Conditions
Every real-world task scene had sub- sub-sample of all within-distributtin objects. Following describes skills and their success condition for Franka experiments:
18

---

In-distribution
Out of distribution
Figure 9: Within distribution objects used at training time and out-of-distribution objects introduced during testing in our experiments.
Unseen Distractors
Seen Objects
Figure 10: Seen objects and unseen distractors used in longitudinal experiments with Stretch.
Close drawer
Close drawee
Open bottom
Pick lemon
drawer
from basket
Figure 11: Snapshot test scenes from Franka evaluations show the range of variation time
1. Open the top drawer
Task: Grab the small loop and pull the drawer open. Drawer configuration within training data face-first with slight orientation changes
Action labeling: Approach the loop, grab the loop, pull the drawer out
Success metric: When the drawer open by 50% or more
2. Open the bottom drawer
Task: Grab the cylindrical handle and pull the drawer open. Drawer configuration within training data face-first with slight orientation changes. Note significantly different grasp is required than for top drawer
Action labeling: Approach the handle, grab it, pull the drawer out
Success metric: When the drawer open by 50% or more
3. Close the drawer
19

---

SCLAI
Pour into bowl task: Showing variability and out-of domain distribution covered by test against training samples. Top row: Training scenes. Note that the bowl was always placed somewhere on this particular region (right of sink) on the counter. Bottom row: Test scenes. Note bowl surrounded by unseen clutter, placed novel unseen environment and relatively different positioning with respect to the camera and robot. Green boundary signifies successful episode, red a failed episode.
Figure 12:
Figure 13: Close drawer task: Showing variability and out-of-domain distribution covered by test against training samples. Top row: Training scenes. Note absence of any clutter and narrow range of relative positioning of drawer with respect to the camera and robot. Bottom row: Test scenes. Note presence of objects used in other tests in the same frame. Green boundary signifies successful episode, red failed episode.
Task: This task is unqualified, i.e. the instructor does say whether to close the top or bottom drawer instead the agent must determine which drawer needs closing from its state and close it. Align the gripper with the front of whichever drawer is open and push it closed. The training set always has only one of the drawers open, in front-facing configuration with small orientation changes
Action labeling: Approach drawer from the front, make contact, push until closed
Success metric: When the drawer is closed to within 10% of its limit when arm is maximally stretched out to its limit (when the drawer is kept far back)
20

---

4. Place inside the drawer
Task: Approach empty spot inside the drawer and place whatever is in hand inside
it
Action labeling: Top-down approach pose on top of the drawer, move make contact with the surface and release the object, move up for retreat
Success metric: Object should be inside the drawer
5. Pick lemon from the basket
Task: Reach into the basket where lemon is placed and pick up the lemon
Action labeling:
Success metric: Lemon should be in robot's gripper
Considerations: Since the roll-out is open-loop and lemon is spherical in nature, trial was assigned success if the lemon rolled out of hand upon contact after the 2nd action. This was done consistently for both PerAct and SLAP.
6. Place in the bowl
Task: Place whatever is in robot's hand into the bowl receptacle
Action labeling: Approach action top of the bowl, interaction action inside the bowl with gripper open, retreat action top of the bowl
Success metric: The object in hand should be inside the bowl now
7. Place in the basket
Task: Place the object in robot's hand into the basket
Action labeling: Approach action on top of the free space in basket, interaction action inside the basket with gripper open, retreat action top of the basket
Success metric: The object is inside the basket
8. Pick up the bottle
Task: Pick up the bottle from the table
Action laheling: Approach pose in front of the robot with open gripper, grasp pose with gripper enclosing the bottle and gripper closed, retreat action at some height from previous action with grippers closed
Success metric: The bottle should in robot's gripper off the table
Notably, success for opening drawers is if the drawer is 50% open after execution; this is because sometimes the drawer is too close to the robot's base for to open fully with fixed-base Franka arm.
Following describes the skills undertaken in Stretch experiments and their success conditions:
Open the drawer
Task: Grab the handle of the drawer and pull the drawer open
Action labeling: Approach the handle, grab the handle, pull the drawer and open
grasp
Success metric: When the drawer open by 100%
2. Close the drawer
Task: Align with the surface of drawer's face and push the drawer close
Success metric: When the drawer closed within 10% of fully-closed configuration
3. Pour into bowl
Task: Skill starts with a cup filled with candies already in robot's gripper. Align cup with the bowl and turn the cup in pouring motion
Success metric: When 2 50% of candies are in the bowl
21

---

Place in drawer
Pick up bottle
Open top drawer
Figure 14: Examples of out of distribution predictions made by æ. We show that it is able to handle heavy clutter around the implicated object to predict interaction points. Note that the prediction for bottle picking is sub-optimal in this example.
4. Take bottle
Task: Approach bottle with gripper orientation in the right configuration, grasp the bottle, lift the bottle and retract keeping the grasp
The bottle is in robot's gripper in stable configuration the end of execution
Success metric:
5. Handover to person
Task: Approach hand of the person with object in hand, align with hand's surface and release the object, finally retracting the gripper back
Success metric: The object is in human's 's hand at the end of execution
Note that we count the success for pouring 2 50% of the candies because we are comparing this task to pouring a liquid. Liquid would pour out completely in the intended final configuration due to different dynamies.
D.3 Language Annotations
In the following, include the list of language annotations used in our experiments. Table 6 shows the language that was used to train the model: we'r re able show some robustness to different language expressions. We performed set of experiments on held-out, diitt--iiistiittionn language despite this not being the focus of our work; this test language is shown in Table 7.
D.4 Out of distribution Results from SLAP
We show more results for the attention point predicted by कy in Fig. 14. For the placement task, the agent has never seen a heavily cluttered drawer inside before, but able to find flat space that indicates placement affordance. For the bottle picking task, this sample has lemon right next the bottle which changes the shape of the point-cloud around the bottle. We see that πy able to find an interaction point albeit with placement different from expert and lower down on the bottle. Similarly, the open-top drawer sample has more heavy clutter on and around the drawer to test robustness.
Fig. shows the prediction and generated trajectory for picking up previously unseen bottle. Note that while the models are able to detect the out of distribution bottle, the trajectory actually fails because the bottle is much wider and requires manipulator domain, observe SLAP performed better than vanilla PerAct every count. Our
accuracy in grasping. For the mobile
22

---

time
Figure 15: A generalization example of success for our model. The new bottle has same shape as the within distribution bottle but much taller, different in color and wider in girth. The model able to predict the interaction site and feasible trajectory around it. We note though the execution of this trajectory was failure; due wider girth of the bottle the predicted grasp was not accurate enough to enclose the object.
hypothesis that SLAPPs better performance due to the addition of semantic features, more efficient training, and higher resolution due a non-grid point-cloud representation.
D.5 Motion Planning Failures
Our evaluation system has a simple motion planner which not collision aware as result we saw number of task failures for both the models. However, note that the frequency of task failures due to motion planning problems was higher for PerAct. We think it is because PerAct predicts each action of the same task as an entirely separate prediction trial, while SLAP forces continuity on the relative motions for the same task by centering them around the interaction point (see Fig. 15), That said, we also note with a collision-aware motion planner PerAct may not run into such issues as seen during our evaluations. However the planner setting and conditions were same across both models in these evaluations. The authors note their own paper their heavy reliance on good motion planning solutions [2].
E
Additional Analysis
E.1
Ablations
Hybrid vs Monolithic Architecture (Table-top). train SLAP and PerAct such that they observe same amount of data. SLAP outperforms PerAct on six of eight tasks when tested in in-istribution settings and five of eight tasks in out-of-distribution settings Franka. PerAct performs equally well model for two of eight tasks on our in-distribution scenes. Similarly, for our "hard" generalization scenes, PerAct performed equally well in two cases, and actually outperformed SLAP when picking bottle. Under similar experimentation conditions, SLAP outperforms PerAct in all four tasks in cluttered scenes for the mobile manipulator environment on Stretch. In failure cases, RR predicted the correct trajectory, but not with respect to the right part the object.
Unseen Scene Generalization. We drop in the success rate for both PerAct and SLAP when tested out-of-distribution settings. PerAct would often predict the correct approach actions, but then it would fail to grasp accurately. With SLAP, however, we saw that was predicted fairly accurately, but the regressor would fail for out-of-distribution object placements specifically because of bad orientation prediction. When क, failed, it was because the position and orientation of the target object was dramatically different, and unseen distractors confused it. We see better results for SLAP under Stretch setting due the addition of semantic features from Detic.
E.2 Visualizing the Learned Attention
Since we use scores choose the final interaction point, our classifier model is naturally inter- pretable, being able to highlight points of interest in scene. We visualize this attention by selecting the points with the highest 5% of interaction score given language command l.
23

---

Place drawer
Place in bowl
time
Figure 16: Examples of failure cases for baseline, PerAct, for the "place in drawer" and "place in bowl" tasks. In the top example, the gripper moved from drawer's side towards inside, instead of from the top demonstrated by expert. The gripper ends up pushing off the drawer the side as our motion-planner is not collision-aware. Note that SLAA does not exhibit such behaviors as л implicitly learns the collision constraints present in demonstrated data. In the bottom example, each action prediction is disjointed from previous and semantically wrong.
An example cutt-f-distributtinn SLAP failure where an extreme sideways configuration of the drawer is paired with unseen distractors for the "open top drawer" skill. Note that the attention mask ranks other distractors in its top 5% and fails to choose an optimal interaction point.
Figure 17:
24

---

E.3
Language Generalization
By using pretrained CLIP language embeddings to learn our spatial attention module ry, our model can generalize to unseen language to some extent. We tested this by running an experiment where we evaluate performance on in-distribution scene settings, prompted by a held-out list of language ex- pressions. We choose three representative tasks for this experiment and run 10 tests with 2 different language phrasings.
F
Additional Related Work
We note some other related work related to the larger language-conditioned mobile manipulation domain that SLAP is situated in, but not as directly relevant.
Vision-Language Navigation. Similar representations are often used to predict subgoals for explo- ration in vision-language navigation [30, 31, 8, 51, 52]. HL.MM builds voxel map [30], whereas FiLM builds 2D representation and learns to predict where to go next [31]. VLMaps proposes an object-centric solution, creating of candidate objects to move [8], while CLIP-Fields learns an implicit representation which can be used to make predictions about point attentions in responds to language queries [51], but does not look at manipulation. Similarly, USA-Net [52] generates a 3D representation with a lot of semantic features including affordances like collision. Such representation can naturally incorporated for collision-awe aware action plans at prediction time.
25

---

Task Name
Training Annotations
pick up the bottle
pick up bottle from the table
pick bottle
grab my water bottle pick the lemon from inside the white basket grab lemon from the basket on the table hand me a lemon from that white basket place the lemon from your gripper into the bowl add the lemon to bowl on the table put the lemon in the bowl place the object in your hand into the basket put the object into the white basket place the thing into the basket the table open bottom drawer of shelf on the table pull the second drawer out open the lowest drawer close the drawers push the drawer close the drawer with your gripper
pick lemon
place lemon in bowl
place in the basket
open bottom drawer
close the drawer
open top drawer
open the top drawer of the shelf on the table pull the first drawer out open the highest drawer put it into the drawer place the object into the open drawer add the object the drawer
place in the drawer
Table 6: Examples of language used to train the model.
Task Name
Held-Out Test Annotations
Pick up the bottle
Grab the bottle from the table Pick up the water bottle Pull top drawer out Open the first drawer Add the drawer
Open the top drawer
Place into the drawer
Put inside the drawer
Table 7: Examples of
language annotations used for evaluation
26


Reviews:
[{"review_id": "hxFRGxOP-S8", "reviewer": "Reviewer_iD6X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Spatial-Language Attention Policies (SLAP), a method for language-guided mobile manipulation. SLAP has two components (1) an interaction prediction module that localizes interaction points in a point cloud (object affordance), and (2) a relative action module that predicts relative action sequence w.r.t. the contact points in the cartesian space. The models are trained with labeled expert demonstrations.\nThe approach uses atomic skills to manipulate objects, defined by the interaction point and subsequent relative waypoints. A language-conditioned policy predicts the atomic skills parametrization given input observations.\nThe authors evaluated the method with 5 test runs per scene setting per skill per model, comparing it to Perceiver-Actor (PerAct) method and demonstrating superior performance.", "summary_of_recommendation": "The paper proposes an interesting method for language-conditioned policies operating on point clouds.\nThe current experimental sections need some revisions to support the paper's claims.\nWe vote for rejecting the current version of the paper, but we will support it for acceptance if the authors can address our comments.\n\n**Update after rebuttal:**\n\nThe authors addressed most of our comments reasonably; we raised our recommendation to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see the weaknesses section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The presentation of the method is clear and easy to follow.\n2. Good coverage of related work.\n3. The design of the two-phase language-conditioned policy to predict the affordance and the waypoints is a viable option.\n4. Using demonstrations from a different robot than the robot used for deployment suggests good generalization abilities. \n5. Real-world experiments on two robots, Franka and Stretch.\n6. Honest discussion of the limitations.\n\nWeaknesses:\n1. In lines 92-94, the authors criticize PerAct for not looking at out-of-domain generalization, but the current evaluation section does not look at it.'\n2. It would be interesting to compare the method to 2D attention-based policies, e.g., SPOT.\n3. The method constructs an input point cloud from pre-defined scanning actions (lines 117-118) followed by voxelization. How much time is needed to build this scene representation? Is it required for each atomic skill? How much does this affect method deployment?\n4. What are the benefits of the proposed set abstraction layer?\n5. The evaluation is reported for 5 tests per scene setting per skill per model. The number of scenes is not reported, and it seems from Table 2 that there were only 5 runs (aside from the 10% for PerAct).\n6. It would be interesting to show the results with the last checkpoint of SLAP. It will be a fair comparison, and it will demonstrate robustness. While using the model with the highest validation loss is equivalent to cherry-picking.\n7. The robot is spawned randomly in the environment (line 222). Are the objects randomized during the experiments?\n8. Section 4.2. how big is the dataset? What are the evaluation settings? Table (Figure 5) suggests potential overfitting.\n9. The ablation study shows that SLAP can not generalize well for out-of-distribution settings which contradicts the claims in the introductions.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "f6aO96Ue5Gd", "reviewer": "Reviewer_zJZk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes Spatial-Language Attention Policies (SLAP), which leverages three-dimensional tokens (features extracted from point clouds) as input presentation to train a single multi-task, language conditioned action prediction policy. The method is comprised of two primary modules: Interaction Prediction Module, which predicts an interaction point given a language instruction and input point clouds, and Relative Aciton Module, which predicts a desired end-effector pose relative to the interaction point and the desired gripper state given the interaction point, cropped point cloud and language instruction. The two modules are trained separately with imitation learning. The whole system demonstrates promising results in table-top pick-and-place tasks (including opening/closing articulated objects) and mobile manipulation pick-and-place tasks, and outperformed previous baselines. ", "summary_of_recommendation": "Based on the review above, I recommend Weak Accept. I am open to changing my mind based on the rebuttal. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Add missing citations to related work such as ReLMoGen and Spatial Action Maps for Mobile Manipulation as mentioned above.\n- Add more error analysis to share more insights about your experiments\n- In Fig 3, what is the top 5% prediction used for? Just for visualization? \n- How are the semantic features from Detic being used exactly? It’s not very well-explained in the paper. Why is it only used in the Stretch experiments, but not the table top experiments? Can we have an ablation study for models with and without semantic features? \n- What would happen if the predicted action (i.e. delta pose) is out of reach for the robot or leads to collision with the environment? Would the episode immediately terminate and labeled as a failure? Is LSTM-based Relative Aciton Module closed-loop, and how? Does it recover from its mistakes? \n- How many demonstrations are collected on Franka and Stretch robot respectively?\n- It’s mentioned that the two modules are trained separately. But isn’t the second module (relative action module) depending on the first module (interaction prediction module) for cropping the input point cloud? More training procedure details are appreciated. \n- What exactly are the hand-specified orthogonal viewing directions for in the Stretch robot? Why is it needed? What if we take them out?\n- How is navigation handled for objects in FoV and outside FoV? \n- Since the relative action module is aggregating global features from SA layers and outputting a single desired 6 dof delta pose, it’s a bit doubtful how fine-grained/dextrous the action can be. Is it really at the “mm” level accuracy already? Have the authors thought of some sort of voting mechanism for neighboring points to collectively vote for an action? See VoteNet [3]\n\n[3] Qi, Charles R., et al. \"Deep hough voting for 3d object detection in point clouds.\" proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a pretty exciting and novel robot learning system that performs well in around 10 real-world robotics tasks and is able to handle unseen distractor objects and novel object configurations to some degree during test time.\n\nThis work has a few noteworthy strengths. First of all, the algorithmic design seems novel and interesting. Although spatial action map / attention-based policies have been studied before, this paper is one of its first kind to directly operate on 3D point cloud with modified set abstraction layers adapted from PointNet++ and also take advantage of transformers’ ability to take in varying-length stream of multi-resolution spatial embeddings. \n\nSecondly, by decomposing the policy into two stages (“finding interaction point”, and “generating relative motion around that point”), the SLAP demonstrates better performance and sample efficiency than previous methods on real world robotics tasks. The experimental evaluation (per skill and multiple skill setting) is relatively thorough. The proposed model is able to generalize to unseen objects and object poses, which is quite impressive. \n\nThirdly, the paper is relatively well-written. Different components of the system are well explained (the two major modules, how LLM is used, how the point clouds are processed, etc). The figures are pretty illustrative, as well as the videos in the supplementary material. \n\nThis work, however, is not without limitations. Overall, although the paper is relatively easy to follow, it seems to be missing some important information (more details can be found in the section “Questions For Rebuttal” below). For example, the paper should probably include more results analysis, especially the failure mode analysis (e.g. which of the two modules fail more often, in seen and unseen settings, and why? What percentage of the failures are due to motion planning failures? Why is the motion planner not collision aware). The paper also misses citations to other attention-based policies, such as ReLMoGen[1] and Spatial Action Maps for Mobile Manipulation[2]. ReLMoGen is particularly relevant because it also predicts a 3D interaction point (a pixel mapped to 3D space from depth sensing) and a local nonprehensile interaction from that point (predicting desired offset), although it’s trained with RL rather than IL. Lastly, the paper should probably tone down its claim about “mobile manipulation”. One of the primary challenges of mobile manipulation, compared to tabletop manipulation, is that 1) the scene is only partially observable (i.e. the desired interaction point might be outside the current FoV), 2) whole body motion (e.g. the robot might need to move the arm and the base holistically when opening a fridge door in a narrow kitchen). This work tackles neither of these two cases, and it’s left unexplained how the robot navigates to objects (those in FoV and outside FoV). \n\nNonetheless, this work seems to be fairly promising. With proper revision, this work can be very impactful for the skill learning and the overall robot learning community\n\n[1] Xia, Fei, et al. \"Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\n[2] Wu, Jimmy, et al. \"Spatial action maps for mobile manipulation.\" arXiv preprint arXiv:2004.09141 (2020).\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "BzwO8pxd69", "reviewer": "Reviewer_91vF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a language-conditioned policy for mobile and table-top manipulation tasks. The proposed policy is based on spatial attention on point clouds and can be decomposed into two stages: i) an interaction point is predicted based on sub-sampled point cloud, and ii) relative action is predicted around the predicted interaction point. The proposed method is shown to perform well in real-world manipulation tasks compared to a competitive baseline PerAct. \n\n ", "summary_of_recommendation": "Good method design with a bit insufficient experiments. Could be improved with rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. More experiments. Maybe add additional baselines like pure BC, multi-task learning setting; more evaluations to improve statistical significance.\n2. More explanations of the atomic skill design.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the paper presents an interesting and well-engineered system for manipulation tasks. The overall method design makes sense but with some small minor limitations on the experiment design.\n\n1.\tIn Table 2, the number of evaluations per task is only 5, which may cause large statistical variations. One success and failure could lead to 20% of the difference. Considering the performance difference for SLAP and PerAct is only 30%, having only 5 trials per tasks is clearly not significant for a thorough comparison.\n\n2.\tPerAct is designed to be a multi-task policy. I am not sure comparing PerAct in a single task learning is fair?\n\n3.\tMore details on atomic task design. Since the training data is composed by the experts, I am a bit confused about how to generate that many 500k samples in a reasonable time and labour budget? How many raw language instructions were collected and how many atomic skills per task is generated?\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5CDrMrgSao", "reviewer": "Reviewer_1D6s", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method based on PerAct to train robots to follow natural language instructions given observed point clouds. One of the main contributions is a hierarchical architecture design which first predicts a reference point using a reference layer and then predicts the actions accordingly. The proposed method is compared with the original PerAct on a set of table-top manipulation tasks and achieves better performances in 11 of the 16 designed tasks.  ", "summary_of_recommendation": "I like the proposed method and some of the demos look nice. However, the current experiments do not seem to be sufficient to justify the claims and contributions in the paper. Therefore, I am leaning towards rejection. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- How many training trajectories were collected?\n\n- What are the hyperparameters and how are they chosen? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n\n- The proposed hierarchical design is reasonable. Empirically, it demonstrates to outperform the original PerAct in 11/16 testing tasks (if counting each instruction with seen and unseen objects to be two tasks).\n\n- The proposed method is clearly motivated and described in the paper. The technical part of the paper is easy to follow.\n\nWeaknesses: \n\n- One of my biggest concerns is about the experiment setup. In the experiments, each of the tasks/instructions is separately designed in a different environment. This does not convincingly demonstrate if the learned policy can control the robot to solve tasks faithful to the comannded instruction. Based on our previous experience working on language-conditioned imitation learning, it is usually relatively easy to find certain scenes and a single instruction in each of those scenes, which allows the learned policy to solve the task. But if the policy is commanded by other different instructions for another plausible task in the same scene, the policy can fall short. For example, when there is an open drawer in front of the robot, closing the drawer can be the obvious thing for the robot to do given the training data. But if the robot is commanded to \"Move the mug into the drawer\" at the same state, the robot might either performs some random irrelevant behaviors or still close the drawer regardless. We believe that a more principled way to evaluate such language-conditioned policies would be to design N environments and, in each environment design M tasks (e.g. N = 3, M = 3) and shows if the robot can successfully follow different instructions at exactly the same initial state. For example, in the \"Close drawer\" tasks shown on the website, other tasks such as \"Move the bottle to the drawer\" and \"Move the stuffed animal to the drawer\" need to be also evaluated. Otherwise, it seems not entirely convincing if the policy successfully gains the ability to follow different instructions.\n\n- Some of the details about the method and the experimental setup seem to be missing. For example, how many training trajectories were collected? I couldn't find it either in the main paper or in Appendix A.1. And What are the hyperparameters and how are they chosen? \n\n- The results show that the proposed method does not generalize very well to unseen objects. The success rates seem to be much lower than those with seen objects.\n\n- It would be better if ablation on the design choices such as the modified set abstraction layer and the combination of the two losses for the interaction prediction module could be included in the experiments. Otherwise, it would be hard to justify the significance of these design choices. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach to allow robots for follow natural language instructions that relies on a multi-resolution tokenization of point-cloud-based features. The review process resulted in a recommendation of "weak accept" by a majority of 3 reviewers and "weak reject" by 1 reviewer. 

The reviewers mentioned the following strengths of the work in particular:
- improved performance vs PerACT was demonstrated
- the overall approach appears to be well-founded and motivated
- the paper is well written and easy to follow
- the decomposition of the policy into two phases was appreciated as novel

Among the opportunities for further improvement, the reviewers remarked on
- potential difficulty of the approach to generalize
- an evaluation showcasing that different instructions can be followed in the same scene and starting configuration was requested
- a comparison to 2D attention-based policies may be valuable
- additional simulation evaluations (as in PerACT0 may have been of value

Given the overall positive reviews, and since the work has promise in a rapidly evolving area of robotics, I believe the paper is above the acceptance threshold and of interest to the CoRL community despite potential areas that could be improved further.
Decision: Accept (Poster)