Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a method based on PerAct to train robots to follow natural language instructions given observed point clouds. One of the main contributions is a hierarchical architecture design which first predicts a reference point using a reference layer and then predicts the actions accordingly. The proposed method is compared with the original PerAct on a set of table-top manipulation tasks and achieves better performances in 11 of the 16 designed tasks.  
review: Strengths: 

- The proposed hierarchical design is reasonable. Empirically, it demonstrates to outperform the original PerAct in 11/16 testing tasks (if counting each instruction with seen and unseen objects to be two tasks).

- The proposed method is clearly motivated and described in the paper. The technical part of the paper is easy to follow.

Weaknesses: 

- One of my biggest concerns is about the experiment setup. In the experiments, each of the tasks/instructions is separately designed in a different environment. This does not convincingly demonstrate if the learned policy can control the robot to solve tasks faithful to the comannded instruction. Based on our previous experience working on language-conditioned imitation learning, it is usually relatively easy to find certain scenes and a single instruction in each of those scenes, which allows the learned policy to solve the task. But if the policy is commanded by other different instructions for another plausible task in the same scene, the policy can fall short. For example, when there is an open drawer in front of the robot, closing the drawer can be the obvious thing for the robot to do given the training data. But if the robot is commanded to "Move the mug into the drawer" at the same state, the robot might either performs some random irrelevant behaviors or still close the drawer regardless. We believe that a more principled way to evaluate such language-conditioned policies would be to design N environments and, in each environment design M tasks (e.g. N = 3, M = 3) and shows if the robot can successfully follow different instructions at exactly the same initial state. For example, in the "Close drawer" tasks shown on the website, other tasks such as "Move the bottle to the drawer" and "Move the stuffed animal to the drawer" need to be also evaluated. Otherwise, it seems not entirely convincing if the policy successfully gains the ability to follow different instructions.

- Some of the details about the method and the experimental setup seem to be missing. For example, how many training trajectories were collected? I couldn't find it either in the main paper or in Appendix A.1. And What are the hyperparameters and how are they chosen? 

- The results show that the proposed method does not generalize very well to unseen objects. The success rates seem to be much lower than those with seen objects.

- It would be better if ablation on the design choices such as the modified set abstraction layer and the combination of the two losses for the interaction prediction module could be included in the experiments. Otherwise, it would be hard to justify the significance of these design choices. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I like the proposed method and some of the demos look nice. However, the current experiments do not seem to be sufficient to justify the claims and contributions in the paper. Therefore, I am leaning towards rejection.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents Spatial-Language Attention Policies (SLAP), a method for language-guided mobile manipulation. SLAP has two components (1) an interaction prediction module that localizes interaction points in a point cloud (object affordance), and (2) a relative action module that predicts relative action sequence w.r.t. the contact points in the cartesian space. The models are trained with labeled expert demonstrations.
The approach uses atomic skills to manipulate objects, defined by the interaction point and subsequent relative waypoints. A language-conditioned policy predicts the atomic skills parametrization given input observations.
The authors evaluated the method with 5 test runs per scene setting per skill per model, comparing it to Perceiver-Actor (PerAct) method and demonstrating superior performance.
review: Strengths:
1. The presentation of the method is clear and easy to follow.
2. Good coverage of related work.
3. The design of the two-phase language-conditioned policy to predict the affordance and the waypoints is a viable option.
4. Using demonstrations from a different robot than the robot used for deployment suggests good generalization abilities. 
5. Real-world experiments on two robots, Franka and Stretch.
6. Honest discussion of the limitations.

Weaknesses:
1. In lines 92-94, the authors criticize PerAct for not looking at out-of-domain generalization, but the current evaluation section does not look at it.'
2. It would be interesting to compare the method to 2D attention-based policies, e.g., SPOT.
3. The method constructs an input point cloud from pre-defined scanning actions (lines 117-118) followed by voxelization. How much time is needed to build this scene representation? Is it required for each atomic skill? How much does this affect method deployment?
4. What are the benefits of the proposed set abstraction layer?
5. The evaluation is reported for 5 tests per scene setting per skill per model. The number of scenes is not reported, and it seems from Table 2 that there were only 5 runs (aside from the 10% for PerAct).
6. It would be interesting to show the results with the last checkpoint of SLAP. It will be a fair comparison, and it will demonstrate robustness. While using the model with the highest validation loss is equivalent to cherry-picking.
7. The robot is spawned randomly in the environment (line 222). Are the objects randomized during the experiments?
8. Section 4.2. how big is the dataset? What are the evaluation settings? Table (Figure 5) suggests potential overfitting.
9. The ablation study shows that SLAP can not generalize well for out-of-distribution settings which contradicts the claims in the introductions.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes an interesting method for language-conditioned policies operating on point clouds.
The current experimental sections need some revisions to support the paper's claims.
We vote for rejecting the current version of the paper, but we will support it for acceptance if the authors can address our comments.

**Update after rebuttal:**

The authors addressed most of our comments reasonably; we raised our recommendation to a weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes Spatial-Language Attention Policies (SLAP), which leverages three-dimensional tokens (features extracted from point clouds) as input presentation to train a single multi-task, language conditioned action prediction policy. The method is comprised of two primary modules: Interaction Prediction Module, which predicts an interaction point given a language instruction and input point clouds, and Relative Aciton Module, which predicts a desired end-effector pose relative to the interaction point and the desired gripper state given the interaction point, cropped point cloud and language instruction. The two modules are trained separately with imitation learning. The whole system demonstrates promising results in table-top pick-and-place tasks (including opening/closing articulated objects) and mobile manipulation pick-and-place tasks, and outperformed previous baselines. 
review: This paper presents a pretty exciting and novel robot learning system that performs well in around 10 real-world robotics tasks and is able to handle unseen distractor objects and novel object configurations to some degree during test time.

This work has a few noteworthy strengths. First of all, the algorithmic design seems novel and interesting. Although spatial action map / attention-based policies have been studied before, this paper is one of its first kind to directly operate on 3D point cloud with modified set abstraction layers adapted from PointNet++ and also take advantage of transformers’ ability to take in varying-length stream of multi-resolution spatial embeddings. 

Secondly, by decomposing the policy into two stages (“finding interaction point”, and “generating relative motion around that point”), the SLAP demonstrates better performance and sample efficiency than previous methods on real world robotics tasks. The experimental evaluation (per skill and multiple skill setting) is relatively thorough. The proposed model is able to generalize to unseen objects and object poses, which is quite impressive. 

Thirdly, the paper is relatively well-written. Different components of the system are well explained (the two major modules, how LLM is used, how the point clouds are processed, etc). The figures are pretty illustrative, as well as the videos in the supplementary material. 

This work, however, is not without limitations. Overall, although the paper is relatively easy to follow, it seems to be missing some important information (more details can be found in the section “Questions For Rebuttal” below). For example, the paper should probably include more results analysis, especially the failure mode analysis (e.g. which of the two modules fail more often, in seen and unseen settings, and why? What percentage of the failures are due to motion planning failures? Why is the motion planner not collision aware). The paper also misses citations to other attention-based policies, such as ReLMoGen[1] and Spatial Action Maps for Mobile Manipulation[2]. ReLMoGen is particularly relevant because it also predicts a 3D interaction point (a pixel mapped to 3D space from depth sensing) and a local nonprehensile interaction from that point (predicting desired offset), although it’s trained with RL rather than IL. Lastly, the paper should probably tone down its claim about “mobile manipulation”. One of the primary challenges of mobile manipulation, compared to tabletop manipulation, is that 1) the scene is only partially observable (i.e. the desired interaction point might be outside the current FoV), 2) whole body motion (e.g. the robot might need to move the arm and the base holistically when opening a fridge door in a narrow kitchen). This work tackles neither of these two cases, and it’s left unexplained how the robot navigates to objects (those in FoV and outside FoV). 

Nonetheless, this work seems to be fairly promising. With proper revision, this work can be very impactful for the skill learning and the overall robot learning community

[1] Xia, Fei, et al. "Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[2] Wu, Jimmy, et al. "Spatial action maps for mobile manipulation." arXiv preprint arXiv:2004.09141 (2020).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on the review above, I recommend Weak Accept. I am open to changing my mind based on the rebuttal.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a language-conditioned policy for mobile and table-top manipulation tasks. The proposed policy is based on spatial attention on point clouds and can be decomposed into two stages: i) an interaction point is predicted based on sub-sampled point cloud, and ii) relative action is predicted around the predicted interaction point. The proposed method is shown to perform well in real-world manipulation tasks compared to a competitive baseline PerAct. 

 
review: Overall, the paper presents an interesting and well-engineered system for manipulation tasks. The overall method design makes sense but with some small minor limitations on the experiment design.

1.	In Table 2, the number of evaluations per task is only 5, which may cause large statistical variations. One success and failure could lead to 20% of the difference. Considering the performance difference for SLAP and PerAct is only 30%, having only 5 trials per tasks is clearly not significant for a thorough comparison.

2.	PerAct is designed to be a multi-task policy. I am not sure comparing PerAct in a single task learning is fair?

3.	More details on atomic task design. Since the training data is composed by the experts, I am a bit confused about how to generate that many 500k samples in a reasonable time and labour budget? How many raw language instructions were collected and how many atomic skills per task is generated?


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Good method design with a bit insufficient experiments. Could be improved with rebuttal.
review_title: null

Meta-review: The authors propose an approach to allow robots for follow natural language instructions that relies on a multi-resolution tokenization of point-cloud-based features. The review process resulted in a recommendation of "weak accept" by a majority of 3 reviewers and "weak reject" by 1 reviewer. 

The reviewers mentioned the following strengths of the work in particular:
- improved performance vs PerACT was demonstrated
- the overall approach appears to be well-founded and motivated
- the paper is well written and easy to follow
- the decomposition of the policy into two phases was appreciated as novel

Among the opportunities for further improvement, the reviewers remarked on
- potential difficulty of the approach to generalize
- an evaluation showcasing that different instructions can be followed in the same scene and starting configuration was requested
- a comparison to 2D attention-based policies may be valuable
- additional simulation evaluations (as in PerACT0 may have been of value

Given the overall positive reviews, and since the work has promise in a rapidly evolving area of robotics, I believe the paper is above the acceptance threshold and of interest to the CoRL community despite potential areas that could be improved further.
Decision: Accept (Poster)