Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes KITE, a method that maps instructions to robot manipulation actions. It consists of two stages: a higher level stage proposes keypoints to act on and skills to execute. Each skill consists of a learned keypoint policy and a controller to generate low-level motion. The experiments show that KITE can solve a number of challenging rearrangement tasks (including openable containers) on a physical robot. 
review: Advantages:
- The problem of mapping natural language instructions to robot manipulation is hard, and this paper solves it reasonably well.
- The method offers generalization to unseen tasks (through LLMs) so long as the tasks can be broken down to a set of skills. It also offers generalization to unseen objects (through the use of CLIP and the particular way the keypoint policies are conditioned).
- Overal convincing real-world results showing efficacy of the system compared to prior state of the art methods.

Disadvantages:
- The writing is currently confusing, mixing definitions, inference, learning and data processing in text without sometimes a coherent structure. This makes it hard to keep track of what is the proposed method vs what is auxiliary processing needed to instantiate it in this paper, what are the inputs and outputs of the different components, and what processing happens during data preparation, training, and inference stages. It's not all bad though - all the needed information is there and the paper can be understood sufficiently well, but it doesn't make it easy - I almost need to draw diagrams while reading the paper to make sense of the system.
- Grounding module identifies a single keypoint for instruction and does not appear to be able to ground instructions referring to more than one thing.
- It would be good to compare the against PerAct on RLBench, a more standard and replicable benchmark. Or otherwise provide replicable simulation experiments.
- Given PerAct's good performance before, it is hard to understand why it performs so poorly in these experiments. I suspect it is not used as intended. PerAct is a keypoint policy, so it should be compared in an ablation when used as a keypoint policy within KITE. Comparing KITE to PerAct directly doesn't really make sense, since PerAct is not an instruction-following policy (it takes instruction input only to disambiguate tasks).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My rating is conditioned on improved presentation of the paper
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper advocates the use of keypoints in 2D images to help disambiguate language for downstream language understanding such
as selecting and parameterizing skills.
review: I would think that a paper on language understanding would have a much richer set of things the language could refer to.
Each of the three evaluation domains had about 4 skills to talk about, not 400. There seem to be up to 10 or so objects or parts
one could refer to, although this was not clear in the paper.

Why is the success rate in all cases 70-75%? Why is that range so low? And why are rates similar
across very different tasks with "varying tiers of difficulty"? Seems weird.

I am not convinced that selecting regions is more difficult than selecting keypoints, and in many cases I would think regions would
be more useful.
Where does the keypoint go if:
- there are multiple handles on the drawer (many have at least two).
- there is a long graspable area the width of the drawer.
- you have cabinets doors or drawers with no handles, which one pushes to open. 
https://www.youtube.com/watch?v=B2xYIQhZloI
Rather than focusing on "keypoints", don't you want to focus on "saliency"?
As the paper says, one is directing attention to particular features, not
commanding contact points.

What is the evidence for "capacity for long-horizon reasoning and generalization."?

Smaller points

"We assume that at least one “grounding” camera can partially see all relevant objects"
This seems unnecessary, and difficult to obtain. Why can't a virtual grounding image or set of images be synthesized instead?

What is the difference between a waypoint policy and a controller to move
between waypoints? Does this detail matter?

In training, why aren't new observations and point clouds obtained at every
point along the robot trajectory, or at least at the waypoints?
Why must a sequence of future waypoints be predicted?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am not impressed by the evaluation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper aims to perform 6-DoF manipulation from RGB-D + language instructions. The motivation/technical problem: existing instruction-following manipulation pipelines (1) don’t effectively allocate attention toward context at both the scene and object levels and (2) output 6-DoF gripper trajectories with limited expressivity. The proposed pipeline aims to address this: (1) from RGB-D + language, a discrete skill and a region for executing the skill are selected (via keypoint prediction, similar to CLIPort + SayCan), (2) from a local point cloud near the predicted region, a network predicts a dense sequence of "waypoint poses" for the gripper to follow. The method is compared to baselines that map RGB-D + language directly to gripper poses and that use frozen vision-language models for language grounding. The results highlight the benefits of continuous waypoint pose prediction, especially in enabling multi-step tasks, and the use of keypoint-prediction grounding over frozen VLMs.
review: **Summary of strengths and weaknesses**

Strengths:

- The real demonstrations are impressive and compelling. Any pipeline that enables the diverse kinds of capabilities shown in the paper (semantic grasping, multi-step tabletop manipulation) directly from language + RGB-D on real-world objects is noteworthy. 
- The paper showcases an overall system implementation that works and that is (mostly) described in sufficient detail to understand how to reproduce and build upon it. 
- The main novel technical components (the locally conditioned "waypoint-pose predictor"/skill policy) make sense, both in their design and in the way they're integrated into the overall pipeline.  

Weaknesses:


1. In my view, beyond the good overall system engineering, the main technical innovation is the local "waypoint pose" prediction. Thus, it’s a critical part of the paper to understand, but as it's written, I found it to be the most difficult part of the paper to understand (at least in terms of technical details). Some more detailed comments are mentioned with regard to this in the "clarity" section below. 
2. A related weakness is the framing of the paper and the difficulty in knowing what the main takeaway messages are. The results are good and the gist of the pipeline is not difficult to understand, but there are so many terms and claimed advantages (*long horizon reasoning*, *generalization*, *instruction following*, *parameterized skills are better than end-to-end*, *need for both scene- and object-level semantic*) that its hard to know what I learned as a reader of the paper. 
    - If we are to break it down a bit, one perspective (that I had as a reader) is below. Take it for what it's worth, but the general suggestion is to find a way to make it easier for a reader to walk away from the paper with a more clear "take-home message" in mind. 
        - The main new technical component is the local skill policies - predicting a sequence of waypoint poses from a local point cloud, parameterizing these as offsets from the point cloud points, using a different pose prediction network for each skill, and training these from a modest # of demonstrations. 
        - The motivation for building these local skill policies is to utilize them within an "RGB-D + language -> 6-DoF EE trajectory" instruction-following manipulation framework. By doing so, this will expand upon the capabilities of previous instances of "manipulation from language instructions" pipelines, so that (i) the executed motions can be more diverse, expressive and accurate, (ii) they can integrate with more kinds of instructions (specifying a desired part to interact with), and (iii) by being more accurate, they can better succeed in multi-step tasks (where compounding errors can cause failure). 


**Details**

*Technical quality -* Good. Strengths: The authors show an impressive full-stack implementation and very nice real-world demos. The point cloud-conditioned skill policies are a valuable technical contribution addition to the language-to-action stack. Weaknesses: More could be done to evaluate the ability of the low-level skills on their own, i.e. how well can they generalize to different objects/shapes/poses of objects, but the focus is more on how these skills integrate into the full stack (rather than trying to make the low-level skills perfect) so this is mostly okay.

*Originality -* Good. Strengths: Incorporating the local point cloud for conditioning a local skill policy that outputs 6-DoF "waypoint pose" is a nice addition to existing full-stack language-conditioned manipulation pipelines, and showing a clever way to parameterize it (i.e., by predicting offsets from the local point cloud) is also valuable. Weaknesses: PerAct/CLIPort part is the same as before, SayCan part is the same. 

*Clarity -* Good. Strength: The overall pipeline makes sense (high- vs. low-level, language to keypoint + skill, local point cloud to dense waypoint-pose sequence). Weakness: The most critical part of the method to understand (the skill-conditioned waypoint policies) was pretty confusing in terms of some key technical details. 
- Details on confusion:
    - My confusion comes from how the local point cloud + mask are used to output the waypoint poses. I cannot tell if it's (1) or (2): (1) The *whole* scene point cloud is used to predict the offset poses, and the predicted mask is appended to this entire point cloud as an extra binary feature, (2) Only the *subset* of the point cloud that comes from the predicted mask is used to predict the offset poses. 
    - i.e., in the bowl example that's discussed, you say “this yields a set of ‘candidate’ points for consider for interaction”… and “the bottom of the bowl points predict the points on the rim”, which makes me think you mask out the local point cloud and **only** use these to represent the gripper pose predictions. 
    - While this might seem like a minor detail, there are two reasons why I believe it's a critical point to clarify. First - masking + conditioning on the local point cloud alone seems like it would support significantly better generalization. Second - representing the poses as offsets from the local point cloud alone seems like it could make the task more difficult, since the gripper might need to be quite far from these masked points. This has implications for tasks where the gripper is using, e.g., a large tool, such as the broomstick task shown in PerAct (this is mentioned in the "questions for rebuttal" section below). 

*Impact/Significance -* Good. Strength: Language-conditioned pipelines are here to stay, so providing more instantiations of how they can work and perform more flexible behaviors is valuable. Further, introducing different kinds of hierarchical decomposition into such pipelines (vs. just end-to-end, direct execution of voxel-predicted actions, etc.) is also beneficial. Weakness (minor): This particular method for low-level skills will probably be built on and improved, but showing the impact of a locally-conditioned component is likely to help expand others’ minds for other ways to do this.


**Other comments, points of confusion**
- In introducing the paper + method, the claimed advantage of “*capacity for long-horizon reasoning”* felt slightly off-putting. IIUC, the main part that enables this is the LLM instruction following, which isn't really a contribution of the work, and the ability to execute individual steps more accurately. This is a contribution of the work, but the claim of enabling "long-horizon reasoning" should then be situated more clearly to this improved ability to perform individual steps more successfully. 
- In the related work, this comparison to RL/IL was pretty confusing and hard to unpack: “*Imitation learning (IL) aims to learn skills from demonstrations in a more sample efficient manner than RL, but these skills still fail to generalize to scene perturbations [35, 36]. Furthermore, in both IL and RL, connecting learned skills to precise language in a generalizable fashion is an open challenge. KITE avoids learning skills from scratch and instead defines a library of keypoint-conditioned skills, where the exact parameters of each skill are learned from demonstration.”*
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the details of the main novel technical components should be made more clear, the full-stack language-to-action system implementation is impressive, the addition of hierarchy, local geometry conditioning, and more expressive low-level motion prediction is valuable, and the real-world demos are very well done.
review_title: null

Meta-review: ### Summary, Strengths and Weaknesses
This paper introduces KITE, a two-stage method that utilizes keypoints to map instructions to robot manipulation actions. Experimental results demonstrate that KITE outperforms models that use visual language or end-to-end control. It achieves high success rates due to precise keypoint-driven execution in real-world environments. While the formulation of the method needs further refinement, the method is novel. Real-world demonstrations showcase the effectiveness of the approach.

The reviewers agree in their recommendation to accept the paper because most of the concerns have been addressed. I agree with their consensus.
Decision: Accept (Poster)