KITE: Keypoint-Conditioned Policies for Semantic Manipulation
Authors: Priya Sundaresan
Suneel Belkhale
Dorsa Sadigh
Jeannette Bohg
Venue: CoRL 2023 Poster (2023)
Keywords: Semantic Manipulation
Language Grounding
Keypoint Perception

Abstract: While natural language offers a convenient shared interface for humans and robots, enabling robots to interpret and follow language commands remains a longstanding challenge in manipulation. A crucial step to realizing a performant instruction-following robot is achieving semantic manipulation – where a robot interprets language at different specificities, from high-level instructions like "Pick up the stuffed animal" to more detailed inputs like "Grab the left ear of the elephant." To tackle this, we propose Keypoints + Instructions to Execution, a two-step framework for semantic manipulation which attends to both scene semantics (distinguishing between different objects in a visual scene) and object semantics (precisely localizing different parts within an object instance). KITE first grounds an input instruction in a visual scene through 2D image keypoints, providing a highly accurate object-centric bias for downstream action inference. Provided an RGB-D scene observation, KITE then executes a learned keypoint-conditioned skill to carry out the instruction. The combined precision of keypoints and parameterized skills enables fine-grained manipulation with generalization to scene and object variations. Empirically, we demonstrate KITE in 3 real-world environments: long-horizon 6-DoF tabletop manipulation, semantic grasping, and a high-precision coffee-making task. In these settings, KITE achieves a 75%, 70%, and 71% overall success rate for instruction-following, respectively. KITE outperforms frameworks that opt for pre-trained visual language models over keypoint-based grounding, or omit skills in favor of end-to-end visuomotor control, all while being trained from fewer or comparable amounts of demonstrations. Supplementary material, datasets, code, and videos can be found on our website: https://tinyurl.com/kite-site.

KITE: Keypoint-Conditioned Policies for Semantic Manipulation
Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, Jeannette Bohg
Stanford University
Abstract: While natural language offers a convenient shared interface for humans and robots, enabling robots to interpret and follow language commands remains a longstanding challenge in manipulation. A crucial step realizing performant instruction-folloiiin robot is achieving semantic manipulation where robot interprets language at different specificities, from high-level instructions like 'Pick up the stuffed animal' to more detailed inputs like 'Grab the left ear of the elephant.' To tackle this, we propose KITE: Keypoints + Instructions to Execution, a two- step framework for semantic manipulation which attends to both scene semantics (distinguishing between different objects in visual scene) and object semanties (precisely localizing different parts within an object instance). KITE first grounds input instruction in visual scene through 2D image keypoints, providing a highly accurate object-centric bias for downstream action inference. Provided an skill
RGB-D scene observation, KITE then executes learned to carry out the instruction. The combined precision of keypoints and parameterized skills enables fine-grained manipulation with generalization to scene and object variations. Empirically, we demonstrate KITE in 3 real-world environments: long- horizon 6-DoF tabletop manipulation, semantic grasping, and a high-precision loffee-making task. In these settings, KITE achieves 75%, 70%, and 71% overall success rate for instruction-followee that opt for pre-trained visual language models over keypoint-baseee grounding, omit skills in favor of end-to-end visuomotor control, all while being trained from fewer or comparable amounts demonstrations. Supplementary material, datasets, code, and videos can be found on our website."
respectively. KITE outperforms frameworks
Keywords: Semantic Manipulation, Language Grounding, Keypoint Perception
Introduction
Language has the potential to serve powerful communication channel between humans and robots in homes, workplaces, and industrial settings. However, two primary challenges prevent today's 's robots from handling free-form language inputs. The first is enabling a robot to reason over what to manipulate. catruction-followinn requires not only recognizing task-relevant objects from a visual scene, but possibly refining visual search to specific features on particular object. For instance, telling a robot to "Open the top shelf" "Yank open the bottom shelf" of cabinet requires not only parsing and resolving any liberties taken with phrasing and localizing the cabinet in the scene (scene semantics), but also identifying the exact object feature that matters for the task - in this case the top or bottom handle (object semantics). In this work, we refer to instruction- following with scene and object awareness as semantic manipulation. Similarly, pick-and-place is a standard manipulation benchmark [1, 2, 3, 4], knowing how to pick up a stuffed animal by the ear versus leg, or a soap bottle by the dispenser versus side requires careful discernment. After identifying what manipulate, the second challenge determining how the robot can accomplish the desired behavior, i.c., low-level sensorimotor control. In many cases, low-level action execution requires planning in SE(3) with 6 degrees-of-freedon (DoF), such as reorienting the gripper sideways to grasp and pull open a drawer.
instruction-follooiig
(priyasun, belkhale] @atanford. edu, dorsauca. stanford. edu, bohgOstanford.edu http://tinyarl..omiiesssst
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Long- Horizon
Instruction Following
"Open the buttum drawer"
"Put the marker away"
"Close drawer"
"Pick the ketchup"
"Tlck
Semantic Grasping
pliers right handle
I-tel: left
plarnç joint
seisome larger hole
marker Label
marker;
belly
elephant; trunk
unicomm leg
Precise Manipulation
Keusig"
grewn
"Laud blue mug up"
"Fill up
Figure Real-World Semantic Manipulation Environments: We visualize our semantic manipulation framework KITE on three real-world environments: long-horizon instruction following, semantic grasping. and coffee-making. Using keypoint-basee based grounding, KITE contextualizes scene-level semantics ('Pick up the green/red/blue/brow coffee pod') as well as object-level semantics ("Pick up the unicorn by the leg/ear/tail', "Open the cabinet by the top/middlle/ottiom shelf") and precisely executes keypoint condi--endiio skills.
Going beyond grasping, we want robots to assist us in many daily real-world tasks which may require even finer-grained precision. Making coffee, for example, a simple daily task for humans, but robot it involves complex steps like reorienting mug from sideways to upright or carefully inserting coffee pod into espresso machine. Thus to achieve semantic manipulation, robots scene and object semantics from input instructions and plan precise low-level actions accordingly.
extract
Leveraging advances in open-vocabulary object detection [5, 6, 7, 8], prior works in language- -aasae manipulation determinine what to manipulate via bounding boxes or keypoints obtained from pre- trained [9] or fine-tuned vision language models (VLMs) [10]. So far, these works operate at the level of scene semantics (distinguishing amongst objects) rather than object semantics (identifying within-ohject features). In addition, these works not apply VLMS any complex manipulation beyond simple pick-and-place. To address these shortcomings, follow-up works couple the what and how subproblems together and learn end-to-end 6-DoF language- innguage-coniitione policies from demonstrations [11, 12, 13]. However, learning high dimensional action spaces from raw sensor inputs such as images voxelized scene representations can require excessive amounts of data [12, 13] and can be difficult in high-precision tasks especially when using discretized actions [1]].
Between approaches that leverage pre-trained visual representations from off aff-the-shelf VLMs, and those that plan directly from pixels or voxels, we lack an intermediate object-centric representation that can link natural language scene and object semantics. Prior work has demonstrated that open-vocabure open-vocabulary VLMs address scene semantics to some extent by locating different objects with coarse bounding boxes [9], but these representations are still too granular precisely locate parts on objects. A suitable visual representation would be one that can represent both across-object or within-object features, and is interpretable enough to inform downstream 6-DoF action planning. We argue that keypoints provide this happy medium by offering way to precisely pinpoint objects at the seene-level or even features within an object (what) and way to condition downstream 6-DoF manipulation (how) on region of interest.
In this work, we present KITE: Keypoints Instructions To Execution, a flexible framework for semantic manipulation. KITE is decoupled into grounding policy which maps input images and language commands to task-relevant keypoints, and an acting module which employs keypoint- conditioned skills to carry out low-level 6-DoF actions. We show that KITE can be trained from just a few hundred annotated examples for the grounding model, and less than 50 demos per skill for the acting module, while outperforming and generalizing better than methods that do not make use of either keypoints or skills. We experimentally evaluate KITE on semantic manipulation across three challenging real-world scenarios with varying tiers of difficulty: 6-DoF tabletop manipulation, se- mantic grasping, and coffee-making (Figure l). Results indicate that KITE demonstrates fine- fine-grained instruction following, while exhibiting capacity for horgzhorizon reasoning and generalization.

---

2 Related Work
Language-Basee Based Manipulation: Many recent works ground language to manipulation skills as means for long-horizon instruction following. Several methods learn language conditioned policies end-to-end using imitation learning; however, end-to-end learning can require many demonstrations and can be brittle to novel scenes or objects [12, 11, 14, 15, 16, 17, 18, 19, 10, 20]. To improve sample efficiency, Shridhar et al. [11] predicts robot waypoints rather than low-level actions, conditioned on language and point cloud inputs. Waypoints alone do not specify how go from one waypoint to another, and thus fail capture dynamic tasks such as peg insertion pouring motions. Other works take a hierarchical approach that first learn or define a library of language- -onouagge-odiiioneed skills, and then plan over skills with large language models (LLM) [21, 22, 23, 24, 25, 26]. However, each skill often requires hundreds of demonstrations. In addition, the LLM planner can only reason about the scene at a high level, lacking visual grounding. Alternatively, Liang et al. [27] query the LLM to generate code using an API for low-level skills, but predicting continuous parameters of these skills challenging for an ungrounded LLM limiting this approach to tasks that do not require precision or dexterity. Vision-Language Models (VLM) are often proposed to ground LLM planners. For instance, Stone et al. [28] leverage pretrained VLM to identify task-relevant objects from language. However, this approach has also been limited to pick and place tasks suggesting that today's pretrained VLMs struggle with more precise language instructions. In contrast, KITE maps language and vision directly to desired keypoints, enabling precise and semantic manipulation over long horizons.
Skill-Based Manipulation: Many prior works study multi-task manipulation by defining skills to represent sub-tasks, and then composing these skills over long horizons. These skills can either be learned to output each action or parameterized by expert-defined features. In reinforcement learning (RL), hierarchy can be imposed the policy to learn both skills and composition end-to-end, but these methods can be sample inefficient and rarely integrate well with natural language [29, 30, 31]. Other RL works parameterize skills to reduce the action space size for sample efficiency, but these skills are usually rigid and cannot generalize to new settings [32, 33, 34]. Imitation learning (IL) aims to learn skills from demonstrations in a more sample efficient manner than RL, but these skills still fail to generalize to scene perturbations [35, 36]. Furthermore, in both II. and RI., connecting learned skills to precise language in a generalizable fashion is an open challenge. KITE avoids learning skills from scratch and instead defines a library of keypoint-condittonee skills, where the exact parameters of each skill are learned from demonstration. We show that keypoint condiint-o skills are sample efficient to learn and generalizable to new objects, while also easily integrated with precise language.
Keypoints for Manipulation: Keypoints have emerged in the literature as more robust skill representation for manipulation [37, 10J. Keypoints are 2D points on images that serve as a natural intermediary between images and low-level behaviors. Several methods use keypoints to force the model to attend to the most important features in the input images [37]. Others predict keypoints and then translate keypoints into 3D points, or directly predict 3D points, to parameterize low level behaviors in a general and visually-grounded fashion [38, 10, 39, 40). For example, Shridhar et al. [10] parameterize pick and place tasks using keypoints learned with image supervision, showing that this keypoint abstraction generalizes better to new objects. Keypoint action spaces have also helped in deformable object manipulation, for example in the domains of cloth folding, rope untangling, and food manipulation [41, 42, 43, 45). For many of these prior works, keypoints have yet to be integrated with language, and the methods that are linked to language are limited to small library of primitives, usually focusing only on pick and place scenarios. Our approach defines a much broader library of keypoint- skills, and integrates keypoints with complex language instructions.
3 KITE: Keypoints + Instructions To Execution
In this work, our goal is to train an instruction-followinn agent capable of performing semantic manipulation at both scene and object-level granularity. We accomplish this with KITE, a sample- efficient and generalizable framework operating in two stages: grounding language into keypoints and acting on those keypoints. In this section, we first formalize the semantic manipulation problem (Section 3.1), discuss data collection (Section 3.2), and then discuss the training procedures for the grounding (Section 3.3) and acting modules (Section 3.4).
3

---

"Open the top drawerr
{pick, place, open, close)
Ul. Ve!
I
Grounding Module
Acting Module
π
(Pr.Me)
Localizing semantic keypoints
Skill inference
Skill execution
Figure 2: KITE System Overview: KITE receives image observation /t along with user instruction in and grounds these inputs 2D semantic keypoint in the image. After inferring which skill type appropriate from of skill labels, KTTE takes RGB-D point cloud observation Pt, annotated with the deprojected keypoint M,, and infers the appropriate waypoint policy for execution. After executing this action, KITE replans based on a new observation (Ie+1, (t41) and repeats whole process.
3.1 Semantic Manipulation Problem Formulation
We aim to tackle instruction- instruction-followinn with scene and object-level semantic awareness using library of skills. We assume each skill can be parameterized by 6-DoF waypoints, and we decouple each skill into a waypoint policy π and controller to move between waypoints in task-specific manner. Additionally, we assume each skill can be represented by a skill label I, e.g., pick, open, etc. We construct a library L of M specialized skills where L {1' from skill label an underlying policy æ and controller p.
.p")...,, .MMM ,M) maps
We assume access to multiple calibrated cameras that provide RGB and depth. We assume that at least one "grounding" camera can partially see all relevant objects. An observation o, (1, P.) where I IR Wxx33 the image from the grounding camera, and Pe E Px6 point cloud. We denote low-level robot actions 0₂ (I. y, 2, v) 0, o) at time & which consist of the end-effector position, yaw, pitch, and roll. We denote waypoints represents a high-level pose (e.g., grasp pose for pick) rather than low-level action.
multi-view
which is also 6-DoF pose, but
At time given an instruction éc. we want know which skill to execute by associating i, corresponding skill label E (?'....,1%h MM. Next, aim to infer 2D keypoint (u, u] in the current visual observation o, which grounds to associated object or object part. Finally, the chosen skill (TT, p) C(l+] is executed (Figure 2). For each skill, we want to find waypoint policy that takes as input the visual observation or and 2D keypoint [u,, Uz] and outputs K waypoints: T: (o,, [u,, vel) + {n. ....,, K) Then, the associated controller P can output a low-level trajectory between waypoints p {I=,,..., ......,K - T {(or, at)..... (O++T-1,
(i.e. via linear interpolation, motion planning, etc.) for the robot to execute. For multi-step manipulation tasks, we restart the above process each step with the new observation o++-7 and paired language input 4,4T.
We consider instructions which refer to scene semantics, such as specifying desired spatial rearrange- ments objects (c.g. "Pick up the lemon"), and object semantics, which reference desired object parts to be manipulated (e.g. "Grab the kangaroo stuffed animal by the tail"). As we not assume access to the interaction history, our space of feasible language inputs excludes post-hoc feedback ("Pick up the other marker") or culine-corrections ("No, to the left!"), which we leave to future work. Next we outline how KITE learns to predict keypoints (grounding) and learns each skill æ (acting).
3.2 Demonstration Collection
To learn both the grounding and acting modules, collect dataset D,, consisting of N expert demonstrations per skill. Fach demonstration has an initial observation, list of K waypoints, and a language instruction: D, = {ox, {Nr K J, ix) [I,...,,NJ, For instance, for pick skill, we record the initial image and point cloud, provide an instruction (e.g.. PPikk up the lemon') ), and then kinesthetically move the robot and record each end-effector waypoint. We use the calibrated robot-to-camera transformation to automatically project each robot end-effector pose ke, to 2D coordinates [u,, va] in the image plane of the camera used for grounding. For each skill, we train the acting module from D Aggregating across all skills yields dataset of paired images, keypoint annotations, and language instructions with which to train the grounding module.

---

3.3 Grounding Module
The grounding module learns to identify 2D keypoints from RGB images that correspond to object features mentioned in an input instruction. We draw inspiration from recent works which use explicitly supervised [10, 38] or self-supervised keypoint attention networks [37, 46] to implement a grounding model ground. Specifically, we learn grounding function Qgoound (u, v, 42,4t) representing the likelihood of keypoint (u, v) given image I, and paired language instruction it. In this work, we attempt learn single-step look-ahead grounding function that takes a language input (e.g. "Put the lemon into the cabinet") and outputs the most immediately relevant keypoint (e.g. the pixel for the lemon if not already grasped, otherwise the pixel for the cabinet drawer to be placed) (see Fig. 2).
Given this grounding function Q Qground, we infer the 2D pixel in the image with highest likelihood:
[u,, Uz] arg max Q ground (u, v, Ie, ic)
(1)
1,,17
In practice, we implement Qground using the two-stream architecture from [10] which fuses pre- trained CLIP [47] embeddings of visual and textual features in a fully-convolutional network to output heatmap of Q ground. The grounding function is trained with binary cross-entropy loss between the predicted heatmaps and 2D Gaussian heatmaps centered at the ground-truth pixel.
3.4 Acting Module
Although keypoints can pinpoint both scene and object semantics, they critically lack the 3D geometric context necessary to recover precise 6-DoF actions for a given task. For instance, the command "Pick up the bowl" may result in a predicted keypoint located at the bottom of bowl, where there is no feasible grasp The exact 6-DoF actions are also dependent not just on the keypoint, but also language: "pick the lemon" and "cut the lemon" have similar keypoints but require completely different actions. We need a way to refine predicted keypoint into candidate 6-DoF actions based on a desired language command, which we discuss next.
Skill Selection: Given free-form language instruction, KITE first leverages the knowledge of LLMs to determine the appropriate skill label (e.g. i, ="Put the lemon in the cabinet" should result in the LLM outputting 4 'pick paackppaae'). following prior work [48, 21J. The procedure entails prompting the LLM, in our case OpenAl's text-davinci-003 [49], with in-context examples of instructions and the appropriate skill type (see Appendix B.3 for examples of prompting strategy). At test,time, we concatenate the example prompt with instruction i, and generate skill label 1, E {(,...,. [.f using the LLM. Then, we obtain the skill, consisting of the waypoint policy π and controller via lookup in the library: (I, p) = [(])..
Learning Waypoint Policies: Given the keypoint (ūe, 0z) predicted by the grounding module and skill label 4,, we need to learn waypoint policy π to perform the skill. KITE learns for each skill The waypoint policy x takes a point cloud P. and keypoint (ū,, 6x] as input, and aims to output K waypoints (xs...... k } to execute the chosen skill.
from demonstrations of keypoints {K² ...., kx
In KITE we align both 3D point cloud and 2D keypoint representations by "annotating" P, with the keypoint. We do this by first taking the depth image D, from the same view as I,, and deprojecting all nearby pixels within a radius R: KR = (lu, w] E Ie, [l[u, w] - [ūg. 6e]ll < R} to their associated 3D points Pr = ((x. y, =) = deproject(( v), V(u, v) E K.n). This yields set of "candidate" points consider for interaction. In the bowl grasping example, PR would be points on the bottom of the bowl. Next, we augment the point cloud P. with 1-channel mask M, E RNII (Fig. 2). For any point (x, y. 2) E P,, the mask channel label is if (x,, v, PR (i.e., the point is in close proximity to the deprojected keypoint) and O otherwise.
Given the pointcloud and keypoint mask, KITE predicts all K waypoints relative to individual points in the point cloud. For each point, we classify which of the K waypoints it is nearest to, along with the offset to the desired 7-DoF end-effector pose (position and quaternion) for each of the K waypoints. To do so, we adapt the PointNet++ [50] architecture, and define Q= (P), + P. where P, Nxd and d K (1 + + 4). Continuing the example of grasping bowl, the predicted pose offsets for each point the bottom of the bowl (ungraspable) should lead to the bowl rim (graspable). See Appendix A for more details about the actor. We supervise Qn using the
5

---

following per per-point loss:
Lykil Ca, + Aori(1 (9.-9))) + Apps Dg. 2il, [atk yk: 2xl)
(2)
The first term corresponds to the 1-hot cross-entropy classification loss between the predicted
waypoint index i and the true nearest waypoint index k. The remaining terms supervise the predicted gripper orientation and position using waypoint n^ for only the points that have classification label k, so as to only penalize points that matter (are in close proximity) to the n².
Action Module Inference: At test time, given a point cloud P. with associated keypoint mask Mr. we use Qx to obtain Ph. By taking the highest likelihood point for each of the K indices (representing the points nearest each of the K waypoints). Then, we index the predicted end-effector poses in Ps by these K indices, resulting in K waypoints {f* "''.....,. .......,,,,, Finally, we obtain the final trajectory to carry out the skill with using the skill-specific controller:
p( (o(""....... ..... A**)).
In summary, KITE's full pipeline first grounds language command i, in observation or via Qground to infer keypoints (Section 3.3), infers the skill label 4 (Section 3,4), maps the label to skill and controller (T,, p) = L/4). then and executes and p. and finally replans.
Experiments
In this section, we aim to answer the following questions: (Q1) How well does KITE handle scene semantic instructions? (Q2) How well does KITE handle precise object-semantic instructions? (Q3) Does KITE's scene and object semantic awareness generalize to unseen object instances? and (Q4) Can KITE's primitives capture precise and dexterous motions beyond pick and place? We first outline KITE's key implementation details and the baseline methods we benchmark against. Finally, we analyze KITE's comparative and overall performance across three real-world environments which stress test scene and object-aware semantic manipulation (Section 4.1).
Implementation Details: Across all evaluation environments, we specify a library of skills and collect 50 kinesthetic demonstrations per-skill. In order to improve precision of the grounding module, and because keypoint supervision is easy obtain compared to kinesthetic teaching, we supplement the grounding dataset obtained from kinesthetic data collection by manually labeling small amount of images with paired language instructions (0.75:1 supplemental to original samples ratio). We implement the grounding module according to the architecture from [10] and each waypoint policy in the acting module as a PointNet++ backbone [50] with a point cloud resolution of 20K points. See Appendix A for more details and visualizations of grounding model predictions.
Baselines: We benchmark KITE's performance against two state-of- of-te-of-thh-arr instraction-followinn following frameworks. The first is PerAct [11], which trains PerceiverIO [51] transformer backbone to predict waypoint poses end-to-end conditioned
voxelized scene representation and language. In comparing against PerAct, we hope to understand whether KITE's use of skills can offer better precision over end-to-end actions. To understand the value of keypoint- based grounding over frozen representations obtained from VLMs, we compare to RobotMoo [9], which extends a library of language-conditione skills to additionally condition on segmentation masks from an open-vocabulary object detector. Since exact models and data were not released, we use state-of-the-art thr- VLM and our set of learn skills for RobotMoo. See Appendix A.3 for more details.
4.1 Real-World Evaluation
We explore three real-world manipulation environments that provide rich testhed to explore KITE's sensitivity to scene and object semantics. Task variations are detailed in Appendix B. Across all experimental trials, we use Franka Emika 7DoF robot and 3 Realsense D435 RGB-D cameras.
Tabletop Instruction-Following We train a library of four skills: fpick, place, open, close} to reorganize a tabletop environment with 15 different household objects and an articulated storage organizer with three pull-out drawers (see Figure 1, Appendix B. While we do not test on completely unseen objects, we randomly vary the positions of objects on the table and the degree of clutter by adding distractor objects to the scene.
Table 1 compares KITE against PerAct and RobotMoo in this setting. We evaluate all approaches with 12 trials of ciltucctionfollowingg across three tiers of difficulty, ranging from a few objects on

---

the table and fairly straightforward language instructions (Tier 1), visually cluttered table (Tier 2). and a cluttered table with more ambiguous instructions (Tier 3). See Appendix B. for examples of the objects considered and variations across tiers.
We first evaluate individual actions (open,
close, pick), finding KITE to be the most bust and repeatable. KITE's 's use of precise key- point grounding enables scene semantic aware- ness (QI) over different objects (pick, place) and object semantic understanding (Q2) by dis- tinguishing amongst different drawer handles with the open and close skills. PerAct's disad- vantage its discrete visual space, where any slight 1-off voxel predictions make it diffi- cult to grasp objects or cabinet handles. Due to the weak classification objectives it is trained on, its most common failure mode is misclassi- fied gripper opening/closing actions. These fail- ures are alleviated by the parameterized skills used in KITE and RobotMoo. Unsurprisingly, RobotMoo does well at grasping different ob- jects referenced in language, VLMs trained on internet-scale data have strong object priors. Still, RobotMoo struggles with object semantics like the distinction amongst top, middle, or bottom drawers when opening or closing. We find that KITE is also the most competitive framework for long-horizon sequential reasoning (last two columns in Table 1), and the most common failures still include grasping the wrong object or with slightly misaligned gripper poses. RobotMoo's inability to reason multiple cabinet handles for opening and closing impedes its long-horizon performance, whereas PerAct's compounding precision errors render long horizon tasks especially difficult.
Rigid Tools
Deformables
Articulated Items
left handle handle
car
left handle
Seen Instances
botton bandle
heil
larger hold
expo label
right handle
shoelaces
cap
handle
Unseen Instances
top handle hammerhean
smaller
hole
bothum
handle
Figure Semantic Grasping Experimental Setup: We evaluate KITE on semantic grasping across rigid tools, deformable objects, and articulated items. We show 17 of the objects tested along with ground- truth semantic labels for different features. The top row includes objects seen during grounding module training. and the bottom consists of unseen object instances.
open
close
pick
pick >place
open +pick place
KITE RobotMoo
1
0.92
0.83
0.75
0.75
Tier
0.33
0.41
0.75
0.41
0.08
PerAct
0.08
0.5
0.33
0.08
N/A
KITE
1 0.36
0.83
0.76
0.66
0.42
Tier 2
RobotMoo
0.36
0.55
0.36
0.09
KITE RobotMoo
0.75
0.83
0.58
0.66
0.58
Tier 3
0.33
0.42
0.5
0.42
0.0
Table 1: Tabletop Instruction Following Results: Across 12 trials per method per tier. KITE outperforms both RobotMoo and PerAct for individual actions (open, close, pick) and chaining together up to four actions in sequence. We test all approaches Tier 1 (fewer objects, straightforward language). Tier (more objects. straightforward language), and Tier (more objects, skills gives an edge with precision over PerAct, which highly susceptible to one-off voxel predictions. This makes skills like opening and picking especially hard, and renders the approach virtually ineffective for higher complexity tiers (2 and 3). RobotMoo the most competitive approach KITE, but its main pitfall is a lack of object semantic awareness such as distinguishing amongst different-level cabinet handles.
free-form language). KITE's use of parameterized
Semantic Grasping: Aside from recognizing and manipulating different objects, we explore in greater detail whether KITE can perform object-semantic manipulation (Q2). We evaluate KITE on the task of semantic grasping, with instructions of the form "Pick up the X by the Y" (i.e. "stuffed bear' and 'car'; 'marker' and 'cap'; 'shoe' and 'laces') (examples in Fig. 1). For these trials, we train Qground on a subset of rigid tools, deformable items, and articulated items (Fig. 3) and retain the keypoin--conditioned pi cick skill from Section 4.1. We summarize the findings in Table with 26 trials per category of items, noting that KITE can achieve precise semantic grasping with generalization to unseen object instances (Q2, Q3). We omit comparison to PerAct as its difficulties
7

---

with bick-and-place in the tabletop environment are only exacerbated in the semantic grasping setting where specific intra-object features matter. In the trials summarized in Table 2, KITE outperforms RobotMoo, suggesting the utility of keypoints to pinpoint specific object parts compared to coarse segmentation masks or bounding boxes oulput by VLMs. We also observe that the majority of KITE's failures in this setting are due to misinterpretations with symmetry (i.e. grasping the left instead of right handle of the pliers), rather than completely erroneous keypoint is common in RobotMoo. We posit that this could be alleviated with more diverse data of object semantic variations.
Rigid Tools
Deformable Objects
Articulated Items
Failures
A
B
C
Seen Instances Unseen Instanees
KITE 0.77
0.77
0.70
KITE 0.70
0.54
0.70
All
RobotMoo 0.23
0.35
0.19
36
7
Table 2: Semantic Grasping Results:
Across 20 total objects. 3 diverse object categories, and 26 trials per method per category. KITE achieves the highest rate of pick success for various object semantic features (Fig. 3), and with the least severity of failures. We categorize failure modes follows, with (A) denoting symmetry error (picking the left instcad of right handle), (B) representing a grounding error with an erroneous keypoint prediction, and (C) indicating a manipulation failure (wrong inferred orientation or slip during grasping).
Coffee- Making: Finally, we seek to answer whether KITE
reorient mug pour .cup refill keurig
load pod
KITE
8/12
9/12
8/12
9/12
can execute fine-grained be-
Table 3: Coffee-Making Results: KITE handles fine- fin-grained manipulation
haviors from instructions (Q4) by studying coffee-making scenario with four skills: {reorient mugg pour cup, refill keurig, load pod] (examples in Fig. 1). We evaluate on the same object instances seen in training, but subject to spatial variations and language variations (i.e. 'Place the blue/ned/green/brown pod in the machine', 'Pour the red/grey pitcher into the mug/Keurig refill area, 'Place the cup/mug that's 's sideways right-side-up.') Even for these very fine-grained motions, KITE is able to follow instructions with 67-75% success (Table 3). The main failures reside with low-level control errors rather than grounding, such as partial coffee pod insertion, misaligned mugs and pitchers during pouring, or slippage during mug reorientation. This suggests that the individual skills could benefit from scaling up demonstration collection, while retaining the existing grounding modules.
across 4 skills requiring highly precise manipulation.
5 Discussion
Summary In this work, we present KITE a framework for semantic manipulation that identifies 2D task-relevant keypoints and extrapolates 6-DoF actions accordingly. By leveraging 2D keypoints to precisely localize semantic concepts, KITE is adept at recognizing semantic labels both across different object instances and on different regions within the same object. KITE does action -ratiing by drawing from library of parameterized skills. Empirically, language-basn based manipulation frameworks along the axes of scene semantic awareness and object semantic awareness. We also find that KITE can be trained from orders of magnitude less data and with large precision gains over end-to-end approaches, while exhibiting an ability to generalize and operate over extended horizons. Finally, we show that KITE offers flexible interface for instruction- following, including tabletop rearrangement, fine-grained grasping, and dexterous manipulation.
find that KITE surpasses existing
Limitations and Future Work One limiting factor of KITE is its reliance building a library of skills. However, we show that a relatively small library of deypoont-parameterrzed skills is expressive enough to accomplish many standard manipulation tasks with object variations over an extended horizon. Additionally, KITE requires less than 50 demonstrations per new skill, meaning that adding new skills is fairly straightforward. We also note that KITE's grounding module is trained from scratch. As VLMs continually improve and in the future may able to pinpoint keypoints in images, it would be interesting replace or enhance KITE's grounding module with these models. Additionally, we acknowledge that KITE currently executes skills in an open-loop manner parameterized by waypoints. In, the future, we excited to extend KITE's skills with closed-loop feedback and extend the complexity of these skills to even more dexterous settings.
8

---

Acknowledgments
This work is in part supported by funds from NSF Awards 2132847, 2006388, 2218760, as well as the Office of Naval Research, and FANUC. Toyota Rescarch Institute provided funds to support this work. We also thank Sidd Karamcheti and Yuchen Cui for their helpful feedback and suggestions. Priya Sundaresan is supported by an NSF GRFP.
References
[1] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition,, pages 9044-9053, 2021.
[2] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Roboticy and Automation (ICRA), pages 13438-13444. IEEE, 2021.
[3] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. arXiv preprint arXiv: 003,09312, 2017.
[4] N. Correll, K. Bekris, D. Berenson, O. Brock, Causo, K. Hauser, Okada, A. Rodriguez, M. Romano, and R. Wurman, Analysis and observations from first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1):172-188, 2016.
[5] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1388 2021.
[6] Minderer. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Amna, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part X, pages 228-755. Springer, 2022.
[7] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühh, and Misra. Detecting twenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022.
[8] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Maryyng dino with grounded pre-training for open-set object detection. arXiv preprint trXiv:2303.05499 2023.
[9] A. Stone, T. Xiao, Y. Lu. K. Gopalakrishnan, K.-H. Lee, Q. Vuong, Wohlhart, Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained lisiun-language models. arXiv preprint arXiv:2303. 00905, 2023.
[10] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[11] M. Shridhar, L. Manuelli, and Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[12] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning, pages 991-1002. PMLR, 2022.
[13] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 2022.
[14] S. Stepputtis, J. Campbell, M. Phielipp. S. Lee, C. Baral, and H. Ben Amor. Language-conditioned imitation learning for robot manipulation lasks. Advances in Neural Information Processing Systems, 33: 13139-131 13150, 2020.
[15] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning. arXiv preprint arXiv:2204.06253 2022.
[16] F. Codevilla, Müller, A. López, Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4693-4700, 2018. doi: 1 109/CCA,2001 84664487.
[17] L. Shao, T. Migimatsu, Q Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14), 2021.

---

[18] A. Akakzia, C. Colas, P.-Y. Oudeyer. M. Chetouani, and O. Sigaud. Grounding language to autonomously- acquired skills via goal generation. In ICLR 2021-Ninth International Conference on Learning Representa- tion, 2021.
[19] P. Goyal, R. J. Mooney, and S. Nickum. Zero-shot task adaptation using natural language. CoRR. abs/2106.02972. 2021. URL https: // arxivv.vvvvv hwtp:/orggaaaab/aeeeee 02972.
[20] H. Hu, D. Yarats, Q. Gong, Y. Tian, and M. Lewis. Hierarchical decision making by generating and following natural language instructions. Advances in neural information processing systems, 32, 2019.
[21] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, Herzog, D. Ho, Ibarz, A. Impan, Jang, R. Julian, al. Do can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[22] Huang. F. Xia, T. Xiao, H. Chan, J. Liang. P. Florence, A. Zeng. J. Tompson, Mordatch, Y. Chebotar, P. Sermanet, Jackson, Brown, Luu, S. Levine, K. Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022. URL https: / /ttp///ppeeeeeeeeeeeeeeeeeeeeeee wwt./fernnmm//// Sirrruidddd33P250005
[23] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.13300 2022.
[24] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118 9147. PMLR, 2022.
[25] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans, arXiv preprint W:Xi::2303.2213, 2023.
[26] Y. Jiang. A. Gupta, Z. Zhang, G. Wang. Y. Dou, Y. Chen, L. Fei-Fei, A.
Anandkumar, Y. Zhu, and L. Fan. Vima: General rohot manipulation with multimodal prompts. arXiv preprint arXiv:22 10.000094 2022.
[27] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman. B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint 201v:2209.07753. 2022.
[28] A. Stone, T. Xiao, Y. La, K. Gopalakrishnan, K.-H. Lee, Q Vuong. P. Wohlhart. B. Zitkovich, F. Xia, C. Finn, and K. Hausman. Open-world object manipulation using pre-trained vision-language model. In arXiv preprint, 2023.
[29] P. Dayan and G. E. Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[30] L. X. Shi, J. Lim, and Lee. Skill-based model-based reinforeement learning. 6th Anmual Conference on Robot Learning, 2022. URL http:::/ pet/forumm http//orrmmidd/i// seerrruiiidiiyyyee wet.ummmdddiiiyyeeeeeee Vxy2e0601U,
[31] S. Mirchandani, S. Karamcheti, and D. Sadigh, Ella: Exploration through learned language ab- straction. M. Ranzato, A. Beygelzimer, Dauphin. P. Liang, and W. Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 29529- Curran Asso- ciates, Inc., 2021. URL https: h/pp://eedng/geeee/ htw.:eeeediinnnnnnni............ wttppps.ccppppeeeeeeee pta.ee//ppper/222/EE12 161154417c466588 115556615830 113111
saper.pdf.
[32] C. Agia, T. Migimatsu, J. Wu, and J. Bohg. Stap: Sequencing task task-agnostic policies. arXiv preprint arXiv:2210.12550, 2022.
[33] S. Nasiriany, H. Liu, and Y. Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[34] M. Dalal, D. Pathak, and R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. In NewrIPS, 2021.
[35] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. CoRR, abs/ 100./9993..99 2019. URL http:// // //axxivvvvv///////// org/abs, abb//90b/9 .1903.0977..
[36] S. Belkhale and D. Sadigh. Plato: Predicting latent affordances through object-centric play. In Conference on Robot Learning, pages 1424-1434. PMLR, 2023.
[37] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1112 16199, 2022.
10

---

[38] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic
manipulation. In Robotics Research: The 19th International Symposium ISRR, pages 132-157. Springer, 2022.
[39] C. Wang, M. Chai, M. He, D. Chen, and J. Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. CoRR, abs/21 12.05139, 2021. URL httpt::t//////.... http:///wwww.....a/aaaaa//aaataa http//ab/2211...00
[40] Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, Lerf: Language embedded radiance fields, 2023.
[41] X. Ma, D. Hsu, and W. S. Lee. Learning latent graph dynamics for deformable object manipulation. CoRR, abs/2104. 12149, 2021. URL attps:///axxia org/ /t//abb////// 12149.
[42] H. Ha and S. Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robotic Learning (CoRL), 2021.
[43] V. Viswanath, J. Grannen, Sundaresan, Thananjeyan, A. Balakrishna, E. Novoseller, J. Ichnowski, Laskey, E. Gonzalez, and K. Goldberg. Disentangling dense multi-cable knots. In 2021 JEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 3731-37355 2021. 1oi:10.1109/ IROS51168.202 2.36666977
[44] P. Sundaresan, J. Grannen, B. Thananjeyan, Balakrishna, J. Ichnowski, E. R. Novoseller, M. Hwang, Laskey, J. E. Gonzalez, and Goldberg, Untangling dense non- non-planar knots by learning manipulation features and recovery policies. CoRR, abs/2107.0894 08942, 2021. 08942.
URL https: //tp:::////xxvvv org/ /a/ab///11 2200/
[45] P. Sundaresan, S. Belkhale, and D. Sadigh. Leaminng visuo-haptic skewering strategies for robot-assisted feeding. In 6th Annual Conference on Robot Learning, 2022. URL https::/coporrevee...ee www.:/wwwwww....w.eeeeeeeeeeeeee hwt.:/ee//ooeeeee ner/forum?
id=1Lq09gVoaTE.
[46] C. Wang. R. Wang, A. Mandlekar, L. Fei-Fei, S. Savarese, and D. Xu. Generalization through hand-eye coordination: An action space for learning spatially
spatially-nvvariant invariant visuomotor control. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 9913-8920. IEEE, 2021.
[47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[48] J. Wu, Antonova, Kan, M. Lepert, Zeng, Song. Bohg. S. Rusinkiewicz, and Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint orXiv22305.05655 2023.
[49] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[50] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnct++: Deep hicrarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[51] A. Jaegle, S. Borgeaud, J.B. Alayrac, C. Doersch, lonescu, Ding. S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14995 2021.
[52]
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint 2rX1v:2304.02643, 2023.

---

KITE: Keypoints + Instructions To Execution Supplementary Material
In this section, outline additional details regarding the implementation of KITE, the real-world environments studied, and qualitative results of all methods. Please refer to our website to best understand the task diversity and qualitative performance through videos of KITE performing real- world semantic manipulation.
A
Implementation Details
A.1 KITE
As discussed in Section 3.4, KITE trains PointNet++ model output all K relative waypoints and one-hot waypoint index for each point in the point cloud. The auxiliary one-hot classification output layer classifies which waypoint each point in the point cloud is most relevant for manipuation (nearest to). In practice, many skills like pick or place can be parameterized by just K = waypoint (where grasp, where to place). In this case, the one-hot classification output is reduced to binary classification of which points are near the graspable object or target location, respectively. For more general skills parameterized by K waypoints, predictions per-pos per-point by taking the loss of the predicted and ground truth gripper pose for that point compared to ground truth. This loss provided in the main text in Eq. (2).
supervise the K-th end-effectee pose
"Grab lemon" "
"Shut top drawer" "Take a peek at 2nd shelf" "Hand me the blue lego"
"laces"
"joint"
"arm" "
"hammerhead"
"left
"tail"
"Put the green pod in"
"Refill the Keurig with the red pitcher"
"Flip up the white mug"
Figure 4: KITE Grounding Predictions: KITE's grounding model is able to accurately predict keypoints for both scene semantic instructions (e.g., "grab the lemon" and "put the green pod in") and object semantic instructions (c.g.. "shut the top drawer" and "take peek at the 2nd shelf".
To artificially scale the data KITE's grounding module is trained on, we apply various random colorspace and affine transformations the dataset collected with skills to augment 8X before training. We train the grounding module and each skill policy using the Adam optimizer with learning rate 0.0001, which takes hours and hour an NVIDIA GeForce GTX 1070 GPU, respectively.
A.2 PerAct
For each evaluation environment, we consolidate each the skill datasets used to train KITE into one multi-task dataset with which to train PerAct. The input to PerAct is 753 voxel grid (although
12

---

the original PerAct implementation used 100$ voxel resolution, we adjust our workspace bounds accordingly retain the same voxel resolution). We represent waypoints 1-hot encodings in this voxel grid, end-effector orientations as a discrete prediction over bins each for yaw, pitch, and roll, and the gripper open/closed state binary indicator variable in [1]]. For each environment, we train PerAct for 7200 iterations.
PerAct Ground Truth
"Open the
top shelf
top shelf
middle cabinet
middle cabinet
Figure 5: PerAct Predictions: We visualize PerAct predictions the task of opening cabinet with multiple drawers. Although PerAct exhibits some reasonable predictions (last column), it struggles with localizing the correct handle (Ist, 3rd columns). Even when localizing the correct handle (2nd column), the slight imprecision of the predict ground truth action lead to downstream manipulation failure.
A.3 RobotMoo
We note that the original implementation of RobotMoo leveraged the RT-1 [13] skill learning framework. This set of skills trained with months of data collection, amassing thousands of trajectories for 16 object categories, and RobotMoo further extended these policies to 90 diverse object categories, As this is not reproducible in our setting, we implement RobotMoo by using KITE's library of skills, but conditioning them VLM predictions instead of our keypoints. Specifically, while the original RobotMoo implementation used OwLViT, we use more recent state-of-the-an open vocabulary object detectors Grounding DINO [8] and Segment Anything [52] jointly. With these models, we obtain segmentation masks for objects referenced in input instruction. We take
center pixel coordinate of these segmentation masks as input to our acting module rather the output of @ground.
white hrbe
White har
grear
ma
Figure 6: RobotMoo Predictions: We visualize the predictions for RobotMoo's 's perception stack on tabletop instruction following images. Although RobotMoo exhibits decent scene awareness and an ability to localize different object instances, it struggles with object semantics. Specifically, RohotMoo struggles to localize all the different drawers in the scene, let alone distinguish amongst the top vs. middle vs. bottom handles.
B
Real- Experimental Details
B.1
Task Variations
For each real environment used in evaluation, we stress-test all methods across task variations ranging from diversity in the input language instructions to amount of clutter and distractor objects. We summarize each axis of variation for long-horizon tabletop instruction following (Table 4), semantic grasping (Table 5), and aafeee-making (Table 6) below.
13

---

Variations
Tier
Skill
Language
Scene
open
"Open the [oop/middle/bottoon cabinet'
randomized position of cabinet randomized position cabinet randomized object positions randomized object positions randomized position distractor objects (clothes strewn) randomized position + distractor objects (clothes strewn)
close
'Close the [top/middlc/hottoom drawer'
pick
'Pick up the remookkrrewdrvver/lggooooo pe marker]
place
'Put the [...] [away. the [...] drawer, in the bowl]'
open
Open the [top/middle/bootoon cabinet"
close
'Close the [lop/miiddlc/bottmm]] drawer'
pick
'Pick up the (emmon/kreewdiivv//gggoooooo
marker, eggplant. carrot, corn, lime, scissors.
randomized position + scene clutter
ketchup, coffee pod]
place
Put the [...] (away, in the [..]] drawer, in the bowl]'
randomized object positions
scene clutter
open
Yank open the top drawer' "Give the 2nd drawer ugg"
randomized position
distractor objects (clothes strewn)
Takke peck at the 3rd drawer pls"
Can you check the top drawer?'
close
'Close
distractor objects (clothes strewn)
'Give it a push'
'Let's shut the drawer'
'Go ahead and close the top drawer'
'Close any open drawer'
'The one 3rd from the bottom needs to be shut'
pick
'Grab me the [...]'
randomized object positions + scene clutter
'Do me a favor and get me the I...I'
'Can you pass me the [..]]?'
'Get the I...I'
'Locate the [...]'
'Could you hand me the [...] please'
place
"Grab the [...] and put it [...]'
randomized object positions + scene clutter
"Take the !...] and place I...]'
Piic up the [...] and put it [...]'
'Ftchh the [..]] and drop it [...]' 'Plop the 1...] into the bowl"
Table 4: Tabletop Instruction Following Environment Variations
Category
Object hammer T-ooo
Language Variations
middle, end. tooltip. hammerhead, metal, wooden handle, center, tip
Rigid Tools
Tools
left side, left right side, right bottom handle, top handle
Deformable Objects
shoe
heel, toe, back, front, shoelaces, laces, lace-up head, nose, car, tail, belly, foot, arm, leg, tummy, elephant trunk joint, left handle, right handle, top handle, bottom handle joint. left handle, right handle, top handle, bottom handle
stuffed animal
pliers
clamp
Articulated Items
scissors
joint, left hole, right hole, smaller hold, bigger hold, larger hold cup. expo label. center, label. end
marker twist-off cap
Table 5: Semantic Grasping Environment Variations
B.2 Primitive Instantiations
In this section, we describe the instantiation of our library of skills for each real-world environment:
Tabletop Instruction Following: We parameterize each skill in the tabletop manipulation setting via a single waypoint (z. y, 2, ub, f, Φ) specifying the primary point of interaction.
open: With its gripper open and predicted orientation (1é, 0, d). the robot approaches 5cm. away from closed drawer handle at predicted position (xr, y. Next, the robot moves to (x, y, 2) in the same orientation and closes the gripper to grasp the cabinet handle. Finally, it executes linear pull by moving to the approach position, keeping the orientation fixed, before releasing the handle.
close: The robot approaches 5cm. away from an opened drawer handle at position (I, y, 2) with orientation (v), 0, Φ), then executes linear push towards (a, y, 2) close the drawer.
pick: The robot approaches the object located at (ar y, z), closes its gripper, and lifts 5cm. place: While holding an object grasped with the pick primitive, the robot moves 5em above the desired place location (Ir, y; 2) with orientation (1/, 0, Φ) and opens the gripper to its maximum width, releasing the object.
14

---

Coffee-Making In the coffee- masking tasks, we implement a library of 4 skills which test KITE's ability to handle precise dynamic movements. Since pour_ refill keurig, and load pod all involve grasping, we finetune the pick skill from tabletop instruction- foollooing with 50 demon- strations across pitchers and coffee pods, respectively. Then, we can parameterize each skill with single waypoint (x, M/, 2, 1/, 0, () follows:
reorient_mm The robot attempts to grasp a mug, initially oriented sideways, with pose before resetting to a canonical upright (untilted) end-effector pose.
refilll keurig: After grasping a pitcher, the robot moves to position (1r, y; =) denoting the position of the vessel be poured into (cup or refill compartment of Keurig). Starting from an untilted end-effector pose, the rohot gradually rotates at a constant velocity to (15, 0, .),, denoting the final pour orientation.
pour_cup/ httpprrffllll
load_pod: After grasping coffee pod with the pick primitive, the robot moves 2 cm. above (z. y, 2), the sensed position of the K-cup slot with orientation (t), 0). Next, the robot releases its grasp to drop the pod into the compartment. As this task requires high precision, it is often the case that after releasing the pod, it is not completely inserted or properly aligned. Thus, the load pod primitive moves downward attempt to push the pod into place. We note that we not evaluate this skill with real liquids for safety reasons, but measure success in terms of visual alignment between the pitcher and vessel.
additional 2cm in
Skill
Language
Scene
Flip the mug right-side up"
randomized mug position, roll (-T/2,
Put the mug upright"
reorient mug
'Grab the mug and put it right-side-up'
"Can you place the mug right side up?"
'Get the mug that's 's laying flat
and flip upright
'Fill up the mug that's right-side up" glass'
randomized pitcher (red gray) cups (compostable, Dixie, mug)
'ouur
pour_cup
'Pour the red pitcher into the mug'
'Grab the silver-handle pitcher and fill
up the brown cup'
RRffill Dixie cup with the red pitcher'
"Refill the espresso machine'
randomized pitcher (red / gruy) randomized Keurig pose
defill.keurig
'Grab the red/silver pitcher and
'fill up the water compartment"
'I Loa the hlue K-cup'
randomized coffee pod (red/blue/greee/rrwwn)
'Can you put the red pod in?'
+ randomized Keurig
poa
'Insert the green pod'
'Start a brew with the hrown pod'
Table 6: Coffee-Making Variations
B.3 LLM Prompting
Skill Label Inference: In this section, we briefly outline how KITE retrieves the skill label 4 for input instruction i, via LLMs. Below, we provide sample prompt which we feed as input to text-davinci-00s obtain 4 tabletop instruction following setting.
Listing 1: LLM Prompting for Skill Label Inference
i_t_=uinput (" ('Enter_inatrucction: unn
Input: :
Input:. http::/PPPuuthhhsrrrreeeee .ie.rree...eeerrrree..aaaa (utput:.."pioon "utpp::..""ppee""""p"""""""
Input:.."Ple grab. me. the screvdii.re. itt. Outpet:_""ppcc'"_ "p"lacee
10
11
Output:-["graas: []]=ppp]]]
15 t
15

---

16
prrputt:""piice "p"acee""
Input: tttpep..t..e.e....dr... ("tput:
Input:.."Pla shut. the.. drawer" Output:-["ccose")
22
24
25
Output::."ppcck" -p=paace"]
26
27
28
Output:-["pick' [="lace"]]
29
30
Input:_'ke:
Output:
32 "i_t
16


Reviews:
[{"review_id": "Xt9W1Ipyzum", "reviewer": "Reviewer_C4rM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to perform 6-DoF manipulation from RGB-D + language instructions. The motivation/technical problem: existing instruction-following manipulation pipelines (1) don’t effectively allocate attention toward context at both the scene and object levels and (2) output 6-DoF gripper trajectories with limited expressivity. The proposed pipeline aims to address this: (1) from RGB-D + language, a discrete skill and a region for executing the skill are selected (via keypoint prediction, similar to CLIPort + SayCan), (2) from a local point cloud near the predicted region, a network predicts a dense sequence of \"waypoint poses\" for the gripper to follow. The method is compared to baselines that map RGB-D + language directly to gripper poses and that use frozen vision-language models for language grounding. The results highlight the benefits of continuous waypoint pose prediction, especially in enabling multi-step tasks, and the use of keypoint-prediction grounding over frozen VLMs.", "summary_of_recommendation": "While the details of the main novel technical components should be made more clear, the full-stack language-to-action system implementation is impressive, the addition of hierarchy, local geometry conditioning, and more expressive low-level motion prediction is valuable, and the real-world demos are very well done. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Address weakness comments above (more clearly explain local skill policies)\n- On website example, PerAct fails on \"Open the bottom drawer\" with a \"wrong handle prediction\". Why wouldn’t KITE have the same issue?\n- How would the waypoint pose policies behave for, i.e., sweeping with a broomstick (as shown in PerAct)?\n    - In PerAct, global attention is used to directly predict the next gripper pose (so this can be a function of something far away, like the tip of the broom). In KITE, does representing waypoint poses as offsets from the point cloud limit the ability to handle a task like this (where the \"task-relevant region\" is far from the gripper, such as if it's holding a tool)?\n- How does the method deal with the K different (sequential) waypoints? Is K manually specified for each skill?\n- Please improve notation/terminology consistency - you have “keypoint-conditioned skills\" and “demonstrations of keypoints/waypoints\", but these waypoints are not just points (they are poses)\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of strengths and weaknesses**\n\nStrengths:\n\n- The real demonstrations are impressive and compelling. Any pipeline that enables the diverse kinds of capabilities shown in the paper (semantic grasping, multi-step tabletop manipulation) directly from language + RGB-D on real-world objects is noteworthy. \n- The paper showcases an overall system implementation that works and that is (mostly) described in sufficient detail to understand how to reproduce and build upon it. \n- The main novel technical components (the locally conditioned \"waypoint-pose predictor\"/skill policy) make sense, both in their design and in the way they're integrated into the overall pipeline.  \n\nWeaknesses:\n\n\n1. In my view, beyond the good overall system engineering, the main technical innovation is the local \"waypoint pose\" prediction. Thus, it’s a critical part of the paper to understand, but as it's written, I found it to be the most difficult part of the paper to understand (at least in terms of technical details). Some more detailed comments are mentioned with regard to this in the \"clarity\" section below. \n2. A related weakness is the framing of the paper and the difficulty in knowing what the main takeaway messages are. The results are good and the gist of the pipeline is not difficult to understand, but there are so many terms and claimed advantages (*long horizon reasoning*, *generalization*, *instruction following*, *parameterized skills are better than end-to-end*, *need for both scene- and object-level semantic*) that its hard to know what I learned as a reader of the paper. \n    - If we are to break it down a bit, one perspective (that I had as a reader) is below. Take it for what it's worth, but the general suggestion is to find a way to make it easier for a reader to walk away from the paper with a more clear \"take-home message\" in mind. \n        - The main new technical component is the local skill policies - predicting a sequence of waypoint poses from a local point cloud, parameterizing these as offsets from the point cloud points, using a different pose prediction network for each skill, and training these from a modest # of demonstrations. \n        - The motivation for building these local skill policies is to utilize them within an \"RGB-D + language -> 6-DoF EE trajectory\" instruction-following manipulation framework. By doing so, this will expand upon the capabilities of previous instances of \"manipulation from language instructions\" pipelines, so that (i) the executed motions can be more diverse, expressive and accurate, (ii) they can integrate with more kinds of instructions (specifying a desired part to interact with), and (iii) by being more accurate, they can better succeed in multi-step tasks (where compounding errors can cause failure). \n\n\n**Details**\n\n*Technical quality -* Good. Strengths: The authors show an impressive full-stack implementation and very nice real-world demos. The point cloud-conditioned skill policies are a valuable technical contribution addition to the language-to-action stack. Weaknesses: More could be done to evaluate the ability of the low-level skills on their own, i.e. how well can they generalize to different objects/shapes/poses of objects, but the focus is more on how these skills integrate into the full stack (rather than trying to make the low-level skills perfect) so this is mostly okay.\n\n*Originality -* Good. Strengths: Incorporating the local point cloud for conditioning a local skill policy that outputs 6-DoF \"waypoint pose\" is a nice addition to existing full-stack language-conditioned manipulation pipelines, and showing a clever way to parameterize it (i.e., by predicting offsets from the local point cloud) is also valuable. Weaknesses: PerAct/CLIPort part is the same as before, SayCan part is the same. \n\n*Clarity -* Good. Strength: The overall pipeline makes sense (high- vs. low-level, language to keypoint + skill, local point cloud to dense waypoint-pose sequence). Weakness: The most critical part of the method to understand (the skill-conditioned waypoint policies) was pretty confusing in terms of some key technical details. \n- Details on confusion:\n    - My confusion comes from how the local point cloud + mask are used to output the waypoint poses. I cannot tell if it's (1) or (2): (1) The *whole* scene point cloud is used to predict the offset poses, and the predicted mask is appended to this entire point cloud as an extra binary feature, (2) Only the *subset* of the point cloud that comes from the predicted mask is used to predict the offset poses. \n    - i.e., in the bowl example that's discussed, you say “this yields a set of ‘candidate’ points for consider for interaction”… and “the bottom of the bowl points predict the points on the rim”, which makes me think you mask out the local point cloud and **only** use these to represent the gripper pose predictions. \n    - While this might seem like a minor detail, there are two reasons why I believe it's a critical point to clarify. First - masking + conditioning on the local point cloud alone seems like it would support significantly better generalization. Second - representing the poses as offsets from the local point cloud alone seems like it could make the task more difficult, since the gripper might need to be quite far from these masked points. This has implications for tasks where the gripper is using, e.g., a large tool, such as the broomstick task shown in PerAct (this is mentioned in the \"questions for rebuttal\" section below). \n\n*Impact/Significance -* Good. Strength: Language-conditioned pipelines are here to stay, so providing more instantiations of how they can work and perform more flexible behaviors is valuable. Further, introducing different kinds of hierarchical decomposition into such pipelines (vs. just end-to-end, direct execution of voxel-predicted actions, etc.) is also beneficial. Weakness (minor): This particular method for low-level skills will probably be built on and improved, but showing the impact of a locally-conditioned component is likely to help expand others’ minds for other ways to do this.\n\n\n**Other comments, points of confusion**\n- In introducing the paper + method, the claimed advantage of “*capacity for long-horizon reasoning”* felt slightly off-putting. IIUC, the main part that enables this is the LLM instruction following, which isn't really a contribution of the work, and the ability to execute individual steps more accurately. This is a contribution of the work, but the claim of enabling \"long-horizon reasoning\" should then be situated more clearly to this improved ability to perform individual steps more successfully. \n- In the related work, this comparison to RL/IL was pretty confusing and hard to unpack: “*Imitation learning (IL) aims to learn skills from demonstrations in a more sample efficient manner than RL, but these skills still fail to generalize to scene perturbations [35, 36]. Furthermore, in both IL and RL, connecting learned skills to precise language in a generalizable fashion is an open challenge. KITE avoids learning skills from scratch and instead defines a library of keypoint-conditioned skills, where the exact parameters of each skill are learned from demonstration.”*", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "EAlsbmQtalL", "reviewer": "Reviewer_pCg6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes KITE, a method that maps instructions to robot manipulation actions. It consists of two stages: a higher level stage proposes keypoints to act on and skills to execute. Each skill consists of a learned keypoint policy and a controller to generate low-level motion. The experiments show that KITE can solve a number of challenging rearrangement tasks (including openable containers) on a physical robot. ", "summary_of_recommendation": "My rating is conditioned on improved presentation of the paper", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Main question I have is regarding comparison to PerAct and the way PerAct has been used in the experiments (see the Review section)\n\nThe below paragraphs provide some feedback from my subjective experience reading the paper. I hope it helps the authors re-organize and improve the presentation. These are just suggestions, there are likely other ways to improve presentation.\n\nSection 3.1, starting line 141 it describes inference, saying that \"at time t ... we want to know which skill to execute\", then \"infer a 2D keypoint\", then suddently \"find a waypoint policy pi\" - do we learn a new policy at every timestep? Do we search among a set of policies? What does \"find\" mean? Does the sentence starting on line 144 refer to learning instead of inference process? That can't be, because on line 149 it says \"restart the above process\", which means everything above was the same process. Additionally, it is confusing when the inputs and outputs of policies and controllers are defined inline in a paragraph that describes an inference process. It would be much clearer to separate definitions, task formulation, inference process, and learning in separate sections and paragraphs. \n\nI suggest moving Section 3.2 after Section 3.3. It refers to things like \"grounding modules\" and \"acting modules\" that have not been clearly defined thus far. Also, it is hard to understand the motivation and formal structure of the data being collected before knowing the inputs and outputs of the modules being trained.\n\nI would say that language grounding refers to converting human-understandable text to a machine-understandable representation for a given system and context. Under this definition, I find it confusing that \"Skill selection\" is under \"Acting Module\", but keypoint proposal is under \"Grounding module\". Both skill selection and keypoint proposal is language grounding. \n\nSection 3.4 again surprisingly mixes learning and inference. \"Skill selection\" talks about inference (which as I said I think should be under the grounding module), the following paragraph is on \"learning\" and the one after that on \"inference\" again. I think moving \"Skill selection\" to the grounding module will fix one of these issues. The paragraph starting on line 196 internally also mixes learning, inference, and data processing in a confusing way.\n\nLine 215 - if you use an equation, you need to define all symbols in the equation. I think there three options:\n1. Define all symbols\n2. remove the equation\n3. reduce the equation to have less symbols.\n\nLine 184 - typo\nLine 35 - is pick and place a benchmark?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Advantages:\n- The problem of mapping natural language instructions to robot manipulation is hard, and this paper solves it reasonably well.\n- The method offers generalization to unseen tasks (through LLMs) so long as the tasks can be broken down to a set of skills. It also offers generalization to unseen objects (through the use of CLIP and the particular way the keypoint policies are conditioned).\n- Overal convincing real-world results showing efficacy of the system compared to prior state of the art methods.\n\nDisadvantages:\n- The writing is currently confusing, mixing definitions, inference, learning and data processing in text without sometimes a coherent structure. This makes it hard to keep track of what is the proposed method vs what is auxiliary processing needed to instantiate it in this paper, what are the inputs and outputs of the different components, and what processing happens during data preparation, training, and inference stages. It's not all bad though - all the needed information is there and the paper can be understood sufficiently well, but it doesn't make it easy - I almost need to draw diagrams while reading the paper to make sense of the system.\n- Grounding module identifies a single keypoint for instruction and does not appear to be able to ground instructions referring to more than one thing.\n- It would be good to compare the against PerAct on RLBench, a more standard and replicable benchmark. Or otherwise provide replicable simulation experiments.\n- Given PerAct's good performance before, it is hard to understand why it performs so poorly in these experiments. I suspect it is not used as intended. PerAct is a keypoint policy, so it should be compared in an ablation when used as a keypoint policy within KITE. Comparing KITE to PerAct directly doesn't really make sense, since PerAct is not an instruction-following policy (it takes instruction input only to disambiguate tasks).\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "BYcKla2Hab", "reviewer": "Reviewer_nGAo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper advocates the use of keypoints in 2D images to help disambiguate language for downstream language understanding such\nas selecting and parameterizing skills.", "summary_of_recommendation": "I am not impressed by the evaluation.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would suggest have a much more extensive set of skills and objects in the evaluations, and maybe even more evaluation domains.\nThese could be evaluated in simulation.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I would think that a paper on language understanding would have a much richer set of things the language could refer to.\nEach of the three evaluation domains had about 4 skills to talk about, not 400. There seem to be up to 10 or so objects or parts\none could refer to, although this was not clear in the paper.\n\nWhy is the success rate in all cases 70-75%? Why is that range so low? And why are rates similar\nacross very different tasks with \"varying tiers of difficulty\"? Seems weird.\n\nI am not convinced that selecting regions is more difficult than selecting keypoints, and in many cases I would think regions would\nbe more useful.\nWhere does the keypoint go if:\n- there are multiple handles on the drawer (many have at least two).\n- there is a long graspable area the width of the drawer.\n- you have cabinets doors or drawers with no handles, which one pushes to open. \nhttps://www.youtube.com/watch?v=B2xYIQhZloI\nRather than focusing on \"keypoints\", don't you want to focus on \"saliency\"?\nAs the paper says, one is directing attention to particular features, not\ncommanding contact points.\n\nWhat is the evidence for \"capacity for long-horizon reasoning and generalization.\"?\n\nSmaller points\n\n\"We assume that at least one “grounding” camera can partially see all relevant objects\"\nThis seems unnecessary, and difficult to obtain. Why can't a virtual grounding image or set of images be synthesized instead?\n\nWhat is the difference between a waypoint policy and a controller to move\nbetween waypoints? Does this detail matter?\n\nIn training, why aren't new observations and point clouds obtained at every\npoint along the robot trajectory, or at least at the waypoints?\nWhy must a sequence of future waypoints be predicted?\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary, Strengths and Weaknesses
This paper introduces KITE, a two-stage method that utilizes keypoints to map instructions to robot manipulation actions. Experimental results demonstrate that KITE outperforms models that use visual language or end-to-end control. It achieves high success rates due to precise keypoint-driven execution in real-world environments. While the formulation of the method needs further refinement, the method is novel. Real-world demonstrations showcase the effectiveness of the approach.

The reviewers agree in their recommendation to accept the paper because most of the concerns have been addressed. I agree with their consensus.
Decision: Accept (Poster)