ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Authors: Wenlong Huang
Chen Wang
Yunzhu Li
Ruohan Zhang
Li Fei-Fei
Venue: CoRL 2024 (2024)
Keywords: Structural Representation
Model-Based Planning
Foundation Models
TL;DR: Large vision models and vision-language models can generate a spatio-temporal series of constraints that map keypoints to a numerical cost, which are optimized to obtain actions for multi-stage, in-the-wild, bimanual, and reactive manipulation tasks.

Abstract: Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep are expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observation. We present system implementations on a mobile single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models.

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Wenlong Huang', Chen Wang"', Yunzhu Li Ruohan Zhang', Li Fei-Fei! Stanford University CColumbia University rekep-robot.github.i
Abstract:
Representing robotic manipulation tasks constraints that associate the robot and the environment is promising way to encode desired robot be- haviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimiz- able by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), visually-grounder repre- sentation for constraints robotic manipulation. Specifically, ReKep is expressed Python functions mapping set of 3D keypoints in the environment numer- ical cost. We demonstrate that by representing manipulation task sequence of Relational Keypoint Constraints, we can employ hierarchical optimization procedure to solve for robot actions (represented by sequence of end-effector poses in SE(3)) with a perception-a actioon loop at a real-time frequency. Further- more, order circumvent the need for manual specification of ReKep for each new task, devise an automated procedure that leverages large vision models and vision-language models produce ReKep from free-form language instruc- tions and RGB-D - observations. We present system implementations single-arm platform and stationary dual-arm platform that can perform large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models.
wheeled
Pour tea into the cup.
Vislon-Language Model
Large Vision Model
Relational Constraints
Multi-Stago
In-The-Wild
Input: O
Output: Cost
Optimization Solver
Actions
Bimanual
Reactive
Figure 1: Relational Keypoint Constraints (ReKep) specify diverse manipulation behaviors opti- mizable spatio-tempont series constraint functions operating on semantic keypoints. one ReKep first constrains the grasping location at the handle of the teapot (blue). A subsequent ReKep pulls the teapot spout (red) towards the top of the cup opening (green) while another ReKep constrains the desired rotation of the tcapot by associating the vector formed by the handle (blue) and the spout (red).
pouring task,
Introduction
Robotic manipulation involves intricate interactions with objects in the environment, which can of- ten be expressed as constraints both spatial and temporal domains. Consider the task of pouring tea into cup in Fig. 1: the robot must grasp at the handle, keep the cup upright while transporting
Denotes equal contribution. Correspondence to Wenlong Huang <wenlongh @ stanford.cduu>.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

it, align the spout with the target container, and then tilt the cup at the correct angle to pour. Here, the constraints encode not only the intermediate sub-goals (e.g.. align the spout) but also the transi- tioning behaviors (e.g., keep the cup upright in transportationn, which collectively dictate the spatial, timing, and other combinatorial requirements of the robot's ' actions in relation to the environment.
However, effectively formulating these constraints for a large variety of real-world tasks presents significant challenges. While representing constraints using relative poses between robots and ob- jects is a direct and widely-used approach [1], rigid-body transformations do not depict geometric details, require obtaining object models a priori, and cannot work on deformable objects. On the other hand, data-driven approaches enable learning constraints directly in visual space [2. 3]. While more flexible, it remains unclear how to effectively collect training data as the number of constraints grows combinatorially in terms of objects and tasks. Therefore, we ask the question: how we represent constraints in manipulation that 1) widely applicable: adaptable to tasks that require multi-stage, in-the-wild, bimanual, and reactive behaviors, 2) scalably obtainable: have the potential to be fully automated through the advances in foundation models, and 3) real-time optimizable: can be efficiently solved by off-the-shelf solvers to produce complex manipulation behaviors?
In this work, we propose Relational Keypoint Constraints (ReKep). Specifically, ReKep repre- sents constraints as Python functions that map a set of keypoints to numerical cost, where each keypoint is task-specific and semantically meaningful 3D point in the scene. Each function is composed of (potentially nonlinear) arithmetic operations on the keypoints and encodes desired "relation" between them, where the keypoints may belong to different entities in the environment, such as the robot arms, object parts, and other agents. While each keypoint only consists of its 3D) Cartesian coordinates in the world frame, multiple keypoints can collectively specify lines, surfaces, and/or 3D rotations if rigidity between keypoints is enforced. We study ReKep in the context of the sequential manipulation problem, where each task involves multiple stages that have spatio-tempora dependencies (e.g., "grasping", "aligning", and "pouring" in the aforementioned example).
While constraints are typically defined manually per task [4], we demonstrate the specific form of ReKep possesses a unique advantage in that they can automated by pre-trained large vi- sion models (LVM) [5] and vision-language models (VLM) [6], enabling in-the-wild specification of ReKep from RGB-D observations and free-form language instructions. Specifically, we leverage L.V propose fine-grained and semantically meaningful keypoints in the scene and VL. to write the constraints as Python functions from visual input overlaid with proposed keypoints. This process can be interpreted as grounding fine-gr ane-grained spatial relations, often those not easily specified with natural language, in an output modality supported by VLM (code) using visual referral expressions.
With the generated constraints, off-the-shelf solvers can be used to produce robot actions by re- evaluating the constraints based on tracked keypoints. Inspired by [7], we employ hierarchical optimization procedure to first solve set of waypoints as sub-goals (represented as SE(3) end- effector poses) and then solve the sccding-horizzo control problem to obtain a dense sequence of actions to achieve each sub-gou uub-gol.. With appropriate instantiation of the problem, we demonstrate that it can be reliably solved at approximately 10 Hz for the tasks considered in this work.
Our contributions are summarized as follows: 1) We formulate manipulation tasks as hicrarchical optimization problem with Relational Keypoint Constraints; 2) We devise pipeline to automatically specify keypoints and constraints using large vision models and rision-language models; 3) We present system implementations on two real-robot platforms that take as input a language instruction and RGB-D observations, and produce multi-stage, in-the-wild, bimanual, and reactive behaviors for a large variety of manipulation tasks, all without task-specific data or environment models.
2 Related Works
Structural Representations for Manipulation. Structural representations determine the orchestra- tion of different modules in manipulation system and yield different implications on the capabili- ties, assumptions, efficiency, and effectiveness of the system. Rigid-body poses are most commonly

---

used given rell-understood rigid-body motions in free space and their efficiency at modeling long- range dependencies of objects [1, 8-18). However, since it often requires both geometry and dynam- ics of environment to be modeled beforehand, various works have studied structural representations using data-driven methods, such as learning object-centric representation [19-34), particle-based dynamics [35-41], and keypoints or descriptors [3, 4, 42-54]. Among them, keypoints have shown great promises given their interpretabilityy efficiency, generalization to instance variations [4], and ability to model both rigid bodies and deformable objects. However, manual annotation is required per task, thus lacking scalability in open-world settings, which we aim to address in this work.
Constrained Optimization in Manipulation. Constraints are often used to impose desired be- haviors on robots. Motion planning algorithms use geometric constraints compute feasible tra- jectories that avoid obstacles and achieve goals [55-60]. Contact constraints can be used plan forceful or contact-rich behaviors [61-74]. For sequential manipulation tasks, task and motion plan- ning (TAMP) [1, 8, 13] is widely used framework, often formulated as constraint satisfaction problems [11, 75-80] with continuous geometric problems as subroutines. Logic-Geometrr Pro- gramming [81-86] alternatively formulates a nonlinear constrained program over the entire state trajectory, taking into account both logical and geometric constraints. Constraints can either be manually written or learned from data in the form of manifolds [87], feasibility model [86, 88], or signed-distande fields [2. 89]. Inspired by [7], we formulate sequential manipulation tasks as an integrated continuous mathematical program that is repeatedly solved in a receding-horiinn horizon fashion, with the key difference being that the constraints are synthesized by foundation models.
Foundation Models for Robotics. Leveraging foundation models for robotics is an active area of research. We refer readers to [90-93] for overview and recent applications. Here, we focus on VLMs that are capable of incorporating visual inputs [6, 94 98] for manipulation. However, despite showing promises for open-world planning and goal specification [99-113], the caption-guided pre- training scheme of VLMs often limits visual details that can be retained about the image [1 1114-117. Self-supervised vision models (e.g., DINO [5, 118)), on the other hand, provide fine-grained pixel- level features useful for various vision and robotic tasks [31, 119-124]. but there lack effective ways for interpreting open-world semantics pivotal for cross-task generalization. In this work, we leverage their complementary strengths by using DINOv2 [5] for fine-grained keypoint proposal and using GPT-4o [6] for its visual reasoning capability in a supported output modality (code). Similar forms of visual prompting techniques are also explored in concurrent works [99-101, 112, 125]. In this work, demonstrate ReKep possesses the unique advantages of performing challenging 6-12 DoF tasks, integrated high-level reasoning for reactive replanning, fighefreyyency closed-loop execution, and generating black-box constraints via visual prompting. More discussion in Appendix A.10.
3 Method
Herein we discuss: (1) What are Relational Keypoint Constraints (Sec. 3.1)? (2) How to formulate
manipulation constrained optimization problem with ReKep (See. 3.2)? (3) What is our algo- rithmic instantiation that can efficiently solve the optimization in real-time (Sec. 3.3)? (4) How to automatically obtain ReKep from RGB-D observations and language instructions (Sec. 3.4y?
3.1 Relational Keypoint Constraints (ReKep)
Herein we define a single instance of ReKep. For clarity, we assume that a set of K keypoints have been specified (discussed later in Sec. 3.4). Concretely, each keypoint k; IR3 refers 3D point on the scene surface with Cartesian coordinates, which is dependent on the task semantics and the environment (e.g., grasp point on the handle, spout).
A single instance of ReKep is a function f R x3 + R that maps an array of keypoints, denoted as kt to an unbounded cost, where f(k) S indicates the constraint satisfied. The function f is implemented as a stateless Python function, containing NumPy [126] operations on keypoints, which may be nonlinear and nonconvex. In essence, one instance of ReKep encodes one desired spatial relation between kcypoints, which may belong robot arm(s), object parts, and other agents.
3

---

RGB-D Observation
Pour into the cup.
oe sibgel L1C0
Vision
poth anagez FEER)e return aback diFF)
Large
Language Model
Vision Model
R(3]E2] en 6.18
return norM(AEZ) KFD)
Relational Keypoint Constraints
gla) sob gool
(E3) path
e2 sublogoul
(33) sublgool
Constrained Optimization Solver
Optimized
Actions
gripper
teapot
Figure 2: Overview of ReKep. DINOv2 [5] first proposes keypoints in the scene, which overlaid on the original RGB image. The image and an instruction are fed into GPT-4o [6] to generate series ReKep constraints as python programs that specify desired relations between keypoints at different task stages (C (Cahi-gon) and any requirement on transitioning behaviors within stages (Crun). Finally, dense sequence of end-effector actions are obtained by solving the optimization problem subject to generated constraints.
However, manipulation task typically involves multiple spatial relations and may have multiple temporally dependent stages where each stage entails different spatial relations. To this end, decompose task into N stages and use ReKep to specify two kinds of constraints for each stage
i E [1....,Ny: set of sub-goal constraints c(s)
"sub-goul (f sub-ggoat, 1
(k:) (k......
f(())
sub-goal, (k)) and
set path constraints Cpan Spun,, ( (k))....,
p (k)). where p(()) soah encodes one keypoint
p(S) path.m.
relation to be achieved at the end of stage i, and ((t) encodes one keypoint relation to be satisfied for every state within stage i. Consider the pouring task Fig. 2, which consists of three stages: grasp, align, and pour. The stage-1 sub-goal constraint pulls the end-effector towards the teapot handle. Then stage-2 sub-goal constraint specifies that the teapot spout needs to be top of the cup opening. Additionally, stage-2 path constraint ensures the teapot stays upright to avoid spillage when transported. Finally, the stage-3 sub-goal constraint specifies the desired pouring angle.
3.2 Manipulation Tasks as Constrained Optimization with ReKep
Using ReKep general tool to represent constraints, adopt the formulation in [7] and show how manipulation task can be formulated constrained optimization problem involving ((
sub-gual
and c(i) We denote the end-effector pose as e E (FF(3).To perform the manipulation task, we aim "path" obtain the overall discrete-time trajectory cı.r by formulating the control problem as follows:
e1 Cinit 90 1, 0 < 91 9++1
N
f(kg. ) < 0, vf E (())
arg min e1:2701:NN
( goal (eg.)
+ t=
Σ
>
pah
(e))
s.t.
sub-goal
s(k)) S 0, Aʃ E path?
((i)
/ = yi ..... :9
Kx+1 h ez), t= -1
T
(1)
where e; denotes end-effector pose at time t, gr E {I...., T) are the timings of the transition from stage to i which are also auxiliary decision variables, k is the array of keypoint positions (i)
(6) time t, h is a forward model of keypoints, and sub-goal and path are auxiliary cost functions (c.g., collision avoidance) for the sub-goal and path problems respectively. Namely, for each stage the optimization shall find an end-effector pose next sub-goal, along with its timing, and a sequence of poses c9.-199. that achieves the sub- uub-goal, subject to the given set of ReKep constraints and auxiliary costs. This formulation be considered direct shooting in trajectory optimization [127].

---

3.3 Decomposition and Algorithmic Instantiation
To solve Eq. in real-time, we employ a decomposition of the full problem and only optimize for the immediate next sub-goal and the corresponding path to reach the sub-goal (pscudo-code in Algorithm 1). All optimization problems are implemented and solved using SciPy [128] with decision variables normalized [0, 1]. They are initially solved with Dual Annealing [129] with SLSQP [130] local optimizer (around second) and subsequently solved with only local opti- mizer based on the previous solution at approximately 10 Hz.
The Sub-Goal Problem: We first solve the sub-goal problem to obtain Cg. for the current stage i:
arg min (i) sub-goal (ey.) s.t. f(ky.)) O, vf sub-goal
(+)
Cg
where Asub-goal subsumes auxiliary control costs: scene collision avoidance, reachability, pose regu- larization, solution consistency, and self-collision for bimanual setup (details in A.8). Namely, Eq. attempts to find sub-goal that satisfies Gaub-gual while minimizing the auxiliary costs. If a stage is concerned with grasping, grasp metric is also included. In this work, we use AnyGrasp [131]'.
The Path Problem: After obtaining sub-goal gg: we solve for trajectory e::g: starting from current end-effector pose e₁ to the sub-goal Cg.:
arg min Apath (et:g:) s.t. f(k;) S. O, Vf E c
(3)
where Aputh subsumes the following auxiliary control costs: scene collision avoidance, reachability, path length, solution consistency, and self-collision in the case of bimanual setup (details in A.9). the distance to the sub-goal eg, within a small tolerance e, we progress the next stage + 1.
Backtracking: Although the sub-problems can be solved at a real-time frequency to react to external disturbances within a stage, it is imperative that the system can replan across stages if any sub-goal constraint from the last stage no longer holds (c.g., cup taken out of the gripper in the pouring task).
Specifically, in every control loop, we check for violation of (()) If one is found, we iteratively paih
backtrack to previous stage j such that clu) is satisfied.
Forward Models for Keypoints: To solve Eq. 2 and Eq. 3, one must utilize a forward model that estimates Ak from Ae in the optimization process. As in prior work [4], we make the rigidity assumption between the end-effector and the "grasped keypoints" (a rigid group of keypoints that belong the same object or part; obtained from the segmentation model described in Sec. 3.4.). Namely, given a change in the end-effector pose Ae, can calculate the change in keypoint po- sitions by applying the same relative rigid transformation k' [grasped]] = Tae A [grasped], while assuming other keypoints stay static. We note that this is a "local" assumption in that it is only as- sumed to hold for the short duration (0. Is) that the problem is solved. Actual keypoint positions are tracked using visual input at 20 Hz and used in every new problem. For more challenging scenarios (e.g., dynamic or contact-rich tasks), a learned or physics-based model may be used.
3.4 Keypoint Proposal and ReKep Generation
To enable the system perform tasks in-the-wild given free-form task instruction, we devise pipeline using large vision models and vision-language models for keypoint proposal and ReKep generation, which are respectively discussed as follows:
Keypoint Proposal: Given an RGB image hhxwx3, we first extract the patch-wise wise features F patch E R& xw/ xd from DINOv2 [5]. Then we perform bilinear interpolation to upsample the features to the original image size, F Pimerp E R4xwxd To ensure the proposal covers all relevant objects in the scene, we extract all masks M {m), m m in the scene using Segment Anything
'Sice AnyGrasp is grasp detector instead of a metric and is computationally expensive to run in optimiza- tion loops, we always return the grasp closest to specified "grasp keypoint" by exploiting the fact that ReKep related to grasping always associates dummy keypoint the end-effector and one actual keypoint.
5

---

(a) Initial State
(b) Solution
(c) Final State
(a) Initial State
(b) Solution
(c) Final State
Stow Book
Pour
Tape Box
Reeyer
Collab. Folding Fold Garment
Pack Shoes
Figure Experiment tasks and visualization of optimization results. Seven tasks are designed to validate different aspects of our system, including in-the-wild specification with commonsense knowledge, multi-stage tasks with spatio-temporal dependencies, bimanual coordination with geometric awareness, and reactiveness when collaborating with humans and under disturbances.
(SAM) [132]. For each mask j. we cluster the masked features F Finterp [my] using k-means with with a cosine-similarity metric. The centroids of the clusters are used as keypoint candidates, which are projected
world coordinate IR3 using calibrated RGB-D camera. Candidates that are within 8em of others filtered out. Overall, find that this procedure is adept identifying large percentage of fine-grained and semantically meaningful regions of objects.
After obtaining the keypoint candidates, we overlay them on the original RGB image with numerical marks. Coupled with the language instruction of the task, we then use visual prompting to query GPT-4o [6] to generate the number of required stages and the corresponding sub- c(f)
ReKep Generation:
(( goal constraints sub-goal and path constraints path for each stage (prompts are in A.6). Notably, the functions do not directly manipulate the numerical values of the keypoint positions. Rather, we exploit the strength of VLM to specify spatial relations as arithmetic operations, such as L2 distance or dot product between keypoints, that are only instantiated when invoked with actual keypoint po- sitions tracked by specialized 3D tracker. Furthermore, an important advantage of using arithmetic operations set of keypoint positions is that it can specify 3D rotations in full SO(3) when suffi- cient points are provided and rigidity between relevant points is enforced, but this is done only when needed depending on task semantic. This enables VLM to reason about 3D rotations with arithmetic operations in 3D Cartesian space, effectively circumventing the need for dealing with alternative 3D rotation representation and the need for performing numerical computation.
Experiments
We aim to answer the following research questions: (1) How well does framework automatically formulate and synthesize manipulation behaviors (Sec. 4.1)? (2) Can system generalize to novel

---

objects and manipulation strategies (Sec. 4.2)? (3) How do the individual components contribute to the failure cases of the system (Sec. 4.3)? We validate ReKep on two real robot platforms: a wheeled single-arm platform, and a stationary dual-arm platform (Figure. 3). Additional implementation details can be found in Appendix, including keypoint proposal (A.5), VLM querying (A.6), point trackers (A.7), sub-goal solver (A.8), and path solver (A.9).
4.1 ReKep for In-the-Wii and Bimanual Manipulation
Tasks. We purposefully select set of tasks (shown in Fig. 3) with the goal of examining the multi-stage (m), in- the-wild (w), bimanual (b), and reactive (r) behaviors of the system. The tasks and their features are Pour Tea (m, w, r), Stow Book (w), Recycle Can (w). Tape Box (w. r), Fold Garment (b), Pack Shoes (b), and Collaborative Folding (b, r). We further evaluate three of the tasks under external disturbances (denoted as "Dist.") by changing poses of task objects during execution.
ReKep
Task
VoxPoser
Auto
Annot.
Mobile Arm
Pour Tea
0/10
3/10
8/10
Recycle Can
3/10
6/10
8/10
Stow Book
0/10
3/10
6/10
Tape Box
4/10
7/10
8/10
Dual-Arm
Fold Garment
0/10
5/10
6/10
Metric and Baselines. Each setting has 10 trials, which object poses are randomized. Success rate is reported in baseline. We
Pack Shoes
0/10
3/10
5/10
Collab. Folding
0/10
4/10
7/10
Tab. 1. We compare VoxPoser [106] evaluate two variants of the system: "Auto" uses foundation models to automatically generate ReKep, and "Annotated (Annot.)" uses human-annotated ReKep.
Total (%)
10.0%
44.3%
68.6%
Table 1: Success rate on wheeled single- arm and stationary bimanual platforms.
Results. Compared to baselines, ReKep can effectively handle core challenges of each task. For example, it can formulate correct temporal dependency in multi-stage tasks (e.g., spout needs to be aligned with the cup before pour- ing), leverage commonsense knowledge (e.g.. coke cans should be recycled), and construct coordination behaviors in both bimanual settings (e.g., folding left sleeve and right sleeve simultaneously) and human-robot collaboration set- ting (e.g., folding a large blanket by aligning the four cor- ners together with human). Coupled with optimization framework, it can also generate kinematically challenging behaviors in confined spaces in the Stow Book task and find a feasible solution that densely fits two shoes within a small volume the Pack Shoes task. Since the keypoints are tracked at high frequency, the system can react to external disturbances and replan both within stage and across stages, Despite promising results, we also identify several limitations which are discussed in Sec. 5.
ReKep
Task (Dist.)
VoxPoser
Auto
Annot.
Pour Tea
0/10
2/10
4/10
Tape Box Collah. Folding
2/10
3/10
5/10
0/10
3/10
5/10
Total (%)
6.7%
26.7%
46.7%
Table 2: Success under external dis- turbances across both robot platforms.
7.7h
Kopolrt Propesa
15.4%
Trucker
Optimiiiiin
Others
Figure System emror breakdown.
4.2 Generalization in Manipulation Strategies
Tasks. We systematically evaluate how novel manipulation strategies can be formulated by focusing on a single task, garment folding, but with 8 unique categories of garments, each demanding a unique way of folding and requiring both geometrical and commonsense reasoning. Evaluation is done on the bimanual platform, presenting additional challenges in bimanual coordination.
Metric. We use GPT-4o with a prompt containing only generic instructions with no in-context examples. "Strategy Success" measures whether generated ReKep is feasible, which tests both the keypoint proposal module and the VLMM, and "Execution Success" measures system success rate given feasible strategies for each clothing. Each is measured with 10 trials. Results. Interestingly, observe drastically different strategies across categories, many of which aligned with how humans might fold each garment. For example, it recognize that two
7

---

sweater
hoodie
pants
shorts
Garment
ReKep
O
sweater
shirt
hoodie
vest
dress
pants
shorts
scarf
Total
Strategy Success
6/10
4/10
4/10
6/10
3/10
7/10
7/10
5/10
52.5%
Execution Success
6/10
5/10
6/10
9/10
7/10
8/10
9/10
9/10
73.8%
Novel bimanual strategies of ReKep for folding different categories of garments and their success rates. Since ReKep in this task always associates two points at time, two keypoints are connected by arrow if they need to be aligned. The coloring of the keypoints denotes the order. In the sweater task, two sleeves first folded simultaneously with two arms, and then two arms grasp the crew neck to align to the bottom.
Figure 5:
sleeves often are folded together, prior to fully folding the clothes. In cases where using two arms is unnecessary, akin to how humans fold clothes, only one arm is being used. However, we do observe that the VLM may miss certain steps to complete the folding as the operator expected, but we recognize that this inherently an open-ended problem often based on one's preferences.
4.3 System Error Breakdown
The modular design of the framework entails an advantage for analyzing system errors due to its interpretability. In this section, we perform empirical investigation by manually inspecting the failure cases of the experiments reported in Tab. 1, which is then used calculate the likelihood of a module causing Results are reported in Fig. 4. Among the different modules, the point tracker produces the largest portion of errors, as frequent and intermittent occlusion poses significant challenges for accurate tracking. Keypoint proposal and VLM also produce considerable portions errors, where common cases include the proposal module missing certain keypoints and the VL.M referring to incorrect keypoints. The optimization module, on the other hand, does not contribute as much to the failures despite given limited time budget, since there often exist many possible solutions for each problem. Other modules, such as segmentation, 3D reconstruction, and low-level controller, also contribute to some failure cases, but they are relatively insignificant compared to other modules.
error while accounting for their temporal dependencies in the pipeline.
5 Conclusion & Limitations
In this work, we presented Relational Keypoint Constraints (ReKep), a structural task representa- tion using constraints that operates on semantic keypoints to specify desired relations between robot arms, object (parts), and other agents in the environment. Coupled with point trackers, we demon- strate that ReKep constraints can be repeatedly and efficiently solved a hierarchical optimization framework to act as a closed-loop policy that runs at a real-time frequency. We also demonstrate the unique advantage ReKep in that can be automatically synthesized by large vision models and vision-language models. Results are shown on two robot platforms and on variety of tasks featuring multi-stage, in-the-wild bimanual, and reactive behaviors, all without task-specific data, additional training, or environment models. Despite the promises, several limitations remained. First, the op- timization framework relies on a forward model of keypoints based on rigidity assumption, albeit a high-frequent feedback loop that relaxes the accuracy requirement of the model. Second, ReKep relies on accurate point tracking to correctly optimize actions in closed-loop. which is itself a chal- lenging 3D vision task due to heavy intermittent occlusions. Lastly, the current formulation assumes fixed sequence of stages (i.e., skeletons) for each task. Replanning with different skeletons requires lighfrreueecy, which poses considerable computational
running keypoint proposal and VLM challenges. An extended discussion of limitations can be found in Appendix A.11.
8

---

Acknowledgments
This work is partially supported by Stanford Institute for Human-Centered Artificial Intelligence, ONR MURI N00014-21-1-2801, and Schmidt Sciences. Ruohan Zhang is partially supported by Wu Tsai Human Performance Alliance Fellowship. The bimanual hardware is partially supported by Stanford TML. We would like to thank the anonymous reviewers, Albert Wu, Yifan Hou, Adrien Gaidon, Adam Harley, Christopher Agia, Edward Schmerling, Marco Pavone, Yunfan Jiang, Yix- uan Wang. Sirui Chen, Chengshu Li, Josiah Wong. Wensi Ai, Weiyu Liu, Mengdi Xu, Yihe Tang, Chelsea Ye, Mijiu Mili, and the members of the Stanford Vision and Learning Lab for fruitful dis- cussions, helps on experiments, and support.
References
[1] L. P. Kaelbling and T. Lozeno-Pérez. Hierarchical planning in the now. In Workshops the Twenty-Fourth AAAI Conference Artificial Intelligence, 2010.
[2] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models functionals of signed- distance fields for manipulation planning. In Conference on robot learning, pages 245-255. PMLR, 2022.
[3] A. Simeonov, Y. Du, A. Tagliasacchi, B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equvvariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 3394-6400. IEEE, 2022.
[4] L. Manuelli, Gao, P. Florence, and R. Tedrake, kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132-157. Springer, 2019.
[5] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[6] OpenAL Gpt-4 technical report. arXiv, 2023.
M. Toussaint, J. Harris, J.-S. Ha, D. Driess, and W. Hönig. Sequence-of-constraime mpc: Reactive timing-optimal control of sequential manipulation. In 2022 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 13753-13760. IEEE, 2022.
[8] L. P. Kaelbling and T. Lozano- Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, 32(9-10): 194-1227, 2013.
[9] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE international conference on robotics and automation (ICRA), 2014. 2014.
[10] A. Byravan and D. Fox. Se3-nets: Learning rigid body motion using deep neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 173-180. IEEE, 2017.
[1]] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E, Kavraki. An incremental constraint- based framework for task and motion planning. The International Journal of Roboties Re- search, 37(10): 134-1151, 2018.
[12] T. Migimatsu and J. Bohg. Object-centric task and motion planning dynamic environments. IEEE Robotics and Automation Letters, 5(2):844-851, 2020.
[13] C. R. Garreut R. Chitnis, Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 2-265-293, 2021.

---

[14] A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-Pérez and C. R. Garrett. Long-horizon ma- nipulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[15] Labbé, Manuelli, A. Mousavian, S. Tyree, S. Birchfield, Tremblay, J. Carpentier, M. Aubry, D. Fox, and J. Sivic. Megapose: 6d pose estimation of novel objects via render & compare. In Proceedings of the 6th Conference Robot Learning (CoRL), 2022.
[16] S. Tyree, J. Tremblay, T. To, J. Cheng, T. Mosier, Smith, and Birchfield. 6-dof pose esti- mation of household objects for robotic manipulation: An accessible dataset and benchmark. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13081-13088. IEEE, 2022.
[17] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. Tax-pose: Task-specific cross-pose estimation for rohot manipulation. Conference on Robot Learning, pages 1783-1792. PMLR, 2023.
[18] B. Wen, W. Yang, Kautz, and S. Birchfield. Foundationpose: Unified pose estimation and tracking of novel objects. arXiv preprint arXiv:2312.08344 2023.
[19] I. Lenz, Knepper, and A. Saxena. Deepmpe: Learning deep latent features for model predictive control. In Robotics: Science and Systems, volume 10. Rome, Italy, 2015.
[20] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341, 2016.
[21] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[22] A. Sanchez- Gonzalzz, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pages 4470-4479. PMLR, 2018
[23] E. Jang, C. Devin, Vanhoucke, and S. Levine, Grasp2vec: Learning object representations from sele-supervised grasping. arXiv preprint arXiv: /0ti::////..0 1811,06961, 2018.
[24] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and Birchfield. Deep ob- ject pose estimation for semantic robotic grasping of household objects. arXiv preprint arXiv:1809.10900. 2018
[25] Z. Xu, J. Wu, A. Zeng, B. Tenenbaum, and S. Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. arXiv preprint arXiv: arXiv:1906.0m 1906603853. 2019.
[26] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904. 12584, 2019.
[27] C. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Ler- chner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.1 (tt::9901.11900, 2019.
[28] L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning-based model pre- dictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269-296, 2020.
10

---

[29] F. Locatello, D. Weissenbomn Т. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 13:11525-1158,, 2020.
[30] N. Heravi, A. Wahid, C. Lynch, Florence, T. Armstrong, J. Tompson, P. Sermanet, Bohg, and D. Dwibedi. Visuomotor control in multi-object scenes using object-aware representa- tions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 95 9515-9522. IEEE, 2023.
[31] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. arXiv preprint arXiv:2310. 14386, 2023.
[32] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representations for sequential manipulation. In Conference on Robot Learning, pages 148-157. PMLR, 2022.
[33] S. Cheng, C. Garrett, A. Mandlekar, and Xu. Nod-tamp: Multi-step manipulation planning with neural object descriptors. arXiv preprint arXiv:231..01530 2023.
[34] J. Hsu, J. Mao, J. Tenenbaum, and J. Wu. What's ss left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[35] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv: 181000160, 2018.
[36] X. Lin, C. Qi, Y. Zhang. Z. Huang, K. Fragkiadaki, Y. Li, C. Gan, and D. Held. Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. arXiv preprint arXiv:22100 15751, 2022.
[37] Y. Wang. Y. Li, K. learning for object pile manipulation. arXiv preprint arXiv:2306. 16700, 2023.
L. Fei-Fei, and J. Wu. Dynamic-resolution model
[38] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto clasto-plastic object manipulation with diverse tools. arXiv preprint arXiv:2306. /4447, 2023.
[39] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference Robot Learning, pages 256-266. PMLR, 2022.
[40] J. Abou-Chakra, K. Rana, F. Dayoub, and N. Sünderhauf. Physically embodied gaussian splatting: A realtime correctable world model for robotics. arXiv preprint wwXiv:2406.10788, 2024.
[41] D. Bauer, Z. Xu, and S. Song. Doughnet: A visual predictive model for topological manipu- lation of deformable objects. arXiv preprint arXiv:2404. 12524, 2024.
[42] T. Schmidt, R. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters, 2(2):420-42)) 2016.
[43] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv: 18X.008006008756 2018.
[44] T. D. Kulkarni, A. Gupta, C. lonescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
[45] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7278-7285. IEEE, 2020.

---

[46] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gon- zale7, and K. Goldherg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9411-9418. IEEE, 2020.
[47] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.0508 05500, 2020.
[48] B. Chen, P. Abbeel, and D. Pathak. Unsupervised learning of visual 3d keypoints for control. In International Conference Machine Learning, pages 1539-1544 PMLR, 2021.
[49] A. Simeonov, Y. Du, Y.C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano-Pérez, and P. Agrawal. (3)equivari relational rearrangement with neural descriptor fields. In Conference on Robot Learning, pages 835-846. PMLR. 2023.
[50] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308. 1597,, 2023.
[51] E. Chun, Y. Du, A. Simeonov, T. Lozano-Perez, and L. Kaelbling. Local neural descriptor fields: Locally conditioned object representations for manipulation. In 2023 IEEE Interna- tional Conference Robotics and Automation (ICRA), pages 1830-1836. IEFF, 2023.
[52] S. Bahl, R. Mendonca, L.. Chen, Jain, and Pathak. Affordances from human videos versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778- 137790, 2023.
[53] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025. 2023.
[54] H. Bharadhwaj. R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation, 2024.
[55] Z. Kingston, M. Moll, and L.. E. Kavraki. Sampling- methods for motion planning with constraints. Annual review of control, robotics, and autonomous systems, 1:159-185, 2018.
[56] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa. Chomp: Gradient optimization tech- niques for efficient motion planning. In 2009 IEEE international conference on robotics and automation, pages
IEEE, 2009.
[57] J. Schulman, Y. Duan, J. Ho, A. Lee, I. Awwal, H. Bradlow, J. Pan, S. Patil, K. Goldberg, and P. Abbeel. Motion planning with sequential convex optimization and convex collision checking. The International Journal of Robotics Research, 33(9):1251-122 2014.
[58] Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, Van Wyk, Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023.
pages 81 111181119
[59] T. Marcueci, Umenberger, Parrilo, and R. Tedrake. Shortest paths in graphs of convex sets. SIAM Journal on Optimization, 34(1):507-532, 2024.
[60] N. D. Ratliff, J. Issac, D. Kappler, S. Birchfield, and D. Fox. Riemannian motion policies. arXiv preprint arXiv:1801.028: 2018.
[61] M. Posa, S. Kuindersma, and R. Tedrake. Optimization and stabilization of trajectories for constrained dynamical systems. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 1366-1373. IEEE, 2016.
12

---

[62] I. Mordatch, E. Todorov, and Z. Popović. Discovery of complex behaviors through contact- invariant optimization. ACM Transactions on Graphics ( 2012.
[63] I. Mordatch, Z. Popović, and Todorov. Contact-invariant optimization for hand manipula- tion. Proceedings of the ACM SIGGRAAPH/Fumographies symposium on computer anima- tion, pages 137-144, 2012.
[64] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, (3(1):69-B)) -1, 2014,
[65] T. Howell, N. Gileadi, S. Tunyasuvunakool, K. Zakka, T. Erez, and Y. Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco. arXiv preprint arXiv:2212.00541, 2022.
[66] Z. Liu, G. Zhou, J. He, T. Marcucei, F.F.. Li, J. Wu, and Y. Li. Model-based control with sparse neural dynamics. Advances in Neural Information Processing Systems, 36, 2024.
[67] K. M. Lynch and M. T. Mason. Stable pushing: Mechanics, controllability, and planning. The international journal of robotics research, 15(6):533-556, 1996.
[68] Hou, Z. Jia, and M. T. Mason. Fast planning for 3d reo-pese-r using pivoting. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1631-1638. IEEE, 2018.
[69] J.-P. Sleiman, F. Farshidian, and M. Hutter. Versatile multicontact planning and control for legged doco-manipulation. Science Robotics, 8(81):eadg5014 2023.
[70] W. Yang and M. Posa. Dynamic on-palm manipulation via controlled sliding. arXiv preprint arXiv:2405.08% 08731. 2024.
[71] B. P. Graesdal, S. Chia, T. Marcucci, S. Morozov, A. Amice, A. Parrilo, and R. Tedrake. Towards tight convex relaxations for contact-rich manipulation. arXiv preprint arXiv:2402.10312 2024.
[72] K. Yunt and C. Glocker. Trajectory optimization of mechanical hybrid systems using sumt. In 9th IEEE International Workshop on Advanced Motion Control, 2006., pages 665-671. IEEE, 2006.
[73] Yunt and C. Glocker. combined continuation and penalty method for the determination of optimal hybrid mechanical trajectories. In lutam Symposium on Dynamics and Control of Nonlinear Systems with Uncertainty: Proceedings of the IUTAM Symposium held in Nanjing, China, September 18-22, 2006, pages 187-196. Springer, 2007.
[74] K. Yunt. An augmented lagrangian based shooting method for the optimal trajectory gen- eration of switching lagrangian systems. Dynamics of Contimuous, Discrete and Impulsive Systems Series B: Applications and Algorithms, 18(5):615 665, 2011.
[75] F. Lagriffoul, D. Dimitrov, A. Saffiotti, and I.. Karlsson. Constraint propagation on interval bounds for dealing with geometric backtracking, In 20/2 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 957-964. IEEE, 2012.
[76] F. Lagriffoul, D. Dimitrov, J. Bidot, A. Saffiotti, and L. Karlsson. Efficiently combining task and motion planning using geometric constraints. The International Journal of Robotics Research, 1726-17777 2014.
[77] T. Lozano-Pérez and L. Kaelbling. A constraint-based method for solving sequential ma- nipulation planning problems. In 2014 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 3684-3691, IEEE, 2014.
13

---

[78] Z. Yang, J. Mao, Y. Du, J. Wu, J. B. Tenenbaum, T. Lozano-Pérez, and L. P. Kaelbling, Com- positional Liffusion-based continuous constraint solvers. arXiv preprint S.Niv:2309000966 2023.
[79] T. Silver, R. Chitnis, J. Tenenbaum, L.. P. Kaelbling, and T. Lozano-Pérez. Learning sym- bolic operators for task and motion planning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3182-3189. IEEE, 2021.
[80] B. Vu, T. Migimatsu, and J. Bohg. Coast: Constraints and streams for task and motion planning. arXiv preprint arXiv:2405.0833 2024.
[81] M. Toussaint. Logic-geometric programming: An optimization-baseee approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[82] M. Toussaint and M. Lopes. Multi-bound tree search for logic-geometric programming in cooperative manipulation domains. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4044-4051. IEEE, 2017.
[83] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation, 2018.
[84] J.-S. Ha, D. Driess, and M. Toussaint. A probabilistic framework for constrained manipula- tions and task and motion planning under uncertainty. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6745-6751. IEEE, 2020.
[85] T. Xue, A. Razmjoo, and S. Calinon. D-lgp: Dynamic logicgeometric program for reactive task and motion planning. arXiv preprint arXiv:2312.0273// 2023.
[86] D. Driess, O. Oguz, J.-S. Ha, and M. Toussaint. Deep visual heuristies: Learning feasibility of mixed-integer programs for manipulation planning. In 2020 IEEE international conference on robotics and automation (ICRA), pages
IEEE. 2020.
[87] G. Sutanto, I. R. Fernández, P. Englert, R. K. Ramachandran, and G. Sukhatme. Learning equality constraints for motion planning manifolds. In Conference Robot Learning, pages 2292-2305. PMLR, 2021.
[88] Z. Yang. C. R. Garrett, T. Lozano-Pérez L. Kaelbling, and D. Fox. Sequence-based plan feasibility prediction for efficient task and motion planning, arXiv preprint arX1::221100157,, 2022.
[89] G. S. Camps, R. Dyro, M. Pavone, and M. Schwager. Learning deep sdf maps online for robot navigation and exploration. arXiv preprint arXiv:2207.1078; 2022.
[90] Y. Hu, Q. Xie, V. Jain, J. Francis, Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Zhao, et al. Toward general purposse robots via foundation models: A survey and meta-analysis. arXiv preprint
2023.
[91] Firoozi, J. Tucker, Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor,
K. Hausman, et al. Foundation models in robotics: Applications, challenges, and the future. arXiv preprint ar arXiv:2312.07884: 2023.
[92] Kawaharazuka, T. Matsushima, Gambardella, Guo, C. Paxton, and A. Zeng. Real- world robot applications of foundation models: A review. arXiv preprint 1 2024.
[93] S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans. Foundation models for decision making: Problems, methods, and opportumities. arXiv preprint arXiv:2303.04129, 2023.
14

---

[94] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language super- vision. In International Conference on Machine Learning, pages 7748-773. PMLR, 2021.
[95] Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[96] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In International conference on machine learming, pages 12888-12 19980. PMLR, 2022.
[97] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087744 2023.
[98] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint vrXi:2231.12597 2023.
[99] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa: General robotic manipulation through spatial constraints parts with foundation models. arXiv preprint arXiv:2403.08244 2024.
[100] F. Liu, Fang, Abbeel, and Levine. Moka: Open-vocabulary robotic manipulation through mark-basee visual prompting. arXiv preprint arXiv:2403.03174, 2024.
[101] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid,
Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:4402.07822 2024.
[102] Y. Hu, Lin, Zhang, L. Yi, and Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.1784/ 2023.
[103] Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning. arXiv preprint 10. 2023.
[104] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 20482-20494, 2023.
[105] Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. Spatialvim: En- dowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[106] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[107] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-actinn models transfer web knowledge to robotic control. arXiv preprint 1.Xiv:2377.18116 2023.
[108] J. Gao, B. Sarkar, Xia, T. Xiao, J. Wu, B. Ichter. A. Majumdar, and D. Sadigh. Physically grounded vision-language models for robotic manipulation. arXiv preprint arXiv:2309.22561, 2023.
[109] Y. Wang, T.-H. Wang, J. Mao, M. Hagenow, and Shah. Grounding language plans in demon- strations through counterfactual perturbations. arXiv preprint arXiv:2403.17124 2024.
15

---

[110] J. Hsu, J. Mao, and J. Wu. Ns3d: Neuro-symbole grounding of 3d objects and relations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2614-2623, 2023.
[111] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh. Physi- cally grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2462-12469 IEEE, 2024.
[112] W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024.
[113] J. Duan, W. Yuan, W. Pumacay, Y. R. Wang, K. Ehsani, D. Fox, and R. Krishna. Manipulate- anything: Automating real-world robots using vision-language models. arXiv preprint arXiv:2406.18)) 2024.
[114] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer 2024.
Vision and Pattern Recognition, pages
[115] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238-5248, 2022.
[116] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision- language models behave like bags-of wo-dss, and what to do about it? In The Eleventh Inter- national Conference on Learning Representations, 2023.
[117] C.-Y. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna. Sugarcrepe: Fixing hackable benchmarks for lision-language compositionality. Advances in neural information processing systems, 36, 2024.
[118] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerg- ing properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[119] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descrip- tors. arXiv preprint arXiv:21 2(3):4, 2021.
[120] L. Melas-Kyriaze C. Rupprecht, I. Laina, and A. Vedaldi. Deep spectral methods: A sur- prisingly strong baseline for unsupervised semantic segmentation and localization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8364 8375, 2022.
[121] Y. Wang, Z. Li, M. Zhang, K. Driggs-Campbel J. Wu, L. Fei-Fei, and Y. Li. Diiields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXiv:2309. 16118 18, 2023.
[122] X. Lin, J. So, S. Mahalingam, F. Liu, and P. Abbeel. Spawnnet: Learning generalizable visuomotor skills from pre-trained networks. arXiv preprint arXiv:2307.03567, 2023.
[123] N. Di Palo and E. Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403. 19578, 2024.
[124] N. Di Palo and E. Johns. Dinobot: Robot manipulation via retrieval and alignment with vision foundation models. arXiv preprint arXiv:2402.11781, 2024.
[125] O. Y. Lee, A. Xie, K. Fang, K. Pertsch, and C. Finn. Affordance- guided reinforcement leamn ing via visual prompting. arXiv preprint arXiv.2407.10341, 2024.
16

---

[126] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Coumapeau,, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, Gohlke, and E. Oliphant. Array programming with NumPy. Nature, 585(7825))357-362, Sept. 2020. doi: 108.1088//1188-0202222222 attps://doi.ooe
URL
1008/ s1555////51556-5
[127] Tedrake. Underactuated Robotics. 2023. URL https: http::://undrraattttteee mit edu.
[128] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Coumaaeau,, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, ((1))261-272)) 2020.
[129] Y. Xiang, D. Sun, W. Fan, and X. Gong. Generalized simulated annealing algorithm and its application to the thomson model. Physics Letters A, 233(3):216-220) 1997.
[130] D. Kraft. A software package for sequential quadratic programming. Forschungsbericht- Deutsche Forschungs- und Versuchsanstali fur Luft- und Raumfahri, 1988.
[131] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
[132] Kirillov, E. Mintun, Ravi, Mao, C. Rolland, L. Gustafson, Xiao, Whitehead, Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.00 02643, 2023.
[133] Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[134] Y. Zhu, Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision- based manipulation with object proposal priors. 6th Annual Conference Robot Learning. 2022.
[135] M. Minderer, A. Gritsenko, Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open- open-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.066.30 2022.
[136] H. K. Cheng, S. W. Oh, B. Price, J.-Y. Lee, and A. Schwing. Putting the object back into video object segmentation. In arXiv, 2023.
[137] T. Darcet, M. Oquab. J. Mairal, and P. Bojanowski. Vision transformers need registers. arXiv preprint mrxiv:2339.16588 2023.
[138] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence, (4(5))603-619 2002.
[139] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-22 2830, 2011.
[140] J. Yang, H. Zhang, F. Li, Zou, C. Li, and Gao. Set-of-mark prompting unleashes ex- traordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. /441, 2023.
[141] W. Harley, Z. Fang, and K. Fragkiadaki. Particle video revisited: Tracking through oc- clusions using point trajectories. In European Conference Computer Vision, pages 59-75. Springer, 2022.
17

---

[142] Q. Wang, Y.-Y. Chang, R. Cai, Z. Li, B. Hariharan, A. Holynski, and N. Snavely. Tracking everything everywhere all at once. In Proceedings the JEEF/CVF International Conference on Computer Vision, pages 19795-19806, 2023.
the
[143] Y. Zheng, A. W. Harley, B. Shen, G. Wetzstein, and L. J. Guibas. Pointodyssey; A large-scale synthetic dataset for long-term point tracking. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 19855-19865, 2023.
[144] N. Karaev, Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: is better track together. arXiv preprint arXiv:2307.07635, 2023.
[145] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pages 10061-10072, 2023.
[146] C. Doersch, Y. Yang, D. Gokay, P. Luc, S. Koppula, A. Gupta, J. Heyward, R. Goroshin, J. Carreira, and A. Zisserman. Bootstap: Bootstrapped training for tracking-any-poiin arXiv preprint 1xXiv:24020008// 2024.
[147] Y. Xiao, Q. Wang. S. Zhang, N. Xue, S. Peng, Y. Shen, and X. Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406-20417, 2024.
[148] J. Luiten, G. Kopanas, B. Leibe. and D. Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint 2rXiv:2308.097 2023.
[149] A. Millane, H. Oleynikova, E. Wirbel, R. Steiner, V. Ramasamy, D. Tingdahl, and R. Sieg- wart. nvblox: Gpu-accelerated incremental signed distance field mapping, arXiv preprint arXiv:2311.000 1.00626, 2023.
[150] X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong. Ma- niplim: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 18061-18070, 2024.
[151] W. Xia, D. Wang, X. Pang, Z. Wang, B. Zhao, and D. Hu. Kinematic-aware prompting for generalizable articulated object manipulation with llms. arXiv preprint S.Niv:3311.0284/ 2023.
[152] S. Huang, H. Chang, Y. Liu, Y. Zhu, H. Dong, P. Gao, Boularias, and H. Li. A3vlm: Ac- tionable articulation-aware vision language model. arXiv preprint wxXi::2406007544 2024.
[153] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin-Martin, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[154] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polosuukii.. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[155] A. Goyal, Xu, Y. Guo, V. Blukis, Y.W.. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694-710. PMLR, 2023.
[156] A. Goyal, V. Blukis, J. Xu, Y. Guo, Y.-W. Chao, and D. Fox. Ryt-2: Learning precise manip- ulation from few demonstrations. arXiv preprint 3Xiv:2406008545 2024.
[157] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
18

---

A Appendix
A.1 Pseudo-code for Sequential Manipulation with Relational Keypoint Constraints
Algorithm Relational Keypoint Constraints for Sequential Manipulation
1: Initialize current stage i t- 1, and current time +- 1
2: while N do
3:
if 35 ((ú)
path s.t. f(k:) > 0 then
4:
i-i-1
5: continue
6: end if
7:
if distance( og.) <
then
8:
i 1-i+1
9:
continue
10:
end
11: Solve sub-goal problem for stage i to obtain eg; (Eqq 2)
Solve path problem for stage to obtain et:g, (Eq. 3)
13:
Execute the next actions e++1:+m
14:
t t m +
15:
end while
Franka Arm
Franka Arm
RGB-D Camera
Wheeled Base
RGB-D Camera
Figure 6: Wheeled Single-Ar Platform.
Figure 7: Stationary Dual-Arm Platform.
A.2 Wheeled Single-Arm Platform
One of our investigated platform is a Franka arm mounted on a wheeled base built with Vention frames (shown in Figure 6). Note that the base does not have motors and thus cannot move au- tonomously, but its mobility nevertheless allows us investigate the proposed method outside of lab environments.
Since our pipeline produces a sequence of 6-DoF end-effector poses, we use position control in all experiments, which running at a fixed frequency of 20 Hz. Specifically, once the robot is given target end-effector pose the world frame, we first clip the pose to the pre-defined workspace. Then we linearly interpolate from the current pose to the target pose with a step size of 5mm for position and degree for rotation. To move each interpolated pose, we first calculate inverse kinematics to obtain the target joint positions based current joint positions (IK solver from PyBullet [133]). Then the joint impedance controller from Deoxys [134] reach the target joint positions.
19

---

Two RGB-D cameras, Orbbec Femto Bolt, are mounted on each side of the robot facing the workspace center. The cameras capture RGB images and point clouds at a fixed frequency of 20 Hz.
A.3 Stationary Dual-Arm Platform
We also investigate the method on a stationary dual-arm platform consisting of two Franka arms mounted in front of tabletop workspace (shown in Figure 7). We share the same controller as the wheeled single-arm platform with the exception that the arms controlled simultaneously at 20 Hz. Specifically, our pipeline jointly solves two 6-DoF end-effector pose sequences, which are sent to the low-level controller together. The controller subsequently calculates IK for both arms and moves the arms using joint impedance control.
Three RGB-D cameras, Orbbec Femto Bolt, are mounted on this platform. Two cameras are mounted the left and right sides and camera is mounted in the back. The cameras simi- larly capture RGB images and point clouds at fixed frequency of 20 Hz.
A.4
Evaluation Details
Below we discuss the evaluation details for the experiments reported in Section 4.1 and Section 4.2.
A.4.1 Details for In-the-Wild and Bimanual Manipulation (Section 4.1)
For each task, 10 initial different configurations of objects are selected, which cover the full workspace but are manually verified to ensure they are kinematically feasible for the robot. For each trial, a human operator restores the scene to the corresponding configuration and initiates the system. Due to the challenge of developing automatic success criteria for the diverse set of objects and environments investigated this work, success rates are measured by the operator with the cri- terion reported under each task description below. For experiments involving external disturbances, the set of disturbances for all trials is pre-selected, and one disturbance is applied to each trial. Specifically, the disturbance is introduced by human operator using hands to change the object's pose. Collision checking is disabled for all tasks involving deformable objects.
The environment consists of teapot and cup placed on a counter table in a kitchen setting. The task involves three stages: grasping the handle, aligning the teapot to the top of the cup, and pouring the tea into the cup. The success criterion requires that the teapot remains upright until the pouring stage, and at the end, the spout must be aligned and tilted on top of the cup opening.
Pour Tea:
The environment includes one three types of cans (Coke, 7ero Coke, Zero Sprite), a recycle bin with a narrow opening (such that the cans may only go in when they are upright), a landfill bin, and compost bin, all situated inside an office building. The task involves two stages: grasping the can and reorienting it top of the recycle bin before dropping it. The success criterion that the can is successfully thrown into the bin.
Recycle Can:
Stow Book: The environment consists of target book placed on a side table and real-size book- shelf with a 15cm opening among the placed books, all inside an office environment. The task involves two stages: grasping the target book on the side and stowing it inside the opening in the shelf. The success criterion is that the target book is placed steadily after the robot releases the gripper, and the robot must not bump into the shelf or other placed books.
Tape Box: The environment includes cardboard box, a packaging tape with a dispenser sitting on top of the box that already has one side taped, and a human user collaborating with the robot. The tape has already been unrolled to be enough for taping because unrolling typically requires a large force that exceeds the limit of the robot arm. The task involves two stages: while human operator squeezing the box, the robot needs grasp the tape and align it to the correct side complete the taping. The success criterion is that the tape must end up in the correct position such that it is aligned with the seam.
20

---

(Bimanual) Fold Garment: The environment consists of sweater placed flat close to the workspace center, with small deformations on the sleeves, neck, and bottom. The task typically requires four stages: grasping both sleeves, folding them to the middle, grasping the neck, and fold- ing it to the bottom. The success criterion does not enforce consistent stages; as long as the sweater folded such that occupies most half of the original surface size, it is regarded as a success.
(Bimanual) Pack Shoes: The environment includes an empty shoe box placed close to the workspace center, with two shoes placed on opposite sides of the box in random poses. The task involves two stages: grasping the shoes simultaneously and placing them in the shoe box. The suc- cess criterion does not enforce consistent stages; as long as the shoes are placed into the box without being stacked together or causing bimanual self-collision, is considered successful.
(Bimanual) Collaborative Folding: The environment consists of large blanket (pre-folded to an appropriate size that occupies about 70% of the workspace due to its size exceeding the workspace limit) and a human user collaborating with the robot. The task involves two stages: the robot must grasp the two corners of the blanket opposite to the human user, and the second stage is aligning the two corners with the two corners that the human has grasped. The success criterion that the robot has grasped the correct corners and can align them with the correct human arms (left-left, right-right).
A.4.2 Details on Baseline Methods
We use VoxPoser [106] as the main baseline method as makes similar assumptions that no task- specific data or pre-defined motion primitives are required. We adapt VoxPoser to our setting with certain modifications to ensure fairness in comparisons. Specifically, we use the same VLM, GPT- 4o [6], that takes the same camera input. We also augment the original prompt from the paper with the prompt used in this work to ensure it has sufficient context. We only use the affordance, rotation, and gripper action value maps and ignore the avoidance and velocity value maps because they are not necessary for our tasks. We also only consider the scenario where the "entity of interest" is the robot end-effector instead of objects in the scene. The latter is tailored for pushing task, which is not being studied in this work. We use OWI-ViT [135] for open-vocabulary object detection, SAM [132] for initial-frame segmentation, and Cutie [136] for mask tracking.
A.4.3 Details for Generalization in Manipulation Strategies (Section 4.2)
The dual-arm robot is tasked with folding eight different categories of clothing. We two metrics for evaluation: "Strategy Success" and "Execution Success," where the former evaluates whether keypoints are proposed and constraints are written appropriately, and the latter evaluates the robotic system's 's execution given successful strategies.
To evaluate "Strategy Success," the garment is initialized close to the center of the workspace. A RGB-D camera captures the RGB image. Then, the keypoint proposal module gen- erates keypoint candidates using the captured image, which are then overlaid top of the original image with numerical marks {O...., K 1}. The overlaid image, along with the same generic prompt, is fed into GPT-4 [6] to generate the ReKep constraints. Since folding garments is itself an open-ended problem without ground truth strategies, we manually judge the proposed keypoints and the generated constraints are correct. Note that since the constraints are to be executed by a bimanual robot, and the constraints are almost always connecting (folding) two keypoints such that they are aligned, correctness is measured by whether it is (potentially) executable by the robot with- out causing self-collision (arms crossing over opposite sides) and whether the folding strategy can fold the garment to at most half of its original surface area.
To evaluate "Execution Success," we take the generated strategies in the previous section that are marked as successful for each garment and execute the sequence on the dual-arm platform, with a total of 10 trials for each garment. Point tracking disabled as we observe that our point tracker predicts unstable tracks when the garment is potentially folded many times. Success is measured by whether the garment is folded such that its surface area is at most half of its original surface area.
21

---

A.5 Implementation Details of Keypoint Proposal
Herein we describe how keypoint candidates in a scene are generated. For each platform, we use one of the mounted RGB-D cameras to capture an image of size h x w X 3, depending on which camera has the best holistic view of the environment, as all the keypoints need to be present in the first frame for the proposed method. Given the captured image, we first use DINOv2 with registers (ViT-S14) [5, 137] extract the patch-wise features F patch TR xw' xd Then perform bilinear interpolation to upsample the features to the original image size, F interp E Rhxwwxd To ensure the proposal covers all relevant objects in the scene, we extract all masks M = {m1. m............................... m, in the seene using Segment Anything (SAM) [132]. Within each mask m,, we apply PCA to project the features to three dimensions, F FPCA = PCA(F resized [m,].3), We find that applying PCA improves the clustering as it often removes details and artifacts related to texture that are not useful for our tasks. For each mask j. we cluster the masked features F Finter my using k-means with k = 5 with the Euclidean distance metric. The median centroids of the clusters used as keypoint candidates, which are projected to world coordinate I& using calibrated RGB-D camera. Note that we also store which keypoint candidates originate from the same mask, which is later used as part of the rigidity assumption in the optimization loops described in Sec. 3.3. Candidates outside of the workspace bounds are filtered out. To avoid many points cluttered in a small region, we additionally use Mean Shift [138, 139] (with a bandwidth Scm) to filter out points that are close to each other. Finally, the centroids are taken as final candidates. Alternatively, one may develop a pipeline using only segmentation models [132, 140], but we leave comparisons to future work.
A.6 Querying Vision-Language Model
After we obtain the keypoint candidates, they are overlaid the captured RGB image with numer- ical marks f0.....K. K 10--, 1}. Then the image and the task instruction are fed into a vision-language model with the prompt described below. The prompt contains only generic instructions with no image-text in-context examples, although a few text-based examples are given to concretely ex- plain the proposed method and the expected output from the model. Note that the majority of the investigated tasks are not discussed in the provided prompt. As a result, the VLM is tasked with generating ReKep constraints by leveraging its internalized world knowledge.
For the experiments conducted in this work, we use GPT-4o [6] as it is one of the latest available models at the time of the experiments. However, due to rapid advancement in this field, the pipeline can directly benefit from newer models that have better vision-language reasoning. Correspond- ingly, we observe different models exhibit different behaviors when given the same prompt (with the observation that newer models typically require less fine- mne-grained instructions). As a result, in- stead of developing the best prompt for the suite of tasks in this work, we focus demonstrating full-stack pipeline consisting key component that can be automated and continuously improved by future development.
Sugpose you are cont iin robot to ferforn nenipalat fan taske by writing constraint funct lons Pythonn Instructionn
The nanipulat Lank is given as an image of the environmat, overlayed with keypuiata nacked with their indices, along with text instruction. The single
starts with parenthesis indicating is binanuel. each given task, please perforn lowing
the
alepas
Determine how many stages are inval in the tack, Grasping nost be an independent stage, Some examples:
" "single ar)) pouring tea fron teagot
stages: "graep teapot", "align seapet wita cup opening", and "pour Liquid"
"(single-am) pat red block on top of blue block*;
3 st stages: "graap red blok,, falign red block or sop of blue bluck", and Frelesse red block*
"ibinanusll told sieeves to the center";
stages: fleft
greepe left sleeve right ame graspe right sleeve" and arms fold sleeves
the centes"
"(binanuall told jacket";
3 stages: fleft
grasge left sleeve and right graspa right sleeve", "both fold sleeves to the conter", and "guasp the neck with ane exm (the other arn stays in placel", and *siign the neck with the botten*
For each stage, write tvo kines of constraints, *sah-goal constreinted and "path constraints", The faub-gow) stagses, while the "path
satistied w*t the end constraints" are constraints that maet be ratistied veitthia the stagers, Sone examples:
constrointe" are constreiats that
(single-am) pouring ligure fron teapot";
"grasp teapot" stage:
500-goe1 censtrsints: *align the end-estector with teapet handle"
path constraincs: Nona
22

---

allgn wich
suo goel conat raints: teagot agout needs to above cup opening"
path cumatiaiina:: "robut za graaping the Leapol", and "Lhe Leapal must abay upright to avoid apilling"
*oou liquid" stage:
oup opening", "the teapot spout path constraints: "tha teapot spout is directly above the eup epening"
goel raintsiinss:
teagot agout
Lilied pour liquid"
"(bimenuel) fold sleeves to the centert;
graspa left sluwve
right geaapa right sieeve" alage:
suo-goel constraints "left am greepe left sleeve", "right arm greeps right sleeve'
path const raints: Nome
*both asma fold alaeves the center" slage:
su3-goel
fleft
aligrs
center", fright aliges
path comstraimass None
Note:
Fathh consiraint
effector paint
keypaints ingul cost snaller that or equal sero.
namerical
cest, sheze conatraint saciafled
For each stage, you my write 0 or more zub-goal corsenaints and 0 or more path constraints. cumstraints.
Avoid using tite stalements
using path constraints
manipolating detomable objects (eig.r clotaing, cowels).
not need consider colliston avoidance. Forms on what is necessery to comp ete the task. Iapuca to the cunatraints ace as fullows:
"end ectestor's
"3,11" representing the end-ettector pasition. regresenting
"'eyyoonns' opverray shape
keypoint et positions.
or each function, you may use native 2ython functions and KanPy functions.
For greaping stage, you should only write one sab-goal constreint that associates the end-etfector with keypoint. No path constraints its non-grasping stage, you
needed.
not reter to ens-ettester position. In order to nove a keypoint, its associated object must be greeped in one the previous stages, tine.
The robot can only greap ane object
Grasping
be keypoints between the vecsor and fixed axis).
ray
vector,
specify
byy speci angle
You may use nultiple keypointe so specity suctace or volune.
the center of muleiple keypoinne speci position, singic tolding action should consist tNo stzges: ane grasp
place.
St ructure your outpat in e single hynhon code block as foll owe for single-are robot: .. "python
Your explenation how nany stages are involved in the bask and what each stage is about.
aun_stages
#+ stage sub-goal constrainte rit any)
rend_effoctor, keypoinze): your explenation hare.
...
seturn cest
conetraints
1ff stage path ccastraints (:f anyl
det httpl_aath_cossssteoe werrrrattleeendd.e rend_sttector, keypoinze): "**put
hare. tee
return cest Add mare path constraiats if naedad
repeat stages
structure your cutput singla pychon codo block as follows robot:
"python
Your explanation many involved the task and
aun_stages
laft-azn stage sub-eoal conatrainca (if any!
def gast_stsqei_puboos constrsoini conee (fff_ccfeeeor,, koypoints):
neppul ixplanation here, rae
#f8 right-arn stade I sub-goal constraints (if any) sight__steel___uung your explanation here.
"fsctoo, keypoimss):
needed
#1$ lwft slage path contraintt (if any)
def patagge__ttt_ (ffectc,,,,,, keypointa):
ae*put your explenation here,
securn cost
#4# right stage constraints (IF
sight_stagel__. patl (f s)ypuin.s):
23

---

bare.
secuen cost
Add nore path constraints if needad
eupeaa muze aLages
Ouery
Query Task: [ENSTROCT TON]
Quexy Image: TIKAGE WITE KEYPOΣNТ3]
A.7 Implementation Details of Point Tracker
We implement simple point tracker following [121] based on DINOv2 (ViT-S14) [5] that leverages the fact that multiple RGB-D cameras are present and DINOv2 efficient to run at a real-time frequeney.
At initialization, array of 3D keypoint positions k E R are given. We first take the RGB-D captures from each present camera. For each RGB image, we obtain the pixel-wise DINOv2 features following the same procedure in Section A.5 and record their associated 3D world coordinates using calibrated cameras. For each 3D keypoint positions, we aggregate all the features from points that are within 2em from all the cameras. The mean of the aggregated features is recorded as the reference feature for each keypoint, which is kept fixed throughout the task.
After initialization, at each time step, we similarly obtain the pixel-wise features from DINOv2 from all cameras with their 3D world coordinates. To track the keypoints, we calculate cosine similarity between features across all pixels and the reference features. The top 100 matches are selected for each keypoint with a cutoff similarity of 0.6. We then reject outliers for the selected matches by calculating median deviation (m = 2). Additionally, as the tracked keypoints may oscillate small region, we apply a uniform filter with window size of 10 in the end. The entire procedure runs at fixed frequency of 20 Hz.
Note that the implemented point tracker is a simplification from [121] for real-time tracking. We refer readers to [121] for more comprehensive discussion on using self-supervised vision models, such as DINOv2, for point tracking. Alternatively, more specialized point trackers can be used [141- 148].
A.8 Implementation Details of Sub-Goal Solver
The sub-goal problems are implemented and solved using SciPy [128]. The decision variable is single end-effector pose (position and Euler angles) in R$ for single-arm robots and two end-effector poses in 12 for bimanual robot. The bounds for the position terms are the pre-defined workspace bounds, and the bounds for the rotation terms are that the half hemisphere where the end-effector faces down (due to the joint limits of the Franka arm, it often likely to reach joint limit when an end-effector pose faces up). The decision variables are normalized to [-1, 1] based on the bounds. For the first solving iteration, the initial guess chosen to be the current end-effector pose. We sampling-basse global optimization Dual Annealing [129] in the first iteration to quickly search the full space, which is followed by gradient-basee local optimizer SLSQQ [130] that refines the solution. The full procedure takes around 1 second for this iteration. In subsequent iterations, we use the solution from previous stage and only use local optimizer as can quickly adjust to small changes. The optimization cut off with a fixed time budget represented as number of objective function calls to keep the system running at a high frequency.
We discuss the cost terms in the objective function below.
Constraint Violation: We implement constraints as cost terms in the optimization problem, where the returned costs by the ReKep functions are multiplied with large weights.
Scene Collision Avoidance: We nvblox [149] with the PyTorch wrapper [58] to compute the ESDF of the scene in separate node that runs at 20 Hz. The ESDF calculation aggregates the
24

---

depth maps from all available cameras and excludes robot arms using cuRobo and any grasped rigid objects (tracked via masked tracker model Cutie [136]). A collision voxel grid is then calculated using the ESDF and used by other modules in the system. In the sub-goal solver module, we first downsample the gripper points and the grasped object points to have a maximum of 30 points using farthest point sampling. Then we calculate the collision cost using the ESDF voxel grid with linear interpolation with threshold of 15em.
Reachability: Since our decision variables are end-effector poses, which may not be always reach- able by the robot arms, especially in confined spaces, we need to add a cost term that encourages finding solutions with valid joint configurations. Therefore, we solve an IK problem in each iteration of the sub-goal solver using PyBullet [133] and use its residual as a proxy for reachability. We find that this takes around 40% of the time of the full objective function. Alternatively, one may solve the problem in joint space, which would ensure the solution is within the joint limits by enforcing the bounds. We find that this is inefficient with our Python-based implementation as we need to cal- culate forward kinematics for a magnitude of more times in the path solver, because the constraints are evaluated in the task space. To address this while ensuring efficiency, future works can consider using
implementations to solve the problems in joint space [58].
Pose Regularization: We also add small cost that encourages the sub-goal to be close to the current end-effector pose.
Consistency: Since the solver iteratively solves the problem at a high frequency and the noise from the perception pipeline may propagate to the solver, we find useful to include a consistency cost that encourages the solution be close to the previous solution.
(Dual-Arm only) Self-Collision Avoidance: To avoid two arms collide with each other, we compute the pairwise distance between the two point sets, each including the gripper points and grasped object points.
A.9 Implementation Details of Path Solver
The path problems are implemented and solved using SciPy [128]. The number of decision variables is calculated based on the distance between the current end-effector pose and the target end-effector pose. Specifically, we define a fixed step size (20cm and 45 degree) and linearly approximate the desired number of "intermediate poses", which are used as decision variables. As in the sub-goal problem, they are similarly represented using position and Euler angles with the same bounds. For the first solving iteration, the initial guess is chosen to be linear interpolation between the start and the target. We similarly use sampling- sumpling-based based global optimization followed by a maudient-based local optimizer in the first iteration and only use local optimizer in subsequent iterations. After we obtain the solution, represented as a number of intermediate poses, we fit a spline using the current pose, the intermediate poses, and the target pose, which are then densely sampled to executed by the robot.
In the objective function, we first unnormalize the decision variables and use piecewise linear in- terpolation to obtain dense sequence of discrete poses to represent the path (referred to as "dense samples" below). A spline interpolation would be aligned with how we postprocess and execute the solution, but we find linear interpolation to be computationally more efficient. Below we discuss the individual cost terms the objective function.
Constraint Violation: Similar to that in the sub-goal problem, we check violation of the ReKep constraints for each dense sample along the path and penalize with large weights.
Scene Collision Avoidance: The calculation is similar to the sub-goal problem, except that it is calculated for each dense sample. We ignore the collision calculation with 5em radius near the start and the target poses, as this tends to stabilize the solution when solved a high frequency due to various real-world noises. We additionally add table elearance cost that penalizes the path from penetrating the table (or the bottom of the workspace for the wheeled single-arm robot).
25

---

Path Length: We approximate the path length using the dense samples by taking the sum of their differences. Shorter paths are encouraged.
Reachability: We solve an IK problem for each intermediate pose inside the objective function as in the sub-goal problem. See the sub-goal solver section for more details.
Consistency: As in the sub-goal problem, we encourage the solution to be close to the previous one. Specifically, we store the dense samples from the previous iteration. To calculate the solution consistency, we use the pairwise distance between the two sequences (treated as two sets) as an efficient proxy. Alternatively, Hausdorff distance can be used.
(Dual-Arm only) Self-Collision Avoidance: We similarly compute self-collision avoidance for the dual-arm platform as in the sub-goal problem. We also use pairwise distance between the two sequences to efficiently calculate this cost.
A.10 Comparisons with Prior Works on Visual Prompting for Manipulation
There has been several concurrent works investigating the application of visual prompting of VLMs robotic manipulation [99-101, 112, 125]. Below we summarized the differences to highlight the contributions of this work.
Task DoF: In this work, we focus on challenging tasks that require 6 DoF (single arm) or 12 DoF (two arms) motions. However, this is not trivial for existing VLMM which operate on 2D images -as as quoted from MOKA [100]. "current VLMs are not capable of reliably predicting 6-DoF motions" and PIVOT [101], "generalizing higher dimensional spaces such as rotation poses even additional challenges". To tackle this, one key insight from ReKep is that VLMs only need to implicitly specify full 3D rotations by reasoning about keypoints in (x, y, 2) Cartesian coordinates. After this, actual 3D rotations are solved by high-precision and efficient numerical solvers, effectively sidestepping the challenge of explicitly predicting 3D rotations. As a result, the same formulation also naturally generalizes to controlling multiple arms.
High-Level Planning: While many works also consider multi-stage tasks via an language- task planners which are independent from their methods, our formulation takes inspiration from TAMP and organically integrates high-level task planning with low-level actions in unified contin- uous mathematical program. As a result, the method can naturally consider geometric dependencies across stages and do so at a real-time frequency. When failure occurs, it would backtrack to previous stage in which its conditions can still be satisfied. For example, in the "pouring tea" task, the robot can only start tilting the teapot when the teapot spout is aligned with the cup opening. However, if the cup is being moved in the process, it should level the teapot and re-align with the cup. Or if the teapot is being taken from the gripper, should instead re-grasp the teapot.
Low-Level Execution: A common issue with using VLMs is that it is computationally expensive to run, hindering the high-frequency perception-action feedback loops often required for many ma- nipulation tasks. As a result, most of existing works either consider the open-loop settings where visual perception is only used the beginning or only consider the tasks where slow execution is acceptable. Instead, our formulation natively supports a high-frequency perception-ace -action loops by coupling VLMs with point tracker, which effectively enables reactive behaviors via closed-loop exccution despite leveraging very large foundation models.
We uniquely consider using visual prompting for rade-generation, where code may contain arbitrary arithmetic operations on a set of keypoints via visual referring expressions. Although a single point is limiting to capture complex geometric structure, multiple points and their relations can even specify vectors, surfaces, volumes, and their temporal dependen- cies. While being conceptually simple, this offers much higher degree of flexibility which can fully specify 6 DoF or even 12 DoF motions.
Visual Prompting Methods:
26

---

A.11 Extended Discusssions on Limitations
Herein we present additional limitations of the existing system.
Prompting and Robustness: Although we have demonstrated that existing VLMs possess rudi- mentary capabilities at specifying ReKep constraints, we have observed that when dealing with tasks that span many stages with several temporally dependent constraints (A.14), the VLMs lack enough robustness obtain consistent success.
Task-Space Planning: To enable efficient planning, in this work we only consider planning in the task space with the end-effector poses as decision variables. However, we have observed that in certain scenarios, it may be kinematically challenging for robots to achieve the optimized poses as the solver does not explicitly account for the kinematics the robot. Planning joint space can likely resolve the issue but we find be less computationally efficient for our tasks.
Articulated Object Manipulation: In this work, we do not investigate tasks involving articulated objects, as we observed this requires advanced spatial reasoning capabilities that are beyond those of existing Vs.Ms. However, the ReKep formulation may be extended to such tasks by representing different types of joints also by "relations of keypoints". For example, ReKep constraints can be written to constrain certain keypoints to move only alongside line (prismatic joints) or curve (revolute joints). To extend these scenarios, finetuning may be required in [150-152].
Bimanual Coordination: Although we demonstrate the application of ReKep to bimanual manipu- lation, we also identify several important limitations in this domain. Notably, the challenges can be roughly categorized into those pertaining to semantic reasoning of keypoint relations by the VLM and those pertaining to solving for bimanual motions by the optimization solver. For semantic rea- soning, achieve bimanual folding, the VLM needs to possess certain spatial knowledge about which steps should/can be performed together by both arms. For example, the bottom of shirt often needs to be grasped by two hands, each one corner, in order to fold it upwards to align with the collar. Another example in blanket folding is to recognize that the bottom-left corner should be aligned with the top-left corner and the bottom-right comerr should be aligned with the top-right corner, as other matching may lead to self-collision. For optimization solver, as bimanual motion planning dramatically increases the search space of possible motions, which slows down the overall pipeline and more frequently produces less optimal behaviors.
A.12 Simulation Experiments
We additionally implement ReKep in OmniGibson [153] for the Pour Tea task. It is compared to monolithic learning-bas based baseline based on the transformer architecture [154] adopted from RVT [155, 156]. The baseline is trained via imitation learning 100 expert demonstrations, where demonstrations are from scripted policies using privileged simulation information. Success rates are averaged across 100 trials and reported below. Although the monolithic policy excels in training scenarios given its access to expert demonstrations, we observe that ReKep performs significantly stronger in unseen settings, and more importantly, without the need of expert demonstrations.
Seen Poses
Unseen Poses
Unseen Objects
Monolithic Policy
0.93
0.31
0.14
ReKep (Zero-Shot)
0.75
0.68
0.72
A.13 Comparisons of Visual Feature Extractors for Keypoint Proposal
Herein we provide qualitative comparisons of different methods for keypoint proposal. We compare three pre-trained visual feature extractors, each of which represents a popular class of pre-training methods: DINOv2 [5] (self-supervised pre-training). CLIP [94] (vision-language contrastive pre- training), and ViT [157] (supervised pre-training). We also compare to variant that does not use
27

---

Segment Anything (SAM) [132] for its objectness prior. In Fig. 8, we show the extracted feature maps (projected to RGB space) and their clustered keypoints for three different scenes.
We would like note two important observations from the comparisons: 1) objectness prior given by SAM is critical to constrain the keypoint proposal on objects in the scene instead of the background, and 2) while most visual foundation models can provide useful guidance, DINOv2 produces sharper features that can better distinguish fine-grained regions of an object. The first observation can be clearly made by comparing the last column with other candidate methods. The second observation can be made by noting the following places: 1) the unique eyan color on the cup handle in the first scene, 2) the unique colors the box panels the second scene, and 3) the blue/green contrast of the top panel and the side panel in the third scene. Similarly, CLIP provides different features between different object parts, but the features (color saturating from one part the other). ViT, the other hand, produces least distinguishable features between object parts, especially when texture is similar. In general, our observations are aligned with other works that also apply DINOv2 for its fine-graune iinegrained object understanding [114, 121, 123].
less sharp than those of DINOv2
DINOV2
DINO only
B
Figure 8: Comparisons of different methods for keypoint proposal.
28

---

A.14 Case Study of Long-Horizon Tasks
To stress test the presented system, we additionally perform a case study of long-horizon bimanual task of preparing breakfast tray. A successful completion requires 10 stages: 1) grasp table cloth, 2) place table cloth inside the tray, 3) grasp bread from the plate, 4) place bread on top of the table cloth, 5) grasp mug and teacup simultaneously with arms, 6) align mug and teapot, pour into the mug. 8) place the mug inside the tray, 9) grasp the two tray handles simultancously with two arms, and 10) lift up the tray and hand it to the human operator.
This task presents significant challenges for many components in the system. Specifically, we find that existing pipeline for keypoint proposal and constraint specification are incapable of generating the full set of correct keypoints and the full sequence of the correct ReKep constraints, Additionally, observe that result of multiple present objects, our point tracker incapable of consistently tracking all required keypoints in the scene. As result, and constraints, well as keypoint detection at only the start of each stage instead of persistent keypoint tracking. With the above modifications, we find that the system can reasonably perform the task. The initial and final configurations are shown below. The solutions for each stage are shown on the next page. The video result is available at rekep-robot..ithubb.o
resort to manually annotated keypoints
Initial
Final
Figure 9; Initial configuration and successful final configuration of the preparing breakfast tray task.
29

---

Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
Stage 6
Stage 7
Stage 8
Stage 9
Stage 10.
Figure 10: Solution visualization of the preparing breakfast tray task.
30


Reviews:
[{"review_id": "rlfr7poknQ", "reviewer": "Reviewer_yq2j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors proposes ReKep, which turns RGBD observations and language instructions into a series of constraints that map keypoints into numerical cost, based on which, optimized actions could be achieved in the action stage.", "summary_of_recommendation": "Good paper overal while some improvements is welcome.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. See Review's Part.\n2. The reason why you choose DINOv2 is not clear. Besides, you used the SAM with DINOv2 to get keypoints with the masks as the initial region, then why cannot we directly use DINOv2?\n3. The error analysis. As mentioned in other API-calling methods like Instruct2Act, CaP, etc, the generated codes would contain syntax errors, how these errors can be handled? Or current GPT4o can already generate much better code lines? \n4. Can you provide some rough token consumption information for each task execution?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TM9haHrRgw", "reviewer": "Reviewer_v9TF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a VLM-based method for robotic manipulation tasks, named Relational Keypoint Constraints (ReKep). ReKep extracts keypoints from camera images using DINOv2 and interpolates them with segmentation masks. A Vision-Language Model (VLM), GPT4-o, is used to decompose the task into stages for high-level reasoning. The VLM is also queried to generate constraints and costs for each stage and keypoint. Costs are expressed with Python. An optimization solver then finds the end-effector pose that minimizes these costs. The method is evaluated in the real world with both a unimanual robot and a bimanual setup.", "summary_of_recommendation": "This is nice work. ReKep is an innovative approach with real-world experiments!", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- If the method works with arbitrary Python code. How can you ensure that the cost function is well specified. E.g. what happens if the output is -1 * np.linalg.norm(x, y) ?\n\n- Could you do bimanual manipulation with rigid objects, e.g. a handover task or lifting an object? \n\n- Why did you choose VoxPoser as baseline and how can you explain the relatively poor performance of VoxPoser? Is this Zero-Shot?\n\n- Can you give more details on the additional challenges and how are those addressed?  “presenting additional challenges in bimanual coordination” (L. 249)?\n\n- (Optional) Would a table in the related works section help to highlight the novelty of this work w.r.t. [80] and [81], [82]. \n\n- How is the bimanual coordination done? While querying the VLM is done for both robots at the same time, it is unclear to me how this step is executed. Are the two robots independently controlled independently or not?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkEjc6bfsw", "reviewer": "Reviewer_1ZB8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel scalable methodology to (a) break up a given task into stages using off-the-shelf LLMs and VLMs, (b) create constrained problem formulation for each stage such that, (c) off-the-shelf solvers can generate solutions good enough for real-time reactivity. The experiments consist of both single-arm and dual-arm configurations on intermediate-to-hard real-world problems, and show promising results.", "summary_of_recommendation": "I have concerns around complexity of the system and the unclear analysis/lack of tools for systemic analysis of gaps. That said, I find the scope of hardware results quite exciting and the system very impressive. I thus vote for a weak accept and look forward to the author's rebuttal.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- What is the definition of \"in-the-wild\" in the context of task characteristics? Why is stowing a book a \"in-the-wild\" experiment?\n- How will your system handle problems where contact is intentional and therefore the collision costs are variable? For example, compare against PerAct [1] with its explicit modeling of \"collide\" boolean which is off for intentional collisions in high-contact tasks.\n\n[1] Shridhar, M., Manuelli, L. and Fox, D., 2023, March. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning (pp. 785-799). PMLR.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPKFDl5soa", "reviewer": "Reviewer_JmTk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an end-to-end pipeline to solve a manipulation problem given input RGBD images and a language description, with no additional inputs. It uses an LVM to generate keypoints for objects of interest. Input image with keypoints along with the language instruction are passed to a VLM to divide the task into stages and create cost functions for constraints in the form of simple Python functions. Finally, the constraints are passed to a motion optimizer to solve the motion planning problem. This framework can replan in case of failure, as well as generalize to novel objects and tasks. It is also shown to work with bimanual manipulation and potentially other multi-agent settings. The authors perform real-world experiments on a variety of settings such as pouring tea and folding clothes. They also provide some analysis of system error.", "summary_of_recommendation": "This paper proposes a method for language-instructed motion planning that is flexible and suitable for real-world applications. The explanation was mostly clear and intuitive, and visuals provided were sufficient and in high quality. For experiments, the authors provide a wide variety of hardware experiments that address several important research questions. However, the method is somewhat lacking in novelty; most of the pipeline consists of readily available tools with little modification or fine-tuning. The results of the experiments are not conclusive due to the low amount of experiment runs and comparison with only a single baseline which fails on almost every experiment task. Most important criticism is that the prompt provided to the VLM contains detailed explanations of many of the tasks used in the experiments, which raises questions about the actual capability of the VLM.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- How would the proposed method perform with (much) less-informed instructions provided as an initial prompt?\n\n- Current formulation uses ReKep costs in certain constraints only (i.e., requires them all to be <= 0). Is it possible to use these costs in other ways?\n\n- Since LVM takes only the image input, the keypoints it generates can be unrelated to the task at hand, or even worse it may not generate some keypoints crucial to the task. If you have the language instructions anyway, why not use something like a VLM rather than a pure vision-based model?\n\n- In your system error analysis, from what I understand you manually inspect failed experiments and select the first failing component as the cause of failure. However, this should naturally introduce a bias to yield higher error rates for earlier stages of the pipeline. Please clarify.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep are expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. The system is implemented on a mobile single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks. Most of the reviewers find this work interesting, while identifying several weaknesses that need to be addressed in the rebuttal. An important criticism is that the prompt provided to the VLM contains detailed explanations of many of the tasks used in the experiments, which raises questions about the actual capability of the VLM.
The reviews for this paper are mixed, with the majority of the reviewers recommending an accept. A reviewer raised the issue of the over-reliance on VLMs/LLMs without rigorous assessment, and the reliance on annotations. Other reviewers share these concerns, but do not seem to think they are strong enough for rejecting the paper.
Decision: Accept