ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

Authors: wenlong  huang
chen  wang
yunzhu  li
ruohan  zhang
li  fei-fei
relational keypoint constraints
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 08:12:19

Abstract: Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep are expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observation. We present system implementations on a mobile single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models.

Main paper: 
 related works: 
 Structural Representations for Manipulation. Structural representations determine the orchestration of different modules in a manipulation system and yield different implications on the capabilities, assumptions, efficiency, and effectiveness of the system. Rigid-body poses are most commonly used given well-understood rigid-body motions in free space and their efficiency at modeling longrange dependencies of objects [1, [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] . However, since it often requires both geometry and dynamics of environment to be modeled beforehand, various works have studied structural representations using data-driven methods, such as learning object-centric representation [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] , particle-based dynamics [35] [36] [37] [38] [39] [40] [41] , and keypoints or descriptors [3, 4, [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] . Among them, keypoints have shown great promises given their interpretability, efficiency, generalization to instance variations [4] , and ability to model both rigid bodies and deformable objects. However, manual annotation is required per task, thus lacking scalability in open-world settings, which we aim to address in this work.Constrained Optimization in Manipulation. Constraints are often used to impose desired behaviors on robots. Motion planning algorithms use geometric constraints to compute feasible trajectories that avoid obstacles and achieve goals [55] [56] [57] [58] [59] [60] . Contact constraints can be used to plan forceful or contact-rich behaviors [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] . For sequential manipulation tasks, task and motion planning (TAMP) [1, 8, 13] is a widely used framework, often formulated as constraint satisfaction problems [11, [75] [76] [77] [78] [79] [80] with continuous geometric problems as subroutines. Logic-Geometric Programming [81] [82] [83] [84] [85] [86] alternatively formulates a nonlinear constrained program over the entire state trajectory, taking into account both logical and geometric constraints. Constraints can either be manually written or learned from data in the form of manifolds [87] , feasibility model [86, 88] , or signed-distance fields [2, 89] . Inspired by [7] , we formulate sequential manipulation tasks as an integrated continuous mathematical program that is repeatedly solved in a receding-horizon fashion, with the key difference being that the constraints are synthesized by foundation models.Foundation Models for Robotics. Leveraging foundation models for robotics is an active area of research. We refer readers to [90] [91] [92] [93] for overview and recent applications. Here, we focus on VLMs that are capable of incorporating visual inputs [6, [94] [95] [96] [97] [98] for manipulation. However, despite showing promises for open-world planning and goal specification [99] [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [110] [111] [112] [113] , the caption-guided pretraining scheme of VLMs often limits visual details that can be retained about the image [114] [115] [116] [117] . Self-supervised vision models (e.g., DINO [5, 118] ), on the other hand, provide fine-grained pixellevel features useful for various vision and robotic tasks [31, [119] [120] [121] [122] [123] [124] , but there lack effective ways for interpreting open-world semantics pivotal for cross-task generalization. In this work, we leverage their complementary strengths by using DINOv2 [5] for fine-grained keypoint proposal and using GPT-4o [6] for its visual reasoning capability in a supported output modality (code). Similar forms of visual prompting techniques are also explored in concurrent works [99-101, 112, 125] . In this work, we demonstrate ReKep possesses the unique advantages of performing challenging 6-12 DoF tasks, integrated high-level reasoning for reactive replanning, high-frequency closed-loop execution, and generating black-box constraints via visual prompting. More discussion in Appendix A.10. 
method: 
 Herein we discuss: (1) What are Relational Keypoint Constraints (Sec. 3.1)? (2) How to formulate manipulation as a constrained optimization problem with ReKep (Sec. 3.2)? (3) What is our algorithmic instantiation that can efficiently solve the optimization in real-time (Sec. 3.3)? (4) How to automatically obtain ReKep from RGB-D observations and language instructions (Sec. 3.4)?Herein we define a single instance of ReKep. For clarity, we assume that a set of K keypoints have been specified (discussed later in Sec. 3.4). Concretely, each keypoint k i ∈ R 3 refers to a 3D point on the scene surface with Cartesian coordinates, which is dependent on the task semantics and the environment (e.g., grasp point on the handle, spout).A single instance of ReKep is a function f : R K×3 → R that maps an array of keypoints, denoted as k, to an unbounded cost, where f (k) ≤ 0 indicates the constraint is satisfied. The function f is implemented as a stateless Python function, containing NumPy [126] operations on keypoints, which may be nonlinear and nonconvex. In essence, one instance of ReKep encodes one desired spatial relation between keypoints, which may belong to robot arm(s), object parts, and other agents. (i) sub-goal = {f (i) sub-goal,1 (k), . . . , f (i) sub-goal,n (k)} and a set of path constraints C (i) path = {f (i) path,1 (k), . . . , f (i) path,m (k)}, where f (i)sub-goal encodes one keypoint relation to be achieved at the end of stage i, and f (i) path encodes one keypoint relation to be satisfied for every state within stage i. Consider the pouring task in Fig. 2 , which consists of three stages: grasp, align, and pour. The stage-1 sub-goal constraint pulls the end-effector towards the teapot handle. Then stage-2 sub-goal constraint specifies that the teapot spout needs to be on top of the cup opening. Additionally, stage-2 path constraint ensures the teapot stays upright to avoid spillage when transported. Finally, the stage-3 sub-goal constraint specifies the desired pouring angle.Using ReKep as a general tool to represent constraints, we adopt the formulation in [7] and show how a manipulation task can be formulated as a constrained optimization problem involving C (i) sub-goal and C (i) path . We denote the end-effector pose as e ∈ SE(3).To perform the manipulation task, we aim to obtain the overall discrete-time trajectory e 1:T by formulating the control problem as follows:arg mine 1:T ,g 1:N N i=1   λ (i) sub-goal (e gi ) + gi t=gi-1 λ (i) path (e t )   s.t.            e 1 = e init , g 0 = 1, 0 < g i < g i+1 f (k gi ) ≤ 0, ∀f ∈ C (i) sub-goal f (k t ) ≤ 0, ∀f ∈ C (i) path , t = g i-1 , . . . , g i k t+1 = h(k t , e t ), t = 1, . . . , T -1(1) where e t denotes end-effector pose at time t, g i ∈ {1, . . . , T } are the timings of the transition from stage i to i + 1 which are also auxiliary decision variables, k t is the array of keypoint positions at time t, h is a forward model of keypoints, and λ (i) sub-goal and λ (i) path are auxiliary cost functions (e.g., collision avoidance) for the sub-goal and path problems respectively. Namely, for each stage i, the optimization shall find an end-effector pose as next sub-goal, along with its timing, and a sequence of poses e gi-1:gi that achieves the sub-goal, subject to the given set of ReKep constraints and auxiliary costs. This formulation can be considered as direct shooting in trajectory optimization [127] .To solve Eq. 1 in real-time, we employ a decomposition of the full problem and only optimize for the immediate next sub-goal and the corresponding path to reach the sub-goal (pseudo-code in Algorithm 1). All optimization problems are implemented and solved using SciPy [128] with decision variables normalized to [0, 1]. They are initially solved with Dual Annealing [129] with SLSQP [130] as local optimizer (around 1 second) and subsequently solved with only local optimizer based on the previous solution at approximately 10 Hz.The Sub-Goal Problem: We first solve the sub-goal problem to obtain e gi for the current stage i:arg min eg i λ (i) sub-goal (e gi ) s.t. f (k gi ) ≤ 0, ∀f ∈ C (i) sub-goal (2)where λ sub-goal subsumes auxiliary control costs: scene collision avoidance, reachability, pose regularization, solution consistency, and self-collision for bimanual setup (details in A.8). Namely, Eq. 2 attempts to find a sub-goal that satisfies C i sub-goal while minimizing the auxiliary costs. If a stage is concerned with grasping, a grasp metric is also included. In this work, we use AnyGrasp [131] 1 .The Path Problem: After obtaining sub-goal e gi , we solve for a trajectory e t:gi starting from current end-effector pose e t to the sub-goal e gi : arg minEQUATIONwhere λ path subsumes the following auxiliary control costs: scene collision avoidance, reachability, path length, solution consistency, and self-collision in the case of bimanual setup (details in A.9). If the distance to the sub-goal e gi is within a small tolerance ϵ, we progress to the next stage i + 1.Backtracking: Although the sub-problems can be solved at a real-time frequency to react to external disturbances within a stage, it is imperative that the system can replan across stages if any sub-goal constraint from the last stage no longer holds (e.g., cup taken out of the gripper in the pouring task). Specifically, in every control loop, we check for violation ofC (i)path . If one is found, we iteratively backtrack to a previous stage j such that C (j) path is satisfied. Forward Models for Keypoints: To solve Eq. 2 and Eq. 3, one must utilize a forward model h that estimates ∆k from ∆e in the optimization process. As in prior work [4] , we make the rigidity assumption between the end-effector and the "grasped keypoints" (a rigid group of keypoints that belong to the same object or part; obtained from the segmentation model as described in Sec. 3.4.). Namely, given a change in the end-effector pose ∆e, we can calculate the change in keypoint positions by applying the same relative rigid transformation k ′ [grasped] = T ∆e • k[grasped], while assuming other keypoints stay static. We note that this is a "local" assumption in that it is only assumed to hold for the short duration (0.1s) that the problem is solved. Actual keypoint positions are tracked using visual input at 20 Hz and used in every new problem. For more challenging scenarios (e.g., dynamic or contact-rich tasks), a learned or physics-based model may be used.To enable the system to perform tasks in-the-wild given a free-form task instruction, we devise a pipeline using large vision models and vision-language models for keypoint proposal and ReKep generation, which are respectively discussed as follows:Keypoint Proposal: Given an RGB image R h×w×3 , we first extract the patch-wise features F patch ∈ R h ′ ×w ′ ×d from DINOv2 [5] . Then we perform bilinear interpolation to upsample the features to the original image size, F interp ∈ R h×w×d . To ensure the proposal covers all relevant objects in the scene, we extract all masks M = {m 1 , m 2 , . . . , m n } in the scene using Segment Anything (SAM) [132] . For each mask j, we cluster the masked features F interp [m j ] using k-means with k = 5 with a cosine-similarity metric. The centroids of the clusters are used as keypoint candidates, which are projected to a world coordinate R 3 using a calibrated RGB-D camera. Candidates that are within 8cm of others are filtered out. Overall, we find that this procedure is adept at identifying a large percentage of fine-grained and semantically meaningful regions of objects.ReKep Generation: After obtaining the keypoint candidates, we overlay them on the original RGB image with numerical marks. Coupled with the language instruction of the task, we then use visual prompting to query GPT-4o [6] to generate the number of required stages and the corresponding subgoal constraintsC (i)sub-goal and path constraints C (i) path for each stage i (prompts are in A.6). Notably, the functions do not directly manipulate the numerical values of the keypoint positions. Rather, we exploit the strength of VLM to specify spatial relations as arithmetic operations, such as L2 distance or dot product between keypoints, that are only instantiated when invoked with actual keypoint positions tracked by a specialized 3D tracker. Furthermore, an important advantage of using arithmetic operations on a set of keypoint positions is that it can specify 3D rotations in full SO(3) when sufficient points are provided and rigidity between relevant points is enforced, but this is done only when needed depending on task semantic. This enables VLM to reason about 3D rotations with arithmetic operations in 3D Cartesian space, effectively circumventing the need for dealing with alternative 3D rotation representation and the need for performing numerical computation. 
experiments: 
 We aim to answer the following research questions: (1) How well does our framework automatically formulate and synthesize manipulation behaviors (Sec. 4.1)? (2) Can our system generalize to novel objects and manipulation strategies (Sec. 4.2)? (3) How do the individual components contribute to the failure cases of the system (Sec. 4.3)? We validate ReKep on two real robot platforms: a wheeled single-arm platform, and a stationary dual-arm platform (Figure . 3). Additional implementation details can be found in Appendix, including keypoint proposal (A.5), VLM querying (A.6), point trackers (A.7), sub-goal solver (A.8), and path solver (A.9). Tasks. We purposefully select a set of tasks (shown in Fig. 3 ) with the goal of examining the multi-stage (m), inthe-wild (w), bimanual (b), and reactive (r) behaviors of the system. The tasks and their features are Pour Tea (m, w, r), Stow Book (w), Recycle Can (w), Tape Box (w, r), Fold Garment (b), Pack Shoes (b), and Collaborative Folding (b, r). We further evaluate three of the tasks under external disturbances (denoted as "Dist.") by changing poses of task objects during execution.Metric and Baselines. Each setting has 10 trials, in which object poses are randomized. Success rate is reported in Tab. 1. We compare to VoxPoser [106] as a baseline. We evaluate two variants of the system: "Auto" uses foundation models to automatically generate ReKep, and "Annotated (Annot.)" uses human-annotated ReKep.Results. Compared to baselines, ReKep can effectively handle core challenges of each task. For example, it can formulate correct temporal dependency in multi-stage tasks (e.g., spout needs to be aligned with the cup before pouring), leverage commonsense knowledge (e.g., coke cans should be recycled), and construct coordination behaviors in both bimanual settings (e.g., folding left sleeve and right sleeve simultaneously) and human-robot collaboration setting (e.g., folding a large blanket by aligning the four corners together with human). Coupled with an optimization framework, it can also generate kinematically challenging behaviors in confined spaces in the Stow Book task and find a feasible solution that densely fits two shoes within a small volume in the Pack Shoes task. Since the keypoints are tracked at a high frequency, the system can react to external disturbances and replan both within stage and across stages. Despite promising results, we also identify several limitations which are discussed in Sec. 5.Tasks. We systematically evaluate how novel manipulation strategies can be formulated by focusing on a single task, garment folding, but with 8 unique categories of garments, each demanding a unique way of folding and requiring both geometrical and commonsense reasoning. Evaluation is done on the bimanual platform, presenting additional challenges in bimanual coordination.Metric. We use GPT-4o with a prompt containing only generic instructions with no in-context examples. "Strategy Success" measures whether generated ReKep is feasible, which tests both the keypoint proposal module and the VLM, and "Execution Success" measures system success rate given feasible strategies for each clothing. Each is measured with 10 trials.Results. Interestingly, we observe drastically different strategies across categories, many of which are aligned with how humans might fold each garment. For example, it can recognize that two sleeves often are folded together, prior to fully folding the clothes. In cases where using two arms is unnecessary, akin to how humans fold clothes, only one arm is being used. However, we do observe that the VLM may miss certain steps to complete the folding as the operator expected, but we recognize that this is inherently an open-ended problem often based on one's preferences.The modular design of the framework entails an advantage for analyzing system errors due to its interpretability. In this section, we perform an empirical investigation by manually inspecting the failure cases of the experiments reported in Tab. 1, which is then used to calculate the likelihood of a module causing an error while accounting for their temporal dependencies in the pipeline. Results are reported in Fig. 4 . Among the different modules, the point tracker produces the largest portion of errors, as frequent and intermittent occlusion poses significant challenges for accurate tracking. Keypoint proposal and VLM also produce considerable portions of errors, where common cases include the proposal module missing certain keypoints and the VLM referring to incorrect keypoints. The optimization module, on the other hand, does not contribute as much to the failures despite given limited time budget, since there often exist many possible solutions for each problem.Other modules, such as segmentation, 3D reconstruction, and low-level controller, also contribute to some failure cases, but they are relatively insignificant compared to other modules. 
conclusion & limitations: 
 In this work, we presented Relational Keypoint Constraints (ReKep), a structural task representation using constraints that operates on semantic keypoints to specify desired relations between robot arms, object (parts), and other agents in the environment. Coupled with point trackers, we demonstrate that ReKep constraints can be repeatedly and efficiently solved in a hierarchical optimization framework to act as a closed-loop policy that runs at a real-time frequency. We also demonstrate the unique advantage of ReKep in that it can be automatically synthesized by large vision models and vision-language models. Results are shown on two robot platforms and on a variety of tasks featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data, additional training, or environment models. Despite the promises, several limitations remained. First, the optimization framework relies on a forward model of keypoints based on rigidity assumption, albeit a high-frequency feedback loop that relaxes the accuracy requirement of the model. Second, ReKep relies on accurate point tracking to correctly optimize actions in closed-loop, which is itself a challenging 3D vision task due to heavy intermittent occlusions. Lastly, the current formulation assumes a fixed sequence of stages (i.e., skeletons) for each task. Replanning with different skeletons requires running keypoint proposal and VLM at a high-frequency, which poses considerable computational challenges. An extended discussion of limitations can be found in Appendix A.11. 
: 
 it, align the spout with the target container, and then tilt the cup at the correct angle to pour. Here, the constraints encode not only the intermediate sub-goals (e.g., align the spout) but also the transitioning behaviors (e.g., keep the cup upright in transportation), which collectively dictate the spatial, timing, and other combinatorial requirements of the robot's actions in relation to the environment. However, effectively formulating these constraints for a large variety of real-world tasks presents significant challenges. While representing constraints using relative poses between robots and objects is a direct and widely-used approach [1] , rigid-body transformations do not depict geometric details, require obtaining object models a priori, and cannot work on deformable objects. On the other hand, data-driven approaches enable learning constraints directly in visual space [2, 3] . While more flexible, it remains unclear how to effectively collect training data as the number of constraints grows combinatorially in terms of objects and tasks. Therefore, we ask the question: how can we represent constraints in manipulation that are 1) widely applicable: adaptable to tasks that require multi-stage, in-the-wild, bimanual, and reactive behaviors, 2) scalably obtainable: have the potential to be fully automated through the advances in foundation models, and 3) real-time optimizable: can be efficiently solved by off-the-shelf solvers to produce complex manipulation behaviors?In this work, we propose Relational Keypoint Constraints (ReKep). Specifically, ReKep represents constraints as Python functions that map a set of keypoints to a numerical cost, where each keypoint is a task-specific and semantically meaningful 3D point in the scene. Each function is composed of (potentially nonlinear) arithmetic operations on the keypoints and encodes a desired "relation" between them, where the keypoints may belong to different entities in the environment, such as the robot arms, object parts, and other agents. While each keypoint only consists of its 3D Cartesian coordinates in the world frame, multiple keypoints can collectively specify lines, surfaces, and/or 3D rotations if rigidity between keypoints is enforced. We study ReKep in the context of the sequential manipulation problem, where each task involves multiple stages that have spatio-temporal dependencies (e.g., "grasping", "aligning", and "pouring" in the aforementioned example).While constraints are typically defined manually per task [4] , we demonstrate the specific form of ReKep possesses a unique advantage in that they can be automated by pre-trained large vision models (LVM) [5] and vision-language models (VLM) [6] , enabling in-the-wild specification of ReKep from RGB-D observations and free-form language instructions. Specifically, we leverage LVM to propose fine-grained and semantically meaningful keypoints in the scene and VLM to write the constraints as Python functions from visual input overlaid with proposed keypoints. This process can be interpreted as grounding fine-grained spatial relations, often those not easily specified with natural language, in an output modality supported by VLM (code) using visual referral expressions.With the generated constraints, off-the-shelf solvers can be used to produce robot actions by reevaluating the constraints based on tracked keypoints. Inspired by [7] , we employ a hierarchical optimization procedure to first solve a set of waypoints as sub-goals (represented as SE(3) endeffector poses) and then solve the receding-horizon control problem to obtain a dense sequence of actions to achieve each sub-goal. With appropriate instantiation of the problem, we demonstrate that it can be reliably solved at approximately 10 Hz for the tasks considered in this work.Our contributions are summarized as follows: 1) We formulate manipulation tasks as a hierarchical optimization problem with Relational Keypoint Constraints; 2) We devise a pipeline to automatically specify keypoints and constraints using large vision models and vision-language models; 3) We present system implementations on two real-robot platforms that take as input a language instruction and RGB-D observations, and produce multi-stage, in-the-wild, bimanual, and reactive behaviors for a large variety of manipulation tasks, all without task-specific data or environment models. 
 
 Appendix: A. Solve sub-goal problem for stage i to obtain e gi (Eq. 2)12:Solve path problem for stage i to obtain e t:gi (Eq. 3)13:Execute the next m actions e t+1:t+m 14:t ← t + m + 1 15: end while Franka Arm Wheeled BaseFranka ArmOne of our investigated platform is a Franka arm mounted on a wheeled base built with Vention frames (shown in Figure 6 ). Note that the base does not have motors and thus cannot move autonomously, but its mobility nevertheless allows us to investigate the proposed method outside of lab environments.Since our pipeline produces a sequence of 6-DoF end-effector poses, we use position control in all experiments, which is running at a fixed frequency of 20 Hz. Specifically, once the robot is given a target end-effector pose in the world frame, we first clip the pose to the pre-defined workspace. Then we linearly interpolate from the current pose to the target pose with a step size of 5mm for position and 1 degree for rotation. To move to each interpolated pose, we first calculate inverse kinematics to obtain the target joint positions based on current joint positions (IK solver from PyBullet [133] ). Then we use the joint impedance controller from Deoxys [134] to reach to the target joint positions.Two RGB-D cameras, Orbbec Femto Bolt, are mounted on each side of the robot facing the workspace center. The cameras capture RGB images and point clouds a fixed frequency of 20 Hz.We also investigate the method on a stationary dual-arm platform consisting of two Franka arms mounted in front of a tabletop workspace (shown in Figure 7 ). We share the same controller as the wheeled single-arm platform with the exception that the two arms are controlled simultaneously at 20 Hz. Specifically, our pipeline jointly solves two 6-DoF end-effector pose sequences, which are sent to the low-level controller together. The controller subsequently calculates IK for both arms and moves the arms using joint impedance control.Three RGB-D cameras, Orbbec Femto Bolt, are mounted on this platform. Two cameras are mounted on the left and right sides and one camera is mounted in the back. The cameras similarly capture RGB images and point clouds at a fixed frequency of 20 Hz.Below we discuss the evaluation details for the experiments reported in Section 4.1 and Section 4. For each task, 10 initial different configurations of objects are selected, which cover the full workspace but are manually verified to ensure they are kinematically feasible for the robot. For each trial, a human operator restores the scene to the corresponding configuration and initiates the system. Due to the challenge of developing automatic success criteria for the diverse set of objects and environments investigated in this work, success rates are measured by the operator with the criterion reported under each task description below. For experiments involving external disturbances, the set of disturbances for all trials is pre-selected, and one disturbance is applied to each trial. Specifically, the disturbance is introduced by a human operator using hands to change the object's pose. Collision checking is disabled for all tasks involving deformable objects.Pour Tea: The environment consists of a teapot and a cup placed on a counter table in a kitchen setting. The task involves three stages: grasping the handle, aligning the teapot to the top of the cup, and pouring the tea into the cup. The success criterion requires that the teapot remains upright until the pouring stage, and at the end, the spout must be aligned and tilted on top of the cup opening.Recycle Can: The environment includes one of three types of cans (Coke, Zero Coke, Zero Sprite), a recycle bin with a narrow opening (such that the cans may only go in when they are upright), a landfill bin, and a compost bin, all situated inside an office building. The task involves two stages: grasping the can and reorienting it on top of the recycle bin before dropping it. The success criterion is that the can is successfully thrown into the bin.Stow Book: The environment consists of a target book placed on a side table and a real-size bookshelf with a 15cm opening among the placed books, all inside an office environment. The task involves two stages: grasping the target book on the side and stowing it inside the opening in the shelf. The success criterion is that the target book is placed steadily after the robot releases the gripper, and the robot must not bump into the shelf or other placed books.Tape Box: The environment includes a cardboard box, a packaging tape with a dispenser sitting on top of the box that already has one side taped, and a human user collaborating with the robot. The tape has already been unrolled to be enough for taping because unrolling typically requires a large force that exceeds the limit of the robot arm. The task involves two stages: while a human operator is squeezing the box, the robot needs to grasp the tape and align it to the correct side to complete the taping. The success criterion is that the tape must end up in the correct position such that it is aligned with the seam.(Bimanual) Fold Garment: The environment consists of a sweater placed flat close to the workspace center, with small deformations on the sleeves, neck, and bottom. The task requires four stages: grasping both sleeves, folding them to the middle, grasping the neck, and folding it to the bottom. The success criterion does not enforce consistent stages; as long as the sweater is folded such that it occupies at most half of the original surface size, it is regarded as a success.(Bimanual) Pack Shoes: The environment includes an empty shoe box placed close to the workspace center, with two shoes placed on opposite sides of the box in random poses. The task involves two stages: grasping the shoes simultaneously and placing them in the shoe box. The success criterion does not enforce consistent stages; as long as the shoes are placed into the box without being stacked together or causing bimanual self-collision, it is considered successful.(Bimanual) Collaborative Folding: The environment consists of a large blanket (pre-folded to an appropriate size that occupies about 70% of the workspace due to its size exceeding the workspace limit) and a human user collaborating with the robot. The task involves two stages: the robot must grasp the two corners of the blanket opposite to the human user, and the second stage is aligning the two corners with the two corners that the human has grasped. The success criterion is that the robot has grasped the correct corners and can align them with the correct human arms (left-left, right-right).We use VoxPoser [106] as the main baseline method as it makes similar assumptions that no taskspecific data or pre-defined motion primitives are required. We adapt VoxPoser to our setting with certain modifications to ensure fairness in comparisons. Specifically, we use the same VLM, GPT-4o [6] , that takes the same camera input. We also augment the original prompt from the paper with the prompt used in this work to ensure it has sufficient context. We only use the affordance, rotation, and gripper action value maps and ignore the avoidance and velocity value maps because they are not necessary for our tasks. We also only consider the scenario where the "entity of interest" is the robot end-effector instead of objects in the scene. The latter is tailored for pushing task, which is not being studied in this work. We use OWL-ViT [135] for open-vocabulary object detection, SAM [132] for initial-frame segmentation, and Cutie [136] for mask tracking.The dual-arm robot is tasked with folding eight different categories of clothing. We use two metrics for evaluation: "Strategy Success" and "Execution Success," where the former evaluates whether keypoints are proposed and constraints are written appropriately, and the latter evaluates the robotic system's execution given successful strategies.To evaluate "Strategy Success," the garment is initialized close to the center of the workspace. A back-mounted RGB-D camera captures the RGB image. Then, the keypoint proposal module generates keypoint candidates using the captured image, which are then overlaid on top of the original image with numerical marks {0, . . . , K -1}. The overlaid image, along with the same generic prompt, is fed into GPT-4 [6] to generate the ReKep constraints. Since folding garments is itself an open-ended problem without ground-truth strategies, we manually judge if the proposed keypoints and the generated constraints are correct. Note that since the constraints are to be executed by a bimanual robot, and the constraints are almost always connecting (folding) two keypoints such that they are aligned, correctness is measured by whether it is (potentially) executable by the robot without causing self-collision (arms crossing over to opposite sides) and whether the folding strategy can fold the garment to at most half of its original surface area.To evaluate "Execution Success," we take the generated strategies in the previous section that are marked as successful for each garment and execute the sequence on the dual-arm platform, with a total of 10 trials for each garment. Point tracking is disabled as we observe that our point tracker predicts unstable tracks when the garment is potentially folded many times. Success is measured by whether the garment is folded such that its surface area is at most half of its original surface area.Herein we describe how keypoint candidates in a scene are generated. For each platform, we use one of the mounted RGB-D to capture an image of size h × w × 3, depending on which camera has the best holistic view of the environment, as all the keypoints need to be present in the first frame for the proposed method. Given the captured image, we first use DINOv2 with registers (ViT-S14) [5, 137] to extract the patch-wise features F patch ∈ R h ′ ×w ′ ×d . Then we perform bilinear interpolation to upsample the features to the original image size, F interp ∈ R h×w×d . To ensure the proposal covers all relevant objects in the scene, we extract all masks M = {m 1 , m 2 , . . . , m n } in the scene using Segment Anything (SAM) [132] . Within each mask m i , we apply PCA to project the features to three dimensions, F PCA = PCA(F resized [m i ], 3). We find that applying PCA improves the clustering as it often removes details and artifacts related to texture that are not useful for our tasks. For each mask j, we cluster the masked features F interp [m j ] using k-means with k = 5 with the Euclidean distance metric. The median centroids of the clusters are used as keypoint candidates, which are projected to a world coordinate R 3 using a calibrated RGB-D camera. Note that we also store which keypoint candidates originate from the same mask, which is later used as part of the rigidity assumption in the optimization loops described in Sec. 3.3. Candidates outside of the workspace bounds are filtered out. To avoid many points cluttered in a small region, we additionally use Mean Shift [138, 139] (with a bandwidth 8cm) to filter out points that are close to each other. Finally, the centroids are taken as final candidates. Alternatively, one may develop a pipeline using only segmentation models [132, 140] , but we leave comparisons to future work.After we obtain the keypoint candidates, they are overlaid on the captured RGB image with numerical marks {0, . . . , K -1}. Then the image and the task instruction are fed into a vision-language model with the prompt described below. The prompt contains only generic instructions with no image-text in-context examples, although a few text-based examples are given to concretely explain the proposed method and the expected output from the model. Note that the majority of the investigated tasks are not discussed in the provided prompt. As a result, the VLM is tasked with generating ReKep constraints by leveraging its internalized world knowledge.For the experiments conducted in this work, we use GPT-4o [6] as it is one of the latest available models at the time of the experiments. However, due to rapid advancement in this field, the pipeline can directly benefit from newer models that have better vision-language reasoning. Correspondingly, we observe different models exhibit different behaviors when given the same prompt (with the observation that newer models typically require less fine-grained instructions). As a result, instead of developing the best prompt for the suite of tasks in this work, we focus on demonstrating a full-stack pipeline consisting a key component that can be automated and continuously improved by future development.Suppose you are controlling a robot to perform manipulation tasks by writing constraint functions in Python.The manipulation task is given as an image of the environment, overlayed with keypoints marked with their indices, along with a text instruction. The instruction starts with a parenthesis indicating whether the robot has a single arm or is bimanual. For each given task, please perform the following steps: -Determine how many stages are involved in the task. Grasping must be an independent stage. Some examples:-"(single-arm) pouring tea from teapot": -3 stages: "grasp teapot", "align teapot with cup opening", and "pour liquid" -"(single-arm) put red block on top of blue block":-3 stages: "grasp red block", "align red block on top of blue block", and "release red block" -"(bimanual) fold sleeves to the center":-2 stages: "left arm grasps left sleeve and right arm grasps right sleeve" and "both arms fold sleeves to the center" -"(bimanual) fold a jacket":-3 stages: "left arm grasps left sleeve and right arm grasps right sleeve", "both arms fold sleeves to the center", and "grasp the neck with one arm (the other arm stays in place)", and "align the neck with the bottom" -For each stage, write two kinds of constraints, "sub-goal constraints" and "path constraints". The "sub-goal constraints" are constraints that must be satisfied ** at the end of the stage ** , while the "path constraints" are constraints that must be satisfied ** within the stage ** . Some examples:-"(single-arm) pouring liquid from teapot":-"grasp teapot" stage:-sub-goal constraints: "align the end-effector with the teapot handle" -path constraints: None -"align teapot with cup opening" stage:-sub-goal constraints: "the teapot spout needs to be 10cm above the cup opening" -path constraints: "robot is grasping teapot", and "the teapot must stay upright to avoid spilling" -"pour liquid" stage:-sub-goal constraints: "the teapot spout needs to be 5cm above the cup opening", "the teapot spout must be tilted to pour liquid" -path constraints: "the teapot spout is directly above the cup opening" -"(bimanual) fold sleeves to the center":-"left arm grasps left sleeve and right arm grasps right sleeve" stage:-sub-goal constraints: "left arm grasps left sleeve", "right arm grasps right sleeve" -path constraints: None -"both arms fold sleeves to the center" stage:-sub-goal constraints: "left sleeve aligns with the center", "right sleeve aligns with the center" -path constraints: None Note:-Each constraint takes a dummy end-effector point and a set of keypoints as input and returns a numerical cost, where the constraint is satisfied if the cost is smaller than or equal to zero. -For each stage, you may write 0 or more sub-goal constraints and 0 or more path constraints. -Avoid using "if" statements in your constraints.-Avoid using path constraints when manipulating deformable objects (e.g., clothing, towels).-You do not need to consider collision avoidance. Focus on what is necessary to complete the task.-Inputs to the constraints are as follows:-`end_effector`: np.array of shape `(3,)`representing the end-effector position.-`keypoints`: np.array of shape `(K, 3)`representing the keypoint positions. -Inside of each function, you may use native Python functions and NumPy functions.-For grasping stage, you should only write one sub-goal constraint that associates the end-effector with a keypoint. No path constraints are needed. -For non-grasping stage, you should not refer to the end-effector position.-In order to move a keypoint, its associated object must be grasped in one of the previous stages.-The robot can only grasp one object at a time.-Grasping must be an independent stage from other stages.-You may use two keypoints to form a vector, which can be used to specify a rotation (by specifying the angle between the vector and a fixed axis). -You may use multiple keypoints to specify a surface or volume.-You may also use the center of multiple keypoints to specify a position.-A single folding action should consist of two stages: one grasp and one place.Structure your output in a single python code block as follows for single-arm robot: ```python # Your explanation of how many stages are involved in the task and what each stage is about. # ... num_stages = ? ### stage 1 sub-goal constraints (if any) def stage1_subgoal_constraint1(end_effector, keypoints):"""Put your explanation here.""" ... return cost # Add more sub-goal constraints if needed ### stage 1 path constraints (if any) def stage1_path_constraint1(end_effector, keypoints):"""Put your explanation here.""" ... return cost # Add more path constraints if needed # repeat for more stages ... ``S tructure your output in a single python code block as follows for bimanual robot: ```python # Your explanation of how many stages are involved in the task and what each stage is about. # ... num_stages = ? ### left-arm stage 1 sub-goal constraints (if any) def left_stage1_subgoal_constraint1(end_effector, keypoints):"""Put your explanation here.""" ... return cost ### right-arm stage 1 sub-goal constraints (if any) def right_stage1_subgoal_constraint1(end_effector, keypoints): """Put your explanation here.""" ... return cost # Add more sub-goal constraints if needed ### left stage 1 path constraints (if any) def left_stage1_path_constraint1(end_effector, keypoints):"""Put your explanation here.""" ... return cost ### right stage 1 path constraints (if any) def right_stage1_path_constraint1(end_effector, keypoints):"""Put your explanation here. We implement a simple point tracker following [121] based on DINOv2 (ViT-S14) [5] that leverages the fact that multiple RGB-D cameras are present and DINOv2 is efficient to run at a real-time frequency.At initialization, an array of 3D keypoint positions k ∈ R are given. We first take the RGB-D captures from each present camera. For each RGB image, we obtain the pixel-wise DINOv2 features following the same procedure in Section A.5 and record their associated 3D world coordinates using calibrated cameras. For each 3D keypoint positions, we aggregate all the features from points that are within 2cm from all the cameras. The mean of the aggregated features is recorded as the reference feature for each keypoint, which is kept fixed throughout the task.After initialization, at each time step, we similarly obtain the pixel-wise features from DINOv2 from all cameras with their 3D world coordinates. To track the keypoints, we calculate cosine similarity between features across all pixels and the reference features. The top 100 matches are selected for each keypoint with a cutoff similarity of 0.6. We then reject outliers for the selected matches by calculating median deviation (m = 2). Additionally, as the tracked keypoints may oscillate in a small region, we apply a uniform filter with a window size of in the end. The entire procedure runs at a fixed frequency of 20 Hz.Note that the implemented point tracker is a simplification from [121] for real-time tracking. We refer readers to [121] for more comprehensive discussion on using self-supervised vision models, such as DINOv2, for point tracking. Alternatively, more specialized point trackers can be used [141] [142] [143] [144] [145] [146] [147] [148] .The sub-goal problems are implemented and solved using SciPy [128] . The decision variable is a single end-effector pose (position and Euler angles) in R 6 for single-arm robots and two end-effector poses in R 12 for bimanual robot. The bounds for the position terms are the pre-defined workspace bounds, and the bounds for the rotation terms are that the half hemisphere where the end-effector faces down (due to the joint limits of the Franka arm, it is often likely to reach joint limit when an end-effector pose faces up). The decision variables are normalized to [-1, 1] based on the bounds. For the first solving iteration, the initial guess is chosen to be the current end-effector pose. We use sampling-based global optimization Dual Annealing [129] in the first iteration to quickly search the full space, which is followed by a gradient-based local optimizer SLSQP [130] that refines the solution. The full procedure takes around 1 second for this iteration. In subsequent iterations, we use the solution from previous stage and only use local optimizer as it can quickly adjust to small changes. The optimization is cut off with a fixed time budget represented as number of objective function calls to keep the system running at a high frequency.We discuss the cost terms in the objective function below.Constraint Violation: We implement constraints as cost terms in the optimization problem, where the returned costs by the ReKep functions are multiplied with large weights.We use nvblox [149] with the PyTorch wrapper [58] to compute the ESDF of the scene in a separate node that runs at 20 Hz. The ESDF calculation aggregates the depth maps from all available cameras and excludes robot arms using cuRobo and any grasped rigid objects (tracked via a masked tracker model Cutie [136] ). A collision voxel grid is then calculated using the ESDF and used by other modules in the system. In the sub-goal solver module, we first downsample the gripper points and the grasped object points to have a maximum of 30 points using farthest point sampling. Then we calculate the collision cost using the ESDF voxel grid with linear interpolation with a threshold of 15cm.Reachability: Since our decision variables are end-effector poses, which may not be always reachable by the robot arms, especially in confined spaces, we need to add a cost term that encourages finding solutions with valid joint configurations. Therefore, we solve an IK problem in each iteration of the sub-goal solver using PyBullet [133] and use its residual as a proxy for reachability. We find that this takes around 40% of the time of the full objective function. Alternatively, one may solve the problem in joint space, which ensure the solution is within the joint limits by enforcing the bounds. We find that this is inefficient with our Python-based implementation as we need to calculate forward kinematics for a magnitude of more times in the path solver, because the constraints are evaluated in the task space. To address this while ensuring efficiency, future works can consider using hardware-accelerated implementations to solve the problems in joint space [58] .Pose Regularization: We also add a small cost that encourages the sub-goal to be close to the current end-effector pose.Consistency: Since the solver iteratively solves the problem at a high frequency and the noise from the perception pipeline may propagate to the solver, we find it useful to include a consistency cost that encourages the solution to be close to the previous solution.(Dual-Arm only) Self-Collision Avoidance: To avoid two arms collide with each other, we compute the pairwise distance between the two point sets, each including the gripper points and grasped object points.The path problems are implemented and solved using SciPy [128] . The number of decision variables is calculated based on the distance between the current end-effector pose and the target end-effector pose. Specifically, we define a fixed step size (20cm and 45 degree) and linearly approximate the desired number of "intermediate poses", which are used as decision variables. As in the sub-goal problem, they are similarly represented using position and Euler angles with the same bounds. For the first solving iteration, the initial guess is chosen to be linear interpolation between the start and the target. We similarly use sampling-based global optimization followed by a gradient-based local optimizer in the first iteration and only use local optimizer in subsequent iterations. After we obtain the solution, represented as a number of intermediate poses, we fit a spline using the current pose, the intermediate poses, and the target pose, which are then densely sampled to be executed by the robot.In the objective function, we first unnormalize the decision variables and use piecewise linear interpolation to obtain a dense sequence of discrete poses to represent the path (referred to as "dense samples" below). A spline interpolation would be aligned with how we postprocess and execute the solution, but we find linear interpolation to be computationally more efficient. Below we discuss the individual cost terms in the objective function.Constraint Violation: Similar to that in the sub-goal problem, we check violation of the ReKep constraints for each dense sample along the path and penalize with large weights.The calculation is similar to the sub-goal problem, except that it is calculated for each dense sample. We ignore the collision calculation with a 5cm radius near the start and the target poses, as this tends to stabilize the solution when solved at a high frequency due to various real-world noises. We additionally add a table clearance cost that penalizes the path from penetrating the table (or the bottom of the workspace for the wheeled single-arm robot).Path Length: We approximate the path length using the dense samples by taking the sum of their differences. Shorter paths are encouraged.Reachability: We solve an IK problem for each intermediate pose inside the objective function as in the sub-goal problem. See the sub-goal solver section for more details.Consistency: As in the sub-goal problem, we encourage the solution to be close to the previous one. Specifically, we store the dense samples from the previous iteration. To calculate the solution consistency, we use the pairwise distance between the two sequences (treated as two sets) as an efficient proxy. Alternatively, Hausdorff distance can be used.(Dual-Arm only) Self-Collision Avoidance: We similarly compute self-collision avoidance for the dual-arm platform as in the sub-goal problem. We also use pairwise distance between the two sequences to efficiently calculate this cost.There has been several concurrent works investigating the application of visual prompting of VLMs to robotic manipulation [99-101, 112, 125] . Below we summarized the differences to highlight the contributions of this work.Task DoF: In this work, we focus on challenging tasks that require 6 DoF (single arm) or 12 DoF (two arms) motions. However, this is not trivial for existing VLMs which operate on 2D images -as quoted from MOKA [100] , "current VLMs are not capable of reliably predicting 6-DoF motions" and PIVOT [101] , "generalizing to higher dimensional spaces such as rotation poses even additional challenges". To tackle this, one key insight from ReKep is that VLMs only need to implicitly specify full 3D rotations by reasoning about keypoints in (x, y, z) Cartesian coordinates. After this, actual 3D rotations are solved by high-precision and efficient numerical solvers, effectively sidestepping the challenge of explicitly predicting 3D rotations. As a result, the same formulation also naturally generalizes to controlling multiple arms.High-Level Planning: While many works also consider multi-stage tasks via an language-based task planners which are independent from their methods, our formulation takes inspiration from TAMP and organically integrates high-level task planning with low-level actions in a unified continuous mathematical program. As a result, the method can naturally consider geometric dependencies across stages and do so at a real-time frequency. When a failure occurs, it would backtrack to a previous stage in which its conditions can still be satisfied. For example, in the "pouring tea" task, the robot can only start tilting the teapot when the teapot spout is aligned with the cup opening. However, if the cup is being moved in the process, it should level the teapot and re-align with the cup. Or if the teapot is being taken from the gripper, it should instead re-grasp the teapot.Low-Level Execution: A common issue with using VLMs is that it is computationally expensive to run, hindering the high-frequency perception-action feedback loops often required for many manipulation tasks. As a result, most of existing works either consider the open-loop settings where visual perception is only used in the beginning or only consider the tasks where slow execution is acceptable. Instead, our formulation natively supports a high-frequency perception-action loops by coupling VLMs with a point tracker, which effectively enables reactive behaviors via closed-loop execution despite leveraging very large foundation models.Visual Prompting Methods: We uniquely consider using visual prompting for code-generation, where code may contain arbitrary arithmetic operations on a set of keypoints via visual referring expressions. Although a single point is limiting to capture complex geometric structure, multiple points and their relations can even specify vectors, surfaces, volumes, and their temporal dependencies. While being conceptually simple, this offers a much higher degree of flexibility which can fully specify 6 DoF or even 12 DoF motions.Herein we present additional limitations of the existing system.Prompting and Robustness: Although we have demonstrated that existing VLMs possess rudimentary capabilities at specifying ReKep constraints, we have observed that when dealing with tasks that span many stages with several temporally dependent constraints (A.14), the VLMs lack enough robustness to obtain consistent success.Task-Space Planning: To enable efficient planning, in this work we only consider planning in the task space with the end-effector poses as decision variables. However, we have observed that in certain scenarios, it may be kinematically challenging for robots to achieve the optimized poses as the solver does not explicitly account for the kinematics of the robot. Planning in joint space can likely resolve the issue but we find it to be less computationally efficient for our tasks.Articulated Object Manipulation: In this work, we do not investigate tasks involving articulated objects, as we observed this requires advanced spatial reasoning capabilities that are beyond those of existing VLMs. However, the ReKep formulation may be extended to such tasks by representing different types of joints also by "relations of keypoints". For example, ReKep constraints can be written to constrain certain keypoints to move only alongside a line (prismatic joints) or a curve (revolute joints). To extend to these scenarios, finetuning may be required as in [150] [151] [152] .Bimanual Coordination: Although we demonstrate the application of ReKep to bimanual manipulation, we also identify several important limitations in this domain. Notably, the challenges can be roughly categorized into those pertaining to semantic reasoning of keypoint relations by the VLM and those pertaining to solving for bimanual motions by the optimization solver. For semantic reasoning, to achieve bimanual folding, the VLM needs to possess certain spatial knowledge about which steps should/can be performed together by both arms. For example, the bottom of a shirt often needs to be grasped by two hands, each at one corner, in order to fold it upwards to align with the collar. Another example in blanket folding is to recognize that the bottom-left corner should be aligned with the top-left corner and the bottom-right corner should be aligned with the top-right corner, as other matching may lead to self-collision. For optimization solver, as bimanual motion planning dramatically increases the search space of possible motions, which slows down the overall pipeline and more frequently produces less optimal behaviors.We additionally implement ReKep in OmniGibson [153] for the Pour Tea task. It is compared to a monolithic learning-based baseline based on the transformer architecture [154] adopted from RVT [155, 156] . The baseline is trained via imitation learning on 100 expert demonstrations, where demonstrations are from scripted policies using privileged simulation information. Success rates are averaged across 100 trials and reported below. Although the monolithic policy excels in training scenarios given its access to expert demonstrations, we observe that ReKep performs significantly stronger in unseen settings, and more importantly, without the need of expert demonstrations. Herein we provide qualitative comparisons of different methods for keypoint proposal. We compare three pre-trained visual feature extractors, each of which represents a popular class of pre-training methods: DINOv2 [5] (self-supervised pre-training), CLIP [94] (vision-language contrastive pretraining), and ViT [157] (supervised pre-training). We also compare to a variant that does not use Segment Anything (SAM) [132] for its objectness prior. In Fig. 8 , we show the extracted feature maps (projected to RGB space) and their clustered keypoints for three different scenes.We would like to note two important observations from the comparisons: 1) objectness prior given by SAM is critical to constrain the keypoint proposal on objects in the scene instead of on the background, and 2) while most visual foundation models can provide useful guidance, DINOv2 produces sharper features that can better distinguish fine-grained regions of an object. The first observation can be clearly made by comparing the last column with other candidate methods. The second observation can be made by noting the following places: 1) the unique cyan color on the cup handle in the first scene, 2) the unique colors of the box panels in the second scene, and 3) the blue/green contrast of the top panel and the side panel in the third scene. Similarly, CLIP provides different features between different object parts, but the features are less sharp than those of DINOv2 (color saturating from one part to the other). ViT, on the other hand, produces least distinguishable features between object parts, especially when texture is similar. In general, our observations are aligned with other works that also apply DINOv2 for its fine-grained object understanding [114, 121, 123] . To stress test the presented system, we additionally perform a case study of a long-horizon bimanual task of preparing breakfast tray. A successful completion requires 10 stages: 1) grasp table cloth, 2) place table cloth inside the tray, 3) grasp bread from the plate, 4) place bread on top of the table cloth, 5) grasp mug and teacup simultaneously with two arms, 6) align mug and teapot, 7) pour tea into the mug, 8) place the mug inside the tray, 9) grasp the two tray handles simultaneously with two arms, and 10) lift up the tray and hand it to the human operator.This task presents significant challenges for many components in the system. Specifically, we find that existing pipeline for keypoint proposal and constraint specification are incapable of generating the full set of correct keypoints and the full sequence of the correct ReKep constraints. Additionally, we observe that as a result of multiple present objects, our point tracker is incapable of consistently tracking all required keypoints in the scene. As a result, we resort to manually annotated keypoints and constraints, as well as keypoint detection at only the start of each stage instead of persistent keypoint tracking. With the above modifications, we find that the system can reasonably perform the task. The initial and final configurations are shown below. The solutions for each stage are shown on the next page. The video result is available at rekep-robot.github.io. Since AnyGrasp is a grasp detector instead of a metric and is computationally expensive to run in optimization loops, we always return the grasp closest to a specified "grasp keypoint" by exploiting the fact that ReKep related to grasping always associates a dummy keypoint on the end-effector and one actual keypoint. 


Decision: This paper introduces Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep are expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. The system is implemented on a mobile single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks. Most of the reviewers find this work interesting, while identifying several weaknesses that need to be addressed in the rebuttal. An important criticism is that the prompt provided to the VLM contains detailed explanations of many of the tasks used in the experiments, which raises questions about the actual capability of the VLM.
The reviews for this paper are mixed, with the majority of the reviewers recommending an accept. A reviewer raised the issue of the over-reliance on VLMs/LLMs without rigorous assessment, and the reliance on annotations. Other reviewers share these concerns, but do not seem to think they are strong enough for rejecting the paper.

Reviews: review: title: title: Reviews for the ReKep
paper_summary: paper_summary: The authors proposes ReKep, which turns RGBD observations and language instructions into a series of constraints that map keypoints into numerical cost, based on which, optimized actions could be achieved in the action stage.
main_review: main_review: ## Strengths:
1. One very interesting trial to combine VLM with classic numerical optimization methods. The optimization methods offer the additional numerical stability that is currently almost impossible for the VLM. 
2. The ReKap pipeline does not require human labelling.
3. I found the implementation details in the appendix quite informative.
4. Real-wrold experiments are conducted to showcase the effectiveness of the ReKep.

## Weakness:
1. There are already some works[1,2] that utilized SoM + GPT4v to deal with the robotic tasks which makes this paper not that impressive.  Could the author make some comparisons among them?

2. Still limited to the tabletop manipulation tasks. Can this method be extended to the 3D ones, or harder ones, for the articulation tasks, as done in [3] where ChatGPT is used or in [4], where the model is fine-tuned?

3. Seems that the proposed system's path constrains mainly rely on cuRobo[5] library. Can the authors emphasize the contributions besides the usage of the cuRobo. 

4. The baseline was only VoxPoser, more baselines are welcome. 


[1] Huang, Haoxu, et al. "Copa: General robotic manipulation through spatial constraints of parts with foundation models." arXiv preprint arXiv:2403.08248 (2024). 

[2] Hu, Yingdong, et al. "Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning." arXiv preprint arXiv:2311.17842 (2023).

[3] Xia, Wenke, et al. "Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs." arXiv preprint arXiv:2311.02847 (2023). 

[4] Huang, Siyuan, et al. "A3VLM: Actionable Articulation-Aware Vision Language Model." arXiv preprint arXiv:2406.07549 (2024).

[5] https://curobo.org/
questions: questions: 1. See Review's Part.
2. The reason why you choose DINOv2 is not clear. Besides, you used the SAM with DINOv2 to get keypoints with the masks as the initial region, then why cannot we directly use DINOv2?
3. The error analysis. As mentioned in other API-calling methods like Instruct2Act, CaP, etc, the generated codes would contain syntax errors, how these errors can be handled? Or current GPT4o can already generate much better code lines? 
4. Can you provide some rough token consumption information for each task execution?
limitations: limitations: 2
review_summary: review_summary: Good paper overal while some improvements is welcome.
title: title: Innovative approach; Some questions about the experiments
paper_summary: paper_summary: This work presents a VLM-based method for robotic manipulation tasks, named Relational Keypoint Constraints (ReKep). ReKep extracts keypoints from camera images using DINOv2 and interpolates them with segmentation masks. A Vision-Language Model (VLM), GPT4-o, is used to decompose the task into stages for high-level reasoning. The VLM is also queried to generate constraints and costs for each stage and keypoint. Costs are expressed with Python. An optimization solver then finds the end-effector pose that minimizes these costs. The method is evaluated in the real world with both a unimanual robot and a bimanual setup.
main_review: main_review: The paper reads well, and I appreciate the real-world experiments. The concept of using relational keypoint constraints combined with an optimization step is particularly compelling. A key limiting factor appears to be the tracking of keypoints, which is acknowledged as a limitation by the authors.

However, some concerns remain. Details about the Voxposer baseline are missing, and the claim of handling "a large variety of manipulation tasks" seems a little bit overstated to me, as the experiments primarily focus on bimanual manipulation with compliant objects. Without some details about the VoxPoser baseline it makes it difficult for me to assess the relative performance of ReKep. Additionally, the experiments have a relatively short horizon, and the bimanual experiments are quite similar, e.g. similar clothes. It appears that the bimanual manipulation is achieved by executing the same method simultaneously with both arms, but the paper does not discuss the coordination between the two arms, which I see as a key challenge of bimanual manipulation. The term “Mobile arm” in this work and “mobile base” in Fig.6 seem to indicate mobile manipulation. I would prefer the terms “wheeled base” in Fig 6 or the term “single arm”.

Minor:

-	Line 36: combinatorially
-	Line 52 nonscientific language: ReKep enjoys …
-	Line 78: reprensetation
-	(Suggestion) Line 213ff: Instead of  a,b,c,d use m,w,b,r this makes the mapping easier. Alternatively, use a table with checkboxes.
-	Line 181: The symbol R has duplicate use in this work. 
-	Video: The video is longer than the recommended length.
-	Video 0:45: Speed-up is missing for some experiments.
-	Fig2 shows a point cloud, but the text mention that keypoints are overlayed in the RGB image, but the Fig2 suggests that it’s the point cloud. 
-	Fig3 is a little bit confusing. Could you overlay the waypoints in a camera image instead of a point cloud? The point cloud seems noisy,and I find it difficult to identify the waypoints.
questions: questions: - If the method works with arbitrary Python code. How can you ensure that the cost function is well specified. E.g. what happens if the output is -1 * np.linalg.norm(x, y) ?

- Could you do bimanual manipulation with rigid objects, e.g. a handover task or lifting an object? 

- Why did you choose VoxPoser as baseline and how can you explain the relatively poor performance of VoxPoser? Is this Zero-Shot?

- Can you give more details on the additional challenges and how are those addressed?  “presenting additional challenges in bimanual coordination” (L. 249)?

- (Optional) Would a table in the related works section help to highlight the novelty of this work w.r.t. [80] and [81], [82]. 

- How is the bimanual coordination done? While querying the VLM is done for both robots at the same time, it is unclear to me how this step is executed. Are the two robots independently controlled independently or not?
limitations: limitations: 2
review_summary: review_summary: This is nice work. ReKep is an innovative approach with real-world experiments!
title: title: Official Review
paper_summary: paper_summary: This paper proposes a novel scalable methodology to (a) break up a given task into stages using off-the-shelf LLMs and VLMs, (b) create constrained problem formulation for each stage such that, (c) off-the-shelf solvers can generate solutions good enough for real-time reactivity. The experiments consist of both single-arm and dual-arm configurations on intermediate-to-hard real-world problems, and show promising results.
main_review: main_review: Strengths:
- Novel system setup connecting VLMs and LLMs to verifiable, and interpretable low-level solvers for solving 3D problems
- Results show promising generalization of the methodology to several classes of problems
- Paper flows well with relevant coverage of literature review. However the formulation is a bit dense, and paper may benefit from simplifying its explanation further.
- The paper outlines an impressive systems effort in creating a full vertical of TAMP-like problem solving but while using much more generic VLMs and LLMs
- The hardware experiments are exciting in their scope to handle both rigid and deformable objects (i.e. clothes)
- The system may also be of great impact as an expert data generator, where non-modular specialized models may be trained on the data generated for complex 3D problems
- Appreciate the high throughput and reactive nature of solution

Weakness:
- Concerned about the hand-engineered nature of auxiliary constraint costs introduced in the system. The system is only evaluated on table-top manipulation setting. Even though the system does well on a range of such tasks, the costs rely heavily on the scoping of the problem space and I worry how this will generalize to non-tabletop manipulation or even harder mobile manipulation settings.
-  There are a number of confounding components in the system with no clear understanding of why this system works as well as it does. While the system will get better with more powerful VLMs and LLMs, I don't see a clear way of analyzing this problem more deeply to inform community of most promising gaps/bits of improvements. One clear systemic insight is the high failure rate of the 3D point tracker, but the reviewer still finds analysis lacking in depth.
- Missing comparisons with comparable non-modular baselines like a backbone-based end-to-end model solution. This will also open up the space to perform deeper analysis of what works/does not work/works better between MLLMs and more targeted backbone instantiation of this problem. Would appreciate more hardware/simulated experiments against baselines.
- Consider adding "Overall success" row to table in Figure 4. for a better understanding of performance
questions: questions: - What is the definition of "in-the-wild" in the context of task characteristics? Why is stowing a book a "in-the-wild" experiment?
- How will your system handle problems where contact is intentional and therefore the collision costs are variable? For example, compare against PerAct [1] with its explicit modeling of "collide" boolean which is off for intentional collisions in high-contact tasks.

[1] Shridhar, M., Manuelli, L. and Fox, D., 2023, March. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning (pp. 785-799). PMLR.
limitations: limitations: 3
review_summary: review_summary: I have concerns around complexity of the system and the unclear analysis/lack of tools for systemic analysis of gaps. That said, I find the scope of hardware results quite exciting and the system very impressive. I thus vote for a weak accept and look forward to the author's rebuttal.
title: title: Initial Review
paper_summary: paper_summary: This paper proposes an end-to-end pipeline to solve a manipulation problem given input RGBD images and a language description, with no additional inputs. It uses an LVM to generate keypoints for objects of interest. Input image with keypoints along with the language instruction are passed to a VLM to divide the task into stages and create cost functions for constraints in the form of simple Python functions. Finally, the constraints are passed to a motion optimizer to solve the motion planning problem. This framework can replan in case of failure, as well as generalize to novel objects and tasks. It is also shown to work with bimanual manipulation and potentially other multi-agent settings. The authors perform real-world experiments on a variety of settings such as pouring tea and folding clothes. They also provide some analysis of system error.
main_review: main_review: Quality: The proposed method is fairly intuitive and is mostly explained well, and the assumptions made are reasonable. There are a good number of demonstrations and visualizations of the method. Experiments address some important research questions, however the proposed model is only compared with a single baseline, and the number of experiment instances per task can be higher (perhaps in a simulated setting).
 
Clarity: The paper was written in an easily understandable manner. The figures provided are high quality and explanatory enough. However, it is not explicitly told in the paper how the framework determines the number of stages the task will have. This is better addressed in the explainer video (VLM decides the number of stages as well, alongside the constraints). I also did not understand how the system error breakdown was done. How do you quantify and compare errors from different modules (while considering their dependencies on each other)?

Originality: A fair number of previous works use VLMs and LVMs to generate keypoints or similar features to solve a manipulation problem (often, these features are either given to an optimizer as in this work, or the motion is also obtained from a learned model). Most of the proposed pipeline consists of off-the-shelf models. However, the method for generation of constraints as VLM-generated Python functions, although somewhat similar to VoxPoser, seems to be novel.

Significance: The significance of the paper is questionable because it’s claimed to provide a method that can take advantage of optimization solvers that can work with minimal human input. However, the prompt provided to the VLM contains overly detailed explanations of the tasks tested, which introduces a significant human bias and puts doubt on the actual capability of the VLM. The low amount of experiments also detract from the certainty of the results proposed.


Strengths:
- End-to-end framework: Given human instruction and RGBD images, the plan can be executed

- Shown to be suitable for online settings

- Robust to disturbances and failures, can replan from a previous stage

- Can generalize to (somewhat) new objects

- Shown to work with bimanual manipulation

Weaknesses:
- Overly detailed prompt that explains how the test tasks should be done in great detail, introducing a significant amount of human bias

- Most of the pipeline is a combination of readily available tools

- Only compared to a single baseline

- Said to be fully autonomous, however the non-annotated model achieves significantly worse results

- Not robust to a satisfactory level; as results in Table 2 show the non-annotated model only succeeds in about a fourth of the runs.

- In its current form, not feasible for complex manipulation problems (i.e., contact-rich), although this is discussed in the paper as well

- Heavy reliance on the performance of underlying VLM, LLM and point tracker, with the latter shown to be the largest contributor to system error. The existing model seems to use a single camera’s input, so this method could be extended to multi-view settings to reduce this error.

- More experiments could be done (possibly in a simulated environment) for a better analysis
questions: questions: - How would the proposed method perform with (much) less-informed instructions provided as an initial prompt?

- Current formulation uses ReKep costs in certain constraints only (i.e., requires them all to be <= 0). Is it possible to use these costs in other ways?

- Since LVM takes only the image input, the keypoints it generates can be unrelated to the task at hand, or even worse it may not generate some keypoints crucial to the task. If you have the language instructions anyway, why not use something like a VLM rather than a pure vision-based model?

- In your system error analysis, from what I understand you manually inspect failed experiments and select the first failing component as the cause of failure. However, this should naturally introduce a bias to yield higher error rates for earlier stages of the pipeline. Please clarify.
limitations: limitations: 2
review_summary: review_summary: This paper proposes a method for language-instructed motion planning that is flexible and suitable for real-world applications. The explanation was mostly clear and intuitive, and visuals provided were sufficient and in high quality. For experiments, the authors provide a wide variety of hardware experiments that address several important research questions. However, the method is somewhat lacking in novelty; most of the pipeline consists of readily available tools with little modification or fine-tuning. The results of the experiments are not conclusive due to the low amount of experiment runs and comparison with only a single baseline which fails on almost every experiment task. Most important criticism is that the prompt provided to the VLM contains detailed explanations of many of the tasks used in the experiments, which raises questions about the actual capability of the VLM.