Language-guided Manipulator Motion Planning with Bounded Task Space

Authors: thies  oelerich
christian  hartl-nesic
andreas  kugi
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 14:09:21

Abstract: Language-based robot control is a powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to reason about the environment. However, the generated robot motions by these controllers often lack safety and performance, resulting in jerky movements. In this work, a novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning-specific training. An LLM is combined with a vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization-based planner, the BoundMPC framework, is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness of the approach is shown on various everyday manipulation tasks in simulation and experiment, shown in the video at www.acin.tuwien.ac.at/42d2.

Main paper: 
 introduction: 
 The rise of large language models (LLMs) is not just a technological advancement, but a significant step towards enabling robots to understand and reason about the world. This development has practical implications, as it allows for the simplified use of robots in everyday tasks and unseen environments. Non-expert users can now control robots using natural language, which is a major stride in human-robot interactions. This technology is particularly important in enabling robots to cooperate and help humans, such as in household tasks or robot-aided medical care. The generalization capabilities of LLMs simplify the process of conveying instructions from a user to a robot, which should act according to these instructions. However, transforming this language understanding into safe, performant robotic motions remains an open topic with many possible solution approaches. This research tackles the crucial issue of translating human-generated instructions into safe and executable robotic actions. Previous studies have proposed various methods, including lexical analysis [2] , step-by-step language instructions [3, 4] , translation of language instructions into program code [5, 6, 7] , and end-to-end learning for low-level robot actions [8] . While these approaches can accomplish the given tasks, they often fall short in terms of robot motion performance, leading to jerky movements [5, 8, 9] or reliance on predefined motion primitives [10, 4] , which restricts robot motions to a predefined set of skills. To overcome these limitations, [11, 12] suggests learning discrete Cartesian waypoints and utilizing existing robot motion planners to traverse them. However, this necessitates training a neural network for each task, which is computationally intensive and lacks generalizability across different tasks. Moreover, the waypoint sequences require training data, which is challenging to provide. This research addresses these issues by proposing a modular motion planner that seamlessly integrates with large language models, resulting in smooth and performant robot trajectories that generalize well across tasks. The controllers of robot manipulators operate at high-frequency (≥ 1 kHz) to control the input torques of the robot joints. In contrast, human-generated instructions are high-level plans provided at a significantly lower frequency.Bridging this gap between high-level plans and low-level control requires considering various cognitive, visual, spatial, and robotic requirements. Our approach tackles this challenge with a modular architecture, see Fig. 1 . This architecture includes an LLM to interpret human instructions, a visual model to localize and reason about objects in the scene, a 3D information model to exploit spatial data, a path planner to create Cartesian reference paths with collision-free bounds, and the model predictive path-following concept BoundMPC [1] to execute a safe and efficient robot trajectory along the reference path.The generalization capabilities of the LLM can translate complex human-generated instructions into program code to communicate robot behavior to the underlying modules. However, it cannot directly deal with collision avoidance or the robot's kinematic and dynamic constraints. The LLM instructs a path planner to find feasible collision-free paths toward different goals, like objects or locations in the scene, which are specified from a zero-shot visual model [13] . The 3D information of the scene is used to construct a graph of convex collision-free sets, which is the basis for planning a Cartesian reference path. Based on these sets, the bounds around the reference paths are defined to avoid collisions with the environment. The predictive path-following controller BoundMPC [1] guides the robot towards the goal. BoundMPC computes and executes locally optimal trajectories in real-time to follow the reference path within the given bounds. Splitting path and trajectory planning has the advantage that no global trajectory optimization is needed, allowing shorter planning times. The convex-set-based path planner allows simple interaction with the generalization capabilities of the LLM, and the optimization-based trajectory planner BoundMPC creates fast and smooth trajectories.The main contribution of this work is a novel modular motion planning pipeline. This pipeline comprises a path planner based on convex sets, which is a unique approach, as well as the interfaces of this planner with an overlaying learning-based task planner, i.e., a pre-trained LLM and a pre-trained vision model, and the underlying model predictive trajectory planner BoundMPC. This unique combination allows reasoning in Cartesian space by leveraging the strength of learning-based models deeply integrated with smooth and safe optimization-based motion planning in the joint space. The capability to create smooth, performant trajectories for unseen tasks is demonstrated in simulation and experiment by using a robot manipulator, showcasing the effectiveness of this approach. The main novelties of the proposed approach can be summarized as follows:1. The modular motion planning framework can plan smooth, safe, and performant motions based on natural language instructions. This framework does not rely on motion primitives but generates via-points and task-specific path constraints. 2. A convex-set-based Cartesian path planner with Cartesian bounding functions describes the obstacle-free region around the path. This builds a crucial component for interfacing the learning-based task planner with the optimization-based trajectory planner BoundMPC. 
related works: 
 The proposed work falls in task and motion planning (TAMP) [14, 15] . TAMP is concerned with breaking down a complex task into, e.g., motion instructions for a robot [16, 17, 18] . Specifically, this work deals with the manipulation among movable objects planning [19] . A recent and important subfield is language-based motion planning, where tasks are specified as natural language instructions and are translated to executable robot trajectories. Task understanding and ordering are assumed to happen within the language model. Extracting information from language has been studied extensively for general applications [20, 21, 22] and specifically for robots [23, 3, 24, 25, 2, 26] . These language-grounding tools can be applied to many problems, such as learning reward functions in reinforcement learning [27, 28] , instructing intelligent agents to obtain action policies [29] , or communicating domain knowledge [30] . This work focuses on grounding task instructions for robotic manipulation tasks [5, 9] . In order to successfully plan motions for a manipulator, the language instruction has to be grounded with visual information. The authors in [24] use a textual description of the scene within a language model. More advanced works [31, 25, 12, 32] use image embeddings in combination with language embeddings as inputs for a network. Training these networks for robotic tasks is computationally expensive and requires large datasets. Another approach combines an LLM for language understanding and a vision model to correlate the language features with the objects in the scene [5, 33, 34] . Since the language and vision models are pre-trained on large general datasets, no additional training is needed, which makes the motion planner inherit the generalization capabilities and zero-shot task understanding of the pre-trained networks without any computational costs for training. Furthermore, the language model programs (LMPs) [6] demonstrate strong coding capabilities, directly translating the natural language input into Python code. However, these works lead to jerky and slow robot motions and do not consider the robot kinematics for motion planning. These undesired motions occur in end-to-end learning frameworks [8] but also for networks outputting discrete actions [5, 9] due to insufficient lower-level motion generation. In contrast, our approach uses a path planner that systematically integrates with the language model and outputs task-constrained Cartesian bounded reference paths executed with the trajectory planner BoundMPC [1] for fast, safe, and smooth motions with bounded path deviations, accounting for the robot kinematics. Safety is an important feature in motion planning and is often addressed by relying on underlying planners that guarantee safety. The work on drone swarms in [35] uses an underlying safety filter, and the manipulation planning framework in [11] uses existing joint-space motion planners with obstacle avoidance. The trajectory planner BoundMPC [1] fulfills a similar role in this work. In addition to safety, it also allows the incorporation of task-specific Cartesian constraints, e.g., keeping a cup upright or moving along a straight line.Manipulator trajectory planning must consider the robot's kinematics, dynamics, and interaction with the environment. Integrating these capabilities directly into LLM-based reasoning is infeasible and does not generalize across tasks. Due to this complexity, different methods for trajectory planning have emerged, which are based on sampling [36, 37] , optimization [38, 1] , learning [39, 40] , or a combination of multiple methods [11] . Obstacles avoidance is commonly realized by parametrizing the space occupied by obstacles and planning in the remaining space [38, 37, 5] . On the contrary, planning based on convex sets finds convex obstacles-free regions and plans a trajectory within these regions [41, 42] . The important difference between these two approaches is that the convex regions do not necessarily cover the whole obstacle-free space but still allow task completion by finding a path through the environment. The work [43] navigates through sets of convex regions using a graph, which is insufficient for motion planning. The works [42, 41, 43] directly optimize a trajectory over the path of sets. This is computationally expensive for robotic Cartesian task-space trajectories. Therefore, we propose a novel Cartesian path planner, which computes a Cartesian reference path inside connected convex regions and determines bounds of obstacle-free space for a robot manipulator to move inside. Planning a path through a graph of convex sets can be done using piecewise linear paths where each linear segment is contained in a convex set. The convexity of the set guarantees that the whole path is within the convex sets. Due to their discontinuities, existing path-following controllers, such as [44, 45, 46] , struggle to follow such piecewise linear reference paths. In this work, a novel path planner based on Cartesian convex sets is proposed and combined with the path-following controller [1] . An alternative approach is planning a global offline trajectory [36, 37, 47, 48, 49, 50, 51] , which is computationally expensive. Furthermore, our path planner provides an intuitive interface with natural language instructions and enables an easy specification of task-specific constraints, while the online trajectory planner handles the robot's kinematics and motion speed. This is computationally more efficient and allows for online adaptions. 
safe language-based trajectory generation: 
 This paper solves the problem of generating a robot manipulator trajectory τ in joint space using a language instruction L given by a user, e.g., "Put the cup on the table". Furthermore, the robot must perform this motion safely such that the joint-space trajectory is contained within the set of safe trajectories τ ∈ τ safe . This can be defined asEQUATIONwhere the indicator function I success indicates a successful task execution. This problem formulation is general and applicable to many different tasks. The generality makes solving the problem difficult as many solutions may exist. Directly searching over τ safe is infeasible as there is a significantly large number of feasible trajectories in general.Therefore, the problem (1) is subdivided into subproblems. Thus, similar to [5] , it is assumed that the instruction L consists of n subtasks l i such that L = {l 1 , l 2 , . . . , l n }, e.g., the instruction "Put the cup on the table" may be subdivided into the subtasks "Detect the cup", "Move to the cup", "Grasp the cup", "Move to the table", and "Release the cup". Each of the subtasks l i is successfully performed by a corresponding trajectory τ i . In this work, each trajectory τ i is represented by a set of via-points W j = {w via,j , j = 1, . . . , m}, where each via-point consists of a desired end-effector pose p via,j , containing position and orientation, and a gripper state g via,j ∈ [0, 1], corresponding to closed (g via,j = 1) or open (g via,j = 0). Additionally, the collision-free space along the path between the via-points is specified using bounds to ensure collision-free trajectories τ i ∈ τ safe .This section presents our developed motion planning framework for solving problem (1) . An overview of the framework is shown in Fig. 1 . It consists of four distinct components, namely:1. An LLM for breaking down the instructions L into actions A defined in Python code, 2. a vision model to align the actions A of the LLM with the RGB image of the scene and retrieve spatial locations from a 3D point cloud, 3. a novel path planner based on convex sets using the IRIS algorithm [52] to plan a piecewise linear bounded and collision-free reference path π, 4. and the model predictive controller BoundMPC [1] to traverse the reference path π.The LLM is instructed to output Python code to perform the necessary actions A, see Fig. 1 . This work uses a combination of language model programs (LMPs) [6] with Chat-GPT4 [53] . To this end, the generated Python code uses the 2D vision model and the LLM to retrieve specific objects and locations from the scene. The corresponding 3D coordinates are obtained by querying the 3D point cloud. In order to execute action A on the robot, the motion planner is employed to move the robot's end-effector to the planned location. In turn, the path planner first determines an obstaclefree path and bounds using convex sets of the scene. Finally, the model predictive path-following controller BoundMPC executes the robot motion. The first three steps are performed offline, and the final trajectory planning using BoundMPC is performed online at 10 Hz. The planned trajectories are sent to a computed-torque controller that computes the necessary joint torques at 8 kHz.This idea is based on the work in [6] also used in [5] for robot motion planning. The main difference to this work is the trajectory planning. The work in [5] uses smoothed 3D cost maps to plan the end-effector trajectory, which is not guaranteed to be executable or collision-free. Furthermore, the Cartesian trajectories are converted to joint-space trajectories by sending the desired Cartesian end-effector pose of the current time step to a controller. This formulation leads to jerky and slow robot motions. The following further explains the individual components of the proposed modular framework.S 1 S 2 S 3 c 1,2 c 2,3 (a) Graph G p s,1 p s,2 p s,3 p 0 p via,1 p via,2 p f S1 S2 S3 O π Ψi and Ψ i (b) 2D environmentPerforming a task requires the robot to understand the surrounding scene visually. The vision model comprises the zero-shot open-vocabulary object detection network OWL-ViT [13] . The model is queried to detect objects of a specific kind, e.g., tables, shelves, and cups, and returns a list of objects with the corresponding areas in the RGB image that make up this object. After refining the 2D areas using the semantic segmentation network SegmentAnything [54] , they are projected into the 3D point cloud to locate the objects in the scene spatially.The LLM creates Python code in this work, invoking the other components to complete the task. The code queries the vision model to get scene information. For example, when the user asks the robot to "move the cup to the table", the code will query the vision model to locate the cup and the table.If several cups are detected, the Python code will call the LLM to clarify the relevant object instance for the task. The combination of the LLM with the vision model is inspired by VoxPoser [5] .In order to successfully perform a task, the LLM generates robot actions that move the end-effector from the current pose p 0 = p via,0 to a desired pose p f = p via,L . A novel path planner based on convex sets is employed to find a collision-free path in Cartesian space. The core idea of the planner is to compute collision-free convex sets in the Cartesian space [52] and connect them to a graph based on their overlaps. A graph planner [43] then finds a path through the graph, which is converted into a linear reference path with Cartesian bounds, to be executed by BoundMPC [1] . The novelty of this planner is the computation of the bounded reference path π to traverse the graph.An alternative approach would be to compute a collision-free reference path using existing path planning methods, e.g., [36, 48] . However, this needs to consider the robot's kinematics which leads to high computational complexity. Our approach defines a bounded Cartesian space along the path in the offline step and utilizes online trajectory planning to account for the robot's kinematics. This is computationally less expensive, and it is possible to incorporate task-specific constraints specified by natural language during the reference path computation, see Appendix E for more details.In the following, the functionality of the planner is explained in detail for an example 2D environment, see Fig. 2 . The 2D environment in domain D ⊂ R 2 contains six convex grey obstacles that constitute the obstacle space O. The path planner must find an obstacle-free path π from the starting point p 0 to the end point p f . The IRIS algorithm [52] is employed to compute collision-free convex sets S k ∈ D\O around the sample points P = {p s,k : k = 1, . . . , K}. Each set is represented as a linear inequality S k = {p ∈ D\O : A k p ≤ b k } with the matrix A k and the vector b k , describing the halfspaces that define the convex set S k . A graph G of convex sets is build as detailed in Appendix A. The optimal path through the graph G is given as the sequence of sets S path,l with l = 1, . . . , L. As the sets describe safe regions for the robot to move in, constructing a reference path is desirable to systematically consider the safe space when planning toward the endpoint p f . Linear reference paths with Cartesian bounds achieve this goal, and their construction based on the sets S path,l is detailed in Appendix B. The resulting piecewise linear path is shown in Fig. 2b . Linear paths have the advantage of being contained within a convex set as long as their start and end points are. This is not the case for more complex formulations such as splines.The robot must remain within the convex sets S path,l to ensure collision freedom. This is ensured using the bounding functions Ψ i and Ψ i around the path segment π l within BoundMPC. In order to interact with the path-following controller in Section 3.1.3, it is desirable to obtain smooth bounding functions, detailed in Appendix C, because BoundMPC is an optimization-based framework.The above approach is used for the position reference path of the end-effector. Finding convex sets for the orientation is more challenging due to the more complex space of orientations in three dimensions. The current planner assumes knowledge of the orientation at the starting point p 0 and end point p f , which can be obtained from grasp pose detection, e.g., [55, 56] . Grasp pose detection is an important feature, especially for complex object geometries. Adding a grasp pose detection model to our framework is straightforward and will be part of future work. At each via-point p via,l , the orientation is a linear interpolation between the start and the end orientation.Remark. The physical extent of the end-effector is taken into account by increasing the size of the obstacles by the size of the end-effector. This works best with a symmetric end-effector.Following the path π from Section 3.1.2 requires a path-following controller that systematically considers the Cartesian bounds Ψ i and Ψ i . In this work, the receeding-horizon planning BoundMPC framework [1] is utilized. It allows the end-effector of a robot manipulator to follow the Cartesian path π with bounds. As proposed in [1] , a piecewise linear reference path formulation as constructed in Section 3.1.2 provides a simplified yet performant choice for the reference path π. The bounds guide the robot along the path π while ensuring collision freedom. Furthermore, the Cartesian path π may be hard to follow exactly since it does not consider the kinematics of the robot. The bounds enable the robot to deviate from the path and find optimal trajectories q(t) within the given collisionfree Cartesian space, defined by the bounding functions Ψ i and Ψ i . The advantages over state-ofthe-art MPC formulations are discussed in Appendices E.1 and E.2. Currently, the bounds ensure collision-freedom for the end-effector. However, it is straightforward to add collision-avoidance terms to enforce safety for the entire kinematic chain of the robot, as shown in Appendix F. 
experimental results: 
 The trajectory planning framework developed in Section 3 is evaluated in simulation and experiment in this section. As a simple example, a cup is picked up and placed on the opposite side of a book, placed upright on a table. Afterward, tasks are executed to show the proposed method's versatility, generalization, and robustness. For the implementation of the path planner based on convex sets in Section 3.1.2, the Drake software framework [57] is employed. Appendix H shows the LLM prompts used in this section, which rely on the functions listed in Appendix G. More simulation results comparing the individual components of our proposed framework with state-of-the-art methods are shown in Appendix E, and collision-avoidance examples are given in Appendix F.The trajectory to pick up the cup for the prompt "Place the cup left to the book" is visualized in Fig. 3a . The path planner from Section 3.1.2 plans a piecewise linear reference path π from the starting point p 0 to the end point p f with one via-point p via,1 . The BoundMPC framework follows π to reach p f without colliding with the book. This is compared to VoxPoser [5] , where the desired end-effector trajectory is planned based on 3D cost maps. The book and the table have a negative cost in repelling the end-effector, while the cup has a positive cost in attracting the end-effector. The planned trajectory gets stuck on the right side of the book and then jumps to the endpoint p f . This results in a collision with the book. It is also not possible to tune the cost functions appropriately to achieve a collision-free trajectory using VoxPoser, as it relies on RRTConnect to create a joint trajectory from the desired Cartesian trajectory shown in Fig. 3 . This procedure does not guarantee that the desired Cartesian trajectory is followed and invalidates any collision-free guarantees in the reference trajectory. BoundMPC ensures that the end-effector remains safe without any cost tuning.The performance of our proposed methods is further evaluated in simulation in the environment shown in Fig. 1 and Fig. 5 with the following tasks: 1 shows that the proposed method outperforms VoxPoser across all tasks. The Simple Move task is challenging for VoxPoser due to the collision avoidance where a situation similar to picking up the cup in Fig. 3 is encountered. The proposed method plans within collision-free convex sets to avoid any collisions. In the Swapping task, the LLM used in VoxPoser returns invalid instructions, which do not solve the task. This is caused by the independence of the subtask execution in VoxPoser, i.e., each subtask l i of the instruction L is assumed to be independent of all other subtasks. This assumption is invalid for the swapping scenario, as the first object must be removed, and the second object has to be moved to the previous position of the first object, which is not considered in this subtask. The Reposition task repositions an object on the table. Both methods perform this task successfully, but the proposed framework does this more efficiently regarding the path length. In the Arrangement task, the block must be grasped from different heights on the shelf and arranged on the table. VoxPoser creates a Cartesian path that the robot cannot follow due to its joint limits, which is visualized in Fig. 3b . Using our method, the bounded deviations from the path allows BoundMPC to utilize the orientation around the y-axis and find a suitable trajectory while avoiding the joint limits. This emphasizes the advantage of considering the robot's kinematic limits during planning and execution. The computed trajectory from VoxPoser leads to a collision with the shelf and requires a constant orientation, which the robot cannot achieve. A similar situation occurs in the Tea Cup task where a tea cup must be moved to a new position without spilling the tea. VoxPoser outputs a non-executable Cartesian trajectory. The LLM instructs our planner to keep the tea cup upright. BoundMPC then exploits the unconstrained rotation direction around the z-axis and the path deviations regarding the position to find a suitable trajectory to accomplish the task. Lastly, the Wiping task is solved by both frameworks. In this task, our method requires fewer stops compared to VoxPoser. The task requires picking up the sponge and moving it to one side of the table, from which the wiping motion starts. VoxPoser stops during the pick-up of the sponge, which is necessary, but also at the beginning and end of the wiping motion, which is not required. With our method, the robot moves smoothly without stopping at the via-points, leading to faster task execution. Furthermore, our method specifies that the wiping segment is constrained in position and orientation to follow the line and stay orthogonal to the table top. The LLM effectively instructs the planner to consider these constraints. Especially in the Tea Cup and Wiping tasks, the constraints specified by the LLM show a deep integration of the learning-based model and the motion planner. The LLM specifies the task-relevant constraints, and the planner finds executable, smooth, and collision-free trajectories that solve the task. The proposed framework performs well in the tasks and outperforms VoxPoser regarding task success, collision avoidance, average path length, and the number of intermediate stops.We also demonstrate the performance of the developed framework in several experiments in the video at www.acin.tuwien.ac.at/42d2. The video showcases the smooth trajectories and the planner's generalizability while staying safe. More information is given in Appendix D. 
conclusion, limitations and future work: 
 This paper presents a modular language-driven zero-shot motion planning framework to create safe and smooth manipulator trajectories. A novel Cartesian path planner is developed to interface the learning-models with the trajectory generator. This allows the learning-based components to reason in Cartesian space while smooth and safe trajectories that respect the robot's kinematic limits are generated in the joint space. The strength of the proposed framework is validated in experiments and simulations. However, some limitations exists. Prompt engineering is needed to query the LLM successfully, which requires the user to invest upfront time. Furthermore, the modularity of the approach requires suitable interfaces between the modules, which can be improved for more diverse scenarios. This is especially important for the LLM and the vision module, where recent work on vision language models can provide a helpful extension. The safety of the current approach is limited by the discretization within BoundMPC [1] . In the literature, continuous-time methods guarantee safety for the whole trajectory, e.g., based on reachability [58, 59, 60] . However, in practice, we never encountered collisions due to the discretization. 
 
 Appendix: The convex sets needed to build a graph G are computed using the IRIS algorithm [52] . The sample points P = {p s,k : k = 1, . . . , K} are hyperparameters of this algorithm. In this work, the sample points P include the starting and end points p 0 and p f , and six additional known collision-free points.Remark. Providing a small set of known obstacle-free points is often easily possible. If this is not the case, a rejection sampling approach can be adopted. More complex strategies based on the knowledge of p 0 and p f can be developed but are not part of this work. The authors of [41] propose to adapt methods developed for probabilistic-random-tree-methods to improve the sampling.Once collision-free convex sets are found around all sample points P, the graph G is constructed in which each convex set S k is represented by a node, similar to [41] . For each pair of convex sets S a and S b found by the IRIS algorithm, an edge is added in graph G between the corresponding nodes, if their intersection is not empty, i.e., S a,b = S a ∩ S b ̸ = ∅. The cost associated with this edge is chosen asEQUATIONwhich is the projection of the end point p f onto the intersection set S a,b . Note that the graph is undirected as c a,b = c b,a , and each edge allows passing in both directions. The constructed graph G for the 2D environment is shown in Fig. 2a . The shortest path through the convex sets is then calculated using the optimizer proposed in [43] . The optimal path through the graph G according to the cost ( 2) is given as the sequence of sets S path,l with l = 1, . . . , L.For constructing the piecewise linear reference path π, the sequence of sets S path,l is used. Each set intersection S inter,l = S path,l ∩ S path,l+1 with l = 1, . . . , L -1 between two consecutive sets S path,l and S path,l+1 in the solution path requires one via-point p via,l . The L -1 intersection sets S inter,l are used to compute L -1 via-points by solving the optimization problemEQUATIONwith the weights α ≥ 0 and β ≥ 0. Together with the starting point p 0 = p via,0 and the end point p f = p via,L , they make up the set of via-points. In (3), the matrix A inter,l and the vector b inter,l define the intersection set S inter,l , and the term ∥max(A inter,l p-b inter,l )+β∥ 2 2 , with the maximum taken over the vector entries, incentivises the via-point p via,l to be β away from any border of the intersection set S inter,l . This moves p via,l from the border of the set into the set and leads to more symmetric bounds later. The piecewise linear reference path π is the linear connection between the starting point p 0 = p via,0 , the via-points p via,l l = 1, . . . , L -1, and the end point p f = p via,L , as shown in Fig. 2 . This formulation ensures that the linear segment between p via,l-1 and p via,l is completely contained in the convex set S path,l , from which the bounding functions are computed using geometric considerations.In the following, the computation of the bounding functions in the general case of three dimensions is described for a given linear path segment π l , which connects the via-points p via,l and p via,l+1 and is contained in the corresponding convex setEQUATION)A visual explanation of the procedure is shown in Fig. 4 . This procedure also applies analogously to the special case of two dimensions and the example in Fig. vectors are obtained using the Gram-Schmidt procedure explained in [1] , such that an orthonormal frame is created, i.e., b 1 ⊥d, b 2 ⊥d, and b 1 ⊥b 2 . Furthermore, the mid point of the considered linear segment π l is p mid . Next, S path,l in ( 4) is transformed into the basis vector coordinates at the point p mid by using the transformationEQUATIONA rectangle that is aligned with the basis vectors b 1 and b 2 in the transformed set ( 5) is defined by the four corner pointsz T rect,1 = [x 1,1 , x 2,1 ], z T rect,2 = [-x 1,2 , x 2,1 ], z T rect,3 = [x 1,1 , -x 2,2 ], and z T rect,4 = [-x 1,2 , -x 2,2 ] with the parameter vector x T = [x 1,1 , x 1,2 , x 2,1 , x 2,2]. The bounds are computed by finding the largest-volume rectangle that is contained in the set S trans,l using the optimizationx opt = arg maxEQUATIONwhere the constraints ensure that the rectangle is contained in the transformed set (5) and that the path π is contained in the bounds at p mid with x ≥ 0. This rectangle is generally a conservative approximation of the transformed set (5) . The rectangle parameters are used as the maximum bounding function values in the linear segment π l . For each line segment, four bounding functions exist, i.e., the upper bounding functions Ψ i and the lower bounding functions Ψ i for both basis directions b 1 and b 2 . The respective maximum and minimum values are chosen as max(Ψ i ) = x i,1 and min(Ψ i ) = -x i,2 , shown as dotted lines in Fig. 2b . The value of the bounding functions at the via-points p via,l is zero, which the path-following controller requires. More information on the bounding function formulation is given in [1] . When choosing the bounding function as piecewise linear functions between zero at the via-points and max(Ψ i ) and min(Ψ i ) at p mid , it is ensured that they remain within the set S path,l because it is convex. However, the path-following controller benefits from smooth bounds, so fourth-order polynomials are used in this work instead. This can result in some parts of the bounds lying outside the set S path,l . However, in practice, this does not cause any issues due to the conservative approximation of the collision-free space of the IRIS algorithm and the conservative approximation in (6) . The bounding functions Ψ i and Ψ i , in combination with the reference path π, are used for the trajectory generation in Section 3.1.3.The scene of the experiments is shown in Fig. 5 . It consists of two shelves and a table in front of the robot. An Azure Kinect RGB-D camera is mounted behind the robot to provide the required data for the vision model, namely the RGB picture and the depth information used to obtain the point cloud. The user provides a prompt at the beginning of an episode, and the LLM creates The convex set planner from Section 3.1.2 requires a convex obstacles description of the scene. For arbitrary scenes, a voxel grid based on the point cloud can be used in the planner. This may lead to many obstacles and increases the computation time of the planner, but it allows the robot to act safely in unknown environments. To speed up the computation, a known convex decomposition of the scene is employed in this work. This results in an environment with 13 obstacles, leading to computation times of about 0.15 s for the path planner.Common failures of the vision model are falsely detecting an object that is not present or not detecting an object that is present. Further failure cases of the LLM and the vision model are discussed in [5] .This section compares the proposed motion planner, consisting of the convex-set-based path planner from Section 3.1.2 and the trajectory planner BoundMPC [1] to existing approaches. The advantages are summarized in Appendix E.4. Additionally, the simulation comparisons are shown in the video at www.acin.tuwien.ac.at/42d3.The current work uses an offline path planner in combination with the BoundMPC framework to compute suitable joint space trajectories. Other formulations exist to plan such trajectories. This section compares our proposed framework to an MPC trajectory tracking formulation [61, 62] . A reference trajectory is created for the robot, and the goal is to follow this trajectory using a receding horizon MPC while avoiding obstacles. The obstacles are commonly added as potential functions to the MPC objective [63] . A reference trajectory consists of a reference path with a time parametrization. A Cartesian reference trajectory determines the end-effector's desired position and orientation reference trajectory. This formulation is known as CT-MPC. The reference trajectory is generated based on a sequence of via-points according to the procedure described in [64] and is not guaranteed to be collision-free.For a detailed simulation comparison, a green block is positioned in the lower shelf part inside the left shelf of the environment shown in Fig. 5 . The goal is given by the prompt "Move the green block onto the left shelf.". The comparison of the created trajectories is shown in Fig. 6 . For the simple CT-MPC formulation, only the desired grasping pose and drop off pose are provided as viapoints. This does not lead to a successful task execution as the robot gets stuck on the lower shelf since it tries to move up straight but is blocked by obstacle avoidance. Thus, a more advanced reference trajectory is needed. For the CT-MPC formulation with more via-points, the via-points of the convex-set-based path planner from Section 3.1.2 are utilized to create a reference trajectory. This leads to a successful task execution with a trajectory similar to our proposed method, showing that a global reference path/trajectory is necessary for the task's success.The CT-MPC formulation needs a reference trajectory but does not require bounds around this trajectory as obstacle avoidance is performed using potential functions in the MPC objective function.The objective of the CT-MPC is to minimize the tracking error between the end-effector pose and the reference pose. This leads to problems when the reference trajectory is not followable due to the robot's kinematics. An example is shown in Fig. 7 . Here, the robot picks up a green block and has to put it on the shelf behind itself. Based on the via-points, the created reference trajectory is hard to follow because it passes very close to the robot's base's vertical axis. Hence, a large tracking error is needed to perform the task, as shown in Fig. 8b . However, the objective of CT-MPC is to minimize this error, and the robot's kinematics hinder a minimization, leading to non-smooth motions. The norm of the end-effector acceleration a c reflects this behavior. Furthermore, CT-MPC does not perform the task successfully because the large reference tracking error e ref leads to the robot getting stuck in a local minimum above the desired shelf space, which leads to a non-zero tracking error e ref at the end of the trajectory. This is avoided by using a bounded reference path π as this method determines which path deviations are suitable at which point in space. Hence, the proposed approach finds a smoother and successful end-effector trajectory for this task. The two comparisons above showcase pick-and-place scenarios. These can also be solved using a joint-space reference trajectory instead of a Cartesian one. We call this formulation JT-MPC. It works well for pick-and-place tasks but struggles with tasks where Cartesian task-specific constraints are given. The reference trajectory uses the inverse kinematics solutions of the via-points that result from the convex-set-based planner outputs is using. The example of the wiping task from Section 4 is considered in Fig. 9 , where the wiping motion along the table surface has to be a straight line. The JT-MPC framework cannot take the Cartesian reference path π into account, leading to an undesired arc-shaped trajectory. The proposed framework considers the task-specific constraints formulated by the LLM and successfully finishes the task. This shows the advantages of a Cartesian reference path to incorporate task-specific constraints.Our approach splits the motion planning problem into offline Cartesian reference path planning and online trajectory planning. However, it is also possible to directly optimize a joint-space trajectory from the starting pose p 0 to the final pose p f . In this section, the turn-around example from Appendix E.1 is considered with different global motion planners. It consists of two planned trajectories: The first trajectories grabs the green block and the second trajectory transfers it to the other shelf. Our approach is compared to VPSTO [47] , RRT* [65] , and STOMP [51] . A comparison of the combined planning times for both trajectories is shown in Table 2 . Our approach computes a reference path π with collision-free bounding functions considerably faster than computing a collision-free joint-space trajectory. Note that all methods manage to solve the task successfully. Furthermore, a change in the environment will require a replanning, which is costly for the global planners but not for our proposed method. This shows the advantages of using our proposed methods instead of global trajectory planners.The proposed approach of motion planning as a combination of convex-set-based Cartesian path planning with trajectory planning using BoundMPC has the following advantages: • Reference A reference path/trajectory is needed to perform the task successfully and provide a global guide for the robot's motion. The example in Fig. 6 elaborates on this point.• Cartesian reference path: A Cartesian reference path/trajectory has the advantage that task-specific constraints along the path are easy to incorporate compared to joint-space reference paths/trajectories, see Fig. 9 .• Bounds around the reference path: Existing sampling-based methods are a powerful tool to plan reference paths, e.g., RRT [66, 36] or PRM [67, 36] . However, they currently have no notion of path deviation bounding functions. The examples in Fig. 6 and 8 show the advantages of using such bounding functions. In these examples, deviating from the reference path is necessary due to the robot's kinematics, which makes minimizing the tracking error unsuitable, leading to non-smooth trajectories. The bounding functions guide the robot towards the goal while keeping it in the collision-free part of the task space. Furthermore, a Cartesian reference path is required to handle task-specific constraints, such as following a straight line exactly or keeping a cup upright. Thus, bounding functions around the reference path are a powerful tool for considering the robot's kinematics and encoding task-specific constraints.• Splitting up path and trajectory planning: Our approach first computes a bounded Cartesian reference path offline and then plans a joint-space trajectory online. Another approach is to directly compute a joint-space trajectory offline using global planners, as shown in Appendix E.3. The advantages of our approach are the faster offline planning time and faster adaption in case of environmental changes.The bounded reference path computed in Section 3.1.2 considers collisions only for the robot's endeffector. In this section, this collision avoidance is extended to the entire kinematic chain of the robot using potential functions in BoundMPC's objective function. This approach is based on the formulation in [63] . Additionally, it is compared to the Cartesian tracking (CT-MPC) and joint-space tracking (JT-MPC) MPC formulation of Appendix E.The example in Fig. 7 , where the robot has to turn around and place an object, is considered again. Six additional obstacles are positioned close to the robot base as shown by the black spheres in Fig. 7 . This is a very demanding task as the space where the robot can turn around is highly constrained due to the obstacles. The resulting end-effector trajectories of our framework, CT-MPC, and JT-MPC are depicted in the x-z and x-y-planes in Fig. 10a and b , respectively. Our framework and JT-MPC successfully perform the task, but the CT-MPC gets stuck at about x = 0.05 while turning around due to the obstacles. This is because the robot must deviate from the Cartesian reference trajectory due to the kinematic constraints but has no guidance on which deviation directions are suitable. Our Table 3 : Comparison of our method with CT-MPC and JT-MPC for a turn-around task with collision avoidance of the robot's full kinematic chain. Nine experiments are compared, in which the six obstacles in Fig. 7 are shifted along the x-axis in their x-coordinate between -0.4 and 0. framework uses the BoundMPC formulation, which defines the Cartesian bounding functions such that the end-effector stays collision-free. It is the only formulation that can completely perform the task successfully. Even though JT-MPC reaches the goal pose p f , a collision of a robot link with one of the collision spheres is detected, as shown in Fig. 10c . In this example, a trade-off between minimizing the trajectory error and the collision avoidance leads to an unsafe trajectory of the robot. Our framework avoids such situations since the reference paths π are collision-free by design. A possible improvement to JT-MPC would be providing a collision-free joint-space trajectory for the JT-MPC. However, this is not trivial and would require a global trajectory planner, which is computationally expensive, as discussed in Appendix E.3.In order to evaluate the robustness of our proposed method, the obstacles were shifted along their x-coordinate in the range -0.4 and 0.4. The evaluation of the resulting nine experiments is shown in Table 3 . JT-MPC always reaches the desired final pose p f but often collides with the environment due to the collisions in the reference trajectory. The CT-MPC formulation never collides with the environment but may get stuck at obstacles or joint limits. Our approach is the most successful framework, which only sometimes gets stuck at joint limits when the reference path bounds do not allow enough freedom to optimize a suitable joint trajectory. This comparison shows the robustness of our approach compared to the state-of-the-art.The LLM generates Python code that can use the following functions:• move to(pose: array, constraint: string, stop: bool) Instructs the motion planner to move to pose. The argument constraint can be used to specify constraints along the path, which are evaluated by an LLM query, and the argument stop specifies whether to stop at pose or continue fluently towards the next motion goal. • detect(object name: string) -> object Invokes the vision module to detect objects with the name object name.• close gripper() Closes the gripper.• open gripper() Opens the gripper.• move home() Move the robot's end-effector to the home position.• select object(object list: list, object name: string, instance: string) -> object Invoked in case multiple objects of the same type are detected with the detect() function. Queries the LLM to clarify which object is relevant to the task based on the instance string, e.g., finding the left shelf when instance is 'left' and multiple shelves are detected.• set speed(max speed: float) Sets the maximum path speed for BoundMPC to be max speed. • find position(object move: object, object relative: object, hint: string) -> object Invokes the LLM with an object placing query. Finds a position for object move relative to object relative specified by the hint. For example, placing a cup relative to a table with the hint 'onto right' finds a position on top of the table to the right where the cup can be placed. The found position is saved in the return object• set position(position: array) Sets the position based on the adaptions from the find position() function.• get end effector() -> object Returns the current end-effector state of the robot.• empty constraint() -> constraint Returns a constraints object where all constraints are unset.• full constraint() -> constraint Returns a constraints object where all constraints are set.• set constraint(constraint) Sets the constraint for the current segment.The LLM prompts used in this work are given in this section. The planner prompt is the highestlevel prompt invoked with the natural language user input. The position prompt is used by the import numpy a s np from a d a p t u t i l s import s e t p o s i t i o n # G i v e n d a t a : # -h i n t : h i n t f o r t h e d e s i r e d p o s i t i o n on g o a l o b j e c t / s t r i n g # Query : " cup " t o " r i g h t s h e l f " , h i n t : " o n t o " cup = o b j e c t s . movable s h e l f = o b j e c t s . g o a l ( max x , max y , max z ) = s h e l f . max bounds # Onto means t h e maximum z p l u s h a l f t h e s i z e o f t h e o b j e c t x = s h e l f . c e n t e r [ 0 ] y = s h e l f . c e n t e r [ 1 ] z = max z + cup . s i z e [ 2 ] / 2 s e t p o s i t i o n ( [ x , y , z ] ) # done # Query : " c u b e " t o " s h e l f " , h i n t : " i n t o " c u b e = o b j e c t s . movable s h e l f = o b j e c t s . g o a l # I n t o means we u s e t h e c e n t e r p o s i t i o n s e t p o s i t i o n ( s h e l f . c e n t e r ) # done # Query : " book " t o " t a b l e " , h i n t : " t o p " book = o b j e c t s . from c o n s t r a i n t l i b import ( e m p t y c o n s t r a i n t , f u l l c o n s t r a i n t , s e t c o n s t r a i n t ) # Query : "no c o n s t r a i n t " c o n s t r a i n t = e m p t y c o n s t r a i n t ( ) s e t c o n s t r a i n t ( c o n s t r a i n t ) # done # Query : " e x a c t " # E x a c t means t h a t e v e r y t h i n g i s c o n s t r a i n e d c o n s t r a i n t = f u l l c o n s t r a i n t ( ) s e t c o n s t r a i n t ( c o n s t r a i n t ) # done # Query : " u p r i g h t " c o n s t r a i n t = e m p t y c o n s t r a i n t ( ) # U p r i g h t means t h e r e s h be no r o t a t i o n a r o u n d t h e x o r y a x i s c o n s t r a i n t . r o t a t i o n x = T r u e c o n s t r a i n t . r o t a t i o n y = T r u e s e t c o n s t r a i n t ( c o n s t r a i n t ) # done # Query : " h o r i z o n t a l p l a n e " c o n s t r a i n t = e m p t y c o n s t r a i n t ( ) # The x-y-p l a n e i s t h e h o r i z o n t a l p l a n e c o n s t r a i n t . p o s i t i o n x = T r u e c o n s t r a i n t . p o s i t i o n y = T r u e s e t c o n s t r a i n t ( c o n s t r a i n t ) # done # Query : " z r o t a t i o n " c o n s t r a i n t = e m p t y c o n s t r a i n t ( ) c o n s t r a i n t . r o t a t i o n z = T r u e s e t c o n s t r a i n t ( c o n s t r a i n t ) # done 


Decision: This paper presents a planning framework that addresses task and motion planning (TAMP) by utilizing a Large Language Model (LLM) for discrete task planning and a graph of convex sets (GCS) for motion planning.

Before the rebuttal, the major concern among reviewers is the lack of novelty. The reviewers noted that while the components of the framework are existing work, the innovation may lie in their combination and significantly more simulation and hardware experiments in more complex scenarios are needed to justify the advancement enabled by this combination.

Other concerns include confusion regarding the terminologies and equations used, unclear figures, an emphasis on safety that only considers the end-effector, the lack of an ablation study with other existing planners, and an insufficient literature review on TAMP methods.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns. As a result, all reviewers now rate the paper as a Weak Accept. While some reviewers still have concerns regarding the novelty of the paper, they acknowledge the clear motivation behind the work and the hardware experiments that demonstrate the proposed framework's effectiveness.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper proposes a modular motion planning framework, consisting of an LLM to generate Python code for the motion task completion, a vision module, a path planner to compute smooth paths of the robot, and a MPC controller to execute the path. For the LLM, vision module and MPC controller, existing approaches are used. The main novely lies in the path planner. It computes collision-free convex sets and connect them via a graph. The path for the robot to follow is then computed based on the shortest path in the graph. The motion planning framework is evaluated using simulated and physical manipulation tasks.
main_review: main_review: The paper is generally easy to follow. The components of the motion planning framework that are not part of this work (LLM, vision module, MPC controller) are mostly described with sufficient detail to understand the whole framework.

The main issue I have with the paper is a poor motivation of the path planner, the only component of the framework that is novel. The idea of constructing a piecewise linear reference path from a graph that connects overlapping convex sets of collision-free states is interesting, but the paper doesn't explain the need for this idea. The goal is to generate smooth trajectories, but the smoothness is achieved by the MPC controller, not by the path planner. In fact, the path planner outputs piecewise linear reference paths, which are not smooth. That being said, it is not clear to me why not any existing off-the-shelf path planner is used, and what the actual advantages of the proposed planner are. Since the main contribution of the paper is the path planner, the paper should discuss this in detail. Ideally, the paper should also provide an ablation by swapping the proposed planner with an existing one. For instance, what happens if VoxPoser's motion planner is used in conjunction with the MPC controller? I have the feeling that this would work just as well as using the proposed planner.

Some other comments:

- The paper uses "path planner", "motion planner" and "trajectory planner" interchangeably, which is confusing. Please be consistent to avoid confusion.
- Section 3.2.1 should be in the related work section. Please mention the used LLM instead.
- Line 182: \mathcal{D} is not defined.
- Line 182: What is matrix A_k and vector b_k? Please explain.

Post rebuttal: The authors have addressed my previous concerns regarding the motivation of the planner and the smoothness of trajectories.
questions: questions: - I'm surprised that a piecewise linear reference path is computed, since the main motivation of the planning and control modules are to generate smooth trajectories. Why not use B-splines or other curves for the reference path? 
- Why does the framework have such a low success rate in the TeaCup problem?
limitations: limitations: 3
review_summary: review_summary: In summary, while the paper contains some interesting ideas, more discussion on the novelty of the path planner must be provided, including an ablation study with existing planners.
title: title: Language-guided Manipulator Motion Planning with Bounded Task Space
paper_summary: paper_summary: The paper appears to be doing language-guided motion planning by combining LLMs and MPC.
main_review: main_review: Strengths
* The authors have identified an interesting area of research, namely zero-shot reasoning using open-vocabulary
* The authors demonstrate their method on hardware
* The authors’ method is outperforming VoxPoser

Weaknesses
* The novelty of the paper is unclear. There are many papers that use LLMs to generate motion plans for robots. In fact, there are methods that have more stringent safety constraints.
* The figures in the main part of the paper are very unclear. Arguably the most important part of Figure 1 is the image of the simulated robot performing the task. The paper could benefit from showing an image of the robot actually solving a task where the time steps of the arm are transparent and overlaid on the image.
* The authors emphasize safety, but only enforce safety of the end-effector. This seems very strange because there are numerous real-time trajectory optimization methods for manipulators that would have allowed the authors to enforce safety for the entire robot.
* The discussion on the novelty of their MPC method is rather limited? 
* Why is the method called MPC? MPC usually refers to some kind of receding-horizon control. The authors description sounds like they are generating the plan offline.
* The comparisons in the paper are limited.
* The scenes in the paper are  rather simple.
questions: questions: * Why not show an example trajectory in the main overview figure?
* Can you motivate the use of GCS more?
* Can you discuss the novelty of your BoundMPC algorithm a bit more? It seems like you are using GCS, but only enforcing collision constraints for the end-effector. Is this because GCS isn’t actually real-time? Furthermore, I do not believe that the linear interpolation is novel. Isn’t this how TrajOpt works?
* Is it possible for you to do evaluate the performance of your method on more challenging scenes? How many obstacles can your method handle?
* How do you think your method would work for tasks requiring longer sequences?
* Could the authors explain the various pieces of their method a little better. They say BoundMPC runs at 10Hz and that their path planner takes 0.15s. Is GCS generating the path and BoundMPC tracking it? Is GCS running online or offline? BoundMPC is only generating torques at 10Hz? This sounds very slow for torque control.
* The grasp detection should be discussed more.
limitations: limitations: 3
review_summary: review_summary: I chose weak accept because I do not understand the novelty of their MPC method. But the combination of MPC with LLMs is still very compelling.
title: title: Review of Language-Guided Manipulation Motion Planning with Bounded Task Space
paper_summary: paper_summary: The presented planning framework tackles task and motion planning by using an LLM for discrete task planning and for motion planning first building a set of of convex sets representing empty space, connects these sets via a graph, and plans over this graph.
main_review: main_review: The presented planning method is validated in simulation and demonstrates modest improvements against a state of the art visual-language planning method. The modular framework provides an interesting perspective on using an LLM for discrete task planning while using an MPC-based controller for motion planning. While the method is interesting, there are several concerns.

Major Comments:
1. The presented framework is fundamentally a Task And Motion Planning (TAMP) method, yet there is no literature review on TAMP methods. In my opinion, this method does not use an LLM for motion planning, rather it uses the LLM for discrete task planning (move to, pick up, drop, etc), and so the lit review on language-based motion planning does not fit within this paper. Without this TAMP lit review, it is hard to evaluate the contributions of the paper. Furthermore, there is no discussion on other safety-based methods in the literature such as control barrier functions, reachability, etc.

2. The authors describe their method as “safe”, however safety is only enforced at discrete time-steps within the MPC pipeline. This does not guarantee the method will be safe while transitioning between discrete, optimal states. The authors may be interested in continuous-time safety methods such as [1] [2].

3. The novelty of the individual elements of the presented framework are not thoroughly discussed. It is unclear whether LLMs have been previously used for discrete task planning, however planning over convex sets using graph optimization (which the authors claim is a contribution) [3] and MPC-based obstacle avoidance have both been presented in the literature. While novelty may lie in the combination of these elements into a single planning framework, the experiments and results of this paper are not yet sufficiently rich enough to demonstrate this novelty.

4. The proposed method would be better validated with real-world hardware experiments.


[1] Jonathan Michaux, Adam Li, Qingyi Chen, Che Chen, Bohao Zhang, and Ram Vasudevan. "Safe Planning for Articulated Robots Using Reachability-based Obstacle Avoidance With Spheres." Robots: Science and Systems, 2024.

[2] Ian Mitchell. "Comparing forward and backward reachability as tools for safety analysis." In International Workshop on Hybrid Systems: Computation and Control, 2007.

[3] Jack Tobia, Pablo Parrilo, and Russ Tedrake. "Shortest paths in graphs of convex sets." SIAM Journal on Optimization, 2024.

[4] Russell Buchanan, Lorenz Wellhausen, Marko Bjelonic, Tirthankar Bandyopadhyay, Navinda Kottege, and Marco Hutter. "Perceptive whole‐body planning for multilegged robots in confined spaces." Journal of Field Robotics, 2021.
questions: questions: 1. A more thorough literature review on safety-based planning methods as well as on task and motion planning (TAMP).

2. More simulation results are required in order to evaluate the novelty of the modular framework, as it is challenging to determine the novelty of each of the individual elements of the proposed method.
limitations: limitations: 2
review_summary: review_summary: It is hard to determine the novelty of the proposed modular task and motion planning framework without significantly more simulation and hardware experiments in more complex scenarios.