Language-guided Manipulator Motion Planning with Bounded Task Space
Authors: Thies Oelerich
Christian Hartl-Nesic
Andreas Kugi
Venue: CoRL 2024 (2024)
Keywords: Vision Language Models
Manipulation Planning
Path-following MPC

Abstract: Language-based robot control is a powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to reason about the environment. However, the generated robot motions by these controllers often lack safety and performance, resulting in jerky movements. In this work, a novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning-specific training. An LLM is combined with a vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization-based planner, the BoundMPC framework, is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness of the approach is shown on various everyday manipulation tasks in simulation and experiment, shown in the video at www.acin.tuwien.ac.at/42d2.

Language-guided Manipulator Motion Planning with Bounded Task Space
Thies Oelerich', Christian Hartl-Nesic', Andreas Kugil.2 Automation and Control Institute (ACIN), TU Wien, Vienna, Austria AIT Austrian Institute of Technology GmbH, Vienna, Austria foelerich, hartl, nugt @acin. tuwien. awwien.ac.at ac.at
Abstract: Language-bast based robot control is powerful and versatile method to control a robot manipulator where large language models (LLMs) are used to rea- son about the environment. However, the generated robot motions by these con- trollers often lack safety and performance, resulting in jerky movements. In this work, novel modular framework for zero-shot motion planning for manipulation tasks is developed. The modular components do not require any motion-planning specific training. An LLM is combined with vision model to create Python code that interacts with a novel path planner, which creates a piecewise linear reference path with bounds around the path that ensure safety. An optimization- based planner, the BoundMPC framework [1], is utilized to execute optimal, safe, and collision-free trajectories along the reference path. The effectiveness the approach is shown on various everyday manipulation tasks in simulation and ex- periment, shown in the video www www..ci... tuwien. ac.aatt....... at/42d2.
Keywords: Vision Language Models, Manipulation Planning, Path-following
MPC
Introduction
The rise of large language models (LLMs) is not just a technological advancement, but signifi- cant step towards enabling robots to understand and reason about the world. This development has practical implications, as it allows for the simplified use of robots everyday tasks and unseen envi- ronments. Non-expert users in human-robot interactions. This technology is particularly important in enabling robots to coop- erate and help humans, such as in household tasks or robot-aided medical care. The generalization capabilities of LLMs simplify the process of conveying instructions from a user to a robot, which should act according to these instructions. However, transforming this language understanding into safe, performant robotic motions remains an open topic with many possible solution approaches.
control robots using natural language, which is major stride
This research tackles the crucial issue of translating human-generated instructions into safe and executable robotic actions. Previous studies have proposed various methods, including lexical anal- ysis [2]. step-by-step language instructions [3, 4], translation of language instructions into program code [5, 6, 7]. and end-to-end learning for low-level robot actions [8]. While these approaches can accomplish the given tasks, they often fall short terms of robot motion performance, leading to jerky movements [5, 8, 9] or reliance on predefined motion primitives [10, 4], which restricts robot motions predefined set of skills. To overcome these limitations, [11, 12] suggests learning discrete Cartesian waypoints and utilizing existing robot motion planners to traverse them. How- ever, this necessitates training neural network for each task, which is computationally intensive and lacks generalizability across different tasks. Moreover, the waypoint sequences require training data, which is challenging to provide. This research addresses these issues by proposing a modu- lar motion planner that seamlessly integrates with large language models, resulting in smooth and performant robot trajectories that generalize well across tasks.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Vision Model
Motion Planner
"Wipe the table from right to left"
Path Planner BoundMPC (offline)
(online)
query
spatial information
query
User
Large Language Model
Python Code
command
generate and explain
Figure 1: Overview of proposed modular motion planning framework.
The controllers of robot manipulators operate at high-frequency (2 kHz) control the input torques of the robot joints. In contrast, human-generated instructions high-level plans provided at significantly lower frequency.
Bridging this gap between high-level plans and low-level control requires considering various cog- nitive, visual, spatial, and robotic requirements. Our approach tackles this challenge with a modular architecture, see Fig. This architecture includes an LLM interpret human instructions, a visual model to localize and reason about objects in the scene, a 3D information model to exploit spatial data, path planner create Cartesian reference paths with collision-free bounds, and the model predictive path-following concept BoundMPC [1] to execute safe and efficient robot trajectory along the reference path.
The generalization capabilities of the LLM can translate complex human-generated instructions into program code to communicate robot behavior the underlying modules. However, it cannot directly deal with collision avoidance the robot's kinematic and dynamic constraints. The LLM instructs path planner to find feasible collision-fre free paths toward different goals, like objects locations in the scene, which are specified from zero-shot visual model [13]. The 3D information of the scene is used to construct graph of convex collision-free sets, which is the basis for planning a Cartesian reference path. Based on these sets, the bounds around the reference paths are defined to avoid collisions with the environment. The predictive path-following controller BoundMPC [I] guides the robot towards the goal. BoundMPC computes and executes locally optimal trajectories real-time to follow the reference path within the given bounds. Splitting path and trajectory planning has the advantage that no global trajectory optimization is needed, allowing shorter planning times. The convex-set-based path planner allows simple interaction with the generalization capabilities of the LLM, and the optimization-basse trajectory planner BoundMPC creates fast and smooth trajectories.
The main contribution of this work is novel modular motion planning pipeline. This pipeline com- prises path planner based on convex sets, which unique approach, as well as the interfaces of this planner with an overlaying learning-based task planner, i.e., a pre-trained L.M and pre-trained vision model, and the underlying model predictive trajectory planner BoundMPC. This unique com- bination allows reasoning in Cartesian space by leveraging the strength of learning learning-based models deeply integrated with smooth and safe optimization-based motion planning in the joint space. The capability to create smooth, performant trajectories for unseen tasks is demonstrated simulation and experiment by using robot manipulator, showcasing the effectiveness of this approach. The main novelties of the proposed approach can be summarized as follows:
1. The modular motion planning framework can plan smooth, safe, and performant motions based on natural language instructions. This framework does not rely on motion primitives but generates via-points and task-specific path constraints.
2. A convex-set- Cartesian path planner with Cartesian bounding functions describes the obstacle-free region around the path. This builds a crucial component for interfacing the learning based task planner with the optimization-bann trajectory planner BoundMPC.

---

2
Related Works
The proposed work falls in task and motion planning (TAMP) [14, 15]. TAMP is concerned with breaking down complex task into, e.g., motion instructions for robot [16, 17, 18). Specifically, this work deals with the manipulation among movable objects planning [19]. A recent and impor- tant subfield is language-based motion planning, where tasks are specified as natural language instructions and are translated to executable robot trajectories. Task understanding and ordering are assumed happen within the language model. Extracting information from language has been stud- ied extensively for general applications [20, 21, 22] and specifically for robots [23, 24, 25, 2, 26]. These language-grooe grounding tools can be applied to many problems, such as learning reward func- tions in reinforcement learning [27, 28], instructing intelligent agents to obtain action policies [29], or communicating domain knowledge [30]. This work focuses grounding task instructions for robotic manipulation tasks [5, 9]. In order to successfully plan motions for a manipulator, the lan- guage instruction has to be grounded with visual information. The authors in [24] use a textual description of the scene within language model. More advanced works [31, 25, 12, 32] use image embeddings in combination with language embeddings as inputs for a network. Training these net- works for robotic tasks is computationally expensive and requires large datasets. Another approach combines an LLM for language understanding and a vision model to correlate the language features with the objects in the scene [5, 33, 34]. Since the language and vision models are pre-trained on large general datasets, no additional training is needed, which makes the motion planner inherit the generalization capabilities and zero-shot task understanding of the pre-trained networks without any computational costs for training. Furthermore, the language model programs (LMPs) [6] demon- strate strong coding capabilities, directly translating the natural language input into Python code. However, these works lead to jerky and slow robot motions and do not consider the robot kinematics for motion planning. These undesired motions occur in end-to-end learning frameworks [8] but also for networks outputting discrete actions [5, 9] due to insufficient lower-level motion generation. In contrast, our approach uses a path planner that systematically integrates with the language model and outputs cask-constrained Cartesian bounded reference paths executed with the trajectory plan- ner BoundMPC [1] for fast, safe, and smooth motions with bounded path deviations, accounting for the robot kinematics. Safety is an important feature in motion planning and is often addressed by relying on underlying planners that guarantee safety. The work on drone swarms in [35] uses an underlying safety filter, and the manipulation planning framework in [11] uses existing joint-space motion planners with obstacle avoidance. The trajectory planner BoundMPC [1] fulfills similar role in this work. In addition to safety, it also allows the incorporation of task-specific Cartesian constraints, e.g., keeping a cup upright or moving along a straight line.
Manipulator trajectory planning must consider the robot's kinematics, dynamics, and interaction with the environment. Integrating these capabilities directly into L..M-based reasoning is infeasible and does not generalize across tasks. Due to this complexity, different methods for trajectory plan- ning have emerged, which are based on sampling [36, 37], optimization [38, 1J, learning [39, 40], or a combination of multiple methods [11]. Obstacles avoidance is commonly realized by parametriz- ing the space occupied by obstacles and planning in the remaining space [38, 37, 5]. On the contrary, planning based on convex sets finds convex obstacles-free regions and plans trajectory within these regions [41. 42]. The important difference between these two approaches is that the convex regions do not necessarily cover the whole obstacle-free space but still allow task completion by finding path through the environment. The work [43] navigates through sets of convex regions using graph, which is insufficient for motion planning. The works [42, 41, 43] directly optimize a tra- jectory over the path of sets. This is computationally expensive for robotic Cartesian task-space trajectories. Therefore, we propose a novel Cartesian path planner, which computes a Cartesian reference path inside connected convex regions and determines bounds of obstacle-free space for a robot manipulator to move inside. Planning a path through a graph of convex sets can be done using piecewise linear paths where each linear segment is contained in a convex set. The convexity of the set guarantees that the whole path is within the convex sets. Due to their discontinuities, existing follwwollowing controllers, such as [44, 45, 46), struggle to follow such piecewise linear reference
3

---

This paper solves the problem of generating robot manipulator trajectory in joint space using a language instruction L given by user, e.g., "Put the cup on the table". Furthermore, the robot must perform this motion safely such that the joint-space trajectory is contained within the set of paths. In this work, a novel path planner based on Cartesian convex sets is proposed and combined with the path-following controller [1]. An alternative approach is planning a global offine trajec- tory [36, 37, 47, 48, 49, 50, 51], which is computationally expensive. Furthermore, our path planner provides an intuitive interface with natural language instructions and enables an easy specification of spsk-spec constraints, while the online trajectory planner handles the robot's kinematies and motion speed. This is computationally more efficient and allows for online adaptions.
3 Safe Language-based Trajectory Generation
safe trajectories T E Tsafe- This can be defined as
find T s.t. Isuccess (C) T E Texde
where the indicator function Success indicates a successful task execution. This problem formulation is general and applicable to many different tasks. The generality makes solving the problem difficult as many solutions may exist. Directly searching over Tsafe is infeasible as there is a significantly large number of feasible trajectories in general.
Therefore, the problem (1) is subdivided into subproblems. Thus, similar to [5], it is assumed that the instruction L consists of n subtasks 4 such that L = {41 .2.,2,,..,4.... e.g., the instruction "Put the cup on the table" may be subdivided into the subtasks "Detect the cup". "Move to the cup", "Grasp the cup". "Move to the table", and "Release the cup". Each the subtasks 4 is successfully performed by a corresponding trajectory T;. this work, each trajectory T; is represented by a set of via-points W; (Wvia.J: j L..... .m.. where each via-point consists of a desired end-effector pose Pvin.j. containing position and orientation, and gripper state Ivia.j [0, 1], corresponding to closed (fvia.. = 1) or open (gvin.j 0). Additionally, the collision-free space along the path between the via-points is specified using bounds to ensure collision-fre trajectories Ti Tsafe-
3.1 Motion Planning Framework
This section presents our developed motion planning framework for solving problem (I). An overview the framework is shown in Fig. It consists of four distinct components, namely:
1. An LLM for breaking down the instructions L into actions A defined in Python code,
2. a vision model to align the actions A of the LLM with the RGB image of the scene and retrieve spatial locations from a 31) point cloud,
3. a novel path planner based on convex sets using the IRIS algorithm [52] to plan a piecewise linear bounded and collision feeiisiio--reeeeee reference path π,
4. and the model predictive controller BoundMPC [1] to traverse the reference path ा.
The LLM is instructed to output Python code to perform the necessary actions A, see Fig. 1. This work uses a combination of language model programs (LMPs) [6] with Chat-GPT4 [53]. To this end, the generated Python code uses the 2D vision model and the LLM to retrieve specific objects and locations from the scene. The corresponding 3D coordinates are obtained by querying the 3D point cloud. In order to execute action A on the robot, the motion planner is employed move the robot's end-effector to the planned location. In turn, the path planner first determines obstacle- free path and bounds using convex sets of the scene. Finally, the model predictive path-following controller BoundMPC executes the robot motion. The first three steps are performed offline, and the final trajectory planning using BoundMPC is performed online at 10 Hz. The planned trajectories are sent to a computed-torque controller that computes the necessary joint torques at 8 kHz..
This idea is based the work in [6] also used in [5] for robot motion planning. The main difference to this work is the trajectory planning. The work in [5] uses smoothed 3D cost maps plan the end-effector trajectory, which is not guaranteed to be executable or collision-free. Furthermore,

---

S
S
S2
P
Prin.
2122
Pai
S3
S
V. and
L
22,3
S3
(a) Graph
(b) 2D environment
Figure 2: 2D planning example using the convex-set- based path planner. (a) graph 9 convex sets S, (b) environment for planning and the resulting reference path π.
the Cartesian trajectories are converted to joint- trajectories by sending the desired Cartesian end-effector pose of the current time step to controller. This formulation leads to jerky and slow robot motions. The following further explains the individual components of the proposed modular framework.
3.1.1 Vision Model
Performing task requires the robot understand the surrounding scene visually. The vision model comprises the zero-shot open-vocabulary object detection network OWL-ViT [13]. The model is queried detect objects of a specific kind, e.g., tables, shelves, and cups, and returns list of objects with the corresponding areas in the RGB image that make up this object. After refining the 2D areas using the semantic segmentation network SegmentAnything [54], they are projected into the 3D point cloud to locate the objects in the scene spatially.
The LLM creates Python code in this work, invoking the other components complete the task. The code queries the vision model get scene information. For example, when the user asks the robot to "move the cup the table", the code will query the vision model to locate the cup and the table. If several cups are detected, the Python code will call the LLM clarify the relevant object instance for the task. The combination of the LLM with the vision model is inspired by VoxPoser [5].
3.1.2 Path Planner based on Convex Sets
In order to successfully perform a task, the LLM generates robot actions that move the end-effector from the current pose Pu = Pvia.0 to a desired pose Pr = Pvia. A novel path planner based on convex sets is employed to find collision-free path in Cartesian space. The core idea of the planner is compute collision-free convex sets in the Cartesian space [52] and connect them graph based their overlaps. A graph planner [43] then finds path through the graph, which is converted into linear reference path with Cartesian bounds, to be exccuted by BoundMPC [1]. The novelty of this planner is the computation of the bounded reference path to traverse the graph. An alternative approach would be to compute a collision-free reference path using existing path planning methods, c.g., [36, 48]. However, this needs to consider the robot's kinematics which leads to high computational complexity. Our approach defines bounded Cartesian space along the path in the offline step and utilizes online trajectory planning to account for the robot's kinematics. This computationally less expensive, and it is possible to incorporate task-specific constraints specified by natural language during the reference path computation, Appendix E for more details.
In the following, the functionality of the planner explained in detail for an example 2D environ- ment, see Fig. 2. The 2D environment in domain D C R2² contains six convex grey obstacles that constitute the obstacle space O. The path planner must find obstacle-fre free path i from the starting point Po the end point pr. The IRIS algorithm [52] is employed compute collision-free convex
sets Sa D around the sample points P = {Pa.k & 1,... K). Each set is represented
a linear inequality Sa {p E DÍO A&p b&} with the matrix A and the vector b
5

---

describing the halfspaces that define the convex set Sg. A graph ς of convex sets is build as detailed in Appendix A. The optimal path through the graph 9 is given as the sequence of sets Spath, with 1 = 1..... L. As the sets describe safe regions for the robot to move in, constructing a reference path is desirable to systematically consider the safe space when planning toward the endpoint pr- Linear reference paths with Cartesian bounds achieve this goal, and their construction based on the sets 1 is detailed in Appendix B. The resulting piecewise linear path is shown in Fig. 2b. Linear paths have the advantage of being contained within convex set as long their start and end points are. This is not the case for more complex formulations such as splines.
The robot must remain within the convex sets Spath. to ensure collision freedom. This is ensured using the bounding functions F and V, around the path segment स. within BoundMPC. In order to interact with the path-following controller in Section 3. 1.3,, is desirable obtain smooth bounding functions, detailed in Appendix C, because BoundMPC is an optimization-basee framework.
The above approach is used for the position reference path of the end-effector. Finding convex sets for the orientation is more challenging due to the more complex space of orientations in three dimensions. The current planner assumes knowledge of the orientation at the starting point po and end point pr, which can be obtained from grasp pose detection, e.g., [55, 56]. Grasp pose detection is an important feature, especially for complex object geometries. Adding a grasp pose detection model our framework is straightforward and will be part of future work. At each via-point Pvia.l. the orientation is linear interpolation between the start and the end orientation.
Remark, The physical extent of the end-effector is taken into account by increasing the size of the obstacles by the size of the end-effector. This works best with a symmetric end-effector.
3.1.3 Model Predictive Path-following Control
Following the path from Section 3.1.2 requires path- path-following controller that systematically considers the Cartesian bounds V, and M In this work, the receeding-horizon planning BoundMPC framework [1] is utilized. It allows the end-effector of a robot manipulator to follow the Cartesian path with bounds. As proposed in [1]. a piecewise linear reference path formulation as constructed in Section 3.1.2 provides a simplified yet performant choice for the reference path क. The bounds guide the robot along the path æ while ensuring collision freedom. Furthermore, the Cartesian path may be hard follow exactly since it does not consider the kinematics of the robot. The bounds enable the robot to deviate from the path and find optimal trajectories q(t) within the given collision- free Cartesian space, defined by the bounding functions V and y The ad advantages over state-of- the-art MPC formulations are discussed in Appendices E. and E.2. Currently, the bounds ensure collisionfrreddm for the end-effector. However, it is straightforward to add collision-avoidance terms to enforce safety for the entire kinematic chain of the robot, as shown in Appendix F.
4 Experimental Results
The trajectory planning framework developed in Section 3 is evaluated in simulation and experiment in this section. As simple example, a cup is picked up and placed on the opposite side of book, placed upright on a table. Afterward, tasks are executed to show the proposed method's versatility, generalization, and robustness. For the implementation of the path planner based on convex sets in Section 3. 3.2. the Drake software framework [57] is employed. Appendix H shows the LLM prompts used in this section, which rely on the functions listed in Appendix G. More simulation re- sults comparing the individual components of our proposed framework with state-of-the-as methods are shown in Appendix E, and
examples are given in Appendix F.
The trajectory to pick up the cup for the prompt "Place the cup left to the book" is visualized in Fig. 3a. The path planner from Section 3.1.2 plans a piecewise linear reference path я from the starting point Po to the end point P& with one via-point Pvia,1- The BoundMPC framework follows reach Dy without colliding with the book. This compared to VoxPoser [5], where the desired end-effector trajectory is planned based 3D cost maps. The book and the table have a negative cost in repelling the end-effector, while the cup has positive cost in attracting the end-effector. The

---

0.6
P
0.5
Po.
m
m
0.4
0.4
Pein.1
Pr
0.2
0.3
Pr
-0.2
0
0.2
0.4
0.6
0.8
y / m
/ m
Cup
Book
Table
Block
Shelf
Ours
BoundMPC
VoxPoser
VoxPoser
(a) Grasp cup behind a book
(b) Grasp block from shelf
Figure 3: Comparison of the end-effector trajectories for BoundMPC and the desired end-effector trajectory VoxPoser to (a) grasp the green cup while avoiding the yellow book and the grey table and (b) grasp the green block from the yellow shelf in the environment shown in Fig. 5, The trajectories are projected into the y-z-plane and z-z-plane, respectively. The light grey shaded area the enclosed
of the bounding functions where the robot's end-effector allowed The starting points do not coincide due to a smoothing filter applied by VoxPoser, resulting in non- smooth behavior at the beginning. The arrows in (b) indicate the end-effector's orientation around the y-axis, where the path bounds are chosen high to allow BoundMPCC to optimize the orientation.
Table 1: Comparison of our method with VoxPoser (VP) [5] for six tasks in terms of successful task execution, collision-freedom, average path length, and intermediate stops.
Task
Success
Collision-free
Avg. path length
Stops
VoxPoser (VP)
Ours
VP
Ours
VP
Ours
VP
Ours
Simple Move
33 %
100 %
83 %
100 %
1.44m
1.48 m
3
2
Swapping
%
100 %
50 %
100 %
4.43 m
11
Reposition
100 %
100 %
100 %
100 %
1.41
0.82 m
3
2
Arrangement
0 %
100 %
0 0%
100 %
4.33 m
3.72m
11
6
Tea Cup
0 %
50%
100 %
100 %
2.11
2.02 m
3
2
Wiping
100 %
100 %
100 %
100 %
1.84
1.22 m
4
1
planned trajectory gets stuck on the right side of the book and then jumps to the endpoint py. This results in a collision with the book. It is also not possible to tune the cost functions appropriately to achieve collision-fire free trajectory using VoxPoser, as relies on RRTConnect to create a joint trajectory from the desired Cartesian trajectory shown in Fig. This procedure does not guarantee that the desired Cartesian trajectory followed and invalidates any collision-free free guarantees in the reference trajectory. BoundMPC ensures that the end-effector remains safe without any cost tuning.
The performance of our proposed methods is further evaluated in simulation in the environment shown in Fig. and Fig. with the following tasks:
1. Simple Move: "Move the [obj] [into/onto] the [left/right] shelf".
2. Swapping: "Swap the [obj] with [obj] by using the [table/right shelff".
3. Reposition: "Put the [obj][slowly/quickky] onto the [leftright] side of the table",
4. Arrangement: "Arrange the three blocks as [line/triangle] on the table".
5. Tea Cup:
"Place the tea cup onto the [left/right] shelf without spilling it",
6. Wiping: "Wipe the table from [left/right] to [right/left]",
where [obj] is chosen among [green block/blue block/yellow block]. This leads total of 42 distinct tasks. The simulation assumes perfect visual information such that the performance of the Python code created by the LLM is evaluated independently of the performance of the visual

---

model. Table shows that the proposed method outperforms VoxPoser across all tasks. The Simple Move task is challenging for VoxPoser due to the collision avoidance where a situation similar to picking up the cup in Fig. is encountered. The proposed method plans within collision-free convex sets to avoid any collisions. In the Swapping task, the LLM used in VoxPoser returns invalid instructions, which do not solve the task. This caused by the independence of the subtask execution in VoxPoser, i.e., each subtask 1 of the instruction C is assumed to be independent of all other subtasks. This assumption is invalid for the swapping scenario, as the first object must be removed, and the second object has to be moved to the previous position of the first object, which is not considered in this subtask. The Reposition task repositions an object on the table. Both methods perform this task successfully, but the proposed framework does this more efficiently regarding the path length, In the Arrangement task, the block must be grasped from different heights on the shelf and arranged on the table. VoxPoser creates a Cartesian path that the robot cannot follow due to its joint limits, which is visualized in Fig. 3b. Using our method, the bounded deviations from the path allows BoundMPC utilize the orientation around the y-axis and find suitable trajectory while avoiding the joint limits. This emphasizes the advantage of considering the robot's kinematic limits during planning and execution. The computed trajectory from VoxPoser leads to a collision with the shelf and requires a constant orientation, which the robot cannot achieve. A similar situation occurs in the Tea Cup task where a tea cup must be moved to a new position without spilling the tea. VoxPoser outputs a non-executable Cartesian trajectory. The LLM instructs our planner to keep the tea cup upright. BoundMPC then exploits the unconstrained rotation direction around the z-axis and the path deviations regarding the position to find suitable trajectory accomplish the task. Lastly, the Wiping task is solved by both frameworks. In this task, our method requires fewer stops compared to VoxPoser. The task requires picking up the sponge and moving it to one side of the table, from which the wiping motion starts. VoxPoser stops during the pick-up of the sponge, which is necessary, but also at the beginning and end of the wiping motion, which is not required. With our method, the robot moves smoothly without stopping at the via-points, leading to faster task execution. Furthermore, our method specifies that the wiping segment is constrained in position and orientation to follow the line and stay orthogonal to the table top. The LLM effectively instructs the planner to consider these constraints. Especially in the Tea Cup and Wiping tasks, the constraints specified by the LLM show a deep integration of the learning- based model and the motion planner. The L.M specifies the task-relevant constraints, and the planner finds executable, smooth, and collision-free trajectories that solve the task. The proposed framework performs well in the tasks and outperforms VoxPoser regarding task success, collision avoidance, average path length, and the number of intermediate stops.
We also demonstrate the performance of the developed framework in several experiments in the video at www. www.acin. tuwien. ac aa.aac..a...... at/42d2. The video showcases the smooth trajectories and the planner's generalizability while staying safe. More information is given in Appendix D.
5 Conclusion, Limitations and Future Work
This paper presents a modular language-driven zero-shot motion planning framework to create safe and smooth manipulator trajectories. A novel Cartesian path planner developed to interface the learning-mod models with the trajectory generator. This allows the learning-based components to reason in Cartesian space while smooth and safe trajectories that respect the robot's kinematic limits are generated in the joint space. The strength of the proposed framework is validated in experiments and simulations, However, some limitations exists. Prompt engineering is needed to query the LLM successfully, which requires the user to invest upfront time. Furthermore, the modularity of the approach requires suitable interfaces between the modules, which can be improved for more diverse scenarios. This especially important for the LLM and the vision module, where recent work on vision language models can provide helpful extension. The safety of the current approach is limited by the discretization within MoundMPC [1]. In the literature, continuous-tine methods guarantec safety for the whole trajectory, e.g., based on reachability [58, 59, 60). However, in practice, we never encountered collisions due to the discretization.
8

---

Acknowledgments
We would like thank Alexander Wachter for the help regarding the implementation of the com- parisons and Florian Beck for the fruitful discussions.
References
[1] T. Oelerich, F. Beck, C. Hartl-Nesic, and A. Kugi. BoundMPC: Cartesian Trajectory Plan- ning with Error Bounds based on Model Predictive Control in the Joint Space. Preprint, 3xXiv:2401.050577 2024. doi:arXiv:2401.5000. 005057.
[2] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. Proceedings of the AAAI Conference on Artificial Intelligence, 25:1507-1514, 2011. doi: 10i10...a//aai. v2511.7979.
[3] W. Huang. P. Abbeel, D. Pathak, and I. Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. Preprint, (arXiv:2001.02207, 2022. 000:100445500/rxx.2220.0077
[4] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, Julian, D. Kalashníkov, Y. Kuang, K.-H. Lee, S. Levine, Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, Rao, Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do As Can, Not As I Say: Grounding Language in Robotic Affor- 2022. doi: 42.50//a
dances. Preprint,
[5] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 540- 562. PMLR, 06-09 Nov 2023. URL / //pppe/dd///cceniggss. mlr, wwtpppeeee////aaeeeeeep html.
[6] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as Policies: Language Model Programs for Embodied Control. Preprint, (arXiv:2209.07753), 2023. doi: (00488500/rxx1.2220.0777
[7] Y. Jin, D. Li, Y. J. J. Shi, P. Hao, F. Sun, J. Zhang, and B. Fang. RobotGPT: Robot Manip- ulation Learning from ChatGPT. IEEE Robotics and Automation Letters, pages 1-8, 2024. doi:
[8] A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan. A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishma, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf,
B. Wulfe, B. Ichter, C. Lu, Xu, C. Le, Finn, C. Wang. C. Xu, C. Chi, Huang. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Wang, H. Su, H.-S Fang, H. Shi, H. Bao, H. B. Amor, H. I. Chris- tensen, H. Furuta, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, Leal, J. Liang, Abou-Chakra,
Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Bohg. J. Bingham, J. Wu, Gao, Hu, J. Wu, Wu, J. Sun, Luo, J. Gu, Tan, J. Oh, Wu, Lu, Yang, J. Malik, J. Silvério, J. Hejna, Booher, Tompson, Yang, J. Salvador, J. J. Lim, J. Han, K. Wang. K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byme, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis,

---

K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, I.. Pinto, I.. Fei-Fei, L. Tan, L., J. Fan, I. Ott, L.. Lee, L.. Weihs, M. Chen, M. Lepert, M. Mem- mel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. Kim, Kanazawa, N. Hansen, N. Heess,
N. Joshi, N. Suenderhauf, N. Liu, N. Di Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q Chen, Q Vuong, R. Rafailov, R. Tian, R. Doshi, R. Martin-Martin, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang. R. Men- donca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, Song. S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, Nasiriany, S. Schaal, S. Welker, Tian, Ramamoorthy, S. Dasari, Belkhale, S. Park, Nair, S. Mirchandani, Osa, Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liang- wei, X. Li, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X- E-mbodimenn: Robotic Learning Datasets and RT-X Models. Preprint, (arXiv:3310.08864) 2024. doi: 10. 48555//ariw.23110006..
[9] A. Stone, T. Xiao, Y. Lu, Gopalakrishnan, K.-H. Lee, Q. Vuong, Wohlhart, Kirmani, Open-World B. Zitkovich, Xia, C. Finn, and K. Hausman. Open- World Object Manipulation using Pre- trained Vision-Language Models. Preprint, (arXiv:2303.00005) 2023. ivv 2303.00905,
[10] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and Bohg. Text2Motion: From Natural Language Instructions to Feasible Plans. Autonomous Robots, 47(8)::345-11355, 2023. ISSN 0929-55933,
1573-7527. doi: 10. 10000000100111-2211003333
[11] S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13739,1373,,2011 2022. doi:
[12] M. Shridhar, Manuelli, and D. Fox. Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation. Preprint, (arXii:2209.05451) 2022. No0::0004550//2xx22000.00
[13] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, Dosovitskiy, Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby. Simple Nven-Vocabulary Object Detection with Vision Transformers. Preprint, (arXiv:2205.06230,
2022. doi: 000:10008550//axx.2205.666 06230..
[14] H. Guo, F. Wu, Y. Qin, R. Li, K. Li, and K. Li. Recent Trends in Task and Motion Planning for Robotics: A Survey, ACM Comput. Surv., 55(13)):289; 12:::))9 2023. doi: 10.1 145/3583
[15] Z. Kai, E. Lucet, J. Alexandre dit Sandretto, S. Kchir, and D. Filliat. Task and motion planning methods:
Applications and limitations. 2022.
[16] X. Zhang, Y. Zhu, Y. Ding, Y. Zhu, P. Stone, and S. Zhang. Visually Grounded Task and Motion Planning for Mobile Manipulation. Preprint, (arXiv:2002.10667), 2022. 1oi. 2202.10667.
[17] R. Takano, H. Oyama, and M. Yamakita. Continuous Optimization- Based Task and Motion Planning with Signal Temporal Logic Specifications for Sequential Manipulation. In 2027 IEEE International Conference on Robotics and Automation (ICRA), pages 8409-8411 2021. 001:00..10090C44444000000.0.....
10

---

[18] S. Cambon, R. Alami, and F. Gravot. A Hybrid Approach to Intricate Motion, Manipulation and Task Planning. The International Journal of Robotics Research, 28(1):104-126, 2009. doi: 101:1011170077864980078888
[19] D. Saxena and M. Likhachev. Planning for manipulation among movable objects: Deciding which objects go where, in what order, and how. Proceedings of the Thirty-Thinn Thiry Interna- fional Conference Automated Planning and Scheduling, volume 33 of ICAPS '23, pages 668-676, Prague, Czech Republic, July 2023. AAAI Press. ISBN 978-1-57735-881-7, doi: 10. http://iccaa..33..222..
[20] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari. W. M. Czar- necki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom. Grounded Language Learning in a Simulated 3D World. Preprint, (arXiv: (00.:.1706065551) 2017. doi: doi:10..4855//arxivv. 1ww.10666...651..
[21] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Vision-and-Languune Navigation: Interpreting Lisually-Grounder Nav- igation Instructions in Real Environments. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 3674-3683, 2018. 100110..109/ 00387.
[22] A. W. Hanjie, V. Y. Zhong, and K. Narasimhan, Grounding Language to Entities and Dynamics for Generalization Reinforcement Learning. Proceedings of the International Conference on Machine Learning, pages 4051-4065 PMLR, 2021.
[23] S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek. Robots That Use Language. Annual Review of Control, Robotics, and Autonomous Systems, 3(Volume 3, 2020):25-55, 2020. ISSN 2573-5144. doi: 10. 1144/amurey-comtrr 14/
101 119-071628.
[24] S. Wang, M. Han, Z. Jiao, Z. Zhang, Y. N. Wu, S. S.-C. Zhu, and H. Liu. LLM3:Large Lan- guage Model-based Task and Motion Planning with Motion Failure Reasoning. Preprint, (arXiv:2403.I) 1552),, 2024. doi: 400104455500xxX12400.11555
[25] Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, An- dreas, and D. Fox. Correcting Robot Plans with Natural Language Feedback. Preprint, (arXiv:2204.55186), 2022. 40:110448550aaxx:224..1111
[26] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and Motion Planning with Large Language Models for Object Rearrangement. Preprint, (arXiv:2303.06227), 2023. doi: 10.45500///r 2303.06247.
[27] J. Fu, A. Korattikara, Levine, and Guadarrama. From Language to Goals: Inverse Re- inforcement Learning for 2019. www.44000////..11922....
Instruction Following. Preprint, (arXiv:1902.07744),
[28] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, Zhong, Y. Yang, and T. Yu. Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning Preprint, (arXiv:2309.114) 11499), 2023. 0tt:10048850//rxx..23300 11489.
[29] H. Hu and D. Sadigh. Language Instructed Reinforcement Learning for Human-AI Coordina- tion. Preprint, (arXiv:2304.07297) 2023. doi: 1004485500//2xi2223000222
[30] J. Luketina, N. Nardelli, G. Farquhar, J. Foerster, Andreas. E. Grefenstette, S. Whiteson, and Rocktäschel. Survey of Reinforcement Learning Informed by Natural Language. Pro- ceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 6309-6311 Macao, China, 2019. International Joint Conferences Artificial Intelligence Organization. ISBN 978-0-9992411-441
do:: 1oi:10024963/jcci.222 2019/8880
11

---

[31] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke,
K. Hausman, M. Toussaint, K. Greff, A. Zeng. Mordatch, and P. Florence. PaLM-E: An
Embodied Multimodal Language Model. Preprint, (arXiv:2303.03379) 2023.
[32] A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and Bonatti. LATTE: LAnguage Trajectory TransformEr. Preprint, (arXiv:2208.0291 2022. doi: 10.45500///rrxi 2208.02918.
[33] D. Shah, B. Osinski, B. Ichter, and S. Levine. LM-Nav: Robotic Navigation with Large Pre- Trained Models of Language, Vision, and Action. Preprint, (arXiv:2207.04429) 2022. doi: 10. 10.48550/arxi..227777 20070444... .0429.
[34] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. Preprint, (arXiv:2209. 11302), 2022. doi: 10.445550/aaXX..222.11333.
[35] A. Jiao, T. Patel, S. Khurana, A.-M. Korol, L.. Brunke, K. Adajania, U. Culha, Zhou, and A. Schoellig. Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design. Preprint, (arXiv:2312.01059) 2023. doi:10.48550/rriii 2312.01059.
[36] S. Karaman and E. Frazzoli. Sampling-based algorithms for optimal motion planning. The International Journal of Robotics Research, 30(7):846-894, 2011. ISSN 0278-3649. doi: 10. 10.1177/02783649 11177002733649114076..
[37] S. M. Persson and I. Sharf. Sampling-based A algorithm for robot path-planning. The International Journal of Robotics Research, 33813)1668-1 2014. doi:10.1177/
0278364914547786.
[38] T. Schoels, P. Rutquist, L. Palmieri, A. Zanelli, K. O. Arras, and M. Diehl. CIAO*: MPC- based safe motion planning in predictable dynamic environments. PPAC-PaperSOnL.ine 53(2): 6555-6562, 2020. ISSN 24058963. doi: 101166 oft.:......clll222222.o.
[39] T. Osa. Motion planning by learning the solution manifold in trajectory optimization. The International Journal of Robotics Research, 41(3):281-311, 2022. ISSN 0278-3649. doi: 10.1177/0278364921 1044405.
[40] J. Carvalho, A. T. Le, M. Baierl, D. Koert, and Peters. Motion Planning Diffusion: Learn- ing and Planning of Robot Motions with Diffusion Models. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1916-1923, 2023. doi: 11111190RRRO555522030404444
[41] T. Mareucci, M. Petersen, D. Wrangel, and R. Tedrake. Motion planning around obstacles with convex optimization. Science Robotics, 8(84):eadf7843, 2023. doi: 126/scirobotsss. adf7843.
[42] R. Deits and R. Tedrake. Efficient mixed-integer planning for UAVs in cluttered environments. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 42-49, 2015. doi: 10. 109/CRA.2015.713899
[43] Marcucei, Umenberger, P. A. Parrilo, and R. Tedrake. Shortest Paths in Graphs of Convex Sets. Preprint, (arXiv:2101.11565), 2023, 401:100885500/xx22111115555
[44] C. Hartl-Nesic, T. Glück, and A. Kugi. Surface-based path following control: Application of curved tapes on 3-D objects. IEEE Transactions Robotics, 37(2):615-66 626. 2021. doi:
12

---

[45] T. Faulwasser, B. Kern, and R. Findeisen. Model Predictive Path-Following for Constrained Nonlinear Systems. In Proceedings of the IEEE Conference on Decision and Control, pages (042-8647, Shanghai, China, 2009. doi:10. 109/9CDD.2220095999974.
[46] A. Astudillo, G. Pipeleers, J. Gillis, W. Decré, and J. Swevers. Varying-Radiun Tunnel- Following NMPC for Robot Manipulators Using Sequential Convex Quadratic Programming. In Proceedings of the Modeling. Estimation and Control Conference, volume 55, pages 345- 352, Jersey City, USA, 2022. doi: doi:10.1016//.faacoooo (f2..l.022...11..2000
[47] J. Jankowski, L. Brudermüller,, N. Hawes, and S. Calinon. VP-STO: Via-point-busel based Stochas- tic Trajectory Optimization for Reactive Robot Behavior. In Proceedings of the IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 10125-10131, Mar. 2023. 10910101//// 1016.111
[48] R. Verschueren, N. van Duijkeren, J. Swevers, and M. Diehl. Time-optimal motion planning for n-DOF robot manipulators using a path-parametric system reformulation. In Proceedings of the American Control Conference (ACC), pages 2092-2097, July 2016. doi:10. 1099/CC 2016.7525227.
[49] M. Zucker, N. Ratliff, A. D. Dragan, M. Pivtoraiko, M. Klingensmith, C. M. Dellin, J. A. Bagnell, and S. S. Srinivasa. CHOMP: Covariant Hamiltonian optimization for motion plan- ning. The International Journal of Robotics Research, 320--00):114411999 2013. doi: 10. 10.1177/027836499 33488805.
[50] J. Schulman, J. Ho, A. Lee, I. Awwal, H. Bradlow, and P. Abbeel. Finding Locally Opti- mal, Collision-Free Trajectories with Sequential Convex Optimization. Robotics: Science and Systems IX, June 2013. 0od::0..16677// 15500//SS..011....0...
[51] M. Kalakrishnan, S. Chitta, E. Theodorou, P. Pastor, and S. Schaal. STOMP: Stochastic tra- jectory optimization for motion planning. In Proceedings IEEE International Conference Robotics and Automation, pages 4569-4574, May 2011. doi: 1TA.L/////AAAA.111900000
[52] R. Deits and R. Tedrake. Computing Large Convex Regions of Obstacle-Free Space Through Semidefinite Programming. In H. L. Akin, N. M. Amato, V. Isler, and A. F. Van Der Stap- pen, editors, Algorithmic Foundations of Robotics XI, volume 107, pages 109-124. Springer International Publishing, Cham, 2015. doi: 100
[53] OpenAl. GPT-4 Technical Report. Preprint, (arXiv:2303.08774) 2024. doi: 10i455 2303.08774.
[54] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, Gustafson, T. Xiao, S. White- head, A. C. Berg, W.Y.. Lo, P. Dollár, and R. Girshick. Segment Anything. Preprint, (arXiv:2304.02643) 2023. 46::1044850aaxxi..32444om
[55] A. Mousavian, C. Eppner, and Di. Fox. 6-DOF GraspNet: Variational grasp generation for object manipulation. Proceedings of the IEEE International Conference on Computer Vision, 20 10110ctob:2901-29100 2019, ISSN 15505499, doi: 109///C/..2211900029999
[56] A. D. Vuong. M. N. Vu, H. Le, B. Huang, B. Huynh, T. Vo, A. Kugi, and A. Nguyen. Grasp- Anything: Large-scale Grasp Dataset from Foundation Models. Preprint, aarXiv2300909888)) 2023.
[57] R. Tedrake and D. D. Team. Drake: Model-based design and verification for roboties, 2019.
[58] I. M. Mitchell. Comparing Forward and Backward Reachability Tools for Safety Analysis. In A. Bemporad, A. Bicchi, and G. Buttazzo, editors, Hybrid Systems: Computation and Con- trol, pages 4484433, Berlin, Heidelberg, 2007. Springer. doi:
13

---

[59] J. Michaux, A. Li, Q. Chen, C. Chen, B. Zhang, and R. Vasudevan. Safe Planning for Artic- ulated Robots Using Reachability-basee Obstacle Avoidance With Spheres. Preprint, 2024.
[60] M. Chen, S. Herbert, H. Hu, Y. Pu, J. Fernandez Fisac, S. Bansal, S. Han, and C. J. Tom- lin. FaSTrack: A Modular Framework for Real-Time Motion Planning and Guaranteed Safe Tracking. IEEE Transactions on Automatic Control, (Cdc):1517-1522, 2021. doi: 10.1 2..1009A..2221.3053355
[61] L. Dai, Y. Yu, D.-H Zhai, I. Huang, and Y. Xia. Robust Model Predictive Tracking Control for Robot Manipulators With Disturbances. Proceedings of IEEE Transactions on Industrial Electronics, 68(5)=48884427, 2021. doi:
[62] Y. Wang, Y. Pang, Q. Li, W. Cai, and M. Zhao. Trajectory Tracking Control for Robot Ma- nipulator Under Dynamic Environment. In H. Yang, H. Liu, J. Zou, Z. Yin, L. Liu, G. Yang, X. Ouyang, and Z. Wang, editors, Proceedings of Intelligent Robotics and Applications, pages 513-524, Singapore, 2023. Springer Nature. doi: 10.
1177788988999-492-5550
[63] M. N. Vu, P. Zips, A. Lobe, F. Beck, W. Kemmetmüller, and A. Kugi. Fast motion planning for laboratory 3D gantry crane in the presence of obstacles. IFAC-PapersOnLLine, 53(2): 9508-9514 Jan. 2020. doi: 10. 101.001166..ffcooo...... .f2.l..f.c........11.. 20..o..2220.1..2227... 12.2427.
[64] M. Toussaint, Harris, J.-S. Ha, Driess, and W. Hönig. Sequence-of- Sequence-of-Constraitts MPC: Reactive Timing-Optinn Control of Sequential Manipulation. Preprint, (arXiv:2203.05390, 2022. doi:10.4888 10..4400//a.....223.....
[65] I. A. Şucan, M. Moll, and L. Kavraki. The Open Motion Planning Library. IEEE Robotics & Automation Magazine, 19(4):72 82, Dec. 2012. doi:10. 109/MRA.2012.22056.
[66] D. Ferguson, N. Kalra, and A. Stentz. Replanning with RRTs. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 1243-1248, Orlando, FL, USA, 2006. doi: 10. 10000000TT...00.611111 1641879
[67] L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Au- tomation, 12(4))566-580 1996. doi: 10. 109/7.09700
14

---

A Graph of convex sets
The convex sets needed to build a graph 9 are computed using the IRIS algorithm [52]. The sample points p - (Ps.: k = I,.... } are hyperparameters of this algorithm. this work, the sample points P include the starting and end points Po and Pr, and six additional known collision-free points.
Remark. Providing small set of known obstacle-free points often easily possible. this is not the case, a rejection sampling approach can be adopted. More complex strategies based the knowledge of Po and can be developed but are not part of this work. The authors of [41] propose to adapt methods developed for
to improve the sampling.
Once collision-free convex sets are found around all sample points P, the graph G is constructed in which each convex set Sş is represented by a node, similar to [41]. For each pair of convex sets S. and S found by the IRIS algorithm, an edge is added graph & between the corresponding nodes, their intersection is not empty, i.c., Sa,b S n Sb + 0. The cost associated with this edge is chosen
Ca.b
min ||p prll PeS.,b
(2)
which is the projection of the end point Pr onto the intersection set S..b. Note that the graph is undirected as Ca,b = Cb and each edge allows passing in both directions. The constructed graph g for the 2D environment is shown Fig. 2a. The shortest path through the convex sets is then calculated using the optimizer proposed in [43]. The optimal path through the graph according to the cost (2) is given as the sequence of sets Spath, with / I....,L.
B Piecewise linear reference path
For constructing the piecewise linear reference path the sequence of sets Spath,/ is used. Each set intersection Sutee. = Spath./ n Spath,4+1 with = 1..... L between two consecutive sets Spath, and Spath,t+1 in the solution path requires one via-point Pvia.-- The I, - intersection sets Sinter, are used to compute L - via-points by solving the optimization problem
Pvia.1 =
arg min |p- Poll + |pp Prllž + a alllmax(AAaaaaaa Aitter. binter.)) + + PESunter.
(3)
with the weights 2 O and B 2 O. Together with the starting point Po = Pvia.0 and the end point PF L. they make up the set of via-points. In (3), the matrix Ainter, and the vector binter define the intersection set Sinter. and the term [| taken over the vector entries, incentivises the via-point Pvia.; to be / away from any border of the intersection set Sinter,t- This moves Pvia./ from the border of the set into the set and leads to more symmetric bounds later. The piecewise linear reference path æ is the linear connection between the starting point Po Pvia,0. the via-points Pvia. I 1,..., L and the end point pr Pvia, shown in Fig. 2. This formulation ensures that the linear segment between Pvia,1- and Pvis. is completely contained in the convex set Spath,,, from which the bounding functions are computed using geometric considerations.
() B with the maximum
C Path bounding functions
In the following, the computation of the bounding functions the general case three dimensions described for given linear path segment Ty, which connects the via-points Pvia.! and Pvia./+1 and is contained in the corresponding convex set
Spath./ {p: A /ptth./P < bpath..).
A visual explanation of the procedure shown in Fig. This procedure also applies analogously to the special of two dimensions and the example in Fig, 2. In 3D, each linear path segment has constant unit-norm direction vector d and two unit-norm basis vectors b and b These basis
15

---

Spath.
b2
Precte
Pract,1
{x: A <b11}
Paid
b1
Prect,
Prect.2
Computation of reference path bounding functions V and V based on the convex set Spach, and the path π.
Figure 4:
vectors are obtained using the -rram-Schmidt procedure explained in [1], such that an orthonormal frame is created, i.e., Ldd Ld, and b, Lbg. Furthermore, the mid point of the considered linear segment Ny is Pmid- Next, Spath, in (4) is transformed into the basis vector coordinates at the point Pmid by using the transformation
(pp Pmid) [b1 by] (p Pmid)
Swans. = {z path, A PT < bpath./ - Apath,/Pmia)
(5)
A rectangle that is aligned with the basis vectors b and b in the transformed set (5) is defined by the four comner points Z [21,1, Z = [-I1,2 22,1), z [21,1 -2222 and 2feet.4 = 21,2, - 22,2] with the parameter vector x" T = [I1,1, 21,2, $2,1, x222 The bounds are computed by finding the largest-volume rectangle that is contained the set Strans,t using the optimization
Xopt = aarg max(x111 + (12) + 22,2)
S.L. 0,
(6)
Zrect,i Suaus. i :,...:4.
where the constraints ensure that the rectangle is contained in the transformed set (5) and that the path æ is contained the bounds at pmid with x 2 0. This rectangle is generally a conserva- tive approximation of the transformed set (5). The rectangle parameters are used as the maximum linear segment my For each line segment, four bounding functions
bounding function values exist, i... the upper bounding functions V, and the lower bounding functions V for both basis di- rections b and b2. The respective maximum and minimum values chosen max(V/, (V,) x4,1 and min(y,, -4,2 shown as dotted lines in Fig. 2b. The value of the bounding functions at the via-points Pvia,! is zero, which the path- following controller requires. More information on the bounding function formulation is given in [1]. When choosing the bounding function as piecewise linear functions between zero at the via-points and max( (V)) and min (1,)) at Pmiid it is ensured that they remain within set Spath. because it is convex. However, the path-following controller ben- efits from smooth bounds, so fourth-order polynomials are used in this work instead. This can result in some parts of the bounds lying outside the set Spath,,. However, in practice, this does cause any issues due the conservative approximation of the collision-free space of the IRIS algorithm and the conservative approximation in (6). The bounding functions V, and Vr in combination with the reference path are used for the trajectory generation in Section 3.1.3.
D Experimental Setup
The scene of the experiments is shown Fig. 5. consists of two shelves and a table in front of the robot. An Azure Kinect RGB-D camera is mounted behind the robot provide the required data for the vision model, namely the RGB picture and the depth information used to obtain the point cloud. The user provides prompt at the beginning of episode, and the LLM creates
16

---

(a) Experimental scene
(b) Simulation scene
Figure 5: A view of the experimental and simulation scenes: The experimental scene picture is taken by the RGB-D Azure Kinect camera, also used by the vision module. The colored axes in the simulation scene at the robot's base indicate the world frame
the necessary Python code execute the task. BoundMPC runs at 10 Hz, and computed torque controller executes the created joint trajectories 8 kHz.
The convex set planner from Section 3.1.2 requires convex obstacles description of the scene. For arbitrary scenes, voxel grid based on the point cloud can be used the planner. This may lead to many obstacles and increases the computation time of the planner, safely in unknown environments. the scene is employed in this work. This results in an environment with 13 obstacles, leading to computation times of about 0.15s for the path planner.
allows the robot to act speed up the computation, known convex decomposition of
Common failures of the vision model are falsely detecting an object that not present or not detect- ing an object that is present. Further failure cases of the L.MM and the vision model are discussed in [5].
E Advantages of convex-set-based path planning with trajectory optimization
This section compares the proposed motion planner, consisting of the convex-set-hased path planner from Section 3.1.2 and the trajectory planner BoundMPC [1] to existing approaches. The advantages are summarized in Appendix E.4. Additionally, the simulation comparisons are shown in the video acin. www.inttiiiiinnnccoo..... ac. ac at/42d3.
E.1 BoundMPC compared to Cartesian tracking MPCs (CT-MPC)
The current work uses an offline path planner in combination with the BoundMPC framework to compute suitable joint space trajectories. Other formulations exist to plan such trajectories. This section compares our proposed framework to an MPC trajectory tracking formulation [61, 62]. A reference trajectory created for the robot, and the goal follow this trajectory using a receding horizon MPC while avoiding obstacles. The obstacles are commonly added as potential functions to the MPC objective [63]. A reference trajectory consists of reference path with time parametriza- tion. A Cartesian reference trajectory determines the end-effector's desired position and orientation reference trajectory. This formulation is known as CT-MPC. The reference trajectory is generated based on a sequence of via-points according to the procedure described in [64] and not guaranteed to be collision-free.
For detailed simulation comparison, green block is positioned in the lower shelf part inside the left shelf of the environment shown in Fig. 5. The goal is given by the prompt "Move the green
17

---

0.9
0.8
0.7
z/m
0.6
0.5
0.4
0.3
0.50
0.55
0.60
0.65
0.70 0.75 0.85 0.90 0.95
x / m
Block
Shelf
Ours
CT-MPC
CT-MPC via-points
Figure 6: Comparison of trajectories of different MPC formulations for the prompt "Move the green block onto the left shelf." in the z-z-pl This example showcases the need for suitable reference path/trajectory execute the task successfully.
block onto the left shelf." The comparison of the created trajectories is shown in Fig. 6. For the simple CT-MPC formulation, only the desired grasping pose and drop off pose are provided as via- points. This does not lead to successful task execution as the robot gets stuck the lower shelf since it tries to move up straight but is blocked by obstacle avoidance. Thus, more advanced reference trajectory is needed. For the CT-MPC formulation with more via-points, the via-points of the convex-set seavee--ee--bseee based path planner from Section 3.1.2 are utilized to create a reference trajectory. This leads successful task execution with trajectory similar to our proposed method, showing that global reference path/trajectory necessary for the task's success.
The CT-MPC formulation needs a reference trajectory but does not require bounds around this tra- jectory obstacle avoidance is performed using potential functions in the MPC objective function. The objective the CT-MPC is to minimize the tracking error between the end-effector pose and reference pose. This leads to problems when the reference trajectory is not followable due to the robot's kinematics. An example shown in Fig. Here, the robot picks up a green block and has to put it on the shelf behind itself. Based the via-points, the created reference trajectory is hard to follow because it passes very close to the robot's base's vertical axis. Hence, large track- ing error is needed to perform the task, as shown in Fig. 8b. However, the objective of CT-MPC is to minimize this error, and the robot's kinematics hinder minimization, leading to non-smooth motions. The norm of the end-effector acceleration a, reflects this behavior. Furthermore, CT-MPC does not perform the task successfully because the large reference tracking eтrоr eref leads to the robot getting stuck local minimum above the desired shelf space, which leads to a non-zero tracking error e,er the end of the trajectory. This is avoided by using a bounded reference path this method determines which path deviations are suitable at which point in space. Hence, the proposed approach finds smoother and successful end-effector trajectory for this task.
18

---

(a) Start position
(b) Possible end position
Figure 7: Simulation setup for a turn-around task. The black spheres around the robot base are only used in the obstacle avoidance example Appendix F, The starting position the same for all formulations. Our framework realize the end position shown. The colored axes at the robot's base indicate the world frame.
Table 2: Comparison of the offline planning time of our method with the global planners VP- STO [47], RRT* [65] and STOMP [51] for the turn-around task in Fig. 8. The planning time is the combined time for the first and second trajectory planning.
Ours VPSTO
RRT
STOMP
Planning time /
0.3
53
30
411
E.2 BoundMPC compared to joint-space tracking MPCs (JT-MPC)
The two comparisons above showcase pick-and-place scenarios. These can also be solved using a joint-space reference trajectory instead of a Cartesian one. We call this formulation JT-MPC. It works well for pick-and-place tasks but struggles with tasks where Cartesian task-specific constraints are given. The reference trajectory uses the inverse kinematics solutions of the via via-points that result from the convex-set-baser planner outputs is using. The example of the wiping task from Section considered in Fig. 9, where the wiping motion along the table surface has be a straight line. The JT-MPC framework cannot take the Cartesian reference path into account, leading to undesired are-shaped trajectory. The proposed framework considers the task-specific constraints formulated by the LLM and successfully finishes the task. This shows the advantages of Cartesian reference path to incorporate task-specific constraints.
E.3
Comparison to global trajectory planning
Our approach splits the motion planning problem into offline Cartesian reference path planning and online trajectory planning. However, it is also possible to directly optimize a joint-space tra- jectory from the starting pose Po to the final pose p:. In this section, the turn-around example from Appendix E. is considered with different global motion planners. It consists of two planned trajectories: The first trajectories grabs the green block and the second trajectory transfers it to the other shelf. Our approach is compared to VPSTO [47], RRT* [65], and STOMP [51]. A compar- ison of the combined planning times for both trajectories shown in Table Our approach com- putes reference path with collision-free bounding functions considerably faster than computing collision-free joint-space trajectory. Note that all methods manage to solve the task successfully. Furthermore, a change in the environment will require a replanning, which is costly for the global planners but not for our proposed method. This shows the advantages of using our proposed methods instead of global trajectory planners.
E.4 Summary of advantages
The proposed approach of motion planning combination of convex-set-based Cartesian path planning with trajectory planning using BoundMPC has the following advantages:
19

---

0.9
0.8
0.7
0.6
m/m
0.5
0.4
0.3
Block CT-MPC
Shelf
Ours
0.2
0.1
0.0
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
m
(a) End-effector trajectories in the z-z-plane.
1.5
Ours
|| //ms"
1.0
CT-MPC
0.5
0.0
ms
|| /
0
||e || m
0.50
0.25
0.00
L/s
(b) Norms of end-effector velocity Ve, acceleration ac, and reference tracking error eref-
Figure 8: Comparison of our proposed framework with CT-MPC for turn-around task.
20

---

0.7
3. m
0.6
Table
Ours
JT MPC
0.5
0.2
0.0 y / m
0.2
Figure 9: Comparison of trajectories of different MPC formulations for the prompt "Wipe the table from left right." in the y-z-plane. Only the part of the trajectories relevant the wiping is shown. This example showcases the need for a Cartesian reference path/trajectory to execute the task successfully.
Reference path: A reference path/trajectory needed to perform the task successfully and provide global guide for the robot's motion. The example Fig. elaborates on this point.
Cartesian reference path: Cartesian reference path/trajectory has the advantage that task-specific constraints along the path are easy to incorporate compared to joint-space reference paths/tajectorriss, Fig. 9.
Bounds around the reference path: Existing sampling-based methods are a powerful tool to plan reference paths, c.g., RRT [66, 36] or PRM [67, 36]. However, they currently have notion of path deviation bounding functions. The examples in Fig. 6 and 8 show the advantages of using such bounding functions. In these examples, deviating from the reference path is necessary due to the robot's kinematics, which makes minimizing the tracking error unsuitable, leading to non-smooth trajectories. The bounding functions guide the robot towards the goal while keeping Furthermore, Cartesian reference path is required to handle task-specific constraints, such following straight line exactly keeping a cup upright. Thus, bounding functions around the reference path are powerful tool for considering the robot's kinematics and encoding task-specific constraints.
the collision-fres part of the task space.
Splitting up path and trajectory planning: Our approach first computes a bounded Carte- sian reference path offline and then plans a joint-space trajectory online. Another approach is to directly compute a joint-space trajectory offline using global planners, as shown in Ap- pendix F.3. The advantages of our approach are the faster offline planning time and faster adaption in case of environmental changes.
F Collision avoidance for the full kinematic chain of the robot
The bounded reference path computed in Section 3.1.2 considers collisions only for the robot's end- effector. In this section, this collision avoidance is extended to the entire kinematic chain of the robot using potential functions in BoundMPC's objective function. This approach is based on the formulation [63]. Additionally, it compared to the Cartesian tracking (CT-MPC) and joine tracking (JT-MPC) MPC formulation of Appendix
The example in Fig. 7, where the robot has to turn around and place an object, is considered again. Six additional obstacles are positioned close to the robot base as shown by the black spheres in Fig. 7. This is very demanding task as the space where the robot can turn around highly constrained due to the obstacles. The resulting end-effector trajectories of our framework, CT-MPC, and JT-MPC are depicted in the z-2 and r-y--lanes in Fig. 10a and b, respectively. Our framework and JT-MPC successfully perform the task, but the CT-MPC gets stuck about æ 0.05 while turning around due to the obstacles. This is because the robot must deviate from the Cartesian reference trajectory due to the kinematic constraints but has guidance on which deviation directions are suitable. Our
21

---

Table Comparison our method with CT-MPC and JT-MPPC for turn-arouna task with collision avoidance of the robot's full kinematic chain. Nine experiments are compared, in which the six obstacles in Fig. 7 are shifted along the x-axis in their z-coordinate between -0.4 and 0.4.
JT-MPC
CT-MPC
Ours
Successes
4/9
3/9
6/9
Collisions
5/9
0/9
0/9
Stuck at joint limits
0/9
3/9
3/9
Stuck at obstacles
0/9
3/9
0/9
framework uses the BoundMPC formulation, which defines the Cartesian bounding functions such that the end-effector stays collision-free It is the only formulation that can completely perform the task successfully. Even though JT-MPC reaches the goal pose Pr. collision of robot link with one of the collision spheres is detected, shown in Fig. 10c. In this example, trade-off' between minimizing the trajectory error and the collision avoidance leads to an unsafe trajectory of the robot. Our framework avoids such situations since the reference paths т are collision-fiee by design. possible improvement to JT-MPP would be providing a collision-free joint-space trajectory for the JT-MPC. However, this is not trivial and would require global trajectory planner, which is computationally expensive, as discussed in Appendix E.3.
In order to evaluate the robustness of our proposed method, the obstacles were shifted along their a-coordiinate in the range -.4 and 0.4. The evaluation of the resulting nine experiments is shown in Table 3. JT-MPC always reaches the desired final pose Pr but often collides with the environment due to the collisions in the reference trajectory. The CT-MPC formulation never collides with the environment but may get stuck at obstacles or joint limits. Our approach the most successful framework, which only sometimes gets stuck at joint limits when the reference path bounds do allow enough freedom optimize suitable joint trajectory. This comparison shows the robustness of our approach compared to the state-of-the-ar
G Functions used in the generated Python Code
The LLM generates Python code that can use the following functions:
to (pose: array, constraint: string, stop: bool)
Instructs the motion planner to move to pose. The argument constraint can be used to specify constraints along the path, which are evaluated by an LLM query, and the argument stop specifies whether to stop at pose or continue fluently towards the next motion goal.
detect (object.naaleee _ame::tttmm:::. string) object Invokes the vision module to detect objects with the name object name.
close chose_gripper( ()
Closes the gripper.
open gripper ()
Opens the gripper.
move h)me()
Move the robot's end-effector to the home position.
select object(object list: list, object name: string, instance:
string) -> object
Invoked in case multiple objects of the same type are detected with the detect function. Queries the L.M to clarify which object relevant to the task based on the instance string, e.g., finding the left shelf when instance 'left' and multiple shelves are detected.
set speed (max speed: float) Sets the maximum path speed for BoundMPC to be max_speed.
22

---

0.8
m
0.6
Pi
0.2
0.0
0.75
-0.50
0.25
0.00
0.25
0.50
0.75
m
Shelf
Block JT-MPC
Obstacle
Ours
CT-MPC
(a) End-effector trajectories in x-2 --laane
0.4
0.2
//m
0.0
-0.2
-0.4
-1.00
- 0.75
-0.50
-0255
0.00
0.25
0.50
0.75
1.00
x: m
On
Shelf
Block
Obstacle
Ours
CT-MPC
JT-MPC
Robot base
(b) End-effector trajectories in z-y-plane
daphare m
Ours
CT-MPC
0.1
JT-MPC
Collision
0.2
0
2
8
10
12
14
16
18
t/s
(c) Minimum distance daphere of any robot link to collision sphere
Figure 10: Comparison our proposed framework with CT-MPC and JT-MPC for turn-around task with additional obstacles. The distance dsphere is computed the minimum distance of any robot link to one the collision spheres and has to fulfill dephere > 0 m to ensure collision freedom.
23

---

find position(objer move: object, object relative: object, hint: string) -> object
Invokes the LLM with an object placing query. Finds position for object move relative to object relative specified by the hint. For example, placing a cup relative to table with the hint 'onto right' finds position on top of the table the right where the cup can be placed. The found position is saved in the return object
set-position (position: array)
Sets the position based on the adaptions from the find Sinditisii function.
get _nt_end_ue effector () -> object
Returns the current end-effector state of the robot.
empty.constraint () -> constraint
Returns constraints object where all constraints are unset.
full_con mall_conttriint () -> constraint
Returns constraints object where all constraints are set.
set.constraint (constraint)
Sets the constraint for the current segment.
H Prompts
The LLM prompts used in this work are given in this section. The planner prompt is the highest- level prompt invoked with the natural language input. The position prompt is used by the f find.position() function. The constraint prompt is invoked within the move_to () function, and the selection prompt is used by the select object () function. The selection prompt lists the objects and their center points and requires the LLM to provide an integer number of the relevant object. All other queries provide examples for the LLM, which are listed below:
Planner prompt:
import numpy as np import copy
from plan- import (move move_to, efffent-e detect, openeer close.gripper, select.object, set speed, find position)
# Query: go back to default. move_home done
# Query: save the current pose of the plant detected detected.objects = detect ((" plant "tree", "chair plant oblect- (detectedd objectss "plant") Laved_plane = copy deepcopy (plant) done
Query: the three green blocks into the left shelf sobctetsobbjets = red block", "ush stick", "green block". "shelf"])
shelf = select_.bjectt (etecclidcttttcoooojjccc "shelf", "left") for i. object in enumerate
objjects ("green green block "]):
move
close -ooiper ()
adapted_shell shelf = find poditiooo (object, relative_to=shelf hint="into")
move to (adapted )
open-gripper ()
move_home ()
24

---

done
Query: move the cup into the right shelf to the left detected_ (etect " "shelf" "plate", " "glass")) cup = deeecedddocc..........e.ccc "cup")
shelf = select_object ddbtetddo "shelf", "right") adapted-shelf find-position (cup, relative.to=shelf hint="into left")
move_to(cup)
close-gripper
# A cup has to be upright to not spill anything move to to(aappted shelf, comstraint="uprigh"") open gripper move_home done
Query: : move the ball from the table into the right shelf and back ball", "shelf", "block", "curtain "]) detected.objecte = (detected.objects ball select-object (deebjeeccddeeted...ooooo Copy the start position of the ball to be able to return to it ball start copy copy. dey.ddeeepppybbuee shelf = select object (detected objects, "shelf", "right") adapted shelf find position (ball, relative to=shelf, hint="into" move_to (ball)
"ball")
close.gripper.
move_to ashlppt
move_to(ball (balllstaat))))
open.gripper() move_home # done
Query: place the block in the right shelf onto the table detected.objects = detect ([" block", "phone", "ball", "shelf", "table"])
block = select_objer (detected_objecct _bjeccts "block", "right shelf") table select.object (detected (detected.objects. "table") adapted.table find-position (block, relative.to=table. hint="onto"))
move_to(block)
close-gripper(
move to (adapted table)
open gripper)pp
move home () done
Query: put the trash onto the floor objects (" "floor", "bin"]) for object in detected "]: "trash")
trash (ebeclobbjcctdeeetetee.d....d.d move.to(trash)
close g
tove open-gripper () move home done
Query: the highest glass slowly the bottom of the left shelf
detected. detect (I" glass", "shelf", "bowl", "umbrella" "])
25

---

glass = select object detected objects, "glass", "highest") shelf select object (detected objects "shelf", "left") adapted shelf find position (glass, relative to=shelf, hint="bottom")
(ove_to(glass) close-gripper. set-speed move_to (shpttdoaadppte__hheee
open.gripper() move.home()
# done
Query: create tower of all blue blocks the table the right side detected obbjects detect blue block", "green block", "red block", "table"]]
table selecttobject (detected "table") for i, block in enumerate detected (" block "J): (ooe_to(blook)
close-gripper
adapted table find position (block, relative to=table.
hint=f" stack height {i} right") move (adapted table) open-gripper(
move_home # done
# Query: move all books to the middle of the left shelf debected.objects = detect flowers". "plate", "book". " "shelf")) shelf select-object deteccted "shelf", "left") for book in detected objects [" ["book"]: move (book)
close gripper ()
adapted_shelf find position (book, relative mint="middle")
move_to aadapted
open-gripper ()
move_home() done
Query: move the green mug to the table and the white mug to the previous position of the green mug (c"ect(("mupp "lamp", "table"]) detected detected_objects (detettedddt table = select select_ white_mus select.object (detecieee "table")
"mug" "white") green_mug = select.object (derected objects. "mug", "green") prev-green.mug copy deepcopy (green_mug)
adapted_table find_position (white_mug. relaaive
hint="
Grasp the green mug move to (ggren_mmg)) el close gripper() # Move the green mug onto the table move adapted open open- (ven-gripper()
Move the white mug to the previous position of the green mug move_to (white_wwttt_uggg close-gripperd. move_to powettoprevv.gren..mug
26

---

open gripper ()
move home # done
Query: put the usb stick close to the flowers detected.objects
(["ect stick", "cupboard", "]o usb = (detecteee flowers = select_object deteetedoocc (etee floweessslowers find-posite (usb, relftive.to=fllwers, hint="Sem next to")
"usb stick")
"flowers")
move to (usb)
close gripper
move to (adapted flowers) open_gripper() move_home() done
Query: place the cup left to the coffee machine and return to the initial position of the cup detected_objert
"coffee machine", "table cup select object objects "cup") coffee machine select object (detected objects, "coffee machine prev cup = copy cop.. deepeoppypppp
Save the position of the cup adapted_coffee_mant -machine = find-position (cup, relative to=coffee relative_t==coffee macchin. hint="10em l)ft")
Grasp the cup move_to (cup)
close-gripper
# Move 10cm next to the coffee machine and place the cup while keeping it upright
move to adapted coffee machine, constraint="upright open gripper
Move back to the initial cup position move_to prev_cup)
done
Query: set up the spoon for my soup debected.objects bowl select.object (dieccedddtcttd.oojjeeett
(("eect("" bowl", "spoon", "knife", "fork "])
"bowl")
Fork and knife are not needed to each soup spoon = select oojetttojcttee (detected objects. "spoon ") object spoon spoon-place find aond-posiiinn (spoon, telative.to=bowe hint="10cm right") nove_to(spoon)
close (osse-gripper
move_to (sve_to(Spoon_placce)
open.gripper move_home() () done
Query: sweep all particles off the table at the left side. detected.oble objects detect particle", "table", "shelf"]) table = sellctt.ojectt (detec end end_effector
(detected objects, "table") for particle in gel.end_effecttr()
(" particle"
# The pre sweep position is to the right of the particle to be # able to push it to the left pre_sweep find-position end_effector, relative.to=partiice..
27

---

hint="3em right")
post sweep find position (end effector, relative to=table, hint="left side")
Move the pre sweep position, stopping there is not # necessary
move_to(pre_swe
stop=False)
# During the sweep the constraint the movement to be exact to # follow straight line to the post sweep position move_to post.sweep.. constraint="exace exxectt))
move_home()
# done
Query: push close the topmost drawer detected, detected_objects = detect drawer", dl"awerr"" "table", "shelf"]) dopmost_drawer select-object "topmost")
dejecctd delect..ojeeccddeeccdd.....ccc
"drawer",
end_effector get.end_effector(
close- find (ffe_tofc,
relative relative_octtopoot..rr.e.
hint="30cm push")
move to (topmost drawer. stop=False)
move to ( (close drawer, constraint="exact") move home done
# Query: wipe the blackboard from left to right detected detect sponge", setonet""",ngee", "blackboard", "shelf"]) sponge = select-object (eeet_oojecc(deteteee..oom objjectteddobjetts blackboard = select.object (detected pre-wipe find-position eessitiionssooge... (sponge, relative.t==blacbborrd hint="left side")
"sponge") "blackboard")
post wipe find position (sponge, relative to=blackboard. hint="right side")
# Pick up the sponge move_to (sponge) close gespperi
Wipe the blackboard move_to prre_wipe, stop=False)
move_to post_wipe, constraint="exact") move_home () done
Position prompt:
import numpy as np
from adips import set.position
Given data:
-hint: hint for the desired position on goal object / string
Query: "cup" to "right shelf", hint: "onto"
cup objects.movable shelf = objects goal
(max.x, max-y, max.z) shelf. max max bounds
Onto means the maximum plus half the size of the object shelf center [0]
shelf center [1]
max_z cup. size [2]/2
sct-position ([x, y. z))
28

---

done
Query: "cube" to "shelf", hint: "into" movable
cube e shelf = objects. goal Into means we use the center position set-position (shelf.. center)
done
Query: "book" to "table", hint: "top" book objects movable
table = objects goal 2) table. max bounds
(max x, max
x = table center [0]
y table center [1]
= max_z + book. size [2]/2
set-position ([x, y, 2])
done
Query: "mug" to "flowers", hint: "10cm left"
flowers objects goal max max y, max 7.) = flower. max bounds
Add 10cm the left border of the flower due to the " "10em left hint
flowers center [0]
= max-y 0.1
flowers. center [2] set-po eetsiiitoo ([x, y. z)) done
Query: "block" to "floor", hint: "stack height 1" First stack height means add only the size of the block in z direction block objects movable
floor = objects.
floor [0]
= conterr
floor max_dounds [2] block. size [2]/2
sct-position ([x, y. z])
Query: "block" to "floor", hint: "stack height 2" Second stack height means we add twice direction block = objects movable
floor = objects.gom
floor. center[0]
y floor center
floor. max_bounde [2] 1.5 block, size [2]
set_position ([x, y. z]) done
# Query: "picture" to "mirror", hint: "in front"
mirror = objects.. goal
mirror center [0] 0.1
y = mirror center [1]
mirror center [2]
set position ([x, y. z))
done
Query: "glass" to "table", hint: "onto"
29

---

glass = objects... movable
table objects goal
(max x, max y, max 7.) = table max bounds
Center [0]
center
z = max-z + glass. size [2]/2
sct-position ([x, y. 2])
done
Query: "phone" to "box", hint: "onto left" phone objects movable
box = objects goal
(max x, max y, (min_x, min_y, min_z) = box. min_ bounds # Onto means that need choose the max_z Left on the box is the maximum in y-direction box. center
2) box. max bounds
y = max-y
max-z glass. size [2]/2 set pos-iiioo ([x, y, z))
done
Query: "stamp" " to "table", hint: "lower left" box = objects movabee
table = objects. goal
(min_x. min-y. min-z) = table mon_dounds
(max_x, max_y, max.z) = table max max_dounds
# The lower left corner of the tabletop is at maximum and the minium x position min_x
y = max
box. size [2]/2 set position ([x, y, zl)
done
Query: "usb" "cuuboardd,, hint: "right" usb = objects. movable cupboard objects goal (min_x, min-y, min-z) = cupboard.. bounds (max_x, max-y, max-z) cupboard, max bounds The right of the cupboard refers to the minimum cupboard. center [0]
min-y
z max_z usb. size [2]/2 pet-pooitoon ([x, y. 2))
done
Query: "pen" to "paper", hint: "10cm right" pen objects movable paper = objects goal
(min min min
paper. min bounds
10cm right of the paper refers to the minimum y of the paper # minus 10cm
x = cupboard center [0]
y min_y 0.1
z = max Z + usb size [2]/2 set_position ([x, y, z]) done
30

---

Query: "end effector" to "drawer", hint: "10cm push" end effector objects movable drawer = objects goal normal drawer normal
10cm push means moving along the negative normal for 10cm push-pos = end_effector. center 0.1 * normal set-position (push-pos) done
Constraint prompt:
from constraint_lib import compty_constraint, full_constraint, sel.constraint)
Query: "no constraint"
constraint empty empty-constraie (onstraint ()
set.constraint (ooosstrin((constaannn) (constraint) done
Query: "exact"
Exact means that everything is constrained constraint full_constraint
set_constraint (consttranttconnsnnnnn (constraint) done
Query: "upright"
constraint ( Upright means there should be no rotation around the or y axis constraint.rotatiiiii constraint rotation y = True set_constraint (constraint) done
True
Query: "horizontal plane" constraint -optyyconstraint() The x-y-plane is the horizontal plane constraint position.x True constraint position = True set constraint constraintcoonstrainn) done
Query: "z rotation"
constraint empty eapty- ccnntrrinnt constraint, rotatiinn... = True ceconssrainntcoonsttiiii done
31


Reviews:
[{"review_id": "DWAT2iEYAV", "reviewer": "Reviewer_fPqJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a modular motion planning framework, consisting of an LLM to generate Python code for the motion task completion, a vision module, a path planner to compute smooth paths of the robot, and a MPC controller to execute the path. For the LLM, vision module and MPC controller, existing approaches are used. The main novely lies in the path planner. It computes collision-free convex sets and connect them via a graph. The path for the robot to follow is then computed based on the shortest path in the graph. The motion planning framework is evaluated using simulated and physical manipulation tasks.", "summary_of_recommendation": "In summary, while the paper contains some interesting ideas, more discussion on the novelty of the path planner must be provided, including an ablation study with existing planners.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- I'm surprised that a piecewise linear reference path is computed, since the main motivation of the planning and control modules are to generate smooth trajectories. Why not use B-splines or other curves for the reference path? \n- Why does the framework have such a low success rate in the TeaCup problem?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8x9WZ3sl9j", "reviewer": "Reviewer_p2nF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper appears to be doing language-guided motion planning by combining LLMs and MPC.", "summary_of_recommendation": "I chose weak accept because I do not understand the novelty of their MPC method. But the combination of MPC with LLMs is still very compelling.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "* Why not show an example trajectory in the main overview figure?\n* Can you motivate the use of GCS more?\n* Can you discuss the novelty of your BoundMPC algorithm a bit more? It seems like you are using GCS, but only enforcing collision constraints for the end-effector. Is this because GCS isn’t actually real-time? Furthermore, I do not believe that the linear interpolation is novel. Isn’t this how TrajOpt works?\n* Is it possible for you to do evaluate the performance of your method on more challenging scenes? How many obstacles can your method handle?\n* How do you think your method would work for tasks requiring longer sequences?\n* Could the authors explain the various pieces of their method a little better. They say BoundMPC runs at 10Hz and that their path planner takes 0.15s. Is GCS generating the path and BoundMPC tracking it? Is GCS running online or offline? BoundMPC is only generating torques at 10Hz? This sounds very slow for torque control.\n* The grasp detection should be discussed more.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KKKuGULAU", "reviewer": "Reviewer_1wvP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The presented planning framework tackles task and motion planning by using an LLM for discrete task planning and for motion planning first building a set of of convex sets representing empty space, connects these sets via a graph, and plans over this graph.", "summary_of_recommendation": "It is hard to determine the novelty of the proposed modular task and motion planning framework without significantly more simulation and hardware experiments in more complex scenarios.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 1, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "1. A more thorough literature review on safety-based planning methods as well as on task and motion planning (TAMP).\n\n2. More simulation results are required in order to evaluate the novelty of the modular framework, as it is challenging to determine the novelty of each of the individual elements of the proposed method.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a planning framework that addresses task and motion planning (TAMP) by utilizing a Large Language Model (LLM) for discrete task planning and a graph of convex sets (GCS) for motion planning.

Before the rebuttal, the major concern among reviewers is the lack of novelty. The reviewers noted that while the components of the framework are existing work, the innovation may lie in their combination and significantly more simulation and hardware experiments in more complex scenarios are needed to justify the advancement enabled by this combination.

Other concerns include confusion regarding the terminologies and equations used, unclear figures, an emphasis on safety that only considers the end-effector, the lack of an ablation study with other existing planners, and an insufficient literature review on TAMP methods.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns. As a result, all reviewers now rate the paper as a Weak Accept. While some reviewers still have concerns regarding the novelty of the paper, they acknowledge the clear motivation behind the work and the hardware experiments that demonstrate the proposed framework's effectiveness.
Decision: Accept