Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skills
Authors: Tianhao Wei
Liqian Ma
Rui Chen
Weiye Zhao
Changliu Liu
Venue: CoRL 2024 (2024)
Keywords: embodied agent
model-based control
LLM
manipulation
TL;DR: a meta-control system can be built to automate the thought process that human experts use to customize control systems for heterogeneous robot skills.

Abstract: The requirements for real-world manipulation tasks are diverse and often conflicting; some tasks require precise motion while others require force compliance; some tasks require avoidance of certain regions while others require convergence to certain states. Satisfying these varied requirements with a fixed state-action representation and control strategy is challenging, impeding the development of a universal robotic foundation model. In this work, we propose Meta-Control, the first LLM-enabled automatic control synthesis approach that creates customized state representations and control strategies tailored to specific tasks. Our core insight is that a meta-control system can be built to automate the thought process that human experts use to design control systems. Specifically, human experts heavily use a model-based, hierarchical (from abstract to concrete) thought model, then compose various dynamic models and controllers together to form a control system. Meta-Control mimics the thought model and harnesses LLM's extensive control knowledge with Socrates' "art of midwifery" to automate the thought process. Meta-Control stands out for its fully model-based nature, allowing rigorous analysis, generalizability, robustness, efficient parameter tuning, and reliable real-time execution.

Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skills
Tianhao Wei Liqian Mat.1.2, Rui Chen', Weiye Zhao', Changliu Liu
Equal contribution, Carnegie Mellon University, Tsinghua University
meta- caa-ronntolppppppe paa.lo..pppr.ttttteeee github.
Abstract: The requirements for real-world manipulation tasks are diverse and often conflicting; some tasks require precise motion while others require force compliance; some tasks require avoidance of certain regions, while others require convergence to certain states, Satisfying these varied requirements with fixed state-action representation and control strategy is challenging, impeding the de- velopment of universal robotic foundation model. In this work, we propose Meta-Control. the first LLM-enabled automatic control synthesis approach that creates customized state representations and control strategies tailored to specific tasks. Our core insight is that meta-control system can be built to automate the thought process that human experts human experts heavily use model-based hierarchical (from abstract to concrete) thought model, then compose various dynamic models and controllers together to form a control system. Meta-Control mimics the thought model and harnesses LLM's extensive control knowledge with Socrates' "art midwifery" to automate the thought process. Meta-Control stands out for its fully model-based nature, al- lowing rigorous analysis, generalizability. robustness, efficient parameter tuning. and reliable real-time execution.
design control systems. Specifically,
Keywords: embodied agent, model-based control, LLM, manipulation
Conflieting Requicamnnts
pricia TEIVEENE
lesteesgenooss Trsks
belascing
explanuaie
fequracy
Castumiring
parcppiioo
Representatoos
Control Strategies
Hypa-pammeier
Codred siffress
dicurce
seit lrpua
tail
armiilee
Task Spece
pealike
abjec
ampic
EOR for
wablle sipt
trading spone
Tricking Space
plyper supporrts join
oiget joiee cusu, disans sbstado frali froguasy git-oeeed oit tiher
sarpn Igleid jos. firez coniniller
vacking convette
teeling
macal-taad truking corriilee
Inseiresl
conyllance cvammo
Figure 1: Opening the door, moving safely, and balancing heterogeneous skills and have conflicting quirements that be difficult satisfy with fixed control strategy and fixed state-action representation. Meta-Control addresses challenge with a composable hierarchical control formulation and LLM,, enabling automatic model-based control synthesis. The synthesized skill has customized state action representations, dynamic models, and controllers that perfectly align with the objectives and requirements.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
From control (for trajectory tracking) to learning (for trajectory generation), robotic systems have gained higher autonomy over the years. However, human designers still play an indispensable role in the creation and tuning of these control and learning algorithms in case-by-case manner, which prevents the realization of general purposs robots. For robots better serve and assist human users in variety of tasks, there is pressing need for higher autonomy that allows robots to autonomously synthesize skills S based on language instructions & to any given task, without manual tuning. Some recent attempts have been made towards this goal using LLM. Such as automatically selecting skill S; from a predefined skill library {Si, S
.... } [1. 2, 3). leveraging generic parameterized policies Sg. where 0 := LLM (?) [4, 5, 6), and reinforcement learning (RL) with LLM generated rewards: == (L)LLM(() 17, 8, 9]. However, the journey towards general-purpose purpose robots, capable of adapting arbitrary tasks without manual tuning, remains incomplete.
A major challenge for general-purpose robots is that real-world tasks have inherently diverse and often conflicting requirements that are difficult to satisfy with a fixed control strategy, For example, as illustrated in fig. I, moving safely requires precise movements for collision avoidance, necessi- tating safe position control. Conversely, position control is unsuitable for opening the door due to the difficulty of planning a trajectory that perfectly aligns with the door's swing path, making force compliance control more favorable. Similarly, balancing demands controllers that ensure the pole's convergence (a position-atttracing goal), which opposes collision avoidance (a position-avoidde goal). Although visuomotor methods can potentially address heterogeneous tasks, they often lack explainability and reliability. Predefined action primitives may cover different task types but require manual construction and have limited applicability and scalability. These challenges restrict existing methods from generalizing to various open- world manipulation tasks with varying constraints.
To advance towards caneral-purposs robot, propose Meta-Control, novel framework F that automatically synthesizes model-based robot skills from instructions for diverse open-world tasks: S = F(E). Our core insight is that higher level autonomy can be achieved by creating a meta- control system to automate the thought process that human experts use to design control systems, with the help of LLMs. More specifically, human experts design different represettion and control strategies for different tasks following madel-based, hierarchical (e.g., from abstract to concrete) thought model, and then compose various dynamic models and controllers into an integrated control system. By mimicking the thought model and the thought process, the autonomous system, Meta- Control, can gain greater coverage in robotics problems with good performance.
Based on these insights, we formulate robot skill design as hierarchical control system synthe- sis problem and leverage LLM to customize representations and control strategies. The hierarchy involves a high-level task space and a low-level tracking space. The task space is an abstracted, intuitive space for accomplishing the task (e.g.. Cartesian space or gripper pose space) while the tracking space usually represents the robot state space where low-level constraints can be specified and task-level commands are followed. Meta-Contur first let the LLM determine the task and track- ing spaces, then design the corresponding dynamic models and controllers utilizing predefined but abstract templates. The abstract model/controller template grounds the behavior of the synthesized system while maintaining high flcxibility. We design set of guidance and checklist of error in the prompt to help the L.LM avoid potential errors and produce the correct code, just as how Socrate helps people draw their own conclusions.
Meta-Control offers several benefits: 1. It enables the synthesis of challenging heterogeneous robotic skills for unseen tasks, allowing each task to be accomplished with the most suitable rep- resentation and control strategy tailored to task-specific requirements. 2. Unlike previous work that primarily utilizes spatial priors from LLMs (c.g., object localization, affordance), Meta-Control leverages the internalized control knowledge of L.LMs which encompasses decades of modeling and optimization efforts for various tasks and skills. 3. The synthesized control system is fully model- based which brings robustness, generalizability. efficient parameter tuning. and formal guarantees (c.g., safety and stability), leading to reliable and trustworthy execution.

---

Clean the whitebourd
Open the cabinet
Meta
Control
suthesiial
Tinjectory based controller
Figure 2: Comparison between Meta-Control and trajectory planning-base method on real robot for wiping the board and opening the cabinet. The trajectory-based method fails to erase the mark because it neglects force requirements. Opening the cabinet with trajectory- planned trajectory does not precisely align with the door's swing path, which may damage the door cabinet fixed. In contrast, Meta-Control addressed these challenges with properly customized control systems.
controller leads cabinet displacement because the
2
Related work
We review four closely related approaches to achieve higher autonomy: skill libraries, optimization, end-to-end methods, and model-based control synthesis. Meta-Control integrates aspects these approaches, offering a flexible, robust, and explainable solution for diverse real-world tasks.
Skill libraries with LLM methods utilize predefined libraries of motion primitives, enabling diverse control strategies through high-level APIs. LLMs can dynamically combine these libraries for task execution [1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. Skill libraries are often constructed using behavior cloning, reinforeement learning, or bootstrapping [5, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]. However, these motion primitives are manually constructed and task-specific. In contrast, Meta- Control is capable of synthesizing new heterogeneous skills on the fly.
Optimization-basee methods use fixed optimization framework such MPC RL with mutat- ing crper-parameters. including objectives [30, 31, 32, 33, 7, 8]. constraints [34, 4, 35], or dynamic models [36, 37, 38, 39, 40, 41]. These typer-parameters can be inferred by LLMs [4, 33, 7, 8] or learned from data [36, 30, 31J. Hierarchical optimization often use LLM inferred high-level com- mands for an instruction instruction-oolloiing controller [42, 29]. A key limitation is that the chosen framework constrains the method's capability. An MPC generating end-effector actions cannot produce force- compliant control in joint space, and RL-based methods are unsuitable for online skill synthesis. Meta-Control, however, dynamically generates flexible policies.
End-to-end models directly map perceptions and instructions to robot actions, using VLMs [6, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52] or energy functions and affordance maps [53, 54, 55, 56). Diffusion models have also been used to learn from demonstration [57, 58, 20]. enabling multimodal action distribution. End-to-end methods often output Cartesian actions, requiring additional motion planning for joint-level movement, and they are data-hungry. Furthermore, these methods lack robustness and explainability. Meta-Control is data-efficient, integrating LLM with model-basee control strategies ensure flexibility, robustness, and explainability.
Model-based control synthesis designs explainable systems with rigorous guarantees [59, 60, 61, 62, 63, 64, 65). Hierarchical formulations simplify control synthesis for complex systems [62, 66, 67, 68, 69, 70, 71]. However, model-based synthesis is often task-specific and lacks generalizability. Meta-Control overcomes this limitation with the help of LLM.
3 Method
In this work, we focus on synthesizing robot skills, defined as unit actions (e.g., grab the eraser, erase the marks) from robot tasks instructed via free-form language L (e.g., clean the whiteboard) [4]. We assume that the decomposition from task to skills: L e,, l2....... ......,,, ln is given by a task-level
3

---

Inper
The aned sakable nooles contella
Key persantes sael aadrio st... optiiii yet
leice the suted Lie
Lavel
Data Flow Level
Parameter
Generic
Reflection Hece tne
Reflerike Foce La the
Few-dha Tariing
Prosupta
Dores Jeewey, pleese
Freet cineersers foe Iommleeltery tutlowinn
bay peranecere and
Fece La
with /Placoholdary/
poapchCa desiner
Fresizle cacoee
Fonaur labile
F-ease redice
LRenes parlatew Juy.
Code Exisector and Evaluator
Estract Evalusic
Extret & Evilunk
Exteast & Eveiantic
(oantrollec)
2D amr
(neecde)
decred
Metn-Coatter Centrol
pesoppios
tock ieger converier
mndking toyut cosvetir
medil
conmriler
controllir
Syothesized Control
Figure 3: Overview of Meta-Control The user only needs to provide a skill description, Meta-Control then leverages the control knowledge of LLMs to synthesize skills through a three-level pipeline: strategy level, data flow level, and parameter level. For each level, we have designed a generic prompt with placeholders, which dynamically replaced with input code extracted from the LLM response during runtime, utilizing control system.
code extractor make the prompt task-specific. The extracted code is also used to construct At cach level, if the LLM- generated code results an error, reflection phase is initiated. We have embedded design principles and checklists of common errors within the design and reflection prompts to assist the LLM in producing correct code. The generic prompt design with placeholders allows Meta-Control generalize unseen tasks without modification.
planner, which be LLM-bbsse search-based [72]. Our focus is to synthesize a control system to perform skill described by l.
Designing control systems from scratch for open-world skills is very difficult even for humans be- cause the design space is infinite. Mimicking the thought model control experts, we propose a composable hierarchical control formulation and introduce the Meta-Control pipeline.
3.1 Meta-Control Formulation: Composable Hierarchical Control
Hierarchical Control The bilevel hierarchical control formulation can represent various skills through a high-level controller in an intuitive task space and low-level controller that tracks the high-level control in the robot state space. Formally, we define
State space: == f(x u)
Measurement space:
==
Task space: 1 = h(2 v)
where æ is the system state, u the state space control input, y is the output or measurement of the system, is the state of the task space, and is the task space control input. The task space contains intuitive and high-level states, such the gripper poses for robot arms and the center-of-mass for quadrupeds. x and may not be directly observable, but has to be measurable or extractable from perception. f. h are dynamic models of the corresponding system. We assume g. the perception algorithms, is given this work, but could also be designed. The task space can sometimes be omitted when it coincides with the state space, such as when a robot arm skill directly specifies joint goals. We denote the task space controller by T. (y), and the tracking space controller by म. (y: v).
Our goal is to use LLM to 1) infer the objectives and constraints of the task; 2) design the proper task state 3) design the dynamic models h2z,, v) and fxr u); 4) design the task and tracking controller Ty)y), Tu Tu(y, v) to achieve the desired performance.
Composable Design Designing z, h, f. n., and n. directly is still very challenging due to infinite possible spaces and dynamic models and the need for an accurate and deep understanding of the robotic system. Similar humans, LLMs excel in making intuitive decisions, but relatively poor extensive reasoning, large-scale design, and implementation [73]. Furthermore, LLM gen-

---

erated code cannot guarantee constraint satisfaction. Therefore, we introduce model and controller templates, which enable composable design and guaranteed constraint satisfaction.
Templates are predefined object classes that need to be instantiated with concrete arguments. For example, offer a dynamic model template called LinearModel which can be instanti- ated by passing in four matrices A, B, C, D describing linear system, and a controller template LQRController requires matrices Q, R and vectors Zo. N0- Templates differ from motion prim- itives in that they are abstract and generic. The usage of templates grounds the behavior of the system, greatly increasing explanability while maintaining flexibility.
3.2 Meta-Control Pipeline: Generic Prompts with Multi-Level Synthesis
As illustrated in fig. 3, we designed a 3-Jevel pipeline for Meta-Control (Meta-Control) to generate hierarchical control systems using generic prompts that adapt unseen tasks without modification. The user provides a task description, and Meta-Control interacts with an LLM in a Socratic manner to synthesize the control system. A detailed query and response can be found in appendix C.
Art of Midwifery LLMs often struggle to write fully functional code without errors. To guide it, we implemented a method similar to Socratic questioning, where instead of directly giving solutions, we prompt the LLM to "give birth" to correct code through structured guidance. Meta-Control inter- acts with the LLM through three dialogues, each representing a level of the pipeline, and after each, it generates summary that informs the next step. These dialogues follow structured template, with task-specific placeholders, filled in by user input or code extracted from the LLM's responses using a code extractor. This process ensures the pipeline can adapt to unseen tasks, simply by pro- viding task descriptions. Each dialogue follows specific design principles, such as defining control spaces based on available mcasurements, designing the task controller with objectives in mind, and aligning the tracking controller with task requirements. If the generated code encounters errors, a reflection phase, depicted in fig. 3., is triggered. This phase uses a checklist of common LLM errors and related error messages to correct them. For example, Drake's [74] unusual rotation- -tatiinntranslation order is common source of errors. A detailed breakdown of prompt design techniques is provided in appendix B.
Strategy Level: Design via Template Meta-Control first queries the L.LM to 1) identify the appro- priate task space, objectives, and constraints; 2) selects the models and controllers from the template library, and 3) instantiate them with the correct parameters based on the task's requirements. The process follows logical sequence: defining the task controller, selecting the task model, and then designing the tracking model and controller. The LLM's responses follow a strict format, allowing us to extract the generated code and evaluate it.
Data Flow Level: Composition via Semantic Understanding emplate inputs and outputs often differ format and semantic meaning, so the LLM must align these interfaces to ensure unimpeded data flow. Meta-Contro queries the LLM to implement two objects: 1) task input converter that composes the task controller's ' input from available measurements y; 2) tracking input converter that composes the tracking controller's 's input from and the task control v. F In the balance cart-pole example, the task input converter extracts the 4D input for the LQRControlles from 20+ available measurements. It then passes the task control (a scalar force) to the tracking input converter, which prepares the inputs for the Hybri t i conoorceController. The tracking input converter has to pad the task control and combine with available measurements to form a 12D input. A checklist of common errors helps correct interface mismatches, such dimensionality or data type issues.
Parameter Level: Alignment via Few-Shot Optimization The designed system may degenerate when deployed because the inferred parameters may not align the reality. To address this, Meta- Control enables few-shot optimization of system parameters by querying the LLM on 1) key param- eters affecting behavior, 2) performance metrics, and 3) strategies for adjusting parameters based on past trajectories. For instance, in the cart-pole task, the LLM identifies the LQR controller's Q and R parameters as crucial, using the cart's position and pole's 's angle as metrics. The process is efficient because LLMs have internalized gurameter-performee relationships in common dynamic
5

---

F22
1-3
Balance cart pole with robot arm
Pick and place with collision avoidance
E-1
E=2
Open the deor
Push the button
Open the drawer
Figure 4: Five manipulation tasks that have inherently different challenges and requirements. For example, the balance task requires accurate and high-frequency feedback controller. The safe pick and place task requires guarantecing collision avoidance for the whole robot The open door task requires properly handling articulated objects; and the executed trajectory has to perfectly match the swing path.
Before tuning
After paramoter tuming
pas
Pole
Cart pm
Centrol
0.3
Pole
155
Pile
Contrel
1.5
0.8
1.3
13
34
2.0
3.0
Time 60
Time
Figure 5: Meta-Control can automatically identify hyper-parameters that require tuning and tune them to ac- complish challenging tasks. The figure shows the trajectory of the arm-held cart-pole system before and after tuning the synthesized controller. The hyper-parameter Q -diag(10, 1, 100, 1), R = 0.01 are chosen and tuned by the LLM with only 2 rounds of trial-and-eroe
systems. A list of possible root causes (e.g., initialization errors or suboptimal parameters) of the degenerated performance helps the LLM diagnose problems more effectively.
Experiment
The experiment is designed to manifest the following features. We first show that Meta-Control enables synthesizing skills for heterogeneous requirements by exploiting the control knowledge of LLM. Then we did an ablation study show that all three levels of the Meta-Control pipeline im- prove the success rate; Finally, we reveal the benefit of the model-based design, including robustness, generalizability. and formal guarantees.
Most of the experiments are conducted on four representative tasks. These tasks cover constraints that appear frequently in the real world, such as constraints on position, velocity, force, compliance, convergence, and safety. We implement our pipeline with Drake[74), a framework designed for model-based control. For the hardware experiment, we used a Kinova Gen3 robot arm. The language model GPT 4.0 with default temperature 1.0.
4.1 Meta control enables synthesizing challenging skills
As shown in fig. and fig. Meta-Control. successfully synthesizes controllers for various challeng- ing tasks with inherently different requirements, both in simulation and in the real world. Figure shows that Meta-Controi can identify core metrics and critical parameters that affect performance, then efficiently and effectively tune the parameters based on the metric through trial and error. A de- tailed description of task challenges and synthesized control systems can be found in appendix A. The complete skill synthesis process can be found in appendix C.
Meta-Control exploits dynamics priors In contrast previous work that focuses more spa- tial relationship priors [1, 5], Meta-Control exploits the LLM's internalized knowledge of dy- namics, enabling rigorous synthesis of high-performance controllers. In the balance cart pole

---

Method
API
API Hierarchy
API Template
Meta-Control
design
30%
90%
60%
100%
Balance
implementation
0%
30%
20%
90%
execution
0%
0%
70%
design
40%
50%
60%
100%
Open door
implementation
10%
20%
10%
100%
execution
0%
0%
0%
80%
design
0%
0%
40%
90%
Safe Pick&place
implementation
0%
0%
0%
90%
execution 0%
0%
0%
90%
Table 1:
The ablation study shows that hierarchical formulation (Hierarchy) and templates synthesis (Templates) improves the success rate, compared to directly synthesis with only the low-level API. API Template can be viewed as a generalized motion-
motiion-rimitivv primitive method.
task, LLM designs the task space dynamics h(z, v) with linear approximation in the form of Az + Bu around the upright position of the pole, where A and B are correctly given by the
LLM: A 1, 0, 0]: [o, O, "pplel 00: [0, O, 1]; [0, O, (
ApaetMen ..]]. B [0; miai 0 - 4pae 4pae where m denotes mass, 1 denotes length, g is the acceleration gravity. Detailed description of the synthesized control system can be found appendix A.1.
mast
4.2 Ablation study of the Meta Control pipeline
To demonstrate the necessity of the hierarchy formulation and simplification using templates, show the success rate of control system synthesis with different ways to query the L.M.M Meta- Control use the query: "output is 7-DoF torque, input is (API) + "consider task space and a tracking space..." (Hierarchy) + "use available templates" (Template). We test the success rate in 3 stages: design (via templates), composition (interface alignment), and execution (after few- shot optimization). We say that a design is successful the LLM LLM-designed control system has the potential perform the skill judged by an expert. We say that the implementation is successful if the system
without errors. We say that the execution successful if the control system finishes the skill desired. We repeat each task 10 times to compute the success rate. Randomness is caused by the LLM. As shown in table 1, we can see that the success rates of all steps in the baseline are lower than Meta-Control. Although LLM gives reasonable architectures to finish the task in the Balance and Open Door task, the LLM fails to provide correct code and parameters to realize the control system due to the complexity of the system and the huge design space. With all the modules, we achieve the highest success rate for all tasks. The Safe Pick&Place task is especially difficult for the baseline because the baseline methods, even though prompted to avoid collision, unable to successfully design a controller that can avoid collision continuously.
4.3 Meta-Control brings benefits of model-based design
The model-based nature of the synthesized controller brings variety of benefits, such as robustness, explainability, generalizability, and rigorous analysis.
Generalization to different attributes/states Meta-Control synthesized control systems can eas- ily generalize to scenarios of different attributes/states due to the model-based nature. Given successfully synthesized control system, test different attributes/states and calculate the success rate. The range of change and the results are shown in table 2. Examples are shown in fig. 6. Meta-Control achieved 100% success rate for all scenarios.
Rigorous Formal Analysis The madel-based design allows rigorous formal analysis for a variety of properties, making autonomous certification by LLM possible. For example, we can guarantee the convergence for the cart-pole stask by solving the Riccati equation, and ensure collision avoidance for the pick-and-place task by certifying safety index. Details can be found in appendix A.3
7

---

Balance
Open Door
Safe Pick&PPacce
Pole Mass
Handle Height 0.3r~0.75
Obstacle Position
10/10
10/10
10/10
0.01~0.5 kg
0.01~0.3 (y-axis)
Carl Mass
Handle Radius 0,3-~0,7 m
Obstacle Size
10/10
10/10
10/10
0.05~0.5 kg
0.1~00.45 m (height)
Initial Angle -0.3~-0.5 rad
Door Mass
Place Position
10/10
10/10
10/10
1~~30 kg
-0.3~0.3 (X-axis)
Table The synthesized controllers easily generalize scenarios with different object states/attributes. The left column lists range the parameters each scenario. The right column indicates the success count of 10 trials for each set of parameters.
intial arigie -0.5 rad
fundle rediza 0.8m
tall coadiec
Figure 7: Meta-Controd generalizes different bodiments because the synthesis fully model- based. Furthermore, a controller synthesized on Kinova can transfer to Franka Pranda simply by replacing the robot dynamic model thanks to the model-based nature.
intiaal anggle
shant
Figure 6: Meta-Control synthesized control systems robust attribute/statte changes because of the model-based design.
Transfer real robot and different embodiments The control system for opening door is syn- thesized in simulation and executed both in simulation and in the real world. As shown in fig. and fig. 4, the behavior consistent and no sim-to-real gap is observed because the synthesized con- troller is model-based, closed-loop. highly explainable, and math-certified. The controller can also generalize to different embodiments with the same low-level API (e.g. 7 DoF joint torque) given the model of the new embodiments as shown in fig.
4.4 Failure analysis
We analyze the failure cases in table and summarize their reasons. Mathematical error: Al- though LLM can give a mathematical description of the approximate dynamic model for an unseen system, it make mistakes in math. For example, for the dynamic model synthesized for the cart-pole system, it can miss term in the A matrix, mess up signs (use when is desired). 2. Failure to follow instructions: We require the LLM to provide a structured response so that program can extract the code and plug it into the robotic system. However, sometimes LLM fails to follow the instructions, leading to responses in the wrong format. 3. Incorrect reasoning: In the open-door task, the LLM infers the target location of the door knob. However, the LLM may infer wrong target given the environmental information. Although these are still challenging for LLMs, believe that they can be overcome with the rapid development of LLMs in the near future.
5 Limitation and Discussion
In this work, we present Meta-Control, a novel framework for automatic model-based control sys- tem synthesis using LLMs, designed for heterogeneous robotic tasks. Simulations and real-world tests demonstrate its potential to enhance robotic autonomy. However, Meta-Control has limitations, such as reliance on accurate system state estimation and predefined models, which limit adaptabil- ity to new tasks. The LLM can also make suboptimal decisions, like selecting PID over LQR, complicating parameter tuning. Future improvements include better reflection across the pipeline, stronger reasoning capabilities, faster synthesis, and the integration of learning- baased templates and automatic perception selection.

---

Acknowledgement
This work was supported by the National Science Foundation under Grant No. 2144489.
References
[1] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[2] W. Huang. P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118 9147. PMLR, 2022.
[3] A. Zeng, M. Attarian, Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00550 2022.
[4] W. Huang, Wang. Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprini arXiv:2307.05973, 2023.
[5] A. Brohan, Y. Chebotar, C. Finn, Hausman, Herzog, D. Ho, J. Ibarz, A. Irpan, Jang, R. Julian, al. Do can, not say: Grounding language in robotic affordances. In Conference Robot Learning, pages 287-318. PMLR, 2023.
[6] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.088 2023.
[7] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306. 280477776700067 2023.
[8] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D) Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Mxxiv-2310.12931, 2023.
[9] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309.1 11489, 2023.
[10] Y. Wang, Z. Xian, F. Chen, T.HH. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation, 2023.
[11] Zhang, J. Zhang, K. Pertsch, Z. Liu, X. Ren, M. Chang, S.-H. Sun, and Lim. Bootstrap your own skills: Learning solve new tasks with large language model guidance. In 7th Annual Conference Robot Learning, 2023. URL https: htttps:/opeerevieee http:://pppmeeeeeeeeee/orrrriiie nwt.:/rumffi aOmFRgadGO.
[12] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
[13] Huang, Z. Jiang, H. Dong, Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint 2305 2023.

---

[14] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-
11530. IEEE, 2023.
[15] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Au- tonomous Robots, 47 2023.
[16] Y. Hu, F. Lin, T. Zhang, L. Yi, and Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842 2023.
[17] Joublin, A. Ceravola, Smirnov, F. Ocker, J. Deigmoeller, A. Belardinelli, C. Wang, Hasler, D. Tanneberg, and M. Gienger. Copal: Corrective planning of robot actions with large language models. arXiv preprint arXiv:2310.07261. 2023.
[18] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 International Conference on Robotics and Automation (ICRA). IEEE, 2024.
[19] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.1213: 2023.
[20] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Proceedings of the 2023 Conference on Robot Learning. 2023.
[21] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03374 2023.
[22] Y. Guo, Y.-J. Wang, L. Zha, Z. Jiang, and J. Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. arXiv preprint arXiv:2307.002022222 2023.
[23] C. Jin, W. Tan, J. Yang, B. Liu, R. Song. L. Wang, and Fu. Alphablock: Embodied finetuning for vision-language reasoning in robot manipulation. arXiv preprint arXiv:2305. S...::::205./8889. 2023.
[24] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, et al. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances Neural Information Processing Systems, 36, 2024.
[25] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0500 055608 2022.
[26] Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma. and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open- open-world multi-task agents. arXiv preprint arXiv:2302.015600 2023.
[27] Luketina, Nardelli, G. Farquhar, Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. survey of reinforcement learning informed by natural language. arXiv preprint arXiv: (e... 03926, 2019.
[28] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. In International conference on machine learning, pages 166-175. PMLR, 2017.
[29] Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn. Language abstraction for hierarchical deep reinforcement learning, Advances in Neural Information Processing Systems, 32, 2019.
10

---

[30] C. Finn, S. Levine, and P. Abbeel. Guided cost learning; Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[31] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint 1rXiv:1710.122444 2017.
[32] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter. Differentiable mpe for end-to-end planning and control. Advances in neural information processing systems, 31, 2018.
[33] Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, An- dreas, and D. Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.051 186, 2022.
[34] D. Driess, O. Oguz, J.-S. Ha, and M. Toussaint. Deep visual heuristics: Learning feasibility of mixed-inte integer programs for manipulation planning. In 2020 IEEE international conference on robotics and automation (ICRA), pages 9563-9569, IEEE, 2020.
[35] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08245 2024.
[36] I. Lenz, R. A. Knepper, and A. Saxena. Deepmpe: Learning deep latent features for model predictive control. In Robolics: Science and Systems, volume 10, page 25. Rome, Italy, 2015.
[37] L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning- model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269-2966 2020.
[38] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenhaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00331, 2016.
[39] Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, al. Interaction networks for learning about objects, relations and physics. Advances neural information processing systems, 29, 2016.
[40] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[41] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.0156: 2018.
[42] Y. Tang, W. Yu, J. Tan, H. Zen, A. Faust, and T. Harada. Saytap: Language to quadrupedal locomotion. 2023. URL /aattps:/sayye
github. io. /taps://saytap. github. io.
[43] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068177 2022.
[44] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-attion models transfer web knowledge to robotic control. arXiv preprint 2rXiv:2307.15878, 2023.
[45] A. Bucker, I.. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and R. Bonatti. Latte: Language trajectory transformer. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 7287-7294. IEEE, 2023.
[46] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, Wohlhart, Kirmani, B. Zitkovich, F. Xia, et al. Open-world object manipulation using pre-trained vision-language models, arXiv preprint arXhv:2303.00900 2023.

---

[47] Octo Model Team, D. Ghosh. H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy, https: / //oc/toooodelle github. io, 2023.
[48] Y. Jiang, A. Gupta, Z. Zhang, G. Wang. Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth Interna- tional Conference on Machine Learning, 2023.
[49] A. Szot, M. Schwarzer, B. Mazoure, H. Agrawal, W. Talbott, K. Metcalf, N. Mackraz, D. Hjelm, and A. Toshev. Large language models generalizable policies for embodied tasks. preprint, 2023.
[50] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.06175. 2022.
[51] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv: 17766, 2023.
[52] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
[53] Florence, C. Lynch, A. Zeng. O. Ramirez, A. Wahid, Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL), 2021.
[54] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRI.), 2020.
[55] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[56] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[57] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[58) I.. Chen, S. Bahl, and D. Pathak. Playfusion: Skill acquisition via diffusion from language- annotated play. In Conference on Robot Learning, pages 2012-2029. PMLR, 2023.
[59] Daafouz, Riedinger, and lung. Stability analysis and control synthesis for switched systems: switched Ιyapunov function approach. IEEE Transactions on Automatic Control, (((11):1883-888)) 2002. doi: 10i:10.1100 100.1110000/TAA22202000404
[60] S. Prajna, P. A. Parrilo, and A. Rantzer. Nonlinear control synthesis by convex optimization. IEEE Transactions on Automatic Control, 49(2):310-314, 2004.
[61] T. Wei, S. Kang, W. Zhao, and C. Liu. Persistently feasible robust safe control by safety index synthesis and convex semi-infinite programming. JEEE Control Systems Letters, 7:1213-1218, 2022.
[62] T. Wei, S. Kang, R. Liu, and C. Liu. Zero-shot transferable and persistently feasible safe control for high dimensional systems by consistent abstraction. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 8618 8614-8619. IEEE, 2023.
12

---

[63] W. Zhao, T. He, R. Chen, T. Wei, and C. Liu. State-wise safe reinforcement learning: A survey. arXiv preprint arXiv:2302.03/22 2023.
[64] R. Chen, W. Zhao, R. Liu, W. Zhang, and C. Liu. Real-time safety index adaptation for parameter-varying systems via determinant gradient ascend. arXiv preprint 2rXiv:2403.144900 2024.
[65] T. Wei and C. Liu. Safe control with neural network dynamic models. In Learning for Dynam- ics and Control Conference, pages 739-750. PMLR, 2022.
[66] C. Ju and H. I. Son. A hybrid systems- based hierarchical control architecture for heterogeneous field robot teams. IEEE Transactions on Cybernetics, 53(3):1802-1155, 2021.
[67] A. Schmuck, Majumdar, and A. Leva. Dynamic hierarchical reactive controller synthe-
sis. Discrete Event Dynamic Systems, 27:261-299, 2017.
[68] L. Sentis and O. Khatib. Synthesis of whole-body behaviors through hierarchical control of behavioral primitives. International Journal of Humanoid Robotics, 2(04))505-518, 2005.
[69] D. Shim, H. Kim, and S. Sastry. Hierarchical control system synthesis for rotorcraft-based unmanned aerial vehicles. In AIAA guidance. navigation, and control conference and exhibit, page 4057, 2000.
[70] E. Firouzmand, H. Talebi, and I. Sharifi. Hierarchical control of linear systems using extended robust approximate simulation. European Journal of Control, page 100987, 2024.
[71] M. Liu, Y. Tan, and V. Padois. Generalized hierarchical control. Autonomous Robots, 40: 17-31, 2016.
[72] X. Luo, S. Xu, and Liu. Obtaining hierarchy from human instructions: an Ilms-based approach. In CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023.
[73] A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09771. 2022.
[74] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL htttps://drall wwi.iieeemi........... mit.miitdee edu.
[75] T. Wei and C. Liu. Safe control algorithms using energy functions: uni ed framework, benchmark, and new directions. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 238-243, IEEE, 2019.
A Experiment details
A.1 Task descriptions
Open the door Opening a door is challenge for robots because door has a fixed swing path that must be followed exactly. As shown in fig. 2, position control can easily lead door damage or failure of action. Therefore, it is preferable open door with compliant controller. With multi- ple rounds of experiments, found that Meta-Control synthesized control system usually involves acting the task controller or the tracking controller. Al- though the trajectory may not be perfectly aligned with the swing path, with the stiffness controller, the robot can still open the door because it complies with the force given by the door.
Car Cartesine anSt itw.siiitiieeeeeeeeeeeeeee wosSstfffnessootrollll
Wipe the board Wiping a board requires certain amount of force be applied on the board while moving the eraser, which involves two different objectives: position tracking and force tracking. As shown in fig. 2, the synthesized controller successfully removes the marks, while control frameworks
13

---

Task
Challenge
Meta-Contrer designed system
Open the door
The robot trajectory must fectly align with the door's swing path. position control can casily lead to damage or failure.
CartesianStiffness Controller, allowing imperfect trajectory planning and tracking with compliant behavior to avoid damage or failure.
Cariesian
Balance the cart pole
The pole is non-actuated. The system is sensitive, requiring high-frequency feedback and convergence guarantee.
LORController
SotriddossiiooooorCCotooooooolo The LQR controller gives the force to be applied on the cart, and the hybrid position/force controller tracks the de- sired force on the y-axis while maintaining neutral pose on x-axis and z-axis.
Collision-fiee free pick and place
whole-body collision free in continuous time during the whole task.
Kinematic KinematiiTTajeeotryyyFlle TrajectoryMPCC SafeController, allowing discrete-time planning and continuous-tina whole- body collision-free tracking.
Wipe the white- board
Two different objectives: track- ing position and maintaining force
Cartesian CranssiaaTTajectooyCoottoooooo + HybridPositionnoree Controller, allowing position tracking while maintain- ing desired force the whiteboard.
Table Experiment tasks, challenges and Meea-CCnntoll synthesized controllers..
that only consider spatial relationships are not suitable for this task because of the lack of force constraints. In most trials, Meta-Control chooses Cartesi CartesianInterpoll www.teerpllllttoooootooooooooooo task controller, and PosaForcaController controller plans the trajectory of the eraser, while the hybrid position/force controller tracks the trajectory while maintaining desired force on the board to erase.
as tracking controller. The Cartesian interpolation
Balance the cart pole Cart pole is a classic control task that has been extensively studied. Attempts were made synthesize simple PID controller with LLM balance a pole with predefined APIs where the cart be controlled directly [1]. However, in this experiment,
robot to hold the cart and ask LLM to balance it by controlling the robot arm. This is significantly more challenging task because only low-level APIs of the robot arm are given, and the pole is attached to the cart with a non-actinn non-actuated free joint. The LLM has to understand the relationship from the arm to the cart, and from the cart to the pole. In most cases, our method chooses an L.QRontrooles as as the tracking controller. The LQR controller
the task controller and the PoseForceControllee gives the force to be applied on the cart along the pole joint direction (y-axis) to balance the pole, and the hybrid position/force controller tracks the desired force on the y-axis while maintaining neutral pose the x-axis and the z-axis. Profile of the pole's angle is shown in fig. 5, which shows that the synthesized controller efficiently balanced the pole.
The synthesized control system is described below:
PPoleg, Pole,, Carty: Cartg]
Joint states, Ellaaget EBiarge
U End-effector (EE) force on y-axis
Joint torques
h(2, v) Linearized Cart Pole dynamics
f (ar, u))
Kinova dynamics model
(T) LQR controller
Tu. (I, y, v) Pose Force Controller
where
0
0
0
0
11(2, v)
0
1
1
+
V.
Tpols
Collision-free pick and place Pick and place is very common skill in daily life. In this task, require the robot reach goal position while maintaining whole-body collision- free. The goal position be either the location of the object the target location. In most
14

---

as the task controller to generate collision-free way-points for reaching the goal, and a SafeController as the tracking controller to guarantee collision-free in continuous time. In most cases, the LLM wtw.ea/aaaaaattoooooooeeeraaaaaa
cases, the LLM chooses KinematicTraject www.rr/rww..aaaaaaaaaeeeetttttta
chooses a
as task controller to gener- collision-free way-points for reaching the goal, and SafeController as the tracking controller guarantee collision-free in continuous time.
A.2 Joint torques during openning door
As shown in fig. 8, openning door with position control can lead to very large joint torques, leading to failure or dangerous behaviors. But with compliant controller, the joint torques are much smaller and the door is opened successfully without damage.
Joist tonque range duilng opening dooc
Peslisn Contrll Strthea Cretral
Seiffees Control
1232
-30
Figure 8: Joint torque range during opening door for the baseline (position control) and the Meta-Control synthesized controller (stiffness control). The baseline has huge torque hecause the planned trajectory is inaccurate, which leads to damage.
A.3 Formal Analysis of Properties
Convergence and Stability: For the balance cart-pole task, the convergence can be guaranteed by solving the Riccati equation for the LQR controller. The closed-loop system matrix A - BK has the following four eigenvalues: -41..29, -9 9.925, - 1.502 + 1. 1.175j, - -1.00 - 1.175j. All of them have negative real parts, which means that the system is guaranteed to converge. More rigorous analysis can be conducted by taking the linearization error into account.
Constraint satisfiability and Forward Invariance: In the pick-and-place task, MPC task troller is tracked by a safe controller. The safe controller is realized with safety index (also known as the barrier function), which guarantees collision avoidance with mathematical proofs [75, 62]: it ensures the system state always satisfies min(dmin - d(x), 100 (0.022 )(x)²)= 10 d(x)) } < 0. where dmin is the allowable minimum distance between the robot and the obstacle, d(x) and d(x) are the relative distance and relative velocity from the robot to the obstacle, respectively.
A.4 Runtime and parameter optimization rounds
In experiments, it usually takes GPT-4o 1-3 minutes to synthesize skill, depending the capacity of the servery.
B Prompt Design
We use variety of techniques to improve the quality of the full dialogue. First, we introduce some techniques in the overall design, followed by the techniques used in each step of the three-level pipeline.
B.1 Overall Design Techniques
1. Divide skill synthesis into three dialogues, corresponding the three-level pipeline. This approach avoids overly long contexts, enables step-by-stey reasoning, and allows reflec- tions to focus on the current step.
15

---

2. At the end of each dialogue, Meta-Control asks the LLM to generate summary to warm- start the next dialogue, including the decisions made in the current step and all necessary information for the next step.
3. Use generic prompt with placeholder design make the prompt zero-shot generalizable while remaining task-specific. Each placeholder is uniquely identified by a name>. The placeholders are dynamically replaced by the LLM generated code.
4. Provide guidance and examples to help the LLM generate code following a specific format. The code should start with a <step name>, e.g., <task model>, followed by Python code block encompassed by triple backticks (*
5. Use <step_nands _amen to insert L.M M---eeerrteed code into the control system. Our code extrac- tor uses regular expressions to search for <step name>, extract code from the LLM output, and then insert the code into the control system based on <step name>.
6. Use closed-loop design procedure. The L.M--ennrated code is executed by our program, which receives feedback, such as error messages or measurements during exccution. Meta- Control then asks the LLM to improve the code based on the feedback.
7. Use code replacement instead of code completion progressively construct the control system. Debugging and improvement are difficult with an incomplete control system code that is only executable after completing it with LLM-generated code, as feedback for inter- mediate steps would be unavailable. Therefore, we initially provide a blank but executable control system code, which outputs all zero joint torques. At each step of the synthesis, part the code replaced based on the identifier. We designed the control system to be always executable during the design process as long as the generated code is correct.
B.1.1 Dialogue 1: Design via Template Prompt
1. Use role-play prompt (e.g.. ... professor in control and proficient programmer) to invoke control knowledge and coding skills.
2. Provide some optional principles to assist the design, such as "design the task controller on simplified dynamics" and "use the task controller to accomplish the primary objective while using the tracking controller satisfy constraints or secondary objectives."
3. Use model templates and controller templates to ground the design space. The templates greatly reduce the design space and anchor the behavior of the synthesized control system, making it explainable and provable.
4. Provide all available measurements and their samples. The sample measurements give the LLM precise idea of what inputs are available and their format, helping decide on the appropriate task controller and tracking controller.
5. Provide example of the code that the LLM should return, including the <step_name> and how to initialize the chosen models and controllers.
6. Provide a checklist of requirements that the LLM should satisfy to reduce errors, such as "choices have been made for all steps" and "<step_name> must be included before each code block."
7. Use reflections. Meta-Control extracts and evaluates the code from the LLM response, then informs the LLM if the initialization of the chosen models and controllers causes any errors, allowing the LLM to decide if any fixes or improvements are needed. This process is repeated until there are no errors or the maximum iteration is reached.
8. Ask the LLM to generate a summary of the design to warm-start the composition dia- logue. The summary includes the chosen models and controllers, a detailed description of the inputs for the controllers, the meaning of each dimension, and any other necessary information.
16

---

B.1.2 Dialogue 2: Composition via Semantic Understanding
1. Use a role-play prompt as in the previous dialogue.
2. Use the design summary to warm-start the dialogue, providing the task, the chosen con- trollers and models, their input and output ports, and how they should be combined into control system to complete the task.
3. Provide examples of how to access the values of the input and output ports of a model or controller.
4. Provide an example of the input converter, demonstrating how to connect the input and pad zeros when necessary. The input converter is set of callback functions that assign values to the input based on the output of other modules.
5. Provide tips and potential errors to avoid when implementing the input converter. For example, "Ensure the dimensions of the output match port. The output value may only be partially available from the input port," and "For pose, velocity, and force vectors, rotation always comes first."
6. Use partial updates. Because input converters involve many functions, if an error is de- tected during reflection, Meta-Control only updates the function that the LLM identifies as causing the eπror, instead of updating all functions, which may cause more eπrоrs if the code is not exactly the same.
7. Ask the LLM to generate a summary of the data flow to warm-start the parameter tuning, including parameters of the controller, parameters the model, input converter structures, and potential performance metrics.
B.1.3 Dialogue 3: Parameter Optimization
1. Use role-play prompt as in the previous dialogues.
2. Provide list of code blocks that the LLM can update to optimize the parameters, covering models, controllers, and input converters.
3. Warm-start the dialogue with the data flow summary, providing detailed implementation of the control system, description, and samples of the module inputs.
4. Provide guide for parameter optimization: identify key parameters, identify performance metrics from task state and task control, and then determine if the time series of the metrics is desired.
5. Provide checklist of errors to identify the cause of degraded performance, including defi- nition, initialization, implementation, and suboptimal parameters.
6. Iteratively optimize the parameters with reflections. Every time LLM updates code or parameters, Meta-Control evaluates the new system, provides time series of the cho- sen metrics, and allows the LLM decide if the performance is desired. This process is repeated until the performance is satisfactory.
C Full conversation
The full conversation skill synthesis is attached below. Some long numerical arrays are omitted for the sake of clarity. The controller templates, the dynamical model templates, and the input port samples attached after the conversation.
17

---

C.1 Balance
C.1.1 Design via Templates
#* Mata-Cantral
GPT professor
control proficient programer of Python PyDrake. can be accomplisted by sequentislly executing several skills. The QPT will help the user write code to conpoаe one of the skilla after reading the requirenenta. The GPT will inatructed atep by step. task controller and tracking controller. The tack controller is respensible for generating high-level and abstract control, and the tracking controller responsible for tracking the high-level control and natisfying constraints.
skill acceaplithed hierarchically
optional principles design the controller 1. Design the task controller for the object of interest or end-effector, and design the tracking controller for the robot.
tracking controller:
Design the task controller simplified approxinated dynemice, such Carterian space
bo approximated linear nodel, and design the tracking controller on the full robot dynamics. of the primary
If skill involves miltiple objectives, design the task controllar objective and design the tracking controller to take care of the rest of the objectives and tracking controller
take
coatroller 4. Task control is often the pesition, velocity, acceleration or force in the Carterian space. But in some cases, the taek control is best to be joint torque.
constraints.
output joint torque.
5. The final output the tracking controller must be joint torques.
GPT proceed
following steps: makk_controller,
task_nodel,
bracking.nodel. In the GPT's reeponse, the nost important part will be <step_name> followed by code Each code indicates the code is sritten for specific step. The GPT can feel free to make some plans to help it think before writing the code blocks. There is no need to explain the code after the code block. The GPT should inport any nodule. The GPT should make up arguzents, variables, anything else o that is not nantionad.
block enclosed by " "pyytoon"
should correspond
single step.
explicitly
## User
stabilize cart pole. + the robot arm. The pole rotates freely around the x-axis and is unactuated. Please stabilize the polo around the upright position (vhen the pole angle, denoted by PolePin, is 0) by exerting ferce on the along the y-axis. secondary objective is to keep the end-effactor pese close possible 17578844594000 D.O, 1.87079455a-00, 5.96598901e-01, 1,34625934--02, 4. 4.35892236-001 ([rall, pitch, yau, x, y, a]).
Please help ne compose
kg, the pole is
kg.
Certainly! I'm goad uriting controllers for robots with Pytraka. will folley your instructions carefully and think carefully.
## Mata-Control
available neaeurenents, dynanic nodels, controllers.
inputs
Hera all available dynamic model satup functiona yanl format:
(dynanic_nodel. model_brief_ett.
Here are all available controllers in yanl format: ccontrollar.breff.oo
Based on the available nessurements, nodels, and controllere, please think step by step to choose the
controllers:
1. Decide the subject of the task controller, is it an object, the end effector, or something alse.
Dacide desirad task control be applied on aubject, it Cartestan position, Cartesian
force, something else.
Decide the task space model that can model the dynemics of the subject with the task control. When the theory
needed, draz upon your extensive knouledge of and systea modeling. Think step by step to nake sure the model is correct according to the tack description.
analytical
the dyuanica
4. Dacide the task controller that can apply the desired task control and realiza the goal of the skill.
Nake aure the controller can realize the geal with the given available nensurements. There are no additional inputs functions provided the controller.
Decide the tracking space nodel.
tracking controller constraints.
output task controller satiety possible
Detail about hou can the chesen modela and controllers realize the skill goal. Ba spacific.
Please return the choice with the folloving fornat.
model>
18

---

1 ppython
task_nodel natup,n arg2)
<tracking.nodal>>
""python
tracking tetup_nodellerm.ppp object_infoo
<taek_controller>
"pyyhon
cont_cont ControlleerCasssarggl, arg2)
stracking.conttrlllas
"'pyhhon
arecking.controllee (ontzollerCleeeaargg))
Nake response complies with the following requirenente:
<stap_ <stap_name> nane> must be included before each code block so the code block can be recognized. step_name must enclosad angle bracketa
Choices have been nade for all stepa. à correct solution exista with the given information. instantiating controller stated in the yanl file (arga). +4Do not* include Itens in "Iaput.porta" (auch as "controller, "controller.paranttter paraneters"") as argunente when instantiating the controllers. They are not argunents for instantiating the controller.
arguments (arge)
argunents needed
The argunents should be defined befare use, except "arm_type" and "object_infa", which can be used directly without definition.
Do not use any placeholder or assune any variable is defined. All information and available variables provided. your
nodel and controller requires additional information, rethink
it.
The nodele and controllers are instantiated correctly as stated in the naterials. chosen model natches the required model the controllers stated the naterials.
Considering the requirenents and the available nodels and controllere, let's proceed step by step to stabilizing cart pole attached the effector the + achiaving the secondary objective of keeping the and-effactor's pase as close possible to the specified position.
accomplish
and
#subjacc of the Task Controller:*s Since the prinary goal is to stabiliza the pole around the upright. the pole
position by exerting
along yraxis the cart, systen", This suggests we require contrel nethod directly influencing the cart to alter the pole's systez-lovel
subject is "the
position, naking
the problen involving
polo.
+eDesired Task Control:*e The control needed is force along the y-axis on the cart, shich indirectly influances the pola's angle achieva stabilization.
accurately represents dynanice for the application of controlling force. Given the problen's nature, a linear model approxination Therafore,
Space
cart-pole systen
upright position Lnnear because it can approxinate the dynamics of the cart-pole systen in affine nanner suitable for designing controller to apply the required force.
pole can
*TTask Controller: To apply the desired task control (force), an appropriate choice is an "LORControllee". (Linear-Quadrest Quadratic Regulator) vell-suited for stabilizing systen around equilibrium point (the upright position), given it requires model of the systen dynanics, shich aligna wall with our chaice
linear model.
#Trrcking Space Model:*+ For the secondary goal, the nodel needs to incorperate the arn dynanica ca along end-effector's
would be suitable choice because it returns a model coneidering the robot arn and static objects, allowing end-effector's position
the constrainta
"setup_arn_vie win.vittobb/ecaa....
control over stabilization.
keeping the prinary
pole
6. <TTrackigg Controller: Considering control forces to stabilize the pale, a PesseForcatontrallarr is selected for the tracking controll lar. and poses, naking perfect nanaging
reepect both desirable end-effector pose and exert
capable handling specific secondary objective of zaintaicing the eud-effector's pose.
Let's proceed with specifying the nodels and controllers as par our choice:
<taek_nodel> python
for the cart-pole system u0 2e,01((1,1)) Inétial controi imput (force)
up (o.2eree((s1))) Instsalising
n_cart 0.1 mass of
n_pole = 0.01 # mass of the pale
1_pole 0.25 tength of the pole
9.81 gravitational acceleration
19

---

and matrices placeholders and must be defined based the specific dynamics the cart-pole systen
up. array([Io,
[0, 0, m_pole
n_cart, 0],
[o, 0, 12,
[0, 0, (a,eart m_pole) + (Lpplee n_cart), o1])
B up. array ([0), (s/t_ [o], (=/(0_polemm_eart))) C eya (4)
D ww..pp..ae ((4, 1))
tode_ 1etup,
ud,
<tracking model>
<rrdelingnoode>>>>
python
tracking.nodel
aatop.amm.itth.hjjjete....dd (arn (arn_type, object_info)
Stask_controllar>
python
dia 10, 10]) # Emphosssing the isportance of the pole's posstion and veloesty R up array(([o. (]]. Control effort cost
taak_controller [aBl,,tllllrttaaa.dddd,,,, x0, u0)
stracking.controllen python
wwt._oottl__riieeeeee up a.rarryyaPale,, False, False, False, True, False]) Fonce control along the y-azis
tracking.controllen Postforeetatrolllertttttttn.nn
poro__cottroL_xxia.mm))))
This sequance of models and controllers fulfills the skill's requirenents by utilizing systematic and + hierarchical control structure.
# Mata-Contral
Here output running
<empty>
expert supplenentary instructions: <empty?
you think it is working as expected? (Eapty output means the system is running without errors) with vord "yes".
If it please If it is not, please idantify which atep(a) vant vrong and explain how to fix it by including whit step(s) nodify (<step_paze>)
updated code (enclosed by ""python"" ""), have proceeded
with the following steps:
tracking_node nodal
task_controlle.
trrccking_controll
up step.
systen working axpacted, you mast modify at step. nodify mitiple stepa. Remember to add "<atep_nane>" before the corresponding coda block. correct solution exista eo with the given information. Try your best.
Here is sample response when it is not working ae expected:
I think this arror occurred because [for sone reasen].
controllee
updated_code:
""yython
Keeftrckking@otttollleraas.. arg2)
## Assistant
yes
Yeta-Control
Please sunnarize the choice of the task controller and the tracking controller. Box sould you expect the input output ports of the controllers to connected to perforn the skil17 This response will be used connector, please include all neceesary infornstion, including:
proupts implement the
controller and tracking controller, including the meaning of each dimension (be careful about the order), and hov to get then from the outputs.
detailed description
input porte
arailable
description of any constant paranater have used that affect the perfornance of the control
aysten.
Everything else you think is necessary implenent the controller by only reading the sumary.
20

---

# Assistant
stabilizing pole attached and offector of robot arm, selected tuo controllers: an "LORController" for the task controller and tracking controller- Hare sumarry expected to interact and bow their imputs and outputa should connected:
the port
perforning
"PossForaaCotrollar"
contrallers are
#** Task Controller (LARController)
*+Iput Ports:**
represents
cart-pole esyaten. the "LURController',
vector is expected to have dinensions [x, x_dot, theta, theta_dot] shere "x" and "x_dot" represent the position and velocity of the cart along the y-axia force applied along the y-axis), and "theta" and "theta_dot" represent the pole's angle (around the x-axie) and angular velocity, respectively. This state can be constructed from the "edasured body_poses""
"neasured_ bady__eod_____ velocitis for the 'Cart', and PPllePin' neasurenents from "neasurad_ maamrrad_joint_ctaree
+*Output
"control"s This port outputa the computed contral inpot based on the LQR control lav. For our application, this control laput is a single scalar value representing the force to be applied along the yraxis to the cart.
Tracking Controller (PenoPorcadontrolllrr)
IIputt Porta:**
[[rll, pitch, yax, to keep the end-affector's 's close as poasible specified target while applying force to stabilize the pole, this target pose is set as constant paraneter [1.57078469, 0.0, 1.57079456, 0.865998905, 0.00134625984, 0.483592036) ", Note that the 4 force along the y-axis, which acts an the cart to stabilize the pole, is managed by this controller's "ea_force_targe
"ee_pose_ te.pose_targett:
input requires desired end-effector pose 2]" Since our prinary
"ee_force, target"; Thie input specifies the desired end-effector force in the fornat [[x-rotation, y-rotation, z-rotation, s-translation, y-translation, z-translatios". controller (LQBCentroller) indicating the force to be applied along the y-axis to the cart will be fed into this port, All other dinensione are set to as ve are only interested in exerting force along the y-axis.
output iron task
postttoon and velocity: Thase inputs provide the current joint pasitions and valocities of the robot arm, respectively. They directly obtained "neasured pospiiitioo and aissured_ann_eeoocig
+*Output Ports:**
"applied_arn, tra_torque' This port outputs the computed joint torques for the robot arn, which are derived based the desired force and pase control objactivas.
Constant
1. For the 'LAConttrlller' we intreduced state cast natrix "Q" with higher weights on the pole's angle angular velocity enphasize inportance stabilizing pola. relatively cost "R" is chosen to allos the application of necassary control afforta without being overly penalized.
2. In the Posaforcatontroller" se used a constant target pose for the end-effector, with force control prinarily along y-axis.
array "force_come con.oocoooooaee arrol_aaiisnen configured apply control only along the y-axis and poze control for all other dimensions.
#*# Summary
To connect these controllers offectivaly:
"state" input port the "LORController" ba assembled from available neasurements,
+ particularly focusing on the cart position and velocity along the y-axis and the pole's angle and
angular velocity.
The output control force from the 'LoRCentroller' must be properly directed to the "ee_force_taaee target" input port the "FoneForcaContraller"" spacifically into the dimansion controlling forca along the y-axia.
The "ee_pose_ target", "arm_positio'' and "arn_ crn_valocity' ports of the "PanePorcelantrollee" require targeta, with end-etfector pose being apecified constant. The final output of "applied_arm, araeedd_rn__trrreeeeeee tra_torque" from the Preeforeebontroller deternines the joint torques to meintaining the end-offecter's desired pose.
direct neaaurenenta
achieve both stabilization the pole
This approach integrates both controllers to zanage primary objective of pola stabilization the secondary objective of maintaining the end-effector pose efficiently.
C.1.2 Composition via Semantic Understanding
#N Meta-Control
The CPT is a proficient programar Python and Pyürake and a professor in control. The GPT will help the requirenenta. The GPT will instructed step step
accomplish the code after reading please only complete nentioned tasks. The code implenents robot skill through hierarchical design. tracking controller given track the task control. The task controller and the tracking controller vill be given. The CPT needs to accomplieh a task controller converter and tracking contreller converter. A converter traneforne arailable neasurements and to the required input ports the controller.
task controller is
to give high-level control,
21

---

The GPT xill proceed with the folloving steps: task,callback, and cackarg.callback. the GFT's respense, the inportant part
by ""yython"" Each code block should correspond to single step. This explicitly indicates the code
<atep_nane> folloved by code block can
to some plana to help writing the code blocke. There need explain the code after the code block. The GPT should not inpert any nedule. The GPT should not nake up argunente, veriables, and anything else that is not mentioned.
vrittan
spacific step.
Meta-Control
Please help ma compase skill to stabilize cart pale. The cart is attached to the and effector of the robot arm. The pole rotates freely around the x-azia and unactuated. Please stabilize the pole around the upright position (wben the pole angle, denoted by PolePin, is 0) exerting ferce on the cart along the y-axis. secondary objective is to keep the end-effactor pese as close as possible to 11.67079499+000, 0.O, 1.57079465=400, 5. 5.96198901e-01, 1.34625984--03, 4.33589238--01] ([rall, pitch, yau, x, y, zl). The cart in 0.1 kg, and the pole in 0.25 B, 0.01 kg-
#* Assistant
Certainly! I'm good at writing callback functions for ports in PyDrake. will follow your instructions carefully and think carefully.
Mata-Control
Now please implement callback functions of "LeafSyater" that acts as a connector which have already (observations controller paranaters). The output porta correspond the input porta of the task controller. Here is yeal file describing what the inpute to the skill are: iatilll_nput___mmare
constructed. input porte correspend inputs
the input porta. get of 4 port by "value - self. GetInputPort (pttppuuPrrppprttaaaat (port_usne)) Eval (context)" when implementing the call back functions:
Hera print port nane, type, sample
The chosen task model, tracking model, task controller, and tracking controller "pyyhon
20 (oreerer(((,1)
Initsalizing the state for the cart-pole system input (force)
up. (op.zoror(11.1)
n_cart = 0.1 # sass of the cart n_pole 0.01 # mass of the pole 1_pole 0.25 length the pole
8 == 9.81 grauutationaa accateration
matrices
placeholders
defined based epeciféc dynanscs of cart-pole
+ system
ap arO, 1,
[o, 0, m_pole
n_cart, 0],
[0, 0, 0, 1),
[o, 0, (r_cart s_pola) g (Lppol s_cart), oll)
8 up. ([[o) [i/n_cart], [o], [-1/(i_polem_cartt]) eye
D zeron((4, 1)) task_nodal setup_ Linep_inearr__eee nocall uo, B, D)
tracking.nodal aatup.arn.vithhoojjeee (arn (arn_type, object_info)
- diag( 1, 10, 10]) # Emphasizing the isportance of the pole's position and velocsty effort
up. array il1)
task cont_conrr (rreentrroleerttakkkeo Q, B, x0, up)
force porve_control_stie axie__eskki ar..aayyaFFlee,, False, False, False, True, False]) # Force control along the yossis
PuaabraeCanttollortttaiiinn..nee force__cnttol cow.eoooroooooi__nataeeee cancking.contollle
Here design sumary explein expected way
by the porte: (eerigy_ (fron Design Templates)
Here the sumnary the input porta task controller: faaek_controlee__r inp.utuuuuu_arrry (from the controller sumnary)
functions you implexent python
taek,s,ate(sel1, context, outgut: BasicVector) pass
22

---

Hore i5 an exampla response:
<task_callbacke
python
def balderrlder, (ell,,,,, context, output: BasicVector):
abstract, cestract_input_peee waaputpport_tto value self Sel.:.GeiapuuPPorrpprr... mane Bea.aaeeeelllooteeeeeea (conteet))
output. SetPronlaactor (abstraac. (sbstractt_input.r... wwwptttprrt..alllee.... pert_valuue"plare "talellatr_tolddr_eeee
def placa_holdas fon.der_tinctin22sses contaxt, output: BasieVactor]
iecto_.nputt.port.t out zeros()))))
self GetInputPort(portt (portt..rna Baae
out[3]] vector_inpatt
Sutput.BetFronVeete (out)
def place_bolder, Stwe__oodor__uuciinnneeeeee function3(self, context, output: AbstractValue):
valua salf GetInpatfort(portt.nngs Eval (contaxt) cntput.set__aee sat_valuuevaalus
Please implement all the task callback functiona in one code block baginning with <task_call callkack..
Here
implement the callback functions: When use a port, nake sure the nane of the port correct and the port is available in the current function.
Kote that sone of the porta should be passed through without any nodification like control.paranete ports
nodified.
Make sure the dinensions of the output match the The output value be only partially available from the input port, You need to decide the value for the remaining. Be careful of which dinension to set value, nake
complies with the description of the For posa, velocity, and farce vectors, rotation alvays comes first, such as [roll, yas, pitch, x, 7. z] [x-rotation, y-rotation, z-rotation, x-translation, y-tranalation, z-tranalation].
the controller sunnary.
* If the input port is dictionary, nake aure only use keys that has shown in the printed port value sample.
* If the output should be an enun type, you can use the anum type mentioned in the controller summary
in the beginning. Make the call back functions correctly to 4 directly without any import.
Recall the task requirenents
solve the task.
avoid using placeholdera simplified examplea require modifications by calculations are coaclusive. In case of any uncertainties about parameters, utilize the information informed inference available.
provided
You can use mun_q to denote the nunber of joints
Please implemant the call-back functions. Please think step by step then write the code.
IN Assistant
<task_callbach
pyyhon
def tark_state(soll) context, output: BasicVector):
Patracting secessery keputa
neasured_boo bodyy__s_odppppeeee self. Getnppattttt body [vall
uessured_body velocities self. Www.Ineuuoorttteeeeeeeeeeee boregureeddody_eelllele velocitiee") vee_oooiiiee)))) Biaiiiiee".ellllonaeaaaa (cantcet))tt
jaauure__jint_stattee self GetInputPort tareeoorrtttttaaaeee aeoasured_joint_saatee Baaatt_stttte""_aac(acaaea (canteet)
# Inctsalsze state vector utth seros state mp. (4)an(4)
# Posstion and Velocsty of Cart along yranss from body poses and velocštšes cart_pose neasured_body bodyypoeebd ('dree tramelstion() cart velt_itl neasured_bect bodyy_veooiiion (Crrt' translational() stata[o] cart.pposa=]
positiin of atate [1] cart_velocity(s]]
veloesty Cart
Pole's angle (aroand s-apis) and angular velocity joent pole, angla
jaauure__oiittettttee ('PoePinn') [ ['positione'] pole, angular_ velociity
jnseured_joint_ttee (" ('PolePin') ['velocities']
state [2] gole_anglefO) # Assuning "peastions" ss numpy annay with one elament
atate [3] www.aanggarrelleeee
Assueing 'velocstens" sunpy array one elenest
output. ineFrroVactor(staatae
Meta-Control
Now please implement callback functions of "LeafSysten" that acts as connector which have already constructed for the tracking controller. The input porta correapond to inputa to the skill (obeervations and controller paraneters) and task control (the output of the tack controller). The output ports correspond to the input ports of the tracking controller.
Recall the imput port of the skill given previously, and the folloving is description of the tack skill imput ports www.toooooootttttaaaaaaaaaaooaee pomt_a (fron the controllar sumnary)
control port.
control port.
(task_control_ppn)
Here the sumnary the input of the tracking controller:
23

---

seracking_controllle htwpllrrr_nnut___marrree (fron the controller sumnary)
functions you implerent python
context, outpat: BasicVector)
def tracking.eo_foree.tte
output: BasicVector)
pass
def tracking.arn.pannn ars.dnng..mmpppttiooaaaa.. context, output: BasicWactor)
pass
def waching.ar..vloocityeeel. context, output: BasicVector)
Here an example response:
stracking.callbacht
python
def place holder, funciionnnnt,,, context, output: BasicVector):
peetract_iaput..oo prruut__ort___eeee
walc.plac___eee saace__ooddr_port bo..de..........a..ccteett
SetrronVeerreeeeeeeeeeeo (abstract_ iortetttttttttttttt...... palluel"placehhllue Xey"))
def place_holded
futtter/ffnttoneeettt context, output: BasicVector):
vector vector_inpus_port_eee
self place_holéer_por (conteee)))))
out (oros(((
out[:]] Berrr
place_boldez
seeeebooddr_fuuciooS3333 functionn(selff, contezt, output: Abstractialue):
value self place_holder_port Bval(contott)
Follow previously given tips inplementing callback functions. Furthernore: Note that output ports of the task controller have been renaned by adding the prefiz: 'task."
port_name.
Now please implement the call-back functions, Please think step by step then write the codo.
## Assistant
<tracking. python
tracking.eo.pp.pp
Becpiig.ee_poss_trreeeee (soes context, outpat: BasicVector):
# Destred end-effector pase sa constant as per design aunnory
desired_ec_pose (777844060,+00, 0.0, 1.87079466e400, 5.96598901e-01, 1.34625884-033,
4.335892388-01] output SetFronVeeerrrrr (tpp.arraydeirrrddeo......
def tracking. ea wot_foocettrgetteeee (sll, context, output: BasicWector):
from controller, directing Obtoining aelf. Get ("aarortttttaak_coootoo""" Evrroi"")..vl(aaaaa Eval (context)
apply force along y-anis
task control self,
# Only force along the g-aris is controlled, the rest is masked out with zeros. se_force_ target mp.array array([0.0, 0.0, 0.0, 0.0, task_control [e], 0.0]) output wewpaa/eccoreeeooteetrrreeee
def tracking. arn pmspoiitionteeff, context, output: BasicVastor):
Directly passing
ane's measered position
neasured_an arn prn_pooition self PerIputPortt Pooppuutoorrteeeeeeeeaaaa ttr.euurdd.ann.pstttooo (araatiioo""_Bllloootttaaat (conteet)
autput SetFronVactor
def tracking, arn, velocity(eelf,, context, output: BssicVector);
Directly passing
amm's measured velocity
neasured, arn_velocity self, DetInppuPort ("noasurod_arn wonnweloocty"") Eral (contaxx))
output. SatFronVacrrrrrrrrrrooeo
(neauureed (neaaured_ vmllocctty
Yeta-Control
Here is the output of running the skill: <empty>
Bunan expert supplenentary instructions: compty?
think it is working expected? (Eapty neans the eystem running without errors) it is, please ansver with one word "yes". not, plesse identify
step(s) vrong explain it by including which
step(s) to nodify (<step_pame>> and the updated code (enclosed by "python"" ")). Vi have proceeded the follosing stops:
task callback tracking.callback
24

---

Please do nonexist step.
syaten working expected, you nodify
step. nodiry miltiple
< steps.
step, you can only nodify the function you urong- kept unchanged. Bemember add "<atep.nane>" nane?" before the corresponding code block. correct solution exista with the by given information. Try your best.
Here 15 sample response when it is not working as expacted:
I think this arror occurred because [sone reason].
updated_code
<tracking.callbach
"pyyhon
tracking. www.king.enn.peeiiioeeeee.e context, output: BasicVector);
Assistant
yea
## Yeta-Control
Please sunnarize the costrroer,,,,,,, tracking. lask_callbac and the baccking.callbeee Explain the data flov. This response vill be used pronpt choose the performance netric from neasurements and tune the controller parazaters, so please include all necessary information.
## Assistant
integration
tracking controllers comprehensive control loop for stabiliziig cart-pele system attached to the end effector of robotic arm. Here's sunnary of the componente and their data flow:
respective callback
#** Task Controllar ('18Rtontroller')
*arppaae: Stabilizea cart-pole system be applied elong the yrazis of the cart.
upright position by computing
IIIput
"state", Accepts state vector [x, x_dot, thata, theta_dot] representing cart position and velocity along the y-azis, and pole angle and angular velocity, respectively. ++0utput Port:**
"control": Dutpats control action (forcs) basad on the LQR control lau.
**Inplemented by Function:** "task_state" "alasz_
Tracking Controller "PssePorceeattelleee")
**Purpose:** Aligns the end effector's 's pose with target while applying the computed force from the task controller to stabilize the pole.
*+Input Ports:**
"ee_pose_ targett: Takes constant target pose for the end effector.
"ee_force, target";
(force slong yrazis)
applied by the end effector.
"'arn.oosition' Provides
joint positions velcctylo Provides current joint velocities of the robot am.
arm.
+*0utput Port:**
N/A (As these inputa directly dictate behavior - tracking control logic).
effector and based internal
**Ipleennnted by Functions:** "tracking_ee se_pose__teeeeeeeeeeeeeeeeeeeeee pose_target', "tracking se__eorc_tarreeeeeeeeeeeeeeeeeee target",
"tracking.arm.posstte" and 'tracking volooocityy in
callback"'.
#*# Data Flow
vobsarvations to Task Controller:*x
pole angle angular extracted from aggrogated poseee, "zeasured body_ velociiil and joiured_jointeettteee These values are vector [x, dod, theta, theta_dot] by "task_stato" task controller callback, which feads into the 'IARControllor..
position and velocity, along
nessured_bbod body_____bdd___o_e
TTaak Controller to TracKing Controller:**
The computed control action (force) from the "LURContraller" is passed to the tracking controller callback. end effector's desired force along the y-axis while keeping forces in other dinensions at zero.
"tracking.ee" wew.iigeeee_eeee se__foree_trres function
3. velbservations to Tracking Controller:ex
Direct observations the robot arm'a joint positions and velocities ("neaaured amm.ureddammppsiiii position and nessured velocity") passed through "tracking por_ponttionn and bracking.arn_velocee functions to their respective input ports on the tracking controller.
4. Constantt Parameters:**
target pose
"trackingges, pocpae_____p___e______ "argeetttrt"
roflecting the sacondary objactive of maintaining the and-effactor pase close to predefined poas.
WM Performance Metric Selection and Controller Tuning:
25

---

the controller paraneters and select appropriate performance
the complete flow of data
< and objectives should be considered; setask controller*s, current pole position/angle and its upright position (zere angle), the root-meam-square error (RMSE)
performance netric night include Buclidean distance
the
pole's angla over tima, the control affort exerted.
Tuning paranetera for the task controller ('IRRonttrolerr) include adjusting the "q" and "R' natrices, which balance between penalizing deviation from the target state and the magnitude of the control effort.
sstracking controlleres, the perforsance netric night locus + affector's pose alignnent with its target and the magnttdee of applied forces.
precision the
Tuning invelves the force control axis mask and potentially adjusting internal pose control paranetera within the PoeeforeeCoatrlller to achieve desired tracking performance and reeponeiveness.
carefully nonitoring these performance netrics and adjusting controller paranaters accordingly, the control schenea can be refined to optinize both the prizary objective of stabilizing the pole and the secondary objective naintaining the desired end-effector pose.
C.1.3 Parameter Optimization
## Meta-Contzol
The QPT is proficient programerr Python and Pyürake and professor in control. The GPT will help the user to revise the code after reading the roquirenents. The GPT will ba instructed step by step so code inplamenta rabot akill through hierarchical design.
please only complete nentionad tasks. A taek controller is chesen to give high-level control, and a tracking controller is given to track tracking controller are connected through
task controller functions. The GPT will holp to tune the paraneters and check variable assignnents of the hierarchical control systens, A time series the task controller state and control vill be given to help you determine the systen running expected. Then GPT needs to decide if the time series dastrad, and then identify paranaters and variables that can affect the time series. In the and, The paranaters variables through multiple
control.
GPT needa
interaction.
can update any stracking_contee controllerr, <task_callbact), strackiig.ccallbeet callbacks. Each time, the GPT needs to pick and reurite one or nore blocks to update the paranetere or variable assigmentss. The paraneters and variables be anything defined in the code. Sonetines, may requira switching the ordar dinensions to correct the ixplenentation arror.
following
blocks: coask_ Ccracking_nddel> cattrclotr.ll,r),
In the GPT's respease, the nost inpertant part will be <block_nane> folloved by a code block enclosed by vritten for specific block.
"python" t* This explicitly indicates the give the complate updated coda black that contains the ravisian. The GPT can feel free to nake soma plans to help it think before vriting the code blocka. There no need explain the code after the code block. The GPT should import any modula. The GPT should not make up argunenta, variables, and anything else that is not nentioned.
needs
## Meta-Control
Please help na compose skill stabilize
pole. The the robot arn. The pole rotates freely around the x-axis and unactuated. Please stabilize the pols around the upright position (vben the pole angle, denoted by PolePin, is 0) by exerting force on the along the y-axis. secondary objective is to keep the and-affactor pesa close possible 5707846660000 0.O, 1.57079455++00, 5. 5.95598901e-01, 1.34625964--33, 4. 3.35889236--011 ([roll, pitch, yau, x, y.
attached
kg, the pole is
kg.
Certainly! I'm good tuning paransters for control systens in PyDrake. will follow your instrustions carefully and think carefully.
## Meta-Controe sunnary
control systen to explain expected vay the controllers work, cs hau the ports are connectod: fimplementations su_nnnay))) (fron Interface Aliganent)
Here is a yeml file describing what the impute to the skill are:
Bakilt_input_ee (ummuryy
print the port nane, type, cample Esnput_port_sanplae
the input ports:
Bere the detailed implementation of the tode_, trackinggnodel, ceek_controller, b gracking.controlll task_callback (the callback functions for the tack input part), and tracking.callbes (callback functione tracking input porte).
pytton
x0 =p.zeros ap.zoros((4,1)) Instializing the state for the cart-pole system
up 20r2e ((1,1) Inttsal control input (force)
26

---

0.1 # mass the cart
n_pole 0.01 # mass of the pole
1.pole gravitational ecceleration
Length
pole
== 9.81
natréces are placeholders
be defined opecifse dynanses of the cant-pole
system
A up. array([[O, 1, 9, 0),
[0, 0, s_pole
n_cart, 0],
[o, 0, D, 11,
[o, 0, (a_cart n.pole) g (Lpole n_cart), oll)
B = [rray[[[]], [irm_ [ol, (-1/0_polorm_aart)) up (4)
D (4, 1)) tack_ setup_linears modo (x0, uo, A, B. D)
<tracking.nodeD>
1* pyython
tracking.nodal aatup.am_vith..hjeeta.... (are_type, objact_info)
cont__oontroler>::: python
q up. di.diag([L, 1, 10, 10]) # Emphasszing the seportance of the pole's posstion and velocity effort
ap aroaa([[0.]])) task task_controller (taoettrollorrtackooooooo Q, B, x0, u0)
controller?;
python
aFaaey (Palee, False, Palse, False, True, Falee]) Force control along the www.ccottoooriisnneeeeee up. np.array(
y-azis
gracking.controllle Raseeraccattlleerttaaatnn...o coroo_oontrol_axlaa.o
<alabkcallbacko:
pytton
tack laek_stat,,self, context, output: BesicVector):
# Estracting secessory impats
neasured_bodt bowuureddbddy_poooo body_poses
CatInpatPorttttttteeeee..tt ("reasured, body_bosss"") [val(context)
neasured_bodd valocities tttueoottatty.alloctttsss= self. www.Geeeaaoootteeaaaaaaae ("neauurddddd body. aady__vlocitiee"" "esiee"")_Vve" Evall (context)
neasured_joine states self, Iolf.uerInputPott ""eeasured_joiii joint_states") stt.ssttttaaaaalllaaaa (rantcott)))) Testialsze state
state up 2eroze))
Velocity of along yrozss from body poses and veiocities Sacdyyppoes[ll "oses["Cart')) translatiinn))
cart_pose neasured_body
cart_ part_valocity neasured_ htceod__ooo____eeeeeeeeee modd__valocitiaa [aartt translationall)
state [0] cart_poae[1] # poastion of Cart
state[i] cart_veloctt cart_velocity(1)
# velocity of Cart
Pote's engle (aroand
asgular velocity from foint states
pole_angle ressured_jointeetttee "PolePin'] ['positione']
pole_ aaggaaanggeeeeeeoooee
neasured_joint_etttee ('PolePin)) ["velocities']
state [2] pole_angle[[] # Assuning "peastions" sa numpy arnay with one elament
atate[s]
velgoiarvllocctt((]] Assueing 'velocsténs" sumpy array one elenest output. (staat)))
<tracking.calllan cabaacc python
def
Dessred end-effector pose context, output: RasicVector):
per design swenary
desired_ee_pocr = (777846666000, 0.0, 1.57079455++00, 5. 5.96598901e-01, 1. 1.34625886--3,, 4.33689236a-01]
output. BatFronVactorr(p.. arraaaaaaattrdddeeee
def tracking. ee Koree_targetceee context, output: BasieVector):
0b Ootasning the control action task controller, directing it apply force along
27

---

space control, please identiffy netrice from tack_control self. CotIaputPorss"tate coatrccoottroooo (van(eent)xt)
# Only force along the yrazia is controlled, the rest sa masked out with zeros,
ee_force target up. ap.array((oo,, 0.0, 0.0, 0.0, task_control [o], 0.0]) output SetPronVacterr (e___ooce__tarreeeeea tracking.
context, output: BastcVactor):
# Directly passing the robot are's measured posstion
neeeured_are pra_iistiionnn self GetIIautPrprrr output. SetPronleactor (awarueeectrceaareee_eeeo.. (neasured_amm_poitttt def tracking.arr www.aaloccttytaaeeooo
context, oatput: BasicVactor):
Directly possing robot
measured velocity
neasured_arma arn_velocity self Iettnuttttttt ("nearaarrr"nee ("neasured_ sem_.vlooctyy").Be "err"""")eeeeeeee (context)
output.SetFronit SetFronVector (neaaured_ aaraaured_amm.eeee veloveloytt)
series data the space co then and datermine if the syaten ia running as axpacted.
{tine series of taak state and task control)
Please first think step by step about the expacted serias and detarnina if the currant series is + desired.
Then please analyze one by
any following happen.
Mismatch of order. The order of the state inconsietent nodel definition, controller definition, or be modeling the task space with [pes_a, vol_a, pes_b, vel_b], but composed the task state by [pes_b, vol_b, pos_a, case, the GPT needa redefine the state, the nodel, the
callback function. Double-check if the order consistent. For exanple,
val_a] in callbacks. callback function.
4
model, state, the + callback functions. Hake sure the signs are correct, the numbers are desired, there is missing
definition. Double-check
assignment correct
assiganents or careless nistakee, Especially pay attention to the dynamic model. Think step by step about the neaning of each elenent in the dynanic function.
Massed up rotation and tranclation arder. In pydrake, for pase, velecity, and force vectors, rotation [roll, yau, pitch, y, [x-rotation, y-rotation, z-rotation, 4 x s-tramslation, y-translation, z-tranelation)
alveya
first,
Modeling inaccurate, that
nodel involves estimated parameters and turns
+ astimation is inaccurate.
The controller paraneter is not optinal to finich the tack. For exemple, the system diverges with the controller
happens, identify paraneters variable seeignnents 4 require tuning to correct errore and ingrove the perforzance. The identified code blocks that contain conversations.
thinke any
these paraneters
updated
Assistant
for task and control, and the goals and + atructure of the control systen, let's break down the evaluation and identification proceas step by step.
Analyzing the provided systen satup, tine
Analysis
The time series of *staak_stataes shows the cart-pole system's state vactor componenta over time: cart
6 position and velocity along the y-axis, and pole angle and angular relocity. The vstank_controll
ahows compated contral actions (farce) to applied along the y-axis of
stabilize
4 pole around the upright position.
pole angle ('theta") initially decreases tovards indicating noveneat tovards upright position, which is desired. Hovever, the absolute value of "theta" should ideally approach zero more clossly over tine, stabilizing around upright pesition.
The a*task, controls force values start positive, then decrease, and becone negative, suggesting efforts to stabilize the pole by noving the cart back and forth. The force values appear to reduce in time, 4 a desirable trend.
nagnitude
night
diminishing
correction
systen stabilizea,
#*4 Expected and Actual Time Series Conparison
The time seriea appeara follow expected trenda: correctiona decrease as the aystem approaches stable state. Bowever, for precise stability and optinal control, further refined tuning of controller o paranaters and systan modaling accuracy night be nacessary.
V*A Identification of Potential Issues
MeMismatc Orderte: definitions provided seen
terns order, especially "task_state" callback, where [x, x_dot, theta, thata_dot] order natches with hou the systen model and controller are defined.
2. *+Error in Definitiones: Initial checks don't reveal abvious errors in sign or incorrect assignnents, but deeper evalustion needed when exanining specific blocks.
reHessed Up Rotation
Orderv:; Given problem scope, prinarily notion and rotation around single axis. The task controller deals with Hinear notion and doean't directly nanipulate rotations, while the tracking controller's force application aligned correctly
along the axiis.
28

---

*Modeling Inaccuracyes: Based on the provided dynamics and control nodal, the fundanental modeling approach aligned with physical bebavior of cart-pole systen. Rosever, specific paranetera (masses, longths) are accurate provided and need no inmodiate correction.
ccontroller Parameter Not Optinsl*s; Parazeters within "Q" and "R" the <task, controller> and Torce control axis mask in stracking, stracking.controllll controller> suggest roon for optinization. For improved performance, positien) and control effort, adjustnents
especially task prioritization (pole stability to these values may offar batter resilta.
#** Prelininary Racomendations for Paransters and Variable kasignnenta Tuning
potentially correct errora and ixprove aysten performance, adjustments to "Q" and "R" natrix values in sectask_ Trrcss__coattlllee>a should be considered to better balance betvesn penalizing the pole's deviation from the upright position and the energy (force) expended for control. The initial eaphasis on the pole's pesition ('theta') and velocity ('theta_dot') hinting the right direction, but fine-tuning will help achiave nore precise and energ--fffciient stabilization.
Next atepa in tuning will be directed towards revising "<tack_controllar>> paranatera with a focus on likely adjusting "Q" for a higher enphasis on reducing the pole's angle deviation and possibly refining "R" to nanage control effort nore effectively.
Mata-Control
Now please decide if the time series is desired. "yee".
please ansver If not, you muat reurite the code blocks that contain paraneters and variables that require tuning to correct errors or to improve perforaance. Do not fear to nake nietakee. Ve will evaluate the code time block, produce
and iteratively improve the paraneters.
<step_name> follosed 4 nust be included before each code block so the code block can be recognized, step_tane nust be <step_nane must one <task_nodel>, stracking_ncdell
respense
enclosed ""python"" <atep_nane>
enclosed angle coackk_onttoller,, <tracking. controller,,,,, <task_callbadh,, <tracking callback>. ccracking.callback.. For <task,callbacko, <tracking, Sxracking.callback. callback?, you can only rewrite chosen functions of the codeblock. But for other steps, make resrite the shole code black because will directly replace the original block with the reuritten one. Bо not mias any variable in the code block. baring "resriting', you are only supposed code block,
nodify the existing content Do not revise functione and variablee not pregented in the existing code block.
values assigned to paraneters and varisbles.
An example output is:
<tracking_callbaad callbacko
python
trackingg. (eeel context, output): # updated function body
controllot
python
updated whole code block
The time series is not entirely desired. While it shows that the pole's angle is noving tovarda stabilization around the upright position, the control effort depicted by task_control is exhibiting a
the pole has reached stabilized reaches and crosses 0 but starts to deviate again). The controontrol should ideally drive the systen croezover. séjustnents needed the tuning of the LQR controller's is and R natrices, shich directly affect the control perfornance by effort.
pattern of diminishing force application
(angle
precisely
balancing between state deviations
coat
<task_controllar>
** python
g disgii 1, 100, 10]) # Increased emphasis on the pole's angle stabslization ap aro Reduced for control effort allow aggressive control actions tack_controller 1gBControllor(tackkkoo Q, B, x0, u0)
## Mata-Contral
Here the tine O
apace
space control the
Now please dacide if the time series is desired.
1s, please anaver with one vord "yes".
If not, you nust revrite the code blocks that contain paraneters and varisbles that require tuning to correct errors or to improve perfornance. Do not fear to nake nistakes. Ve will evsluate the code series, iteratively improve paraneters.
block, produce
29

---

GPT should import any module. The respense with catep_rame> follezed code block enclosed "python" " <step_nane> must he included before anch coda block so the code block can be recognized. step_rane must <atep_nane> enclosed angle <tracking, controllar,,, <task_callbadd,, <tracking. callback>. For <task_callbacds <tracking, Iback,, you can only resrite chosen functiona of codablock. rearite the vhole code block because will directly replace the original block with the reuritten one. Do not miss mny variable in the code block. During 'resriting' you are only supposed values assigned to
staak_nodel>, <tracking_nodel>
for other stepa, make
modify the existing content the code block, Do not revise functions and variables not presented in the existing code black.
and variables.
An exampla output 1s:
srrecking_callbacd
python
def foee, # updated funstion body
output)):
<taek_ <taek_ controller>
python
spdated whole code black
Assistant
C.2 Pick and Place
C.2.1 Design via Templates
## Systen
GPT professor can be accomplished by soquentially executing several skills. The GPT xill help the user urite coda to compose one of the skills after reading the requirenents. The GPT will be instructed step by step. task controller and tracking controllor. The task controller ia responsible for generating high-laval and abstract control, and the tracking controllar reaponsible for tracking higb-level control eatislying constraints.
control proficient programer Python PyDrake.
skill accomplished hierarchically
optional principles design the Dasign the task controller for the objact of interest or and-affector, and dasign the tracking controller for the robot.
tracking controller:
simplified approxinated dynanics, such Cartesian space 4 approxinated linear nodel, and deaign the tracking controller on the full robot dynanics. multiple objectives, deeign objective and design the tracking controller to take care of the rest objectives and constraints. The controllar
gn the task controllar
controller
of prinary
tracking controller
output joint torque.
4. Task control is often the position, velocity, acceleration or force in the Cartesian space. But in some cases, the taek control is best to be joint torque. joint torques.
The final output the tracking
following stepe: coek_controlerr tracking controller, tack_nodel, tracking.nodi model. In the GPT's response, the nost impertant part will be <step_name> followed by code should correspond
proceed
enclosed by ""python""" Each code indicates the code is written for specific step. The GPT can feel free to nake plan help it think before writing the code blocks. There is no need to explain the code after the code block. The up arguzents,
aingla step. This axplicitly
ahould
anything
4 that is not nentioned.
## User
Please help na urita code that is part robot skill robot to reach goal position Cartesian space with collision avoidance. This sxill will be used pick and place objects "cotrollle_paraneeere paranetere port.
clutterd scene with obstacles. Goal positions will be provided in the "controller
"N Assistant
Certainly! good vriting controllers carefully and think carefully.
PyDrake.
your
#F Weta-Control
Lot's a first take look all available measurenents, dynanic modele, and controllers.
Here are all available inputs to the skill: {skill iaaillliinnutt_cm_ppaa
Here are all available dynanic model setup functions in yaml formt:: www.aaic_odeebbireefeoo
all available controllers yenl format:
30

---

(controller, beaellerrbbref_ttr
on
neasurementa, nodela, controllera, please think atep by step to
nodels and controllers:
Decide the subject of the tack controller, is it an object, the end effector, or something elee. Decide the desired task control be applied the subject, it Cartesian position, Cartesian force, or sonathing else.
Decide
space nodel that nodel dynenica
subject
analytical form of the dynamics model is needed, draz upon your extensive kmovladge in control theory and system nodeling. Think step by step to make aure nodel is correct according to the taak description.
Decide the task controller that can apply the desired task control and realize the goal the skill. Nake the controller realize the goal
the given available neasurenents. There additional inputs or functiona provided to the controller.
tracking space Dacida the tracking controller to track the output of the task controllar and to satisty passible constraints.
Detail about hov can the chosen models and controllers realize the skill goal, Be specific.
Please return the choice with the following fornat.
<tdex_nodel>
"pyyhonn
tack_nodel setup.nodel arg2)
brracking_nodell
* python
tracking.model
catup_nodellrrm.tppp. objact_inff)
<task_controllar>
"* ppython
task_controller ControllerClasssergg srg2)
** "python
tracking..omtrollar
(atrollerCClassarrg))
sure your response comply
folloring requirenents:
*
<atap_nanes nane> must he included bafore each coda block sa the code block can be racognized. stap_name must
be enclosed in angle brackets c,
Choices been nada for all steps. à
solution
the given infomation.
Pass the argunenta (args) uben instantiating the controller and the nodel. The argunenta needed are "input_porte" (such
the yanl "controller._ parameterr"") as argunents when instantiating the controllers. They are not argunents for "coatroller.parrmeteree
(arge).
not*. include
instantiating the
The argunenta should be defined before except "arm_type" and "object_info", which can be used
6 directly vithout dafinition.
any variable is dafinad. informations and available variables + needed are provided. If your choice of model and controller requires additionsl information, rethink
Do any placeholder
The nodels and controllers are instantiated correctly as stated in the naterials. raquirad
chosen
the controllers
naterials.
## Assistant
goal position Cartesian space collision avoidance in cluttered scene, let's derive the appropriate choices step by step.
goal
4
1. The subject of the task controllor will be the rabot's and-offactor since we need to move it apecific position in the Cartestan space.
2. The desired taek control to be applied on the subject is Cartesian position, aining to the end
< affector to the geal pose.
space nodel, considering
requirenents,
+ and static objects is the scens to allow for consideration of obstacles for collision avoidance. goal
generate trajectory position. Based the available options, a Kinanatic Trajectory Model Predictive Controller seens appropriate because calculatea collision-free trajectory in joint apace achieve goal in Cartesian space.
coatroller
The tracking space model sill be the nodel the robot are alone as the tracking controller's purpose controller.
follow trajectory planned
6. The tracking controller should implement the folloving operation of the robst arm's actuators based on controller's output.
night be controller directly joint positions and computes necessary torques, such Jonnstiifneaatotrrrllle:
Here's the detailed instantiation:
31

---

<task_nodel>
"" "pythonn
task_nodel
wwt.m//wwhh...jtt...oooloooooood (oject_info)
<tracking.nodell
** "pythonn
tracking_nodal satup.arn_nodal sattp..am..daaammm..ppp.
<task_ controlle>>
"" "pyyhonn
tack_controller
sbanacchhhsainnaaaaaarraeeeereed resolve_poried-9.00 aun_stops=20)
<tracking.controlleer
python
sracking.comtrollk
# Meta-Control
output running
Hunan export supplanentary instructiens:
Do you think it is vorking as expected? (Eapty output means the system is running without errors) word "yes".
please ansver
If it in not, please identify shich stap(s) vent vrong and explain how to fix it by including which
step(s) to modify (<step_name>) and the updated code (enclosed by ""python"""" "), We have proceeded the following steps:
task_nodel
tracking
task controller
Shackinggcentrollne
Please do not nakeup an inexist step.
If the systen is not working as axpactad, you must modify as least one step. You can nodify mitippe add "<step_oste>" before corresponding with the given infomation. Try your best.
stepe.
solution existe
Here is sample response shen it is not working expacted:
I think this occured bacause [sone reason].
<tracking. controller>
updated_code:
ppython
Kertrackinngoattrollareerr arg2)
# Assistant
yes
# Meta-Control
Please sumnarize choice
tracking controller. sould you expect input output ports of the controllere will be connected to perform the skill. This response will be used proupt implesent port connector, so please include necessary information.
including:
The detailed description of the input for the task controller and the tracking controller, including the neaning of each dinenston (ba careful about the ordar), and haw to get then from the available neseurenents outpute.
description of any constant paraneter have used that may affect the performance of the control systen.
Everthing else think is necessary implement the controllar by only reading the sumnary.
## Assietant
For the task controller, ve chose the **Kinenatic Trajectory Nodel Predictive Controllerts:
#** Tack Controller Xinenatic Trajectory Hodel Predictive Controller:
#2#2 Input: Porta:
32

---

wecnnrolllr proatrollor.parreeteeeee Cones fron the skill inputs under the 'controller, paaerooller..paeeeeeeeeee contains information about the desired goal posa for the and affector, expressed in the forn of "Rigidrransform" shich speciriea target poaition and orientation Cartesian space.
It
reneauured crm.posttiones inputa, representing the current joint positions of the robot
imput
connected
'reasured ara.poatioon from
eeeasuree_arre voloortt This imput should be linked the 'neasured amm_oioctttt from the skill inputs, indicating the current joint velocities of the robot arn.
+
#**# Notable Paramaters:
sereeolve_periieee This parazeter affects hov often the XPC recalculates the trajectory. lover value will nake the trajectory nore responsive to changes but will require more computational resources.
**mun * batermines the nunber of discrate steps within the control horizon of the MPC. A higher finely detailed trajectory
number steps allovs
computationel demanda.
**# Tracking Controller Joint Stiffness Controller:
#888 Input Ports:
1. secontroller, paraneters**: parantttse:tp.raattrrrs::::: While this specific tracking controller (Joint Stiffneas Controller) does generally
inbereatly require "controller_ prrranler.prranteerss epecified accepta dynanic adjustnents auch as 'jifnt_atifffesa! and joonin.dnppig'. which can be applied via an AbstractValue laput if controller custonization is necessary during runtine.
2. Harn, parget_pesitiom::: Should sired to the autput of the task controller (Kinematic Trajactory Nodel Predictive Controller), which generates target peaitions reach the desired end-effector pose in collieion-fres nanner.
should follow
3. warr. posittosition::: This is connected to "ncasured_arn_pon" positionn from the skill measurements, chich indicates the current positions of the robot arm's joints.
Harn_ velocity::: This is connacted to "neamred__am_veeestt from the skill measurements, providing velocities of robat jointa.
Parameters:
Specific paraneter settinge like joint stiffness and damping coefficients (if they were to dynamically adjustable through "controllar, parannarnttrrr' affect hav the robot physically bohaves its compliance and responsivanesa to control signala.
VM Implementation Details:
For affective contral, the output of the *<Kinamatic Trajectory Model Predictive Controllerss (target joint positicns) is connected to the input port for rearm_targe sarget_positiont of the **Joint Stiffness Controllertx. The neasured positiona and valocities, which essential for both controllars, cone directly from the robot's sensory reedback.
Additionally, the task controller's 'controller paraneters' imput is based the skill's requirenent (o.g., collision avaidance reaching goal posa), shich includes spocifying the and-affestor target pose that must be extracted fron the catrroller.prrnette skill Input.
This combination
planned, colliston-fres trajectory reach the targat position in Cartesian space by adjusting its joint torques according to the deviations from the target joint positions computed by task controller, considering current poeitions accurate and efficient control.
the
C.2.2 Composition via Semantic Understanding
System.
GPT proficient programmer Python PyDreke professor control. The will help user to accomplish the code after resding the requirenents. The GPT will be instructed step by step so code implements
please only complete nentioned
skill through hierarchical design. A task controller is chesen to give high-level control, and tracking controller is given to track the tracking controller will given. The GPT needs to sccomplieh task controller converter and tracking controller converter. A converter traneforne available neasurements and parts to the required inpat ports the controller.
the task control.
task controller
The GPT xill proceed the following steps: tack_callback, tracking.callback callback. In the GPT's response, the inportant part
<atep_nane> felloved by block abould correspond to single step. This explicitly indicates the coda is urittan for specific step. The GPT can feel free to nake sone plan help it think before kriting the code blocks. There need explain the code after the code block. The GPT should import any module. The GPT should not make up argunents, variables and anything alse that is not mentiened.
enclesed by """ython"" "1, Bach
33

---

# User
Please help na urite code that is part of robot skill for a robot arn to reach a geal pasition in pick place objecta clutterd scone with obstacles. Goal positions xill be provided in the "controller "controller.praanee""" paranetere port.
Carteaian space collision avoidance. This will
# Assistant
Certainly! I'a good at vriting callback functions for ports in PyDrake. will follow your instructions carefully and think carefully.
Neta-Control
Now please implement callback functions of "LeafSysten" that acts as connector which have already constructed. The input ports correapond to inputa skill (observations and controller pareneters). The output ports correspond the input of the task controller. Here is yaml file describing what the inputs to the skill are:
print port nane, type and semple port "value self self wtw...GttnnatPPotppprr...c paort. Eval (context) when inpalomenting the call back functions:
input port. You get
the
(empul_port_emmple)
The chosan task nodel, tracking model, controller, and tracking controller "ppytoon
tadk_nodel aetup.arn, vitup.arn.vtth..oo obtjortttttottttoo.oooooo model (arn, (arn_type, object_into)
tracking.nodel (orppaarr
cack_controller sieetsccaaccceeerrrrrddottaaarrd resolve_poriod-3. stn_stop=)
tracktng.controllar Lanettrfaanddeelllleloaaaaararaa
Here is the dasign aumary to explain the expacted of hoz do the controllers works, and haw connact the ports:
(dmmign_cummar) (fron Besign via Templates)
Here sunnary
input porta
controller:
caack_controllrr.rm conttrolerriaputteee. htrp__ippuuuaarrrte (frem the controller summary)
The functions you need to implement are:
python
def www.roolll..prraneerrr....e..... pass
output: AbstractValus)
def mew.urredarrappoeeeee.o
context, output: BasicWector)
pass
def task back_neasured www.:aanrrood.a..oocctoo....o.oe wrw.oooiiityytsllf context, output: BasicVactor)
pass
exampla response:
<task_callbachb
*** python
def Slace_bolded
htw.eoe..mmttioaeeelllllle fumction(eeff,, context, output: BssicVector):
pestract_iaput_pee porp__orrr_ll value
(contex))
output, SetFronWacter (abstract_ (abstract_iippat, inpat_port__vat "art_vaale""plac__ael paplacc_holder__oae
def placa_holder. here___odde_rreeeeeeeeeeeee_eedd www.eroromncttttteeeeooeeeeeeeee fumetion2(seirr, context, outpat: BasicVactor):
pector_input.port.
self. petInputPort(port.coe Eval (context)
out (aroa()))))
out[3] pector_inptt_ppe (ortua
output, GetFraaVeetor((ut))
blace_oolder_fucccionnieeee context, output: AbstractValue):
value = self. self. Cat.nnpttoort.port.mmo Eval (contaxt) output. wet..eeee_tteeeeeeeeeeee
Please eeplenent all the task callback functions in one code block begin with <tack callbacks,
Here are some tips to implement the callback functions:
port correct port avialble Sone the need to modified.
port, function.
the
Note that some of the ports should be passed through without any modification like control paramttrr.
34

---

Make the dimensions of the output matches the port. The output value may only partially available from imput port. You need to decide the value for the remainings. Be careful which complies
dinenaion value, nake aumaary.
description the port
pose, valocity
vactora, rotation alwaya [x-rotation, y-rotation, 2-rotation, s-tranelation, y-tranalation, e-translation].
firat,
[roll, yau, pitch, x, y,
If the imput port is dictionary, make sure only use keys that has shown the printed pert value
sample.
If the output should be a amun typa, you can use the amm type nantioned in the controller sumnary
directly without any import.
Recall the task requirenents given in the beginning- Make sure the call back functions corractly solve the task.
Please avoid using placeholders simplified examples that require modifications by othere. Ensure all calculations are conclusive. In case of any uncertainties about parametere, utilize the information provided derive the most informed inforence available.
You can use self-nun_q to denote the nunber of jointa
Please implement the call-back functions. Please think step by step then write the code.
Assistant
<task_callbacko
python
tack parlneeereeee,,, context, output: AbstractValue):
# Directly the control Lerrooteeeeeeee Ler_peruneters from the suput to the output. parens self GetInputPort "ootrroller.paraneeert Eval(context) sepptteet_raabue (parans)
def task measuneddddddda aaa.uuad_.annpoait arn_poattionseell context, output: BasicVactor):
# Directly pass the measured, aauureddannpppeeeeeee posittooiion from the sxput to the oudput.
GetInputPort arn output SetFronVacter mmasurre__em_posttee.
arn prsppion
Eval(contert)
def task_nestee meakunesddedddar (see
output: BasicVector):
# Directly pass the measured, aaasured_ann, ane, velocity from the input to the output. velocity
(ntpuuuoott"eeesurerr vellloccttt))) Eval output (essuud (ueasured_ arm velocity
Meta-Control
Now please implement callback functions "LeafSysten" that
connector which have already constructad for the tracking controllar. The inpat porta correspond to inputa to the akill control (the output output ports correspond to the input ports of the tracking controller.
(observatione controller paraneters)
controller).
Recall the input part of the skill given previously, and the follaving is description of the task control port. You can use all skill input ports and the task control port.
yanl
deacription: Targat joint positions for the robot amm.
nane: taok_q_oom
size: nun_q
type: BasicVactor
Here is the sunnary of the input porta of the tracking controller:
yanl
description: Input port for dynanic adjustment of controllar paraneters, such "joint_stiffneat"
'joint_damping'.
name: cestroller.paranttee
type: AbstractValue
deacription: Target position
nane: ton.taagett.ooition
type: BasicVector
description: Current position of the arm.
nane: arn_poaition
type: BasicVector
description: Current velocity of the arm.
arn_velocity
type: BasicVector
The functions you need to implement python
prackinn.cottelleer.paaeeeeeee context, output: AbstractValue) pass
def tracking.arn.ccn ttt.arrrrrr....oattioaaaaatt context, output: BastaVactor)
pass
35

---

def tracking.arm, am..pasiiiooeeefff output: BasicVector)
pass
def tracking.ar vm.ooooittyalllll context, output: BasicWactor)
pass
Hare is an exampla response:
<trackinggcalllaccb
python
def
(el,,,,,, context, output: BasicVector):
abstract abstract_input_por value self place_hoeee..eeeppppp_ Eral (context)
output. SetFronVactor (abstracc, (abstract_input. inp.tt.arttttttauulll polppaa_olldrr.eeeee def honlddrrfumciin22e6 function2(self,, contert, output: BasicVector):
vector intorr_nuutt.por..eee
self place_holder, wtw.c./oooeer.pt...aaaaoooooo (canecet))))
out np. zeroa(6)
out[3] vector supat.port.vvauee
output. BetrreaVecerrrr (out)
def place_holder wtwe__ooodr__uuciioon33333. function3(selff, context, outpat: abstractValue);
valua salf soice_holdrr_prtt (calttot))))
sntput.aet__eee aet_value(vaaue)
Follow the previously given tipe for inplementing the callback functions. Further more: Noto that output ports the task controller have been renaned adding prefix: 'task_' port_nane.
Now please implement call-back functiona. Please think step step urite code.
serracknng.callbace
python
def controlllrrrrrooooooooocoooceeee wtw.errrllle..prrreeeereeeeeeeee (aren context, output: AbstractValue):
# Since the specific tracking controlter (Josnt Staffness Controller) sight not isherently require way
prisery function, # we can decide to either pass defasil configuration or dynanically adjust the paraneters if needed. dynomically adjust paraseters
"controller_ centrolter.pannntters
controller does
the skill saput. ("coatroller.pparaceeeee"
cottroller.paranetee output set_
self Pocclnnntootttccootttt ("controller paranetars") "aaaanttrr")"_allllllla Erall (context)
def tracking.arg arreiinggarn_ttaeee wtt.aaa.aree...ttttnaaaall
context, output: BasicVactar):
This shouid be wired the output of task controtier, provides target joint posstsons. tack_ poea = self (ttpuuttPrrttttt.q.... Baal output SetFronVactor (tatkcttttttttta.oonooo
def tracking.are ara_pasitionteeff, context, output: BasicVector):
Pass the seasured pro_pocition from the ieput output. neasured_ aorddarnn_ppol arn_position self, GetInpttPort (epputooorttnsaareerr output SetFrosVector )mtiiition
tracking. (sal context, output: BasicVactor):
# Pass the sessured, ues_wtt from the saput to the output.
neasured neanured_arn_velocti. velociiy self GorInnutPort ("neasurdd www.urrdddran_elleeeee vellooccitty) (ranltt output. SatFroVVactor (neauurrd aaaaurad_amm_vllccty
## Meta-Control
output running
Reached the cube! Suitch to stage
Grasped the cube! Svitch to stage 2
Hunan expart supplanentary instructions:
Do you think it is vorking as expected? (Eapty output means the eystem is running without errors) sord "yes".
is, please ansver If it is not, please identify which step(s) vent vrong and explain how to fix it by including sbich
atep(a) to nodify (<atep_paze>) the updated code (enclosed by ""python"""" "), have proceeded the following stepe:
task_callback
callback
not nakeup
stop.
the system not working as expected, you must modify as least one step. You can nodify multiple steps.
36

---

For each step, you only nodify the function that think is vrong. Others will be kept unchanged. Renanber to add "catap_names name> before the corresponding code block. A correct solution exists with the given infonation. your
sample reaponse khen
working as expacted:
I think this errer occured because (sone reasen).
updatad_code: code:
<tracking.callback
python
def tracking.arn prel context, output: BasicWector);
## Assistant
yes
## Meta-Control
Please sumnariza cakk_controllrr, sracking_controlll task_callback and the tracking.call callback Explain 4 the data flos. This response will be used as pronpt to choose the performance metric from nessurements,
pareneters, eo please
necessary information.
Assietant
The robot skill implemented involves controlling robot
reach goal position Cartesian space shile aveiding collisions, shich is particularly useful for tasks like pick and place in cluttared To achieve this, hierarchical control approach adopted, consiating and tracking controller, interconnected through callback functions that serve a8 data converters.
controller
#*# Task Controller
**Type:** Kinenstic Trajectory Model Predictive Controller (aaemnttcccieccto/oolllrrrrrtttr
collision-free 4 trajectories to reach the target end-effector pase, given as goal in the Cartesian space.
**Role:* controller responsible high-level decision-naking
**Input Ports:**
*cconroller_ppe sraatroller.parmneeerr- Receives dynanic controller paranetere such as the desired goal pose for the and affector (expressed "Rigidtranaform" specifying target pasition and oriantation) and possible grippar connanda.
peneeurelanm.ooeltions Received the joint positions robot 3. aeearured_aar arn, wrr.oiii...a...ootttt:.......... Receives the current joint velocities of the robot are.
**Notable Paranatera:*
*eresolve_periot Datermines how often the controller recalculates the trajectory.
*tmun_atops:** number of steps the control horizon the MPC, influencing the granularity of the + planned trajectory.
*** Tracking Controller
**Tуpa:** Joint Stiffneas Controller (oontBBiiffneadoottrrolll
**Roles*e This controller with lovar-leval enauring that robot's jointa follow + target positions generated by the taek controller to execute the trajectory accurately.
*>Input Ports:
#*controller.
earanntarrs:* coontroller.parantterr...
necessary, receives dynanic adjustnents as "joint_stiffnes"
goint_ddapping
Harn_ Marn_targat, pergat_pesition=* Receives target joint pasitions to follow, as computed by the task controller. robat's sensory feedback obtain current joint positions. **rn, velocityc Also connected to the robot's seneory feedback for current joint velocities.
Marrn.ossiiio:: Connected
### cask.callback
Tbis callback functions prepares the data fron the robot's sensory impute and controller paraneters ( for the task controller. This includen:
*stask noutroller.parantttrn Directly forvarde the "peasured, "peasured_ermpoeetiie prr_toiiiion to the task controller. "ttark_eessured arn_pocition:** baookk_neasurd_arr.. vrooooccty. Directly forzards "neasured arr velociiy
"controller, paraattrrss directly
controllor.
tracking.cal callback
This set callback functions converts outputs from the task controller and zensory inputs into the required by tracking controller, This includes:
*stracking conpiiig..otttoleee.o. www.rollee..paraeeeeeeooooeeeee. controller.
Optionslly passes dynamic controller adjustnents to the tracking
*stracking haapttng_rn__trrraee www.yrrgtnttttpppoooo posttio.os Forvarda the target joint positions from the task controller to the
tracking coutroller.
37

---

GPT should import any module. The should wrackingg Baacckingamm.ppattiion Passes the joint pesitions fron sensory feedback the tracking joint velocities aensory feedback the
controller.
*ttacking vravooctt..
tracking controller.
**# Data Flow
The data flow begins with sensory inpute and dynanic paraneters being fed into the task controller via the callback functions designed in "taaik_callback'. The task controllar uses this data to compate a the goal pose. target joint positions generated by contraller are then passed to the tracking controllar through trackinggccallban callback'. Additionally, the current joint atates are also provided to the tracking controller, enabling to compute how to nove the robot's 's jeinte to follow the planned trajectory accurately.
collision-free trajectory
This bierarchical and modular design allows for high-level planning and lox-leval execution be optinized individually, shile the callback functions ensure they are conpatible and can work together
seanlesely.
C.3
Open Door
C.3.1
Design via Templates
## System
GPT professor
control proficient programer of Python PyDrake. can be accomplished by soquentially executing several skills. The GPT xill help the user write code to compose one of the skills after reading the requirenents. The GPT will be instructed step by step. task controller and tracking controller. The task controller is responsible for genarating high-lavel and abstract control, and the tracking controllar reaponsible for tracking higb-level control satislying constrainta.
skill accompliched hierarchically
optional principles design the 1. Design the task controller for the objact of interest or and-offector, and dasign the tracking
and tracking controller:
controller for the robot.
simplified approxinated dynamics, such Cartesian space C approximated liner nodal, and dasign the tracking controller on the full rahot dynamics.
Dasign the task controllar
skill involves multiple objectives, desigu objective and design the tracking controller to take care of the rest objectives and constraints. The controller the tracking controller
of prinary
output joint torque.
4. Task control is often the position, valocity, acceleration or force in the Cartesian space. But in some cases, the task control is best to joint torque.
The final output the tracking controller must be joint torques.
following steps: meek_controleer, trackingg controller, tack_nodel, pracking_nodel. In the GPT's response, the nost impertant part vill be <step_name> followed by code Each code indicates the code is written for specific step. The GPT can feal free to nake sone plan help it think before writing the code blocks. There is no need to explain the code after the code block. The up arguzents, variables anything
proceed
enclosed by ** ""python""
correspond
single step.
axplicitly
+ that is not nentioned.
## User
Please halp na Coordinate Syaten Orientation: The x-axis points forward.
code
designad to poll open door.
y-axia points The 2-axis points upvard. Currently,
euccessiully grasped handle. vieved above, the robot base is at [0, 0], the door handle's position is approxinately [0.8, 0.2], 2 the hinge (rotation axis) positioned [0.8, 0.2 radiual, with "radiua" being handle's rotation radius, shich fluctuates betseen 0.3 and 0.5 netere, and the door opens rotating anticlockvise around the binge. Therefore the vector from the hinge to the handle is currently [0, radius), The which should renain unchanged during the pulling process. uncertainty in the radius, the skill should not only deternins rough trajectory to pull the door compliant
closed,
height the
open approxinatoly degrees, procass.
danaging the during pulling
# Assietant
Certainly! I'm good at vriting controllers for robots with PyDrake. will follow your instructions carefully and think carefully.
Mata-Contral
Let's first take look of all available neasurementa, dynamic nodels, and controllers.
Here all available inputs the skill:
(skil) inputs_cample)
38

---

Here all available dynamic model setup functions in fornat:
ddynamic_nodel. modee__biafffttt
Here are all available controllers in yanl format:
fcontroller arir_brreefaar
Basad on the availahle neasurenents, nodels, and controllers, please think step by step to choose the
nodels controllers:
1. Dacide the aubject of the task controller, is it an object, the end effactor, something else.
Decide the desired task control be applied on aubject, It Cartesian position, Cartesian
force, something elee.
3. Decide the task space model that can model the dynemics of the subject with the task control. Khen the analytical form of the dynamics model in naeded, draz upan your extensive knouladge in control
+ and systen nodeling. Think stap by step to nake sure the nodel is correct according to the task description.
Dacida the task controller that can apply the desired task control and realize the goal of the skill.
Nake aure the controller can realize the goal with the given available meseurements. There is no additional inputs functions provided the controller.
Decide the tracking space nodel.
Dacida tracking controller track the outpat constrainta.
task controllar and satisfy passible
+
Detail about hov the chosen nodela and controllers realize the skill goal. Be specific.
Please return the choice with the following format.
<task_nodel>
"pyyhonn
task_ eetup_nodelaargl arg2)
<trackinggnodell>
""python
tracking.nodel
aetup.nodellarr.typpe object_infoo
<task_controllar>
python
task_controller Controllerdlass(argy. arg2)
brrckking.ontrolller>
controller>
* python
conckiin.cottoolar CantrollerClassarggll
Make sure your reaponse comply with the following requirenenta:
included before -o be enclored in angle brackets o,
<etep_ nane>
block code
recognized. step_name
steps. correct Pass the argunenta (args) uben instantiating the controller and the nodel. The argunenta needed are stated in the yaml file (args). +Do not** include itens in "input.ports" (such as
given
argunents when instantiating controllers. Thay argunenta for 4 instantiating ing the controller,
"controller, paraaarrrraaaarr")
argunents should
use, except "arm_type" and "object_info", which
+ directly without definition.
use any placeholder asaura any choice nodel and controller requires additionsl information, rethink
needed are provided,
< about it.
nodels controllera instantiated correctly stated naterials.
The chosen model natches the required nodel of the controllers stated in the naterials.
## Assistant
accomplishing robat destgned pull open door, let'a the atructured stapa the appropriate models and controllers.
**Bubject
end-effector because the task involves pulling the door open. Given this action is highly dependent on the position and holding door handla), precise
Controller4:; subject
controller
orientation the end-effector
case, control over ita Cartesian pasition crucial.
+4Desired Task Control=+: The desired task control will be the Cartesian position. This choice is because the robot needs to maintain a certain trajectory to pull the door open effectively. The task (due
involves noving specific psth approxinates segnent around its hinge) while keeping the height constant.
39

---

+Tfack Space Model+e; Since the dynanics of moving the door handle be simplified to the Cartesian space (ohare the robot needs to fallow a spacific trajectory to pull open the doar), space model because the spacific dynanica aren't relevant to computing the desired ond-offector trajectory.
placeholder model auffices
*4Task Controllerse:
The Gartesiaalltrrrollttoooooott trejectory for the end-effector to reach the target position, considering the need to pull the door. points, aligning with the need adjust
is apt for the taek because it generate
towards the + the palling trajactory according to the door handle's position and maintaining compliance.
This controller interpolate
6. reTracking Space Modalxe: For the tracking model, since the robot's interaction with the envirenment
(1.8., the door) also needs consideration, especially enauring that notion does lead to excersive force shich could danage the door, should utilize the "setup_ara" wtt_aan___tttttoteeea tit___eeeeeectt modell
This model allovs for considering the robot arm and static objects in the scene, facilitating avoiding inposition the door during pulling.
unnecessary
s<Trecking Controller==:
the requirenents force to accommodata the door's opening nation and ita varying radius, CartestanStirfnesstoottt suitable. This controller can regulate forces exerted by the robot, naking the interaction more compliant and safer, which aligns with the goals of being compliant and avoiding danage.
compliance
potential adjustmenta
pulling
Let's instantiate the chosen nodels and controllers with provided argunents.
nodel>
ppython
tack_nodel setup.placeholder_.ooo
<tracking_nodel>
""yython
tracking oettg..t..iiibbbhejjeee..e.e.ee object_info)
controlllo
"'yython
task_controller
<tracking
python
IN Yeta-Control
Please sumnarize
tracking controller. sould you expect input output ports of the controllers will be connected to perforn the skill. This reaponse sill be used as proupt to implement the port connector, so please include all necessary information,
including:
controller and tracking coatroller, including the meaning of each dimension (be careful about the order), and how to get then from the arailable neasurenents outputs.
description
input ports
description of any constant paramater have used that affect the perfornance of the control systen.
Evarthing alse you think is necessary implenent controller by only reading the sunnary.
To impl emplement the skill of pulling door with our robot, we've selected anta
the controller w as the tracking controller. Below the detailed information their input and hov these controllers will be intercomected to achiave the desired skill.
**# Task Controller: Garteesieinterrplaiiiaaooooooood
*>Input Ports and Their Sources:**
#ccontroller.
pooatrooleer.praatteeeee paraneters::: This input contains paranetere for trajectory generation like "new trajectory" flag, "say_points" with "tines" and "points", and "exacution_tine These can be directly sourced from "controller, paraneterarrrerr provided
inputs.
remeasured_od po_seree:
[roll, pitch, yau, x, 7, z]". This comas directly from "maaurod_es_pose"" in the available neasuraments.
pose
effecter
3. sesxecation times::
This input is required evaluste the current position along the interpolated trejectory. It should be calculated based on the current time vithin the execution of a trajectory and could require separate tracking
utilization systen/skill
netadats.
target position [X, Y, darived fron controller peranetara under "way-points" or adjusted based real-tine neasurenents conditions, including door hinge position and estinated radius.
Snnarget_pasitionee
Initially,
40

---

++Output Port:**
-eee_pose_nonee output providea (deaired) end-effector pose [rrll,, pitch, yas,
z] based the interpolation of the saypoints over the given times. This will be used as input the tracking controller
execution.
**Input and Their Gatrossaahttfffeesoottroo..r
#** Tracking Controller:
target (from taak, controller's output)*x: The deaired end-effector target that includea both position and orientation in the form [roll, pitch, yeu, y, 2]", This from the ee__osee_oot output part of our tack controller.
2. *488, Mea__argee saarat__typaae For this inplenentation, se need to specify that our target type ia a pose than
(position orientation), ixplying the controller should focus tracking the pose valocity or force. This is constant paramater based on our control stratagy to pull the door open by following a pose trajectory.
3. poritionition::: Current joint positions the robot arm, obtained from aannurrd_amm_oositiinn in the available
velocityes; the neasurements.
joint velocitiee
arm,
"neseured aneeureedann_eeoociig in
5. +econtroller,
Paraneters such certesian_stiffeeee and "cartesian, danging that define the compliance characteristicn the control. Values like [50, 60, 50, 500, 500, 500] for stiffnans 40]" damping adopted the "controller, parranttr.a the
caaerrooleer.pprneterrrrte parameterst::
and [s, 5, inputa to suit the compliance and control behavior requirenents.
**Outgut Port:**
*tapplied_are araiiidurn__trquett The calculated torques be applied the robot follow the < dastrad trajectory whila maintaining compliance with the inposed stiffness and damping paranetars.
### Interconnecting the Controllers:
The output of the wabartccininteerpaatttoooottt Saaarreeisstifffessoottroleeee
( ('es_pose_non" should be connected to the "ee_target" imput port, This connection is crucial for the execution flow vhere the interpolated pose trajectory (dasired beharior) informs the compliant control, resulting in the actual joint torques appliad to the robot's 's actuators.
Parameters such as 'ee_ target_type and "controller, cootrollee.pprreetrrr for the #CCressiantsffnnssotttllll carefully determined danage) and precise enough to acconplish the task of pulling the door without requiring forceful interactions.
ensure
compliant prevent
By carefully adjuating these paraneters and ensuring accurate port comnections, the robot should be able adjusting
pull door open by folloving desired trajectory, real-tine dynamics and constraints of the task environment.
bseed
C.3.2 Composition via Semantic Understanding
The GPT is proficient programer of Python and PyDrake and a professor in control. The GPT will help the requirenents.
accomplish the code please only complete mentioned tasks. The code inplements rohot skill through hierarchical design. task controller ia cheaen to give high-level control, and tracking controller given track the task control. The task controller and the tracking controller vill given. The GPT needs to accomplieh a task controller converter and tracking controller converter. A converter traneforne required input ports
reading
will instructed step step
available
ports
controller.
response, nost inportant will be <atep_nana> folloved by a code black enclosed by ""python" "" Each code a apecific
proceed folloring stepe: talk_callback, macking.callbact
block should correspond to single step. This explicitly indicatea the code vritten help it think before kriting the code blocke. There no need to explain the code after the code block. The GPT should not import any nodule. The GPT up arguzeute,
step. The GPT feel free to nake
sbould
anything
nentioned.
Please help na urita coda for robot akill designed to pull open door.
Coordinate System Orientation: The x=axis points forward. The y-axis points to the left. z-axis points upvard.
41

---

Currently, the door closed, and the robot has successfully grasped the handle. When vieved from above, the robot base is at [o, 0], the door handle's ponition is approzinately [0.8, 0.2], the hinge radiual,
"radius" being handle's rotation radius, which fluctuates betseen 0.3 and 0.5 neters, and the door opens by rotating anticlockrise around hinge. Therefore vactor from hinga to handla currently [0, radiua]. height of the door handle 0.5, which should rensin unchanged during the pulling process. Given the uncertainty in the radius, the ekill should not only deternine rough trejectory to pull the door open approxinatoly 90 degrees, but also compliant avoid danaging the door daring the pulling procass.
(rotation aria) positioned
## Assistant
Certainly! good writing callback functions for ports PyDrake. will follow your inatructions carefully and think carefully.
Meta-Control
Now please implenent callback functiona of "LeafSyater" that acta as a connector which have already (observations controller paranaters). The output porta correspand the input porta of the task controller. Here yeml file describing what the inpute to the skill are: tekill_input_e Lnput_eummary"
constructed. input porte correspond inputa
Hera print the port nane, type and sample value for all the input part. You Eval (context) when inpelementing ing the call back
value the
+ port by "value self self
functions:
(anput.port.sampla)
The chosen task model, tracking model, task controller, and tracking controller ppython
task_nodel eetup.plaaceoldeer.seellll
tracking setup.ar..wibb.cjjeees...oo
object_info)
task_controller Caressaeeteppttttaooeeeeeeeeeeer
tracking.controller farretatsssttacccttrralaaaaaaaae
design sumarry explein expected way
works,
by the ports: (design_rummary) (fron Design Templates)
Hore is the sunnary the input ports of the task controller: wtwee/oooooolllrrptttt..oo aumaary) (from the controllar sumnary)
you implezent
python wtt.:/octrrollrrrrrraanrrraaaaae context, output: AbstractValue)
def task maak__nasurdd..oo (sale, contaxt, output: BasicWactor)
pass
def back_execution,tinn (inlat,,,, context, output: BasicHector) pass
def task tww.taat..pprttonnellllllt context, output: BasicVactor)
pass
Here an example response:
<task_callbachb
*** python
def place_ halder_ functionnetn, context, output: Basic/ector):
abstract_input._g intpt_ppotttttttt..ooopppp
porettt_oottttteeeeeeeeeeteeeeee value
(contee))
output. output.SetFrroVaattnn SwtFronVactor (abstract, (abstract__inpat.tt.. sartaactt__pat__pp___p witptt_port_vaateeetepppp "irt__aaleppllaeohaaa sallacc_hllde__eeeee._
def place_bolder
htwp.booddr.ftciieeeeeee.. ioneion context, output: BasicVector):
vector_ingut_pootttuee
self. GetInputPort (port__aa)) Eval (context)
out zp.zeros(6)
out[3] vector_input. _nputt_port.valuel0
SeaputtSetFro//ectoorc (out)
(out)
place_holdor. sfalddrrffncciinn(saa... context, output: AbstractValue):
value = self. self. www.aaaaaaorrtpprrtttooo Eval (context)
sutput.set..vlluevaall
42

---

Please implenent all the task callback functions code block begin with <task, callbacks.
Here
inplement callback functions:
*
Whan use port, nake sure the nane of the part is correct and the port is avialble in the current
function.
Note that sone of the porta should be passed through without any modification like control, par.reter...eeerr.....
-
Sone of the ports need to be modified.
Make the dimensions of the output matches the port. The output value ma only partially
available from input port. You naed to decide the value for the ramainings. Be careful which
dinension
value, nake
complies
description the port in
aumaary.
For pose, velocity force vectora, rotation always comea firat, auch aa [roll, yau, pitch, x, y, [x-rotation, y-rotation, z-rotation, s-tranelation, trtanslltioon, z-translation).
If the input port is dictionary, make sure only use keys that has shown in the printed port value sample.
* If the output should be a anun type, you can use the emm type nentioned in the controller aumnary
directly without any import.
Racall the task requirenents given in the baginning- Make aure the call back functions correctly solve
the task.
Please avoid using placeholders simplified examples that require motifications by othere. Ensure all colculations are conclusive. In case of any uncertainties about paraneters, utilize the information provided derive the informed inference available.
* You can use "self-nune nun_q" to denote the nunber of joints
Please implement the call-back functions. Please think step by step then write the code.
# Assistant
<tasx_callbacko
python
pat.zoolles.pprnnterreeeeee context, output: AbstractValue):
# Simply pass through the controller parameters from the input to the output
controller.peee paranns self GetIaputPort coottroller.pareeeeeeeeeeeeeeeee (val(centext) output. wwl
task nensured ee, paae pa..aeedle.ppaeeelr,, # The output directly maрs from the sapul measured ES pose, nodefication needed "Bra
context, output: BasicVector):
co_pose output. wwwFronVaattorreeeepooooooaaa
(caltcoot)
def task (recutiio_timm(aeeff,, context, output: BasicVector):
# For this placsholder, we staslate a staple enscution time. In practice, thsa calculated
the eystem # Here we just oudput a constant value, in real applicatione, this needs to be dynomic. [1.0] placehoider output. SetPronVactor (exeectttnectt
trajectory progress.
execution,
for execution
def task www.starrttttttteeeee pssitionnself, context, output: BasicVector):
# Cateviate the target posétson based on the door's Aendle information GetInpatPort ("reessrredddddbbree body_poses")) " (val(contextt) handle_pose neasured_body. htweeedddddd___ppo__eeeee 'bandle") ' translation()
nessured_ bessuured_bdd__poo bosee
Given the task to open the door, Let's 's assune st suinga about 90 degrees. # Ve know the door's handle as roughly at [0.0, 0.2] when closed.
stmplsfscation, during suseg, radius # Hence, ae adjust the target possison to minie the door's handle at about 90 degrees open. fine-buned based on
constant, but doer ratates.
This aseplsficatio
mechanics.
Ascuning average radéns for caleutation aseplicity. radius (0.3 0.5) / targat_x 0.8 avg_radius Subtract rugiua because we are rotating antsclockwise target_y avg_radius Assune handle stays
height
target_position [target_x, target_y, bandle_pose(2]] Maindaining handle's height output. Satrrraaaeetteetttrageeee (tareeooooo
## Mata-Contral
please implement callback constructed for the tracking controller. The input ports correspend to inputs to the skill control (the output
"LeafSysten'
which slready
(observations controller paraneters) output porta correspond to the input ports of the tracking controller.
controller).
Becall the imput port of the skill given previously, and the folloving description of the tack control port. You can use all skill input ports and the task control port.
yanl
description: Hominal end-effector pase based on the trajectory evaluated at the current RPY_XYZ
name: task_ea_ppaa___
size:
type: BasicWector
43

---

Here is the sumnary of the imput ports of the tracking controller:
yanl
description: Desired end effector target (poze wist), [roll, pltch, yau,
y, z]
ee_target
size:
type: BasicVector
dascription: Type of the and effecter target. This type is anun. It must be EndEfffactorTargt..eee kPase EuddffectorTarget-TTWiss Do not use sat",, use enun directly.
ee_target__yy
type: AbstractValue
description: Current joint position of the robot nane: arn_position
nun_q
type: BasicVactor
deacription: Current joint velocity of the robot arm.
arn_velocity
size: nun_q
type: BasicVactor
deacription: The controller can be paraneterized through an abstract input port "controller, paranttere zodify bebavior during runtine, alloving changes cartesian_stiffmess and "cartesien_dampin" matrices. [roll, pitch, y, 2]
example_value
ceraasan_stiifneee [[0.1]+3 [300]
controller.
pearoollr.paranetars paranetera
type: AbstractValue
The functions you need implement python
tracking.eo.targereeell context, output: BasicVector) pass
def
context, outpat: AbstractValue)
pass
def
context, output: BasicVactor)
pass
wwa.king.ar..vvoocitt...ooo. context, output: BasicVastor)
def controle. cor.eerroootro..rrrarrrrrraaaal context, output: AbstractValus)
pass
Bere is exanple response:
<tracking_cal calacncckcall python
place blace_bolder function(eelf, context, output: BasicWector):
abstract input_ppppppp poouttport..om pllppporrrrrrreeeeee value self place_holdor_port Eval (context)
output. SatFraaacctor (abstractt tttprracttnnpattprtttoooo "aalue""pacc_hhoderr
def slace_holden function2(seif,, context, output: BasicVector):
vector pettor_nppttpprt....
self place_holder_pp BaalddrrporrtBval...l. (context)
out[3] vector certtr_iipat_pprt.ttt [ort_
.utput.SetPraatt SetFroaVeetor
def blace_holdor, holddr__fncctln iumerion3(sel1, context, output: AbstractValue):
value self, place_holderr (cattee output.eet, set, set_value(vvauue)
Follow the previously given tipe for implenenting the callback functions. Further more: output ports the controller
by adding prefix: 'task_ port_name..
Now please implement the call-back functions. Please think step by step then write the code.
Assistant
<tracking.
python
context, output: BasicVecter):
# For the tracking controlter, the ee. target comes dérectly from the task controllen's output "aax__ee_poe___oa)) Eval (context)
ee_target self.
output. SetFrenVecter (ee_targee)
44

---

def tracking.no, to_ggaoottttttt.ee wot..raggg...pppeeaattt context, outpat: AbstractValue):
per the task, target type sa specsfically pase costent # Directly setting the target type to indSffactor/arget..e kPose kPose as regusred output. patput.sat_vaaiee
kPose)
def tracking ura_positioneelff context, output: BasicVector):
The position the tracksnng controller arn_position self.
the skill's input
anaaaaaaaaaeeeaaaaatataaaa "..vll BBvrl (context)
output cesFroaVactor(emtpolllli
def tracking, arn, rrn_velocityseell context, output: BaaicVector):
# The velocity for the fracking controller also venoveyy self output SetPron/Vactor (om_eeoctty)
the absil's input www.:rorrrnnanrrrr..a..oo.oo.oo Beraeoiitt""".aaalecetttettt""" (val(coneet))
bracking.coutrollee controller, parenttersseelf, cottroller.prrmeeeerseeeee
context, output: AbstractVelue): # Passing through the contraller panematers directly from the skell'ss input controller.peet paramerrrr self (eeinuutorttooottroleee
.vval (context)
Assuwing aunipulation is needed for comp compisane characlersetice. éf could look like:
# Nodsfy for stiffress and demping 1f needed, here directly passing through for simplicsty pararaattarra
output.sat__aa
## Meta-Control
Here the output of running the skill:
Hunan expert supplementary instructions:
Do think it is working as expected? (Enpty output means the system is running without orrors) it ie, please ansver with one vord "yes". please identify which stap(s) < step(a) to modify (<step_nama>) and the updated code (enclosed by "python"" ")) Ve have proceaded the tollowing stepa:
and explain fix it including which
task callback trackinggcallee
Please do not nakeup an inexist step.
If the systen is not working as axpactad, you mast modify as least one step. You can nodify mitiple steps.
For each step, you can only modify the function that you think is urong. Others will be kept unchanged. "catep_nane>" given infomation. Try your best.
corresponding
with
Here sample response
working expacted:
think error
(eone reason].
updated_code:
stracking.callbach
python
def tracking.ara,annoo.oorora propooeiiiin,,, context, output: BasicVector):
Assistant
yea
Yeta-Control
Please sumnarize the task task_controller, tracking.controll, task_callback and the backing_callbeee Explain used aa prompt choose the perfornance natric from
flow. response nessurements, and tune the controller paraneters, please include all necessary information.
Assistant
To accomplish the task of pulling open a door using robotic syeten, we've implemented bierarchical structure comprising
tracking controller, along their callback functions to connect and preprocesa inputs and autputa accordingly. Bere's comprahensive va sumaary
### Task Controller: Cateeriainterplaaiienotetteel
**Purposess;
eud-effector pose trajectory initial and target positions, ensuring the door opens approxinately 90 degrees without requiring interaction.
on
InInut Porte and Callbackess;
45

---

controller, p Directly passed through from the skill inputs, it includes dynanic adjustnents such as trajectory flags and stiffness paraneters.
through "controller_ parrpaaanttera **Calbbacke:: Froridas the current pose of the end effector in RPY format.
nodification.
measured_eo_ peepuree_e__poee...
<xCallbacker: Directly napa "neasured eo__ees from skill inputa to task controller inputa without nodification.
execution time': An abetract concept used to evaluate the current position along the interpolated trajectory.
>sCallbackex; Sets conatant execution time as placaholder; in practical applications, this abould dynanically reflect progresa slong
trajectory.
"target_position Datermines the final target pesition "Dx, Y, 23 of the end effactor, essential for calculating the trajectory that guidea the door opening.
==Calbbecke:; Calculates "target_ "target_poeition position and adjusts for 90-degree opening novement.
based on the
radius estination from the door hinge
**Dutput Portes:
_ee__pee__pe
desired end-effector pose [[rll,, pitch, yex, x, y, robot achieve, acting as the targat for the tracking controllar.
VNA Tracking Controller: CartesiaSSiifffeeeCatttroo
tewards targat pase specified compliance prevent danage, 4 utilizing joint torques calculated based on the current and desired poses.
**Purposa*:: Executes
**Input Ports and Callbacksss:
"ee verget"""" The desired end-effector pose for the robot to track.
+=Calbbackew: Receives "ee_pose_nen from the task controller and directly passes it.
"po_target_type Specifias that the target is pose (instead of tuist), dictating the controller's
focus.
*=Callback*=; Sete the target type to "EddEffectoraarge: XPoee" xPoee"
"arn "arn_position" arr__velcityy'
robot's +=Galbbacke>; Direct mappings from skill imputs to tracking controller imputs without nodifications.
current joint positions and velocities
controller, caatroller.parmnteers Allows runtine nodifications to control behavior, particularly to stiffnese and damping natrices.
bbackeb Passen cantroller.paraeetii from skill inputs to tracking controller inpats diractly.
**# Data Flow:
1. #1kill Inputa*:: Inconing observations and parameters (like neasured joint states, arm pocitions/velocitie. and controller configurations) feed into both the task and tracking controllers. *Task Callbacks Process*:: These functions preprocess and appropriately
data fron skill inputs to the task controller. They are respenatble for satting up the nominal trajactory by deternining target positions.
3. **Tack Controlleree: Vith its imputs configured, calculates the desired end-effector pase. Process*e; Receives the nominal pose necessary paranatera and skill inputa. Prepares data for the tracking controller.
*eTracking
controller along with
*TTracknn Controlleres; Utilizes the preprocessed inputs to compute the appropriate control actions, the specified and-effector pose trajactory compliantly.
aining
This entire process ensures
adjust its approach dynanically open approximately 90 degrees, based en the initial observations and ongoing neasurenents, all shile being fron neasurenents,
compliant avoid danage. performacce netric, be in reaching the target position, compliance in execution to avoid axcessive force, and officiency in trajectory exccution tine. Tuning controller paramaters, both task tracking atagea, will ba crucial
accuracy of the and-effector
particularly within optinising these aspecte.
C.4
Model template summary
setup linelr_nodel:
description: I function
affine dynemic This model is especially useful when the systen is can be approximated by affine dynanice. ui
The dynanic equation x_dot
Example usage:
.**
x0 ap.zeroz((4.1) 2p.2eroe((1,1)) a 0.01
0.81
np. ap..arra azray([[o,1,0,0) 0], [o,0,arg,0], (., 10.0,0.11,
array
[1/m], [0] (1/))
(-1/a]]
C mp.eye(4)
np. (e,
46

---

nodel setup_lineat winear_nneeeeee padeeerr_nddeleeee (x0, wo, A, B, D)
arga:
name: x0
type: numpy.ndarras
size: (n_x, 1)
description: Initial state u0
type: mumpy.ndarray
(n_u, 1)
description: Initial input
name:
type: nunpy.darrray
size: (n_x, n_x)
description: State natrix
name:
type: numpy. addrray
size: (n_x, n_u)
description: Input natrix
type: numpy.adan adarryy
(n_z, n_x)
deacription: Outpat natrix
nunpy.ndarray
size: (n_y, n_u)
description: Feedforvard natrix
setup_arm_node
description:
function returns model
arm.
This model is aspecially useful when the controller only needs to consider the robot arn
Because this function will be called in skill's init function, ar type is directly available arn_type.
Usaga:
n**
nodel zetup_ (arupparr_ttoeelarr.... .**
arga:
name: arn_type
type:
deacription: The robot are typa. It ahould be consistant as the skill's
setup_arm_ haup_arm_aae sitt__obeetts_odell
description:
This function returns nodel of the rohot arn and static objacts in the scane. This nodel is especially useful when the controller needs to consider the interaction betxeen the robot and objects, such grasping and collision avoidance.
It can be used for controlling in the cartesian space or in the joint space.
init function, type object_info are directly available as arm_type and object_info.
Usage:
nodel setup_arn wttppparrrwttbchatttta.. (arn (arn_type, abject_info)
...
args:
arn_type
type: str
description:
arm type. consistent as
name: ibject_info
type: dict
deacription: The dictionary objecta
should akill's.
setup_ petup_placeheldrr.nod
description: |
This function raturna
This model especially useful when the controller does not need any dynemic model. Erample usage:
n**
nodel = patpppplaccoodder_nneell
u**
C.5 Controller template summary
10RController:
description:
clase implemente Regulstor (LQR) given systen.
47

---

The controller computes control inputs based the deviations from equilibriun.
arga:
nodel
Diagran
deacription: Diagran containing the AffineSysten LQR controllar designed.
name:
type: nunpy. adarray
description: State natrix.
nama:
type: numpy.ndarray
description: Control cost matrix.
name: x0
type: sunpy.nderray)
description: Equilibriun state around which the controller designed. ub
type: nunpy.ndarray
description: Equilibrium
input.
input_porta:
name: state
type: BazicWector
size: "xon
deacription: Representa the
of the aysten.
output.porta
control
BasicVector
size: "1en(u0)"
description: Represents the computed control based the LQR control laz.
Idonttyyontrolllr:
description:
class implenents identity controller directly outpute torque receives.
It is the best choice shen the task controller is already enough to accomplish the task and only need a placeholder for the tracking controller.
args:
name: model
type: Diagran
description: & Diagran containing the MultiBedyPlant model of the robotic systen.
input_ports:
name: applied_amm_tonye
type: BasicVector
"plant.num, "
deacription: applied_arn__one compated by the task controller.
output.ports:
name: applied_erm_
type: BasicVector
size:
description: directly output the inpat applied_art,
PoseForceController::
description: |
combines pose The controller compates torques based on the desired end-effector pose and force.
class implenents
is best suitable for the situation when ve have a desired force on sone dinensions and desired other dinenstens. Note that controller track and posa sane time for dinension.
position
args:
name: model
type: Diagran
deacription: & Diagran containing the MultiBodyPlant model the robotic aysten.
name: force_control control_axis_aast
type: numpy.ndarea, ndarray
description: Boolean array of length to specify which axes are controlled by force. The order is Ik-rotation, y-rotation,
x-tranalation, y-translation, z-tranelation), dimension, True reprasents force control, False represents pose control.
input.ports:
ee_pose_target
type: BasicVactor
description: Desired end-effector pase (roll, pitch, yau, x, 2).
se_force_targer
BasicWactor
size:
deecription: Desired end-effector force. (x-rotation, y-rotation, z-rotation, x-tramslation, t-translation, z-tranelation).
y
arn_position
type: BasicWector
size: pplaan..um.pooittiii "
deacription: Gurrent joint positions of the arn.
neme: srn_velocity
type: BasicWector
48

---

size: rplaat.num.volocitieen velm
description: Current joint volocities of the arm. output.porta.
name: applind_arm__orre
type:
size:
description: Computed joint torquee for the robot arm.
Carteciansti stiffness
deacription:
using impedance control Nakes the robst behave as if spring-dampar system is attached to the end-affector. The stiffness can adjusted dynanically by providing input to the controller paraaerrrr port. The advantage this controller it provide complient behavior with adjustable stiffness.
torques.
args:
madel
type: Diagran
description: Diagran containing
model (NultiBodyPlant) conputing dynanica
input_porta:
name: sergettet
type: BasicVector
size:
deacription: Denirad affector target (posa twist). [roll, pitch, yau, x] name: ee, se_target_ttpp
type: AbstractValue
deacription: Type of the end effector target. This type is enun. The nust be EndEffectorTarget.t kPoae or EudEffectocTarggee.Tiies Do not use "str", use emun directly.
arn_position
type: BasicVactor
plant's arm) description: Current joint position of the robot arn
"(umber positions
srn_velocity
type: BasicVector
size: "(aumber of velocities the plant's ara)"
description: Current joint velocity of the robot nama: controller_pp saarrolleepprranatrr
type: AbatractValue
example_ value:
certesian.stiffeeee +[ [200]
description: The controller can be parassterized through an abstract imput port ceatroller.paranteers to modify its behavior during runtine, alloving changes to cartesian_ttiffaee and "cartasian_dapping" natrices. [roll, pitch, yov, y, z]
output.ports.
applied,emm,torqu torque
type: BasicVector
plant) deacription: Control torquea applied to the jointa of the rahot arn.
SafaController:
deacríption: A controller that projects nominal control torque to aafe control torque using a control collision avoidance. adventage controller is to eafeguard noninal control imput to realize reactive collision avoidance.
barrier function. Currently,
arga:
name: model
type: Diagran
deacription: Diagran containing multihody plant graph,
information
obstacles for collision avoidance.
type: Meshcat
description: arg optional. visualization purposes.
input_ports:
name: arn_target
type: BaaicVector
size: "mun_q"
description: Desired joint target arm. name: arm arn_target__ype
type: AbatractValue
desoription: Type of the target. This type is The value must be S.AutTargett.oosttaa manntTargat.kWalocceo or JointTarget. Do not use "str", use enun directly. arn_position
type: BasicWector
"nun_q"
description: Current joint position of the arm.
name: arm_velocity
type: BasicWector
size: "nun_q"
description:
joint velocity
output_ports:
name: applisd_amm_toqqu
BasicVactor
size: "nun_q"
description: Computed joint torque for the
49

---

description: Constructs trajectory in cartesian and publishes vaypoints by interpolating the trajactory. Dapanding the polynonial order, can produce firat-order holds or splinea for the trajectory.
arge:
model
type: Diagran
deacription: nodel useful this controller, should passed consistency
name: polynonial_order
default:
deecription: Order the polynonisl for trajectory optinization. Supported valuee (for First
Order Hold) and 3 (for Cabic Spline).
mashcat
type: Meshcat:
description: visualization input_porta:
(optionsl)
name: controller, controller.paranetter
type: AbstractValue
example_value:
neu_trajectory:
way_points:
"[iist tines]"
pointa: "[list pose in RPY_XYZ fornat]"
description: Controller paraneters that can indicate the need for new trajectory and provide waypointe.
name: keasured_es_per
type: BasicVector
si2e:
description: Neasured pose
effector fornat.
name: reasure__es_tuiit
type: BasicVector
description: Neasured tuist of the and affector. RPY_XV2 output.porte.
name: ee_pose_con
type: Basic/ector
size:
description: Noninal ead-effector pose based on the last solved trejectory spline evaluated at the time. RPY_XYZ
description: controller using trajectory optimization to trajactory joint space. The objective of this controller is to reach goal in cartesian space with the end effector. Flease note that the output in joint space.
collision-free
arga:
name: model
type: Diagrsa
description: Diagran containing the multibody plant and scene graph, xell as the information of obstacles for collision
name: reaolve_period
type: float
default: 3.0
description: Trajectory optinization resolution period. nun_steps
type: int
deacription: Number of optinization atepa. name: nashcat
type: Meabcat Optional
description: For visualization input_portes
name: cattroller_parmette
type: AbatractValue
example_value:
goal_pose: Rigidfranaform
description: Controller paraneters including goal pose name: reasured_ara_porittie
type: BasicWector
size: "(uunaar of positions of the rohot arn)"
deacription: Current joint position of the robot arn.
nessured_arn, sessured_arr_veloccty
type: BasicVector
velocities
amm)
description: Current joint velocity of the robot arn.
output_portes
name: q_non
type: BasicVector
50

---

size: "(nunber of plant)
description: Noninal joint positions of the arn.
C.6 Skill imput summary
controller, paraneeereeeess
description: Controllar paranatars that can be adjusted symanically during ruaning. type: AbstractValue :aba_structure: dict
port_name: measured_arn, arn arn_passtiin
description: Observation of the robot arn's joint position. type: BasicVactor
dinansions: man_q
part_
neasured_ arn arn_vellcity
description: Observation the type: BasteVactor
joint velocity.
dinensions: nun_q
nort_ neasured_ee,
description: Observation of effector's pose. first correspond rotation. The last three elements correspend to translation. type: BasicVactor
dinensions: 6
neasured_es
deacription: Ubservation of the end effector's twist. The first three elenenta correspond to rotation. elemente correspond type: BasieVactor
The
port_name: reaeured_joint_statee
deacription: Ubservation of joint (including joint articulated objects). type: AbstractValue data_ptruc
port_name:
neasurad, headurad_bady_aa body pady-poses
deacription: Observation of the poses the rigid bodies the scene.
type: AbstractValue
port_name:
neasured velliveloc
description: Ubservation of the velacities of the rigid bodies in the scene.
type: AbstractValue
data_structure: dict
C.7 Input port sample
TreckingCortrolleecooveeee input port contents:
catrolllerpparantter <class 'dict'> X' d'gripper_co command'; 'release
"gripper_command"
<class 'mumpy campy.ddarray's O.
0.26179839
3. 9414169266
-2. 26918361
0.9599
1.57079638]
neasured_ara_ee srn_veloiity <class 'mumpy adarray's ?> [o. 0. 0. o. o. o. 0.]
nessured_ ee eeaeeee <class 'numpy adary.ndarray'> 11.77188460e100 33.92246874-005 1.57079455er00
5. 5.96598901e-01 4.33589236e-01]
1.34625984e-03 neasured_ee_tet
<class 'nuxpy. 'numpy.ndarray" ndarray'> ,
jonnured_jonntsttee states <olass 'dict'> {'Actuator1'; {'positions': array([o.]), 'velocities":
array([O.])), Mottutor2': ('positions's ( 26179939]), "velocities': array([o.)))] 'Actuator3';
{ ('positions's (311961..)1992)5),
14159265)),
velocition': array([o.]))J] 'Actuatord'; ('positions".
(xxxy((-2.29118533) 'velocities's array([o.1)), 'ActustorS", {'positions": array((o.)),
'velocities's arr.y((o.))), 'Actuatrr'' ('positions': (0.ay((0_9999) 'velocities': array([o.)))]
'Actuator7's Uppositinn'' array((1.57079633)) 57079633)), 'velocities' array((o.))))) 'EndEffertor':
['positiona' array([], dtypa=rloat64), 'valocities': array([], dtype=flont64))]
voodid_velds_to tt_eldd__tobbee_____ baw.__eeek.... (aaa":('''peee ('poeitionn': array((), dbype-float64), 'velocities". array
drype-float64)]] 'PolePin': ('positions': array((0.1)). 'velocities'; arrey([0.]13,,
'end_offector: link welds_to_Caart" ('positions', array([], dtypo-floaté4), 'velocition': array(1),
dbype-float64)Jj oberac__ 2 ppooitionss: arrey(l 0, 0.99875026,
0.
-0. 00.04997917, 0.4
51

---

0.3
0.101
D), 'velocition' array([o., o.. o.. o., o., 0.J13.] '$world_goal_
('positions' : array([1.
G. O. 0.6 0. 0.0261), "velocities's array([o., 0.,
0., 0.1)
neasured_ bod.__ood______e
<class 'dict'>, ('Cart': Rigid(ranssormi
ReRota:1on/atrik((
[1.7716140821722224000 0.9899995582032777, (.000889681130020516))
[0. 999999999997771,
1.7706928834778896-666 1.922978601902294660001
[3. 06 0.0002893611000996667,, (.9999999344422224)
1),
p- p-[o. 48659660657829427,, 9.001366459611425565,,
0-4336296659980773)
>}
neasured. wtwpedd_boooooeeeee pody_velocities <clasa 'dict'>
Cart': SpatialVelocity(
v-(0.0, 0.0, 0.0], v=,0, 0.0, 0.0].
cost_olntrol
<clase 'aumpy adarrsy'>
52


Reviews:
[{"review_id": "8YqopUxrtb", "reviewer": "Reviewer_9Tq5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents an LLM-guided synthesis process for hierarchical model-based controllers. In a first query, the LLM is queried with a task provided in natural language and asked to select a high-level task model and task controller, as well as a low-level tracking model and tracking controller from a library provided in-context. This first step also provides parameters for controllers and models. The second query asks the LLM to synthesize interfaces between perception and the two controllers. These two queries are then followed by an optimization loop in which the proposed controller is executed, and the LLM is presented with the resulting trajectory, and asked to update the paramters it previously selected. The combination of the three steps results in a method, named MetaControl. The resulting controller is able to solve three tasks in simulation (an arm-bound pendulum, a door opening and a pick-place task), as well as a board erasing task on real hardware. Each of the three components is ablated, highlighting in particular the necessity of few-shot prompting to improve the initial parameters suggested by the LLM. As the final product of MetaControl is an established model-based controller, the method benefits from interpretability and generalization (e.g. by replacing the robot's model).", "summary_of_recommendation": "Despite the good presentation and significance of this work, there is currently lack of clarity on how the prompts were designed (on a single task, or on multiple) and on how the method would be able to generalize in a zero-shot fashion. Thus, I would lean towards rejection, but I remain open to updating my evaluation.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "**Questions:**\n- How many shots are necessary for few-shot finetuning in each task? This information is currently missing and would be very helpful.\n- How would the author scale as more controllers/models are provided, and choosing the right controller/model becomes harder?\n- The author mention slow synthesis as a limitation. Could they also provide the runtime for the method?\n- What do the authors mean by \"art of midwifery\"? This term has a rather clear method in philosophy, but it is not entirely clear how MetaControl embodies this principle.\n- Line 125: why is $g$ considered a dynamic model? To the best of my knowledge, it does not represent a state change.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pa1v0dndTr", "reviewer": "Reviewer_5Tm1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an approach to use LLM for controller synthesis, using classical concepts like state-space linear systems, LQ regulators, MPC etc", "summary_of_recommendation": "Very interesting paper but, in its current version, very difficult to replicate systematically", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "As a reviewer, I couldn't really understand how to replicate the proposed results. The body of the paper doesn't give enough details so I have tried to retrieve these details from the appendix. However, the appendix is unclear. The conversation seems to have 3 different users (named: \"## Assistant\", \"## User\" and \"## Meta-Control\"). Also, the authors do not give a full description of how MetaControl works (i.e. produces its own text output): some messages seem to be just a stereotyped prompts \"The GPT is a professor in robot control and a proficient programmer...\"; some other messages seem to be a copy-paste of carefully chosen previous assistant answers \"The chosen task model, tracking model, task controller, and tracking controller are...\"). In order for others to replicate the given results, authors should give a complete description of how Meta-Control works, possibly describing it in the form of finite state machine (with parameters), so that it can be coded in some programming language.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAC8Qs8nUa", "reviewer": "Reviewer_U2Js", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents LLM-enabled automatic control synthesis approach that  creates customized state representations and control strategies tailored to specific tasks. The pipeline is implemented with PyDrake.", "summary_of_recommendation": "Overall, it is interesting and impressive, but the level of presentation is somewhat skewed towards saying how good all is.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was initially evaluated with mixed reviews, but after the rebuttal, we the reviewers suggest accepting the paper, although without strong backing.

Here is a high level overview of the reviews. (before rebuttal)
### Strengths:
- Novel approach using LLMs for model-based controllers with clear practical applications. (Reviewer 9Tq5, 5Tm1)
- Overall polished and clear presentation (Reviewer 9Tq5), although critique of lacking details.
- Review of LLM-based control approaches. (Reviewer U2Js)
- Impressive results. (Reviewer U2Js)

### Weaknesses:
- Lack of clarity in the prompt design process and generalization (Reviewer 9Tq5) and lack of essential technical details for replication (Reviewer 5Tm1)
- Insufficient evaluation across diverse tasks and models. (Reviewer 9Tq5)
- Unclear application details and presentation. (Reviewer U2Js)

In the rebuttal, additional evidence was added and the presentation was improved. Still, reviewer 5Tm1 things the explanation of how the automated dialog could be improved. 
Given the potential of the method and the overall good presentation, I suggest accepting the paper.
Decision: Accept