Text2Interaction: Establishing Safe and Preferable Human-Robot Interaction
Authors: Jakob Thumm
Christopher Agia
Marco Pavone
Matthias Althoff
Venue: CoRL 2024 (2024)
Keywords: Human-Robot Interaction
Human Preference Learning
Task and Motion Planning
Safe Control
TL;DR: Text2Interaction integrates human preferences into the task, motion, and control level of the robotic stack.

Abstract: Adjusting robot behavior to human preferences can require intensive human feedback, preventing quick adaptation to new users and changing circumstances. Moreover, current approaches typically treat user preferences as a reward, which requires a manual balance between task success and user satisfaction. To integrate new user preferences in a zero-shot manner, our proposed Text2Interaction framework invokes large language models to generate a task plan, motion preferences as Python code, and parameters of a safety controller. By maximizing the combined probability of task completion and user satisfaction instead of a weighted sum of rewards, we can reliably find plans that fulfill both requirements. We find that 83% of users working with Text2Interaction agree that it integrates their preferences into the plan of the robot, and 94% prefer Text2Interaction over the baseline. Our ablation study shows that Text2Interaction aligns better with unseen preferences than other baselines while maintaining a high success rate. Real-world demonstrations and code are made available at [sites.google.com/view/text2interaction](sites.google.com/view/text2interaction).

Text2Interaction: Estabishhing Safe and Preferable Human-Robot Interaction
Jakob Thumm
Christopher Agia Stanford University cagia@stanford.cm
Technical University of Munich jakob. btobb.thr thummu tum. de
Marco Pavone Stanford University
Matthias Althoff Technical University of Munich althoff faatooff@uu.cce
pavone@stanford.edd
Adjusting robot behavior human preferences require intensive human feedback, preventing quick adaptation to new users and changing cir- cumstances. Moreover, current approaches typically treat user preferences as a reward, which requires manual balance between task success and user satis- faction. To integrate new user preferences in a zero-shot manner, our proposed Text2Interaction framework invokes large language models to generate task plan, motion preferences as Python code, and parameters of safety controller. By maximizing the combined probability of task completion and user satisfac- tion instead of weighted sum of rewards, we can reliably find plans that ful- fill both requirements. We find that 83 % of users working with Text2Interaction agree that it integrates their preferences into the plan of the robot, and 94 % prefer Text2Interaction over the baseline. Our ablation study shows that Text2Interaction aligns better with unseen preferences than other baselines while maintaining a high success rate. Real-world demonstrations and code are made available sites. www..goggg...omvve////eeeeeeeaae
Abstract:
Keywords: Human-Robot Interaction, Human Preference Learning, Task and Motion Planning, Safe Control.
Introduction
We are moving toward a future where robots are fully integrated into our everyday lives, from manu- facturing [1] to healthcare [2] to households [3]. In these settings, robots must quickly adapt to indi- vidual human preferences and changing circumstances. However, current robotic platforms severely lack in this aspect, which prevents their application to daily tasks and widespread acceptance. Re- cent works [4-6] present promising approaches to seamlessly incorporate task-level preferences, answering the question, "What should the robot do?" In human-robot interaction (HRD, must further address two key user preferences about the behavior of the robot: motion preferences, which determine "Which path should the robot choose?", and control preferences, which dictate "How fast, soft, or precise should the robot be?" Recently proposed methods in motion planning [7-9] and safe control [10-12] tend to require labor-intensive human feedback to adapt to such preferences. Hence, these solutions learn a preferable behavior offline and, therefore, fall short when comes situational awareness and fast-changing preferences. Additionally, most recent works [9, 13-17] treat human preferences additive reward, requiring them carefully balance task success and user satisfaction.
In this work, we propose Text2Interaction, a framework to incorporate preferences from a single user instruction in three levels of the robot software stack: task planning, motion planning, and control. exemplified in Fig. 1, we react to user preferences online by querying a large language
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Hand me screwdriver, make sure can comfortably grab
Text2lnteraction
LLM (GPT4)
Task preference
Motion preference
Control preference
Pick(screwdrrivrr
Umax
Qmax
Handover(screwdriver)
prafacance prob
Stiffness, Damping
Plan not feasible/preferable
Planning
Feasibility
Execute
Figure I: Preference-alignedd planning with Text2Interaction.
The user asks the robot to hand them the screwdriver so that they comfortably grab it. Text2Interaction queries an LLM to return (a) sequence of primitives that satisfy the task preferences, (b) a set of motion preference parameters that adjust the safety controller to
functions executable Python code, and (c) a the current situation and control preferences of the user. Our planner than aims to find plan that satisfies the user preferences and is feasible to execute. If the planning step fails, we query the LLM to return the optimal skill for the next timestep only, as discussed in Sec. 6.
model (LLM) to obtain (a) sequence of primitives task preference, (b) preference functions written in Python code that evaluate how well an action aligns with the instructed motion prefer- ences, and (c) a set of parameters that adapt our provably safe controller to the control preferences of the user. Our formulation results in the maximization of the combined probability that the plan is feasible and the user satisfied instead of the commonly used weighted sum of rewards [9, 13-16]. Our ablation study showed that our formulation leads robot behavior that is twice as preferable the baseline while maintaining high success rate in unseen tasks. Of the 18 participants in real-world user study, 83 % stated that Text2Interaction considers their preferences, and 94 % preferred Text2Interaction over the baseline [5]. To summarize, our core contributions are:
1. We propose Text2Interaction, framework for integrating human preferences in robot task planning, motion planning, and control.
2. We derive task and motion planning formulation that optimizes the likelihood human satisfaction and task success online.
3. We evaluate Text2lnteraction (a) user study on a real human-robot collaboration task with 18 participants and (b) in simulation on set of geometric object rearrangement tasks.
Related work
Early works in adaptable task planning use known world model that can be adjusted online to human feedback [18-20]. A number of more recent works propose using LLMs to directly con- vert language instructions into task plans, either using text [4, 5, 21] or code [22, 23). Subsequent works attempt to improve the planning robustness of LLMs through iterative re-prompting [24] or by leveraging formal logic representations [25-28], e.g., in the form of planning domain definition language (PPDL) [29] description. The works in [30-32] extend such PPDL formulation to col- laborative scenarios. Multiple works [14, 33-36] further show how these approaches can allocate tasks more preferably in HRI.
common way to incorporate human preference on the motion level to learn a reward function that represents the human intent, typically by asking comparative questions [17, 37-43J. The authors in [12, 44] demonstrated how this form of preference learning perform preferable robot-human handovers. The main disadvantage of this type of reward learning is that it either only reflects simple

---

linear reward functions or requires thousands of human queries, As such, several recent works con- struct reward models with L.Mss and use them to train [15, 16, 45-44] or directly synthesize [13, 48] robot skills. Notably, these approaches consider preferences for individual skills, whereas we focus on entire skill sequences. Additionally, these methods generally produce reward functions as linear combinations of reward terms, which requires careful weighing of the reward terms. Closely related to work is that of Wang et al. [49], which proposes to incorporate human feedback into long- horizon planning by extracting skills from demonstrations and training two models: one to predict the correct sequence of skills and one to predict the parameterization of these skills. However, their approach relies on a labor-intensive offline labeling phase and assumes that the preferences of all future users align with these offline labels.
A broad range of works [10, 11, 50-52] further proposes learn the parameters of admittance controllers from human feedback achieve comfortable object handling. Huang al. [53] demon- strated the effectiveness of this approach in an interactive handover task.
3 Notation
We denote a skill as the tuple & = (ó, a, E) The primitive & stems from a library of K prim- itives _ = {dtt... .......... each of which parameterized by set of continuous parameters E Me...,,..... K (hereafter referred to as actions). The controller a E _ = {ar .o....., aM with parameters & E m I...., M returns the system inputs 0 = a(d", a ,,E)) fol- low the parameterized primitive at time t. For our problem, process (MDP) with the tuple M = (S, V, Vrr, Here, S is the continuous state space, I = = x A x _ x E is the set of possible skills, T(s+41 5t, 1h) is the transition distribu- tion, and Sp C S is the set of initial states. The reward function r: S x V x S + {0, 1} returns r= if the execution of skill feasible, and r = 0 otherwise.
define Markov decision
Problem statement
The user gives instruction i, which may include their task, motion, and control preferences. Thus, define two events? for any environment history ($1:#+1, $1.H) ) := [$1, 1/1, 82, th2 .... $H+1]: r(SH, 8H+1) 1, i.e., the execution of 4/1: feasible, Spreference $h. satisfies the preferences in when starting in $1.
Steasible r(11, 1/, 82)
We assume that a user satisfied with an environment history if the event Swse Spreference Steasible occurs. Therefore, our goal is to maximize the probability of user satisfaction based the instruc- tion and initial state 81 with
pSSwer i, $1, v1.H) = p( preeferrence 81, 1/1. Stessible) p( P(Seassible 81, v/1.H))
which represents the objective defined in [5, Eq. 2]. Contrary to Lin et al. [5], which only adhere to task preferences, we assume that the user only satisfied with the execution of a skill sequence th their task, motion, and control preferences are satisfied. Therefore, we define the three events:
Sunk 01.H satisfies the task preferences in i when starting in 81,
Smation 01.H satisfies the motion preferences in i,
Scontol O1::H ε1:H satisfy the control preferences in i.
Using these events, we can redefine the preference satisfaction event as Specternnce Sjask A Scontrol A Smotion: and its probability as
0SSpefenence (,, $1, 11.H Steusible) = Suakk Scomools Smetion 81, 1/1.H, Steasible)
(2)
p( (Smotion i, 31, 1hI. Stensible; Susk Soontrll) p p(Senntrol i, $1, dI.H, 1. E1.H, Steasible: Stask)
(3)
pmSnkk i, 91, 0I.H, Steasible)
Note that we denote le-th primitive the library _ as de and the primitive active at time i as di.
All events defined this work the "success" event of its Bernoulli trial.
3

---

where we assume that Susk is independent of d1:H, 01-H, and f1.H and Sccnrol is independent of #1:H- Thus, to find the skill sequence with the highest likelihood of user satisfaction, we must solve
Vi.H = arg max P(Stcasible 81, (1..)) p(Smotion i, 81, 1..H, Stasiile Stskk Scontol)
(4)
01.8
p(Suomoot i, 81, P1:11, 01:J1 E1.H Steasible, Stssk) p(Stask i, 81, 01.11 Steusible)
The goal this work is to derive a framework including reasonable assumptions and approximations to efficiently find a solution to (4) that adheres to the given preferences in an HRI setting.
Supporting methodology
This section introduces the primitive learning method and safety controller used in work.
Sequencing task-agnostic policies For each primitive oe E L we use sequencing task- agnostic policies (STAP) [54] to learn a policy (a(a|s) from which one can sample actions for planning and a Q-value function Qh: S x A + [0, 1] which estimates the probabil- ity that executing primitive &e parameterized by action a in state s is feasible. Additionally, learn the transition distribution T(4++1 St, 1/x) to predict the next state. Given dataset of transitions (so; dps To) D = S, A x S x (0, STAP learns the functions x², Q*, and T using offline reinforcement learning. In our experiments, we reduce sample complex- ity by training these components independently from the controller E)::), i.c.. we assume that the distribution of skill success is approximately constant across most controller configurations: Vat, , a², 5² PSSeesibble 81, Φ (2,,5",a) a) Rs p(Siensible 88, Φ a², u², a). This assumption holds in our experiments as the controller parameters mainly impact the speed with which the robot executes the primitive, but not its path.
Safety controller To ensure human safety, we use a provably safe controller [55-57] that adheres to ISO 10218-2 [58] and ISO/TS 1,066 [59]. Our controller comes in three variants: Oksop [57, 60),
Ocenttctt [60, 61], and Ocoopllaat [62]. Using assopp we guarantee that the robot comes to a complete stop before any contact with human could occur. The Occontuer mode allows the robot to have a low speed close to the human and thereby enables active contact. Finally, the Occoppliant mode ensures low contact forces between the end-effector and the human using compliant Cartesian control. Generally, we can use the parameter vector to adapt the maximal velocity, acceleration, jerk, stiffness, and damping of the controller. To simplify the control parameter selection for the LLM, we predefine set of parameter vectors: Écocxistence for non-interactive scenarios, Éeritical for high-risk scenarios, and three parameter vectors for interaction with varying user experience Ebeginners Eimermediates and Eaxpert-
The Text2Interaction framework
This section details how Text2Interaction satisfies the user preferences and achieves feasible plans. To find feasible skill sequences in long-horizon tasks, Lin al. [5] propose two modes: shooting and greedy-search. In the following paragraphs, we derive the assumptions and approximations necessary to incorporate motion and control preferences into their approach and solve the problem in (4) efficiently. Fig. 2a summarizes our resulting planning objectives, and Fig. 2b exemplifies how Text2Interaction links the shooting and greedy-search modes.
Shooting In the shooting step, first let an LLM generate all elements necessary plan a quence of skills that fulfills the instruction of the user. For the task and control preferences, the LLM directly generates the sequence of primitives @1.H and controller settings ac,, {..H). Since these elements are now fixed, our planner cannot influence them. Therefore, we set the probabil- ity of satisfying the task and control preferences in (4) to p(Suusk $1, P1.H Steasible) zs Crask and pl ((Seumm i, 81, Φ1.HH O1::H, Ç1:H, Steusible: Smus) Countoll with Cuak, Ceuntrol E (0, 1). Hereby, we as- sume that the suggested task plan and controllers have non-zero probability of satisfying the In the next paragraph, will discuss how we proceed if this assumption fails. To approxi-
mate the probability that the execution of the plan is feasible, Lin et al. [5, Eq. 5] have derived that

---

Vi arg max pl Seasible 11, plSmotion 1,81,11115 Steasible Staek: Setra) Wi.r
p(Scaatcol 81, 91.11 0n.8 H. Steasible: Stank) p(Stasi $1.H Stessible)
K
Y
Shooting step
Greedy-search step
de
arg max Cinsk Coontrol
II Pr Pr 0+)
Q-1sr, a.)
(07,a)) arg max Countrol Pussle (0)) ofa
Fi(s ae)
Qilse, as)
greedy-score
score
(a) Planning objectives for the shooting and greedy search step.
1. Shooting step
1 E F1
EH.
Fh
LLM
Planning
O?
X
2. Greedy-search step
@ Stask (01). E1.
F1
Planning
greedy-score 0.2
LLM
o Stask (ot),
E1.
FF
Planning
+ greedy-score=
3. Shooting step
Planning
score > 0?
HH EH
FH
E2
F2
LLM
execute plan
(b) Example of the interplay between shooting and greedy-search. After receiving an instruction i, we first perform shooting step, where an LLM generates task plan $1 the controller settings a(;, {1:H), and the preference functions Fi.H- In this example, the first shooting step failed to find valid solution. Therefore, we exccute greedy-search step the first step, where we generate multiple candidate solutions and select the top candidate. Afterwards, the shooting step starting from $2 succeeds.
Figure 2: Detailed overview of the Text2Interaction framework.
pl (Sessible $1, 11:H) II'", Ce(s,, az). where Q, denotes the Q-value function associated with the primitive de at time t. Analogously, derive in Appendix D that approximate the proba- bility that the user is satisfied with the action sequence using set of preference functions
p(Smotion i 81, $1.H Steasible; Susk, Scontrol) RJ II (s(s ac);
(5)
where S1+1 ~ Tr(-1 St, whr) for t 2 1. A preference function F(s, a) returns the likelihood that executing action a in state s fulfills the motion preferences of the user. We later discuss how we can construct such functions online and give a number of examples in Appendix C. We can now approximate our objective in (4) and find the optimal sequence of actions ai that both satisfy the motion preferences I Ft (St, ax) > 0 and are feasible I (s oe) > 0, by using a cross- entropy method planner [63] to solve the optimization problem (see Fig. 2a)
H
1: arg max Grask Coontrol
II t=1
Fi(se, at) Qe(se,
(6)
a1:H1
Greedy-search If we cannot find shooting plan that adheres to the human motion preferences and is feasible, i.e., I[ Fe(st, a²) Qe(st, a²) O, we have to expect that our assumption that the task sequence valid Cusk > 0 failed. this case, we execute a greedy-search [5], where we try to find the optimal next skill 1/ instead of the entire skill sequence - We define the simplified event Si Seassible Stask St " St S which occurs if the skill dh, is feasible and adheres the user preferences time step t. Thus, the problem (4) simplifies
dbg = arg max P (8,, 8/e, 51:1-1) p(S i, St, 1/2, St Sicusible: Stusk: S² St 51:t-1)
(7)
p(St St St, Pr, E,, Stasiible Stack s1:t-1) P i, Øz, Feasiible: Suser
st
s1:t-1),
5

---

where we made the Markov assumption. In greedy- we let an LLM return L S K candidate primitives for the next skill. We then approximate the probability that the user is satisfied with the next selected primitive with the sum of token log- probabilities of the language description of each primitive p(SSusk i St, Dt, Scasibbee ussr= ~ Piax(())) E I,.... as proposed in [4, 5]. These scores represent the likelihood that the textual label of a primitive is a valid next step for the instruc- tion i [4]. We could proceed with the control preferences in the same way, but generating 2 pairs of primitives and control candidates is time-consuming. Thus, we only generate one set of control pa- rameters per candidate primitive and set p(S i St, (e, Or.: St, feasible: Slask Suer ~ Coontrol- Then, we generate one preference function per candidate primitive to obtain an approximation of p(S i, St, 2)2. ST Steasible Susk: 5 control: Snerr = (st, at), and approximate the probability of feasibility with pl St leusible 8t, 4/)) Rs Qi(s,, at). Thus, the problem in (7) simplifies to
67 a = arg max
d Comarol Pusk (o)) (st, as) (s az),
(8)
which gives us the optimal one-step primitive and action. After finding a solution to the greedy- search problem in (8) using cross-entropy method planner, we return to the regular shooting strat- egy starting from state 8.+1-
Generating motion preferences from text To approximate the probability that human would approve action d₂ at time 6, i.e., p(S i St, 1hH S Stsk (mmorr)) we query LLM to return preference function Fi 0e) as Python code based on the user instruction i, the primitive sequence 01.H, and the controller structure, we prompt with a system and task description, a number of in-context examples [13], and set of programmatic building blocks. These building blocks provide the LLM with
)1. To align the return of the LLM with our desired
1. the predicted next state $t+1 Es..1 [T(5++1 1 Sts sbe)).
2. the functionality to retrieve the pose of an object in a given frame from given state,
3. functions to calculate metrics given of poses, e.g., the Euclidean distance of two objects,
4. set of monotonic functions 9p R + j0, 1] that map the value of given metric to utility.
By restricting the codomain of the utility function [0, 1], we can interpret its image as the probabil- ity of user satisfaction. The LLM can evaluate multiple logical connectives in the preference function probabilistically using the AND(PI, P2) = P1P2 and OR(PI. operators, where we assume that the events are stochastically independent. We provide more de- tails about our prompt design Appendix B. One key assumption that we make is that the pref- erence functions Fi.#1 (s, a) returned by the LLM accurately reflect the true underlying probability pl Smmmtioo i, 81, 1IHH, Steasible, Stnsk, Scontro),, which evaluate in our ablation study in Sec. 7.2.
P1P2 + Pi (1- (1- + P2 PI)
7 Experimental evaluation
We performed user study and an ablation study to investigate three main hypotheses:
H1 Users have preferences regarding the motion and control of interactive robots.
H2 Text2Interaction integrates these preferences the execution of the plan of the robot.
H3 Text2Interaction generates preference functions that align with the instructions the user.
Our experiments focused on preferences on the motion and control level, as previous work [4, 5] already investigated task-level preferences extensively.
7.1 Real-world user study
To evaluate our research hypotheses, we set up our running example in Fig. I on Franka Research 3 robot. Hereby, the robot had to pick up screwdriver from the desk and hand it over to the user, as demonstrated in supplementary video. Our user demographic consisted of 18 participants, of

---

Btoonb
Noutrll
Text2nntencttion integrates preferrences into the robot's plan.
have strong preferences how the robot executes its plan. Communicating my preferences was casy.
preferrod Text2Interaction (predofind)) over bascline.
preferred my instructions Text2lateraction (predefined).
have strong preference in the sufety shield mode.
-15
-10
-5
0
10
15
Number of answers
Figure 3: Main takeaways from our user study. The answers of the 18 participants are centered around zero for better comparison.
which 16 stated that they had previous experience in robotics. Our experiments then comprised four experimental stages, with the users performing the same task multiple times per stage and answering questions on a five-point Likert scale after each stage.
In stage 1, we demonstrated our safety shield to the user. For this, we predefined three sets of controller parameters: Sbeginmer, Sinermediatee and fexpert listed in Appendix A, with each mode being faster and more reactive than the previous one. We then let the user decide which mode they would like to work with in the upcoming interactive task.
In stages 2 and 3, the users performed the task of the screwdriver handover, where the robot had two different modes. The first mode was baseline [5]. which only optimized for task success and did not incorporate motion level preferences. The second mode Text2Interaction (predefined), which we previously queried with the instruction, "Hand me the screwdriver, and make sure the handle is pointing towards me so that I can comfortably grab the handle.".
In stage 4, we asked each user for personal instructions to the robot for this task. We then queried Text2Interaction for new custom preference functions and performed the task in a zero-shot manner.
After stages two to four, the users answered five questions about trust, intelligence, cooperativeness, comfort, and awareness [64] to evaluate the quality the HRI. By asking the same questions repeat- edly, we can evaluate if there is a difference in the distribution of answers between the stages using the Wilcoxon signed-rank test [65] and report the p-values. Fig. further summarizes the answers set of general questions not associated with any specific stage. Our user study confirmed our hypotheses in the following ways:
HI: First, users tend to agree that they have a strong preference for the way the robot executes the task, see Fig. 3. Second, with statistical evidence, users perceived Text2Interaction (predefined) as more intelligent (p S 0.005), more cooperative (p S 0.01), and more trustworthy (p S 0.05) than the baseline. Additionally, they were more comfortable with Text2Interaction (predefined) (p 0.025) and reported that the robot more accurately perceived what the goals of the users were (p < 0.01) than with the baseline. Third, from chose Eexpen controller parameters, which indicates general preference towards faster robot for this task. Furthermore, 33 % of users have stated strong preference for the safety shield mode, indicating that although a majority of users might be indifferent to the controller parameters, is still relevant to integrate control preferences in HRI.
group, 50 % chose Einermediatte and 50 %
83 % of users agreed that Text2lnteraction integrates their preferences into the plan of the robot, and 94 % of users preferred the execution of Text2Interaction (predefined) over the baseline. The participants further rated the execution of their personal preferences, Tex:2Interaction (personal), as more cooperative (p < 0.005) and comfortable (p < 0.025) than the baseline and stated that the robot more accurately perceived what their goals are than the baseline (p S 0.025).
H2:
H3: 67 % of users preferred the robot behavior following their instruction over Text22nteraction (predefined). Additionally, 83 % of users stated that it was easy to communicate their preferences with Text2Interaction.
7

---

7.2 Ablation study
To further quantify the validity of hypothesis H3, we performed an ablation study on a set of object rearrangement tasks. Our study aims to validate if Text2Interaction can generate valid preference func- tions from limited set of in-context examples. We used 15 tasks consisting of instruction, task plan, and hand-crafted oracle preference functions Fi.H- For each task, we defined three trials, and for each trial, we randomly selected three in-context examples out of the other 14 tasks construct the LLM prompt'. We evaluated each trial 100 times with random initial states and calculated their aver- age preference score based on the oracle preference functions Fi.H- Finally, we calculated the mean suc- cess rates and preference scores together with their 95 confidence intervals using bootstrapping. Our experiments included four agents:
Oraele: 11, Q:
100
Baseline 1: scoce II
Broelins: woxe 0. F:
80
TexC2loteraction: sove
11, 9.4
Probability 1%
60
40
0
Success
Human preference
Figure 4: Mean results of our object rangement experiments. The whiskers dis- play the 95 % confidence interval in the ported mean metric.
Oracle, which used the hand-crafted preference functions (score = II. Q.F7).
Baseline I [5], which only optimized for task success (score = II, Q.).
Baseline 2 was inspired by [13, 15, 16]. which treats preferences and the feasibility of the plan as rewards, and aimed to maximize the score = >, Qi + F..
Text2Interaction with the objective defined in (6) (score II, Q:Fi).
Text2Interaction successfully generated executable preference functions for 45 trials. Fig. shows that our approach achieves approximately the same or higher task success rates than both baselines but achieves significantly higher preference scores. Empirically, Text2Interaction mainly struggled with vaguely communicated preferences, such as, "Place object A as far left of the table as possible." This is because there are many different ways of interpreting the instruction, which leads to natural discrepancy between Text2Interaction and the oracle. Overall, the results of our ablation support hypothesis H3 and indicate robust generalization of Text2Interaction to new problems.
Conclusion and Limitations
We presented a framework to include human preferences from a single user instruction three levels of the robotic software stack: task, motion, and control. By optimizing over the combined probability that the plan feasible and satisfies the user instruction, we find task and motion plans that are more likely to fulfill both criteria than related baselines. The overwhelming majority of users found that Text2Interaction integrates their preferences easily.
One limitation of our work the dependence on in-context examples for the LLM prompts. the user request is semantically out of distribution of these examples [66], the LLM unlikely to out- put reasonable results. Unfortunately, as of now, LLMs tend hallucinate solutions such cases instead of returning an empty preference function. Possible solutions could be to perform out-of- distribution detection for in-context examples fine-tuning LLM large set of preference examples improve generalization. Furthermore, assume the user preference is fully communi- cated through language instructions. Most human-human interactions, however, include non non-verbal communication. which Text2lnteraction currently does not capture. Future work could integrate other forms of communication through methods like gesture recognition and vision language mod- els.
We are using the OpenAl opt--401255-reviia model with context length of 128 000 tokens. The average query time 29 + 9.s.
8

---

Acknowledgments
The authors gratefully acknowledge financial support by the Horizon 2020 EU Framework Project CONCERT under grant 101016007, and by the Deutsche Forschungsgemeinscaff (German Re- search Foundation) under grant number AL 1185/33-1. This research was funded in part by the Federal Ministry of Education and Research (BMBF), and supported by fellowship within the IFI programme of the German Academic Exchange Service (DAAD). Toyota Research Institute pro- vided funds to support this work. This work was also supported by the National Acronautics and Space Administration (NASA) under the University Leadership Initiative (ULI) program.
Fedecal Miritry Educcasion and Research
DAAD
Deutscher Mkddeniadhe Austousch Exchange
References
[1] Semeraro, A. Griffiths, and A. Cangelosi. Human-robot collaboration and machine learning: systematic review of recent research. Robotics and Conputer-Integratee Manufacturing, 79:
1-16, 2023.
[2] S. Nertinger, R. Kirschner, A. Naceri, and Haddadin. Acceptance of remote assistive robots with and without human-in-the-loop for healtheare applications. International Journal of Social Robotics, 2022.
[3] M. Prats, S. Wieland, T. Asfour, A. P. del Pobil, and R. Dillmann. Compliant interaction in household environments by the Armar-IIl humanoid robot. In Proc. of the IEEE/RAS Int. Conf. on Humanoid Robots (ICHR), pages 475-480, 2008.
[4] B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. To- shev, V. Vanhoucke, F. Xia, Xiao, P. Xu, M. Yan, N. Brown, Ahn, O. Cortes, N. Sievers, C. Tan, Xu, D. Reyes, Rettinghouse, Quiambao, Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as can, not as I say: Grounding language in robotic affordances. In Proc. of the Conf. on Robot Learning (CoRL), 2022.
[5] K. Lin, C. Agia. T. Migimatsu, M. Pavone, and J. Bohg. Text2Motion: From natural language instructions to feasible plans. Autonomous Robots, 2023.
[6] Z. Yang, L. Ning, H. Wang. T. Jiang, S. Zhang, S. Cui, H. Jiang, C. Li, S. Wang, and Z. Wang. Text2Reaction Enabling reactive task planning using large language models. IEEE Robotics and Automation Letters, (5):4003-4400
2024.
[7] D. Losey, A. Bajesy, M. K. O' O'Malley, and A. D. Dragan. Physical interaction as communi- cation: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41 41(1):20 2022.
[8] G. R. Ghosal, M. Zurek, D. S. Brown, and A. D. Dragan. The effect of modeling human rationality level on learning rewards from multiple feedback types. In Proc. of the AAAI Conf. on Artificial Intelligence (AAAI), volume 37, pages 5983-5992, 2023.
[9] T. Fitzgerald, P. Koppol, P. Callaghan, R. Q. H. Wong, R. Simmons, O. Kroemer, and H. Ad- moni. INQUIRE: Interactive querying for user-aware informative reasoning. Proc. of the Conf. on Robot Learning (CoRL). volume 205, pages 2241-2250 2022.
[10] X. Yu, J. Wu, C. Xu, H. Luo, and L. Ou. Adaptive human-robot collaboration control based on optimal admittance parameters. Journal of Shanghai Jiaotong University, 27(5):589-600 601, 2022.

---

[1]] W. Lu, Z. Hu, and J. Pan. Human-roke collaboration using variable admittance control and hu- man intention prediction. In JEEE Int. Conf. on Automation Science and Engineering (CASE), pages 1116-1121, 2020.
[12] A. Bestick, R. Pandya, R. Bajesy, and A. D. Dragan. Learning human ergonomic preferences for handovers. In Proc. of the IEEE Int. Conf. Robotics and Automation (ICRA), pages 3257-3264, 2018.
[13] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. Gonzalez Arenas, H.-T. Lewis Chiang, Erez, L. Hasenclever, J. Humplik, B. Ichter, Xiao, P. Xu, A. Zeng, T. Zhang, N. Heess, D. Sadigh, Tan, Tassa, and Xia. Language rewards for robotic skill synthesis. In Proc. of the Conf. Robot Learning (CoRL), pages 1-31, 2023.
[14] M. D. Zhao, R. Simmons, and H. Admoni. Learning human contribution preferences in col- laborative human-robot tasks. In Proc. of the Conf. on Robot Learning (CoRL.) 2023.
[15] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, Zhong, Y. Yang, and T. Yu. Text2Reward: Reward shaping with language models for reinforcement learning. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2024.
[16] Y. J. Ma, W. Liang, G. Wang. D.-.. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2024.
[17] M. Hwang, L. Weihs, C. Park, K. Lee, A. Kembhavi, and K. Ehsani. Promptable behaviors: Personalizing multi-objective rewards from human preferences. In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 16216-16226, 2024.
[18] C. Liu, J. Hamrick, Fisac, Dragan, K. Hedrick, S. Sastry, and T. Griffiths. Goal inference improves objective and perceived performance human-robot collaboration. In Proc, of the Int. Conf. on Autonomous Agents and Multi Agent Systems (AAMAS), pages 940-948, 2016.
[19] N. Wilde, D. Kulić, and S. L. Smith. Learning user preferences in robot motion planning through interaction. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 619 626, 2018.
[20] Gerard Canal, G. Canal, Guillem Alenyå, G. Alenyå, Carme Torras, and C. Torras. Adapting robot task planning user preferences: An assistive shoe dressing example. Autonomous Robots, 1333-13566 2019.
[21] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, Wahid, J. Tompson, Q Vuong, Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-E: An embodied multimodal language model. In Proc. the Int. Conf. on Machine Learning (ICML), 2023.
[22] J. Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 9493-9500, 2023.
[23] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Generating situated robot task plans using large language models. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 11523-11530, 2023.
[24] N. Yoshikawa, M. Skreta, K. Darvish, Arellano-Rubach, Z. Ji, L. Bjørn Kristensen, A. Z. Li, Y. Zhao, H. Xu, A. Kuramshin, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Large language models for chemistry robotics. Autonomous Robots, -078 2023.
10

---

[25] L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Proc. of the Int. Conf. on Neural Information Processing Systems (Neur!PS), volume 36, pages 79081-790944 2023.
[26] Z. Zhao, W. S. Lee, and D. Hsu. Large language models as commonsense knowledge for large-scale task planning. In Proc. of the Int. Conf. on Neural Information Processing Systems (NeurlPS), volume 36, pages 31967-31987, 2023.
[27] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. LLM+P: Empowering large language models with optimal planning proficiency, 2023. URL http://arxiv. org/ /ag//bb//a aaalaba//233 2304. 11477.
[28] T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. Kaelbling, and M. Katz. Generalized planning in PDDL domains with pretrained large language models. In Proc. of the AAAI Conf. on Artificial Intelligence (AAAT), volume 38, pages 20264,00244 2024.
[29] C. Aeronautiques, A. Howe, Knoblock, D. McDermott, Ram, M. Veloso, D. Weld, D. W. Sri, A. Barrett, and D. Christianson. PDDL the planning domain definition language. Technical Report, 1998.
[30] S. Singh, K. Swaminathan, R. Arora, R. Singh, A. Datta, D. Das, S. Banerjee, M. Sridharan, and M. Krishna. Anticipate collab: Data-driven task anticipation and knowledge-driven planning for human-robot collaboration, 2024. URL http: http///axvvvv/ee///e//eee/eaaav/ 03587.
http:///axxxxvx..aab/aa wrw.:/o..rrr/bbeeeeeeeeeeee ab2/2444
[31] R. Arora, S. Singh, K. Swaminathan, A. Datta, S. Banerjee, B. Bhowmick, K. M. Jatavallab- hula, M. Sridharan, and M. Krishna. Anticipate & act: Integrating Ilms and classical planning for efficient task execution in household environments. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2024.
[32] A. Narcomey, N. Tsoi, R. Desai, and M. Vázquez. Learning human preferences over rohot be- havior as soft planning constraints, 2024. URL BuRLhtt:p//axxii http:////aa......g///////4.0..0
[33] T. Munzer, M. Toussaint, and M. Lopes. Preference learning on the execution of collaborative human-robot tasks. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 879-885, 2017.
[34] R. Schulz, P. Kratzer, and M. Toussaint. Preferred interaction styles for human-robot collabo- ration vary over tasks with different action types. Frontiers in Neurorobotics, 12, 2018.
[35] Y. Cheng, L. Sun, and M. Tomizuka. Human-aware robot task planning based on a hierarchical task model. IEEE Robotics and Automation Letters, 6(2)::136-111 2021.
[36] A. Noormohammadi-Ase S. L. Smith, and K. Dautenhahn. To lead or to follow? adaptive robot task planning in human-robot collaboration, 2024. URL 2401. . 010014444 144..
[37] H. J. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS), volume 33, pages 4415 4426, 2020.
[38] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan. Feature expansive reward learning: Rethinking human input. In Proc. of the ACM/IEEE Int. Conf. on Human-Ron Robot Interaction (HRI), pages 216-224, 2021.
[39] E. Biyik, D. Losey, M. Palan, N. C. Landoli, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67, 2022.
11

---

[40] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Proc. of the Conf. on Robot Learning (CoRL), pages 342-352, 2021.
[41] A. Xie, F. Tajwar, A. Sharma, and C. Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS), 2022.
[42] D. Hejna III and D. Sadigh. Few-shot preference learning for human in- un-the-loop RL. In Proc. of the Conf. on Robot Learning (CoRL), pages 2014 2025, 2022.
[43] Hejna and D. Sadigh. Inverse preference learning: Preference-based RL without a reward function. In Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS), 2023.
[44] S. Choi, K. Lee, A. Park, and S. Oh. A nonparametric motion flow model for human robot cooperation. Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 7211-7218, 2018.
[45] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2022.
[46] A. Peng, A. Bobu, B. Z. Li, T. R. Sumers, I. Sucholutsky, N. Kumar, T. L. Griffiths, and J. A. Shah. Preference- language-guided abstraction. In Proc. of the ACM/IEEE Int. Conf. on Haamann-Robo Interaction (HRI), pages 572-581, 2024.
[47] Y. Zeng and Y. Xu. Learning reward for physical skills using large language model. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
[48] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models. Proc. of the Conf. on Robot Learning (CoRL), 2023.
[49] X. Wang, K. Lee, K. Hakhamaneshi, P. Abbeel, and Laskin. Skill preferences: Learning extract and execute robotic skills from human feedback. In Proc. of the Conf. on Robot Learning (CoRL), pages 1259-1268, 2022.
[50] L. Roveda, J. Maskani, Franceschi, A. Abdi, Braghin, L. Molinari Tosatti, and N. Pe- drocchi. Model-based reinforcement learning variable impedance control for human-robot collaboration. Journal Intelligent & Robotic Systems, 100(2):417-433, 2020.
[51] M. Wu, Y. He, and S. Liu. Shared impedance control based on reinforcement learning in huban-robot collaboration task. In K. Berns and D. Görges, editors, Advances in Service and Industrial Robotics, pages 95-103, 2020.
[52] A. Ghadirzadeh, X. Chen, W. Yin, Z. Yi, M. Björkman, and D. Kragic. Human-centered collaborative robots with deep reinforcement learning. IEEE Robotics and Automation Letters, 6(2):566-53) 571, 2021.
[53] Y. Huang, J. Silvério, L. Rozo, and D. G. Caldwell. Generalized task task aas-paarmeterrzzed skill learning. In Proc. of the IEEE Int. Conf. on Roboties and Automation (ICRA), pages 5667- 5474, 2018.
[54] C. Agia. T. Migimatsu, J. Wu, and Bohg. Stap: Sequencing task-agnostic policies. In Proc. of the IEEE Int. Conf. Robotics and Automation (ICRA), pages 7951-7958, 2023.
[55] A. Pereira and M. Althoff. Safety control of robots under computed torque control using reachable sets. In Proc. of the IEEE Int. Conf. Roboties and Automation (ICRA), pages 331-338, 2015.
12

---

[56] S. Schepp, J. Thumm, S. B. Liu, and M. Althoff. SaRA: A tool for safe human-row robot co- existence and collaboration through reachability analysis. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 4312-4317, 2022.
[57] J. Thumm and M. Althoff. Provably safe deep reinforcement learning for robotic manipulation in human environments. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pages 6344-6350, 2022.
[58] ISO. Robotics safety requirements - -art 1: Industrial robots. Technical Report DIN EN ISO 10218-1:2021-09 DC, International Organization for Standardization, 2021.
[59] ISO. Robots and robotic devices collaborative robots. Technical Report ISO/TS 15066:2016(E), International Organization for Standardization, 2016.
[60] D. Beckert, A. Pereira, and M. Althoff. Online verification of multiple safety criteria for a robot trajectory. Proc. of the IEEE Conf. on Decision and Control (CDC), pages $454-6461 2017.
[61] J. Thumm, G. Pelat, and M. Althoff. Reducing safety interventions in provably safe reinforce- ment learning. In Proc. of the IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pages 7515-7522, 2023.
[62] S. B. Liu and M. Althoff. Online verification of impact-force-limitte control for physical human-robot interaction. In Proc. tfe IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pages 777-783, 2021.
[63] I. Botev, D. Kroese, optimization. In Handbooo of Statistics, volume 31, pages 35-59, Elsevier, 2013.
Rubinstein, and L'Ecuyer. The cross-entropy method for
[64] G. Hoffman. Evaluating fluency in human-robot collaboration. IEEE Transactions on Human- Machine Systems, 49(3):209-1)) 218, 2019.
[65] R. Lowry. Concepts and Applications of Inferential Statistics. Online available at http://waasarrtaas.eeeeeeoooodd 2014.
[66] A. Elhafsi, Sinha, C. Agia, Schmerling, I. D. Nesnas, and M. Pavone, Semantic anomaly detection with large language models. Autonomous Robots, 2023.
13

---

Appendices
A Controller parameters
The controller parameters used our user study are listed in Table We define two different max- imal joint acceleration and jerk values. The values dg and ja define how fast the robot accelerates along its primitive. The values 4max and Jmax define how fast the robot is allowed decelerate in case human intersects its path. The value safe defines the maximal Cartesian velocity that point on the robot geometry is allowed to have upon contact with human.
Table 1: Controller parameters used in user study
Parameter
Description
Sheginner
Cinternediate
Sexpert
Umax
Maximal joint velocity [rad/s]
0.25
0.4
2.0
ag
Maximal joint acceleration on the primitive [rad/s²]
0.5
2.5
Omax
Maximal joint acceleration [rad/s*]
2
5
20
je
Maximal joint jerk on the primitive [rad/s*]
2
5
15
Jmax
Maximal joint jerk [rad/s*]
100
200
400
Usafe
Maximal contact velocity (Cartesian) [m/s]
0.05
0.15
0.2
B Prompt design
Our prompts contain the following elements, which are further summarized Fig. 5.
Place the red box left of the blue box
Prompt
Introduction
Purpase:
You are an interactive robot pianner... State description: There table with a red box blue box it...
Objective: Write Python function that evaluates the given action would satisfy the user Instruction...
Building blocks
Geometric functionaiity
pase getPPase (obj)
p(x)
Measurement functions
+
metric posNorm (pose_1, pose_2)
Probabiitty functions
probability linear_prob (metric, t_1, 2_2)
12
In-context examples
Instruction: Arrange the mug the plate straight line.
Preference function:
def E..aalghtLieeeerraaggeeet (state, action):
Figure 5: Structure of the prompt used to generate the output of Text2lnteraction.
First, we use fixed system prompt comprising:
Introduction and purpose of the system.
Definition of terminology: task plan, primitive, control parameters, and preference func- tion.
State definition: objects, predicates, and relationships.
Available manipulation primitives.
14

---

Objective: Structure and constraints of: task sequence, controller parameters, and prefer- ence functions.
Preference function signature in Python code.
Then, provide number of building block functions as listed in Fig. 6 that the LLM use to construct the preference function in Python code.
the pose of object in speci fied frame from the current envi ronment state.
pose = getPose (state, obj, frame-'word world")
# Get the L1, 12, or iiiff norm of the positional difference of the two poses. Select axis evaluate the on. metric positienNorm (pose_ pose_2, norm=' axis=l'x', 'y', 1)
Calculate the difference in retation of two poses using the great circle distance.
metric greatcireleDistance (pese_l, pose_2)
Evaluate if an object is pointing in a given direction. Rotates the
given main axis by pose_l.orientation and calculates the greatCircleDistance between the rotated axis and pose_2. position.
metric poiniingginireetiineettt. (pose__,, pose_2, main_ axis= 0, 01)
# Calculate the rotational difference between pose_ 1 and pose_ around the given axis.
metric rotationAngle (pose_l, pose_ axis) Return 1.0 if metric > t, 0.0 otherwise. vice versa if not direction.
12 prob = (hresholddeetrts # Return 1.0 if metric >= t_2, 0.0 if metric < 1, and linearly interpolate otherwise. And vice versa if not direction.
t, directionstrue)
14
prob linear (metric, t_t, t_2, direction-truc)
15
Normal cummulative distribution function with given mean and standard deviation
16.
prob normal (metric, mean, std_dev, di directionttrue) prob AND (probb_,, prob_2)
18
preb OR (prob__l, 2rop_2)
19
Figure 6: Helper functions defined in experiments construct the motion preference function.
number of in-context examples in the prompt serve as a guideline for the LLM to construct outputs the correct format. Our in-context examples start with an instruction with the following elements:
objects in the scene, predicates and relations of the objects to each other. Orientation definition: where is front/behind, left/right, up/down.
State description:
User instruction
If the task plan is already fixed, like in our experiments, we provide here.
If the controller parameters are already fixed, like in our experiments, we provide them here.
Each in-context example defines hand-scripted solution to its given instruction consisting of task plan d1.H, the controller parameters {i.H, and a sequence of preference functions Fi.H- Finally, we add the actual instruction to the prompt consisting of the same elements as the example instructions the in-context examples.
C Examples for preference representations
In this section, provide examples how Text2Interaction integrates human preference effectively.
15

---

Object arrangement When given instruction è: "Place the object left of the blue box", Tex:2Interaction returns a preference function Ft (s 0z) of the place primitive as exemplified Fig. 7.
def 1eLeftorBluuBloxxeears action): Get the blue box pose in the current state blue_box_pose set_pose "blue_box")
Get the predicted object pose after executing the action next_state cransition_fastattf object_pose
action)
(ne_x_.snnet_sstaaae "object")
Evaluate if the object is placed left of the blue box
left [0.0, 1.0, 0.0]
diff_leff = weaatioondiffallloo direction blue_box.pose, left)
The direction difference should be greater zero.
t_o = 0.0
A distance of 10cm is preferred.
t_1 = 0.1
is_left..proabbili = linear, baa.ar.poobbilityy(ffff.... t_0, t_1) return is_left.probabill
Figure 7: Example preference function for the instruction "Place the object left of the blue box".
Safety preference In our recurring example Fig. 1, the user might instruct do not want to get hit by the tip of the screwdriver". Text2Interaction could respect this safety concern in two different ways:
1. A restrictive controller is selected, e.g., O'soo('- Ferneal), which guarantees that the robot comes full stop early before collision could occur. Then, do not need account for any discomfort due to collisions in the preference function. For example, we could pick an action, where the tip of the screwdriver faces the human most of the time and only switches directions in the end.
2. A less restrictive controller is selected, e.g., Ocootacct' Sintermediat). We could then account for the instructed preference on the motion level by requiring all handover actions to have the tip of the screwdriver face away from the human during the entire trajectory.
Carefulness Another example, where Text2Interaction has multiple ways of incorporating human preference into the plan of the robot follows the instruction "Put the vase on the desk carefully!". As an example, Text2Interaction could integrate the instructed preference on two different levels:
On the motion level as a preference function: We could take the angle between the base of the vase and the desk surface as a metric. small angle could then relate a more careful, and therefore preferred, touchdown.
2. On the control level: Text2Interaction could choose an admittance controller (Acoopliantt) instead PID-controller with a low stiffness and low overall low velocity, which would lead softer touchdowns.
Comparison to baseline objectives Our objective in (6) leads an optimization of the product of Q-functions and preference functions. In our ablation study, compare approach to the common weighted sum of rewards formulation used in many baselines [13-16]. In Fig. 8, we give an intuition why our formulation outperforms the baselines.
D Derivations
In this section, derive the approximations of (5) under the assumptions of the Markov property, ie., all information of $14-11 01:t-1 is contained in Additionally, we assume that the user has
16

---

User
Skill sequence
Baseline
Baseline
Text2lnteraction
screwdhiver
grab
3.0,
U.8, F 0.0
Q
Σ
2.
+ F
score II
Q.F.
Trsk planner
2. Handoverssrewwddive hand)
3. Pick(ycrewdrier)
0.8
1.8
= 0..0
Safety controller
Stop
Score
I
score Σ
+
5000e II
Q.F
Preleence function
2.5
0.0
F. RickAtlodProbbdilsscc sl Fg HannDDerr/hianddeehhaaiit. a
Q1 = 02, Fi 0.7
C = 044 Pt
09
I Σ
50000 Q.P.
0.18 2.4
= 0.08
The user asks the robot to hand them the screwdriver so that they can comfortably grasp Previous methods would only output task plan and optimize it for task success (baseline [5]). or treat preference as an additive reward, leading to unsuccessful executions (baseline 2 [13-16]). Text2Interaction takes task-, motion- and control-level preferences into account. In this example, our system outputs two preference functions that evaluate if the screwdriver picked by the rod and if it then handed over by the handle, encouraging plans toward their satisfaction. The skill sequence in the first row is the most likely to succeed, but not preferable as the handle is not easily graspable. The second sequence includes an action that is most likely going to fail, as the robot would try to grasp the very tip of the screwdriver. The last row depicts a skill sequence that is preferable and executable, From these three skill se- quences, baseline 1 would select the first sequence, as it is the most likely to succeed, and baseline 2 would seleet the second sequence, as maximizes the sum over Q-function and preference func- tions. Only Text2Interaction would correctly select the third skill sequence.
Figure 8: Preference-aligned planning with Text2Interaction.
motion preference in each individual skill of the sequence resulting in the events
Vt 1...., H Smation the action a satisfies the motion preferences at time step t starting in st- Hereby, assume that the motion preference St does not depend on the future environment history ($1+1-H+1, v1+1:1). This assumption was reasonable in our experiments, but there might be more complex long-horizon dependencies for which this assumption fails. The overall motion preferences of the user are only fulfilled if their motion preferences at each time step are fulfilled, i.e. Smotion S1 S1 mation A ... A H motioon We give a more detailed explanation of the steps and approximations involved in (9) following the equation.
pl Smation 81, a1:H, i, 01.H 01.H. E1. Steasible Stask; Soontrol
(9a)
p
(SL motion |1 (1, K) p mation 11 a1:H, Smotioon k:)
(9b)
(St (Shotion
(81, x) p (52:H 82, 01:H, S1 S 13) (82 181 01, S x) da2 mation
(9c)
(PS) (Shotion
|81, 11,,a,, x) E.g [p (S2tH $2, 02.H Smotionn xi)
(9d)
Ti (-11.1 v1)
p ( (S
81, a1, k2) (S2.H Shooto
$2; O2.H Imotion Rt)
(9c)
Repeat steps (9b)-(9e) with 8141 T₁
St, 1/2):
p (S)
mntion $1, x))
p
S motion 82, 022 S1 mmotion 15) ..... (S Iiotion
84H 0.1 mution
k)
(9f)
H
II (Sthotion Sz, az, Stition x)) [=1
(gg)
H Fr at),
2 1=1
II
(9h)
where Es, [ refers the expected value all possible states 82. Note, that in (9d), the pected value over all possible states 52 is conditioned on the action 01 state 5 and on the events
17

---

that the execution of primitive ti is feasible and preferable. To derive (9c), we use the assump- tion that S1 motion is independent of 02:H- For (9e) we use the Markov assumption. To retrieve the estimate in (9f), we follow the proposed procedure single Monte-Carlo sample estimate of (9e) under the state transition Ti (#2 81, th). Here, the key insight is that we only execute skill sequences if the problem in (6) finds a valid action se- quence that fulfills the preferences and is feasible. Therefore, it is reasonable to assume that the condition events S mation and S Reasibility in (82 81, 01 S motionn k) occurred. The approxima- tion E. [p (S $2, : St mmotion x)] Rs p (Smoiion $2; Q2: 122. Smotion: k) was reasonable our experiments the learned transition distributions had a low variance. We then repeat steps (9b)- (9e) to get the approximation in (9f), which can be written (9g). Finally, for (9h) assume that the probability functions (s, a) returned by the LLM approximate the true underlying probabilities of user satisfaction by the given actions. In our ablation study, we saw that this assumption held if the new task was semantically in distribution [66] of the given in-context examples. For example, if we give Tex:2Interaction the screwdriver handover as an in-context example, produces reasonable preference functions for the request "Hand me an ice cream", but fails for the request "fold the piece of paper square".
[5, Appendix C.I], where we compute a
18


Reviews:
[{"review_id": "OzWn2zshMT", "reviewer": "Reviewer_dJ5U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Text2Interaction is a method that outputs user preference function, which is used as a multiplier for optimization objective to integrate user preference into task and motion planning.", "summary_of_recommendation": "This paper has its technical merit for the design that makes user preference easily integrates into planning. This is useful for HRI research community. The major concern is around the generalizability: how would this be applied to other settings including different scenarios, and user preference that cannot be expressed through the building block functions.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. I am confused about the concrete implementation of $S_{task}$ and why *sum of token log-probabilities of each primitive’s\nlanguage description* is a good optimization objective? \n2. I might have missed this, but why didn't you compare with $\\Sigma Q + F$ in user study? And would $\\Sigma Q+F$ be different from $\\Pi Q+F$?\n3. How could user preference describing motion be expressed? For example, *put the vase on the desk carefully*. This may be out of scope of Fig. 5", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kR3VXvRNU7", "reviewer": "Reviewer_BygU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a framework named Text2Interaction, which aims to integrate human preferences into robot task planning, motion planning, and control in a seamless manner. The approach leverages large language models (LLMs) to generate task plans, motion preferences as Python code, and parameters for a safe controller. By optimizing for both task completion and user satisfaction, Text2Interaction aims to provide a more adaptable and responsive human-robot interaction. The framework is validated through a user study on real robots and an ablation study in simulation, showing promising results in preference integration and user satisfaction.", "summary_of_recommendation": "This paper presents a novel framework for integrating human preferences into robot task planning, motion planning, and control using LLMs. While the approach is shows promising results, the paper needs clearer explanations on the formulation of the proposed method and the simulation experiments.", "technical_quality": 3, "clarity_of_presentation": 1, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Can the authors provide more detailed examples of how the preference functions generated by the LLMs align with user instructions?\n2. When does the shooting objective become zero? Could the authors report numbers on how often greedy-search happens and what those mean?\n3. Could the authors elaborate on the computational efficiency of Text2Interaction, particularly in real-time applications?\n\n[1] Xie, Tianbao, et al. \"Text2Reward: Reward Shaping with Language Models for Reinforcement Learning.\" The Twelfth International Conference on Learning Representations.\n\n[2] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\" arXiv preprint arXiv:2310.12931 (2023).\n\n[3] Hwang, Minyoung, et al. \"Promptable behaviors: Personalizing multi-objective rewards from human preferences.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2Motion: From natural language instructions to feasible plans. Autonomous Robots, 2023\n\n**Update after rebuttal:**\nI have carefully read the authors' rebuttal and, in particular, the answers to the questions I raised. I am glad that the authors have experimented with the correct OR implementation and also fixed minor errors in the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSsjHtlorI", "reviewer": "Reviewer_VQtk", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method to improve the way in which human preferences are accounted for in robot planning, in addition to task completion. The proposed method jointly optimizes for human preferences at each of the three levels of abstraction: task planning, motion planning and control. The idea is that the task preferences account for finding the correct plan, the motion preferences for finding the desired way, and control preferences to enable a feeling of safety and comfort. The key to this method is using an LLM to translate a human instruction into a more detailed reward function, which can be used to optimize against. Through a user study, the authors show that Text2Interaction outperforms baselines in satisfying user preferences, without a decrease in overall task completion success.", "summary_of_recommendation": "The paper proposes a novel method to better align robot behavior to human preferences by relying on LLMs to detail such preferences from human instructions. The paper requires more theoretical or empirical justification to support two claims: first, how the sum v.s. product formulations are fundamentally distinct, and second, how the preferences at all three levels of abstraction are accounted for.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "In context of the above review:\n- How are preferences for each of three levels of abstraction are/are not taken into account?\n- How does the distinction between sum of rewards vs product of likelihoods stand for tasks beyond the evaluation setting?\n\nOther clarification questions and suggestions:\n- Since a large number of symbols are required to formulate the system, a block diagram annotated with the symbols would help serve as reference in parsing and tying together sections from notation to methodology.\n- Does the Q function only represent geometric feasibility only, or does it also take into account the task goal? Section 3.2 claims that geometric feasibility can be approximated as the product of Q-functions over the trajectory, but if it is trained on the target task, does it also account for task completion probability in addition to geometric feasibility?\n- The role of controllers in the overall system is unclear. The initial narrative suggests that the aim of the controller is to follow a trajectory, and can be parametrized by ‘eta’. But in the ‘Safe controller’ section, it is hard to connect what ‘eta’ actually constitutes. Are the control modes the ‘eta’? Relatedly, in the results of H1, it is unclear what ‘have stated a preference in the safety shield mode’ means.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of the Paper

The paper presents "Text2Interaction," a framework designed to seamlessly integrate human preferences into robot task planning, motion planning, and control. This approach employs large language models (LLMs) to generate task plans, motion preferences in Python code, and safe controller parameters. By maximizing the combined probability of task completion and user satisfaction, Text2Interaction aims to provide adaptable and responsive human-robot interaction. The framework is validated through both a real-world user study and an ablation study in simulation, which show promising results in user satisfaction and preference integration.

Strengths
- The method of using LLMs to integrate human preferences across task, motion, and control levels is novel and represents a major advance over existing methods that typically focus on a single aspect.
- The framework is evaluated through a combination of real-world user studies and simulation-based ablation studies, providing a strong empirical foundation. The results demonstrate statistically significant improvements in user satisfaction and preference integration.
- The ability of Text2Interaction to react to new user preferences in a zero-shot manner highlights its potential for adaptable and flexible human-robot interaction, which is a significant challenge in the field.
- The framework places a strong emphasis on aligning with human preferences, which is crucial for enhancing the efficacy and acceptance of robots in various applications.

Weaknesses
- The user study was conducted with a single task scenario, raising concerns about the generalizability of the approach across diverse setups and tasks. Additional experiments with a wider variety of tasks would strengthen the validation.
- While the paper is technically sound, the presentation, particularly in the methodology section, could be clearer. The paper employs complex notations and lacks intuitive explanations, which may hinder understanding.
- There is a lack of theoretical or empirical justification for certain claims, particularly regarding the distinctiveness of the sum versus product formulations in optimizing task completion and human preference scores. This needs further exploration.
- The preference functions are constrained by the primitives available, which may limit the framework's applicability to open-domain tasks or preferences not easily expressed by the predefined building blocks.
- The reliance on LLMs raises questions about robustness and generalizability to diverse and unexpected user instructions, which could impact the framework's applicability in dynamic real-world settings.

Summary of the rebuttal phase

While the authors' revisions appear to have improved the quality of the paper, the rebuttal did not change the reviewers' opinions. Although not all reviewers agree, accepting as a poster presentation is considered reasonable.
Decision: Accept