Text2Interaction: Establishing Safe and Preferable Human-Robot Interaction

Authors: jakob  thumm
christopher  agia
marco  pavone
matthias  althoff
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 10:54:29

Abstract: Adjusting robot behavior to human preferences can require intensive human feedback, preventing quick adaptation to new users and changing circumstances. Moreover, current approaches typically treat user preferences as a reward, which requires a manual balance between task success and user satisfaction. To integrate new user preferences in a zero-shot manner, our proposed Text2Interaction framework invokes large language models to generate a task plan, motion preferences as Python code, and parameters of a safety controller. By maximizing the combined probability of task completion and user satisfaction instead of a weighted sum of rewards, we can reliably find plans that fulfill both requirements. We find that 83% of users working with Text2Interaction agree that it integrates their preferences into the plan of the robot, and 94% prefer Text2Interaction over the baseline. Our ablation study shows that Text2Interaction aligns better with unseen preferences than other baselines while maintaining a high success rate. Real-world demonstrations and code are made available at [sites.google.com/view/text2interaction](sites.google.com/view/text2interaction).

Main paper: 
 introduction: 
 We are moving toward a future where robots are fully integrated into our everyday lives, from manufacturing [1] to healthcare [2] to households [3] . In these settings, robots must quickly adapt to individual human preferences and changing circumstances. However, current robotic platforms severely lack in this aspect, which prevents their application to daily tasks and widespread acceptance. Recent works [4] [5] [6] present promising approaches to seamlessly incorporate task-level preferences, answering the question, "What should the robot do?" In human-robot interaction (HRI), we must further address two key user preferences about the behavior of the robot: motion preferences, which determine "Which path should the robot choose?", and control preferences, which dictate "How fast, soft, or precise should the robot be?" Recently proposed methods in motion planning [7] [8] [9] and safe control [10] [11] [12] tend to require labor-intensive human feedback to adapt to such preferences. Hence, these solutions learn a preferable behavior offline and, therefore, fall short when it comes to situational awareness and fast-changing preferences. Additionally, most recent works [9, [13] [14] [15] [16] [17] treat human preferences as an additive reward, requiring them to carefully balance task success and user satisfaction.In this work, we propose Text2Interaction, a framework to incorporate preferences from a single user instruction in three levels of the robot software stack: task planning, motion planning, and control. As exemplified in Fig. 1 , we react to user preferences online by querying a large language 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. S�ffness, Damping LLM (GPT4)Figure 1 : Preference-aligned planning with Text2Interaction. The user asks the robot to hand them the screwdriver so that they can comfortably grab it. Text2Interaction queries an LLM to return (a) a sequence of primitives that satisfy the task preferences, (b) a set of motion preference functions as executable Python code, and (c) a set of parameters that adjust the safety controller to the current situation and control preferences of the user. Our planner than aims to find a plan that satisfies the user preferences and is feasible to execute. If the planning step fails, we query the LLM to return the optimal skill for the next timestep only, as discussed in Sec. 6 .model (LLM) to obtain (a) a sequence of primitives as task preference, (b) preference functions written in Python code that evaluate how well an action aligns with the instructed motion preferences, and (c) a set of parameters that adapt our provably safe controller to the control preferences of the user. Our formulation results in the maximization of the combined probability that the plan is feasible and the user is satisfied instead of the commonly used weighted sum of rewards [9, [13] [14] [15] [16] .Our ablation study showed that our formulation leads to a robot behavior that is twice as preferable as the baseline while maintaining a high success rate in unseen tasks. Of the 18 participants in our real-world user study, 83 % stated that Text2Interaction considers their preferences, and 94 % preferred Text2Interaction over the baseline [5] . To summarize, our core contributions are:1. We propose Text2Interaction, a framework for integrating human preferences in robot task planning, motion planning, and control.2. We derive a task and motion planning formulation that optimizes the likelihood of human satisfaction and task success online.3. We evaluate Text2Interaction (a) in a user study on a real human-robot collaboration task with 18 participants and (b) in simulation on a set of geometric object rearrangement tasks. 
related work: 
 Early works in adaptable task planning use a known world model that can be adjusted online to human feedback [18] [19] [20] . A number of more recent works propose using LLMs to directly convert language instructions into task plans, either using text [4, 5, 21] or code [22, 23] . Subsequent works attempt to improve the planning robustness of LLMs through iterative re-prompting [24] or by leveraging formal logic representations [25] [26] [27] [28] , e.g., in the form of a planning domain definition language (PPDL) [29] description. The works in [30] [31] [32] extend such a PPDL formulation to collaborative scenarios. Multiple works [14, [33] [34] [35] [36] further show how these approaches can allocate tasks more preferably in HRI.A common way to incorporate human preference on the motion level is to learn a reward function that represents the human intent, typically by asking comparative questions [17, [37] [38] [39] [40] [41] [42] [43] . The authors in [12, 44] demonstrated how this form of preference learning can perform preferable robot-human handovers. The main disadvantage of this type of reward learning is that it either only reflects simple linear reward functions or requires thousands of human queries. As such, several recent works construct reward models with LLMs and use them to train [15, 16, [45] [46] [47] or directly synthesize [13, 48] robot skills. Notably, these approaches consider preferences for individual skills, whereas we focus on entire skill sequences. Additionally, these methods generally produce reward functions as linear combinations of reward terms, which requires careful weighing of the reward terms. Closely related to our work is that of Wang et al. [49] , which proposes to incorporate human feedback into longhorizon planning by extracting skills from demonstrations and training two models: one to predict the correct sequence of skills and one to predict the parameterization of these skills. However, their approach relies on a labor-intensive offline labeling phase and assumes that the preferences of all future users align with these offline labels.A broad range of works [10, 11, [50] [51] [52] further proposes to learn the parameters of admittance controllers from human feedback to achieve comfortable object handling. Huang et al. [53] demonstrated the effectiveness of this approach in an interactive handover task. 
notation: 
 We denote a skill as the tuple ψ = (ϕ, a, α, ξ). The primitive ϕ stems from a library of K primitives L ϕ = {ϕ1 , . . . , ϕ K } 1 , each of which is parameterized by a set of continuous parameters a ∈ A k , k ∈ 1, . . . , K (hereafter referred to as actions). The controller α ∈ L α = α 1 , . . . , α M with parameters ξ ∈ Ξ m , m ∈ 1, . . . , M returns the system inputs θ = α(ϕ k , a k , t, ξ) to follow the parameterized primitive at time t. For our problem, we define a Markov decision process (MDP) with the tuple M = (S, Ψ, T, r, S 0 ). Here, S is the continuous state space,Ψ = L ϕ × A k × L α × Ξ m is the set of possible skills, T (s t+1 | s t , ψ t )is the transition distribution, and S 0 ⊆ S is the set of initial states. The reward function r : S × Ψ × S → {0, 1} returns r = 1 if the execution of a skill is feasible, and r = 0 otherwise. 
problem statement: 
 The user gives an instruction i, which may include their task, motion, and control preferences. Thus, we define two events2 for any environment history (s 1:H+1 , ψ 1:H ) := [s 1 , ψ 1 , s 2 , ψ 2 . . . , s H+1 ]:S feasible : r(s 1 , ψ 1 , s 2 ) = • • • = r(s H , ψ H , s H+1 ) = 1, i.e., the execution of ψ 1:H is feasible, S preference : ψ 1:H satisfies the user preferences in i when starting in s 1 .We assume that a user is satisfied with an environment history if the event S user : S preference ∧ S feasible occurs. Therefore, our goal is to maximize the probability of user satisfaction based on the instruction i and initial state s 1 withEQUATIONwhich represents the objective defined in [5, Eq. 2] . Contrary to Lin et al. [5] , which only adhere to task preferences, we assume that the user is only satisfied with the execution of a skill sequence ψ 1:H if their task, motion, and control preferences are satisfied. Therefore, we define the three events:S task : ϕ 1:H satisfies the task preferences in i when starting in s 1 , S motion : a 1:H satisfies the motion preferences in i, S control : α 1:H , ξ 1:H satisfy the control preferences in i.Using these events, we can redefine the preference satisfaction event as S preference : S task ∧ S control ∧ S motion , and its probability asp(S preference | i, s 1 , ψ 1:H , S feasible ) = p(S task , S control , S motion | i, s 1 , ψ 1:H , S feasible ) = (2) p(S motion | i, s 1 , ψ 1:H , S feasible , S task , S control ) p(S control | i, s 1 , ϕ 1:H , α 1:H , ξ 1:H , S feasible , S task ) (3) p(S task | i, s 1 , ϕ 1:H , S feasible ) ,where we assume that S task is independent of a 1:H , α 1:H , and ξ 1:H and S control is independent of a 1:H . Thus, to find the skill sequence with the highest likelihood of user satisfaction, we must solve The goal of this work is to derive a framework including reasonable assumptions and approximations to efficiently find a solution to (4) that adheres to the given preferences in an HRI setting.ψ ⋆ 1:H = arg 
supporting methodology: 
 This section introduces the primitive learning method and safety controller used in our work.Sequencing task-agnostic policies For each primitive ϕ k ∈ L ϕ , we use sequencing taskagnostic policies (STAP) [54] to learn a policy π k (a | s) from which one can sample actions for planning and a Q-value function Q k : S × A → [0, 1] which estimates the probability that executing primitive ϕ k parameterized by action a in state s is feasible. Additionally, we learn the transition distribution T (s t+1 | s t , ψ t ) to predict the next state. Given a dataset of transitions (s 0 , a 0 , s1 , r 0 ) ∈ D k = S 0 × A k × S × {0, 1}, STAP learns the functions π k , Q k ,and T using offline reinforcement learning. In our experiments, we reduce sample complexity by training these components independently from the controller α(•, ξ), i.e., we assume that the distribution of skill success is approximately constant across most controller configurations:∀α 1 , ξ 1 , α 2 , ξ 2 : p(S feasible | s 1 , ϕ, α 1 , ξ 1 , a) ≈ p(S feasible | s 1 , ϕ, α 2 , ξ 2 , a).This assumption holds in our experiments as the controller parameters mainly impact the speed with which the robot executes the primitive, but not its path.To ensure human safety, we use a provably safe controller [55] [56] [57] that adheres to ISO 10218-2 [58] and ISO/TS 15066 [59] . Our controller comes in three variants: α stop [57, 60] , α contact [60, 61] , and α compliant [62] . Using α stop , we guarantee that the robot comes to a complete stop before any contact with a human could occur. The α contact mode allows the robot to have a low speed close to the human and thereby enables active contact. Finally, the α compliant mode ensures low contact forces between the end-effector and the human using compliant Cartesian control. Generally, we can use the parameter vector ξ to adapt the maximal velocity, acceleration, jerk, stiffness, and damping of the controller. To simplify the control parameter selection for the LLM, we predefine a set of parameter vectors: ξ coexistence for non-interactive scenarios, ξ critical for high-risk scenarios, and three parameter vectors for interaction with varying user experience ξ beginner , ξ intermediate , and ξ expert . 
the text2interaction framework: 
 This section details how Text2Interaction satisfies the user preferences and achieves feasible plans.To find feasible skill sequences in long-horizon tasks, Lin et al. [5] propose two modes: shooting and greedy-search. In the following paragraphs, we derive the assumptions and approximations necessary to incorporate motion and control preferences into their approach and solve the problem in (4) efficiently. Fig. 2a summarizes our resulting planning objectives, and Fig. 2b exemplifies how Text2Interaction links the shooting and greedy-search modes.Shooting In the shooting step, we first let an LLM generate all elements necessary to plan a sequence of skills that fulfills the instruction of the user. For the task and control preferences, the LLM directly generates the sequence of primitives ϕ 1:H and controller settings α(•, ξ 1:H ). Since these elements are now fixed, our planner cannot influence them. Therefore, we set the probability of satisfying the task and control preferences in (4) to p(S task | i, s 1 , ϕ 1:H , S feasible ) ≈ c task and p(S control | i, s 1 , ϕ 1:H , α 1:H , ξ 1:H , S feasible , S task ) ≈ c control , with c task , c control ∈ (0, 1]. Hereby, we assume that the suggested task plan and controllers have a non-zero probability of satisfying the user. In the next paragraph, we will discuss how we proceed if this assumption fails. To approximate the probability that the execution of the plan is feasible, Lin et al. [5, Eq. 5] have derived that p(S feasible | s 1 , ψ 1:H ) ≈ H t=1 Q t (s t , a t ), where Q t denotes the Q-value function associated with the primitive ϕ t at time t. Analogously, we derive in Appendix D that we can approximate the probability that the user is satisfied with the action sequence using a set of preference functions asEQUATIONwheres t+1 ∼ T t (• | s t , ψ t ) for t ≥ 1.A preference function F (s, a) returns the likelihood that executing action a in state s fulfills the motion preferences of the user. We later discuss how we can construct such functions online and give a number of examples in Appendix C. We can now approximate our objective in (4) and find the optimal sequence of actions a ⋆ 1:H that both satisfy the motion preferences H t=1 F t (s t , a t ) > 0 and are feasible H t=1 Q t (s t , a t ) > 0, by using a crossentropy method planner [63] to solve the optimization problem (see Fig. 2a )EQUATIONGreedy-search If we cannot find a shooting plan that adheres to the human motion preferences and is feasible, i.e., H t=1 F t (s t , a ⋆ t ) Q t (s t , a ⋆ t ) = 0, we have to expect that our assumption that the task sequence is valid c task > 0 failed. In this case, we execute a greedy-search [5] , where we try to find the optimal next skill ψ ⋆ t instead of the entire skill sequence ψ ⋆ 1:H . We define the simplified event S t user = S t feasible ∧ S t task ∧ S t motion ∧ S t control , which occurs if the skill ψ t is feasible and adheres to the user preferences at time step t. Thus, the problem in (4) simplifies to where we made the Markov assumption. In greedy-search, we let an LLM return L ≤ K candidate primitives for the next skill. We then approximate the probability that the user is satisfied with the next selected primitive with the sum of token log-probabilities of the language description of each primitive p(S t task | i, s t , ϕ t , S t feasible , S 1:t-1 user ) ≈ P task (ϕ l t ), l ∈ 1, . . . , L as proposed in [4, 5] . These scores represent the likelihood that the textual label of a primitive is a valid next step for the instruction i [4] . We could proceed with the control preferences in the same way, but generating L 2 pairs of primitives and control candidates is time-consuming. Thus, we only generate one set of control parameters per candidate primitive and set p(S t control | i, s t , ϕ t , α t , ξ t , S t feasible , S t task , S ψ ⋆ t = arg max ψt p(S t feasible | s t , ψ t , SEQUATIONwhich gives us the optimal one-step primitive and action. After finding a solution to the greedysearch problem in (8) using a cross-entropy method planner, we return to the regular shooting strategy starting from state s t+1 .To approximate the probability that a human would approve action a t at time t, i.e., p(S t motion | i, s t , ψ 1:H , S t feasible , S t task , S t control ), we query an LLM to return a preference function F t (s t , a t ) as Python code based on the user instruction i, the primitive sequence ϕ 1:H , and the controller α(•, ξ 1:H ). To align the return of the LLM with our desired structure, we prompt it with a system and task description, a number of in-context examples [13] , and a set of programmatic building blocks. These building blocks provide the LLM with 1. the predicted next states t+1 = E st+1 [T (s t+1 | s t , ψ t )],2. the functionality to retrieve the pose of an object in a given frame from a given state, 3. functions to calculate metrics given a set of poses, e.g., the Euclidean distance of two objects, 4. a set of monotonic functions g p : R → [0, 1] that map the value of a given metric to a utility.By restricting the codomain of the utility function to [0, 1], we can interpret its image as the probability of user satisfaction. The LLM can evaluate multiple logical connectives in the preference function probabilistically using the AND(p 1 , p 2 ) = p 1 p 2 and OR(p 1 , p 2 ) = p 1 p 2 + p 1 (1 -p 2 ) + p 2 (1 -p 1 ) operators, where we assume that the events are stochastically independent. We provide more details about our prompt design in Appendix B. One key assumption that we make is that the preference functions F 1:H (s, a) returned by the LLM accurately reflect the true underlying probability p(S motion | i, s 1 , ψ 1:H , S feasible , S task , S control ), which we evaluate in our ablation study in Sec. 7.2. 
experimental evaluation: 
 We performed a user study and an ablation study to investigate three main hypotheses: H1 Users have preferences regarding the motion and control of interactive robots. H2 Text2Interaction integrates these preferences in the execution of the plan of the robot. H3 Text2Interaction generates preference functions that align with the instructions of the user.Our experiments focused on preferences on the motion and control level, as previous work [4, 5] already investigated task-level preferences extensively.To evaluate our research hypotheses, we set up our running example in Fig. 1 on a Franka Research 3 robot. Hereby, the robot had to pick up a screwdriver from the desk and hand it over to the user, as demonstrated in our supplementary video. Our user demographic consisted of 18 participants, of I preferred Text2Interaction (predefined) over the baseline.Communicating my preferences was easy.I have strong preferences how the robot executes its plan.Text2Interaction integrates preferences into the robot's plan.Strongly disagree Somewhat disagree Neutral Somewhat agree Strongly agree which 16 stated that they had previous experience in robotics. Our experiments then comprised four experimental stages, with the users performing the same task multiple times per stage and answering questions on a five-point Likert scale after each stage.In stage 1, we demonstrated our safety shield to the user. For this, we predefined three sets of controller parameters: ξ beginner , ξ intermediate , and ξ expert , listed in Appendix A, with each mode being faster and more reactive than the previous one. We then let the user decide which mode they would like to work with in the upcoming interactive task.In stages 2 and 3, the users performed the task of the screwdriver handover, where the robot had two different modes. The first mode was baseline 1 [5] , which only optimized for task success and did not incorporate motion level preferences. The second mode was Text2Interaction (predefined), which we previously queried with the instruction, "Hand me the screwdriver, and make sure the handle is pointing towards me so that I can comfortably grab the handle.".In stage 4, we asked each user for personal instructions to the robot for this task. We then queried Text2Interaction for new custom preference functions and performed the task in a zero-shot manner.After stages two to four, the users answered five questions about trust, intelligence, cooperativeness, comfort, and awareness [64] to evaluate the quality of the HRI. By asking the same questions repeatedly, we can evaluate if there is a difference in the distribution of answers between the stages using the Wilcoxon signed-rank test [65] and report the p-values. Fig. 3 further summarizes the answers to a set of general questions not associated with any specific stage. Our user study confirmed our hypotheses in the following ways:H1: First, users tend to agree that they have a strong preference for the way the robot executes the task, see Fig. 3 . Second, with statistical evidence, users perceived Text2Interaction (predefined) as more intelligent (p ≤ 0.005), more cooperative (p ≤ 0.01), and more trustworthy (p ≤ 0.05) than the baseline. Additionally, they were more comfortable with Text2Interaction (predefined) (p ≤ 0.025) and reported that the robot more accurately perceived what the goals of the users were (p ≤ 0.01) than with the baseline. Third, from our user group, 50 % chose ξ intermediate , and 50 % chose ξ expert as controller parameters, which indicates a general preference towards a faster robot for this task. Furthermore, 33 % of users have stated a strong preference for the safety shield mode, indicating that although a majority of users might be indifferent to the controller parameters, it is still relevant to integrate control preferences in HRI.H2: 83 % of users agreed that Text2Interaction integrates their preferences into the plan of the robot, and 94 % of users preferred the execution of Text2Interaction (predefined) over the baseline. The participants further rated the execution of their personal preferences, Text2Interaction (personal), as more cooperative (p ≤ 0.005) and comfortable (p ≤ 0.025) than the baseline and stated that the robot more accurately perceived what their goals are than the baseline (p ≤ 0.025).H3: 67 % of users preferred the robot behavior following their instruction over Text2Interaction (predefined). Additionally, 83 % of users stated that it was easy to communicate their preferences with Text2Interaction. To further quantify the validity of hypothesis H3, we performed an ablation study on a set of object rearrangement tasks. Our study aims to validate if Text2Interaction can generate valid preference functions from a limited set of in-context examples. We used 15 tasks consisting of an instruction, a task plan, and hand-crafted oracle preference functions F ⋆ 1:H . For each task, we defined three trials, and for each trial, we randomly selected three in-context examples out of the other 14 tasks to construct the LLM prompt 3 . We evaluated each trial 100 times with random initial states and calculated their average preference score based on the oracle preference functions F ⋆ 1:H . Finally, we calculated the mean success rates and preference scores together with their 95 % confidence intervals using bootstrapping. Our experiments included four agents: ... • Oracle, which used the hand-crafted preference functions (score = t Q t F ⋆ t ), • Baseline 1 was [5] , which only optimized for task success (score = t Q t ), • Baseline 2 was inspired by [13, 15, 16] , which treats preferences and the feasibility of the plan as rewards, and aimed to maximize the score= t Q t + F t ,• Text2Interaction with the objective defined in ( 6) (score= t Q t F t ).Text2Interaction successfully generated executable preference functions for all 45 trials. Fig. 4 shows that our approach achieves approximately the same or higher task success rates than both baselines but achieves significantly higher preference scores. Empirically, Text2Interaction mainly struggled with vaguely communicated preferences, such as, "Place object A as far left of the table as possible." This is because there are many different ways of interpreting the instruction, which leads to a natural discrepancy between Text2Interaction and the oracle. Overall, the results of our ablation support hypothesis H3 and indicate a robust generalization of Text2Interaction to new problems. 
conclusion and limitations: 
 We presented a framework to include human preferences from a single user instruction in three levels of the robotic software stack: task, motion, and control. By optimizing over the combined probability that the plan is feasible and satisfies the user instruction, we find task and motion plans that are more likely to fulfill both criteria than related baselines. The overwhelming majority of users found that Text2Interaction integrates their preferences easily.One limitation of our work is the dependence on in-context examples for the LLM prompts. If the user request is semantically out of distribution of these examples [66] , the LLM is unlikely to output reasonable results. Unfortunately, as of now, LLMs tend to hallucinate solutions in such cases instead of returning an empty preference function. Possible solutions could be to perform out-ofdistribution detection for in-context examples or fine-tuning an LLM on a large set of preference examples to improve generalization. Furthermore, we assume the user preference is fully communicated through language instructions. Most human-human interactions, however, include non-verbal communication, which Text2Interaction currently does not capture. Future work could integrate other forms of communication through methods like gesture recognition and vision language models.• Objective: Structure and constraints of: task sequence, controller parameters, and preference functions.• Preference function signature in Python code.Then, we provide a number of building block functions as listed in Fig. 6 that the LLM can use to construct the preference function in Python code.1 # Get the pose of an object in a specified frame from the current environment state. • Orientation definition: where is front/behind, left/right, up/down.• User instruction i.• If the task plan is already fixed, like in our experiments, we provide it here.• If the controller parameters are already fixed, like in our experiments, we provide them here.Each in-context example defines a hand-scripted solution to its given instruction consisting of a task plan ϕ 1:H , the controller parameters ξ 1:H , and a sequence of preference functions F 1:H . Finally, we add the actual instruction to the prompt consisting of the same elements as the example instructions in the in-context examples. 
 
 Appendix: In this section, we provide examples how Text2Interaction integrates human preference effectively.Object arrangement When given instruction i: "Place the object left of the blue box", Text2Interaction returns a preference function F t (s t , a t ) of the place primitive as exemplified in Fig. 7 . Safety preference In our recurring example of Fig. 1 , the user might instruct "I do not want to get hit by the tip of the screwdriver". Text2Interaction could respect this safety concern in two different ways:1. A restrictive controller is selected, e.g., α stop (•, ξ critical ), which guarantees that the robot comes to a full stop early before a collision could occur. Then, we do not need to account for any discomfort due to collisions in the preference function. For example, we could pick an action, where the tip of the screwdriver faces the human most of the time and only switches directions in the end.2. A less restrictive controller is selected, e.g., α contact (•, ξ intermediate ). We could then account for the instructed preference on the motion level by requiring all handover actions to have the tip of the screwdriver face away from the human during the entire trajectory.Carefulness Another example, where Text2Interaction has multiple ways of incorporating human preference into the plan of the robot follows the instruction i: "Put the vase on the desk carefully!".As an example, Text2Interaction could integrate the instructed preference on two different levels:1. On the motion level as a preference function: We could take the angle between the base of the vase and the desk surface as a metric. A small angle could then relate to a more careful, and therefore preferred, touchdown.2. On the control level: Text2Interaction could choose an admittance controller (α compliant ) instead of a PID-controller with a low stiffness and low overall low velocity, which would lead to softer touchdowns.Comparison to baseline objectives Our objective in (6) leads to an optimization of the product of Q-functions and preference functions. In our ablation study, we compare our approach to the common weighted sum of rewards formulation used in many baselines [13] [14] [15] [16] . In Fig. 8 , we give an intuition why our formulation outperforms the baselines.In this section, we derive the approximations of (5) under the assumptions of the Markov property, i.e., all information of s 1:t-1 , a 1:t-1 is contained in s t . Additionally, we assume that the user has a The user asks the robot to hand them the screwdriver so that they can comfortably grasp it. Previous methods would only output a task plan and optimize it for task success (baseline 1 [5] ), or treat preference as an additive reward, leading to unsuccessful executions (baseline 2 [13] [14] [15] [16] ). Text2Interaction takes task-, motion-and control-level preferences into account. In this example, our system outputs two preference functions that evaluate if the screwdriver is picked by the rod and if it is then handed over by the handle, encouraging plans toward their satisfaction. The skill sequence in the first row is the most likely to succeed, but not preferable as the handle is not easily graspable. The second sequence includes an action that is most likely going to fail, as the robot would try to grasp the very tip of the screwdriver.The last row depicts a skill sequence that is preferable and executable. From these three skill sequences, baseline 1 would select the first sequence, as it is the most likely to succeed, and baseline 2 would select the second sequence, as it maximizes the sum over Q-function and preference functions. Only Text2Interaction would correctly select the third skill sequence.motion preference in each individual skill of the sequence resulting in the events ∀t = 1, . . . , H : S t motion : the action a t satisfies the motion preferences at time step t starting in s t . Hereby, we assume that the motion preference S t motion does not depend on the future environment history (s t+1:H+1 , ψ t+1:H ). This assumption was reasonable in our experiments, but there might be more complex long-horizon dependencies for which this assumption fails. The overall motion preferences of the user are only fulfilled if their motion preferences at each time step are fulfilled, i.e., S motion = S 1:H motion = S 1 motion ∧ • • • ∧ S H motion . We give a more detailed explanation of the steps and approximations involved in (9) where E s2 [•] refers to the expected value over all possible states s 2 . Note, that in (9d), the expected value over all possible states s 2 is conditioned on the action a 1 , state s 1 and on the events that the execution of primitive ψ 1 is feasible and preferable. To derive (9c), we use the assumption that S 1 motion is independent of a 2:H . For (9e) we use the Markov assumption. To retrieve the estimate in (9f), we follow the proposed procedure in [5, Appendix C.1] , where we compute a single Monte-Carlo sample estimate of (9e) under the state transition T 1 (s 2 | s 1 , ψ 1 ). Here, the key insight is that we only execute skill sequences if the problem in (6) finds a valid action sequence that fulfills the user preferences and is feasible. Therefore, it is reasonable to assume that the condition events S 1 motion and S 1 feasibility in p s 2 | s 1 , a 1 , S 1 motion , κ occurred. The approximation E s2 p S 2:H motion | s 2 , a 2:H , S 1 motion , κ ≈ p S 2:H motion | s 2 , a 2:H , S 1 motion , κ was reasonable in our experiments as the learned transition distributions had a low variance. We then repeat steps (9b)-(9e) to get the approximation in (9f), which can be written as (9g). Finally, for (9h) we assume that the probability functions F t (s, a) returned by the LLM approximate the true underlying probabilities of user satisfaction by the given actions. In our ablation study, we saw that this assumption held if the new task was semantically in distribution [66] of the given in-context examples. For example, if we give Text2Interaction the screwdriver handover as an in-context example, it produces reasonable preference functions for the request "Hand me an ice cream", but it fails for the request "fold the piece of paper to a square".Note that we denote the k-th primitive in the library L ϕ as ϕ k and the primitive active at time t as ϕt.All events defined in this work are the "success" event of its Bernoulli trial.We are using the OpenAI gpt-4-0125-preview model with a context length of 128 000 tokens. The average query time was 29 ± 9 s.prob = linear(metric, t_1, t_2, direction=true) 


Decision: Summary of the Paper

The paper presents "Text2Interaction," a framework designed to seamlessly integrate human preferences into robot task planning, motion planning, and control. This approach employs large language models (LLMs) to generate task plans, motion preferences in Python code, and safe controller parameters. By maximizing the combined probability of task completion and user satisfaction, Text2Interaction aims to provide adaptable and responsive human-robot interaction. The framework is validated through both a real-world user study and an ablation study in simulation, which show promising results in user satisfaction and preference integration.

Strengths
- The method of using LLMs to integrate human preferences across task, motion, and control levels is novel and represents a major advance over existing methods that typically focus on a single aspect.
- The framework is evaluated through a combination of real-world user studies and simulation-based ablation studies, providing a strong empirical foundation. The results demonstrate statistically significant improvements in user satisfaction and preference integration.
- The ability of Text2Interaction to react to new user preferences in a zero-shot manner highlights its potential for adaptable and flexible human-robot interaction, which is a significant challenge in the field.
- The framework places a strong emphasis on aligning with human preferences, which is crucial for enhancing the efficacy and acceptance of robots in various applications.

Weaknesses
- The user study was conducted with a single task scenario, raising concerns about the generalizability of the approach across diverse setups and tasks. Additional experiments with a wider variety of tasks would strengthen the validation.
- While the paper is technically sound, the presentation, particularly in the methodology section, could be clearer. The paper employs complex notations and lacks intuitive explanations, which may hinder understanding.
- There is a lack of theoretical or empirical justification for certain claims, particularly regarding the distinctiveness of the sum versus product formulations in optimizing task completion and human preference scores. This needs further exploration.
- The preference functions are constrained by the primitives available, which may limit the framework's applicability to open-domain tasks or preferences not easily expressed by the predefined building blocks.
- The reliance on LLMs raises questions about robustness and generalizability to diverse and unexpected user instructions, which could impact the framework's applicability in dynamic real-world settings.

Summary of the rebuttal phase

While the authors' revisions appear to have improved the quality of the paper, the rebuttal did not change the reviewers' opinions. Although not all reviewers agree, accepting as a poster presentation is considered reasonable.

Reviews: review: title: title: A new method that integrates user preference in task and motion planning
paper_summary: paper_summary: Text2Interaction is a method that outputs user preference function, which is used as a multiplier for optimization objective to integrate user preference into task and motion planning.
main_review: main_review: # Summary

The authors propose *Text2Interaction*, a framework integrating human preferences in task, motion, and control planning of robots.

## Method
This method prompts LLMs to generate user preference function $F_t(s_t, a_t)$ as Python code with a list of building block functions, e.g. pose, metric, and probability functions. This function is multiplied by Q function and optimize the trajectory with shooting method from Text2Motion. If shooting fails, the method falls back to a greedy search that optimizes $S_{task}F_tQ_t$

## Experiments
With experiments, the authors tested two scenarios: user study and simulation ablation. 
### User study: Franka pick up and hand over task. 
The authors set up three stages:
1. User choose the reactive mode;
2. The robot performs based on $\arg\max\Pi Q_t$ w/ predefined instruction
3. The robot performs based on $\arg\max\Pi Q_tF_t$ w/ predefined instruction
4. 3 but w/ personal instruction 

The questionnaire results demonstrated 83% of users agree Text2Interaction integrates their preferences, 94% prefer it over the baseline.

### Simulation: Rearrangement
On 10 instructions, the authors compare their method also with groundtruth hand-crafted $QF^*$ and $Q + F$, and showed that their method is more preferable against $Q+F$ and a headroom towards using $F^*$. 

# Strength
1. The idea of generating optimization method for human preference is pretty novel and could work for a variety of natural language expressions. 
2. The experiments are convincing for the scenarios that the authors designed. But this could be hard to generalize as will be mention later. 


# Weakness
1. User study is only performed on one single scenario. It is better to consider diverse setups. 
2. Related to 1, the preference function is limited to the primitives in Fig. 5. I am not sure if this is generalizable for open domain tasks.
questions: questions: 1. I am confused about the concrete implementation of $S_{task}$ and why *sum of token log-probabilities of each primitive’s
language description* is a good optimization objective? 
2. I might have missed this, but why didn't you compare with $\Sigma Q + F$ in user study? And would $\Sigma Q+F$ be different from $\Pi Q+F$?
3. How could user preference describing motion be expressed? For example, *put the vase on the desk carefully*. This may be out of scope of Fig. 5
limitations: limitations: 2
review_summary: review_summary: This paper has its technical merit for the design that makes user preference easily integrates into planning. This is useful for HRI research community. The major concern is around the generalizability: how would this be applied to other settings including different scenarios, and user preference that cannot be expressed through the building block functions.
title: title: A well-researched paper with a novel approach but needs refinement in presentation of the proposed method and experiments.
paper_summary: paper_summary: The paper proposes a framework named Text2Interaction, which aims to integrate human preferences into robot task planning, motion planning, and control in a seamless manner. The approach leverages large language models (LLMs) to generate task plans, motion preferences as Python code, and parameters for a safe controller. By optimizing for both task completion and user satisfaction, Text2Interaction aims to provide a more adaptable and responsive human-robot interaction. The framework is validated through a user study on real robots and an ablation study in simulation, showing promising results in preference integration and user satisfaction.
main_review: main_review: The paper presents a novel approach to integrating human preferences into robotic systems using LLMs. The use of LLMs to generate not only task plans but also motion preferences as Python code, and controller parameters for safety is well-motivated and demonstrates potential for integrating human preferences into robotics.

Strengths:

1. The approach addresses both task-level and motion-level preferences, which is a novel aspect compared to existing methods that often focus on one level.
2. The experiments include real-world user study with statistically significant results that provide empirical evidence of the proposed method.
3. The paper addresses a key challenge in human-robot interaction—adapting to individual user preferences quickly and efficiently. The paper presents a promising approach, demonstrating its potential in integrating new user preferences in a zero-shot manner.

Weaknesses:

1. OR operation is wrong in section 3.1. I believe $OR(p_1, p_2)$ should be calculated as $p_1p_2 + p_1(1-p_2) + p_2(1-p_1) = p_1+p_2-p_1p_2$, not $max(p_1, p_2)$, where the two events are stochastically independent. How does this error affect the proposed algorithm and its performance, or any of the derivations?
2. There exist numerous recent work using LLMs to output parameters on robot control (oftentimes reward function or its parameters) to align with human preferences given as language instructions. Citing previous works such as [1, 2, 3] and comparing them with the proposed method would be essential.
3. Although the paper is well-structured, the paper's clarity can be significantly improved, especially in the method section. Simplifying the notations and providing more intuitive explanations would help.
4. While the user study and ablation study provide valuable insights, additional experiments with a larger and more diverse set of tasks would strengthen the validation of the approach.
5. Addressing the terminologies clearly would help readers understand the paper easily, even though the paper builds its formulation on top of Text2Motion [4]. For instance, what does $S_{task}(\phi)$ refer to in L189? Is $H$ in equation 1 time horizon? Describing what 'controller parameters' mean in section 1.2 or 1.3 or having examples would improve the readability of the paper since its meaning appears later in the experiments section for the first time.
6. As mentioned in the limitations, the proposed framework's reliance on LLMs raises questions about its robustness and generalizability to diverse and unexpected user instructions.
7. There are some typos - L38 (optimizes -> optimize) and L200 (investigated -> investigate).
questions: questions: 1. Can the authors provide more detailed examples of how the preference functions generated by the LLMs align with user instructions?
2. When does the shooting objective become zero? Could the authors report numbers on how often greedy-search happens and what those mean?
3. Could the authors elaborate on the computational efficiency of Text2Interaction, particularly in real-time applications?

[1] Xie, Tianbao, et al. "Text2Reward: Reward Shaping with Language Models for Reinforcement Learning." The Twelfth International Conference on Learning Representations.

[2] Ma, Yecheng Jason, et al. "Eureka: Human-level reward design via coding large language models." arXiv preprint arXiv:2310.12931 (2023).

[3] Hwang, Minyoung, et al. "Promptable behaviors: Personalizing multi-objective rewards from human preferences." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2Motion: From natural language instructions to feasible plans. Autonomous Robots, 2023

**Update after rebuttal:**
I have carefully read the authors' rebuttal and, in particular, the answers to the questions I raised. I am glad that the authors have experimented with the correct OR implementation and also fixed minor errors in the paper.
limitations: limitations: 3
review_summary: review_summary: This paper presents a novel framework for integrating human preferences into robot task planning, motion planning, and control using LLMs. While the approach is shows promising results, the paper needs clearer explanations on the formulation of the proposed method and the simulation experiments.
title: title: LLMs to improve alignment of robot behavior with human preferences. Requires better justification of claims.
paper_summary: paper_summary: The authors propose a method to improve the way in which human preferences are accounted for in robot planning, in addition to task completion. The proposed method jointly optimizes for human preferences at each of the three levels of abstraction: task planning, motion planning and control. The idea is that the task preferences account for finding the correct plan, the motion preferences for finding the desired way, and control preferences to enable a feeling of safety and comfort. The key to this method is using an LLM to translate a human instruction into a more detailed reward function, which can be used to optimize against. Through a user study, the authors show that Text2Interaction outperforms baselines in satisfying user preferences, without a decrease in overall task completion success.
main_review: main_review: The paper identifies the need for adaptation to preferences at different levels of abstraction of robot actions, i.e. task-level, motion-level and control-level. It clearly breaks down the formulation of the role of human preferences in combined task and motion planning and control, and how the joint optimization can be inserted into this mathematical formulation. The authors identify the ability of an LLM to elaborate on human preferences beyond what is explicitly expressed in the instruction, and leverage it to optimize robot behavior to follow human preferences. However, I have the following concerns:

- Regarding the breakdown of Task (correct plan) + Motion (desired way) + Control (safe and comfortable), and it’s adaptation in the proposed method:
     - The claim that motion preferences are equivalent to ‘desired way’, and control preferences are equivalent to ‘safety and comfort’ is not well-substantiated; in that preference and safety can play a role in both of those abstractions. For instance the desire to have the robot perform quick control actions is a preference and not necessarily a safety or comfort concern, and avoiding action sequences where the pointed end of a tool faces the user is a safety concern.
     - The breakdown in eq (5) includes the task completion component, as well as three preference components for task, motion and control. Later in Section 3.2, the task completion term  (represented using Q), and one of the three preference terms (the motion term, represented using F) are used to obtain the optimization criterion. It is unclear if the other two terms dropped, or somehow implicitly included in F. In general, it is unclear how control preferences are included as a part of this formulation, especially since F only depends on the trajectory and not the control input.

- Sum of rewards v.s. Task completion probability
     - The claim that by multiplying, as opposed to adding, the task completion score (Q) and human preference score (F) results in a completely different underlying mathematics is not properly substantiated. Especially so because the numbers Q and F are a result of learned networks and not probabilities by design. For instance, one could suppose that instead of probabilities, Q and F are log probabilities, in which case Q+F would result in maximizing the combined probability as well. 
     - In this particular formulation of F, the preferences are constrained to be expressed as a probability through the monotonic functions and logical functions defined in section 3.1, but there is no strong evidence to claim that this formulation would generalize beyond such a setting.

**Edit after rebuttal**: The authors' clarifications including the Q-function, details regarding the controller, the distinction between the three levels, and the sum v.s. product were helpful. However, my concerns on the minimal distinction between sum and product remain. In addition, the simplification to using constant probability terms for two of the three levels, when the main story is adapting at multiple levels of hierarchy seems like a major one. My overall recommendation remains the same.
questions: questions: In context of the above review:
- How are preferences for each of three levels of abstraction are/are not taken into account?
- How does the distinction between sum of rewards vs product of likelihoods stand for tasks beyond the evaluation setting?

Other clarification questions and suggestions:
- Since a large number of symbols are required to formulate the system, a block diagram annotated with the symbols would help serve as reference in parsing and tying together sections from notation to methodology.
- Does the Q function only represent geometric feasibility only, or does it also take into account the task goal? Section 3.2 claims that geometric feasibility can be approximated as the product of Q-functions over the trajectory, but if it is trained on the target task, does it also account for task completion probability in addition to geometric feasibility?
- The role of controllers in the overall system is unclear. The initial narrative suggests that the aim of the controller is to follow a trajectory, and can be parametrized by ‘eta’. But in the ‘Safe controller’ section, it is hard to connect what ‘eta’ actually constitutes. Are the control modes the ‘eta’? Relatedly, in the results of H1, it is unclear what ‘have stated a preference in the safety shield mode’ means.
limitations: limitations: 3
review_summary: review_summary: The paper proposes a novel method to better align robot behavior to human preferences by relying on LLMs to detail such preferences from human instructions. The paper requires more theoretical or empirical justification to support two claims: first, how the sum v.s. product formulations are fundamentally distinct, and second, how the preferences at all three levels of abstraction are accounted for.