Learning Human Contribution Preferences in Collaborative Human-Robot Tasks
Authors: Michelle D Zhao
Reid Simmons
Henny Admoni
Venue: CoRL 2023 Poster (2023)
Keywords: human-robot collaboration
reward learning
human-robot interaction

Abstract: In human-robot collaboration, both human and robotic agents must work together to achieve a set of shared objectives. However, each team member may have individual preferences, or constraints, for how they would like to contribute to the task. Effective teams align their actions to optimize task performance while satisfying each team member's constraints to the greatest extent possible. We propose a framework for representing human and robot contribution constraints in collaborative human-robot tasks. Additionally, we present an approach for learning a human partner's contribution constraint online during a collaborative interaction. We evaluate our approach using a variety of simulated human partners in a collaborative decluttering task. Our results demonstrate that our method improves team performance over baselines with some, but not all, simulated human partners. Furthermore, we conducted a pilot user study to gather preliminary insights into the effectiveness of our approach on task performance and collaborative fluency. Preliminary results suggest that pilot users performed fluently with our method, motivating further investigation into considering preferences that emerge from collaborative interactions.

Learning Human Contribution Preferences in Collaborative Human-Robot Tasks
Michelle Zhao
Reid Simmons
Robotics Institute
Robotics Institute
Carnegie Mellon University United States
Carnegie Mellon University United States rs rsimmons@andrew. cmu edu
mzhao2@andrew. cmu.cmu edu
Henny Admoni
Robotics Institute
Carnegie Mellon University United States
henny@cmu. edu
Abstract: In human-robot collaboration, both human and robotic agents must work together achieve set of shared objectives. However, each team mem- ber may have individual preferences, or constraints, for how they would like to contribute to the task. Effective teams align their actions to optimize task per- formance while satisfying each team member's constraints to the greatest extent possible. We propose a framework for representing human and robot contribu- tion constraints in collaborative human-robot tasks. Additionally, we present an approach for learning a human partner's contribution constraint online during a collaborative interaction. We evaluate our approach using a variety of simulated human partners in collaborative decluttering task. Our results demonstrate that our method improves team performance over baselines with some, but not all, simulated human partners. Furthermore, we conducted pilot user study to gather preliminary insights into the effectiveness of our approach on task performance and collaborative fluency. Preliminary results suggest that pilot users performed fluently with our method, motivating further investigation into considering prefer- ences that emerge from collaborative interactions.
Keywords: human-robot collaboration, reward learning, human-robot interaction
Introduction
As robots transition from machines in controlled environments to collaborative partners integrated into our daily lives, it is essential for them to align their behavior with our preferences and objectives. Human preferences can reflect constraints based on individual differences in capabilities [1, 2, 3], knowledge [4], intentions [5, 6]. or preferred strategies for accomplishing shared goals [7, 8]. In interactions where the human's constraints dictate the team objectives, robots must align themselves those constraints by learning and maximizing the objectives of the human [9, 10, 11, 12). A crucial challenge in robot learning is to develop algorithms that learn reward function representing human preferences in collaborative human-robot tasks [13].
Robots also have a variety of task capabilities, requirements, and activity deadlines, which impose constraints on their behaviors [14, 15, 16). These machine constraints require efficient allocation of appropriate tasks to appropriate agents, through adaptive scheduling [2, 3], market-based methods [3, 17, 18]. Robots that capable of self-assessment [19, 20, 21, 22] may even be able identify constraints for their actions. In worldworld interactions, humans collaborating with robots must also need to align themselves to the constraints of the robot [23, 24).
Consider the task of collaboratively unloading a dishwasher with a robot. In this scenario, let's say the human partner does not trust the robot to handle fragile items, like glassware and china. However, they are comfortable allowing the robot handle non-fragile objects, such as silverware. As such, the human's 's preferred contribution to the task is unloading glassware and china, while they would prefer the robot to unload non-fragile items. Similarly, the robot assesses that it can most
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

safely unload plates due to their upright position in the dishwasher. An effective team strategy that plays to each team member's ss strengths [25] is for the robot to unload plates and silverware, while the human unloads glassware and other china. In order to achieve this strategy, the robot must learn the human's preference through interaction, and determine how to accomplish the task while maximally adhering to the human's preferences and its own constraints.
In this work, we capture both human and robot contribution constraints via reward function based on the actions an agent performs towards some shared goal in given state. There have been a variety of reward-based frameworks, including Cooperative Inverse Reinforcement Learning (CIRL) [12] which align robots to human constraints by modeling the human's constraints as a reward function the robot needs to learn and assist in maximizing [9, 10, 11]. We combine the notion of human contribution constraints with machine constraints and use reward-based framework to represent form of interaction: Interactive Contribution Preference Learning (ICPL), in which a robot partner must learn the human's 's contribution preferences under its own constraints and the objectives of the task. The robot and human operate within the same environment to complete a collaborative task under both the human's 's and robot's individual contribution reward functions. While the robot's individual reward is known to the team, the human's 's preferences must be learned by the robot. The observed human behavior, which provides insights into their preferences, is influenced by the robot's actions, its constraints, and the task objectives.
Our framework captures novel type of collaboration in which successful teams maximize the task objectives while adhering to every partner's 's individual constraints. We propose candidate approach to the collaborative task: Bayesian Leamer (Bal.. Our approach models the human as maximizing reward under the assumption that the robot will take the optimal action, setting expectations such that the robot will eventually learn the optimal action. This human model informs online Bayesian inverse reinforcement learning [26]. The robot then plans using the expected reward under updated term to
beliefs. We also consider an extension of this method, where we add an information the planning objective: Bayesian Information Seeking Learner (BaISL). BaISL then plans using the expected reward under updated beliefs towards states with high information gain potential given the actions available to the human. We evaluate BaL and BalSL in experiments with simulated human models and in a pilot user study with human partners. Our results indicate BaL and BaISL often improves team performance with some, but not all simulated human partners. The pilot results suggest that BaL and BaISL improves perceived collaborative fluency of human partners.
2 Related Work
Human Robot Collaboration Human-robot collaboration the process in which humans and robots work together to achieve common goals [27, 13, 28]. Human-robot collaborations that are assistive in nature, like home service [29, 30] and healthcare robots [31, 32], align closely with the assistance paradigm proposed in [33], where the goals of the task entirely defined by the human. Conversely, collaborations without rigid principal-assistant roles can allow the human and robot to differ in their approaches for achieving shared goals [24, 7J. For example, robots designed for collaborative manipulation of physical objects [28] or collaborative games [34, 35, 36] seek to contribute to shared goals, which are not necessarily dictated by human preferences. ICPL considers learning the human's 's preferences in the context of a collaborative task, where the objectives of the task are partially defined by the human's preferences.
Value Alignment Value aligmment in robotics refers to the design of collaborative agents that can learn and infer a human partner's objective and act towards achieving human's 's objective. Ensuring that robots have the correct reward function is an important problem in designing safe and reliable autonomous systems [37, 38, 9]. Existing approaches for value alignment focus on robot learn- ing through interactions [12, 9]. active learning of human preferences [39, 40], and teaching through demonstrations [41, 42). Cooperative Inverse Reinforcement Learning (CIRL) [12] formulates value alignment as two-player Markov game, in which human and robot must maximize a common re- ward function, but only the human knows the reward function. While the robot and human partner in CIRL are focused entirely on the task of ensuring the learning agent asymptotically converges to the same values as the human teacher, the robot and human in ICPL seek to balance multiple objectives. The actions taken by the human in ICPL are also in service of completing the collaborative task and take into account the robot's s constraints.
2

---

Reward Learning A key part of value alignment and collaboration is understanding another agent's values, or utility functions, through observation of their actions [9, 43, 44, 45, 46, 6, 47, 48). Inverse reinforcement learning (IRL) [49] infers an expert's underlying reward function from its ob- served behavior, typically a single-agent setting [50, 51, 52]. Furthermore, when interacting with robots seeking to learn their reward functions. humans may convey their rewards through a number of different information modalities [53], including demonstrations [54], corrections [55], observa- tions [56], and comparisons [39, 57]. Our reward learning occurs implicitly and online [58], where both agents must collaborate within the same two-agent MDP, where the robot agent lacks partial knowledge of the shared reward function.
3 Interactive Contribution Preference Learning
We formulate the problem of learning human contribution preferences subject task objectives and individual constraints
two-player Markov game. At high level, Interactive Contri- bution Preference Learning (ICPL) game involves both the robot and human operating simultane- ously within the same environment to accomplish known task objective. Each agent's ' behavior additionally constrained by an individual reward function. The robot must learn a composite reward function that captures the task reward, human individual reward, and robot individ- ual reward for the team in this collaborative task. While the human has full knowledge of the composite reward, known to the robot are only the task reward and robot individual reward. Human individual reward is unknown to the robot and is to be learned through interaction. An ICPL game is a two-player Markov game between a robot and human, h, defined by the tuple M = (S, A") {0r, 0*, 0"),, Rgr ,rn ,rr Po. Po.
S the set of possible states: E S
A": action set available the robot r: a" E A"
A action set available to the human h: a² E A
T(h:.s)): transition function, which is conditional distribution on the next state, given pre- vious state and action for both agents: T(s'ls, a²,a") a²)
0":; the space of possible task reward parameters.
h the space of possible human individual reward parameters.
i": the space of possible robot individual reward parameters.
RøT ar.0h (......)) = C(Rgr (.),,))), Rg- (.,:;), ,), Ron (): the composite reward function that both players must maximize. The reward function is parameterized by C, combination function of the individual and task reward functions. In our experiments, C' is the summation of the 3 reward components. Rgr aor,,r.00 00. ga. S A A x 6" + R. Rgr is known to both h and r.
Rgr (+ : a parameterized reward function representing how both players contribute to the group's 's task reward. Ryr represents both players' task-related payoffs, which we refer to as task reward. RØr : S x A" x AA x S @ +
Rox :): parameterized reward function representing human individual reward, which represents the actions the human would prefer to perform in contribution to the task. Rgs
S Ah x S x 6 + R. Rga is known to only h.
Ro- (:r(s))))) a parameterized reward function representing how r's soft constraints enable it to contribute. Raa, robot individual reward, represents how the robot is rewarded for the actions that it robot is able to take. The robot's 's capabilities may dictate these soft constraints. Rg- : S x A" x S x er + R. Rg- is known to both h and r. Po(:--r.-): initial state distribution. Po(so, 0", or, 0h)..
7y: a discount factor E 1]
The game begins in an initial state (s,07.0% (0T sampled from Pg- Robot r observes 8", "T, and human h observes 8h. 8", 0", At each timestep t, robot r and human observe the current state 8, and select their actions a/', ab according to policies n" and hh The environment transitions to a","). The team receives, but does observe, composite reward RgT or ar (St. af, of $1+1) Rgr (st, af at Se+1) Rg- (st, af St+1) + Roi (s,, at st+1). The two agents observe each other's action selection. The process continues until the team reaches
the next state S1+1
3

---

Collaborative Task Environment
Figure I: Upon observing the action of the human, current state, and next state, the robot updates its beliefs (b) over the hypothesis space (0") and produces a predicted policy for the human (f*). The robot then replans its actions seeking information gain.
nt'
s,a", a", s'
Planning Towards States with High Information Gain Potential
b, n
Bayesian Inverse Reinforcement
Learning
BalsL
terminal state, 8p, where the final reward for the team Rs -Et RUT oo an (B di al, 8141) is received. The team only observes the total reward the end of the interaction.
Approach: Bayesian Learner (BaL) and Bayesian Information Seeking Learner (BaISL)
We propose an algorithm for the robot in ICPL, called Bayesian Learner (BaL), and an extension of the algorithm called Bayesian Information Seeking Learner (BaISL). BaL consists of two pri- mary online steps: (1) Bayesian Inverse Reinforcement Learning (BIRL) [26] using a human model with optimistic expectations of the robot, (2) planning given expected reward under beliefs. BaISL consists of the same two steps, but plans under beliefs towards informative states (Fig. 1). Imple- mentation details in Appendix.
Human Model for Bayesian Inverse Reinforcement Learning. The robot estimates a belief, b, over the hypothesis space @ of possible human individual rewards. In order to update its be- liefs, the robot relies on a model human decision-making in order to evaluate the likelihood of human actions. We model an optimistic reward human as noisily rational actor who se- lects actions based on the expectation that the robot will take the optimal complementary action to maximize the team's reward. By understanding these expectations, the robot is able to de- duce which human individual reward function would lead to such expectations. We also adopt the Boltzmann rational model of human behavior [59, 60]. which models that the probability of choosing a particular option increases exponentially as its utility, or reward, increases compared to other available options. Over t timesteps, the robot observes a trajectory of human actions, 7r = {(51, af 82), (82, a², s3), ..., (st, af 8t+1)) Since the human has full knowledge of the com- posite reward, we assume the human's 's policy is stationary expert policy and can make the following independence assumption for all candidate human reward functions 6hh O (Eq. 10).
b(θh) (())(T)) P((s1, at) 10") P((22, d)))@">. ( (0)) P((*)
(1)
Best-case reward, Tbe. the reward Ror ar.0a the team would receive, computed with candidate 0h. if the robot had taken the action that would maximize the composite reward (Bq. 2).
(St dt 10))) = max
ERap as (St, at (441) (s, ",, a.))
(2)
a'EAA-
Under the Boltzmann model, (he af (0t) serves as a potential function (Eq. 3), where 3 is a rationality coefficient. The BIRL update yields a predicted human policy over all beliefs is
a (a (s))s) > ((()))((ss)))))
gh
P((se, x exp(3 rbc(s ((1)))
(3)
Planning Under Expected Reward (BaL). BaL robot acts according to the resulting policy л², where Q(s, af, ah, b) represents expected discounted future composite team rewards given & (Eq 5).
Σ IuOh (an""(aa)ss (s) (Rls, ,',,"""""" dd,b))
n (s) + max
(4)
A'EA
AAEA 0h
Planning Towards States with High Information Gain Potential (BaISL). We consider an ex- tension BaL where leverage the key insight that the robot's 's actions in current state will affect the available actions of the human in the next state, which will in turn affect the human's 's ability to provide informative actions. Hence, the robot should seek to maximize team reward under its

---

Optimistic
Expected Human, Retional
Pedagogic Human, Rational
Rational
1.00
Pavental Optival Roward
0.95
0.00
0.85
DaEL
0.80
Bail.
MaxinntIL
Round Number
Round Number
Round Number
Figure 2: The human models are fully rational with 8 = oc. Though BalSL's information seeking incurs cost during the first round, BaISL. and Bal. perform optimally in later rounds with optimistic reward human and the expected reward human. When paired with the pedagogic human, Prag- Ped outperforms the other three methods, due to the human model being the one Prag-Ped expects. BaISL and BaL perform similarly with the pedagogic human. Error bars represent standard error.
beliefs while seeking to allow the human to provide informative demonstrations or actions, First, we perform a Bellman backup to compute a task-based Q given the current beliefs b (Eq. 5).
(88, a", a², b)) = Σ b(0%) Σ T(b'ss, a",,a",,, aaa)") Rag lih,err," (8, a", ,,",,') + +V(('')
(5)
gi.
S'ES
We want some measure of information gain I(b,,, s, a²) the robot would receive if it takes action a" in state with the current set of beliefs 6. (b)) represents the entropy of the current beliefs b over O". We can easily measure the entropy of the new beliefs, Hb) (s, a²)), given the human takes action a in state 8. We compute this value by updating the beliefs using the likelihood in Eq. 3, which yields updated belief b' and estimated human policy *². Then, the conditional entropy (b) a²)) is measured by computing the entropy over b', H (s,, a²)) (bb).. Since want to evaluate the information gain for a" instead of a we need some measure of H s, a"), where I(b, a²) = Hb)) (b]ss a²). However, the robot and human act simultaneously, so the human will not be impacted by the action of the robot until the next timestep, once both actions of the human and robot are revealed and the environment transitions to the next state. Thus, the robot evaluates information gain potential, as the info gain achieved by the human giving the best entropy- minimizing action in the next state.
Eyr(H((lsss')]- = Σ ET aa") ()) ",a",a")
(6)
E'E Hoold' a,
a. a²)
min H(bl(s',a)) a EAA
(7)
We compute expected entropy using Equation 6. Hhe represents the best-case entropy achieved from action in the next state (Eq- 7). It's important to note that the robot does not necessarily expect the human to always take the most entropy- merropp-mmimmizigg.. action in every state; the robot simply seeks states that would provide the human the opportunity give more informative decisions. Subsequently, the information gain of a" in state is added to the Q-value the highest potential info gain (Eq. 8), where T(b, 8, aa)"= = (bb) E** (H is a") a²)]. The term (Eq. 9) is a switch that turns off the information gain boost when the hypotheses with maximum weight converge to predicting the same human action. BaISL robot acts according T":
updating the beliefs given the human takes the most
boost to actions that would guide the team towards states with
n" (s) f- max a'Ea" A'EAA 8a n {al T (als; max # A (b|s; 0^) A b(0O) = max b(0")) gREeR
ΣΣ 10(0) (am]s; 0n)) (Q(s, a", d",b) + al 8, ",,"")
(8)
a a=
6h
Simulated Evaluation
Task Scenario. We evaluate our approach on a simulated collaborative task, with a series of simu- lated human partners. The human-robot team must clean up objects from a shared workspace. There exist K 3 object types, where objects differ by fragility and weight. The three types of objects are (1) heavy and fragile, (2) heavy and non-fragile, and (3) light and fragile. The action space avail- able to each player is {cleanup (heavy, fragile) object, cleanup (heavy, non-fragile) object, cleanup
5

---

Optimistic Human,
Expected Reward Human, Beta
Pedagogic Human, Beta=1
Beta=1
0.925
0.050
0.925
0.900
0.075
Purcent
0.850
Earst
Eal
Irag-Ped
MaxEntRRL
Round Numrzer
Bound Number
Found
Number
Figure 3: Boltzmann 3 = 1. With all 3 human models, BaISL incurs cost due to the information gain objective during the first two rounds but performs similarly marginally better than BaL in later rounds. Only when paired with the pedagogic human, Prag-Ped outperforms the other methods.
(light, fragile) object, No-Op). The heavy objects require both agents to jointly select the same action. Since the task objective is to declutter all objects, task reward is +2 for each object, with a -1 step cost. The human's 's individual reward, 0h, for each object type is randomly sampled from (111 10). The human's individual reward 8h is observed only by the human. However, the robot has does access to Oh; the hypothesis space for the human individual reward, which is the set of all permutations of 6 values. The robot's individual reward #" is a random permutation of the same set of four reward values in 0h, and is known both agents. In each instance of this game, the human and robot collaborate over N interactions. At the start of each interaction, there exists one object of each type to be decluttered. When all objects have been cleaned up, the interaction ends and the team receives the total composite reward achieved.
Performance Metric. We ran 100 random configurations of individual rewards in this collaborative task, We solve the full-information MDP with joint actions to obtain the optimal composite reward. Performance is measured by the percentage of optimal reward achieved in the final Ntt round.
Baselines. We compare our methods BaL (an ablation of BaISL) and BaISL to an alternative learning approach: Maximum Entropy Inverse Reinforcement Learning (MaxEnt [RL) which approximates the composite reward, bypassing the need for individual components, while maintain- ing the same value-based planning method. This baseline examines the efficacy of separating out the three reward components versus learning single joint reward function to approximate the compos- Value Alignment (eng-Ped)[1O],
ite. Lastly, we consider a second baseline: Pragmatic-Pedagee solution to CIRL in which the robot policy assumes that the human will act pedagogically accounting for the robot's 's beliefs. Technical details can be found in the Appendix.
Simulated Human Partner. We experiment with three types of simulated humans. For each agent type, we experiment with temperatures / > ou (rational), and the other with 3 = (noisy). (1) The optimistic reward human selects actions that maximize composite reward assuming the human takes the optimal complementary action. (2) The expected reward human selects actions that maximize expected composite reward under a uniform probability over all robot actions. (3) The pedagogic human uses the Prag-Ped solution in Fisac al. [10] to act pedagogically by rolling out the robot's belief update.
Results. Our experiments with the noiseless, rational simulated human models (3 = oo) demon- strate that the robot policies which pair with human models that match their prediction of human behavior outperform the misaligned human-robot teams. BaISL, BaL, and Prag-Ped require that the robot assume some model of human behavior in order to define the likelihood function for human actions. BaISL and BaL model the human as being of the type optimistic reward human. Prag-Pet models the human as being of the type pedagogic human. Naturally, when BaISL and Bal. pair with the optimistic reward human, BaISL and BaL have the correct model of the human, giving these ap- proaches an advantage over Pra.-Ped. Likewise, when Prag-Ped robot pairs with pedagogic human, Prag-Ped Ped has the advantage of using the correct human model to make belief updates. Because of this advantage with the Pedagogic human, BaISI. and Bal. do not perform as well as Prag-Ped.
To combat the advantage of an approach having the correct human model, we include human partner (Expected Reward human) which none of the robot approaches Importantly, with this third human type, we still that our approach performs well and outperforms the
their human model.

---

Human
Robot
Number Timestops Remaining: 4
Click the object you wish to pick up.
Your Rewards:
3
3
Aerker
3
Figure 4:
The user study interface requires the human to control one of two manipulators. The robot partner the manipulator on the right. Users select actions the object level using an interface.
baselines. BaISI, and Bal. outperform Prag-Peit with the rational, expected reward human, achieving optimal performance. As the matching would lead us to expect, the Prag-Ped robot outperforms BaISL and BaL with the rational, pedagogic human (Fig 2).
In our second set of experiments, 8 = 1 simulated humans make more noisy decisions, causing the robot's beliefs to become prone to erroneous updates, which may lead to worse performance. BaISL and Bal. both perform well with the optimistic reward human and expected reward human. The approaches perform similarly to each other but not as well Prag-Ped when paired with the pedagogic human. Generally, the BaISL robot performs more informntion-seeking actions, which sometimes No-Op actions, early during the first round. This tends to lead to lower perfor- mance in the first round than Bal., which immediately seeks to maximize expected reward (Fig 3). We find that our method with the information seeking term (BaISL) does not significantly improve performance over our method without the information seeking term (BaL). While not reflected in all randomly sample task configurations, when the rewards induce the human and robot to share the same preferred object, but the human's 's reward for picking up an item is higher than the robot's, we observe that our method beneficial (Fig 6 in Appendix).
6
User Study
We ran pilot user study, with five participants, to compare the collaboration of human partners with our BaL. and BaISI. agent against the two baselines. We investigated the impact of the independent variable: agent type, on dependent variables: task performance, and collaborative fluency. Task per- formance defined by the percentage of optimal reward achieved. We measured four collaborative fluency variables [61]; overall fluency, the improvement of fluency over time, affinity towards the robot a collaborative partner, and the perceived ability of the robot to learn the correct human indi- vidual reward, using four 5-point Likert scale questions whose values aggregated in order to treat as continuous variable [62]. As with the simulated evaluation, we evaluate BaL and BaISL against Prag-Ped and MaxEnt IRL. Each participant performed an ICPL task four times, once with each agent. The ICPL task was a decluttering task similar to the simulated task scenario, with K = object types, where objects differ by color (representing fragility) and weight (Fig 4). Fach rounds began with 1-2 objects of each type, where the starting configuration of objects was identical per round. After each task, participants given two scaffolded training tasks to familiarize themselves with the online game's controls.
asked series of survey questions. Participants were
While we did not perform statistical analyses due to the small sample size, we found general trends indicating that users performed closest to optimally when working with the BaISL and BaL robots. Teams with BaISL robot were able to more consistently maintain close to optimal rewards over the three rounds. On average, in the third and final round, the BaISL teams achieved 86% of the total reward, while BaL teams achieved 72%, Prag-Ped teams achieved 48%, and MaxEnt IRL teams achieved 62%. This indicates that the optimistic reward human model might more closely resemble human reasoming in this collaborative task than the pedagogic human model. The MaxEnt baseline suffers in the first round because it relies on successful demonstrations order to begin learning the reward. Its performance improves throughout the three rounds but never reaches the performance of BaISI. and Bal.. Participants also self-reported higher collaborative fluency with the BaISL and BaL models than the other two baselines. They also perceived the largest improvement in fluency over time with BaISL and BaL, with BaISL slightly higher than BaL. This may be related to our expectation that the BaISL model begins by taking information-seckinn actions, rather than
7

---

Performance
Subjective Collaborative Fluency
26.0
Prag
13.0
10.0
1:5
3.0
2.5
6.0
Game
Geme 2
Game
Cofabbrativve
Afliniy robit
Perceived Abillty
Improsvmment
Over Time
Figure 5: Participants achieved higher percent of optimal reward and higher subjective collaborative fluency with BaISL and BaL. over Prag-Ped and MaxEnt IRI., In the second and third rounds, teams with BaISL outperformed BaL in performance.
optimizing for reward, which may result in lower fluency in earlier rounds. Participants rated their affinity towards the robot as a collaborative partner the highest with the BaISL agent on average, and most strongly perceived that BaISL had learned their correct individual reward functions (Figure 5).
Discussion and Limitations
In this work, formulate type of collaborative interaction where the robot must align itself to the human's 's contribution preferences while adhering to task objectives and its own constraints as an ICPL game. ICPL games are limited to interactions that are based on reward learning, which does not span all types of human-robot collaborations. In ICPL,, the robot's constraints are assumed to take the form of a reward function. However, this is not the only form robot constraint possible; there include activity deadlines, probable errors, and task specifications. Furthermore, this formu- lation limits the human individual reward as only captures the notion of the human's preferences for their own contribution, since only the human action is taken parameter. However, humans may have preferences expectations for how they want the robot to contribute. Capturing this con- tribution preference requires modifying the human reward function to include other agents' actions as input. Despite its limitations, ICPL poses a flexible, interaction paradigm in which robots can represent and learn human contribution preferences in collaborative human-robot tasks.
In our simulated evaluations, find that, naturally, teams consisting of a rational human partner matching the robot's hypothesized human model yields the highest team rewards. A drawback in testing with simulated human models is that they rely predefined decision models. Future work involves rumning larger user studies with human participants, whose decision- variable and not captured by our predefined models. Another limitation is that our information gain objective does not construct long-horizon multi-state plans towards states with high information potential. Rather, BaISI. evaluates whether the next state can be informative and seeks out next states with high information gain potential. BaISL myopic in its information gain computation, in which the algorithm can only reason about the next state, without more long-term plan. While this is one limitation of our approach, we believe this also affords the algorithm simplicity and interpretability of its learning process to human partners, demonstrated by the perceived robot's ability to learn in our preliminary user study (Fig. 5). Future work includes active learning methods [53] which consider longer-horizon plans for learning about human partners in collaborative tasks. Lastly, our model evaluated on a simple task domain for both simulated and real user experiments. While the task includes a coordination complexity in which particular actions must be performed jointly in order for the team to receive reward, it still remains rather simple in the state and action space. Though include second task domain (sce Appendix), in future work, we plan to scale up task complexity to include more agent interdependencies and preconditons for different actions.
may be more
8 Conclusion
We introduce collaborative paradigm, ICPL, where successful teams optimize task objectives while adhering to individual agent constraints. The ICPL framework aims to equip robots with the ability to represent, learn, and accommodate human contribution preferences in collaborative human-robot tasks. We also contribute a candidate approach, BaL, and extension BaISL, which models humans as maximizing rewards with the assumption that the robot will choose the optimal action, facilitating the robot's 's eventual learning of optimal actions. We evaluated our approach through experiments with simulated human models and preliminary user study involving human partners.

---

Acknowledgements:
This research is supported by the DARPA award FP00002636 and NSF IIS-
21 2112333.
References
[1] M. Dunnette. Critical concepts in the assessment of human capabilities. Human Performance and Productivity: Volumes 2. and 2014.
[2] S. Zhang, Y. Chen, J. Zhang, and Y. Jia. Real-time adaptive assembly scheduling in human- multi-robot collaboration according to human capability. In 2020 JEEE International Confer- ence on Robotics and Automation (ICRA), pages 3860-3866. IEEE, 2020.
[3] N. lijima, A. Sugiyama, M. Hayano, and T. Sugawara. Adaptive task allocation based on social utility and individual preference in distributed environments. Procedia computer science, 112: 91-98, 2017.
[4] S. J. Motowildo, W. C. Borman, and M. J. Schmit. A theory of individual differences in task and contextual performance. Human performance, 10(2)71--8, 1997.
[5] G. Nicholls, P. Cheung, Lauer, and M. Patashnick. Individual differences in academic motivation: Perceived ability, goals, beliefs, and values. Learning and individual differences, 1(1):63-84, 1989.
[6] T. Bandyopadhyay, S. Kok, E. Won, D. Frazzoli, W. Hsu, D. Lee, and D. Rus. Intention-aware motion planning. Springer Tracts Advanced Robotics, Algorithmic Foundations of Robotics 2...
X, 86, 02 2013.
[7] F. Gobet, H. Richman, J. Staszewski, and H. A. Simon. Goals, representations, and strategies in concept attainment task: The epam model. 1997.
[8] S. Nikolaidis, Y. X. Zhu, D. Hsu, and S. S. Srinivasa. Human-robot mutual adaptation in shared autonomy. 2017 12th ACM/IEEE International Conference on Human Hunan-Robot Interaction (HRI, pages 294-302, 2017.
[9] D. S. Brown, J. Schneider, A. Dragan, and S. Niekum. Value alignment verification. In Inter- national Conference on Machine Learning, pages 1105-1115. PMLR, 2021.
[10] J. F. Fisac, M. A. Gates, J. B. Hamrick, C. Liu, D. Hadeield-Menell, M. Palaniappan, D. Malik, S. S. Sastry, T. L. Griffiths, and A. D. Dragan. Pragmatic-pedagogic value alignment. In Robotics Research, pages 49-57. Springer, 2020.
[11] T. Munzer, M. Toussaint, and M. Lopes. Preference learning the execution of collabora- tive human-robot tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 879-883 IEEE, 2017.
[12] D. Hadfield-Menell, S. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
[13] A. Ajoudani, A. M. Zanchettin, Ivaldi, A. Albu-Schäffer, K. Kosuge, and O. Khatib. Progress and prospects of the human collaboration. Autonomous Robots, 42(5):957- 975, 2018.
[14] T. T. Nguyen, M. Roos, and J. Rothe. A survey of approximability and inapproximability re- sults for social welfare optimization multiagent resource allocation. Annals of Mathematics and Artificial Intelligence, 68(1-3):65-90. 2013.
[15] P. Krysta, O. Telelis, and C. Ventre. Mechanisms for multi-unit combinatorial auctions with few distinct goods. Journal of Artificial Intelligence Research, 53:721-744, 2015.

---

[16] V. Singhal and D. Dahiya. Distributed task allocation in dynamic multi-agent system. In In- ternational Conference on Computing, Communication & Automation, pages 643-648. IEEE, 2013.
[17] A. Asadpour and A. Saberi. An approximation algorithm for max fair allocation of indivis- ible goods. Proceedings of the thirty-ninth annual ACM symposium on Theory of computing. pages 114-121, 2007.
[18] H. Aziz, Biró, Lang. J. Lesca, and J. Monnot. Optimal reallocation under additive and ordinal preferences. 2016.
[19] A. Norton, H. Admoni, Crandall, T. Fitzgerald. Gautam, M. Goodrich, A. Saretsky, M. Scheutz, R. Simmons, A. Steinfeld, al. Metries for robot proficiency self-assessment and communication of proficiency in human-robot teams. ACM Transactions on Hluman-Robot Interaction (THRI), 1(3):1-38, 2022.
[20] A. Jauffret, C. Grand, N. Cuperlier, P. Gaussier, and P. Tarroux. How can robot evaluate its own behavior? a neural model for self-assessment. In The 2013 International Joint Conference on Neural Networks (IJCNN,, pages 1-8. IEEE, 2013.
[21] K. Odette, J. Rivera, E. K. Phillips, and F. Jentsch. Robot self-assessment and expression: A learning framework. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 61, pages 1188-1192. SAGE Publications Sage CA: Los Angeles, CA, 2017.
[22] Dai, K. Sycara, and M. Lewis. game theoretic queueing approach to self-assessment in human-robot interaction systems. In 2011 IEEE International Conference on Roboties and Automation, pages 58-63. IEEE, 2011.
[23] L. Yuan, X. Gao, Z. Zheng, M. Edmonds, Y. N. Wu, F. Rossano, H. Lu, Y. Zhu, and S.-C. Zhu. In situ bidirectional human-robot value alignment. Science robotics, 7(68):eabm4183, 2022.
[24] T. Fong and R. Nourbakhsh. Peer-to-peer human-robot interaction for space exploration. In AAAI Technical Report (5), pages 87-90, 2004.
[25] K. d. Bosch, T. Schoonderwoerd, R. Blankendaal, and M. Neerinex. Six challenges for human-ai co-learning. In International Conference Human- Caaman-Computer Interaction, pages 572-589, Springer, 2019.
[26] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586-2591, 2007.
[27] A. Bauer, D. Wollherr, and M. Buss. Human-robot collaboration: a survey. International Journal of Humanoid Robotics, 5(01):47-66, 2008
[28] A. Levratti, A. De Vuono, C. Fantuzzi, and C. Secchi. Tirebot: A novel tire workshop assistant robot. 2016 IEEE international conference on advanced intelligent mechatronics (AIM), pages 733-738. IEEE, 2016.
[29] K. Kosuge, Sato, and Kazamura. Mobile robot helper. In Proceedings 2000 ICRA. Mil- lennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 583-588. IEFE, 2000.
[30] H. S. Koppula, A. Jain, and A. Saxena. Anticipatory planning for human-robot teams. In Experimental robotics, pages 453-470. Springer, 2016.
[31] M. Montemerlo, J. Pineau, N. Roy, S. Thrun, and V. Verma. Experiences with mobile robotic guide for the elderly. AAAI/IAII, 2002:587-592, 2002.
10

---

[32] A. Mertens, U. Reiser, B. Brenken, M. Lüdtke, M. Hägele, A. Verl, C. Brandl, and C. Schlick. Assistive robots eldercare and daily living: Automation individual services for senior citizens. In International Conference on Intelligent Robotics and Applications, pages 542- 552. Springer, 2011.
[33] B. A. Newman, R. M. Aronson, K. Kitani, and H. Admoni. Helping people through space and time: Assistance as perspective on human-robot interaction. Frontiers in Robotics and AI, 8, 2022. ISSN 2244-9144. doi: 10. 00t:::00...99/rrrb00011000000000
URL https: //www frontiersin.
org/articles/10. 33w.:///s...33.////// frobt. 2021 720319.
[34] M. Zhao, Simmons, and H. Admoni. Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9116-9123. IEEE, 2022.
[35] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34:14502-14515, 2021.
[36] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[37] D. Amodei, C. Olah, Steinhardt, Christiano, J. Schulman, and D. Mané. Concrete prob- lems in ai safety. arXiv preprint arXiv:1606.06565 2016.
[38] S. Russell, D. Dewey, and M. Tegmark. Research priorities for robust and beneficial artificial intelligence. Ai Magazine, 36(4):105-111 2015.
[39] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-bused learning of reward functions. 2017. doi: 15111...107//S5.201.... W0I.P/SS....1.III....0
[40] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[41] M. S. Lee, H. Admoni, and R. Simmons. Machine teaching for human inverse reinforcement learning. Frontiers in Robotics and AI, 8:188, 2021.
[42] Zhu. Machine teaching for bayesian learners in the exponential family. Advances Neural Information Processing Systems, 26, 2013.
[43] J. Perner. Understanding the representational mind. The MIT Press, 1991.
[44] A. Tabrez, M. B. Luebbers, and B. Hayes. A survey of mental modeling techniques in human- robot teaming. Current Robotics Reports, 1(4):259 267, 2020.
[45] H. Wimmer and J. Perner. Belief' about beliefs: Representation and constraining function of wrong beliefs in young children's 's understanding of deception. Cognition, 13(1):103-12) 128, 1983.
[46] Cahon,Cohen, A. M. Leslie, and U. Frith. Does the autistic child have "theory of mind"? Cognition, 21(1):37-466 1985.
[47] S. M. Carlson, M. A. Koenig, and M. B. Harms. Theory of mind. Wiley Interdisciplinary Reviews: Cognitive Science, 4(4):391-40) 402, 2013.
[48] C. Baker, R. Saxe, and J. Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.

---

[49] A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In in Proc. 17th International Conf. on Machine Learning, pages 663-670. Morgan Kaufmann, 2000.
[50] B. D. Ziebart, A. L.. Maas, A. Bagnell, A. K. Dey, al. Maximum entropy inverse reinforce- ment learning, In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[51] C. Baker, J. B. Tenenbaum, and R. Saxe. Bayesian models of human action understanding. Advances in neural information processing systems, 18:99, 2006.
[52] B. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. Bagnell, M. Hebert, A. Dey, and S. Srinivasa. Planning- based prediction for pedestrians. pages 3931-3936, 12 2009, doi: 10. 10.1109/ROS.oo 2009,5354147.
[53] E. Biyik, A. Talati, and D. Sadigh. Aprel: A library for active preference-based reward learning algorithms. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robe Interaction, pages 113-617,22 2022.
[54] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, (1112-44:1119-14444 2021.
[55] M. Li, A. Canberk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical corrections. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2877-2883, IEEE, 2021.
[56] B. Stadie, Abbeel, and I. Sutskever. Third- thiro-person imitation learning. International COn- ference on Learning Representations (ICLR), 2017.
[57] D. J. Hejna III and D. Sadigh. Few-shot preference learning for human-in-the-loop rl. In 6th Annual Conference on Robot Learning, 2022.
[58] T. K. Büning, A.M. George, and C. Dimitrakakis. Interactive inverse reinforcement learning for cooperative games. In International Conference on Machine Learning, pages 2393-2413 PMLR, 2022.
[59] H. J. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: unifying formal- ism for reward learning. Advances in Neural Information Processing Systems, 33:4415-4426, 2020.
[60] C. Laidlaw and Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204. 10759, 2022.
[61] G. Hoffman. Evaluating fluency in human-robot collaboration. IEEE Transactions on Human- Muchine Systems, 49(3):209-218, 2019.
[62] J. Carifio and R. J. Perla. Ten common misunderstandings, misconceptions, persistent myths and urban legends about likert scales and likert response formats and their antidotes. Journal of social sciences, 3(3): 2007.
[63] P. Shafto, N. D. Goodman, and T. L. Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55-89, 2014.
[64] H. Li, Ni, S. Agrawal, F. Jia, S. Raja, Gui, D. Hughes, M. Lewis, and Sycara. Indi- vidualized mutual adaptation in human-agent teams. IEEE Transactions on Hluaan-Machine Systems, 51(6))706 -14066774,,,0 2021.
12

---

Appendix
9.1 Intuitively Understanding Individual Rewards as Individual Costs
Task Reward represents the task objective. Human Individual Reward represents the human partner's collaborative preferences. And Robot Individual Reward represents the robot's soft con- straints. The individual rewards are based on the actions each agent individually takes. We can interpret these as negative costs, where an action with lower rewards is straining on the agent per- forming it. In the dishwasher loading example, picking up an object, like a bowl or glass cup, are determined to be especially costly for the robot because of the difficulty in picking up objects with smooth surfaces with a two finger gripper, and the risk of the dropping and shattering the object. These high cost actions involved in picking up those objects would give low Robot Individual Reward.
9.2 Bayesian Inverse Reinforcement Learning: Implementation Details
We model the human as a noisily rational actor who selects actions based on the assumption that the robot will take the optimal complementary action to maximize the team's reward. The human implicitly sets expectations for the robot by assuming the robot will take the optimal action. For example, the predicted robot action is optimal under Rgz "er__,r,., which the human has full knowl- edge of. By understanding these expectations, the robot is able to deduce which human individual reward function Rgs would lead to such expectations. We also adopt the Boltzmann rational model of human behavior [59, 60], which models that human decisions are exponentially likely with spect to reward. According to this model, the probability of choosing particular option increases exponentially as its utility, or reward, increases compared to other available options.
Over t timesteps, the robot observes the human perform some trajectory of actions, T {(81, at, 82). (#2, ab; 83), .... (st, ab 81+1)) Since the human has full knowledge of the composite reward, we assume the human's 's policy is stationary expert policy. Thus, we can make the following independence assumption for all candidate human reward functions 6th O (Equation 10).
P(OM)T)) x P((s1, P((s),a))))) 10") (B())))))(((())))))))))))))))) P.. (lllss,,))))0 )
(10)
We intuit that since the human has full knowledge of all three reward functions: task, human individ- ual, and robot individual reward, and in absence of a model the robot's behavior, the human will try to achieve the task objective while acting pedagogically by taking actions that seek to demon- strate complementary action the human expected the robot have performed. This best-case reward, Tbe the reward the team would receive from the composite function Ryr, or år. if the robot had taken the action a" A" that would maximize the composite reward (Equation 2). The composite function under candidate @ uses 6h to parameterize the human individual reward term.
(be at a0'00) max Σ Rør or en (st, of 88+1 8441) (8++1) af; of) a"EA"
P((se, (((s,,a))) 0²)) x exp(3 (oe d(1))))
Consider state in which the human has two actions to choose from a, and b. The robot's evaluates its best-case reward for two hypotheses: (I) best-case reward to a, 3 to 6, and (2) The 2 to a and 3 to b. The human chooses action b; however, we do not want to necessarily update based on a dis- tribution derived from the reward values themselves. Since our robot models the human a reward maximizer, the human views no effective differences between the two strategies. Thus, we want ensure the probabilities reflect equal probability for selecting b by thresholding the Boltzmann potential if the action yields a best-case reward equal to the maximum.
roc df100)
= max (bc (0)))) a EA
r(st, af (0")
otherwise
13

---

Algorithm (BIRL) Bayesian Inverse Reinforcement Learning: Online Update
Input: State 8t, Human Action at. Belief prior (0t) Vgh O
Rationality threshold X, Hypothesis space O", Temperature B Output: Updated belief's Predicted human policy nh
Parameter:
1: for 6h E A do
2: (st, (l.a0/10))
= max E areAt 6x-1
RoT or gn (8t, af, 8 ( at, of)
X,
if roc (0) max ( a 18h)
3: r(si, al ab100)=
a"EAA"
1-
otherwise
4: Z, Σ (rl(enaf10)
Dean
5: P((s), )(0)00))) = to2 e
6:
6(0h) + P(g st,, al)))
"10=
P(oniat)
7: end for
8: (a 0h)) e
9: (a Σ 6(0")) (a")ss; gr
Algorithm below describes the Bayesian inverse reinforcement learning approach. The predicted human policy is the expected policy under the updated beliefs of the robot. We use À 0.9 in our agents for all simulated and human study experiments.
9.2.1 Optimistic Information Gain: Implementation Details
In updating its beliefs, the robot assumes that the human will optimistically choose the action that would achieve maximum reward given that the robot takes the ideal action which would facilitate achieving the maximum reward. The robot updates its beliefs over the possible values of the human's individual reward, 6 using likelihood function P((st, ( 16*h) built on this assumption. [10] examines a solution the CIRL problem in which the human teacher expected act pedaog- ically, while the robot learner, aware and expecting this pedagogy, acts practically under its learned beliefs. Research on human pedagogical reasoning demonstrates that when teaching, humans en- gage in actions aimed at influencing or altering the beliefs of learners [63]. Our assumption that humans will take actions expecting the optimal, complementary robot action interprets the expected human pedagogy as being through setting expectations of the robot.
While the robot uses the composite reward Ryg _00rōh .h to learn from the human's 's actions, the robot seek out states that will give the human the opportunity to demonstrate its true composite reward Rgr, gr. Our key idea is that the robot's s actions affect the state, and in turn affect the human's 's next actions, the robot can leverage this to actively take actions that will lead the team to states in which the human can provide more informative demonstrations. For example, consider the dishwasher unloading task, where the robot and human must collaboratively unload 3 bowls and cup (see Figure 6). If the robot reaches for the cup, it leaves the human with no choice but unload one of the three remaining bowls. Had the robot reached first for one of the bowls, the human would have the option of choosing between the cup one of the three bowls. Opting for this more informative state would have given the robot more information about the human's 's preference between bowls and cups, since the human would have made decision between the two objects. We will next formalize this desire to maneuver the team into informative state using an information gain metric.
Algorithm represents the planning algorithm which seeks out next states with high potential infor- mation gain. In line 5, perform Bellman backup to compute a task-based Q given the current beliefs (Equation 5). Q(s, a", a², b) represents expected discounted future composite team rewards given b. The robot's 's information gain
T(b, a²) = H(b) E (b)),, a")]
14

---

Rewards
0
er
Lh
4
4 10
(A) Without
Information Gain
R
R 5
(B) With
Information Gain
Figure 6: This toy example demonstrates example of the effect of the information gain measure on the behavior of the robot. The robot and human must collaborate to load a dishwasher with bowls and cup: the team gets +1 1 reward for loading any object. The robot's 's individual reward is +7 for the cup, and 44 for the bowl. The human's individual reward is +10 for the cup, and +4 for the bowl. The robot's hypothesis space is two options: (1) +10 for the cup and +4 for the bowl, or (2) +4 for the cup and + 10 for the bowl. Without information gain (A), the robot reaches for the cup, but it leaves the human with no choice but unload one of the three remaining bowls, achieving reward of 8 + 9 + 5 = 22. With information gain (B), the robot opts for a more informative next state, where the robot able to learn the human's preference of cups over bowls, achieving higher reward of 16 + 9 25.
with expected entropy F (H a")] under the predicted human policy ah and the transition dynamics T, using Equation 6. In line 11 of Algorithm 2, the information gain of a" in state is added to the Q-value as boost to actions that would guide the team towards most informative states (Equation 8). If the next state is not informative given any human action, then there will be no information gain boost given to any action. The a term (Equation 9) is a switch that turns on and off the information gain boost. The information gain objective seeks to reduce uncertainty over the human individual reward functions in O However, multiple reward functions can lead to the same predicted human action. When the beliefs begin converging on the same predicted human action, the robot will deprioritize seeking information gain. a measures the intersection of the shared predicted human actions by the top probability human individual reward functions in the beliefs b.
a =
n {a n (als; 0*) max # (b|s; 0')) b(0*)) gth
max b(0*)) H
(11)
@R
The robot thus replans under updated beliefs, while seeking informative states in which the human has opportunity to provide informative decision. If the next state is not informative given any human action, then there will be no information gain boost.
Upon observing the action of the human, current state, and next state, the robot updates its beliefs (b) over the hypothesis space (@*) and produces a predicted policy for the human (**).. The robot then replans its actions seeking information gain. This online decision process comprises our full algorithm: Bayesian Information Seeking Learner (BaISI.) See Algorithm BaISI. is an approach a robot may take the ICPL task. While the task not complete, in each timestep, the robot updates its beliefs based the previous human action and state using Algorithm 1. Then, the robot replans secking potential information gain using Algorithm 2 (line 8). The robot samples action from its computed policy (line 9), while the human chooses an action as well. The environment transitions to the next state. Once the task is complete, the taem receives the total composite reward obtained over the interaction (line 14).
For the Bayesian Learner (BaL), we leave out the information seeking term. See Algorithm 4. While the task is not complete, in each timestep, the robot updates its beliefs based on the previous human action and state using Algorithm 1. Then, the robot replans using expected reward under the current
15

---

Algorithm (PSIG) Plan Seeking Information Gain
Input: Beliefs over human reward functions b, Predicted human policy &^ Parameter: 0",
Output: x²"
1: V(s) < Initialize (s) randomly
2: while not converged do
3: for E S do
4: for (a', E A do
5:
(s a b) Σ
b(0))
Σ (( ,aa) Ran or +9V(2)) s'ES
@h
6: end for
7: V(s)
(a",a (a",a") AAA"XA" a(s,
8: end for
9: end while
10: a Naa car fa x² (a)s;60") = max th (b)s; 6h) A b(0%) = max 6(8*)
11: T' (s) f arg max Σ Σ Eb(0") (s'1ss; 0²)) (Q(s, a² ,) allb,
a'EA" ahEAk @
Algorithm (BaISL) Bayesian Information Seeking Learner: An approach ICPL
Input: so. 8", 0", 6h Output: Updated beliefs b, Predictive model of human action f
1: b 4 Initialize uniform prior over 0% E O
2: for task not over do
3: Human observes 8T, ,0,""""
4: Robot observes 8", _8, and O², but not 8h
5:
8 +- 80
6:
while game n not over do
7:
f t BIRL a",
{update beliefs about human utility)
8:
n' < Pb
{plan seeking info gain using updated beliefs
9:
a" (-(s))
{ (ample robot action a" from policy
10:
a + Human decides a based on
[human takes a
11:
8 + s' Tss' ((88'18,0) 25,aa,a a²)
(environment transitions to state
12:
end while
13:
end for
Return RF Σ Rgr or (st, of, dl sc-1) {team observes final reward once task complete)
beliefs using Algorithm 5 (line 8). The robot samples an action from its computed policy (line 9),
while the human chooses an action as well.
9.3
Simulated Evaluation
9.3.1 Simulated Human Models
We experiment with three types of simulated humans. For each agent type, we experiment with temperatures 3 (rational), and the other with 1.
1. The optimistic reward human selects actions that maximize composite reward assuming the human takes the optimal complementary action. This model is the one used in our BIRL likelihood function, making it easier for Ours and Ours IG to perform well with. The simulated human model selects action a with probability proportional to the reward, with
16

---

Algorithm (BaL) Bayesian Learer: An approach to ICPL
Input: Su. 8", 8",0"" 0",0" gh Output: Updated beliefs 6, Predictive model of human action f
Initialize uniform prior over 6h Oh
2: for task not over do
18"""" 4: Robot observes 8", 8"," and @², but not 8h
3: Human observes 8T, "",0",00
5:
$ St So
6:
while game not over do
7:
f + BIRL((s (s, a², b)
{update beliefs about human utility) (plan seeking info gain using updated beliefs
8:
n¹ < (bAN((b,)
9:
a" ~ ('(sss
{ (sample robot action a' from policy {human takes a (environment transitions state s')
a² Human decides a based on
3+ s' T "s,a""a")) a²)
12:
end while
end for
14:
Return R = Σ Roy or on (st ai, al st-1) {team observes final reward once task complete)
Algorithm 5 (PLAN) Plan Under Expected Rewards Given Current Belief's
Input:
Beliefs over human reward functions b, Predicted human policy #* Parameter: 0", 8T
Output: n²
1: V(s) < Initialize (s) randomly
2: while not converged do
3: for ESo do
4: for (a',a)) A AA do
5: (s, a b)
Σ
b(0n)
Σ
R Ra Rr +VV()
s'ES
6: end for
7: V(s)
max
a(s
(a",a* E
8: end for
9:
end while
Σ Σ (a (1ts)
10: T² (s) &- arg max
a"EA" ahEAk on
best-case prediction for a".
Th (aM)s) & exp(8 max a' E.E.A"
Bor or (9r, s41))))))) at aạ)
9441
2. The optimistic reward human selects actions that maximize expected composite reward under uniform probability over all robot actions. This human decision making function does reflect Prag-Ped nor Ours, making this human model out of distribution for both. The human model computes its reward by marginalizing out a".
<a(ass))) x exp B
(BED Σ
Rgg ar ar. ar. (8t, of 8 ) aF))
a'EA" 8841
3. The pedagogic human uses the sponding to the true human individual reward. Since the human computes the expected Q- value for its own actions by marginalizing over robot actions. This model is the one used in the Prag-Ped robot's i likelihood function, making easier for Prag-Ped work well with. The Pragmatic-Pedaaogie Value Alignment solution to CIRL leverages the assumption that the hu- observe the robot's 's action at the current timestep before selecting its own action. The
Value Alignment [10] Q-value corre-
17

---

human policy maximizes the best expected outcome for each available action:
T" (a"als, (s,",,,b,u a", 0") exp(3Q(ss, b, a", aa;
In order to compute Q, the human considers the belief update of the robot, where the update of the robot's belief is determine given by the Bayesian update:
b' 8000's, a", a²)) x" (a"|s, a", (0,0)000)
The robot's policy under the new beliefs maximizes the expected Q under the new beliefs.
(5' B) arg max
Σ Q(s,b,aa, )(000)))
at.00
The Bellman equation for the human is follows:
Q(s, a., a (t)) Ror or (s, (s, a", a²") + E ho'(s' ( B', a" ,=
The human is pedagogic because the Bellman equation takes into account how the robot's beliefs will change based the actions of the human. Since the human cannot actually the robot's action ahead of time during this collaborative decluttering task, the human marginalizes out a":
(a (s, a. 8%)) x exp( Σ
3Q(s, b, ("a,,,,"",a';0")
a' """
and acts according this policy.
9.3.2 Approaches
1. BaL: Bayesian Learner The robot policy x" is defined by Equation 8, but without the information gain term. The robot models optimistic reward human and plans using expected reward under current beliefs. The robot selects the action a' maximizing the expected-reward-bee Q Q-values only. The robot selects the action a", according to Algo- rithm 4, maximizing the expected Q-values.
t (s) + max a"EAA" EEEAA
ΣE6 b(
6h
2. BaISL:
Bayesian Information Seeking Learner We consider extension to BaL. where the robot plans towards states that have high information gain potential. The robot acts according to x" (Eq. 8), modeling the human as an optimistic reward teacher and planning towards informative states. The robot selects the action a", according to Algorithm 3, maximizing the expected-reward-te reward- Q-values and information gain boost.
x <'(s) + max a' AA"
Σ ahEAA ah
Σ16 b(0)) (a"ss; 00)) (Q(s, aa" + al(b, s,a")
3. Prag-Ped: Pragmatic-Pedagen
Value Alignment [10] As baseline, compare the performance of our agent against a solution to the CIRL [12] problem in which the human acts pedagogically while the robot reasons practically. The robot policy assumes that the human will act pedagogically with Q value function that accounts for the robot's beliefs. The Pragmatic- Pragmatic-Pedagogic value alignment solution further assumes the human observes the robot's action before selecting their own action. The human policy maximizes the best expected outcome for each available action:
<² <"(a"|s b, a", 0hh) erp(3Q(s, a², a"; (²:0%)
Consider state in which the human has only one action. incorrect hypothesize reward achieved will be higher for since under Thus, want to ensure the probabilities reflect
18

---

Performance vs. Update Threshold Lambda
1.00
0.90
BalSL
Bal
Prag-Ped
0.75
0.9
1.0
update Threshold Lambda
Performance evaluation of our models across different values our hyperparameter used in BIRL (line 3, Algorithm 1) shows that the BaISI and Bal methods are unaffected by À when partnering with the = optimistic reward human.
Figure 7:
equal probability for selecting by thresholding the Boltzmann potential if the action yields a best-case reward equal to the maximum.
if (5 (f,, max (be a²
r(s,, ab (0"))
a EAA
otherwise
In order to compute Q. the human considers the belief update of the robot, where the update of the robot's belief is determine given by the Bayesian update:
b'(@"|s, b, a", a²) x (a" [s, a, (b(0))
The robot's policy under the new beliefs maximizes the expected Q under the new beliefs.
(8' B) arg max
Σ Q(,, a², a²; 00))((0)))
aht6"
The Bellman equation for the human is follows:
Q b, a², a"; 00 Rgr er Bh (8, a", a²) Eggaa (2 B' a" ,,((6,0):0))
While the Prag-Ped solution computes both a policy for the human and robot, we take and execute the policy of the robot for this robot baseline. The robot takes the action maximizing expected reward under the heliefs, using (8', The pedagogic human simulated model acts according to the human policy part of the Prag-Ped solution.
4. MaxEnt Our second baseline Maximum Entropy Invese Reinforcement Learning [50], followed by replanning using the learned reward. The robot learns reward function [s, a", ",,""" a²) representing Rgr ar (s, a" a²) using the demonstrated states and joint ac- tions previously seen. The robot evaluates Q-values based on the learned reward function f:
Q(s,a",a" ff: = f(s, a", a s') +y Σ T(s'ss, a²) max ass',a", aa"""""""""" ad
sESS
π (8) arg max
Σ Q(s,a"",a"1) a"
9.3.3 Hyperparameter Analysis
We analyze the performance of our models across the hyperparameter used in BIRL (line 3, Algo- rithm 1). We test from 0.6 to 1.0 in 0.1 intervals, using the 3 1 optimistic reward human with robots BaISI, Bal. Prag-Ped, using 50 random game configurations. We find that performance
19

---

Performance vs. Human Rationality Coefficient
1.00
0.90
0.85
0.90
BalSL
Bal
Prag-Ped
0.75
Simulated Human Rationality Confficiett
Figure 8: We find that performances of BaISI, Bal., and Prag-Ped are not affected by true 3 value for the simulated human.
Optimistic Reward Human,
Expected Reward Human, Rational
Pedagogic Human,
Rational
Rational
100
95
Composte
90
85
Bal:
Prag- Ped
MaxEntIRL
Boand Number
Bound Number
Round Namber
Figure 9: Evaluation with Rational Human Models in Team Space Fortress task. We evaluate the agents across rounds of interaction with the same 3 human models used in the decluttering task. Across all 3 models, the BaISL agent reliably outperforms the BaL and MaxEnIRL models. BalSL and Prag-Ped perform similarly on the optimistic reward human and expected reward human. Prag- Ped outperforms all other methods when paired with the Pedagogic human model. The error bars represent standard error.
of our algorithm BaISL is not affected by the choice of value. The Prag-Ped robot mostly unaffected as well, but performance drops slightly with X 1.0. We use 0.9 in our agents for all other simulated experiments. Additionally, analyze the performance of our models across the 3 hyperparameter used in the simulated human model for the optimistic reward human with robots BaISL,, BaI., Prag-Ped, using 50 random game configurations. The robots update their beliefs us- ing a beta of 1, while the TRUE 3 values for the simulated human are plotted on the x-axis in Figure 8. We find that performances of BaISL, BaL, and Prag-Ped are not affected by true value for the simulated human.
10 Additional Domain: Team space fortress (TSF)
Team space fortress (TSF) is an abstracted variant of the task in Li et al. [64]. The human and robot team must defeat an enemy spaceship. To do so, one agent must act bait, while the other shoots the enemy. Unknown robot, the human prefers to serve as either the bait the shooter. The robot's constraints represent a proficiency to either shoot or bait as well. The action space is h = A'" {act as bait, position to bait, wait, position to shoot, shoot). This task introduces preconditions, which are dependent on prior actions and the agent who performed such actions. For instance, in order to shoot the enemy, an agent must have positioned themselves to shoot. Agents must position to bail before they act as bait. An agent cannot act as bait if the other agent has already positioned themselves to bait, and similarly for the shooter. Compared the initial table decluttering task, this task introduces several complexities: history-dependent state preconditions for actions, larger action space, and a larger state space.
20

---

Optimistic Beta=1
Expected
Pedagogic Human.
Betam1
104
Composte 80
100
BaiSk
Boll
96
MaxEnURL
92
Round Number
Round Number
Bound Number
Figure 10: Evaluation with 3 Human Models in Team Space Fortress task. We evaluate the agents across rounds of interaction with the same 3 human models used in the decluttering task. After the interactions, for the Optimistic Reward and Expected Reward humans, the BaISL agent outperforms the BaL model. Prag-Ped outperforms the other models when paired with the Pedagogic human model and generally outperforms the others on the optimistic reward human and expected reward human in the first two rounds.
Weit
Position to Shoot
Poaltion
BaISL
Intiaal Conditions
Reward: 1.0
Shoot
Balt
0.0 0.2 0.4 0.6 0.8
1.0
Shoot
Halt
Shoot
R Ratiof
Balt
Shoot
R Riciled
Balt
Shoot
A Palket
Beit
0.0
0.2 0.4 0.8 0.8
1.0
Posilion Bait
Ast as Beit
BaL, Prag-Ped Ped
Postlion- Beit
Posttion Shoot
Shoot
player
The other player must shoot.
Reward:
0.6 X
Shoot
IR Bellef
Balt
Shoot
R Belief
Beit
Shoot
Belief
Beit
Figure 11: Illustration of BaISL in Team Space Fortress task. Initially, the human has strong preference for acting the bait, while the robot has only slight preference for acting bait. The BaISL robot (top row) selects to wait on the first turn, and observes the human position themselves as bait. The robot is able to update its belief of the human's preference, denoted R helief. On the other hand, the BaL and Prag-Ped robot (bottom row) goes straight for acting as bait, which prohibits the human from doing as well. The human has choice but to act as the shooter, which does not enable the robot to learn the human's 's reward. Thus, the BaISL robot able to achieve optimal reward.
To evaluate, we randomly sample the reward values for the individual constraints of each agent. In the TSF environment, we demonstrate that our method outperforms the baseline methods, over 20 random configurations (see Figure & 10). In particular, BaISL consistently outperforms BaL and MaxEntILL across all human models. BaISL performs similarly to Prag-Ped on the Optimistic Re- ward and Expected Reward humans, and Prag- consistently outperforms the others when paired with the pedagogic human. In Figure 11, we illustrate particular instance of the TSF task in which the BaISL robot seeks to learn before diving into the task, demonstrating the impact of the information gain term.
21

---

10.1 Elaboration on Viewing Human and Robot Constraints as Separate Reward Functions
Since the human knows the robot constraint, we in theory could learn 'human constraint+robot constraint' as one reward function for the human, which the robot will seek to learn. Viewing the two as separate allows for each constraint to represent conceptually distinct preferences. For example, the robot's constraint Ra- (s, 0y, 0,, 8') could be temporal constraint, where includes time. The human's 's constraint may not involve time, may be featurized represented differently.
Additionally, viewing the constraints as separate allows for the robot to evaluate the human's 's prefer- ences and transfer the knowledge to other robots with potentially different constraints. For instance, when interacting with robots with different capabilities, the underlying human preferences don't change, but the human's policy, which includes robot constraints, may result in different actions.
Separating the two also allows for this framework to be more generalized when we take knowledge into account. The representations of individual constraints in the collaborative context can be easily used to allow for situations where the human does not know the robot's constraints.
22


Reviews:
[{"review_id": "tHM8ttkGtp", "reviewer": "Reviewer_Fwxy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper models human-robot collaboration as a two-agent game where the agents share rewards but the full reward is only known to the human. They introduce a Bayesian approach to infer the human-known component of the reward by explicitly including information gain in the robot's policy objective. They evaluate this method in simulation with both simulated and real humans and demonstrate improved team performance. ", "summary_of_recommendation": "The paper tackles an important problem and introduces an interesting method to model human-robot interaction. The evaluation domain and the baselines are appropriate. I would like the authors to make parts of the paper clearer.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please refer to review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n- Interesting, novel approach to model human-robot collaboration\n- Experiments with real humans\n- Baselines include ablations and prior work\n\n**Weaknesses**\n- Method not described clearly and the intuitive understanding of the advantage over IRL is not well-defined.\n- Experiments do not include real robot.\n\n**Detailed Comments**\n- Related work: can discuss more work in information gathering and contrast with approaches used in the past.\n- Section 4 is not easy to understand from 170. Apart from improving the flow of information, authors can address the following. \n  - A number of symbols are mentioned but not defined. E.g., $V$, $f$.    \n  - Summations sometimes have set mentioned but not always. E.g. $s' \\in S$ is there but $\\hat{\\theta}^h \\in$?.\n  - Equation 5. What is the expectation over? Not sure what $f, T$ are referring to here.\n  - Paragraph 192 is very difficult follow. I'm also not sure how the human is expected to maximize the robot's information gain when it also optimistically assumes the robot to know $\\theta^h$.\n  - line 211. No-op not defined.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "tFV_E2n-zj", "reviewer": "Reviewer_TiNh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a new form of human-robot interaction, Interactive Contribution Preference Learning (ICPL), that brings human constraints, robot constraints, and their common objectives into consideration. The paper also proposes an approach, Bayesian Information Seeking Learner (BaISL), to solve this new form of task. The method uses a human model that assume robot will choose optimal actions. The robot updates its belief and predict human policy based on observation of human action, and then plan its action aiming for information gain. The authors conduct experiments with both simulated human and real human with promising results. ", "summary_of_recommendation": "The paper proposes to define and solve human-robot collaboration in which human and robot contribution constraints are considered. The paper is clearly written, well-motivated, and technical sound. The literature review is thorough. The experiment evaluation is simple, but it could be a starting point.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In paper’s example of unloading dishwasher, the authors mention human constraints is: prefer the robot to unload non-fragile item, and robot constraints is: it can safely unload the plates. However, it seems the robot constraint can also be viewed as another form of human constraint: the human does not trust robot to unload fragile item but knows the robot can comfortably handle plate’s upright position. What would be the difference of viewing them separately as ‘human constraint +robot constraint’ versus a more strict ‘human constraint’ since human knows robot constraint? What would be some examples that these two could be clearly divided? What would be the reason of separating them if there are no clear differences? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n•\tThe paper is well motivated in such that it considers a new interaction setting in which both human contribution constraints and robot contribution constraints are considered. \n\n•\tThe experimental evaluation includes both simulated human and real human users, and samples 100 different reward configuration for simulation experiments. The authors conduct user study. \n\n•\tThe paper is clearly written in detail and the problem is clearly defined.\n\nWeaknesses:\n\n•\tThe experiment is conducted in only one task setting for both simulated human and real human. In addition, the task is cleaning up three objects, which is not complex as the paper admitted. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2oXqM1Sr-F", "reviewer": "Reviewer_92c8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Optimizing for preferences in human contribution in a human-robot collaborative task requires inferring the human’s preferences and goals to optimize the robot’s contribution in the task.  This paper addresses this by contributing: (1) a Markov game representation for human and robot collaboration, (2) a Bayesian inference procedure for active learning the human’s preferences at runtime, and (3) evaluation of the procedure on simulated human partners and a pilot with human users in a manipulation task.", "summary_of_recommendation": "Overall, I believe this paper offers sufficient novelty in the Bayesian information-seeking approach and clear improvements over baselines in shared task setting.  As stated above, I would like to see more extensive evaluations in more than one task setting, if possible one with higher complexity, as it is difficult to assess the approach.  I therefore currently rate as weak accept. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In the Pedagogic Human evaluation, could the authors please comment why the BaISL / BaL approaches do not outperform (or very marginally outperforms) the Prag-Ped approach?\n\nCould the authors comment on the scalability of their approach, and how they may apply the Bayesian approach in the continuous setting?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The formulation used in the paper is, for the most part, sensible.  From a problem formulation point-of-view, it seems sensible to assume there are situations in which the human has complete observability of the task, the robot’s constraints, and their own preferences, while the robot has complete observability of all but the human’s preferences.  The Bayesian inference approach seems sound in the context of the problem setup.  This is common to many papers dealing with information gain, and use of information to improve the legibility of one of the player’s contributions as observed by another player.\n\nIn the evaluation, use of the three simulated human actors are well-motivated.  The baselines are also well-selected, and the results show improvement over these baselines.  However, it would be interesting to see more evaluations with other realistic types of humans, including those that do not follow Boltzmann rationality, and those that are adversarial, choosing actions that degrade the task.\n\nThe primary concern with work is that the authors only evaluate in simplistic task scenarios with only high-level actions considered (block clean up task with three object types).  The action space is extremely simplistic and the transition function is trivial.  It is difficult to assess the scalability of the approach, and understand precisely how it may handle continuous tasks that are of higher complexity.  The authors should aim to evaluate on another task with larger action space, richer set of rewards and continuous reasoning.  It is also puzzling why only three rounds of interaction were evaluated, and not more.  It would be informative to see the asymptotic behavior of repeated interactions.\n\nThe writing in the various sections of the paper could also be improved.  The problem setup in section 3 is quite vague as written, and could benefit from a running example to ground the nomenclature and factorization of the reward structure.  Additionally, it is unclear, without more detail, the motivation behind the information gain switch term in eq. 8 and what motivates the choice.  Is this empirically motivated, or are there theoretical properties? \n\nMinor comments:\nThe authors should fix all  typos, e.g. “that is robot is able to take” in the definition of the robot individual reward, etc.  The authors should also fix the broken links to equation references in the appendix.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0nE05um6deD", "reviewer": "Reviewer_cJgK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper under review proposes a Bayesian framework for learning collaborative robot policy. The framework consists of two stages: Bayesian Inverse Reinforcement Learning (IRL) and planning based on the obtained beliefs. The authors evaluate their approach through simulations of collaborative tasks and a user study.", "summary_of_recommendation": "Overall, the paper exhibits clear presentation and logical flow. However, to strengthen the technical contribution, a more substantial and distinctive advantage over existing methods should be demonstrated, potentially through more intricate evaluation scenarios that more closely resemble real-world collaborative tasks. I'm leaning toward rejecting the paper at the current moment but would happy to change my score if the authors provide good explanations regarding the technical novelty and more comprehensive experimental results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Clarification of the technical novelty of the work\n2. More complicated evaluation tasks in addition to the simple MDP used in the current manuscript.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\nThe paper is well-written and easy to understand. The use of Bayesian IRL to derive beliefs about human rewards and incorporating these beliefs into the planning process is intuitive and logical.\nThe problem formulation is clearly presented.\n\nWeaknesses:\nOne potential concern lies in the technical contribution of the paper. While both Bayesian IRL and planning under beliefs are widely used techniques, the authors introduce an entropy term into the planning objective, representing the entropy deduction based on the assumption of humans taking optimal actions in the next time step. However, the impact of this entropy term on the robot's collaborative policy performance appears to be marginal, as observed in the comparison of BaL and BaISL in Figure 2, 3, and 5. Clarification from the authors is needed to understand the technical novelty and significance of this contribution.\n\nFurthermore, the simplicity of the Markov Decision Process (MDP) used in the evaluation is a potential limitation. With a small action space and a short horizon (T=3), more complex evaluation scenarios would be beneficial to demonstrate the effectiveness and robustness of the proposed method.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for robots to infer human preferences via interaction as a reward function. Three reviewers voted to accept the paper and found the literature review to be thorough, the technical contribution to be solid, but raised issues around experimental evaluation. cJgK voted to reject the paper based on weak experimental results. In their rebuttal, authors included an additional domain which other reviewers appreciated, but cJgK didn't respond despite several messages. I believe the authors have sufficiently addressed the concerns raised by the reviewer. As such, I recommend the paper be accepted. 

Some thoughts for the final revision: 
-  Please include a discussion about continuous action space / include experiments showing the applicability of method in these domains. - - Clarify the technical contributions as done in the response to cJgK and also suggested by Fwxy.
- Discussion about comaprison with Prag-Ped in response to 92c8 is insightful and should be included in the paper. 
- A pedagogical example in the paper is a scenario is where in dishwasher unloading task, the human picks up fragile items whereas the robot picks up silverware/plates. In such scenarios, it seems plausible that the human can express the preferences to the robot in natural language or in formal language. Why should the intent be inferred based on interactions? A discussion on when it is appropriate to directly communicate preferences, v/s inference based on interactions would be very meaningful and help to better contextualize the work.
Decision: Accept (Poster)