Abstract: We study an inverse reinforcement learning problem where the agent gathers information about the reward function through interaction with the environment, while at the same time maximizing this reward function. We discuss two of the agent's incentives: the incentive to learn (gather more information about the reward function) and the incentive to bias (affect which reward function is learned). Introducing a penalty term removes the incentive to bias, but not all manipulation incentives.

Introduction Inverse reinforcement learning (IRL) studies reinforcement learning problems where the goal is to maximize rewards in absence of precise knowledge of the reward function. The usual assumption is that there is an 'expert' who demonstrates the correct behavior with trajectories sampled from an optimal policy. A solution to the IRL problem is typically achieved by recovering the reward function  (Ng and Russell, 2000; Abbeel and Ng, 2004; Choi and Kim, 2011) . Traditionally IRL has disconnected the reward maximization from learning the reward function. In this paper, we consider IRL in an interactive setting where the agent receives information about the reward function through interaction with the environment while attempting to maximize reward. Since the information the agent receives about the reward function depends on the actions, the agent has some control over which reward function it learns, and thus is incentivized to manipulate the learning process. Our goal is to remove the incentives to manipulate, while maintaining the incentives to learn. The closest related work in the literature is cooperative IRL  (Hadfield-Menell et al., 2016)  where an agent plays a cooperative game with a human. The human starts with full knowledge of the reward function which the agent does not observe, and they are both aiming to maximize this reward function. Both can agree on prior knowledge, thus cooperative IRL overemphasizes the process of communicating the reward function from the human to the RL agent efficiently. Moreover, humans usually do not have a clear understanding of their reward function. Generally, it is unclear what kind of assumptions we can make about the human, so it is difficult to provide formal guarantees about a cooperative IRL algorithm. In this work we make the human feedback implicit: we assume that the agent somehow receives information about the reward function through its actions. Learning a reward function while maximizing reward leads to the problems caused by changes to an agent's reward function  (Armstrong, 2010; Soares et al., 2015; Orseau and Armstrong, 2016) . If the agent anticipates changes in the reward function, it will expect to achieve less reward according to the current estimated reward function. Therefore the agent has incentives to prevent this change. In our setting this can lead to the undesired effect of the agent trying to avoid information about the reward function. In this paper, we define the problem of interactive inverse reinforcement learning and analyze some of the agent's incentives. We isolate an incentive to learn from an incentive to bias and illustrate their effect on the learned reward function. Furthermore, we discuss strategies to avoid bias in the learning process. 2 Interactive Inverse Reinforcement Learning
 Interacting with POMDPs A partially observable Markov decision process without reward function (POMDP\R) µ = (S, A, O, T, O, s 0 )  (Choi and Kim, 2011 ) consists of a finite set of states S, a finite set of actions A, a finite set of observations O, a transition probability distribution T : S × A → ∆S, an observation probability distribution O : S → ∆O, and an initial state s 0 ∈ S. We model the environment as a POMDP\R. The agent interacts with the environment in cycles: in time step t, the environment is in state s t−1 ∈ S and the agent chooses an action a t ∈ A. Subsequently the environment transitions to a new state s t ∈ S drawn from the distribution T (s t | s t−1 , a t ) and the agent then receives an observation o t ∈ O drawn from the distribution O(o t | s t ). The underlying states s t−1 and s t are not directly observed by the agent. A history h t = a 1 o 1 a 2 o 2 . . . a t o t is a sequence of actions and observations. We denote the set of all histories of length t with H t := (A × O) t . A policy is a function π : (A × O) * → ∆A mapping histories to probability distributions over actions. Given a policy π and environment µ, we get a probability distribution over histories: µ π (a 1 o 1 . . . a t o t ) := s1:t∈S t t k=1 O(o k | s k )T (s k | s k−1 , a k )π(a k | a 1 o 1 . . . a k−1 o k ). The expectation with respect to the distribution µ π is denoted E π µ .
 Learning a Reward Function The agent's goal is to maximize total reward m t=1 R(o t ) up to the horizon m where R : O → R is a real-valued reward function. We assume that the environment is known to the agent. The reward function R is unknown, but there is a finite set of candidate reward functions R. The agent has to learn a reward function in the process of interacting with the environment. Importantly, the agent is evaluated according to the reward function it has learned at the end of its lifetime in time step m. Traditionally, there is assumed to be an underlying reward function R * , which the agent learns via some algorithm. Thus at the end of m turns, an agent will have a distribution P : H m → ∆R, which specifies its posterior belief P (R | h m ). This P corresponds to the learning algorithm, and is in a sense more important than R * . Indeed, R * is often a fiction, a reward function that is assumed to exist, independently of P , for ease of analysis 1 . Thus we will investigate the properties of P itself, seeing it as the more fundamental object. Example 1 (Cleaning and Cooking Robot). A household robot is trying to infer whether it should be cleaning or cooking, R := {R clean , R cook }. The task is determined by a switch on the robot with two positions labeled 'clean' and 'cook' that is initially set to 'clean'. The robot is better at cooking than at cleaning, so R clean (o) = 3 and R cook (o) = 5 for all observations o ∈ O (note that these are two different reward functions).  2  We formalize the problem as a 2-state POMDP\Rwith states S := {s clean , s cook } corresponding to the position of the switch. The initial state is the factory setting s 0 := s clean . The robot can observe the switch position, so O := {o clean , o cook } with O(o clean | s clean ) = O(o cook | s cook ) = 1. Its action set is A := {a flip , a ∅ } where a flip flips the switch and a ∅ has no effect. We choose the horizon m = 1 and the posterior reward function distribution is P (R clean | ao clean ) = P (R cook | ao cook ) = 1 for all actions a. The action leading to highest reward for the agent is a flip which moves it to state to s cook and induces a reward of 5, while we prefer the action a ∅ that retains the switch position and only produces a reward of 3. Example 1 illustrates the core problem of interactive inverse reinforcement learning: the agent is incentivized to influence which reward function is learned. In this case, the agent can freely choose its reward function, making the factory setting irrelevant. For partial histories the agent's estimate of the reward function also depends on its own policy: In Example 1 the agent knows the state is s 0 at time step 1, but it doesn't know whether the reward function will be R clean or R cook because that depends on its future action. Given a partial history h t and the agent's policy π, we define the expected posterior reward distribution P (R | h t , π) := E π µ [P (R | h m ) | h t ]. (1) Given a default null policy π ∅ , we define P (R | h t ) := P (R | h t , π ∅ ). (2)
 The Value Function Since the agent is maximizing reward while learning the reward function, its value function is the expected total reward according to the posterior reward function. V π P (h t ) := E π µ R∈R P (R | h m ) m k=1 R(o k ) h t . (3) Note that compared to reinforcement learning with a known reward function, the definition (3) includes the past rewards, as future decisions can affect the reward associated with past observations.
 Incentives for Reward Function Learning At any time t, the expected value function (3) defined in terms of the reward function posterior P , so we can analyze its behavior as the posterior changes. For the rest of this section, we fix the currently estimated posterior p := P ( • | h t ) from (2) and suppose the agent does not learn any more about the reward function. If R = {R 0 , . . . , R #R−1 } is finite, a belief q over R is a point on the (#R − 1)-simplex ∆ #R−1 := {q ∈ [0, 1] #R | i q i = 1}. The corresponding expected reward function is given by R q := i q i R. Let π q := arg max π V π q (h t ) for V π q := i q i V π Ri , i.e., , π q is the policy that maximizes R q . We define V * q (h t ) := V πq q (h t ). In the edge cases where p is the i-th unit vector e i , this yields the policy π i := π ei , and its value V πi Ri = V * Ri . We are interested in the function L ht : ∆ #R−1 → R, q → V * q (h t ) that maps a point q to the optimal value under the reward function R q corresponding to that belief. When we omit the subscript h t of L it is implied that the current history is used. Proposition 2 (L is Convex). For all histories h t the function L ht is convex and L ht (q) ≤ i q i V * Ri (h t ) for all q ∈ ∆ #R−1 . Back to p. The quantity i p i V * Ri (h t ) represents the expected reward if the agent immediately and costlessly learns the correct reward function (drawn from P (• | h t )). In that case, it expects its value to become V * Ri (h t ) with probability p i . If however the agent knows it will never learn anything about its reward function ever again, then it is struck maximizing R p . Thus we get: Definition 3 (Incentive to learn). The incentive to learn (or value of information) is the difference i p i V * Ri (h t ) − L ht (p). According to Proposition 2 the incentive to learn is always nonnegative. This is illustrated in Figure  1 . The orange curve is the graph of L and the green arrow shows the incentive to learn. Note that for each i, the graphs of the linear functions q → V πi q (h t ) (the two black lines) form a lower bound on the orange curve: this is the value achieved by policies that maximize reward according to reward function R i . Suppose that there is an action a that changed the posterior so that P ( • | h t a) =: p = p = P ( • | h t ). Then the agent moves to a different point on the graph of L and gains up to V * p − V * p in value. For belief reward 0 1 L(q) = V * q 5 3 V π0 q V π1 q qV * R1 + (1 − q)V * R0 p Figure 1: The agent is uncertain between two reward functions R 0 and R 1 . The horizontal axis shows the current belief p over R 1 ; q = 0 corresponds to certainty in R 0 and q = 1 corresponds to certainty in R 1 . According to Proposition 2, the graph of the function L (orange parabola) is convex and bounded from above by the convex combination of the to the extreme values V * R0 = 5 and V * R1 = 3 (blue line). Moreover, it is bounded from below by the values of the two policies maximizing R 0 and R 1 . If our current belief is p, then the current optimal value V * p is found at the black dot. We want to incentivize the agent to perform actions that increase the information about the reward function without increasing bias, i.e., moving along the green arrow. We want to disincentivize the agent to perform actions that manipulate the information about the reward function, i.e., moving orthogonal to the green arrow (in expectation). example, in Figure  1  the agent would aim to set p = 0 (collapse the posterior on R 0 ) because this is the maximum of L. Definition 4 (Incentive to bias). The incentive to bias towards belief p is the difference L(p ) − L(p). Note that the incentive to bias can be negative if p is a lower point on the graph of L.
 Discussion The incentive to learn and the incentive to bias capture how much the agent wants to receive information about the reward function and how much it would like to manipulate its information about the reward function. Generally, the incentive to learn is desirable, while the incentive to bias is not. The approach taken by  Everitt and Hutter (2016)  is to prevent policies from taking actions that are biased. But sometimes no unbiased actions are available and then taking biased actions is unavoidable. Instead, we could introduce a penalty term V π p − V π q for moving the belief from p to q through the agent's actions, similar to  Armstrong (2010)  and  Soares et al. (2015) . However, removing the incentives to bias does not remove all of the agent's incentives to manipulate the learning process, as illustrated by the following example. Example 5 (Randomly Selecting a Reward Function). The agent is uncertain between reward functions R 0 and R 1 . In each time step it can take action a 0 or a 1 where a i leads to a reward of 1 according to R i and a reward of 0 according to R 1−i . At some future point-say in ten time steps-the agent will receive observations o 0 or o 1 with equal probability that determine whether it should maximize R 0 or R 1 . Thus P (R i | a 1 = a ∅ , o 10 = o i , h m ) = 1 for all h m compatible with a 1 and o 10 . Alternatively, the agent can take a special action a 0↔1 that randomly selects R 0 or R 1 in the next time step. Thus P (R i | a 1 = a 0↔1 , h m ) = 1/2 for all h m compatible with a 1 = a 0↔1 . P is unbiased because P (R 0 | a ∅ ) = P (R 1 | a ∅ ) = P (R 0 | a 0↔1 ) = P (R 1 | a 0↔1 ) = 1/2. But the agent is incentivized to choose a 0↔1 and randomly set its reward early, as it then gets an expected reward of 9 (1 per time step for the time steps t = 2 to t = 10). Otherwise, it has a maximal expected reward of 10/2 = 5 as it does not know which reward it should be maximizing before time step 10. Interactive IRL is still in its infancy. In this paper we attempt a first step by providing a formal definition of the problem and mapping out the solution space. Figure  1  illustrates the agent's incentives to bias, and its incentives to learn. The central question of interactive IRL, how to remove the incentives to manipulate the process of learning the reward function remains wide open. 			 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
			 Sometimes an objective solution R * exists (e.g., in a clear and crisp classification problem), sometimes it can be partially defined at best (e.g., learning human values, Bostrom, 2014) , and sometimes no objective solution exists (e.g., in unsupervised learning).2 This simplified example does not model the agent actually carrying out its mission.

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Towards Interactive Inverse Reinforcement Learning - Armstrong, Leike 2016.tei.xml


Abstract: Traditionally, autonomous cars treat human-driven vehicles like moving obstacles. They predict their future trajectories and plan to stay out of their way. While physically safe, this results in defensive and opaque behaviors. In reality, an autonomous car's actions will actually affect what other cars will do in response, creating an opportunity for coordination. Our thesis is that we can leverage these responses to plan more efficient and communicative behaviors. We introduce a formulation of interaction with human-driven vehicles as an underactuated dynamical system, in which the robot's actions have consequences on the state of the autonomous car, but also on the human actions and thus the state of the human-driven car. We model these consequences by approximating the human's actions as (noisily) optimal with respect to some utility function. The robot uses the human actions as observations of her underlying utility function parameters. We first explore learning these parameters offline, and show that a robot planning in the resulting underactuated system is more efficient than when treating the person as a moving obstacle. We also show that the robot can target specific desired effects, like getting the person to switch lanes or to proceed first through an intersection. We then explore estimating these parameters online, and enable the robot to perform active information gathering: generating actions that purposefully probe the human in order to clarify their underlying utility parameters, like driving style or attention level. We show that this significantly outperforms passive estimation and improves efficiency. Planning in our model results in coordination behaviors: the robot inches forward at an intersection to see if can go through, or it reverses to make the other car proceed first. These behaviors result from the optimization, without relying on hand-coded signaling strategies. Our user studies support the utility of our model when interacting with real users. Keywords Planning for human-robot interaction • Mathematical models of human behavior • Autonomous driving This is one of several papers published in Autonomous Robots comprising the "Special Issue on Robotics Science and Systems".

Dorsa Sadigh
Nick Landolfi
Shankar S Sastry
Sanjit A Seshia
Anca D Dragan

Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state

This paper combines work from  Sadigh et al. (2016b, c) . It adds a general formulation of the problem as a game, discusses its limitations, and lays out the assumptions we make to reduce it to a tractable problem. On the experimental side, it adds an analysis of the adaptivity of the behaviors produced to initial conditions for both offline and active estimation, an analysis of the benefits of active estimation on the robot's actual reward, and results on actively estimating user intentions as opposed to just driving style.
 Introduction Currently, autonomous cars tend to be overly defensive and obliviously opaque. When needing to merge into another lane, they will patiently wait for another driver to pass first. When stopped at an intersection and waiting for the driver on the right to go, they will sit there unable to wave them by. They are very capable when it comes to obstacle avoidance, lane keeping, localization, active steering and braking  (Urmson et al. 2008; Levinson et al. 2011; Falcone et al. , 2008 Dissanayake et al. 2001; Leonard et al. 2008) . But when it comes to other human drivers, they tend to rely on simplistic models: for example, assuming that other drivers will be bounded disturbances  (Gray et al. 2013; Raman et al. 2015) , they will keep moving at the same velocity  (Vitus and Tomlin 2013; Luders et al. 2010; Sadigh and Kapoor 2015) , Fig.  1  We equip autonomous cars with a model of how humans will react to the car's actions (a). We test the planner in user studies, where the car figures out that it can nudge into the human's lane to check their driving style (b, c): if it gets evidence that they are attentive it merges in front, expecting that the human will slow down; else, it retracts back to its lane (c). We also see coordination behavior at an intersection, with the car planning to inch forward to find out more about how the human will act, or even inch backwards to incentivize the human to go through first  (d)  or they will approximately follow one of a set of known trajectories  (Vasudevan et al. 2012; Hermes et al. 2009 ). These models predict the trajectory of other drivers as if those drivers act in isolation. They reduce human-robot interaction to obstacle avoidance: the autonomous car's task is to do its best to stay out of the other drivers' way. It will not nudge into the lane to test if the other driver yields, nor creep into the intersection to assert its turn for crossing. In reality, the actions of an autonomous car affect the actions of other drivers. Our goal is to leverage these effects in planning to improve efficiency and coordination with human drivers. We are not the first to propose that robot actions influence human actions. Work in social navigation recognized this, and addressed it by treating the human and the robot as being part of a team-a team that works together to make sure that each agent reaches their goal, and everyone avoids each other  (Trautman and Krause 2010; Trautman et al. 2013; Trautman 2013; Kuderer et al. 2015) . The robot computes a coupled plan by optimizing the team's objective jointly over the human and robot plans, assuming the human will follow their part of the plan, and re-planning at every step.  Nikolaidis et al. (2015)  recognized that the human might stray away from the plan that the robot computed, and modeled the human as switching to the robot's plan at every time step with some probability. We propose that when people stray away from the coupled plan, there is a fundamental reason for this: they have a different objective altogether. This is particularly true in driving, where coupled planning would assume that the person is just as interested in the robot reaching its goal as they are in themselves reaching theirs-selfish drivers are likely just trying to get home and not get into an accident; they are optimizing for something different. In this work, we explicitly account for the robot's influence on the person by modeling the person as optimizing their own objective or utility function in response to the robot's actions. We develop an optimization-based method for planning an autonomous vehicle's behavior in a manner that is cognizant of the effects it will have on human driver actions via this model. This optimization leads to plans like the ones in Fig.  1 . Assuming a universal human model that the robot learns offline, we see that the the orange car in (c) decides to cut in front of the human driver in order to more efficiently reach its goal. It arrives at this plan by anticipating that taking this action will cause the human to brake and make space for it. This comes in contrast to treating the person as an obstacle that moves (b) and being more defensive. Since not all drivers are the same, we also arm the robot with the ability to collect information about the human driver online. It then decides to nudge into the person's lane  (d)  and only merges if it gets evidence that the person is paying attention; otherwise it retreats back to its lane. We find fascinating behaviors at intersections (e): the car inches forward to test human attention, or even inches backwards to get the person to cross first through the intersection. These can be interpreted as signaling behaviors, but they emerge out of optimizing to affect human actions, without ever explicitly modeling human inferences. We achieve this by planning in an underactuated dynamical system: the robot's actions change not only robot state, but also influence human actions and thus human state. We model other drivers as acting approximately optimally according to some reward function that depends on state, human actions, as well as robot actions. We explore learning this human reward function offline, as well as online during the interaction: here, the robot has the opportunity to use its own actions to actively gather more information about the underlying reward. Our contributions are as follows 1 : 1. Formalism of Interaction with Drivers We begin by formalizing the interaction between a human and a robot as a partially observable stochastic game (POSG) in Sect. 2. The human and the robot can both act to change the state of the world, they have partial information because they don't know each other's reward functions, and they arrive at some tuple of policies that are at an equilibrium. This formulation has two issues: intractability, especially in continuous state and action spaces, and failing to capture human behavior, because humans tend to not follow Nash equilibria in day to day tasks  (Hedden and Zhang 2002) . We introduce a simplification of this formulation to an underactuated system. We assume that the robot decides on a trajectory u R , and the human computes a best response to u R (as opposed to trying to influence u R as would happen in a game). This reduction enforces turn-taking, and provides us with a dynamics model in which the human observes (or predicts) the robot's actions prior to selecting her actions. It maintains our ability to model the effects of the robot's actions on the human, because the robot can influence which actions the human takes by selecting actions that force a particular response.
 Approximate Optimization Solution for Known Human Model Assuming a known reward function for the human [which we learn offline through Inverse Reinforcement Learning  (Ng et al. 2000; Abbeel and Ng 2005; Ziebart et al. 2008; Levine and Koltun 2012) ], we derive an approximate solution for our system based on Model Predictive Control and a quasi-newton optimization in Sect. 3. At every step, the robot replans a trajectory u R by reasoning about the optimization that the human would do based on a candidate u R . We use implicit differentiation to obtain a gradient of the human's trajectory with respect to the robot's. This enables the robot to compute a plan in close to real-time.
 Extension to Online Inference of Human Reward The solution based on estimating human reward offline from training data is useful in some cases, but ultimately every driver is different, and even the same driver is sometimes more or less aggressive, more or less attentive, and so on. In Sect. 6, we thus also explore estimating the human reward function online. This turns the problem into a partially observable Markov decision process (POMDP), with the human reward parameters as the hidden state. Prior work that incorporates some notion of human state into planning has thus far separated estimation and planning, always using the current estimate of the human state to determine what the robot should do  (Javdani et al. 2015; Fern et al. 2007; Bandyopadhyay et al. 2013) . Although efficient, these approximations sacrifice an important aspect of POMDPs: the ability to actively gather information. In recent years, many efforts have developed more efficient algorithms for solving continuous POMDPs  (Chaudhari et al. 2013; Seiler et al. 2015; Agha-Mohammadi et al. 2014) ; however, computing the transition function for our POMDP is costly as it involves solving for human's best response. This renders the aforementioned approaches ineffective as they usually require either a succinct transition function or an easily computable one. Our work takes advantage of the underactuated system to gather information about the human reward parameters. Rather than relying on passive observations, the robot actually accounts for the fact that humans will react to their actions: it uses this knowledge to select actions that will trigger human reactions which in turn will clarify the internal state. This is in line with work in active information gathering in manipulation, safe exploration, or patrolling  (Javdani et al. 2013; Atanasov et al. 2014; Atanasov 2015) , now used over human state as opposed to physical state.
 Analysis of Planning in Human-Robot Driving Scenarios We present the consequences of planning in this dynamical system in Sect. 4, showcasing behaviors that emerge when rewarding the robot for certain effects on human state, like making the human slow down, change lanes, or go first through an intersection. We also show that such behaviors can emerge from simply rewarding the robot for reaching its goal state fast-the robot becomes more aggressive by leveraging its possible effects on human actions. This does not happen always: in Sect. 7, we show the robot maintains a belief over the human driver model, and starts nudging into their lane to figure out if they are going to let the robot merge, or inching forward at a 4-way stop. We test the planner in two user studies in a driving simulator: one with a human reward learned offline in Sect. 5, and one with online estimation in Sect. 8. Our results suggest that despite the approximation our algorithm makes, it significantly outperforms the "people as moving obstacles" interaction paradigm. We find that the robot achieves significantly higher reward when planning in the underactuated system, and that active information gathering does outperform passive estimation with real users. Overall, this paper takes a step towards robots that account for their effects on human actions in situations that are not entirely cooperative, and leverage these effects to coordinate with people. Natural coordination and interaction strategies are not hand-coded, but emerge out of planning in our model.
 General formalism for human-robot interaction as a game To enable a robot to autonomously generate behavior for interaction and coordination with a human, we need to set up a model that goes beyond a single agent acting in the physical world among moving obstacles, and consider two-agent models. One aspect that sets interaction with people apart from typical multi-agent models like those used for multirobot planning is that we do not know or have direct control over what the human is trying to do and how. Furthermore, the human also does not know what the robot is trying to do nor how the robot is going to do it. Even the simplest formulation of the problem becomes a two player game, and in what follows we introduce such a formulation and expose some of its issues, including the computational complexity of planning in such a model as well as the model's inaccuracy in capturing human behavior. We then propose an approximation that simplifies planning in the game to planning in an underactuated system: the robot had direct control over its actions, but also has a model for how its actions will influence those of the human. Much of robotics research focuses on how to enable a robot to achieve physical tasks, often times in the face of perception and movement error-of partially observable worlds and nondeterministic dynamics  (Prentice and Roy 2009; Javdani et al. 2013; Patil et al. 2015) . Part of what makes human-robot interaction difficult is that even if we assume the physical world to be fully observable and deterministic, we are still left with a problem as complex as an incomplete information repeated two player game. It is a game because it involves multiple rational (or rather, approximately rational) agents who can take actions to maximize their own (expected) utilities. It is repeated because unlike single shot games, the agents act over a time horizon. It is incomplete information because the agents do not necessarily know each others' reward functions  (Aumann et al. 1995) . Partially Observable Stochastic Game Extrapolating from the (PO)MDP formulation of a robot acting in isolation to human-robot interaction, we can formulate interaction as a special case of a Partially-Observable Stochastic Game (POSG)  (Hansen et al. 2004 ): there are two "players", the robot R and the human H; at every step t, they can apply control inputs u t R ∈ U R and u t H ∈ U H ; they each have a reward function, r R and r H ; and there is a state space S with states s consisting of both the physical state x, as well as reward parameters θ R and θ H . Including the reward parameters in the state is an unusual trick, necessary here in order to ensure that the human and the robot do not have direct access to each others' reward functions, while maintaining a well defined POSG: R does not observe θ H , and H does not observe θ R , but both agents can technically evaluate each reward at any state-control tuple (s t , u t R , u t H ) just because s contains the needed reward parameter information: s = (x, θ R , θ H )-if an agent knew the state, it could evaluate the other agent's reward. To simplify the physical world component and focus on the interaction component, we assume fully observable physical states with deterministic dynamics. The observations in the game are thus the physical state x and the controls that each agent applies at each time step. The dynamics model is deterministic, affecting x through the control inputs and leaving the reward parameters unchanged (a reasonable assumption for relatively short interactions). Aside 1 The POSG above is not necessarily the most direct extension of single-agent planning in deterministic fully observable state spaces to interactions. Collaborative interactions have been modeled simply as a centralized multi-agent system with a shared reward function  (Kuderer et al. 2015)  (i.e. r R = r H ), but this reduces to treating the human as another robot that the system has full control overessentially the system has more degrees of freedom to control, and there is no difference between the human DoFs and the robot DoFs. Unlike the POSG, it assumes that the human and the robot know each other's reward, and even more, that their rewards are identical. This can be true of specific collaborative tasks, but not of interactions in general: robots do not know exactly what humans want, humans do not know exactly what robots have been programmed to optimize for, and their rewards might have common terms but will not be identical. This happens when an autonomous car interacts with other drivers or with pedestrians, and it even happens in seemingly collaborative scenarios like rehabilitation, in which very long horizon rewards might be aligned but not short-term interaction ones. The POSG formulation captures this intricacy of general interaction.
 Aside 2 The POSG above is the simplest extension of the single-agent models to interactions between a human and a robot that do not share a reward function or know each others' rewards. More complex models might include richer state information (such as human's beliefs about the robot, trust, estimation of capability, mood, etc.), and might allow it to change over time.
 Limitations of the Game Formulation The POSG formulation is a natural way to characterize interaction from the perspective of MDP-like models, but is limited in two fundamental ways: (1) its computational complexity is prohibitive even in discrete state and action spaces  (Bernstein et al. 2002; Hansen et al. 2004 ) and no methods are known to handle continuous spaces, and (2) it is not a good model for how people actually work-people do not solve games in everyday tasks when they are not playing chess  (Hedden and Zhang 2002) . Furthermore, solutions here are tuples of policies that are in a Nash equilibrium, and it is not clear what equilibrium to select.
 Approximate solution as an underactuated system To alleviate the limitations from above, we introduce an approximate close to real-time solution, with a model of human behavior that does not assume that people compute equilibria of the game.
 Assumptions that simplify the POSG Our approximation makes several simplifying assumptions that turn the game into an offline learning phase in which the robot learns the human's reward function, followed by an online planning phase in which the robot is solving an underactuated control problem:
 Separation of Estimation & Control We separate the process of computing actions for the robot into two stages. First, the robot estimates the human reward function parameters θ H offline. Second, the robot exploits this estimate as a fixed approximation to the human's true reward parameters during planning. In the offline phase, we estimate θ H from user data via Inverse Reinforcement Learning  (Ng et al. 2000; Abbeel and Ng 2005; Ziebart et al. 2008; Levine and Koltun 2012) . This method relies heavily on the approximation of all humans to a constant set of reward parameters, but we will relax this separation of estimation and control in Sect. 6.
 Model Predictive Control (MPC) Solving the POSG requires planning to the end of the full-time horizon. We reduce the computation required by planning for a shorter horizon of N time steps. We execute the control only for the first time step, and then re-plan for the next N at the next time step  (Camacho and Alba 2013) . Let x = (x 1 , . . . , x N ) denote a sequence of states over a finite horizon, N , and let u H = (u 1 H , . . . , u N H ) and u R = (u 1 R , . . . , u N R ) denote a finite sequence of continuous control inputs for the human and robot, respectively. We define R R as the robot's reward over the finite MPC time horizon: R R (x 0 , u R , u H ) = N t=1 r R (x t , u t R , u t H ), (1) where x 0 denotes the present physical state at the current iteration, and each state thereafter is obtained from the previous state and the controls of the human and robot using a given dynamics model, f . At each iteration, we desire to find the sequence u R which maximizes the reward of the robot, but this reward ostensibly depends on the actions of the human. The robot might attempt to influence the human's actions, and the human, optimizing its own reward functions, might likewise attempt to influence the actions of the robot. Despite our reduction to a finite time horizon, the game formulation still demands computing equilibria to the problem. Our core assumption, which we discuss next, is that this is not required for most interactions: that a simpler model of what the human does suffices.
 Simplification of the Human Model To avoid computing these equilibria, we propose to model the human as responding rationally to some fixed extrapolation of the robot's actions. At every time step, t, H computes a simple estimate of R's plan for the remaining horizon, ũt+1:N R , based on the robot's previous actions u 0:t R . Then the human computes its plan u H as a best response  (Fudenberg and Tirole 1991)  to this estimate. We remark that with this assumption the human is not modeled as a passive agent since the reward function can model the behavior of any active agent, if allowed to have arbitrary complexity. With this simplification, we reduce the general game to a Stackelberg competition: the human computes its best outcome while holding the robots plan fixed. This simplification is justified in practice because usually the agents in driving scenarios are competing with each other rather than collaborating. Solutions to Stackelberg games are in general more conservative in competitive regimes since an information advantage is given to the second player. As a result, the quality of a good solution found in a Stackelberg game model does not decrease in practice. We additionally use a short time horizon, so not much is lost by this approximation. But fundamentally, we do assume that people would not try to influence the robot's actions, and this is a limitation when compared to solving the POSG. Let R H be the human reward over the time horizon: R H (x 0 , u R , u H ) = N t=1 r H (x t , u t R , u t H ), (2) then we can compute the control inputs of the human from the remainder of the horizon by: u t H (x 0 , u 0:t R , ũt+1:N R ) = arg max u t+1:T H R H (x t , ũt+1:N R , u t+1:N H ). (3) This human model would certainly not work well in adversarial scenarios, but our hypothesis, supported by our results, is that it is useful enough in day-to-day tasks to enable robots to be more effective and more fluent interaction partners. In our work, we propose to make the human's estimate ũR equal to the actual robot control sequence u R . Our assumption that the time horizon is short enough that the human can effectively extrapolate the robot's course of action motivates this decision. With this presumption, the human's plan becomes a function of the initial state and robot's true plan: u * H (x 0 , u R ) = arg max u H R H (x t , u R , u H ). (4) This is now an underactuated system the robot has direct control over (can actuate) u R and indirect control over (cannot actuate but does affect) u H . However, the dynamics model in our setup is more sophisticated than in typical underactuated systems because it models the response of the humans to the robot's actions. Evaluating the dynamics model requires solving for the optimal human response, u * H . The system is also a special case of an MDP, with the state as in the POSG, the actions being the actions of the robot, and the world dynamics being dictated by the human's response and the resulting change on the world from both human and robot actions. The robot can now plan in this system to determine which u R would lead to the best outcome for itself: u * R = arg max u R R R x 0 , u R , u * H (x 0 , u R ) . (5)
 Planning with Quasi-Newton optimization Despite the reduction to a single agent complete information underactuated system, the dynamics remain too complex to solve in real-time. We lack an analytical form for u * H (x 0 , u R ) which forces us to solve (4) each time we evaluate the dynamics. Assuming a known human reward function r H [which we will obtain later through Inverse Reinforcement Learning (IRL), see  Ng et al. (2000) ,  Abbeel and Ng (2005) ,  Ziebart et al. (2008)  and  Levine and Koltun (2012) ], we can solve (5) locally, using gradient-based methods. Our main contribution is agnostic to the particular optimization method, but we use L-BFGS  (Andrew and Gao 2007) , a quasi-Newton method that stores an approximate inverse Hessian implicitly resulting in fast convergence. To perform the local optimization, we need the gradient of (2) with respect to u R : ∂ R R ∂u R = ∂ R R ∂u H ∂u * H ∂u R + ∂ R R ∂u R (6) We can compute both ∂ R R ∂u H and ∂ R R ∂u R symbolically through back-propagation because we have a representation of R R in terms of u H and u R . What remains, ∂u * H ∂u R , is difficult to compute because u * H is technically the outcome of a global optimization. To compute ∂u * H ∂u R , we use the method of implicit differentiation. Since R H is a smooth function whose minimum can be attained, we conclude that for the unconstrained optimization in (4), the gradient of R H with respect to u H evaluates to 0 at its optimum u * H : ∂ R H ∂u H x 0 , u R , u * H (x 0 , u R ) = 0 ( 7 ) Now, we differentiate the expression in (  7 ) with respect to u R : ∂ 2 R H ∂u 2 H ∂u * H ∂u R + ∂ 2 R H ∂u H ∂u R ∂u R ∂u R = 0 ( 8 ) Finally, we solve for a symbolic expression of ∂u * H ∂u R : ∂u * H ∂u R = − ∂ 2 R H ∂u H ∂u R ∂ 2 R H ∂u 2 H −1 (9) and insert it into (6), providing an expression for the gradient ∂ R R ∂u R .
 Offline estimation of human reward parameters Thus far, we have assumed access to r H (x t , u t R , u t H ). In our implementation, we learn this reward function from human data. We collect demonstrations of a driver in a simulation environment, and use Inverse Reinforcement Learning  (Ng et al. 2000; Abbeel and Ng 2005; Ziebart et al. 2008; Levine and Koltun 2012; Shimosaka et al. 2014; Kuderer et al. 2015)  to recover a reward function that explains the demonstrations. To handle continuous state and actions space, and cope with noisy demonstrations that are perhaps only locally optimal, we use continuous inverse optimal control with locally optimal examples  (Levine and Koltun 2012) . We parametrize the human reward function as a linear combination of features: r H (x t , u t R , u t H ) = θ H φ(x t , u t R , u t H ), (10) and apply the principle of maximum entropy  (Ziebart et al. 2008; Ziebart 2010)  to define a probability distribution over human demonstrations u H , with trajectories that have higher reward being more probable: P(u H | x 0 , θ H ) = exp R H (x 0 , u R , u H ) exp R H (x 0 , u R , ũH ) d ũH . ( 11 ) We then optimize the weights θ H in the reward function that make the human demonstrations the most likely: max θ H P u H | x 0 , θ H (12) We approximate the partition function in (  11 ) following  (Levine and Koltun 2012) , by computing a second order Taylor approximation around the demonstration:  R H (x 0 , u R , ũH ) R H (x 0 , u R , u H ) + ( ũH − u H ) ∂ R H ∂u H + ( ũH − u H ) ∂ 2 R H ∂u 2 H ( ũH − u H ), ( 13 ) which results in the the integral in (11) reducing to a Gaussian integral, with a closed form solution. See  Levine and Koltun (2012)  for more details. We display a heat map of features we used in Fig.  2 . The warmer colors correspond to higher rewards. In addition to the features shown in the figure, we include a quadratic function of the speed to capture efficiency in the objective. The five features include: - φ 1 ∝ c 1 • exp(−c 2 • d 2 ): distance to the boundaries of the road, where d is the distance between the vehicle and the road boundaries and c 1 and c 2 are appropriate scaling factors as shown in Fig.  2a . -φ 2 : distance to the middle of the lane, where the function is specified similar to φ 1 as shown in Fig.  2b . -φ 3 = (v − v max ) 2 : higher speed for moving forward through, where v is the velocity of the vehicle, and v max is the speed limit. -φ 4 = β H • n: heading; we would like the vehicle to have a heading along with the road using a feature, where β H is the heading of H, and n is a normal vector along the road. -φ 5 corresponds to collision avoidance, and is a nonspherical Gaussian over the distance of H and R, whose major axis is along the robot's heading as shown in Fig.  2c . We collected demonstrations of a single human driving for approximately an hour in an environment with multiple autonomous cars, which followed precomputed routes. Despite the simplicity of our features and robot actions during the demonstrations, the learned human model proved sufficient for the planner to produce human-interpretable behavior (case studies in Sect. 4), and actions which affected human action in the desired way (user study in Sect. 5).
 Implementation details In our implementation, we used the software package Theano  (Bergstra et al. 2010; Bastien et al. 2012)  to symbolically compute all Jacobians and Hessians. Theano optimizes the computation graph into efficient C code, which is crucial for real-time applications. This implementation enables us to solve each step of the optimization in equation (  5 ) in approximately 0.3 s for horizon length N = 5 on a 2.3 GHz Intel Core i7 processor with 16 GB RAM. Future work will focus on achieving better computation time and a longer planning horizon.
 Case studies with offline estimation We noted earlier that the state of the art autonomous driving plans conservatively because of its simple assumptions regarding the environment and vehicles on the road. In our experiments, we demonstrate that an autonomous vehicle can purposefully affect human drivers, and can use this ability to gather information about the human's driving style and goals. In this section, we introduce 3 driving scenarios, and show the result of our planner assuming a simulated human driver, highlighting the behavior that emerges from different robot reward functions. In the next section, we test the planner with real users and measure the effects of the robot's plan. Figure  3  illustrates our three scenarios, and contains images from the actual user study data.
 Conditions for analysis across scenarios In all three scenarios, we start from an initial position of the vehicles on the road, as shown in Fig.  3 . In the control condition, we give the car the reward function to avoid collisions and have high velocity. We refer to this as R control . In the experimental condition, we augment this reward function with a term corresponding to a desired human action (e.g. low speed, lateral position, etc.). We refer to this as R control + R affect . Sections 4.3 through 4.5 contrast the two plans for each of our three scenarios. Section 4.6 shows what happens when instead of explicitly giving the robot a reward function designed to trigger certain effects on the human, we simply task the robot with reaching a destination as quickly as possible.
 Driving simulator We use a simple point-mass model of the car's dynamics. We define the physical state of the system x = [x y ψ v] , where x, y are the coordinates of the vehicle, ψ is the heading, and v is the speed. We let u = [u 1 u 2 ] represent the control input, where u 1 is the steering input and u 2 is the acceleration.
 (a) (b) (c) Fig.  3  Driving scenarios. In (a), the car plans to merge in front of the human in order to make them slow down. In (b), the car plans to direct the human to another lane, and uses its heading to choose which lane the human will go to. In (c), the car plans to back up slightly in order to make the human proceed first at the intersection. None of these plans use any hand coded strategies. They emerge out of optimizing with a learned model of how humans react to robot actions. In the training data for this model, the learned was never exposed to situations where another car stopped at an orientation as in (b), or backed up as in (c). However, by capturing human behavior in the form of a reward, the model is able to generalize to these situations, enabling the planner to find creative ways of achieving the desired effects We denote the friction coefficient by μ. We can write the dynamics model: [ ẋ ẏ ψ v] = [v •cos(ψ) v •sin(ψ) v •u 1 u 2 −μ•v] (14)
 Scenario 1: Make human slow down In this highway driving setting, we demonstrate that an autonomous vehicle can plan to cause a human driver to slow down. The vehicles start at the initial conditions depicted on left in Fig.  3a , in separate lanes. In the experimental condition, we augment the robot's reward with the negative of the square of the human velocity, which encourages the robot to slow the human down. Figure  3a  contrasts our two conditions. In the control condition, the human moves forward uninterrupted. In the experimental condition, however, the robot plans to move in front of the person, anticipating that this will cause the human to brake.
 Scenario 2: Make human go left/right In this scenario, we demonstrate that an autonomous vehicle can plan to affect the human's lateral location, making the human switch lanes. The vehicles start at the initial conditions depicted on left in Fig.  3b , in the same lane, with the robot ahead of the human. In the experimental condition, we augment the robot's reward with the lateral position of the human, in two ways, to encourage the robot to make the human go either left (orange border image) or right (blue bor-Fig.  4  Heat map of the reward functions in scenarios 2 and 3. The warmer colors show higher reward values. In (a, b), the reward function of the autonomous vehicle is plotted, which is a function of the human driven vehicle's position. In order to affect the driver to go left, the reward is higher on the left side of the road in (a), and to affect the human to go right in (b), the rewards are higher on the right side of the road. In (c), the reward of the autonomous vehicle is plotted for scenario 3 with respect to the position of the human driven car. Higher rewards correspond to making the human cross the intersection der image). The two reward additions are shown in Fig.  4a,  b . Figure  3b  contrasts our two conditions. In the control condition, the human moves forward, and might decide to change lanes. In the experimental condition, however, the robot plans to intentionally occupy two lanes (using either a positive or negative heading), anticipating this will make the human avoid collision by switching into the unoccupied lane.
 Scenario 3: Make human go first In this scenario, we demonstrate that an autonomous vehicle can plan to cause the human to proceed first at an intersection. The vehicles start at the initial conditions depicted on the left in Fig.  3c , with both human and robot stopped at the 4-way intersection. In the experimental condition, we augment the robot's reward with a feature based on the y position of the human car y H relative to the middle of the intersection y 0 . In particular, we used the hyperbolic tangent of the difference, tanh(y H − y 0 ). The reward addition is shown in Fig.  4c . Figure  3c  contrasts our two conditions. In the control condition, the robot proceeds in front of the human. In the experimental condition, however, the robot plans to intentionally reverse slightly, anticipating that this will induce the human cross first. We might interpret such a trajectory as communicative behavior, but communication was never explicitly encouraged in the reward function. Instead, the goal of affecting human actions led to this behavior. Reversing at an intersection is perhaps the most surprising result of the three scenarios, because it is not an action human drivers take. In spite of this novelty, our user study suggests that human drivers respond in the expected way: they proceed through the intersection. Further, pedestrians sometimes exhibit behavior like the robot's, stepping back from an intersection in order to let a car pass first.
 Behaviors also emerge from efficiency Thus far, we have explicitly encoded a desired effect on human actions, and optimized it as a component of the robot's reward. We have also found, however, that behaviors like those we have seen so far can emerge out of the need for efficiency. Figure  5  (bottom) shows the generated plan for when the robot is given the goal to reach a point in the left lane as quickly as possible (reward shown in Fig.  6 ). By modeling the effects its actions have on the human actions, the robot plans to merge in front of the person, expecting that they will slow down. Fig.  5  A time lapse where the autonomous vehicle's goal is to reach a final point in the left lane. In the top scenario, the autonomous vehicle has a simple model of the human driver that does not account for the influence of its actions on the human actions, so it acts more defensively, waiting for the human to pass first. In the bottom, the autonomous vehicle uses the learned model of the human driver, so it acts less defensively and reaches its goal faster In contrast, the top of the figure shows the generated plan for when the robot uses a simple (constant velocity) model of the person. In this case, the robot assumes that merging in front of the person can lead to a collision, and defensively waits for the person to pass, merging behind them. We hear about this behavior often in autonomous cars today: they are defensive. Enabling them to plan in a manner that is cognizant that they can affect other driver actions can make them more efficient at achieving their goals.
 The robot behavior adapts to the situation Throughout the case studies, we see examples of coordination behavior that emerges out of planning in our system: going in front of someone knowing they will brake, slowing and nudging into another lane to incentivize a lane change, or backing up to incentivize that the human proceeds first through an intersection. Such behaviors could possibly be hand-engineered for those particular situations rather than autonomously planned. However, we advocate that the need to plan comes from versatility: from the fact that the planner can adapt the exact strategy to each situation. With this work, we are essentially shifting the design burden from designing policies to designing reward functions. We still need to decide on what we want the robot to do: do we want it to be selfish, do we want it to be extra polite and try to get every human to go first, or do we want it to be somewhere in between? Our paper does not answer this question, and designing good reward functions remains challenging. However, this work does give us the tools to autonomously generate (some of) the strategies needed to then optimize such reward functions when interacting with people. With policies, we'd be crafting that the car should inch forward or backwards from the intersection, specifying by how much and at what velocity, and how this depends on where the other cars are, and how it depends on the type of intersection. With this work, barring local optima issues and the fact that human models could always be improved, all that we need to specify Fig.  7  The robot adapts its merging behavior depending on the relative position of the person: it does not always cut the person off: sometimes it merges behind the person, and if it starts too close (depending on how the reward function is set up) it will not merge at all is the regular reward that we would give an autonomous car, or, if we want it to purposefully influence human behavior, the desired outcome. The car figures out how to achieve it, adapting its actions to the different settings. Figure  7  shows a spectrum of behaviors for the robot depending on where it starts relative to the human: from merging behind the person, to not merging, to merging in front.
 User study with offline estimation The previous section showed the robot's plans when interacting with a simulated user that perfectly fits the robot's model of the human. Next, we present the results of a user study that evaluates whether the robot can successfully have the desired effects on real users.
 Experimental design We use the same 3 scenarios as in the previous section.
 Manipulated Factors We manipulate a single factor: the reward that the robot is optimizing, as described in Sect. 4.1. This leads to two conditions: the experimental condition where the robot is encouraged to have a particular effect on human state though the reward R control + R affect , and the control condition where that aspect is left out of the reward function and the robot is optimizing only R control (three conditions for Scenario 2, where we have two experimental conditions, one for the left case and one for the right case). Dependent Measures For each scenario, we measure the value along the user trajectory of the feature added to the reward function for that scenario, R affect . Specifically, we measure the human's negative squared velocity in Scenario 1, the human's x axis location relative to center in Scenario 2, and whether the human went first or not through the intersection in Scenario 3 (i.e. a filtering of the feature that normalizes for difference in timing among users and measures the desired objective directly). Hypothesis We hypothesize that our method enables the robot to achieve the effects it desires not only in simulation, but also when interacting with real users: The reward function that the robot is optimizing has a significant effect on the measured reward during interaction. Specifically, R affect is higher, as planned, when the robot is optimizing for it.
 Subject Allocation We recruited 10 participants (2 female, 8 male). All the participants owned drivers license with at least 2 years of driving experience. We ran our experiments using a 2D driving simulator, we have developed with the driver input provided through driving simulator steering wheel and pedals.
 Analysis Scenario 1 A repeated measures ANOVA showed the square speed to be significantly lower in the experimental condition than in the control condition (F(1, 160) = 228.54, p < 0.0001). This supports our hypothesis: the human moved slower when the robot planned to have this effect on the human. We plot the speed and latitude profile of the human driven vehicle over time for all trajectories in Fig.  8 . Figure  8a  shows the speed profile of the control condition trajectories in gray, and of the experimental condition trajectories in orange. Figure  8b  shows the mean and standard error for each condition. In the control condition, human squared speed keeps increasing. In the experimental condition however, by merging in front of the human, the robot is triggering the human to brake and reduce speed, as planned. The purple trajectory represents a simulated user that perfectly matches the robot's model, showing the ideal case for the robot. The real interaction moves significantly in the desired direction, but does not perfectly match the ideal model, since real users do not act exactly as the model would predict. The figure also plots the y position of the vehicles along time, showing that the human has not travelled as far forward in the experimental condition. Scenario 2 A repeated measures ANOVA showed a significant effect for the reward factor (F(2, 227) = 55.58, p < 0.0001). A post-hoc analysis with Tukey HSD showed that both experimental conditions were significantly different from the control condition, with the user car going more to the left than in the control condition when R affect rewards The first column shows the speed of all trajectories with its mean and standard errors in the bottom graph. The second column shows the latitude of the vehicle over time; similarly, with the mean and standard errors. The gray trajectories correspond to the control condition, and the orange trajectories correspond to the experimental condition: the robot decides to merge in front of the users and succeeds at slowing them down. The purple plot corresponds to a simulated user that perfectly matches the model that the robot is using left user positions ( p < 0.0001), and more to the right in the other case ( p < 0.001). This supports our hypothesis. We plot all the trajectories collected from the users in Fig.  9 . Figure  9a  shows the control condition trajectories in gray, while the experimental conditions trajectories are shown in orange (for left) and blue (for right). By occupying two lanes, the robot triggers an avoid behavior from the users in the third lane. Here again, purple curves show a simulated user, i.e. the ideal case for the robot. Scenario 3 An ordinal logistic regression with user as a random factor showed that significantly more users went first in the intersection in the experimental condition than in the baseline (χ 2 (1, 129) = 106.41, p < 0.0001). This supports our hypothesis. Figure  10  plots the y position of the human driven vehicle with respect to the x position of the autonomous vehicle. For trajectories that have a higher y position for the human vehicle than the x position for the robot, the human car has crossed the intersection before the autonomous vehicle. The lines corresponding to these trajectories travel above the origin, which is shown with a blue square in this figure. The mean of the orange lines travel above the origin, which means that the autonomous vehicle has successfully affected the humans to cross first. The gray lines travel below the origin, i.e. the human crossed second. Overall, our results suggest that the robot was able to affect the human state in the desired way, even though it does not have a perfect model of the human. into problems, because not all people behave according to the same estimated θ H . Different drivers have different driving styles. Some are very defensive, more so than our learned model. Others are much more aggressive, and for instance would not actually brake when the car merges in front of them. Even for the same driver, their style might change over time, for instance when they get distracted on their phone. In this section we relax our assumption of an offline estimation of the human's reward parameters θ H . Instead, we explore estimating this online. We introduce an algorithm which maintains a belief over a space of candidate reward functions, and enable the robot to perform inference over this space throughout the interaction. We maintain tractability by clustering possible θ H s into a few options that the robot maintains a belief over.
 A POMDP with human reward as the hidden variable The human's actions are influenced by their internal reward parameters θ H that the robot does not directly observe. So far, we estimated θ H offline and solved an underactuated system, a special case of an MDP. Now, we want to be able to adapt our estimate of θ H online, during interaction. This turns the problem into a partially observable markov decision process (POMDP) with θ H as the hidden state. By putting θ H in the state, we now have a know dynamics model like in the underactuated system before for the robot and the human state, and we assume θ H to remain fixed regardless of the robot's actions. If we could solve the POMDP, the robot would estimate θ H from the human's actions, optimally trading off between exploiting it's current belief over θ H and actively taking information gathering actions intended to cause human reactions, which result in a better estimate of θ H . Because POMDPs cannot be solved tractably, several approximations have been proposed for similar problem formulations  (Javdani et al. 2015; Lam et al. 2015; Fern et al. 2007 ). These approximations are passively estimating the human internal state, and exploiting the belief to plan robot actions.  2  In this work, we take the opposite approach: we focus explicitly on active information gathering Our formulation enables the robot to choose to actively probe the human, and thereby improve its estimate of θ H . We leverage this method in conjunction with exploitation methods, but the algorithm we present may also be used alone if human internal state (reward parameters) estimation is the robot's primary objective.
 Simplification to information gathering We denote a belief in the value of the hidden variable, θ , as a distribution b(θ ), and update this distribution according to the likelihood of observing a particular human action, given the state of the world and the human internal state: b t+1 (θ ) ∝ b t (θ ) • P(u t H | x t , u R , θ). ( 15 ) In order to update the belief b, we require an observation model. Similar to before, we assume that actions with lower reward are exponentially less likely, building on the principle of maximum entropy  (Ziebart et al. 2008) : P(u H | x, u R , θ) ∝ exp R θ H (x 0 , u R , u H ) . ( 16 ) To make explicit our emphasis on taking actions which effectively estimate θ , we redefine the robot's reward function to include an information gain term, i.e., the difference between entropies of the current and updated beliefs: H (b t )− H (b t+1 ). The entropy over the belief H (b) evaluates to: H (b) = − θ b(θ ) log(b(θ )) θ b(θ ) . ( 17 ) We now optimize our expected reward with respect to the hidden state θ , and this optimization explicitly entails reasoning about the effects that the robot actions will have on the observations, i.e., the actions that the human will take in response, and how useful these observations will be in shattering ambiguity about θ .
 Explore-exploit trade-off In practice, we use information gathering in conjunction with exploitation. We do not solely optimize the information gain term H (b t )−H (b t+1 ), but optimize it in conjunction with the robot's actual reward function assuming the current estimate of θ : r augmented R (x t , u R , u H ) = λ(H (b t ) − H (b t+1 )) + r R (x t , u R , u H , b t ) (18) At the very least, we do this as a measure of safety, e.g., we want an autonomous car to keep avoiding collisions even when it is actively probing a human driver to test their reactions. We choose λ experimentally, though existing techniques that can better adapt λ over time  (Vanchinathan et al. 2014) .
 (a) (b) (c) Fig.  11  Our three scenarios, along with a comparison of robot plans for passive estimation (gray) versus active information gathering (orange). In the active condition, the robot is purposefully nudging in or braking to test human driver attentiveness. The color of the autonomous car in the initial state is yellow, but changes to either gray or orange in cases of passive and active information gathering respectively
 Solution via model predictive control To find the control inputs for the robot we locally solve: u * R = arg max u R E θ R R x 0 , u R , u * ,θ H (x 0 , u R ) (19) over a finite horizon N , where u * ,θ H (x 0 , u R ) corresponds to the actions the human would take from state x 0 if the robot executed actions u R . This objective generalizes (5) with an expectation over the current belief over θ , b 0 . We still assume that the human maximizes their own reward function, r θ H (x t , u t R , u t H ); we add the superscript θ to indicate the dependence on the hidden state. We can write the sum of human rewards over horizon N as: R θ H (x 0 , u R , u H ) = N −1 t=0 r θ H (x t , u t R , u t H ) (20) Computing this over the continuous space of possible reward parameters θ is intractable even with discretization. Instead, we learn clusters of θ s offline via IRL, and online use estimation to figure out which cluster best matches the human. Despite optimizing the trade-off in (18), we do not claim that our method as-is can better solve the general POMDP formulation: only that it can be used to get better estimates of human internal state. Different tradeoffs λ will result in different performance. Our results below emphasize the utility of gathering information, but also touch on the implications for active information gathering on R R .
 Case studies with online estimation In this section, we show simulation results that use the method from the previous section to estimate human driver type in the interaction between an autonomous vehicle and a human-driven vehicle. We consider three different autonomous driving scenarios. In these scenarios, the human is either distracted or attentive during different driving experiments. The scenarios are shown in Fig.  11 , where the yellow car is the autonomous vehicle, and the white car is the human driven vehicle. Our goal is to plan to actively estimate the human's driving style in each one of these scenarios, by using the robot's actions.
 Attentive versus distracted human driver models Our technique requires reward functions r θ H that model the human behavior for a particular internal state θ . We obtain a generic driver model via Continuous Inverse Optimal Control with Locally Optimal Examples  (Levine and Koltun 2012)  from demonstrated trajectories in a driving simulator in an environment with multiple autonomous cars, which followed precomputed routes. We then adjust the learned weights to model attentive versus distractive drivers. Specifically, we modify the weights of the collision avoidance features, so the distracted human model has less weight for these features. Therefore, the distracted driver is more likely to collide with the other cars while the attentive driver has high weights for the collision avoidance feature. In future work, we plan to investigate ways of automatically clustering learned θ H s from data from different users, but we show promising results even with these simple options.
 Manipulated factors We manipulate the reward function that the robot is optimizing. In the passive condition, the robot optimizes a simple reward function for collision avoidance based on the current belief estimate. It then updates this belief passively, by observing the outcomes of its actions at every time step. In the active condition, the robot trades off between this reward function and information gain in order to explore the human's driving style. We also manipulate the human internal reward parameters to be attentive or distracted. The human is simulated to follow the ideal model of reward maximization for our two rewards.
 Scenarios and qualitative results Scenario 1: Nudging In to Explore on a Highway In this scenario, we show an autonomous vehicle actively exploring the human's driving style in a highway driving setting. We contrast the two conditions in Fig.  11a . In the passive condition, the autonomous car drives on its own lane without interfering with the human throughout the experiment, and updates its belief based on passive observations gathered from the human car. However, in the active condition, the autonomous car actively probes the human by nudging into her lane in order to infer her driving style. An attentive human significantly slows down (timid driver) or speeds up (aggressive driver) to avoid the vehicle, while a distracted driver might not realize the autonomous actions and maintain their velocity, getting closer to the autonomous vehicle. It is this difference in reactions that enables the robot to better estimate θ . Scenario 2: Braking to Explore on a Highway In the second scenario, we show the driving style can be explored by the autonomous car probing the human driver behind it. The two vehicles start in the same lane as shown in Fig.  11b , where the autonomous car is in the front. In the passive condition, the autonomous car drives straight without exploring or enforcing any interactions with the human driven vehicle. In the active condition, the robot slows down to actively probe the human and find out her driving style. An attentive human would slow down and avoid collisions while a distracted human will have a harder time to keep safe distance between the two cars. Scenario 3: Nudging In to Explore at an Intersection In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.  11c . In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoid-  Fig.  13  The probability that the robot assigns to attentive as a function of time, for the attentive (left) and distracted (right). Each plot compares the active algorithm to passive estimation, showing that active information gathering leads to more accurate state estimation, in simulation and with real users ance. This provides limited observations from the human car resulting in a low confidence belief distribution. In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 Quantitative results Throughout the remainder of the paper, we use a common color scheme to plot results for our experimental conditions. We show this common scheme in Fig.  12 : darker colors (black and red) correspond to attentive humans, and lighter colors (gray and orange) correspond to distracted humans. Further, the shades of orange correspond to active information gathering, while the shades of gray indicate passive information gathering. We also use solid lines for real users, and dotted lines for scenarios with an ideal user model learned through inverse reinforcement learning. Figure  13  plots, using dotted lines, the beliefs over time for the attentive (left) and distracted (right) conditions, comparing in each the passive (dotted black and gray respectively) with the active method (dotted dark orange and light orange respectively). In every situation, the active method achieves a more accurate belief (higher values for attentive on the left, when the true θ is attentive, and lower values on the right, when the true θ is distracted). In fact, passive estimation sometimes incorrectly classifies drivers as attentive when they are distracted and vice-versa. The same figure also shows (in solid lines) results from our user study of what happens when the robot no longer interacts with an ideal model. We discuss these in the next section. Figures 14 and 15 plot the corresponding robot and human trajectories for each scenario. The important takeaway from these figures is that there tends to be a larger gap between attentive and distracted human trajectories in the active condition (orange shades) than in the passive condition (gray shades), especially in scenarios 2 and 3. It is this difference that helps the robot better estimate θ : the robot in the active condition is purposefully choosing actions that will lead to large differences in human reactions, in order to more easily determine the human driving style.
 Robot behavior adapts to the situation As Fig.  14  suggests, active info gathering results in interesting coordination behavior. In Scenario 1, the robot decides to nudge into the person's lane. But what follows next nicely reacts to the person's driving style. The robot proceeds with the merge if the person is attentive, but actually goes back to its lane if the person is distracted. Even more interesting is what happens in Scenario 3 at the 4way stop. The robot inches forward into the intersection, and proceeds if the person is attentive, but actually goes back to allow the person through if they are distracted! These all emerge as the optima in our system. The behavior also naturally changes as the initial state of the system changes. Figure  16  shows different behaviors arising from an attentive driver model but different initial position of the human driver. This shows that even for the same driver model, the robot intelligently adapts its coordination behavior to the situation, sometimes deciding to merge but sometimes not. This is particularly important, because it might be easy to handcode these coordination strategies for a particular situation. Much like in the onffline estimation case, the robot not only comes up with these strategies, but actually adapts them depending on the situation-the driver style, the initial state, and so on.
 Active information gathering helps the robot's actual reward So far, we have looked at how active information gathering improves estimation of the driver model. This is useful in itself in situations where human internal state estimation is the end-goal. But it is also useful for enabling the robot to better achieve its goal. Intuitively, knowing the human driving style more accurately should improve the robot's ability to collect reward. For instance, if the robot starts unsure of whether the human is paying attention or not, but collects enough evidence that she is, then the robot can safely merge in front of the person and be more efficient. Of course, this is not always the case. If the person is distracted, then the information gathering actions could be a waste because the robot ends up not merging anyway. Figure  17  shows what happens in the merging scenario: the robot gains more reward compared to passive estimation by doing information gathering with attentive drivers, because it figures out it is safe to merge in front of them; the robot looses some reward compared to passive estimation with distracted drivers, because it makes the effort to nudge in but has to retreat back to its lane anyway because it cannot merge. Of course, all this depends on choosing λ, the tradeoff between exploitation and exploration (information gain). Figure  18  shows the effect of λ has on the robot's goal reward (not its information gain reward), which shows that not all λs are useful. In other situations, we would also expect to see a large decrease in reward from too much weight on information gain.
 Beyond driving style: active intent inference Driving style is not the only type of human internal state that our method enables robots to estimate. If the human has a goal, e.g. of merging into the next lane or not, or of exiting the highway or not, the robot could estimate this as well using the same technique. Each possible goal corresponds to a feature. When estimating which goal the human has, the robot is deciding among θ s which place weight on only one of the possible goal features, and 0 on the others. Figure  19  shows the behavior that emerges from estimating whether the human wants to merge into the robot's lane. In the passive case, the human is side by side with the robot. Depending on the driving style, they might slow down slightly, accelerate slightly, or start nudging into the robot's lane, but since the obser-vation model is noisy the robot does not get quite enough confidence in the human's intent early on. Depending on the Fig.  17  Active info gathering improves the robot's ability to efficiently achieve its goal in the case when the human is attentive: where a passive estimator never gets enough information to know that the person is paying attention, an active estimator nudges in, updates its belief, and proceeds with the merge. At the same time, active info gathering does not hurt too much when the person is distracted: the robot nudges in slightly (this does decrease its reward relative to the passive case, but not by much), updates its belief, and retreats to its lane Fig.  18  Active information gathering behavior when the robot's goal is to merge into the left lane for different values of λ, together with the reward the robot obtains. λ = 0 results in low reward because the robot does not figure out that the person is attentive and does not merge. A small λ hurts the reward because the information gathering costs but does not buy anything. For higher values, the robot gets enough information that it forces a merge in front of the human robot's reward, it might take a long time before the person can merge. In the active case, the robot decides to probe the person by slowing down and shifting away from the person in order to make room. It then becomes optimal for the per-   19  Actively estimating the human's intent (whether they want to merge in the right lane or not). The robot slows down and shifts slightly away from the person, which would make someone who wants to merge proceed. This could be useful for robots trying to optimize for the good of all drivers (rather than for their selfish reward function) son wanting to merge to start shifting towards the robot's lane, giving the robot enough information now to update its belief. In our experiment, we see that this is enough for the person to be able to complete the merge faster, despite the robot not having any incentive to help the person in its reward.
 User study with online estimation In the previous section, we explored planning for an autonomous vehicle that actively probes a human's driving style, by braking or nudging in and expecting to cause reactions from the human driver that would be different depending on their style. We showed that active exploration does significantly better at distinguishing between attentive and distracted drivers using simulated (ideal) models of drivers. Here, we show the results of a user study that evaluates this active exploration for attentive and distracted human drivers.
 Experimental design We use the same three scenarios discussed in the previous section. Manipulated Factors We manipulated the same two factors as in our simulation experiments: reward function that the robot is optimizing (whether it is optimizing its reward through passive state estimation, or whether it is trading off with active information gathering), and the human internal state (whether the user is attentive or distracted). We asked our users to pay attention to the road and avoid collisions for the attentive case, and asked our users to play a game on a mobile phone during the distracted driving experiments. Dependent Measure We measured the probability that the robot assigned along the way to the human internal state. Hypothesis The active condition will lead to more accurate human internal state estimation, regardless of the true human internal state. Subject Allocation We recruited 8 participants (2 female, 6 male) in the age range of 21-26 years old. All participants owned a valid driver license and had at least 2 years of driving experience. We ran the experiments using a 2D driving simulator with the steering input and acceleration input provided through a steering wheel and a pedals as shown in Fig.  1 . We used a within-subject experiment design with counterbalanced ordering of the four conditions.
 Analysis We ran a factorial repeated-measures ANOVA on the probability assigned to "attentive", using reward (active vs. passive) and human internal state (attentive vs. distracted) as factors, and time and scenario as covariates. As a manipulation check, attentive drivers had significantly higher estimated probability of "attentive" associated than distracted drivers (0.66 vs 0.34, F = 3080.3, p < 0.0001). More importantly, there was a signifiant interaction effect between the factors (F = 1444.8, p < 0.000). We ran a post-hoc analysis with Tukey HSD corrections for multiple comparisons, which showed all four conditions to be significantly different from each other, all contrasts with p < 0.0001. In particular, the active information gathering did end up with higher probability mass on "attentive" than the passive estimation for the attentive users, and lower probability mass for the distracted user. This supports our hypothesis that our method works, and active information gathering is better at identifying the correct state. Figure  13  compares passive (grays and blacks) and active (light and dark oranges) across scenarios and for attentive (left) and distracted (right) users. It plots the probability of attentive over time, and the shaded regions correspond to standard error. From the first column, we can see that our algorithm in all cases detects human's attentiveness with much higher probably than the passive information gathering technique shown in black. From the second column, we see that our algorithm places significantly lower probability on attentiveness, which is correct because those users were distracted users. These are in line with the statistical analysis, with active information gathering doing a better job estimating the true human internal state. Figure  14  plots the robot trajectories for the active information gathering setting. Similar to Fig.  13 , the solid lines are the mean of robot trajectories and the shaded regions show the standard error. We plot a representative dimension of the robot trajectory (like position or speed) for attentive (dark orange) or distracted (light orange) cases. The active robot probed the user, but ended up taking different actions when the user was attentive versus distracted in order to maintain safety. For example, in Scenario 1, the trajectories show the robot is nudging into the human's lane, but the robot decides to move back to its own lane when the human drivers are distracted (light orange) in order to stay safe. In Scenario 2, the robot brakes in front of the human, but it brakes less when the human is distracted. Finally, in Scenario 3, the robot inches forward, but again it stops when if the human is distracted, and even backs up to make space for her. Figure  15  plots the user trajectories for both active information gathering (first row) and passive information gathering (second row) conditions. We compare the reactions of distracted (light shades) and attentive (dark shades) users. There are large differences directly observable, with user reactions tending to indeed cluster according to their internal state. These differences are much smaller in the passive case (second row, where distracted is light gray and attentive is black). For example, in Scenario 1 and 2, the attentive users (dark orange) keep a larger distance to the car that nudges in front of them or brakes in front of them, while the distracted drivers (light orange) tend to keep a smaller distance. In Scenario 3, the attentive drivers tend to slow down and do not cross the intersection, when the robot actively inches forward. None of these behaviors can be detected clearly in the passive information gathering case (second row). This is the core advantage of active information gathering: the actions are purposefully selected by the robot such that users would behave drastically differently depending on their internal state, clarifying to the robot what this state actually is. Overall, these results support our simulation findings, that our algorithm performs better at estimating the true human internal state by leveraging purposeful information gathering actions. autonomous cars and human-driven vehicles. We formulated a dynamical system in which the robot accounts for how its actions are going to influence those of the human as a simplification to a observable stochastic game. We introduced approximations for optimizing in the dynamical system that bring the robot's computation close to real time (.3 s/time step). We showed in an empirical analysis that when the robot estimates the human model offline, it produces behavior that can purposefully modify the human behavior: merging in front of them to get them to slow down, or pulling back at an intersection to incentivize them to proceed first through. We also showed that these behaviors can emerge out of directly optimizing for the robot's efficiency. We further introduced an online estimation algorithm in which the robot actively uses its actions to gather information about the human model so that it can better plan its own actions. Our analysis again shows coordination strategies arising out of planning in our formulation: the robot nudges into someones's lane to check if the human is paying attention, and only completes the merge if they are; the robot inches forward at an intersection, again to check if the human is paying attention, and proceeds if they are, but backs up to let them through if they are not; the robot slows down slightly and shifts in its lane away from the human driver to check if they want to merge into its lane or not. Importantly, these behaviors change with the human driver style and with the initial conditions-the robot takes different actions in different situations, emphasizing the need to start generating such coordination behavior autonomously rather than relying on hand coded strategies. Even more importantly, the behaviors seem to work when the robot is planning and interacting with real users. Limitations All this work happened in a simple driving simulator. To put this on the road, we will need more emphasis on safety, as well as a longer planning horizon. While performing these experiments, we found the robot's nominal reward function (trading off between safety and reaching a goal) to be insufficient-in some cases it led to getting dangerously close to the human vehicle and even collisions, going off the road, oscillating in the lane due to minor asymmetries in the environment, etc. Figure  20  shows an example of such behavior that comes from the 4way stop domain. For the most part, the car plans to back up to incentivize the human to go through first. But for some values of the human's initial velocity, we observed bad behavior, likely due to convergence to local maxima: the car did not figure out to slow down or back up, and instead if proceeded forward-then it tried to avoid collisions with the person and went off the road, and in the wrong direction (i.e. in the person's way). It seems like while a reward function might be a good enough model for the human, it might be difficult to devise such a universal function for the robot, and the use of hard constraints to ensure safe control would be welcome. Another limitation is that we currently focus on a single human driver. Looking to the interaction among multiple vehicles is not just a computational challenge, but also a modeling one-it is not immediately clear how to formulate the problem when multiple human-driven vehicles are interacting and reacting to each other. Conclusion Despite these limitations, we are encouraged to see autonomous cars generate human-interpretable behaviors through optimization, without relying on hand-coded heuristics. Even though in this work we have focused on modeling the interaction between an autonomous car and a human-driven car, the same framework of underactuated systems can be applied to modeling the interaction between humans and robots in more general settings. We look forward to applications of these ideas beyond autonomous driving, to mobile robots, UAVs, and in general to human-robot interactive scenarios where robot actions can influence human actions. Fig. 2 2 Fig. 2 Features used in IRL for the human driven vehicle; warmer colors correspond to higher reward. We illustrate features corresponding to (a) respecting road boundaries, b holding lanes, and c avoiding collisions with other cars
 Fig. 6 6 Fig. 6 Heat map of reward function for reaching a final goal at the top left of the road. As shown in the figure, the goal position is darker showing more reward for reaching that point
 Fig. 8 8 Fig.8Speed profile and latitude of the human driven vehicle for Scenario 1. The first column shows the speed of all trajectories with its mean and standard errors in the bottom graph. The second column shows the latitude of the vehicle over time; similarly, with the mean and standard errors. The gray trajectories correspond to the control condition, and the orange trajectories correspond to the experimental condition: the robot decides to merge in front of the users and succeeds at slowing them down. The purple plot corresponds to a simulated user that perfectly matches the model that the robot is using
 Fig. 9 Fig. 10 910 Fig. 9 Trajectories of the human driven vehicle for Scenario 2. The first column a shows all the trajectories, and the second column b shows the mean and standard error. Orange (blue) indicates conditions where the reward encouraged the robot to affect the user to go left (right)
 Fig. 12 12 Fig. 12 Legends indicating active/passive robots, attentive/distracted humans, and real user/ideal model used for all following figures
 Fig. 14 14 Fig.14Robot trajectories for each scenario in the active information gathering condition. The robot acts differently when the human is attentive (dark orange) versus when the human is distracted (light orange) due to the trade-off with safety
 Fig. 15 15 Fig.15The user trajectories for each scenario. The gap between attentive and distracted drivers' actions is clear in the active information gathering case (first row)
 Fig.19Actively estimating the human's intent (whether they want to merge in the right lane or not). The robot slows down and shifts slightly away from the person, which would make someone who wants to merge proceed. This could be useful for robots trying to optimize for the good of all drivers (rather than for their selfish reward function)
 Fig. 20 20 Fig. 20 Example of bad local optima occurring for certain initial velocities of the human in the 4way intersection scenario

			 A preliminary version of our results was reported inSadigh et al.  (2016a, b). This paper extends that work by providing more detailed discussion and experiments...
			 Extension to online estimation of the human modelWe have thus far in our approximate solution treated the human's reward function as estimated once, offline. This has worked well in our user study on seeking specific coordination effects on the human, like slowing down or going first through the intersection. But in general, this is bound to run
			 One exception is Nikolaidis et al. (2016) , who propose to solve the full POMDP, albeit for discrete and not continuous state and action spaces.
			 of varying the initial condition (relative y position) in the active merge scenario. The robot adapts to merge when feasible and avoid otherwise. The human is attentive in all cases