The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models
Authors: Cassidy Laidlaw
Anca Dragan
Venue: ICLR 2022 Poster (2022)
Keywords: human model
boltzmann rationality
suboptimality
HRI
human-robot collaboration
generative models
reinforcement learning
deep RL

Abstract: Models of human behavior for prediction and collaboration tend to fall into two categories: ones that learn from large amounts of data via imitation learning, and ones that assume human behavior to be noisily-optimal for some reward function. The former are very useful, but only when it is possible to gather a lot of human data in the target environment and distribution. The advantage of the latter type, which includes Boltzmann rationality, is the ability to make accurate predictions in new environments without extensive data when humans are actually close to optimal. However, these models fail when humans exhibit systematic suboptimality, i.e. when their deviations from optimal behavior are not independent, but instead consistent over time. Our key insight is that systematic suboptimality can be modeled by predicting policies, which couple action choices over time, instead of trajectories. We introduce the Boltzmann policy distribution (BPD), which serves as a prior over human policies and adapts via Bayesian inference to capture systematic deviations by observing human actions during a single episode. The BPD is difficult to compute and represent because policies lie in a high-dimensional continuous space, but we leverage tools from generative and sequence modeling to enable efficient sampling and inference. We show that the BPD enables prediction of human behavior and human-AI collaboration equally as well as imitation learning-based human models while using far less data.

Published as a conference paper at ICLR 2022
THE BOLTZMANN POLICY DISTRIBUTION: ACCOUNTING FOR SYSTEMATIC SUBOPTIMALITY IN HUMAN MODELS
Cassidy Laidlaw University of California, Berkeley
Anca Dragan
University of California, Berkeley
anca@berkeley.ce
ABSTRACT
Models of human behavior for prediction and collaboration tend to fall into two categories: ones that learn from large amounts of data via imitation learning, and ones that assume human behavior to be noisily-optimal for some reward function. The former are very useful, but only when possible to gather a lot of human data in the target environment and distribution. The advantage the latter type, which includes Boltzmann rationality, is the ability to make accurate predictions in new environments without extensive data when humans are actually close to optimal. However, these models fail when humans exhibit systematic suboptimality, i.e. when their deviations from optimal behavior not independent, but instead consistent over time. Our key insight is that systematic suboptimality can be modeled by predicting policies, which couple action choices over time, instead of trajectories. We introduce the Boltzmann policy distribution (BPD). which serves a prior over human policies and adapts via Bayesian inference to capture systematic deviations by observing human actions during a single episode. The BPD difficult to compute and represent because policies lie in a high-dimensional continuous space, but we leverage tools from generative and sequence models to enable efficient sampling and inference. We show that the BPD enables prediction of human behavior and human-Al collaboration equally well as imitation learning-bai based human models while using far less data.
INTRODUCTION
Understanding human preferences, predicting human actions, and collaborating with humans all require models of human behavior. One way of modeling human behavior to postulate intentions, or rewards, and assume that the human will act rationally with regard to those intentions or rewards. However, people rarely perfectly rational; thus, some models relax this assumption include random deviations from optimal decision making. The most common method for modeling humans in this manner, Boltzmann rationality (Luce) 1959 1977 Ziebart al., 2010), predicts that human will act out a trajectory with probability proportional to the exponentiated return they receive for trajectory. Alternatively, some human models eschew rewards and rationality for a data-driven approach. These methods extensive data of humans acting in the target environment to train (typically high capacity) model with imitation learning to predict future human actions (Ding et al. 2011 Mainpric & Berenson, 2013) Koppula & Saxena 2013 Alahi et al. 2016 Ho & Ermon 2016
decision-making,
Ma et al. 2017 S/Smmrrlin et al. 2018 Chai et al. 2019 Wang et al. 2019) Carroll et al.) 2020).
Both these methods for modeling humans successful in many settings but also suffer drawbacks.
Imitation learning has been applied to modeling humans driving (Sun et al.) 20188) arm motion (Ding et al., 2011), and pedestrian navigation (Ma al.) 2017). among other areas. However, it requires a lengthy process of data collection, data cleaning, and feature engineering. Furthermore, policies learned in one environment may not transfer to others, although there is ongoing work to enable models to predict better out-of-distribution (Torrey et al. 2005 Brys et al. 2015) or in many
Our code and pretrained models available ittpa://githmn com cas.iacsssss.. dy 1ayylailo laidlau bo boltzmann-Pe wwtt--ooolccyyeee di

---

Published as a conference paper at ICLR 2022
Day
Day 5
A
rew=4
Day
Route A
50%
Day
Roate A
rew=5
Day
Route A
0%
Day
Rnate A
A B C
Day
???
Prediction: B (72%)
Prediction: A (73%)
rew=1
(b) Boltzmann
(c) Boltzmann policy distribution
(a) Prediction problem
rationality
Figure 1: We propose extension of the Boltzmann rationality human model to distribution over policies instead of trajectories. (a) person walks work each day, choosing route A, B, or C and receiving varying rewards based on route length. Route B has the highest reward but for 4 days observe the person take route A. (b) In this setting, the Boltzmann trajectory model still predicts with high confidence that the person will take route on day 5 (assuming the reward function is known and cannot be updated). (c) In contrast, our model defines a distribution over policies, the Boltzmann policy distribution (BPD), whose PDF is shown here the left. Using the BPD as a prior for the person's policy, we can calculate a posterior over policies after the first 4 days (shown on the right) and predict they will take route A with high probability day 5. In other settings consider, the BPD adapts in matter of minutes to human's 's policy.
environments simultancously (Duan et al. 2017 Singh et al., 2020). On the other hand, reward- models often learn reward in low- now-dimensional space from limited data (Ziebart et al.) 2009
Kuderer et al. 2013: Kretzschmar et al. 2016) and tend to transfer more easily to new environments and be robust to distribution shifts (Sun et al. 2021). For many useful tasks, most elements of the reward function might even be clear a prion; in thoses cases, Boltzmann rationality has enabled successful human-Al collaboration based only on data observed during execution of single task (Bandyopadhyay et al.] 2013), e.g. assisting a person to reach an unknown goal (Dragan & Srinivasa Javddaii al.j (015)). However, Boltzmann rationality fails capture human behavior that is systematically suboptimal. As we show, in some environments this makes it no better at predicting human behavior than a random baseline.
To solve Auman-AI collaboration tasks, some methods seek to bypass the issues of Boltzmann rationality while still avoiding collecting large amounts of human data. These methods train cooperative Al policy which is robust to wide range of potential human behavior via population- based training (Stone et al., 2010 Knott et al. 2021 Strouse et al. 202T), or attempt to find a "natural" cooperative equilibrium, in the work on zero-shot coordination (Hu et al., 20200 Treutlein et al.) 2021). However, such approaches have drawbacks as well. First, they do not define an explicit predictive human model, meaning that they cannot be used to predict human behavior to infer human preferences. Furthermore, population-bal based training methods struggle to include enough diversity in the population transfer to real humans but not much that it is impossible to cooperate. And, while zero-shot coordination has shown promise in cooperative games such as Hanabi (Hu al. (021), it still assumes human rationality and cannot account for suboptimal behavior. Thus, Boltzmann rationality is still widely used despite its shortcomings.
We argue that Boltzmann rationality struggles because it fails capture systematic dependencies between human actions across time. Consider the setting in Figure I Boltzmann rationality predicts the person will take the shortest path, route B, with high probability. Conditioned on the (known) goal of getting to the office, observing the person choose route A gives information about how the person will in the future. Thus, the following day, the probability under the Boltzmann trajectory distribution of choosing route A remains unchanged. Similarly, in the setting of Figure 3 observing person walk to the right around an obstacle gives no information about how they will navigate around it in the future. However, in reality people consistent over time: they tend to take the same action in the same state, and similar actions in similar states, whether because of consistent biases, muscle memory, bounded computational resources, myriad other factors. Such consistency is supported well by behavioral psychology (Funder & Colvin 1991 Sherman et al. 2010).
Our key insight is that systematic suboptimality can be captured by going beyond predicting distribu- tions over trajectories, and coupling action choices over time by predicting distributions over policies instead. Assuming that people will choose policy with probability proportional to its exponentiated
2

---

Published as a conference paper at ICLR 2022
Before deployment
During deployment
h Prior policy distribution
R(s, a)
S -r. Aar 9n1
Reward function
Past human actions in episode
Environment
Posterior policy distribution
Prior policy distribution
Take robot action to maximize expected shared reward posterior
Mla i 9p .n 9,r %.)
Predict next human action
(a) Calculating the Roltzmann policy distribution
(b) Human prediction
(c) Human-Al collaboration
Figure 2: (a) Before deployment, calculate the Boltzmann policy distribution (BPD) as prior human policies by using a model of the environment and the human't 's reward function. Crucially, this step requires human data given that know the human's 's reward function. Even if the objective is not known, but represented in a low capacity class, it can be inferred via IRI. form very limited data in different environments from the target one (Ziebart et 2010p. Note that using high capacity model for the reward would become equivalent to imitation learning (Ho & Ermon 2016) and lose the advantages of Boltzmann rationality. (b) At deployment time, we infer posterior distribution over human's 's policy by updating the BPD prior based on human actions within the episode. This enables adaptive, online prediction of future human behavior. (c) We can also take actions adapted to the posterior distribution in a human-Al collaboration setting.
expected return induces a Boltzmann policy distribution (BPD), which we use as a prior over human policies. Observing person taking actions updates this distribution a posterior via Bayesian inference, enabling better prediction over time and capturing systematic deviations from optimality. overview of the process of calculating and updating the BPD shown in Figure 2 In the example in Figure [ the BPD predicts that the person will continue to take route A even though route B is shorter. Figure the BPD predicts that the person is more likely navigate around the obstacle to the right in the future after watching them do it once.
One could argue that updating the BPD in response observed human actions is no different than training a human model via imitation learning. However, we show that the BPD can give predictive power over the course of single episode lasting less than three minutes similar to that of a behavior cloning-bar based model trained on far more data. Thus, the BPD has both the advantages of both Boltzmann rationality- that it works with very little data as well as imitation learning that it can adapt systematically suboptimal human behavior.
One challenge is that calculating the Boltzmann policy distribution for an environment and using it for inference more difficult than computing the Boltzmann rational trajectory distribution. Whereas the Boltzmann trajectory distribution can be described by a single stochastic policy (the maximum-entropy policy), the BPD is distribution over the space of all policies-a high dimensional, continuous space. Furthermore, the complex dependencies between action distributions at different states in the BPD that make it effective at predicting human behavior also make difficult to calculate.
To approximate the BPD, we parameterize a policy distribution by conditioning on a latent vector-an approach similar generative models for images such VAEs (Kingma & Welling 2014) and GANs (Goodfellow et al. 2014). A policy network takes both the current state and the latent vector and outputs action probabilities. When the latent vector is sampled randomly from a multivariate normal distribution, this induces a distribution over policies that can represent near-arbitrary dependencies across states. See Figure 3 for visualization of how elements of the latent vector can correspond to natural variations in human policies. In Section 2 we describe how to optimize this parameterized distribution to approximate the BPD by minimizing the KL divergence between the two. During deployment, observed human actions correspond to the human's 's policy. We investigate approximating this posterior both explicitly using mean-field variational inference and implicitly using sequence modeling.
be used to calculate a posterior
which latent vectors
We explore the benefits of the Boltzmann policy distribution for human prediction and human-AI collaboration in Overcooked, game where two players cooperate to cook and deliver soup in
3

---

Published as a conference paper at ICLR 2022
Latent variable
21
Observed actions
Posterior over action
Predicted
Latent variahle 21
latent
probabilities
(c) Inference with Boltzmann
(a) Boltzmann policy distribution (BPD)
(b) Inference with BPD
trajectories
Figure 3: Unlike Boltzmann rationality, the Boltzmann policy distribution captures a range of possible human policies and enables deployment-time adaption based observed human actions. In this gridworld, the agent must repetitively travel to the apple tree in the upper right corner, pick an apple, and drop it in the basket in the lower left corner. (a) Using the methods in Section an approximation to the BPD as a policy conditioned on two latent variables 2 and 22. After the distribution is caleulated, z2 controls which direction the agent travels the way the tree and ø1 controls which direction the agent travels on the way back, capturing natural variations in human behavior. (b) After observing 0, 2, and actions the environment, we show the inferred posterior over 2 and the resulting predicted action probabilities. The model quickly adapts to predict which way the person will go around the obstacle in the future. (c) contrast, the Boltzmann trajectory distribution (Boltzmann rationality) cannot adapt its predictions based on observed human behavior.
calculate
kitchen (Carroll et al. 2020). The BPD predicts real human behavior far more accurately than Boltzmann rationality and similarly to a behavior cloned policy, which requires vast amounts of human data in the specific target environment. Furthermore, policy using the BPD human model performs better than one using Boltzmann rationality at collaboration with a simulated human; approaches the performance of collaborative policy based a behavior cloned human model.
Overall, the BPD appears to be more accurate and useful human model than traditional Boltzmann rationality. Further, by exploiting knowledge of the reward function in the task, it achieves perfor- mance par with imitation learning methods that use vastly more data in the target environment. But while our setting the reward function known (i.e. the task objective is clear), this is not always be the case. There are many tasks in which it needs to be inferred from prior data (Christiano al. 2017). or in which reward learning itself the goal (Ng Russell 2000). However, any reward- reward-condee conaadd-oodiiionedd human model like the BPD can be inverted to infer rewards from observed behavior using algorithms like inverse reinforcement learning. We are hopeful that using the (more accurate) BPD model will in turn lead to more accurate reward inference given observed behavior compared to using the traditional Boltzmann model.
THE BOLTZMANN POLICY DISTRIBUTION
We formalize our model of human behavior in the setting of an infinite-horizon Markov decision process (MDP). We assume a human acting in an environment, taking actions E A in states E S. Given a state and action a, timestep t, the next state is reached via Markovian transition probabilities p(3 St, az). We assume that the person is aiming to optimize some reward function R S A RR. This may be learned from data in another environment specified priori; the introduction for discussion. Rewards are accumulated over time with a discount rate E [0, 1).
Let trajectory consist of sequence of states and actions ($1, ......, ST, @r). Let policy be mapping π S A(A) from a state to a distribution over actions taken at that state. The Boltzmann rationality model, or Boltzmann trajectory distribution, states that the probability the human will take
4

---

Published as a conference paper at ICLR 2022
trajectory is proportional to the exponentiated return of the trajectory times "rationality coefficient" or "inverse temperature" B:
PER PBR(T) oc exp { R(s,, an)
This is in fact the distribution over trajectories induced by a particular policy, the maximum-entropy policy, which we call MaxFun (Ziebart et al. 2010). That is,
T-1
( 01, (...........)) = p(s1)
II
p(s St, MT)
(2)
1=1
Say we are interested using the Boltzmann trajectory distribution to predict the action at time t given the current state 8; and all previous states and actions. This can be calculated using 22:
PBR(AL $1, 01 St-1, 41-1, PBR 01,--. ,t, as)
TMaxtin
|s)
(3)
PBR (BR .......... 5t)
The implication of (3 is that, under the Boltzmann trajectory distribution, the action at particular timestep is independent of all previous actions given the current state. That is, we cannot use past behavior to better predict future behavior.
However, argue, humans consistent in their behavior, and thus past actions should predict future actions. We propose to assume that the human is not making a choice over trajectories, but rather over policies in a manner analogous to [
(B)D(T) x exp { (x,) [E=nn' R(s a]]] = exp [B](
(4)
(4) defines the Boltzmann policy distribution (BPD), our main contribution. Note that (xx denotes the expected return of the policy æ. Under the Boltzmann policy distribution, previous actions do help predict future actions:
PaPD [a, $1, aj....., , Q1-1, = (a, st) PDPD $1, ,4-11111 di (5)
That is. previous actions a1..... a at states $1..... 8 induce posterior over policies PBpd (π $1, 01. Cg..... St-1, a,1)). Taking the expectation over this posterior gives the predicted next action probabilities under the BPD.
Both Boltzmann rationality and the BPD assume that the human is near-optimal with high probability, and thus neither may perform well if person very suboptimal. However, as we will show in our experiments, people generally are close enough to optimal for the BPD to be useful model.
Approximating the distribution While have defined the BPD in (... we still need tractable algorithms to sample from it and to calculate the posterior based on observed human actions. This is difficult because the BPD is a distribution over policies in the high dimensional continuous space (A(A)) (8) Furthermore, the density of the BPD defined in (4) is a nonlinear function of the policy and induces dependencies between action probabilities at different states that must be captured.
We propose to approximate the BPD using deep generative models, which have been successful modeling high-dimensional distributions with complex dependencies in the computer vision and graphics literature (Goodfellow et al. [2014] Kingma & Welling 2014 Dinh et al. 2015). In particular, let NO0 I,,) be a random Gaussian vector R. Then we define our approximate policy distribution as
T(. 8) = fo
where fo is a neural network which takes a state s and latent vector 2 and outputs action probabilities. Each value of corresponds to different policy, and the policy depend nearly arbitrarily z via the neural network fe- To sample policy from the distribution, we sample a latent vector (O, In) and use it to calculate the action probabilities of the policy at any state s. Let ge go(m) denote this distribution over policies induced by the network fe- Figure gives an example of policy distribution approximated using latent vectors 2 E R.?.
To predict the human's next action given previous actions using (TT, it suffices to calculate a posterior over 2, i.c.
(e(a: $1, 01 ........ St-1, $ a,-1, s1) ) f f(( qa(2 $1, ........... at-1, dz
(6)
We discuss at the end of this section how to approximate this inference problem.
Optimizing the distribution
While we have shown how represent policy distribution using
5

---

Published as a conference paper at ICLR 2022
0.2
Boltz. traj.
0,3
01
d1
=
0.2
1
Boltz. traj.
[ arl St. Se.
20
G6
02
06
02
08
02
10
0
05
G3
Gg
45
20 0
10
20
0.0
t 8
Figure 4: A visualization of the Boltzmann tra- jectory and Boltzmann policy distributions state. The density of the policy distribution is shown over hexagon; each point corresponds to a distribution over the six actions. For small (on the left), the policies are close to determin- istic: here policies almost always take either 42, 04, or ag. but rarely randomize much between them. For large (the middle), policies tend to be random across all actions. The Boltzmann trajectory distribution corresponds to single point at the MaxEnt policy.
Figure 5: A visualization of the mutual infor- mation (aka information gain) / [ dx St, St) between actions taken at different timesteps and t' given the states at those timesteps under the Boltzmann policy distribution and Boltzmann trajectory distribution (Boltzmann rationality). Unlike Boltzmann rationality, the BPD allows actions taken in the past to help predict actions taken in the future. The information gain is higher the lower the parameter since more deterministic policies are more predictable.
deep generative model, still need to ensure that this policy distribution is close the BPD that we are trying to approximate. To do so, we minimize the KL divergence between the approximation to (T)) and (TT):
0 = arg min (ge(g)(T)) || (BPD(T))
To optimize
expand the KL divergence:
0* arg min - En (m) (PD + (x) llg
= arg max E (x) [3](T)] + (())(TT)
(8)
Here, H(q)(T)) denotes the entropy of the generative model distribution. Note that although the density (T) includes a normalization term, it is a constant with respect to 0 and thus can be ignored for the purpose of optimization. Equation (8) has intuitive interpretation: maximize the expected return of policies sampled from qe (T), but also make the policies diverse (i.e., the policy distribution should be high entropy). The first term can be optimized using reinforcement learning algorithm. However, the second term is intractable to even calculate directly, let alone optimize; entropy estimation is known to be very difficult high dimensional spaces (Han et 2019).
To optimize the entropy of the policy distribution, we rewrite it as a KL divergence and then optimize that KL divergence using discriminator. Let μ(²) denote the base measure with which the density of the BPD is defined in (4). Let Powe (T) = /(TT// du(()), i.e. (u(T) normalized to probability measure. Then the entropy of the policy distribution (T) equal to its negative KL divergence
from Phose (T) plus constant:
(()o(=))= = log (fdu(T)) DRL. (go
|| PPasse (T))
We include full derivation of (9) in Appendix B. While replaces the intractable entropy term with a KI. divergence, we still need to optimize the divergence. This is difficult as we only have cannot compute the divergence in closed form. Instead,
access samples from (T)),, an adversarially trained discriminator to approximate the KI. divergence. The discriminator d assigns score d(()) E to any policy æ. is trained to assign low scores to policies drawn from the base distribution and high scores to policies drawn from (T(::
d arg min Ex) log (1 exp (-d(T)}) Em log (1 exp [d(+)))
(10)
Huszár (017)) show that if (10) is minimized, then DKL. (go(ga(T) Phase (T) Ex [d(T))
That the expectation of the discriminator scores for policies drawn from the distribution go (T)
approximates its KL divergence from the base distribution Prase ( Similar to the training of GANs.
the policy network fo attempts to "fool" the discriminator by making more difficult to tell its

---

Published as a conference paper at ICLR 2022
policies apart from those in the base distribution, thus increasing the entropy of go (
Putting all together, the final optimization objective is
o" arg max Be (x)]g(())83366 d(())]
(11)
where d is optimized in 10 We optimize 0 in 11 by gradient descent. Specifically, at each iteration of optimization, we approximate the gradient of the expectation in (IT) with a Monte Carlo sample of several policies Ii...... TM qo (T)) The gradient of the discriminator score d(x,)) can be calculated by backpropagation through the discriminator network. The gradient of the policy return (Tts) is approximated by a policy gradient algorithm, PPO (Schulman al. 2017
Choice of base measure An additional parameter to the BPD is the choice of base measure (u(=) and the derived base distribution Phase (T) - p(xx// J du(=)). These define the initial weighting of policies before considering the reward function. In our experiments, we define the base measure as a product of independent Dirichlet distributions each state:
(T) Phase (T) II P.((( ss)
Ps ind. Dir(a,..... a) AsES
(12)
sES
The concentration parameter a used in the Dirichlet distribution controls the resulting distribution
over policies; we explore the effect of a in Section 2 Figures[4and and Appendix C.2
Parameterization of policy and discriminator We modify existing policy network architectures to use as fo(8, z) by adding an attention mechanism over the latent vector z. We use a transformer (Vaswani 2017) neural network as the discriminator. See Appendix A.2 for more details.
Online prediction with the Boltzmann policy distribution Predicting actions with the BPD requires solving an inference problem. In particular, we need to solve (6), predicting the next action by marginalizing over the posterior distribution of latent vectors z. We explore approximating this inference both explicitly and implicitly. First, we approximate the posterior over 2 as product of independent Gaussians using mean-field variational inference (MFVI, see Appendix A (Blei) et al. 2017). This approach, while providing an explicit approximation of the posterior, is slow and cannot represent posteriors which are multimodal or have dependencies between the components of z. Our second approach ignores the posterior and instead directly approximates the online prediction problem (a(ar $1, ai..... st). To do this, generate thousands of trajectories from the BPD by sampling a latent z for each and rolling out the resulting policy in the environment. Then, we train a transformer (Vaswani et al. 2017) these trajectories to predict the next action a, given the current state 8, and all previous actions. This avoids the need to compute a posterior over 2 explicitly.
Exploring the Boltzmann policy distribution Using the methods described above, we compute approximate Boltzmann policy distribu- tions for three layouts in Overcooked: Cramped Room, Coordination Ring, and Forced Coordination. A visualization of the resulting policy distribution for Cramped Room at one state can be seen in Figure [4] which highlights the effect of the parameter a. We also visualize the BPD for a simple gridworld environment in Figure a). This figure shows that components of the latent vector z capture natural variation in potential human policies.
Cross entropy
0.2
0.1
0.5
1.0
Consistency
In Figure 5] we show that, unlike Boltzmann distribution over tra- jectories, the BPD allows actions in previous states to be predictive of actions in future states. The predictive power of past actions is higher for lower values a, which also motivates our choice of a = 0.2 for the remaining experiments. Figure & shows that the BPD use the dependence between actions over time to predict systematically suboptimal actions in Overcooked.
BR
BPD
Figure 6: Cross entropy (lower is better) of Boltz- mann rationality (BR) and the BPD on synthetic hu- man data in the apple pick- ing gridworld from Figure B
EXPERIMENTS
See Section 3 for details.
We evaluate the Boltzmann policy distribution in three settings: predict- ing simulated human behavior in a simple gridworld, predicting real human behavior in Overcooked, and enabling human-Al collaboration
7

---

Published as a conference paper at ICLR 2022
Cramped Room
Forced Coordination
Coordination Ring
Data-intensive approach
Behavior cloning
Naive approaches
Cross entropy
2
2
2
Random prediction
Sclf-play policy
Reward-conditioned human models
Boltzmann rationality
BPD (MFVI)
BPD (transforme)
Figure 7: Prediction performance (cross-entropy, lower is better) of various human models on real human data for three Overcooked layouts. Error bars show 95% confidence intervals for the mean across trajectories. See Section 3 for details and discussion.
BPD: (25%))
(45%)
(66%)
BPD:
(52%)
(45%)
BPD:
(34%)
(64%)
(71%)
BR:
(31%)
(31%)
(31%)
BR:
(21%)
(21%)
BR:
(32%)
(32%)
(32%)
(a) Cramped Room: optimal
Forced Coordination: the
(c)
Coordination Ring: the
action is to move to get
optimal action is to turn to the left to pick up an onion, but the human always moves down instcad.
human consistently tries to interact with the pot alrcady full of soup to put in another onion.
onion, but the human repcatedly goes to the left.
Figure 8: The Boltzmann policy distribution predicts real human behavior in Overcooked better than Boltzmann rationality (BR) by adapting systematically suboptimal humans. In each of the three layouts, the shown state is reached multiple times a single episode. The human repeatedly takes the suboptimal action overlaid the game. The predictions of the BPD and BR for each timestep when the state is reached are shown below the game, ignoring "stay" actions (see Appendix C.3 BPD adapts to the human's consistent suboptimality while BR continues predict an incorrect action. Note that actions at one state can also help predict actions at different states; see Figure
in Overcooked. The full details of experimental setup in Appendix A We provide further results and ablations in Appendix
Simulated human data First, investigate if the BPD predict synthetic human trajectories better than Boltzmann rationality in the apple picking gridworld shown in Figure In this environ- ment, the human must travel back and forth around an obstacle pick apples and bring them to basket. To generate simulated human data, we assume the human has usual direction to travel around the obstacle on the way to the tree and usual direction to travel on the way back. We assign different simulated humans "consistency" values from 0.5 to 1, and these control how often each human takes their usual direction. consistency of 0.5 means the human always picks random direction, while consistency of means the human always takes their usual direction around the obstacle.
The mean cross-entropy assigned to trajectories sampled from these simulated humans by the BPD and Boltzmann rationality at each consistency value is shown Figure with the shaded region indicating the standard deviation. As expected, the BPD is particularly good at predicting human behavior for the most consistent humans. However, even does better than Boltzmann rationality predicting the humans which randomly choose direction proceed around the obstacle each time. This because our simulated trajectories always go straight to the tree and back to the basket. Boltzmann rationality gives chance for the human to reverse direction, but the BPD picks up the consistent behavior and gives it higher probability, leading to lower cross-entropy.
Human prediction in Overcooked Next, we evaluate the predictive power of the Boltzmann
8

---

Published as a conference paper at ICLR 2022
Mean return w/ human proxy
Cramped Room
Forced Coordination
Coordination Ring Data-intensive approaches
150
Behavior cloning
60
100
Human-aware RL
75
Naive approach
100
40
Self-play
50
Human model best responses
50
20
Boltzmann rationality
25
BPD (no memory)
BPD (memory)
Figure Performance of various "robot" policies at collaborating with behaviorally cloned "human proxy" policy on three Overcooked layouts. Error bars show 95% confidence intervals for the mean return over trajectories. See Section 3 for details and discussion.
policy distribution on trajectories of real Hamman-with-hune play collected by Carroll et al. (2020) in Overcooked. As in the gridworld, we evaluate online prediction: given human's 's trajectory until time - 1, predict the action the human will take at time t.
We compare three approaches to the BPD for online prediction Overcooked. First, we train a behavior cloning (BC) model set of human trajectories separate from those evaluated This model allows us to compare a data-intense prediction method to our approach which adapts over the course of single trajectory (less than 3 minutes of real time). Second, we train an optimal policy using reinforcement learning (PPO) with self-play. We expect this policy to have low predictive power because humans tend to be suboptimal. Third, calculate Boltzmann rational policy.
The predictive performance of all models is shown in Figure 7 The self-play and Boltzmann rational models are both no more accurate than random at predicting human behavior in all three Overcooked layouts. In contrast, the BPD predicts human behavior nearly as well as the behavior cloned model. As expected, the MFVI approach to prediction with the BPD performs worse because its representation of the posterior over is explicitly restricted. These results validate the BPD as an accurate reward-condin conditiond human model.
Human-AI collaboration Predicting human behavior is not an end in itself; one of the chief goals of human model research to enable AI agents collaborate with humans. Thus, we also evaluate the utility of the Boltzmann policy distribution for human-Al collaboration. The Overcooked environment provides a natural setting to explore collaboration since human-Al collaborative performance, we pair various collaborative "robot" policies with simulated "human proxy" policy, which is trained with behavior cloning on held-out human data. Carroll et al. (2020) found that performance this proxy correlated well with real human-A performance.
two-player game. test
We compare several collaborative policies based on similar approaches to the prediction models. First, evaluate the behavior cloned (BC) model. Since and the human proxy are both trained from human data, this approximates the performance of a human-humaa team. Second, we train human-awart RL policy (Carroll al., 2020), which is a best-response the BC policy. This approach requires extensive human data to train the BC policy. We also evaluate the self-play optimal policy; this policy optimized for collaboration with itself and not with any human model.
Next, we train collaborative policies based on both the Boltzmann rational and BPD models. During each episode of training, the collaborative policy paired with simulated human policy from the human model the MaxEnt policy for Boltzmann rationality, or a randomly sampled policy from the BPD. The human model's s actions are treated as part of the dynamics of the environment; this allows the collaborative policy to be trained with PPO as in single-agent environment. This approach is similar to human-aware RL, but we train with reward-conditioned human models instead of behavior cloned ones. Over the course of training, each collaborative policy maximizes the expected return of the Auman-AI pair, assuming the human takes actions as predicted by the respective human model.
Because human actions are correlated across time under the BPD, train BPD-based col- laborative policies both with and without memory. The policies with memory have the form Ta $1, aft : ...., St-1. a.- st); that is, they choose robot action am based both the current state 8, and also human actions off taken prior states. This allows adaption to the human't 's policy.

---

Published as a conference paper at ICLR 2022
Figure 9 shows the results of evaluating all the collaborative policies with human proxies the three Overcooked layouts. In all layouts, the human-aware RL policy does better than either a self-play policy or the BC policy, matching the results from Carroll et al. (2020). The collaborative policies based on Boltzmann rationality tend to perform poorly, which is unsurprising given its weak predictive power. The collaborative policies based on the BPD perform consistently better across all three layouts. Surprisingly, the BPD-based collaborative policies with memory do not perform much differently than those without memory. This could be due more the difficulty of training such policies than to the inability for adaption using the BPD.
4 CONCLUSION AND FUTURE WORK
We have introduced the Boltzmann policy distribution, alternative reward-conditione human model to Boltzmann rationality. Our novel model is conceptually elegant, encompasses systematically suboptimal human behavior, and allows adaption human policies over short episodes. While focused on human prediction and human-Al collaboration, reward-conditioned human models are also used for reward learning, by inverting the model to infer rewards from behavior. Our results have important implications for reward learning: if Boltzmann rationality is unable to explain human behavior when the reward is known, it will impossible infer the correct reward given only observed behavior. Since the BPD better predicts human behavior in our experiments, it should lead better reward learning algorithms as well. We leave further investigation to future work.
ACKNOWLEDGMENTS
We would like to thank Jacob Steinhardt, Kush Bhatia, and Adam Gleave for feedback on drafts, and the ICLR reviewers for helping us improve the clarity of the paper. This research supported by the ONR Young Investigator Program and the NSF National Roboties Initiative. Cassidy Laidlaw is supported by National Defense Science and Engineering Graduate (NDSEG) Fellowship.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and S. Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016. doi: 10. 10.1109/CVPR.201 10.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv: 1409.440444 0473 [cx, stat], May 2016. URL hwt.hhttp//aaee org/abs /10bs/14440 1409.44440 0473
arXiv: 1409.0473.
Tirthankar Bandyopadhyay. Kok Sung Won, Emilio Frazzoli, David Hsu, Wee Sun Lee, and Daniela Rus. Intention-Aware Motion Planning. In Emilio Frazzoli, Tomas Lozano-Perez, Nicholas Roy, and Daniela Rus (eds.), Algorithmic Foundations of Robotics X, Springer Tracts in Advanced Robotics, pp. 475-491, Berlin, Heidelberg, 2013. Springer. ISBN 978-3-642-3627988 doi: (-0-778---642--26777 29.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Re- view for Statisticians. Journal of the American Statistical Association, 112(518):59-887) April 2017. ISSN 0152-1459. doi: 10. 10.100000162145 45..2 URL https::/
foo..org//// 11..or/g.............. 1080 00115166211519 2017. 1285773 Publisher: Taylor & Francis .eprint:
http::/dbol.ogg111..ooooooooooea http::/0abaaagg///00000011111222
Tim Brys, Anna Harutyunyan, Matthew E. Taylor, and Ann Nowé. Policy Transfer using Reward Shaping. In Proceedings of the 2015 International Conference Autonomous Agents and Multiagent Systems, AAMAS 15,, pp. 181-188, Richland, SC, May 2015. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978- 1--50--3413-4
Micah Carroll, Rohin Shah, Mark K. Ho. Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anea Dragan. On the Utility of Learning about Humans for Human-Al Coordination. 05789 [cs. stat), January 2020. URL httt::/aaxxv..oto wwwp::iwww.raaa////ee.eeaae 05789 arXiv: 1910.05789.
10

---

Published as a conference paper at ICLR 2022
Yuning Chai, Benjamin Sapp, M. Bansal, and Dragomir Anguelov. MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction. CoRL, 2019.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep 03741 [cs, stat], July 2017. URL arXiv: 1706.03741.
reinforcement learning from human preferences.
abs lattp://arxi>: org/ abs 1706...00 03741
Hao Ding, Reissig, Kurniawan Wijaya, D. Bortot, K. Bengler, and O. Stursberg. Human arm motion Saaa---oooooottteeee
modeling and long-term prediction for safe and efficient International Conference on Robotics and Automation, 2011. doi: 10. 10.109/ICRA2201..999222
2011 IEEE
Compooen Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components
Estimation. arXiv:1410.8516 [cs], April 2015. arXiv: 1410.8516.
Anca D Dragan and Siddhartha S Srinivasa. A policy-blending formalism for shared control. The International Journal of Robotics Research, 32(7):790- June 2013. ISSN 0278-3649. doi: 10.1177/02783649 13490324. /ttp////////////.o///////0/111/1
Publisher: SAGE Publications Ltd STM.
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-Shot Imitation Learning. /003v00003.01 [cs], December 2017. URL
arXiv: 1703.07326.
David C. Funder and C. Randall Colvin. Explorations in behavioral consistency: Properties of persons, situations, and behaviors. Journal of Personality and Social Psychology, 60(5):773-794 1991. ISSN 1939-1315. doi: 10.100370002--3544.6000 Psychological Association.
Place: US Publisher: American
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. arXiv preprins 2661, 2014.
Yanjun Han, Jiantao Jiao, Tsachy Weissman, and Yihong Wu. Optimal rates of entropy estimation over Lipschitz balls. 171.0:1111.02111 [cs, math, stat), November 2019. URL httt:://arxy
org/abs 1111 02141
arXiv: 1711.02141.
Jonathan Ho and Stefano Ermon.
Generative Adversarial Imitation Learning. In Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
wwwpeeiiiiinnnnn..iieeeeeeeeeeee cc 6c12287755ccce52221174333555 atst://sttttt--tatttttttttttt. wttppppp///11.cccccc petperr/201/hhaae 6/hash/
URL https: http::::://///ooeeennsssss
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):
1735-1780, 1997. Publisher: MIT Press.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play" for Coordination. In International Conference on Machine Learning, pp. 4399-4410, PMLR, 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off- Belief Learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 4369 4379. PMLR, July 2021. URL hat.https... http://ptoceciceeeccooooc http.///..pppppppeeeeeeiiii html
ml /
ISSN: 2640-3498.
Ferenc Huszár. Variational Inference using Implicit Distributions. 1702 [es, stat], February 2017. URI.
08235
arXiv: 1702.08235.
Shervin Javdani, Siddhartha S. Srinivasa, and J. Andrew Bagnell. Shared Autonomy via Hindsight Optimization. arXiv: MoXivv/5033000 /50.00769 [cs], April 2015. URL. http://axxiv..ogaaa////ss/a/ arXiv: 1503.07619.
07619 9
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, arXiv preprint arXiv: 22622,90, 2014.
11

---

Published as a conference paper at ICLR 2022
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. 131fv11312.6114 [cs, stat], arXiv: 1312.6114.
May 2014. URL htLh:tt:/// htt.:tt::pppa...crottta http://awww.....aa/a/aaa... 6114
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, A. D. Dragan, and Rohin Shah. Evaluating the Robustness of Collaborative Agents. arXiv:2101.05500 [cs], January 2021.
URL http::tttt//aat org/ http////s//110 05507 arXiv: 2101.05507.
H. Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013. doi: 10.1109/IROS. 2013.6696634.
Henrik Kretzschmar, Markus Spies, C. Sprunk, and W. Burgard. Socially compliant mo- bile robot navigation via inverse reinforcement learning. Int. J. Robotics Res., 2016. doi:
10.1 00.1177/027334491551197777
M. Kuderer, S. Gulati, and W. Burgard. Learning driving styles for autonomous vehicles from demonstration. 2015 IEEE International Conference Robotics and Automation (ICRA), 2015. doi: 12110////R...1.....7...
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for Distributed Reinforcement Learning. arXiv: 1712.09381 [cs]. June 2018. URL htl.:/cce///rrvv...r//////1111.0
arXiv: 1712.09381.
R. Duncan Luce. Individual choice behavior. 1959. Publisher: John Wiley.
R. Duncan Luce. The Choice Axiom After Twenty Years. Journal of Mathematiial Psychology, 15 (3):215-233, June 1977. ISSN 0022-2496. doi: 10.00166002--29444779002223 |RLhttps
//www. www.sssiieeeeiirreeeo wec.t.eiieeee..e.. com htt.a//ww.........o/ooo../.o./a. 79000323
Wei-Chiu Ma, De-An Huang, Namhoon Lee, and Kris M. Kitani. Forecasting Interactive Dynamies of Pedestrians with Fictitious Play. 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2017. doi: N0.1109/CVPR.2007.4443
Jim Mainprice and D. Berenson. Human-robot collaborative manipulation planning using early prediction of human motion. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013. doi: 10. 109/I0OOOOO.. 200.0O....1..6666 3666.666.36..
Andrew Y. Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. In ICML, volume 1, pp. 2, 2000.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py PyTorch: An Imperative Style, Performeroom Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL httpt:pps.//pp.oceeeeeie neur cttp:r/ppppppppppp////11//9/
html
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.064 [cs]. January 2016. URL
httt:://arxiv wwtp::xwivwvg//..boooo
org/ abs 06434 arXiv: 1511.06434.
E. Schmerling, Karen Leung. Wolf Vollprecht, and M. Pavone. Multimodal Probabilistic Model- Based Planning for Human-Rod Robot Interaction. 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1 10.1109/CRA.2018..8 0011000....666....
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.0000
2017.
Ryne A. Sherman, Christopher S. Nave, and David C. Funder. Situational similarity and personality predict behavioral consistency. Journal of Personality and Social Psychology, 99(2):330-3 2010. ISSN 1939-1315. doi: 10.1037/a0019796. Place: US Publisher: American Psychological Association.
12

---

Published as a conference paper at ICLR 2022
Avi Singh, Eric Jang, Alexander Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levinev, Mohi Khansari, and Chelsea Finn. Scalable Multi-Task Imitation Learning with Autonomous Improve- ment. In 2020 JEEE International Conference Robotics and Automation (ICRA), pp. 2167-2173, May 2020. doi: 10. 200.///
ISSN: 2577-087X.
Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad Hoc Autonomous Agent Teams: Collaboration without Pre- Pre-Coordination. volume 3, January 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Col- laborating with Humans without Human Data. In Advances in Neural Informa- tion Processing Systems, volume 34, pp. 14502-14515. Curran Associates, Inc., 2021.
URL bttps://proceediin nwwrips httppappr/2222////// html
Liting Sun, W. Zhan, and M. Tomizuka. Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning. 2018 2/st International Conference on Intelligent Transportation Systems (ITSC), 2018. doi: 10.1 BLL1099ITSC.221188566644
Liting Sun, Xiaogang Jia, and A. Dragan. On complementing end-to-end human behavior predictors with planning. Robotics: Science and Systems, 2021. doi: 10.15607/RSS 2021.XI1.037.
Lisa Torrey, Trevor Walker, Jude Shavlik, and Richard Maclin. Using advice to transfer knowledge acquired in one reinforcement learning task to another. In Proceedings the 16th European conference on Machine Learning, ECML '05, pp. $12-424. Berlin, Heidelberg, October 2005. URL https://do3
Springer- Verlag. ISBN 978-3-540-22244-22 doi: 10. 1007/1 1564096_40
6xc/100.0007111564653
Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A New Formal- ism, Method and Open Issues for Zero-Shor Coordination. Proceedings of the 38th Interna- tional Conference on Machine Leurning, pp. 10413-10423. PMLR, July 2021. URL /tthtt ISSN: 2640-3498.
//proceedings... mlr
https//ww////tttteeeeeeeee html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv: 100::776.000 03762 [cs], December 2017.
URL http::ttt///aat
org abs/ 03762
arXiv: 1706.03762.
Borui Wang, Ehsan Adeli, Hsu-kuang Chiu, De-An Huang, and Juan Carlos Niebles. Im-
itation Learning for Human Pose Prediction. pp. 7124-7133, 2019. URL https
com content_iccty -CC_ CCt_c______eeelllm__ Maitemm//aanggg /Wang_ Imitation_ Learning_for for_ Hwmoor____aa___ao BaciiiaaanPPseeeeeee PowpaaaooooePPecccoooooooCCC (BCCC__201__00 2wwee_111e.______ee1____ hhmpl
dlopeaacces..thede
Brian D. Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J. Andrew Bagnell, Martial Hebert, Anind K. Dey, and Siddhartha Srinivasa. Planning-based prediction for pedestrians. In 2009 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3931-3936, October 2009. doi: 10.1 N0.1109//ROS.2009 200.......00955114477777 ISSN: 2153-0866.
Brian D. Ziebart, Andrew Bagnell, and Anind Dey. Modeling interaction via the principle of maximum causal entropy. 2010. Publisher: Carnegie Mellon University.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A EXPERIMENT DETAILS
Here, we give further details about our experimental setup, hyperparameters, and network architec- tures. We implement the calculation of the BPD and all collaborative training using RLlib (Liang) ct al. 2018 and PyTorch (Paszke et al. 2019). We use RLlib's PPO implementation with the hyperparameters given in [
Gridworld environment We implemented the apple-picking gridworld used in Section 3 and Figures and 6 The Boltzmann rational (MaxEnt) policy is calculated with soft value iteration. The Boltzmann policy distribution is calculated using the methods described in Section 2
Overcooked environment We use the implementation of Overcooked from Carroll et al. (2220). We also use the human data they collected; the train set is used for training the BC policy and the test set used for training the human proxy policy and evaluating all predictive models. The implementation of Overcooked has changed slightly since the data was collected: players now have use an "interact" action start cooking soup. Previously, soup automatically started cooking. We insert"interact actions in each episode where the players would have had to start cooking the soup to make up for this.
Overcooked is a two-player game while our formalism underlying the BPD applies to single agent environments. However, since Overcooked is perfectly cooperative, the two-player game equivalent to single-player game where the agent picks two actions at each timestep independently. Thus, except for when training collaborative policies, we use a single policy network for both players.
We use episode length of 400 for most experiments; this corresponds to about minute of real- weme-time play. For human prediction, we use the episode length of the human-human data, which around 1200 for each episode (about 3 minutes).
Additionally, we implement one small change to the Overcooked environment that we find speeds up reinforcement learning. The environment includes an auxiliary reward with better shaping whose weight is annealed to zero over the course of training. In the original implementation, this auxiliary reward is only given to the agent which takes particular action, like placing an onion in pot. We modify this to give the auxiliary reward to both agents such that the environment is fully cooperative. This particularly improves training Forced Coordination, where one agent is otherwise unable to receive any auxiliary reward.
We anneal the auxiliary reward to zero over 2.5 million timesteps, except when calculating the Boltzmann policy distribution. In that case, we anneal over 20 million timesteps because estimating the policy gradient over distribution of policies results in slower training.
Calculating the Boltzmann policy distribution To calculate the BPD, we use the methods described in Section 2 We set the temperature 1/3 = 0.1 for the BPD well as the Boltzmann trajectory distribution, as we found that this is the maximum that does not lead to near-zero return. We set the concentration parameter of the Dirichlet base distribution described in 12 We also explore using = in Figures and and give quantitative results in Appendix
0.2.
We use latent vectors of dimension 1,000 for Overcooked. We also tried using a vector of dimension 100, but found this did not give enough flexibility to capture human behavior; see Appendix C.1 We use the same latent vector during each episode for both players.
Since our method for approximating the BPD is similar to GAN training, we find it is prone to instability. Following RRafford al. (2016), we set the Adam momentum term Bn 0.5 when calculating the BPD to reduce this instability.
Overall, we generally found that the BPD required a larger-than-usual RL batch size compared to training single policy, since the BPD is essentially an infinite population of policies. However, besides this, we were able to use the hyperparameters, network architectures, and reward shaping largely unchanged from the original Overcooked paper. Thus, while the BPD is somewhat more computationally demanding than typical RL methods, we did not find it needed much additional hyperparameter tuning.
14

---

Published as a conference paper at ICLR 2022
Approximate inference with mean-field variational inference BPD requires computing a posterior distribution over latent vectors & conditioned on sequence of past states and actions:
Online action prediction with the
go(= 81, 41,-........ -, 4T-1)
(13)
We approximate this inference problem with mean-field variational inference (Blei et al. 2017) We approximate the posterior by a parameterized distribution qu.o which is product of independent Gaussians:
qu.a (2) =
II
2, j4s
(14)
O2
Here, o6-) is the density of a standard normal distribution and u E IK.. We minimize the KL divergence between (2) and the true posterior by maximizing the expectation lower bound (ELBO):
-T-1, aT 12)] -DL (N((,,)) E Ez)gn. log qo "1,..,,,,.
max
2.0
max E. Egngge (2)
Σ log Solae 81s 2),22
Σ
(NL(N( (O),))
(15)
:=1
That is, we optimize and both (i) to maximize the sum of the probaabiiiiisss assigned each observed action by policies using latent vectors z ~ (p)r(2), and (ii) to minimize KL divergence to the prior over We use SGD to optimize (15) by estimating the expectation with a Monte Carlo draw of z values and calculating the KI. divergence in closed form.
Once the posterior has been approximated by maximizing 115). probabilities by marginalizing over the posterior:
estimate the next action
(a $1 .i........ ST-1. QT-1. ST)
ST, 2)]
We approximate this expectation via Monte Carlo sampling.
For repeated online prediction over a long episode, performing the optimization in 1.5) from scratch at every timestep is computationally expensive. We speed up inference by using the y and values from the previous timestep to initialize optimization at the following timestep. Using this initialization, we find we only need one SGD iteration per timestep to effectively optimize (15).
Training the predictive sequence models online action prediction problem
As describe in Section
we also approximate the
(16) with the BPD by training transformer rollouts from the distribution. Specifically, sample 50,000 policies from the BPD and roll out a trajectory for each policy. Then, we train both a trans- former and an LSTM (Hochreiter & Schmidhuber] 1997) to predict (16) over this set of trajectories
go(a; 81, 11,,0.,..1111.... Q4-1,
by standard supervised learning. We describe the specific training hyperparameters in Table and architectures in Appendix(A.2)
Behavior cloning We use similar behavior cloning (BC) procedure to Carroll et al. (2020). BC policies are trained minimize the cross entropy over human trajectories with Adam (Kingma Ba 2014). We lower the learning rate by a factor ten if the cross entropy has not meaningtully improved epochs. We regularly evaluate the BC policies playing with themselves in Overcooked. We chose an iteration to stop training for each layout based on when the policies perform best. We use the manually designed features from Carroll et al. (2020) as input to the BC policy network. They also hardcode their BC policies to take random action when stuck in the same state for too long; do not do this.
Training the collaborative policies As described in Section 3 train collaborative policies as best response to the BC policy (i.e., human-aware RL), the Boltzmann rational (MaxEnt) policy, and the BPD. We train the collaborative policies using PPO for one player while using fixed policy for the other player.
Since the BPD enables better prediction of future behavior given past behavior, tried training policy with memory to cooperate with policies sampled from BPD. This policy takes as input not only the current state, but also previous states and the human actions those states:
(aa 81, a
8t (t)
15

---

Published as a conference paper at ICLR 2022
We found that directly trying to optimize such policy with PPO did not lead good performance. Instead, we use the hidden state of the LSTM network trained for online prediction as an extra imput to the policy network (in addition to the state). We hypothesize that the LSTM's hidden state already contains all necessary information to adapt to the human's policy, so we do not need to further train the LSTM during policy optimization. We find that this leads to much more stable training. However, learning policy with memory is still slower, so we use more training iterations of PPO (see Table [
Evaluation collaborative performance We evaluate the performance of collaborative policies with a human proxy policy behavior-cloned from held-out data. Since each Overcooked layout has different starting positions for the two players, we roll out 1,000 episodes with each starting position permutation and average all the returns.
A.1 HYPERPARAMETERS
PPO Hyperparameter
Value (Gridworld)
Value (Overcooked)
Training iterations
500
500
(policies with memory)
1,000
Batch size
2,000
100,000
SGD minibatch size SGD epochs per iteration
2,000
8,000
Optimizer
Adam
Adam
Learning rate
10 3
10-3 3
Gradient clipping
0.1
0.1
Discount rate (7)
0.9
0.99
GAE coefficient (A)
0.98
0.98
Entropy coefficient
0
KL target
0.01
0.01
Clipping parameter (c)
0.05
0.05
Table PPO hyperparameters.
Hyperparameter
Value (Overcooked)
Training epochs
Batch size (num. episodes) Optimizer
40
Adam
Learning rate
10* 10-3
Table 2: Sequence model training hyperparameters.
BC Hyperparameter
Value (Overcooked)
Training epochs
(Cramped Room)
500
(Forced Coordination)
150
(Coordination Ring)
250
SGD minibatch size
64
Optimizer
Adam
Initial learning rate
10
Table 3: Behavior cloning hyperparameters.
A.2 NETWORK ARCHITECTURES
Here, we describe the architectures used for policy networks, the discriminator from Section 2 and sequence models from Section 3
16

---

Published as a conference paper at ICLR 2022
Behavior cloning For our behavior cloned policies in Overcooked, use multilayer perceptron (MLP) with two hidden layers of size 64.
Overcooked policies For all other policies in Overcooked, input the state as two-dimensional grid with 26 channels corresponding to various features in the environment, e.g. counters, players, etc. These states first processed through convolutional neural network with three layers of 25 hidden units each and filter sizes of (5, 5), (3. 3), and (3, 3) respectively. Then, the resulting activations are flattened and passed through three fully-connected layers with 64 hidden units. We use leaky ReLUs with negative slope -0.01 for all activation functions.
Conditional policy network The conditional policy network [o(s, used approximate the BPD additionally takes in as input E R", the random latent vector. We found that directly concatenating to the state representation or the input of the first fully-connected layer led slow and unstable training. Instead, we used an attention mechanism (Bahdanau et al., (016) over the latent vector. Let a be the activations after the first fully- fully-connected layer. We transform by a fully connected layer to matrix of size x n. and then take the softmax over each row such that the elements add to This gives a matrix W, our attention weights. Then, we concatenate Wa to a and feed the resulting vector through the remainder of the network. This allows the network to attend to arbitrary combinations of the values in & and improves training considerably. We set m = 4 for the gridworld experiments and m 10 for Overcooked.
Discriminator We give as input to the discriminator network d a sequence of states and actions $1; .1....... ..... Sk; Og- We let k 10, finding that using more actions and states leads to instability because it is too easy for the discriminator to tell apart policies from the base distribution and the approximated BPD.
The actions at each state are one-hot encoded and appended to the state representation. Then, for Overcooked, the states are passed through convolutional state encoder, which creates a separate representation for each state-action pair. Next, the resulting representations are passed through three-layer transformer with one head and dmodel 64. This allows the discriminator to pool information about the policy across states. Finally, the outputs of the transformer are averaged and fed through a final linear layer produce the discriminator score of the policy.
Sequence models for prediction As described above, we train both LSTMs and transformers to estimate (16). i.e. to predict the next human action ar given the current state s, and past states and actions 81, .i,,..,,. -T--1 aT 1. As input to these sequence models, we give a tuple of (8t-1. 0t- 8)) at each timestep i, and train them by minimizing the cross entropy between the output and the true action de. We find that giving input both the previous state St, and previous action d;-1, opposed to just the previous action a;-1, helps the models to better associate past actions with the states in which they were taken.
The remainder of a the sequence model for Overcooked consists of a policy network as described above, with the fully connected layers replaced by either an LSTM or transformer. We use 3 layers and 256 hidden neurons for both the LSTM and transformer.
B DERIVATIONS
B. ENTROPY KL DIVERGENCE
Here, we derive the equation used in Section 2 to relate the entropy of the policy distribution (o(m) to its KL divergence from the normalized base measure Prese (TTT/// du(().. First, note that
dp (TT) {dp(T)) dprase
17

---

Published as a conference paper at ICLR 2022
where
du dpane
is the Radon-Nikodee derivative between ju(T) and Pease (T). Now,
H(qo(T)) / -log log du
dgo
(x)) (
log (Jdj((T) - / log (fdu(()) + log du
dqo
(T)) dgo(a)
log (Jdj((T)) / log
dqp
(T) (()s(((()))) dgo(T)
du
log (fdj((=)) - - log
dap (TT)
du
(T)) deo((T)
dl dymuue
log (Jdju(T)) - / log
dge
(TT) dgo
dpaase
= log (fdu(()) DxL (go(T) |l Poase (T)).
B.2 DISCRIMINATOR APPROXIMATES KL DIVERGENCE
For completeness, we also derive the result of Huszár 2017) showing that minimizing the discrimina- tor objective given in 10 approximates the KL divergence between (ạ( and Phase (T). Recall that the discriminator is chosen by solving the following optimization problem:
d = arg min (x log (1 exp (-d()))) E Errppae (T) log (1 + exp {d(T)))]
(17)
d
Assuming that qo (T)) and Paase (TT) both have density with respect to the Lebesgue measure, rewrite (17) as
d = arg min / log (1 exp (- + log (1 + exp {d(TT)) Pease dr.
(18)
Clearly, the integral in (18) is minimized if and only if the integrand is minimized for all n. That is.
we have that
d(T) min log (1 exp (-d(T))) (T()) + log (1 + exp {d(==)) Prase (TT). d(
is straightforward to show that the value of d((r) which minimizes this
ga(TT)
d(TT) log
Pouse (TT
Thus, taking the expectation of d(T) gives the KI. divergence between (T()) and Pruse (TT):
Pbase (TT) ) DxL (go || Poase Pbase(TT) (T)).
(TT)
Ex [d(T)] = (T)
log
C ADDITIONAL EXPERIMENTS
In this appendíx, present results from additional experiments that did not fit in the main text. First, we explore alternative choices of hyperparameters for the BPD. We also include an ablation of prediction with sequence models using the BPD. We conclude with closer look at prediction of human behavior in Overcooked and what makes different human models perform better or worse.
C.I LATENT VECTOR DIMENSION
We explore the effect of altering the dimensionality n the latent vector z used approximate the BPD through the policy network fo 2). We use 1, 000 throughout the rest of our experiments, but we also tried using n 100. The effects on prediction and collaboration are shown Figure 10 We find that reducing has negative effects collaborative performance and also on prediction using MFVI. Surprisingly, setting n = 100 does not affect the performance of a transformer trained to perform prediction using the BPD. We are not sure why this happens the transformer may simply be good at predicting oft-of- distribbbbu.... Even if explicitly optimize over values of z for low cross entropy human data when 100, we unable to reach the performance of the transformer.
18

---

Published as a conference paper at ICLR 2022
Cramped Room
Forced Coordination
Coordination Ring
Random prediction
1.5
2.0
1,000
Cross entropy
2
BPD (MFVI)
1.5
1.0
BPD (transforme)
1.0
100
0.5
BPD (MFVI)
0.5
BPD (transfomer)
0.0
0.0
Cramped Room
Forced Coordination
Coordination Ring
Mean return human proxy
150
60
20
n 1,000
100
BPD (no memory)
40
100
50
10
BPD (no memory)
20
Figure 10: The effect on human prediction and human-Al collaboration of changing the dimension n of the latent vector used to approximate the BPD. See Appendix C.I for details and discussion.
Cramped Room
Forced Coordination
Coordination Ring
2.0
Random prediction
1.5
1.5
0.2
Cross entropy
1.5
BPD (MFVI)
1.0
1.0
BPD (tranffrmer)
1.0
0.5
0.5
BPD (MFVI)
BPD (transformer)
0.0
0.0
0.0
Figure 11: The effect on human prediction changing the concentration a of the Dirichlet base distribution (T) using to calculate the BPD. See Appendix C.2 for details and discussion.
C.2 BASE DISTRIBUTION CONCENTRATION
Next, we try changing a, the parameter of the Dirichlet distribution used to define Pbuse Recall from Section aan and Figures 4 and 5 that lower values of a correspond while higher values correspond to more random human. We use a = 0.2 for most experiments throughout the paper, but here explore the effects of using
more consistent human
for predicting human behavior. The results are shown in Figure TT Using a transformer with a = 0.2 predicts human behavior the best for all layouts, validating our choice of and our assumption that humans are consistent.
C.3 FURTHER PREDICTION EXPERIMENTS
Here, we investigate what makes different models better or worse at human prediction in Overcooked. In particular, we noticed that the majority of the actions taken in Overcooked by humans are "stay" actions-that is, actions that do not do anything. Thus, simply predicting a "stay" action with high probability can lead good predictive performance. In Figure 12 we separate out the stay actions and, in the right two columns, show the cross entropy and accuracy of models excluding these actions.
That is, remove all timesteps when stay action was taken and remove the stay probability from the output of the human model. After removing stay actions, Boltzmann rationality is better predictor of human behavior, although behavior cloning and the BPD still superior. We also note
19

---

Published as a conference paper at ICLR 2022
that while the BPD has similar cross entropy to behavior cloning, its accuracy is slightly lower non-stay actions. This could explain why collaborative policies trained with the BPD do not always outperform those trained with behavior cloned human model, e.g. Forced Coordination (see Figure 9.
We also explored some other baselines and ablations of the BPD. First, try removing all reward information and simply trying to predict based on a random distribution over policies- that is, using a uniform prior over policies and doing online inference. Using the same procedure for the BPD, sample trajectories from thousands of random policies drawn from Phase (=) and train a transformer for online prediction over these trajectories. The resulting prediction performance on real human data is shown in Figure 12 as the white bars. At first, it appears that this technique is quite successful, rivaling behavior cloning and the BPD in cross entropy and accuracy. However, this largely seems to be because it consistently predicts the human will take stay action. Removing these actions, is clear that using a random policy distribution does not results in good predictive performance. These experiments validate the utility of knowing the human's 's reward function for predicting human behavior and serve as an ablation of our method online prediction.
Next, explored alternatives to behavior cloning. As discussed in Appendix A we used hand- crafted features to train all the behavior cloning models, following Carroll et al. (2020). We additional tried BC directly using the raw observations and found it performed slightly worse. Furthermore, we explored using generative adversarial imitation learning (GAIL.) (Ho & Ermon (2016) as a more robust alternative BC. We found that both these approaches performed similarly worse than BC; see Figure 12 for the full results.
slightly
Finally, we explored alternative online prediction with the BPD. For this baseline, we simply started with a random initialized policy network and updated it after each timestep of a test episode using the BC loss. The results for this method using either hand-crafted features or raw observations are shown in Figure 12 with the label "Online BC." Online BC is directly comparable to online inference with the BPD since they both do not require prior human data. However, we generally found that online BC does not perform as well as BC or the BPD, particularly when using raw observations.
20

---

Published as a conference paper at ICLR 2022
Cross entropy
Accuracy
Cross ent. stay)
Acc. (no stay)
Cramped Room
60%
2
60%
40%
20%
20%
0%
0%
Forced Coordination
60%
60%
2-
40%
40% 2
3
Coordination Ring
3
60%
40%
2
2
40%
20%
20%
0%
0%
Data-intensive approaches
Naive approaches
Behavior cloning
Random prediction
Behavior cloning (raw observations)
Sclf-play policy
GAIL (raw observations)
Random policies (transformer)
Online learning
Reward-conditioned human models
Online
Boltzmann rationality
Online BC (raw observations)
BPD (MFVI) BPD (transformer)
Figure 12: variety of metrics for predictive power of various human models on the three Overcooked layouts. Lower entropy and higher accuracy is better. The right two columns exclude "stay" actions, which make up the majority of the human data, from consideration. See Appendix C.3 for details and discussion.
21


Reviews:
[{"review_id": "gyH7wCnLX4", "reviewer": "Reviewer_Cv9m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is pretty interesting and the approximation makes it possible to train a rather implausible model BPD. The paper is well written. I find it easy and clear to read through, without the need to go back and forth. The hyperparameters and the setup of the experiments are clearly provided. It would be good if the code can be shared if accepted.\n\nThough the model is only tested on one complicated task and more experiments are always better, I find the experiments in the paper serve their purpose to illustrate the effectiveness of the model. I also find the simulation very useful for understanding the model, and as a sanity check to make sure after approximation the model is doing something as expected.\n\nThe intuition of the model is sort of based on “people are consistent over time”. It makes sense to me, but I’m wondering if there are any references on this. The better performance of BPD compared to BR sort of supports this, but it’s not clear whether it is the consistency of choice that actually benefits BPD. BPD is clearly a much larger and complicated model.\n\nIt would be good to put the size of the training set for BPD compared to BC. Is the BPD only directly trained/adapted on the testing data? It would be good to briefly discuss how easy/hard to train BPD.", "summary_of_the_paper": "The paper proposes a new model BPD to account for the suboptimality of human behavior, and provide an approximation inference method for BPD. The authors illustrate BPD through a simple simulation and compare BPD with existing methods on the overcooked game. They show that their model can match the performance of the data-extensive BC method and outperform BR. They further evaluate their model on human-AI collaboration and show that their model can match or even outperform the human-aware RL model.", "summary_of_the_review": "I’m overall positive about this paper. The idea is interesting and the experiments illustrate the effectiveness of the new model. The model has the potentials to be applied and further adapted to other settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MungH13s3Fk", "reviewer": "Reviewer_Hd8J", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I recommend accepting this paper because it provides a theoretically clean analysis of a novel insight that is relevant to important problems. The key insight is that one can use the Boltzmann distribution over policies as a prior, to make behavioral cloning more data efficient. \n\nSignificance:\n\nThe proposed approach is computationally expensive, but it makes sense to use it when compute is abundant but human demonstrations are scarce or expensive. This is likely to be a common situation in the future as human time stays equally expensive while computation budgets grow. Hopefully, paper’s insights can spur more computationally efficient approaches.\n\nThe paper provides insights relevant mainly to two important issues. Firstly, imitation learning from systematically suboptimal human behavior. Secondly, human-AI coordination, which benefits from predicting human behavior and is increasingly relevant in areas such as robotics and autonomous driving. \n\n\nExperiments:\n\nThe authors empirically demonstrate clear advantages over Boltzmann rationality. They also show that their method better predicts human behavior, and that it is more data-efficient than behavioral cloning.\n\nAside from a toy environment, the evaluation is done in Overcooked, a complex and interesting environment. Overcooked is fittingly a key benchmark for human-AI collaboration which is a goal of this paper.\n\nThe paper is overall very well written and illustrated.\n\n\nConcerns:\n\n1) The authors left it unclear why BPC should be more data efficient than behavioral cloning (BC). While BPC needs less samples empirically, this could just be because the BPC was allowed to learn online during each episode whereas BC was not. In fact, the BC baseline is left underspecified. Is the proposed method a special case of BC that uses a specific prior (the BPC)? For a fair comparison, the authors should use the same models, inference algorithms, online learning, etc for BC and the proposed method. Arguably, the only difference should be the prior (BPC) or any other aspect that the authors claim is novel.\n\n2) One of the motivations of the paper is to understand human preferences, but the proposed method assumes that the human reward function is already known. This seems contradictory or at best requires explanation. I’m referring to the first sentence in the paper here. If the motivation is not to understand human preferences, I’d suggest to clarify the actual motivations throughout the introduction. The other motivations (predicting human actions and collaborating with humans) make more sense to me so you could simply remove ‘understanding human preferences’ to partially address my comment. However, I also think these motivations deserve more than one sentence of elaboration. Additionally, the conclusion says that the the BPD could be useful for reward learning, even though it assumes that the reward is known. There’s also explanation needed here.\n\n3) A further limitation that should be discussed is that the BPC assumes the human policy to be close to optimal with high probability. If the human is systematically irrational they may achieve a much lower reward than what is optimal. Consequently, the BPC may assign very low probability to the true human policy.\n\n\n\n\n\n———————— Detailed comments ————————\n\n\nThe experiments don’t seem to answer if the BPD prior is actually needed for good performance. It might be enough to do online inference with a different prior (e.g. uniform)?\n\n“Understanding human preferences, predicting human actions, and collaborating with humans all require models of human behavior. “; “However, Boltzmann rationality fails to capture human behavior that is systematically suboptimal.”: some of these key claims currently lack support in the introduction. They’ve been supported e.g. in arXiv:1712.05812 and arXiv:1512.05832.\n\nFigure 2 lacks a bracket after R(s,a). It also lacks the labels b) and c).\n\nThe introduction was a bit long, e.g. the last paragraph does not seem essential.\n\nThe paper repeatedly says that BPD is hard to compute because policies lie in a (high-dimensional) continuous space. It’s unclear why continuity would be a problem here, and I suspect this is not true. Indeed a high-dimensional discrete space seems more difficult for optimization and inference.\n", "summary_of_the_paper": "The goal of this paper is to train models that imitate human behavior using limited samples. A human reward function is given but human behavior may be suboptimal according to this reward function and therefore we cannot use the standard assumption, Boltzmann rationality. The authors propose an imitation learning algorithm that uses the known reward function to form a prior over human policies, the Boltzmann policy distribution (BPD). A posterior over policies is then inferred from human behavior. This approach is empirically more data efficient than behaviorally cloning human behavior.\n", "summary_of_the_review": "Overall, I recommend accepting this paper because it provides a theoretically clean analysis of a novel insight that is relevant to important problems. The key insight is that one can use the Boltzmann distribution over policies as a prior, to make behavioral cloning more data efficient. The experiments and presentation are overall convincing. I have concerns about the behavioral cloning baseline and the relevance to learning preferences.\n\n\nUPDATE: Although we were not able to resolve some confusion during the discussion period, my substantive comments have been addressed so I have updated my score now. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Kx43Qsk4xn", "reviewer": "Reviewer_Bv5s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n* Pros\n    * Nice writing and figures.\n    * Very clear difference between Boltzmann rational model and the proposed method.\n    * Real world (Overcooked game) dataset evaluation.\n    * The authors not only evaluated prediction power of the proposed method in Overcooked, but also combine the proposed method with RL agent to do the Overcooked game.\n\n* Majors\n    * I think (please correct me if I am wrong :) the proposed approach is very much similar to GAIL (`Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. Advances in neural information processing systems, 29, 4565-4573.`)\n        * The generator, `f_\\theta` in this work = `\\pi_\\theta` in Alg.1 in GAIL.\n        * The discriminator, `d` in this work = `D_w` in Alg.1 in GAIL.\n        * The sampled path, `q_\\theta = {s1,a1,…,s_{t-1},a_{t-1}}` in this work = `\\tau_i ~ \\pi_\\theta_i`, in Alg.1 in GAIL.\n        * The base measure, `p_base` in this work = the expert policy, `\\tau_E ~ \\pi_E`, in Alg.1 in GAIL.\n        * In my opinion, this work is similar to GAIL. The only differences are the followings:\n            * This work captures the suboptimality in the human behavior in the base measure, while GAIL assumes that the human is optimal.\n              * This work's reward is implicitly hidden in the base measure, while GAIL's reward can be interpreted as `D_w`.\n        * If this is true, it would be great if the authors could add some text to explain the relation between the proposed method and GAIL. This could be formally and/or empirically comparison.\n\n\n* Minor:\n    * The comparison between the proposed method and behavior cloning is a bit unfair, since the proposed method has the knowledge about the human reward, while the behavior cloning does not.\n    * Page 2: `Conditioned on the (known) goal of getting to the office, observing the person choose route A gives no information about how the person will act in the future.`\n      * This sentence makes sense assuming that the human reward function is correct and fixed. I think if the human reward can be updated, then observing that the human choosing route A could provide information. It would be great to clarify this here.\n    * Page 5: `We discuss in Section 3 how to approximate this inference problem.`\n      * Section 3 is a typo?\n    * Page 6: Eq9: `p_unif` should be `p_base`?\n", "summary_of_the_paper": "\n* This paper proposes an interesting approach to model and predict human behavior.\n\n* The approach is called Boltzmann policy distribution (BPD). It improves the Boltzmann rational model in that BPD considers the systematic suboptimality in human behavior.\n    * Systematic suboptimality means that the human could be consistent in producing suboptimal behavior. Hence, to capture systematic suboptimality, it is necessary to combine the human's reward function (optimal behavior) and trajectory data (deviation from optimality) in human modeling and prediction.\n\n* This approach predict human policies, rather than trajectories, so that it can capture the (systematically suboptimal) human behavior that is reflected in the human action choices over time.\n\n* Approach detail: the approach follows GAN (generative adversarial networks):\n    * The goal is to compute the human BPD policy `= \\integral \\pi(a|s) * p_PBD(\\pi | s1,a1,...,s_{t-1},a_{t-1})` (Eq5).\n        * `\\pi(a|s)` is approximated as a **generator**, `f_\\theta(s,z)` (Eq.6).\n        * `p_PBD(\\pi | s1,a1,...,s_{t-1},a_{t-1})` is approximated as sampled human trajectories, `q_\\theta(z | s1,a1,...,s_{t-1},a_{t-1})` (Eq.6).\n    * The base measure, `p_base(\\pi)`,  is defined as the optimal human behavior based on the known human reward function with suboptimality parameter \\beta (Eq4).\n    * The **discriminator**, `d`, used to distinguish between the policy generated by the base measure, `p_base(\\pi)`, and the policy generated by the sampled human trajectories, `q_\\theta(\\pi)`.\n    * The networks can be seen as: `\\pi(s) -> d(\\pi) -> z -> f_\\theta(s,z) -> \\pi(s) reconstructed`.\n    * The training for the generator, `f_\\theta(s,z)`, is in Eq8,9 and the training for the discriminator, `d`, is in Eq10.\n\n\n* Results\n    * Apple picking gridworld with simulated human data\n        * Cross entropy: BPD < Boltzmann rational models\n    * Overcooked (prediction only) with real human-with-human play data\n        * Cross entropy: BPD = behavior cloning < random < PPO+self-play, Boltzmann rational models\n    * Human-AI collaboration (prediction and control in Overcooked) with \"human proxy\" policy from previous work\n        * Mean return: human-aware RL policy (prev work) > policy based on Boltzmann rationality, self-play policy, behavior cloning policy\n        * Mean return: policy based on BPD > policy based on Boltzmann rationality, self-play policy, behavior cloning policy\n        * Mean return: policy based on BPD > Human-aware RL policy (prev work) in 2 out of 3 cases.\n", "summary_of_the_review": "* I think it would be great if the connection between the proposed method and GAIL could be stated more clearly (if there are any). And empirical comparison with GAIL could be helpful, too. This is why I currently give 2 for the `2: The contributions are only marginally significant or novel.`\n\n# Post-rebuttal\nThank you for your clarification! I have adjusted my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4e1SjqjDE9G", "reviewer": "Reviewer_9Tye", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Very well written.\n- Contains many insightful examples and figures.\n- Most of the statements are very well supported\n", "summary_of_the_paper": "The paper addresses the problem of modeling human behavior in a case where their deviation from the optimal behavior is consistent over time, rather than independent (systematic suboptimality). They claim that systematic suboptimality can be modeled by predicting policies rather than trajectories. To this end, they introduce Boltzmann policy distribution (BPD), as an alternative to Boltzman rationality, as a prior over human policies. BPD enables adaption to human policies over time. While in Boltzmann rationality past behavior cannot be used to predict future behavior (since current action is independent of all the previous actions), the paper claims that since humans are consistent one can use past actions to predict future ones which leads to introducing BPD. They consider that human is sampling over policies rather than trajectories which results in the previous actions and states inducing a posterior over policies. By taking the expectation over the posterior they predict the next action. They further use deep generative models in order to sample from the BPD and minimize the KL divergence between the two distributions (P_BPD and the distribution induced by the generative network) to ensure the predicted distribution is close to the BPD.\n\nThrough experiments, they demonstrate cases where Boltzmann rationality is not able to predict the behavior of a suboptimal human while BPD can adapt to the behavior over time and predict the correct next action. They also show that their method is able to predict human behavior as well as the baseline while using fewer data.", "summary_of_the_review": "The paper addresses and motivates an interesting problem and further solves modifies the previously known Boltzmann rationality, introducing Boltmann Policy Distribution, in order to adapt to the human behaviour over time. The technical details of the approach are well written and supported and experiments are well designed to give illustrations of the cases where the Boltmann rationality will fail while BPD manages to predict human behaviour. I believe the paper is a good paper and authors have taken time to write it in an elegant way and design insightful examples to motivate their problem and show the advantages of their method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new approach to learning human behavior by observing a small number of interactions. To this end, it proposes a Bayesian learning framework where a Boltzmann-type prior over human policies, based on an available reward function, governs default behavior. The prior is updated by observing actual trajectories taken by (human) agents, in principle. In practice, a full-fledged implementation using Gaussian priors and features from a neural architecture is proposed and shown to be effective in practical benchmarks. 

The reviewers are all positive about the paper's contributions. One remaining concern is that the effect of the quality of the prior (Boltzmann vs. other type vs. features designed in a different way) on the learning process is not explored to a significant depth. Yet, the results and approach proposed in the paper are valuable to merit acceptance.
Decision: Accept (Poster)