The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models

1 INTRODUCTION . Understanding human preferences , predicting human actions , and collaborating with humans all require models of human behavior . One way of modeling human behavior is to postulate intentions , or rewards , and assume that the human will act rationally with regard to those intentions or rewards . However , people are rarely perfectly rational ; thus , some models relax this assumption to include random deviations from optimal decision-making . The most common method for modeling humans in this manner , Boltzmann rationality ( Luce , 1959 ; 1977 ; Ziebart et al. , 2010 ) , predicts that a human will act out a trajectory with probability proportional to the exponentiated return they receive for the trajectory . Alternatively , some human models eschew rewards and rationality for a data-driven approach . These methods use extensive data of humans acting in the target environment to train a ( typically high capacity ) model with imitation learning to predict future human actions ( Ding et al. , 2011 ; Mainprice & Berenson , 2013 ; Koppula & Saxena , 2013 ; Alahi et al. , 2016 ; Ho & Ermon , 2016 ; Ma et al. , 2017 ; Schmerling et al. , 2018 ; Chai et al. , 2019 ; Wang et al. , 2019 ; Carroll et al. , 2020 ) . Both these methods for modeling humans are successful in many settings but also suffer drawbacks . Imitation learning has been applied to modeling humans in driving ( Sun et al. , 2018 ) , arm motion ( Ding et al. , 2011 ) , and pedestrian navigation ( Ma et al. , 2017 ) , among other areas . However , it requires a lengthy process of data collection , data cleaning , and feature engineering . Furthermore , policies learned in one environment may not transfer to others , although there is ongoing work to enable models to predict better out-of-distribution ( Torrey et al. , 2005 ; Brys et al. , 2015 ) or in many environments simultaneously ( Duan et al. , 2017 ; Singh et al. , 2020 ) . On the other hand , reward-based models often learn a reward in a low-dimensional space from limited data ( Ziebart et al. , 2009 ; Kuderer et al. , 2015 ; Kretzschmar et al. , 2016 ) and tend to transfer more easily to new environments and be robust to distribution shifts ( Sun et al. , 2021 ) . For many useful tasks , most elements of the reward function might even be clear a priori ; in thoses cases , Boltzmann rationality has enabled successful human-AI collaboration based only on data observed during execution of a single task ( Bandyopadhyay et al. , 2013 ) , e.g . assisting a person to reach an unknown goal ( Dragan & Srinivasa , 2013 ; Javdani et al. , 2015 ) . However , Boltzmann rationality fails to capture human behavior that is systematically suboptimal . As we show , in some environments this makes it no better at predicting human behavior than a random baseline . We argue that Boltzmann rationality struggles in these cases because it fails to capture systematic dependencies between human actions across time . Consider the setting in Figure 1 . Boltzmann rationality predicts the person will take the shortest path , route B , with high probability . Conditioned on the ( known ) goal of getting to the office , observing the person choose route A gives no information about how the person will act in the future . Thus , the following day , the probability under the Boltzmann trajectory distribution of choosing route A remains unchanged . Similarly , in the setting of Figure 3 , observing a person walk to the right around an obstacle gives no information about how they will navigate around it in the future . However , in reality people are consistent over time : they tend to take the same action in the same state , and similar actions in similar states , whether because of consistent biases , muscle memory , bounded computational resources , or myriad other factors . Such consistency is supported well by behavioral psychology ( Funder & Colvin ; Sherman et al. ) . Our key insight is that systematic suboptimality can be captured by going beyond predicting distributions over trajectories , and coupling action choices over time by predicting distributions over policies instead . Assuming that people will choose a policy with probability proportional to its exponentiated expected return induces a Boltzmann policy distribution ( BPD ) , which we use as a prior over human policies . Observing a person taking actions updates this distribution to a posterior via Bayesian inference , enabling better prediction over time and capturing systematic deviations from optimality . An overview of the process of calculating and updating the BPD is shown in Figure 2 . In the example in Figure 1 , the BPD predicts that the person will continue to take route A even though route B is shorter . In Figure 3 , the BPD predicts that the person is more likely navigate around the obstacle to the right in the future after watching them do it once . One could argue that updating the BPD in response to observed human actions is no different than training a human model via imitation learning . However , we show that the BPD can give predictive power over the course of a single episode lasting less than three minutes similar to that of a behavior cloning-based model trained on far more data . Thus , the BPD has both the advantages of both Boltzmann rationality—that it works with very little data—as well as imitation learning—that it can adapt to systematically suboptimal human behavior . One challenge is that calculating the Boltzmann policy distribution for an environment and using it for inference are more difficult than computing the Boltzmann rational trajectory distribution . Whereas the Boltzmann trajectory distribution can be described by a single stochastic policy ( the Prior policy distribution . maximum-entropy policy ) , the BPD is a distribution over the space of all policies—a high dimensional , continuous space . Furthermore , the complex dependencies between action distributions at different states in the BPD that make it effective at predicting human behavior also make it difficult to calculate . To approximate the BPD , we parameterize a policy distribution by conditioning on a latent vector—an approach similar to generative models for images such as VAEs ( Kingma & Welling , 2014 ) and GANs ( Goodfellow et al. , 2014 ) . A policy network takes both the current state and the latent vector and outputs action probabilities . When the latent vector is sampled randomly from a multivariate normal distribution , this induces a distribution over policies that can represent near-arbitrary dependencies across states . See Figure 3 for a visualization of how elements of the latent vector can correspond to natural variations in human policies . In Section 2 , we describe how to optimize this parameterized distribution to approximate the BPD by minimizing the KL divergence between the two . During deployment , observed human actions can be used to calculate a posterior over which latent vectors correspond to the human ’ s policy . We investigate approximating this posterior both explicitly using mean-field variational inference and implicitly using sequence modeling . We explore the benefits of the Boltzmann policy distribution for human prediction and human-AI collaboration in Overcooked , a game where two players cooperate to cook and deliver soup in a kitchen ( Carroll et al. , 2020 ) . The BPD predicts real human behavior far more accurately than Boltzmann rationality and similarly to a behavior cloned policy , which requires vast amounts of human data in the specific target environment . Furthermore , a policy using the BPD as a human model performs better than one using Boltzmann rationality at collaboration with a simulated human ; it approaches the performance of a collaborative policy based on a behavior cloned human model . Overall , the BPD appears to be a more accurate and useful human model than traditional Boltzmann rationality . Further , by exploiting knowledge of the reward function in the task , it achieves performance on par with imitation learning methods that use vastly more data in the target environment . But while in our setting the reward function is known ( i.e . the task objective is clear ) , this is not always be the case . There are many tasks in which it needs to be inferred from prior data ( Christiano et al. , 2017 ) , or in which reward learning is itself the goal ( Ng & Russell , 2000 ) . However , any reward-conditioned human model like the BPD can be inverted to infer rewards from observed behavior using algorithms like inverse reinforcement learning . We are hopeful that using the ( more accurate ) BPD model will in turn lead to more accurate reward inference given observed behavior compared to using the traditional Boltzmann model . 2 THE BOLTZMANN POLICY DISTRIBUTION . We formalize our model of human behavior in the setting of an infinite-horizon Markov decision process ( MDP ) . We assume a human is acting in an environment , taking actions a ∈ A in states s ∈ S . Given a state st and action at at timestep t , the next state is reached via Markovian transition probabilities p ( st+1 | st , at ) . We assume that the person is aiming to optimize some reward function R : S ×A → R. This may be learned from data in another environment or specified a priori ; see the introduction for discussion . Rewards are accumulated over time with a discount rate γ ∈ [ 0 , 1 ) . Let a trajectory consist of a sequence of states and actions τ = ( s1 , a1 , . . . , sT , aT ) . Let a policy be a mapping π : S → ∆ ( A ) from a state to a distribution over actions taken at that state . The Boltzmann rationality model , or Boltzmann trajectory distribution , states that the probability the human will take a trajectory is proportional to the exponentiated return of the trajectory times a “ rationality coefficient ” or “ inverse temperature ” β : pBR ( τ ) ∝ exp { β ∑T t=1γ tR ( st , at ) } ( 1 ) This is in fact the distribution over trajectories induced by a particular policy , the maximum-entropy policy , which we call πMaxEnt ( Ziebart et al. , 2010 ) . That is , pBR ( s1 , a1 , . . . , sT ) = p ( s1 ) T−1∏ t=1 p ( st+1 | st , at ) πMaxEnt ( at | st ) ( 2 ) Say we are interested in using the Boltzmann trajectory distribution to predict the action at time t given the current state st and all previous states and actions . This can be calculated using ( 2 ) : pBR ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = pBR ( s1 , a1 , . . . , st , at ) pBR ( s1 , a1 , . . . , st ) = πMaxEnt ( at | st ) ( 3 ) The implication of ( 3 ) is that , under the Boltzmann trajectory distribution , the action at a particular timestep is independent of all previous actions given the current state . That is , we can not use past behavior to better predict future behavior . However , as we argue , humans are consistent in their behavior , and thus past actions should predict future actions . We propose to assume that the human is not making a choice over trajectories , but rather over policies in a manner analogous to ( 1 ) : pBPD ( π ) ∝ exp { β Eat∼π ( st ) [ ∑∞ t=1γ tR ( st , at ) ] } = exp { βJ ( π ) } ( 4 ) ( 4 ) defines the Boltzmann policy distribution ( BPD ) , our main contribution . Note that J ( π ) denotes the expected return of the policy π . Under the Boltzmann policy distribution , previous actions do help predict future actions : pBPD ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = ∫ π ( at | st ) pBPD ( π | s1 , a1 , . . . , st−1 , at−1 ) dπ ( 5 ) That is , previous actions a1 , . . . , at−1 at states s1 , . . . , st−1 induce a posterior over policies pBPD ( π | s1 , a1 , . . . , st−1 , at−1 ) . Taking the expectation over this posterior gives the predicted next action probabilities under the BPD . Both Boltzmann rationality and the BPD assume that the human is near-optimal with high probability , and thus neither may perform well if a person is very suboptimal . However , as we will show in our experiments , people generally are close enough to optimal for the BPD to be a useful model . Approximating the distribution While we have defined the BPD in ( 4 ) , we still need tractable algorithms to sample from it and to calculate the posterior based on observed human actions . This is difficult because the BPD is a distribution over policies in the high dimensional continuous space ( ∆ ( A ) ) |S| . Furthermore , the density of the BPD defined in ( 4 ) is a nonlinear function of the policy and induces dependencies between action probabilities at different states that must be captured . We propose to approximate the BPD using deep generative models , which have been successful at modeling high-dimensional distributions with complex dependencies in the computer vision and graphics literature ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; Dinh et al. , 2015 ) . In particular , let z ∼ N ( 0 , In ) be a random Gaussian vector in Rn . Then we define our approximate policy distribution as π ( · | s ) = fθ ( s , z ) where fθ is a neural network which takes a state s and latent vector z and outputs action probabilities . Each value of z corresponds to a different policy , and the policy can depend nearly arbitrarily on z via the neural network fθ . To sample a policy from the distribution , we sample a latent vector z ∼ N ( 0 , In ) and use it to calculate the action probabilities of the policy at any state s. Let qθ ( π ) denote this distribution over policies induced by the network fθ . Figure 3 gives an example of a policy distribution approximated using latent vectors z ∈ R2 . To predict the human ’ s next action given previous actions using qθ ( π ) , it suffices to calculate a posterior over z , i.e . qθ ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = ∫ fθ ( st , z ) qθ ( z | s1 , a1 , . . . , st−1 , at−1 ) dz ( 6 ) We discuss at the end of this section how to approximate this inference problem . Optimizing the distribution While we have shown how to represent a policy distribution using a deep generative model , we still need to ensure that this policy distribution is close to the BPD that we are trying to approximate . To do so , we minimize the KL divergence between the approximation qθ ( π ) and pBPD ( π ) : θ∗ = arg min θ DKL ( qθ ( π ) ‖ pBPD ( π ) ) ( 7 ) To optimize ( 7 ) , we expand the KL divergence : θ∗ = arg min θ −Eπ∼qθ ( π ) [ log pBPD ( π ) ] + Eπ∼qθ ( π ) [ log qθ ( π ) ] = arg max θ Eπ∼qθ ( π ) [ βJ ( π ) ] +H ( qθ ( π ) ) ( 8 ) Here , H ( qθ ( π ) ) denotes the entropy of the generative model distribution . Note that although the density pBPD ( π ) includes a normalization term , it is a constant with respect to θ and thus can be ignored for the purpose of optimization . Equation ( 8 ) has an intuitive interpretation : maximize the expected return of policies sampled from qθ ( π ) , but also make the policies diverse ( i.e. , the policy distribution should be high entropy ) . The first term can be optimized using a reinforcement learning algorithm . However , the second term is intractable to even calculate directly , let alone optimize ; entropy estimation is known to be very difficult in high dimensional spaces ( Han et al. , 2019 ) . To optimize the entropy of the policy distribution , we rewrite it as a KL divergence and then optimize a1 a2 a3 a4 a5 a6 α = 0.2 a1 a2 a3 a4 a5 a6 α = 1 a1 a2 a3 a4 a5 a6 Boltz . traj . Figure 4 : A visualization of the Boltzmann trajectory and Boltzmann policy distributions at one state . The density of the policy distribution is shown over a hexagon ; each point corresponds to a distribution over the six actions . For small α ( on the left ) , the policies are close to deterministic ; here , policies almost always take either a2 , a4 , or a6 , but rarely randomize much between them . For large α ( the middle ) , policies tend to be random across all actions . The Boltzmann trajectory distribution corresponds to a single point at the MaxEnt policy . 0 10 20 t 0 10 20 t′ α = 0.2 0 10 20 t α = 1 0 10 20 t Boltz . traj . 0.0 0.3 I ( a t , a t′ |s t , s t′ ) Figure 5 : A visualization of the mutual information ( aka information gain ) I ( at , at′ | st , st′ ) between actions taken at different timesteps t and t′ given the states at those timesteps under the Boltzmann policy distribution and Boltzmann trajectory distribution ( Boltzmann rationality ) . Unlike Boltzmann rationality , the BPD allows actions taken in the past to help predict actions taken in the future . The information gain is higher the lower the parameter α since more deterministic policies are more predictable . that KL divergence using a discriminator . Let µ ( π ) denote the base measure with which the density of the BPD is defined in ( 4 ) . Let pbase ( π ) = µ ( π ) / ∫ dµ ( π ) , i.e . µ ( π ) normalized to a probability measure . Then the entropy of the policy distribution qθ ( π ) is equal to its negative KL divergence from pbase ( π ) plus a constant : H ( qθ ( π ) ) = log ( ∫ dµ ( π ) ) −DKL ( qθ ( π ) ‖ pbase ( π ) ) ( 9 ) We include a full derivation of ( 9 ) in Appendix B.1 . While ( 9 ) replaces the intractable entropy term with a KL divergence , we still need to optimize the divergence . This is difficult as we only have access to samples from qθ ( π ) , so we can not compute the divergence in closed form . Instead , we use an adversarially trained discriminator to approximate the KL divergence . The discriminator d assigns a score d ( π ) ∈ R to any policy π . It is trained to assign low scores to policies drawn from the base distribution and high scores to policies drawn from qθ ( π ) : d = arg min d Eπ∼qθ ( π ) [ log ( 1 + exp { −d ( π ) } ) ] + Eπ∼pbase ( π ) [ log ( 1 + exp { d ( π ) } ) ] ( 10 ) Huszár ( 2017 ) show that if ( 10 ) is minimized , then DKL ( qθ ( π ) ‖ pbase ( π ) ) = Eπ∼qθ ( π ) [ d ( π ) ] . That is , the expectation of the discriminator scores for policies drawn from the distribution qθ ( π ) approximates its KL divergence from the base distribution pbase ( π ) . Similar to the training of GANs , the policy network fθ attempts to “ fool ” the discriminator by making it more difficult to tell its policies apart from those in the base distribution , thus increasing the entropy of qθ ( π ) . Putting it all together , the final optimization objective is θ∗ = arg max θ Eπ∼qθ ( π ) [ βJ ( π ) − d ( π ) ] ( 11 ) where d is optimized as in ( 10 ) . We optimize θ in ( 11 ) by gradient descent . Specifically , at each iteration of optimization , we approximate the gradient of the expectation in ( 11 ) with a Monte Carlo sample of several policies π1 , . . . , πM ∼ qθ ( π ) . The gradient of the discriminator score d ( πi ) can be calculated by backpropagation through the discriminator network . The gradient of the policy return J ( πi ) is approximated by a policy gradient algorithm , PPO ( Schulman et al. , 2017 ) . Choice of base measure An additional parameter to the BPD is the choice of base measure µ ( π ) and the derived base distribution pbase ( π ) = µ ( π ) / ∫ dµ ( π ) . These define the initial weighting of policies before considering the reward function . In our experiments , we define the base measure as a product of independent Dirichlet distributions at each state : µ ( π ) = pbase ( π ) = ∏ s∈S ps ( π ( · | s ) ) ps ind.∼ Dir ( α , . . . , α ) ∀ s ∈ S ( 12 ) The concentration parameter α used in the Dirichlet distribution controls the resulting distribution over policies ; we explore the effect of α in Section 2 , Figures 4 and 5 , and Appendix C.2 . Parameterization of policy and discriminator We modify existing policy network architectures to use as fθ ( s , z ) by adding an attention mechanism over the latent vector z . We use a transformer ( Vaswani et al. , 2017 ) neural network as the discriminator . See Appendix A.2 for more details . Online prediction with the Boltzmann policy distribution Predicting actions with the BPD requires solving an inference problem . In particular , we need to solve ( 6 ) , predicting the next action by marginalizing over the posterior distribution of latent vectors z . We explore approximating this inference both explicitly and implicitly . First , we approximate the posterior over z as a product of independent Gaussians using mean-field variational inference ( MFVI , see Appendix A ) ( Blei et al. , 2017 ) . This approach , while providing an explicit approximation of the posterior , is slow and can not represent posteriors which are multimodal or have dependencies between the components of z . Our second approach ignores the posterior and instead directly approximates the online prediction problem qθ ( at | s1 , a1 , . . . , st ) . To do this , we generate thousands of trajectories from the BPD by sampling a latent z for each and rolling out the resulting policy in the environment . Then , we train a transformer ( Vaswani et al. , 2017 ) on these trajectories to predict the next action at given the current state st and all previous actions . This avoids the need to compute a posterior over z explicitly . Exploring the Boltzmann policy distribution Using the methods described above , we compute approximate Boltzmann policy distributions for three layouts in Overcooked : Cramped Room , Coordination Ring , and Forced Coordination . A visualization of the resulting policy distribution for Cramped Room at one state can be seen in Figure 4 , which highlights the effect of the parameter α . We also visualize the BPD for a simple gridworld environment in Figure 3 ( a ) . This figure shows that components of the latent vector z capture natural variation in potential human policies . In Figure 5 , we show that , unlike a Boltzmann distribution over trajectories , the BPD allows actions in previous states to be predictive of actions in future states . The predictive power of past actions is higher for lower values of α , which also motivates our choice of α = 0.2 for the remaining experiments . Figure 8 shows that the BPD can use the dependence between actions over time to predict systematically suboptimal actions in Overcooked . 3 EXPERIMENTS We evaluate the Boltzmann policy distribution in three settings : predicting simulated human behavior in a simple gridworld , predicting real human behavior in Overcooked , and enabling human-AI collaboration in Overcooked . The full details of our experimental setup are in Appendix A . We provide further results and ablations in Appendix C. Simulated human data First , we investigate if the BPD can predict synthetic human trajectories better than Boltzmann rationality in the apple picking gridworld shown in Figure 3 . In this environment , the human must travel back and forth around an obstacle to pick apples and bring them to a basket . To generate simulated human data , we assume the human has a usual direction to travel around the obstacle on the way to the tree and a usual direction to travel on the way back . We assign different simulated humans “ consistency ” values from 0.5 to 1 , and these control how often each human takes their usual direction . A consistency of 0.5 means the human always picks a random direction , while a consistency of 1 means the human always takes their usual direction around the obstacle . The mean cross-entropy assigned to trajectories sampled from these simulated humans by the BPD and Boltzmann rationality at each consistency value is shown in Figure 6 , with the shaded region indicating the standard deviation . As expected , the BPD is particularly good at predicting human behavior for the most consistent humans . However , it even does better than Boltzmann rationality at predicting the humans which randomly choose a direction to proceed around the obstacle each time . This is because our simulated trajectories always go straight to the tree and back to the basket . Boltzmann rationality gives a chance for the human to reverse direction , but the BPD picks up on the consistent behavior and gives it higher probability , leading to lower cross-entropy . 0 1 2 C ro ss en tr op y Cramped Room 0 1 2 Forced Coordination 0 1 2 Coordination Ring Data-intensive approach Behavior cloning Naive approaches Random prediction Self-play policy Reward-conditioned human models Boltzmann rationality BPD ( MFVI ) BPD ( transformer ) ( a ) Cramped Room : the optimal action is to move up to get an onion , but the human repeatedly goes to the left . ( b ) Forced Coordination : the optimal action is to turn to the left to pick up an onion , but the human always moves down instead . ( c ) Coordination Ring : the human consistently tries to interact with the pot already full of soup to put in another onion . Figure 8 : The Boltzmann policy distribution predicts real human behavior in Overcooked better than Boltzmann rationality ( BR ) by adapting to systematically suboptimal humans . In each of the three layouts , the shown state is reached multiple times in a single episode . The human repeatedly takes the suboptimal action overlaid on the game . The predictions of the BPD and BR for each timestep when the state is reached are shown below the game , ignoring “ stay ” actions ( see Appendix C.3 ) . BPD adapts to the human ’ s consistent suboptimality while BR continues to predict an incorrect action . Note that actions at one state can also help predict actions at different states ; see Figure 5 . Human prediction in Overcooked Next , we evaluate the predictive power of the Boltzmann policy distribution on trajectories of real human-with-human play collected by Carroll et al . ( 2020 ) in Overcooked . As in the gridworld , we evaluate online prediction : given a human ’ s trajectory until time t− 1 , predict the action the human will take at time t. We compare three approaches to the BPD for online prediction in Overcooked . First , we train a behavior cloning ( BC ) model on a set of human trajectories separate from those evaluated on . This model allows us to compare a data-intense prediction method to our approach which adapts over the course of a single trajectory ( less than 3 minutes of real time ) . Second , we train an optimal policy using reinforcement learning ( PPO ) with self-play . We expect this policy to have low predictive power because humans tend to be suboptimal . Third , we calculate a Boltzmann rational policy . The predictive performance of all models is shown in Figure 7 . The self-play and Boltzmann rational models are both no more accurate than random at predicting human behavior in all three Overcooked layouts . In contrast , the BPD predicts human behavior nearly as well as the behavior cloned model . As expected , the MFVI approach to prediction with the BPD performs worse because its representation of the posterior over z is explicitly restricted . These results validate the BPD as an accurate reward-conditioned human model . Human-AI collaboration Predicting human behavior is not an end in itself ; one of the chief goals of human model research is to enable AI agents to collaborate with humans . Thus , we also evaluate the utility of the Boltzmann policy distribution for human-AI collaboration . The Overcooked environment provides a natural setting to explore collaboration since it is a two-player game . To test human-AI collaborative performance , we pair various collaborative “ robot ” policies with a simulated “ human proxy ” policy , which is trained with behavior cloning on held-out human data . Carroll et al . ( 2020 ) found that performance this proxy correlated well with real human-AI performance . We compare several collaborative policies based on similar approaches to the prediction models . First , we evaluate the behavior cloned ( BC ) model . Since it and the human proxy are both trained from human data , this approximates the performance of a human-human team . Second , we train a human-aware RL policy ( Carroll et al. , 2020 ) , which is a best-response to the BC policy . This approach requires extensive human data to train the BC policy . We also evaluate the self-play optimal policy ; this policy is optimized for collaboration with itself and not with any human model . Next , we train collaborative policies based on both the Boltzmann rational and BPD models . During each episode of training , the collaborative policy is paired with a simulated human policy from the human model—the MaxEnt policy for Boltzmann rationality , or a randomly sampled policy from the BPD . The human model ’ s actions are treated as part of the dynamics of the environment ; this allows the collaborative policy to be trained with PPO as in a single-agent environment . This approach is similar to human-aware RL , but we train with reward-conditioned human models instead of behavior cloned ones . Over the course of training , each collaborative policy maximizes the expected return of the human-AI pair , assuming the human takes actions as predicted by the respective human model . Because human actions are correlated across time under the BPD , we train BPD-based collaborative policies both with and without memory . The policies with memory have the form π ( aRt | s1 , aH1 , . . . , st−1 , aHt−1 , st ) ; that is , they choose a robot action aRt based both on the current state st and also human actions aHt′ taken at prior states . This allows adaption to the human ’ s policy . Figure 9 shows the results of evaluating all the collaborative policies with human proxies on the three Overcooked layouts . In all layouts , the human-aware RL policy does better than either a self-play policy or the BC policy , matching the results from Carroll et al . ( 2020 ) . The collaborative policies based on Boltzmann rationality tend to perform poorly , which is unsurprising given its weak predictive power . The collaborative policies based on the BPD perform consistently better across all three layouts . Surprisingly , the BPD-based collaborative policies with memory do not perform much differently than those without memory . This could be due more to the difficulty of training such policies than to the inability for adaption using the BPD .

Summary: The paper proposes a new model BPD to account for the suboptimality of human behavior, and provide an approximation inference method for BPD. The authors illustrate BPD through a simple simulation and compare BPD with existing methods on the overcooked game. They show that their model can match the performance of the data-extensive BC method and outperform BR. They further evaluate their model on human-AI collaboration and show that their model can match or even outperform the human-aware RL model.