The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models

1 INTRODUCTION . Understanding human preferences , predicting human actions , and collaborating with humans all require models of human behavior . One way of modeling human behavior is to postulate intentions , or rewards , and assume that the human will act rationally with regard to those intentions or rewards . However , people are rarely perfectly rational ; thus , some models relax this assumption to include random deviations from optimal decision-making . The most common method for modeling humans in this manner , Boltzmann rationality ( Luce , 1959 ; 1977 ; Ziebart et al. , 2010 ) , predicts that a human will act out a trajectory with probability proportional to the exponentiated return they receive for the trajectory . Alternatively , some human models eschew rewards and rationality for a data-driven approach . These methods use extensive data of humans acting in the target environment to train a ( typically high capacity ) model with imitation learning to predict future human actions ( Ding et al. , 2011 ; Mainprice & Berenson , 2013 ; Koppula & Saxena , 2013 ; Alahi et al. , 2016 ; Ho & Ermon , 2016 ; Ma et al. , 2017 ; Schmerling et al. , 2018 ; Chai et al. , 2019 ; Wang et al. , 2019 ; Carroll et al. , 2020 ) . Both these methods for modeling humans are successful in many settings but also suffer drawbacks . Imitation learning has been applied to modeling humans in driving ( Sun et al. , 2018 ) , arm motion ( Ding et al. , 2011 ) , and pedestrian navigation ( Ma et al. , 2017 ) , among other areas . However , it requires a lengthy process of data collection , data cleaning , and feature engineering . Furthermore , policies learned in one environment may not transfer to others , although there is ongoing work to enable models to predict better out-of-distribution ( Torrey et al. , 2005 ; Brys et al. , 2015 ) or in many environments simultaneously ( Duan et al. , 2017 ; Singh et al. , 2020 ) . On the other hand , reward-based models often learn a reward in a low-dimensional space from limited data ( Ziebart et al. , 2009 ; Kuderer et al. , 2015 ; Kretzschmar et al. , 2016 ) and tend to transfer more easily to new environments and be robust to distribution shifts ( Sun et al. , 2021 ) . For many useful tasks , most elements of the reward function might even be clear a priori ; in thoses cases , Boltzmann rationality has enabled successful human-AI collaboration based only on data observed during execution of a single task ( Bandyopadhyay et al. , 2013 ) , e.g . assisting a person to reach an unknown goal ( Dragan & Srinivasa , 2013 ; Javdani et al. , 2015 ) . However , Boltzmann rationality fails to capture human behavior that is systematically suboptimal . As we show , in some environments this makes it no better at predicting human behavior than a random baseline . We argue that Boltzmann rationality struggles in these cases because it fails to capture systematic dependencies between human actions across time . Consider the setting in Figure 1 . Boltzmann rationality predicts the person will take the shortest path , route B , with high probability . Conditioned on the ( known ) goal of getting to the office , observing the person choose route A gives no information about how the person will act in the future . Thus , the following day , the probability under the Boltzmann trajectory distribution of choosing route A remains unchanged . Similarly , in the setting of Figure 3 , observing a person walk to the right around an obstacle gives no information about how they will navigate around it in the future . However , in reality people are consistent over time : they tend to take the same action in the same state , and similar actions in similar states , whether because of consistent biases , muscle memory , bounded computational resources , or myriad other factors . Such consistency is supported well by behavioral psychology ( Funder & Colvin ; Sherman et al. ) . Our key insight is that systematic suboptimality can be captured by going beyond predicting distributions over trajectories , and coupling action choices over time by predicting distributions over policies instead . Assuming that people will choose a policy with probability proportional to its exponentiated expected return induces a Boltzmann policy distribution ( BPD ) , which we use as a prior over human policies . Observing a person taking actions updates this distribution to a posterior via Bayesian inference , enabling better prediction over time and capturing systematic deviations from optimality . An overview of the process of calculating and updating the BPD is shown in Figure 2 . In the example in Figure 1 , the BPD predicts that the person will continue to take route A even though route B is shorter . In Figure 3 , the BPD predicts that the person is more likely navigate around the obstacle to the right in the future after watching them do it once . One could argue that updating the BPD in response to observed human actions is no different than training a human model via imitation learning . However , we show that the BPD can give predictive power over the course of a single episode lasting less than three minutes similar to that of a behavior cloning-based model trained on far more data . Thus , the BPD has both the advantages of both Boltzmann rationality—that it works with very little data—as well as imitation learning—that it can adapt to systematically suboptimal human behavior . One challenge is that calculating the Boltzmann policy distribution for an environment and using it for inference are more difficult than computing the Boltzmann rational trajectory distribution . Whereas the Boltzmann trajectory distribution can be described by a single stochastic policy ( the Prior policy distribution . maximum-entropy policy ) , the BPD is a distribution over the space of all policies—a high dimensional , continuous space . Furthermore , the complex dependencies between action distributions at different states in the BPD that make it effective at predicting human behavior also make it difficult to calculate . To approximate the BPD , we parameterize a policy distribution by conditioning on a latent vector—an approach similar to generative models for images such as VAEs ( Kingma & Welling , 2014 ) and GANs ( Goodfellow et al. , 2014 ) . A policy network takes both the current state and the latent vector and outputs action probabilities . When the latent vector is sampled randomly from a multivariate normal distribution , this induces a distribution over policies that can represent near-arbitrary dependencies across states . See Figure 3 for a visualization of how elements of the latent vector can correspond to natural variations in human policies . In Section 2 , we describe how to optimize this parameterized distribution to approximate the BPD by minimizing the KL divergence between the two . During deployment , observed human actions can be used to calculate a posterior over which latent vectors correspond to the human ’ s policy . We investigate approximating this posterior both explicitly using mean-field variational inference and implicitly using sequence modeling . We explore the benefits of the Boltzmann policy distribution for human prediction and human-AI collaboration in Overcooked , a game where two players cooperate to cook and deliver soup in a kitchen ( Carroll et al. , 2020 ) . The BPD predicts real human behavior far more accurately than Boltzmann rationality and similarly to a behavior cloned policy , which requires vast amounts of human data in the specific target environment . Furthermore , a policy using the BPD as a human model performs better than one using Boltzmann rationality at collaboration with a simulated human ; it approaches the performance of a collaborative policy based on a behavior cloned human model . Overall , the BPD appears to be a more accurate and useful human model than traditional Boltzmann rationality . Further , by exploiting knowledge of the reward function in the task , it achieves performance on par with imitation learning methods that use vastly more data in the target environment . But while in our setting the reward function is known ( i.e . the task objective is clear ) , this is not always be the case . There are many tasks in which it needs to be inferred from prior data ( Christiano et al. , 2017 ) , or in which reward learning is itself the goal ( Ng & Russell , 2000 ) . However , any reward-conditioned human model like the BPD can be inverted to infer rewards from observed behavior using algorithms like inverse reinforcement learning . We are hopeful that using the ( more accurate ) BPD model will in turn lead to more accurate reward inference given observed behavior compared to using the traditional Boltzmann model . 2 THE BOLTZMANN POLICY DISTRIBUTION . We formalize our model of human behavior in the setting of an infinite-horizon Markov decision process ( MDP ) . We assume a human is acting in an environment , taking actions a ∈ A in states s ∈ S . Given a state st and action at at timestep t , the next state is reached via Markovian transition probabilities p ( st+1 | st , at ) . We assume that the person is aiming to optimize some reward function R : S ×A → R. This may be learned from data in another environment or specified a priori ; see the introduction for discussion . Rewards are accumulated over time with a discount rate γ ∈ [ 0 , 1 ) . Let a trajectory consist of a sequence of states and actions τ = ( s1 , a1 , . . . , sT , aT ) . Let a policy be a mapping π : S → ∆ ( A ) from a state to a distribution over actions taken at that state . The Boltzmann rationality model , or Boltzmann trajectory distribution , states that the probability the human will take a trajectory is proportional to the exponentiated return of the trajectory times a “ rationality coefficient ” or “ inverse temperature ” β : pBR ( τ ) ∝ exp { β ∑T t=1γ tR ( st , at ) } ( 1 ) This is in fact the distribution over trajectories induced by a particular policy , the maximum-entropy policy , which we call πMaxEnt ( Ziebart et al. , 2010 ) . That is , pBR ( s1 , a1 , . . . , sT ) = p ( s1 ) T−1∏ t=1 p ( st+1 | st , at ) πMaxEnt ( at | st ) ( 2 ) Say we are interested in using the Boltzmann trajectory distribution to predict the action at time t given the current state st and all previous states and actions . This can be calculated using ( 2 ) : pBR ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = pBR ( s1 , a1 , . . . , st , at ) pBR ( s1 , a1 , . . . , st ) = πMaxEnt ( at | st ) ( 3 ) The implication of ( 3 ) is that , under the Boltzmann trajectory distribution , the action at a particular timestep is independent of all previous actions given the current state . That is , we can not use past behavior to better predict future behavior . However , as we argue , humans are consistent in their behavior , and thus past actions should predict future actions . We propose to assume that the human is not making a choice over trajectories , but rather over policies in a manner analogous to ( 1 ) : pBPD ( π ) ∝ exp { β Eat∼π ( st ) [ ∑∞ t=1γ tR ( st , at ) ] } = exp { βJ ( π ) } ( 4 ) ( 4 ) defines the Boltzmann policy distribution ( BPD ) , our main contribution . Note that J ( π ) denotes the expected return of the policy π . Under the Boltzmann policy distribution , previous actions do help predict future actions : pBPD ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = ∫ π ( at | st ) pBPD ( π | s1 , a1 , . . . , st−1 , at−1 ) dπ ( 5 ) That is , previous actions a1 , . . . , at−1 at states s1 , . . . , st−1 induce a posterior over policies pBPD ( π | s1 , a1 , . . . , st−1 , at−1 ) . Taking the expectation over this posterior gives the predicted next action probabilities under the BPD . Both Boltzmann rationality and the BPD assume that the human is near-optimal with high probability , and thus neither may perform well if a person is very suboptimal . However , as we will show in our experiments , people generally are close enough to optimal for the BPD to be a useful model . Approximating the distribution While we have defined the BPD in ( 4 ) , we still need tractable algorithms to sample from it and to calculate the posterior based on observed human actions . This is difficult because the BPD is a distribution over policies in the high dimensional continuous space ( ∆ ( A ) ) |S| . Furthermore , the density of the BPD defined in ( 4 ) is a nonlinear function of the policy and induces dependencies between action probabilities at different states that must be captured . We propose to approximate the BPD using deep generative models , which have been successful at modeling high-dimensional distributions with complex dependencies in the computer vision and graphics literature ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; Dinh et al. , 2015 ) . In particular , let z ∼ N ( 0 , In ) be a random Gaussian vector in Rn . Then we define our approximate policy distribution as π ( · | s ) = fθ ( s , z ) where fθ is a neural network which takes a state s and latent vector z and outputs action probabilities . Each value of z corresponds to a different policy , and the policy can depend nearly arbitrarily on z via the neural network fθ . To sample a policy from the distribution , we sample a latent vector z ∼ N ( 0 , In ) and use it to calculate the action probabilities of the policy at any state s. Let qθ ( π ) denote this distribution over policies induced by the network fθ . Figure 3 gives an example of a policy distribution approximated using latent vectors z ∈ R2 . To predict the human ’ s next action given previous actions using qθ ( π ) , it suffices to calculate a posterior over z , i.e . qθ ( at | s1 , a1 , . . . , st−1 , at−1 , st ) = ∫ fθ ( st , z ) qθ ( z | s1 , a1 , . . . , st−1 , at−1 ) dz ( 6 ) We discuss at the end of this section how to approximate this inference problem . Optimizing the distribution While we have shown how to represent a policy distribution using a deep generative model , we still need to ensure that this policy distribution is close to the BPD that we are trying to approximate . To do so , we minimize the KL divergence between the approximation qθ ( π ) and pBPD ( π ) : θ∗ = arg min θ DKL ( qθ ( π ) ‖ pBPD ( π ) ) ( 7 ) To optimize ( 7 ) , we expand the KL divergence : θ∗ = arg min θ −Eπ∼qθ ( π ) [ log pBPD ( π ) ] + Eπ∼qθ ( π ) [ log qθ ( π ) ] = arg max θ Eπ∼qθ ( π ) [ βJ ( π ) ] +H ( qθ ( π ) ) ( 8 ) Here , H ( qθ ( π ) ) denotes the entropy of the generative model distribution . Note that although the density pBPD ( π ) includes a normalization term , it is a constant with respect to θ and thus can be ignored for the purpose of optimization . Equation ( 8 ) has an intuitive interpretation : maximize the expected return of policies sampled from qθ ( π ) , but also make the policies diverse ( i.e. , the policy distribution should be high entropy ) . The first term can be optimized using a reinforcement learning algorithm . However , the second term is intractable to even calculate directly , let alone optimize ; entropy estimation is known to be very difficult in high dimensional spaces ( Han et al. , 2019 ) . To optimize the entropy of the policy distribution , we rewrite it as a KL divergence and then optimize a1 a2 a3 a4 a5 a6 α = 0.2 a1 a2 a3 a4 a5 a6 α = 1 a1 a2 a3 a4 a5 a6 Boltz . traj . Figure 4 : A visualization of the Boltzmann trajectory and Boltzmann policy distributions at one state . The density of the policy distribution is shown over a hexagon ; each point corresponds to a distribution over the six actions . For small α ( on the left ) , the policies are close to deterministic ; here , policies almost always take either a2 , a4 , or a6 , but rarely randomize much between them . For large α ( the middle ) , policies tend to be random across all actions . The Boltzmann trajectory distribution corresponds to a single point at the MaxEnt policy . 0 10 20 t 0 10 20 t′ α = 0.2 0 10 20 t α = 1 0 10 20 t Boltz . traj . 0.0 0.3 I ( a t , a t′ |s t , s t′ ) Figure 5 : A visualization of the mutual information ( aka information gain ) I ( at , at′ | st , st′ ) between actions taken at different timesteps t and t′ given the states at those timesteps under the Boltzmann policy distribution and Boltzmann trajectory distribution ( Boltzmann rationality ) . Unlike Boltzmann rationality , the BPD allows actions taken in the past to help predict actions taken in the future . The information gain is higher the lower the parameter α since more deterministic policies are more predictable . that KL divergence using a discriminator . Let µ ( π ) denote the base measure with which the density of the BPD is defined in ( 4 ) . Let pbase ( π ) = µ ( π ) / ∫ dµ ( π ) , i.e . µ ( π ) normalized to a probability measure . Then the entropy of the policy distribution qθ ( π ) is equal to its negative KL divergence from pbase ( π ) plus a constant : H ( qθ ( π ) ) = log ( ∫ dµ ( π ) ) −DKL ( qθ ( π ) ‖ pbase ( π ) ) ( 9 ) We include a full derivation of ( 9 ) in Appendix B.1 . While ( 9 ) replaces the intractable entropy term with a KL divergence , we still need to optimize the divergence . This is difficult as we only have access to samples from qθ ( π ) , so we can not compute the divergence in closed form . Instead , we use an adversarially trained discriminator to approximate the KL divergence . The discriminator d assigns a score d ( π ) ∈ R to any policy π . It is trained to assign low scores to policies drawn from the base distribution and high scores to policies drawn from qθ ( π ) : d = arg min d Eπ∼qθ ( π ) [ log ( 1 + exp { −d ( π ) } ) ] + Eπ∼pbase ( π ) [ log ( 1 + exp { d ( π ) } ) ] ( 10 ) Huszár ( 2017 ) show that if ( 10 ) is minimized , then DKL ( qθ ( π ) ‖ pbase ( π ) ) = Eπ∼qθ ( π ) [ d ( π ) ] . That is , the expectation of the discriminator scores for policies drawn from the distribution qθ ( π ) approximates its KL divergence from the base distribution pbase ( π ) . Similar to the training of GANs , the policy network fθ attempts to “ fool ” the discriminator by making it more difficult to tell its policies apart from those in the base distribution , thus increasing the entropy of qθ ( π ) . Putting it all together , the final optimization objective is θ∗ = arg max θ Eπ∼qθ ( π ) [ βJ ( π ) − d ( π ) ] ( 11 ) where d is optimized as in ( 10 ) . We optimize θ in ( 11 ) by gradient descent . Specifically , at each iteration of optimization , we approximate the gradient of the expectation in ( 11 ) with a Monte Carlo sample of several policies π1 , . . . , πM ∼ qθ ( π ) . The gradient of the discriminator score d ( πi ) can be calculated by backpropagation through the discriminator network . The gradient of the policy return J ( πi ) is approximated by a policy gradient algorithm , PPO ( Schulman et al. , 2017 ) . Choice of base measure An additional parameter to the BPD is the choice of base measure µ ( π ) and the derived base distribution pbase ( π ) = µ ( π ) / ∫ dµ ( π ) . These define the initial weighting of policies before considering the reward function . In our experiments , we define the base measure as a product of independent Dirichlet distributions at each state : µ ( π ) = pbase ( π ) = ∏ s∈S ps ( π ( · | s ) ) ps ind.∼ Dir ( α , . . . , α ) ∀ s ∈ S ( 12 ) The concentration parameter α used in the Dirichlet distribution controls the resulting distribution over policies ; we explore the effect of α in Section 2 , Figures 4 and 5 , and Appendix C.2 . Parameterization of policy and discriminator We modify existing policy network architectures to use as fθ ( s , z ) by adding an attention mechanism over the latent vector z . We use a transformer ( Vaswani et al. , 2017 ) neural network as the discriminator . See Appendix A.2 for more details . Online prediction with the Boltzmann policy distribution Predicting actions with the BPD requires solving an inference problem . In particular , we need to solve ( 6 ) , predicting the next action by marginalizing over the posterior distribution of latent vectors z . We explore approximating this inference both explicitly and implicitly . First , we approximate the posterior over z as a product of independent Gaussians using mean-field variational inference ( MFVI , see Appendix A ) ( Blei et al. , 2017 ) . This approach , while providing an explicit approximation of the posterior , is slow and can not represent posteriors which are multimodal or have dependencies between the components of z . Our second approach ignores the posterior and instead directly approximates the online prediction problem qθ ( at | s1 , a1 , . . . , st ) . To do this , we generate thousands of trajectories from the BPD by sampling a latent z for each and rolling out the resulting policy in the environment . Then , we train a transformer ( Vaswani et al. , 2017 ) on these trajectories to predict the next action at given the current state st and all previous actions . This avoids the need to compute a posterior over z explicitly . Exploring the Boltzmann policy distribution Using the methods described above , we compute approximate Boltzmann policy distributions for three layouts in Overcooked : Cramped Room , Coordination Ring , and Forced Coordination . A visualization of the resulting policy distribution for Cramped Room at one state can be seen in Figure 4 , which highlights the effect of the parameter α . We also visualize the BPD for a simple gridworld environment in Figure 3 ( a ) . This figure shows that components of the latent vector z capture natural variation in potential human policies . In Figure 5 , we show that , unlike a Boltzmann distribution over trajectories , the BPD allows actions in previous states to be predictive of actions in future states . The predictive power of past actions is higher for lower values of α , which also motivates our choice of α = 0.2 for the remaining experiments . Figure 8 shows that the BPD can use the dependence between actions over time to predict systematically suboptimal actions in Overcooked . 3 EXPERIMENTS We evaluate the Boltzmann policy distribution in three settings : predicting simulated human behavior in a simple gridworld , predicting real human behavior in Overcooked , and enabling human-AI collaboration in Overcooked . The full details of our experimental setup are in Appendix A . We provide further results and ablations in Appendix C. Simulated human data First , we investigate if the BPD can predict synthetic human trajectories better than Boltzmann rationality in the apple picking gridworld shown in Figure 3 . In this environment , the human must travel back and forth around an obstacle to pick apples and bring them to a basket . To generate simulated human data , we assume the human has a usual direction to travel around the obstacle on the way to the tree and a usual direction to travel on the way back . We assign different simulated humans “ consistency ” values from 0.5 to 1 , and these control how often each human takes their usual direction . A consistency of 0.5 means the human always picks a random direction , while a consistency of 1 means the human always takes their usual direction around the obstacle . The mean cross-entropy assigned to trajectories sampled from these simulated humans by the BPD and Boltzmann rationality at each consistency value is shown in Figure 6 , with the shaded region indicating the standard deviation . As expected , the BPD is particularly good at predicting human behavior for the most consistent humans . However , it even does better than Boltzmann rationality at predicting the humans which randomly choose a direction to proceed around the obstacle each time . This is because our simulated trajectories always go straight to the tree and back to the basket . Boltzmann rationality gives a chance for the human to reverse direction , but the BPD picks up on the consistent behavior and gives it higher probability , leading to lower cross-entropy . 0 1 2 C ro ss en tr op y Cramped Room 0 1 2 Forced Coordination 0 1 2 Coordination Ring Data-intensive approach Behavior cloning Naive approaches Random prediction Self-play policy Reward-conditioned human models Boltzmann rationality BPD ( MFVI ) BPD ( transformer ) ( a ) Cramped Room : the optimal action is to move up to get an onion , but the human repeatedly goes to the left . ( b ) Forced Coordination : the optimal action is to turn to the left to pick up an onion , but the human always moves down instead . ( c ) Coordination Ring : the human consistently tries to interact with the pot already full of soup to put in another onion . Figure 8 : The Boltzmann policy distribution predicts real human behavior in Overcooked better than Boltzmann rationality ( BR ) by adapting to systematically suboptimal humans . In each of the three layouts , the shown state is reached multiple times in a single episode . The human repeatedly takes the suboptimal action overlaid on the game . The predictions of the BPD and BR for each timestep when the state is reached are shown below the game , ignoring “ stay ” actions ( see Appendix C.3 ) . BPD adapts to the human ’ s consistent suboptimality while BR continues to predict an incorrect action . Note that actions at one state can also help predict actions at different states ; see Figure 5 . Human prediction in Overcooked Next , we evaluate the predictive power of the Boltzmann policy distribution on trajectories of real human-with-human play collected by Carroll et al . ( 2020 ) in Overcooked . As in the gridworld , we evaluate online prediction : given a human ’ s trajectory until time t− 1 , predict the action the human will take at time t. We compare three approaches to the BPD for online prediction in Overcooked . First , we train a behavior cloning ( BC ) model on a set of human trajectories separate from those evaluated on . This model allows us to compare a data-intense prediction method to our approach which adapts over the course of a single trajectory ( less than 3 minutes of real time ) . Second , we train an optimal policy using reinforcement learning ( PPO ) with self-play . We expect this policy to have low predictive power because humans tend to be suboptimal . Third , we calculate a Boltzmann rational policy . The predictive performance of all models is shown in Figure 7 . The self-play and Boltzmann rational models are both no more accurate than random at predicting human behavior in all three Overcooked layouts . In contrast , the BPD predicts human behavior nearly as well as the behavior cloned model . As expected , the MFVI approach to prediction with the BPD performs worse because its representation of the posterior over z is explicitly restricted . These results validate the BPD as an accurate reward-conditioned human model . Human-AI collaboration Predicting human behavior is not an end in itself ; one of the chief goals of human model research is to enable AI agents to collaborate with humans . Thus , we also evaluate the utility of the Boltzmann policy distribution for human-AI collaboration . The Overcooked environment provides a natural setting to explore collaboration since it is a two-player game . To test human-AI collaborative performance , we pair various collaborative “ robot ” policies with a simulated “ human proxy ” policy , which is trained with behavior cloning on held-out human data . Carroll et al . ( 2020 ) found that performance this proxy correlated well with real human-AI performance . We compare several collaborative policies based on similar approaches to the prediction models . First , we evaluate the behavior cloned ( BC ) model . Since it and the human proxy are both trained from human data , this approximates the performance of a human-human team . Second , we train a human-aware RL policy ( Carroll et al. , 2020 ) , which is a best-response to the BC policy . This approach requires extensive human data to train the BC policy . We also evaluate the self-play optimal policy ; this policy is optimized for collaboration with itself and not with any human model . Next , we train collaborative policies based on both the Boltzmann rational and BPD models . During each episode of training , the collaborative policy is paired with a simulated human policy from the human model—the MaxEnt policy for Boltzmann rationality , or a randomly sampled policy from the BPD . The human model ’ s actions are treated as part of the dynamics of the environment ; this allows the collaborative policy to be trained with PPO as in a single-agent environment . This approach is similar to human-aware RL , but we train with reward-conditioned human models instead of behavior cloned ones . Over the course of training , each collaborative policy maximizes the expected return of the human-AI pair , assuming the human takes actions as predicted by the respective human model . Because human actions are correlated across time under the BPD , we train BPD-based collaborative policies both with and without memory . The policies with memory have the form π ( aRt | s1 , aH1 , . . . , st−1 , aHt−1 , st ) ; that is , they choose a robot action aRt based both on the current state st and also human actions aHt′ taken at prior states . This allows adaption to the human ’ s policy . Figure 9 shows the results of evaluating all the collaborative policies with human proxies on the three Overcooked layouts . In all layouts , the human-aware RL policy does better than either a self-play policy or the BC policy , matching the results from Carroll et al . ( 2020 ) . The collaborative policies based on Boltzmann rationality tend to perform poorly , which is unsurprising given its weak predictive power . The collaborative policies based on the BPD perform consistently better across all three layouts . Surprisingly , the BPD-based collaborative policies with memory do not perform much differently than those without memory . This could be due more to the difficulty of training such policies than to the inability for adaption using the BPD .

Summary:  * This paper proposes an interesting approach to model and predict human behavior.  * The approach is called Boltzmann policy distribution (BPD). It improves the Boltzmann rational model in that BPD considers the systematic suboptimality in human behavior.     * Systematic suboptimality means that the human could be consistent in producing suboptimal behavior. Hence, to capture systematic suboptimality, it is necessary to combine the human's reward function (optimal behavior) and trajectory data (deviation from optimality) in human modeling and prediction.  * This approach predict human policies, rather than trajectories, so that it can capture the (systematically suboptimal) human behavior that is reflected in the human action choices over time.  * Approach detail: the approach follows GAN (generative adversarial networks):     * The goal is to compute the human BPD policy `= \integral \pi(a|s) * p_PBD(\pi | s1,a1,...,s_{t-1},a_{t-1})` (Eq5).         * `\pi(a|s)` is approximated as a **generator**, `f_\theta(s,z)` (Eq.6).         * `p_PBD(\pi | s1,a1,...,s_{t-1},a_{t-1})` is approximated as sampled human trajectories, `q_\theta(z | s1,a1,...,s_{t-1},a_{t-1})` (Eq.6).     * The base measure, `p_base(\pi)`,  is defined as the optimal human behavior based on the known human reward function with suboptimality parameter \beta (Eq4).     * The **discriminator**, `d`, used to distinguish between the policy generated by the base measure, `p_base(\pi)`, and the policy generated by the sampled human trajectories, `q_\theta(\pi)`.     * The networks can be seen as: `\pi(s) -> d(\pi) -> z -> f_\theta(s,z) -> \pi(s) reconstructed`.     * The training for the generator, `f_\theta(s,z)`, is in Eq8,9 and the training for the discriminator, `d`, is in Eq10.   * Results     * Apple picking gridworld with simulated human data         * Cross entropy: BPD < Boltzmann rational models     * Overcooked (prediction only) with real human-with-human play data         * Cross entropy: BPD = behavior cloning < random < PPO+self-play, Boltzmann rational models     * Human-AI collaboration (prediction and control in Overcooked) with "human proxy" policy from previous work         * Mean return: human-aware RL policy (prev work) > policy based on Boltzmann rationality, self-play policy, behavior cloning policy         * Mean return: policy based on BPD > policy based on Boltzmann rationality, self-play policy, behavior cloning policy         * Mean return: policy based on BPD > Human-aware RL policy (prev work) in 2 out of 3 cases.