SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We compare offline RL as described in Section 5 to two baselines that also do not assume access to online interactions, nor simulators of the environment: naive behavior cloning (BC) on the dataset, and filtered BC that takes the subset of 10 trajectories that result in the highest reward. We also include a comparison to Fictitious Co-Play (FCP) that assumes an environment simulator and is a common baseline in collaborative tasks [29]. FCP trains a population of randomly-initialized agents via self-play.
Evaluation. We deployed each policy alongside a human player. We recruited 10 different human users for this evaluation. For each layout and method, we collected 30 trajectories of length  $H = 400$ .
Figure 4: Improvement in reward (taken as difference in cumulative reward between last 100 and first 100 timesteps) when a human is paired with our learned agent (via offline RL) across 30 trajectories of  $H = 400$  timesteps.
In Figure 5, we reported the mean and standard error of reward across all trajectories when solving both the standard task, as well as generalization to the new ones. We see that across all domains, offline RL outperformed both BC and filtered BC, as well as FCP. This is because both BC and filtered BC cannot exceed human-human performance, which is often suboptimal due to imperfect allocation of tasks for coordination. The difference is exaggerated when evaluating generalization, where human-human performance in the dataset solves different tasks. In addition, self-play approaches fail when the hu
man partner is strategically suboptimal, whereas our learned agent via offline RL can account for that by influencing the partner to change their strategy.
Next, we ask whether this improvement in reward is solely due to the agent performing well on their part of the task, or due to an actual impact on human behavior. Qualitatively, in Figure 6, we show that offline RL is able to learn a sensible influencing strategy to influence the human partner to plate
Figure 5: Average rewards over 30 trajectories of  $H = 400$  timesteps of learned agents paired with real humans, with standard error across humans.
Figure 6: Trajectory where the ego agent influences their partner to plate the soup by moving a plate on the counter nearby. The partner ends up dropping their onion in order to plate the soup. This leads to more optimal coordination as the partner is in a more advantageous position to deliver soups.
the dish, which is crucial for success in Asymmetric Advantages (Human Deliver). In the visualized trajectory, the learned agent moves a plate near the human and leaves to pick up ingredients for the next soup; the human realizes this and plates the soup on the stove. We verify that this behavior does not appear at all in the prior data. However, components of this behavior (such as mistakenly picking up a plate and subsequently placing it on a counter) appear in the prior data. In addition, in Figure 4, we show that the mean reward achieved in the last 100 timesteps greatly exceeds that of the first 100 timesteps, suggesting that coordination is improving over time between our agent and a human.
6 Achieving Long-term Influence of Latent Strategies
In this section, we consider how offline RL can learn to effectively achieve long-term influence on the human's policy itself. Accomplishing this type of adaptive influence requires knowledge of what the other human intends to do, or their latent strategy. In contrast to Section 5, we are not trying to show that offline RL can learn influence strategies to affect a human's actions that do not appear in the data, but rather that offline RL can influence a human's policy, while adapting to changes in the policy.
We propose an offline RL method that learns a low-dimensional representation of the human's latent strategy, and condition the learned policy's behavior on that strategy. The method we propose involves separately learning such representations, then a policy that conditions on these representations as part of its state. We describe the high-level approach but defer implementation details to Appendix A.
6.1 Learning Representations of Latent Strategy
First, we describe how we learn representations of latent strategy. Recall from the definition of Hi-MDPs in Section 3 that the other human's behavior follows latent dynamics  $P^z (\mathbf{z}'\mid \mathbf{s},\mathbf{a},\mathbf{z})$  where  $\mathbf{z},\mathbf{z}^{\prime}\in \mathcal{Z}$  are representations of the human's latent strategy; we use  $d$ -dimensional embeddings as the representations, with  $d$  as a hyper-parameter.
Additionally, we introduce  $h$  to be the agent's history of previous  $c$  steps of interactions, where  $c \in \mathbb{N}$  is a tunable parameter. Namely, let  $\tau = \{(\mathbf{s}_1, \mathbf{a}_1), \dots, (\mathbf{s}_H, \mathbf{a}_H)\}$  denote a trajectory taken by the ego agent, then we denote the history at step  $i$  corresponding to state  $\mathbf{s}_i$  and action  $\mathbf{a}_i$  as
Figure 7: Example trajectory where the ego agent influences their partner to pick up tomatoes rather than onions by blocking off access to the onions. Note that the agent should only block their partner if they believe their partner is going towards the onions. Once their partner appears to be going to the tomatoes instead, the agent returns to their own task.
$h_i = \{(\mathbf{s}_{i - c},\mathbf{a}_{i - c}),\ldots ,(\mathbf{s}_{i - 1},\mathbf{a}_{i - 1})\}$ . Note that it is easy to augment our offline dataset such to now include such histories  $\mathcal{D} = \{\mathbf{s},\mathbf{a},r,\mathbf{s}',h\}$
We approximate the latent dynamics by learning an encoder  $f(\mathbf{z}|h)$  that uses the history of interactions to predict a distribution over representations of the human's latent strategy. Note that we use history instead of the human's previous strategy because we anticipate that history has enough information to reconstruct the latter. This greatly simplifies training the encoder. In practice, the encoder can be of the form  $f(\mathbf{z}|h) = \mathcal{N}(\mathbf{z};f_{\mu}(h),f_{\Sigma}(h))$ , where  $f$  is a neural network that outputs both the  $d$ -dimensional mean of  $\mathbf{z}$  as well as the  $d\times d$  covariance matrix.
Although we do not observe the ground truth representations of latent strategy, we can still learn these representations in an unsupervised way, since we know that they should be predictive of future states. Recall that the dynamics of Hi-MDPs are given by  $P(\mathbf{s}'|\mathbf{s},\mathbf{a},\mathbf{z})$ . We propose jointly training a decoder  $d(\mathbf{s}'|\mathbf{s},\mathbf{a},\mathbf{z})$  that estimates the transition dynamics and reconstructs the next state. We ultimately propose the following joint training objective:
$$
\arg \max  _ {f, d} \mathbb {E} _ {\substack {\mathbf {s}, \mathbf {a}, \mathbf {s} ^ {\prime}, h \sim \mathcal {D} \\ \mathbf {z} \sim f (\mathbf {z} | h)}} [ \log d (\mathbf {s} ^ {\prime} \mid \mathbf {s}, \mathbf {a}, \mathbf {z}) ] - \mathbb {E} _ {h \sim \mathcal {D}} [ D _ {\mathrm {K L}} (f (\mathbf {z} \mid h) \mid p _ {0} (\mathbf {z})) ]. \tag{1}
$$
The first term in (1) is the reconstruction loss of the next state, and the second term is an information bottleneck [2] that regularizes the encoder towards prior  $p_0(\mathbf{z})$ . In our work, we use predicted distribution for the previous timestep as the prior, starting with  $p_0(\mathbf{z}) = \mathcal{N}(0,I)$ . By doing so, the inferred representations from the encoder are encouraged to be sequentially consistent across time, discouraging abrupt changes in behavior. Note that Xie et al. [34] propose a similar objective to model other agents in the environment, but only consider the reconstruction loss because each agent's latent strategy is fixed within an episode.
6.2 Offline RL Conditioned on Latent Strategy
Recall the example of directing the human towards picking up tomatoes, where the agent's strategy to influence the human depends on how the human behaves. This naturally means that we should condition the agent's policy on the human's latent strategy. We therefore train a policy parameterized as  $\pi (\mathbf{a}|\mathbf{s},\mathbf{z})$ , where  $\mathbf{z}\sim f(\mathbf{z}|h)$  is the representation of latent strategy sampled by the encoder  $f$  from Section 6.1. By doing so, the agent utilizes the learned dynamics of human strategy, and reasons how to influence the human's strategy towards behaviors that better maximize reward.
Our algorithm is a modification of CQL. Namely, because the learned policy  $\pi$  additionally conditions on human behavior, so does the Q-function  $Q(\mathbf{s},\mathbf{a},\mathbf{z})$ . Our modified actor-critic algorithm learns Q-functions via an objective similar to CQL, but where  $\mathbf{z}$  is sampled from the encoder, and serves as an additional input to the Q-function and policy.
6.3 Experiment
Figure 8: Layouts Open Asymmetric Advantages (left) and Counter Circuit (right).
Task description. We again evaluate our described approach in the Overcooked domain. We consider a different set of layouts, which we call Open Asymmetric Advantages and Counter Circuit (Figure 8). In contrast to the layouts in Figure 3, these layouts allow for players to block each other,
which we view as an influence strategy that requires knowing the partner's behavior. We consider the modified task where a soup that is delivered containing only tomatoes yields double the reward of other soups. We do so because this encourages influence by the ego agent on their human partner. If the partner appears to be going to pick up onions, the agent should try to redirect their focus (i.e., by blocking the path towards the onions).
Data Collection. For each layout, we collect a dataset of 20 human-human trajectories of length  $H = 1,200$ . We inform the humans that is the ego agent about the modified objective, and let the other human know that they are missing information. This biases the ego agent to try various influence strategies, and for their partner to respond accordingly.
Baselines. We compare our proposed behavior-conditioned offline RL approach, which we call latent offline  $RL$ , to several baselines. The first two are standard BC and offline RL, which simply condition on the current state and do not try to infer the human's behavior. The remaining baseline, which we call memory offline  $RL$ , implicitly conditions on the human's behavior by conditioning on a history of states. The difference between this baseline and what we propose is that memory offline  $RL$  does not have an additional objective to learn representations of latent strategy as we propose in Section 6.1. Our final baseline is an offline variant of the LILI algorithm [34], which we dub offline LILI. We choose LILI as a baseline because they similarly model human behavior by learning latent representations. However, LILI assumes (1) access to online data and an environment simulator and (2) that changes in behavior solely happen between episode. Thereofre, we modify LILI to match our problem statement as follows: (1) we discard the recollection of data steps in the LILI algorithm, and (2) we re-estimate the human strategy every  $c$  timesteps within an episode.
Evaluation. We deployed each policy alongside a human player, using the same 10 human users for the evaluation in Section 5.1. We collected on 30 trajectories of length  $H = 400$ .
In Figure 10 (left), we report the mean and standard error of reward across all trajectories in both layouts when the learned agents are paired with human partners. We notice that taking into account human behavior in some way is beneficial, as both memory offline RL and latent offline RL greatly outperformed naive BC and offline RL. Surprisingly, our approach (latent offline RL) noticeably improved upon memory offline RL, even though both were given the same amount of information. We hypothesize that disentangling the learning of human behavior and optimal influence leads to more effective training when the policies are neural networks. Our method also improves greatly over offline LILI due to modeling more nuanced transitions in behavior.
Figure 9: Improvement in reward (taken as difference in cumulative reward between last 100 and first 100 timesteps) when a human is paired with our agent (latent offline RL).
In addition, we provide evidence that when agents are successful, it is due to the human partner changing their behavior. In Figure 10 (right), we evaluate all the considered agents when paired with a scripted policy that unconditionally chooses the suboptimal action, e.g., picking up onions when tomatoes are superior, and cannot be influenced to do otherwise. We see that since all evaluated approaches
roughly perform the same, we rule out the possibility that agents learned via our proposed approach are solving tasks more successfully in isolation. Then, in Figure 9, we show that the mean reward our learned agent achieves with a human partner in the last 100 timesteps greatly exceeds that of the first 100 timesteps; since our learned agent does not change behavior unless their partner does, this shows that the human partner was successfully influenced towards more optimal coordination. Qualitatively, in Figure 7, we show that our proposed latent offline RL learns an adaptive influencing strategy to influence the human partner to pick up tomatoes in the Open Asymmetric Advantages layout. In the trajectory, the learned agent blocks access to the onions, only until it infers that the human is not going for them.
Humans often behave suboptimally, either due to systematic biases or incomplete knowledge of the environment. In this paper, we investigate whether offline RL on human-human interactions is capable of learning policies that can influence humans toward more desirable behavior. First, we showed that given a dataset with enough variability in human behaviors, existing model-free offline
Figure 10: Average rewards over 30 trajectories of  $H = 400$  timesteps of learned agents paired with real humans (left) and with a scripted policy that will always pick up onions (right). Note that the scripted policy cannot be influenced.
RL algorithms can learn to intentionally perform new influence strategies that are not explicitly performed by the humans in the dataset. However, humans will often adapt their policy based on past experiences, meaning agents need to recognize how the human is currently behaving, or their latent strategy, and then adapt influence strategies to account for changes in behavior. Therefore, we also proposed an offline RL algorithm that estimates and conditions on latent strategy, and show that it can successfully influence humans even when their latent strategy changes.
Limitations and future work. We focus our evaluation on the Overcooked game, because it has interpretable rules and dynamics, while capturing the fundamental challenges to learning human influence. However, it is still substantially simpler than real-world applications that require collaboration with humans. An important direction of further investigation is to investigate whether our findings hold in more realistic domains (e.g., dialogue systems).
In this work, we separately consider two challenges of human influence: learning unseen influence strategies, and how to achieve long-term influence that co-adapts with changes in human policies. It remains to be seen if offline RL can solve both challenges simultaneously, in one end-to-end approach – from diverse human-human interactions, learn new strategies to influence human actions, then extend these strategies (potentially in simulation or online) to achieve long-term influence.
Ethical implications. Perhaps most importantly, the notion of influence and "improvement" of human behavior is a double-edged sword. On the one hand, humans behavior is suboptimal and people might lack knowledge that agents have, and so agents can serve implicitly as coaches that help people arrive at better strategies. On the other hand, when agents are themselves lacking knowledge (especially of human values and preferences), influencing people to perform "better" under the wrong objective or wrong world state can very much hinder their performance. This is painfully obvious in recommender systems, where influence with the wrong objective leads to much wasted time on social medial platforms and, worse, polarization and other negative societal effects. Endowing the agent with proper uncertainty is a crucial next step for this line of work.
[1] Mohammad Mehdi Afsar, Trafford Crump, and Behrouz H. Far. Reinforcement learning based recommender systems: A survey. CoRR, abs/2101.06286, 2021.
[2] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
[3] DeepMind AlphaStar. Mastering the real-time strategy game starcraft ii. URL: https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii.
[4] Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 2022.
[5] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885-890, 2019.
[23] Dylan P. Losey, Mengxi Li, Jeannette Bohg, and Dorsa Sadigh. Learning from my partner's actions: Roles in decentralized robot teams. In Conference on Robot Learning (CoRL), 2019.
[24] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, 2017.
[25] Dorsa Sadigh, Shankar Sastry, Sanjit A. Seshia, and Anca D. Dragan. Planning for autonomous cars that leverage effects on human actions. In Robotics: Science and Systems, 2016.
[26] Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine learning, 84(1-2):109-136, 2011.
[27] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
[28] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
[29] DJ Strouse, Kevin R. McKee, Matt M. Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In Advances in Neural Information Processing Systems, 2021.
[30] Suzanne C. Thompson. Illusions of control: How we overestimate our personal influence. Current Directions in Psychological Science, 1999.
[31] Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. AI Magazine, 2012.
[32] L. Wang, Wei Zhang, Xiaofeng He, and H. Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
[33] Woodrow Z. Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on Robot Learning, 2021.
[34] Annie Xie, Dylan P. Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference on Robot Learning, 2020.
[35] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pages 25611-25635, 2022.
[36] Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. CoRR, abs/1911.10635, 2019.