SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joey Hong Sergey Levine Anca Dragan
{joeyhong,sergey.levine,anca}@berkeley.edu
When interacting with people, AI agents do not just influence the state of the world – they also influence the actions people take in response to the agent, and even their underlying intentions and strategies. Accounting for and leveraging this influence has mostly been studied in settings where it is sufficient to assume that human behavior is near-optimal: competitive games, or general-sum settings like autonomous driving alongside human drivers. Instead, we focus on influence in settings where there is a need to capture human suboptimality. For instance, imagine a collaborative task in which, due either to cognitive biases or lack of information, people do not perform very well – how could an agent influence them towards more optimal behavior? Assuming near-optimal human behavior will not work here, and so the agent needs to learn from real human data. But experimenting online with humans is potentially unsafe, and creating a high-fidelity simulator of the environment is often impractical. Hence, we focus on learning from an offline dataset of human-human interactions. Our observation is that offline reinforcement learning (RL) can learn to effectively influence suboptimal humans by extending and combining elements of observed human-human behavior. We demonstrate that offline RL can solve two challenges with effective influence. First, we show that by learning from a dataset of suboptimal human-human interaction on a variety of tasks – none of which contains examples of successful influence – an agent can learn influence strategies to steer humans towards better performance even on new tasks. Second, we show that by also modeling and conditioning on human behavior, offline RL can learn to affect not just the human's actions but also their underlying strategy, and adapt to changes in their strategy.
In many AI applications, including games [27], healthcare [26, 32], recommender systems [1], and robotics [15], agents interact with people and end up influencing human behavior. Accounting for and leveraging this influence has mainly been studied in settings where it is sufficient to model human behavior as near-optimal [13, 8], like in games such as Go [27], or in autonomous driving settings where agents try to influence people to slow down and make space [25].
In contrast, we target settings where influence is important but real humans might not behave in strategically optimal or even rational ways, such as cooperative scenarios with other non-expert humans, or social scenarios that are not inherently strategic, like dialogue. For instance, imagine a human and robot collaborating to cook a meal. The human might start chopping the tomatoes to make a salad because there are tomatoes conveniently nearby, but it might be much more efficient to let the robot make the salad while the human plates the main course (because the robot is less adept at plating). To try to get the human to plate, the robot might put a plate right next to them to make the plating task more appealing, or it might even block the human from accessing the tomatoes. Such
Figure 1: Informative example of how an agent may deduce influencing strategies from offline data.
strategies would not be needed with optimally rational humans who understand precisely what they should do, and therefore they must account for the statistical properties of real suboptimal human behavior that allow humans to be influenced by the robot's behavior.
Prior work in influencing suboptimal behavior in humans has thus far approximated human behavior with scripted or heuristic policies [34, 16]. However, humans have a myriad of cognitive biases [30, 10] that can make their behavior sub-optimal. This makes hand-designing a simulator for how humans behave very difficult, as one cannot write heuristics for all these possible biases. In turn, it implies that learning influence will require relying on actual human data. But interacting with people online to figure out what works and what does not is unsafe or unrealistic in many domains. So how can we attain effective influence when given only a data set of (suboptimal) human-human interaction?
Our key observation is that an agent can learn influencing policies even when these are not demonstrated in the data. In particular, offline reinforcement learning (RL) can learn to influence by "stitching" or combining different components of human behavior seen in a diverse human-human interaction dataset. A simple but illustrative example that illustrates this in the collaborative cooking setting is in Figure 1: in some human-human interaction, a human makes a mistake and realizes that they want to go grab onions instead of plating, and drops the plate on the counter; in a (potentially different) interaction, a human may pick up the plate that is conveniently placed right next to them, abandoning their current next goal in favor of plating. By extending those two behaviors, an agent can glean that placing a plate next to the human might influence them towards plating. As an added benefit, this can happen without access to a simulator or model of the physical world.
The main contributions of our work is to demonstrate how offline RL can address two important challenges when it comes to influencing suboptimal human behavior. First, we show that existing offline RL methods can deduce strategies to influence a human's actions despite not seeing any such strategies in the training data, either because the humans were doing different tasks, did not attempt influence, or did not think of a successful influence strategy. Second, by augmenting offline RL algorithms with an estimate of the human's latent strategy, we can enable the RL agent to go beyond influencing human actions only, to influencing their actual strategy. For example, the RL agent might figure out that if it repeatedly blocks a person from reaching a particular ingredient, that person will change their strategy to avoid that ingredient in the future, even when it is no longer blocked, thus altering their strategy for the rest of the episode. Our work can be seen as expanding the potential of offline RL by showing how it can be leveraged in human interaction, trained from human-human data, and conducive to good performance when deployed alongside real humans.
Multi-agent RL. RL has been applied extensively to multi-agent settings, where multiple agents take actions in a competitive or cooperative game [31, 24, 14, 36]. A standard technique in MARL involves modeling the joint effect of the all agents' actions in the environment [11]. Many such methods utilize centralized training, which accounts for the actions of other agents through a centralized critic [24, 12]. Other related approaches add explicit communication channels between agents so that agents can share their policy parameters or gradient updates [7, 23]. These methods are typically concerned with jointly training a collection of autonomous agents. In contrast, our work focuses
on training agents that interact with humans, who are not under our direct control (and thus cannot access their policies), and often behave suboptimally or irrationally.
RL with humans. In recent years, increased attention has been directed toward designing agents that interact with humans. Early works in this setting train agents that play against humans in competitive games, such as Go [3], Poker [5], and Diplomacy [4], and aim to exceed the skill level of their human opponents. Such methods typically rely on computing best responses against a (near-) optimal agent. However, in many realistic tasks, modeling the human as near-optimal is insufficient, such as in collaborative or social settings [6]. We focus on tasks where modeling and improving suboptimal human behavior is important for success. Related to our work, Carroll et al. [6] investigated coordination with humans in the Overcooked environment. However, we deviate from this in several substantial ways. First, we focus on influence to improve suboptimal human behavior rather than simply accounting for it; we do this by considering tasks where an agent must make its human partner behave differently in order to succeed. Second, we do not assume access to an environment simulator, and must learn completely from prior human-human interactions.
Multi-agent influence. We investigate whether we can train agents that learn to influence humans to behave more optimally. Prior works have also considered using influence to improve competition or cooperation, and have proposed both model-free and model-based approaches. Model-free approaches anticipate how other agents update their policy without explicitly learning a model of the dynamics. Notably, in competitive games, LOLA [8] and its more recent improvements [9, 21] estimate the opponent's policy updates, and use it to inform the policy updates of the agent in a recursive manner. Model-based approaches, on the other hand, consider a dynamics model of how other agents' policies change. Existing works such as LILI do so practically by learning a low-level representation of other agents' behavior, and condition the learned agent's behavior on these representations [16, 33]. However, LILI is (1) trained and evaluated on scripted and heuristic policies, which display substantially less complex behavior than real humans, (2) require online interactions, and (3) assume behavior changes only between episodes. In this paper, we address all these limitations by proposing offline RL solely on prior human-human interactions, but can also learn agents that adapt its influence even when human behavior changes within episodes.
The goal in RL is to learn a policy  $\pi (\cdot |\mathbf{s})$  that maximizes the expected cumulative discounted reward in a Markov decision process (MDP), which is defined by a tuple  $(S,A,P,R,\gamma ,H)$ .  $S,A$  represent state and action spaces,  $P(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})$  and  $R(\mathbf{s},\mathbf{a})$  represent the dynamics and reward distribution,  $\gamma \in (0,1)$  represents the discount factor, and  $H$  (optionally) is the horizon of the episode.
RL with hidden information. In our problem formulation, we are concerned with tasks that require interacting another agent, such as a person, whose policy and behavior is unknown. We model the other agent as having some low-dimensional latent strategy,  $\mathbf{z} \in \mathcal{Z}$ , which determines their current behavior (e.g., goal, plan, etc.). We formulate this problem as a special case of a partially observable MDP (POMDP), which we refer to as a hidden MDP (Hi-MDP). A Hi-MDP is defined by  $(S, \mathcal{A}, \mathcal{Z}, P, P^z, R, \gamma, H)$ , where  $\mathcal{Z}$  is the space of internal states that represent the other agent's behavior,  $P(\mathbf{s}'|\mathbf{s}, \mathbf{a}, \mathbf{z})$  additionally depends on the latent behavior, and  $P^z(\mathbf{z}'|\mathbf{s}, \mathbf{a}, \mathbf{z})$  represents the dynamics. We refer to the agent controlled by the learned policy in the Hi-MDP as the ego agent. A similar MDP is studied in Xie et al. [34], but consider a  $\mathbf{z}$  that is fixed within a single episode.
Offline RL Our approach employs offline RL to learn coordination strategies from data, without requiring human interaction or simulation. In offline RL, we are given access to a static dataset consisting of tuples  $\mathcal{D} = \{(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\}$ . Standard model-free offline RL algorithms are often actor-critic algorithms, which attempt to learn a  $Q$ -function  $Q^{\pi}:S\times \mathcal{A}\to \mathbb{R}$  of policy  $\pi$  at all state-action pairs  $(\mathbf{s},\mathbf{a})\in S\times \mathcal{A}$ , Specifically, for a policy  $\pi$ , its Q-value  $Q^{\pi}(\mathbf{s},\mathbf{a}) = \mathbb{E}_{\pi}\left[\sum_{t = 0}^{\infty}\gamma^{t}r_{t}\right]$  is its expected mean return starting from that state and action. The Q-function is the unique fixed point of the Bellman operator  $\mathcal{B}^{\pi}$  given by:  $\mathcal{B}^{\pi}Q(\mathbf{s},\mathbf{a}) = r(\mathbf{s},\mathbf{a}) + \gamma \mathbb{E}_{s^{\prime}\sim P(s^{\prime}|\mathbf{s},\mathbf{a}),\mathbf{a}^{\prime}\sim \pi (\mathbf{a}^{\prime}|\mathbf{s}^{\prime})}[Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime})]$ , meaning  $Q^{\pi} = \mathcal{B}^{\pi}Q^{\pi}$ . In the offline setting, where we are limited to interactions that appear in the dataset  $\mathcal{D}$ , we do not have access to the true Bellman operator  $\mathcal{B}^{\pi}$ . Instead, we use the empirical Bellman operator that backs up a single sample  $(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\in \mathcal{D}$ , denoted as  $\hat{B}^{\pi}$ . In an actor-critic algorithm, a separate policy  $\pi$  is trained to maximize the Q-value of its chosen actions. To do this, actor-critic methods switch between updating a policy-dependent Q-function
$Q^{\pi}$  via:  $\hat{Q}^{k + 1} = \arg \min_{Q}\mathbb{E}_{\mathbf{s},\mathbf{a}\sim \mathcal{D}}\left[\left(Q(\mathbf{s},\mathbf{a}) - \hat{B}^{\hat{\pi}_k}\hat{Q}^k (\mathbf{s},\mathbf{a})\right)^2\right]$  and improving the policy  $\pi (\mathbf{a}|\mathbf{s})$  by solving for:  $\hat{\pi}^{k + 1} = \arg \max_{\pi}\mathbb{E}_{\mathbf{s}\sim \mathcal{D},\mathbf{a}\sim \pi_k(\mathbf{a}|\mathbf{s})}\left[\hat{Q}^{k + 1}(\mathbf{s},\mathbf{a})\right]$ . Standard RL algorithms such as Q-learning suffers from distributional shift [17, 22] because the actions in  $\mathcal{D}$  are derived from some likely suboptimal policy. In our paper, we use conservative Q-learning (CQL) [18], which additionally penalizes Q-values on out-of-distribution actions when learning the Q-function:
$$
\hat {Q} ^ {k + 1} = \arg \min _ {Q} \mathbb {E} _ {\mathbf {s}, \mathbf {a} \sim \mathcal {D}} \left[ \left(Q (\mathbf {s}, \mathbf {a}) - \hat {\mathcal {B}} ^ {\hat {\pi} _ {k}} \hat {Q} ^ {k} (\mathbf {s}, \mathbf {a})\right) ^ {2} \right] + \alpha \left(\mathbb {E} _ {\mathbf {s} \sim \mathcal {D}} \left[ \log \sum_ {\mathbf {a}} \exp (Q (\mathbf {s}, \mathbf {a})) \right] - \mathbb {E} _ {\mathbf {s}, \mathbf {a} \sim \mathcal {D}} [ Q (\mathbf {s}, \mathbf {a}) ]\right).
$$
In this work, we use CQL to accomplish cooperative tasks that involve influencing other agents in the environment. However, our work is not specific to CQL or any particular offline RL method, and could well be adapted to other effective offline RL methods.
4 Human Influence in the Overcooked Environment
We hypothesize that offline RL can learn how to influence suboptimal human behavior, using solely a dataset of human-human interactions. We test this hypothesis on an environment that is a simplified version of the game Overcooked, originally proposed in Carroll et al. [6]. We choose this environment due to its popularity in human-aware RL, and because humans are known to behave suboptimally due to miscoordination with their partner.
In this environment, two players are placed into an environment consisting of 2D tiles, with objects such as stoves, ingredients (tomatoes or onions), and obstacles, and tasked with delivering as many cooked dishes composed of either tomatoes or onions as possible within an episode. This involves a series of sequential high-level actions to which both players can contribute: collecting ingredients, depositing them into cooking pots, letting the ingredients cook into soup, collecting a plate, getting the soup on the plate, and delivering it.
Each player observes an egocentric view of the world that includes the layout of the kitchen and
Figure 2: Illustration of the Overcooked environment. The goal is to place three ingredients (tomatoes or onions) in a pot, plate the soup from the pot, and deliver it, as many times as possible within the time limit. The ego agent is the one in the blue hat.
location of the other player, and at every step can perform one of six actions: stand still, move {up, down, left, right}, interact. The effect of interact varies based on the cell which the player is facing. To effectively complete the task, players must navigate the kitchen and interact with objects in the correct order, all while coordinating with what their partner is doing. While the dynamics of the environment are relatively simple, the introduction of real humans as partners allows for complex and diverse influence strategies that cannot be found in prior works that use scripted or heuristic policies [34, 16, 33].
In this work, we consider two complementary challenges of learning to influence suboptimal human behavior in this environment and propose how offline RL can address them. We describe and motivate both challenges below, and defer details of how we assess whether offline RL succeeds at each challenge in Section 5 and 6, respectively.
Challenge 1: Deducing New Influence Strategies. While it is relatively straightforward to collect human-human interactions in the Overcooked environment, it is comparatively hard to observe effective influence strategies. This is because in this environment, humans are passive and will often only respond to what their partner is doing, e.g., by completing the next available task, and not try to develop more effective coordination by influencing their partner's behavior. For instance, imagine that the agent knows its human partner is much better suited to plating and delivering soups, e.g.,
due to being closer to the delivery station. An intelligent agent should try to elicit this behavior, potentially by trying to "pass" the plate to its partner. However, because in this setting, humans do not tend to intentionally try to change what their partner is doing, this specific interaction may not appear in the data. This means that influence strategies must be deduced, by extending and generalizing the suboptimal human behaviors found in the data.
We hypothesize that naive offline RL algorithms can learn new influence strategies, even using a dataset of diverse suboptimal behavior that contains no evidence of influence, by combining, or "stitching", components of human behaviors to reason about unseen strategies. This is done by identifying new ways to reach intermediate states that will result in successfully changing their partner's behavior, by propagating the reward signal from trajectories that contain the desired behavior through the Q-function for those states. Recall the example of influencing a human to plate a soup. The dataset may not contain trajectories where plate is passed between humans, but instead contains ones where a human puts a plate on the counter, and (potentially different) ones where the human partner picks up a nearby plate. Then, offline RL can learn that moving a plate to a counter near the human may influence them to pick it up and plate the soup.
Challenge 2: Long-term Influence of Latent Strategies. When influencing humans, it is often not sufficient to simply learn an influence strategy and apply it unconditionally on the human partner. Rather than getting the human to execute particular actions, it can be more desirable to achieve long-term influence by changing their underlying policy, or latent strategy. This is also true in the Overcooked environment. For example, consider the case where the agent wants the human to always transfer tomatoes to the pot rather than onions. By explicitly blocking the human from reaching the onions for an extended period of time, the human may adapt its policy and decide to only pick up tomatoes in the future. Furthermore, the agent should recognize that the human's policy has now changed, and deem that blocking the onions any further is a waste of resources. This means that effective influence requires knowing how to achieve long-term sway on the human's latent strategy.
It is evident that naive offline RL cannot learn adaptive policies. But via a natural and simple modification, i.e., modeling the human's latent strategy, and training policies conditioned on that strategy, we hypothesize that offline RL can train agents that can influence and adapt to changes in the human's latent strategy. We believe this can be done even on data where humans overwhelmingly fail to coordinate successfully, likely due to incorrectly predicting their partner's behavior. This is because offline RL can still use trajectories of failed coordination to identify the different behaviors that are present and how to optimally respond to them.
5 Learning Influence Strategies from Diverse Behaviors
Our goal in this section is to demonstrate that an agent trained using offline RL can deduce new strategies to influence and improve human behavior, by smartly recombining parts of behaviors seen in the data. The setting that we consider in this set of experiments is one where the agent utilizes diverse prior data to solve a task, even when the prior data does not actually succeed at solving it. Offline RL has shown success at generalizing behaviors to new tasks in robotic manipulation [28], but not yet to improve human behavior.
We have a very simple goal: use offline RL to learn a policy that knows how to influence humans behaving suboptimally to take better actions. As described in Section 3, we use CQL [18] as our offline RL algorithm due to its effectiveness in reducing distribution shift. We defer implementation details, i.e., architecture and hyperparameter choices, to Appendix A.
One of the main benefits of offline RL over imitation learning is its ability to leverage diverse prior data to generalize to solving new tasks, even when the data does not contain any complete examples for the new tasks [28, 19, 35]. In our work, we also collect and train on a multi-objective dataset, but unlike prior works, our dataset consists entirely of episodes where humans interact with other humans. Different objectives are created by changing the reward function of the environment. Such data can be used for generalization by simply relabeling the rewards to match that of the new task.
5.1 Experiment
Task description. We empirically evaluate our approach in the Overcooked domain. We consider two different layouts, which we call Asymmetric Advantages and Forced Coordination (Figure 3), that often require coordination in the form of passing ingredients to improve efficiency. The RL-controlled agent is the one with the blue hat.
Most prior works on Overcooked both train and evaluate on the standard task [6, 20]. In this work, we want to demonstrate that offline RL can accomplish something more complex - we can train an agent from multi-task data to solve new tasks by influencing its human partner in clever ways, even when the prior data contains no evidence of intentional and successful
Figure 3: Layouts Asymmetric Advantages (left) and Forced Coordination (right).
influence for the new task. Our new tasks use a modified reward function, e.g., soups with tomatoes are more valuable to the customers than with onions, such that humans who are optimizing the standard reward (which rewards any type of soup equally) behave very suboptimally. Due to this suboptimality, it is important for the learned agent to influence its human partner, e.g., to pick up tomatoes instead of onions. We evaluate on two new tasks using the aforementioned layouts: Asymmetric Advantages (Human Deliver) offers doubled reward if the soup is delivered specifically by the human partner of the ego agent, and Forced Coordination (Tomato Bonus) offers doubled reward if the soup contains only tomatoes.
Data collection. We collected a dataset of human-human play where the human players were provided with one of several different instructions, in order to gather a diverse dataset that illustrates a variety of behaviors and human-human interactions. In the first set of instructions, humans are playing the game under the standard objective. We collected 20 human-human trajectories of length  $H = 1$ , 200. In the second, humans are playing the game but one of the humans is given the additional task of moving ingredients to counters for a small reward; interestingly, the other human is not aware of this objective, which induces various suboptimalities because they cannot predict their partner's actions. Under this modified objective, we collect 5 human-human trajectories of length  $H = 1$ , 200.