Learning to Influence Human Behavior with Offline Reinforcement Learning
Authors: Joey Hong
Sergey Levine
Anca Dragan
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
human-aware reinforcement learning
multi-agent influence
TL;DR: We learn agents that can influence suboptimal human behavior using purely offline RL on human-human interactions.

Abstract: When interacting with people, AI agents do not just influence the state of the world -- they also influence the actions people take in response to the agent, and even their underlying intentions and strategies. Accounting for and leveraging this influence has mostly been studied in settings where it is sufficient to assume that human behavior is near-optimal: competitive games, or general-sum settings like autonomous driving alongside human drivers. Instead, we focus on influence in settings where there is a need to capture human suboptimality. For instance, imagine a collaborative task in which, due either to cognitive biases or lack of information, people do not perform very well -- how could an agent influence them towards more optimal behavior? Assuming near-optimal human behavior will not work here, and so the agent needs to learn from real human data. But experimenting online with humans is potentially unsafe, and creating a high-fidelity simulator of the environment is often impractical. Hence, we  focus on learning from an offline dataset of human-human interactions. Our observation is that offline reinforcement learning (RL) can learn to effectively influence suboptimal humans by extending and combining elements of observed human-human behavior. We demonstrate that offline RL can solve two challenges with effective influence. First, we show that by learning from a dataset of suboptimal human-human interaction on a variety of tasks -- none of which contains examples of successful influence -- an agent can learn influence strategies to steer humans towards better performance even on new tasks. Second, we show that by also modeling and conditioning on human behavior, offline RL can learn to affect not just the human's actions but also their underlying strategy, and adapt to changes in their strategy.

Learning to Influence Human Behavior with Offline Reinforcement Learning
Joey Hong
Sergey Levine Anca Dragan
UC Berkeley
[joey hong, sergey wevineeieene @berkeel. edu
Abstract
When interacting with people, AI agents do not just influence the state of the world they also influence the actions people take in response to the agent, and even their underlying intentions and strategies. Accounting for and leveraging this influence has mostly been studied in settings where it is sufficient to assume that human behavior is near-optinnn opttinal:l::: competitive games, or general-sum settings like autonomous driving alongside human drivers. Instead, focus influence in settings where there is a need to capture human suboptimality. For instance, imagine collaborative task in which, due either cognitive biases lack of information, people do not perform very well - how could an agent influence them towards more optimal behavior? Assuming near-optimal human behavior will not work here, and so the agent needs to learn from real human data. But experimenting online with humans is potentially unsafe, and creating a high-fidelity simulator of the environment is often impractical. Hence, we focus on learning from an offline dataset of numan-humas interactions. Our observation is that offline reinforcement learning (RL) can learn to effectively influence suboptimal humans by extending and combining elements of observed human-human behavior. We demonstrate that offline RI. can solve two challenges with effective influence. First, we show that by learning from a dataset of suboptimal numan-human interaction on a variety of tasks none of which contains examples of successful influence an agent can learn influence strategies to steer humans towards better performance even on new tasks. Second, we show that by also modeling and conditioning on human behavior, offline RL can learn affect not just the human's actions but also their underlying strategy, and adapt to changes in their strategy.
Introduction
In many AI applications, including games [27], healthcare [26, 32), recommender systems [1], and robotics [15], agents interact with people and end up influencing human behavior. Accounting for and leveraging this influence has mainly been studied in settings where it is sufficient to model human behavior as near-optimal [13, 8], like in games such as Go [27], or in autonomous driving settings where agents try to influence people to slow down and make space [25].
In contrast, we target settings where influence is important but real humans might not behave in strategically optimal or even rational ways, such as cooperative scenarios with other non-expert humans, or social scenarios that are not inherently strategic, like dialogue. For instance, imagine human and robot collaborating to cook meal. The human might start chopping the tomatoes to make a salad because there are tomatoes conveniently nearby, but might be much more efficient to let the robot make the salad while the human plates the main course (because the robot is less adept at plating). To try to get the human to plate, the robot might put a plate right next to them to make the plating task more appealing, or it might even block the human from accessing the tomatoes. Such
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fess poing poyleet. Iiv Frow resfizr sead to anal oxioon aralnad.
Faoee plate ahap ofler angra shor.
an The place asag right gaara I'm soing toplate this seal
drop plase
Vanny plac
get place
for
Ituran-lteran Dara
Offine RL.
Figure I: Informative example of how an agent may deduce influencing strategies from offline data.
strategies would not be needed with optimally rational humans who understand precisely what they should do, and therefore they must account for the statistical properties of real suboptimal human behavior that allow humans to be influenced by the robot's 's behavior.
Prior work in inftuencing suboptimal behavior in humans has thus far approximated human behavior with scripted or heuristic policies [34, 16]. However, humans have myriad of cognitive biases [30, 10] that make their behavior sub-optimal. This makes hand-designing simulator for how humans behave very difficult, as one cannot write heuristics for all these possible biases. In turn, implies that learning influence will require relying on actual human data. But interacting with people online to figure out what works and what does not is unsafe or unrealistic in many domains.So how can we attain effective influence when given only a data set of (suboptimal) human-humar interaction?
Our key observation is that an agent can learn influencing policies even when these are not demon- strated in the data. In particular, offline reinforcement learning (RL) can learn to influence by "stitching" combining different components of human behavior seen in diverse human- human interaction dataset. A simple but illustrative example that illustrates this in the collaborative cooking setting is in Figure 1: in some numan-human interaction, human makes mistake and realizes that they want to go grab onions instead of plating, and drops the plate on the counter; in a (potentially different) interaction, human may pick up the plate that is conveniently placed right next them, abandoning their current next goal in favor of plating. By extending those two behaviors, an agent can glean that placing a plate next to the human might influence them towards plating. As an added benefit, this can happen without access to simulator or model of the physical world.
The main contributions of our work is to demonstrate how offline RL can address two important challenges when it comes to influencing suboptimal human behavior. First, we show that existing offline RL methods can deduce strategies to influence human's 's actions despite not seeing any such strategies in the training data, either because the humans were doing different tasks, did not attempt influence, did algorithms with an estimate of the human's latent strategy, we can enable the RL agent to go beyond influencing human actions only, to influencing their actual strategy. For example, the RL agent might figure out that if it repeatedly blocks person from reaching a particular ingredient, that person will change their strategy to avoid that ingredient in the future, even when it no longer blocked, thus altering their strategy for the rest of the episode. Our work can be seen as expanding the potential of offline RL by showing how can be leveraged in human interaction, trained from data, and conducive to good performance when deployed alongside real humans.
think of successful influence strategy. Second, by augmenting offline RL
2 Related Work
Multi-agent RL. RL has been applied extensively to multi-agent settings, where multiple agents take actions in a competitive or cooperative game [31, 24, 14, 36). A standard technique in MARL involves modeling the joint effect of the all agents' actions in the environment [11]. Many such methods utilize centralized training, which accounts for the actions of other agents through a centralized critic [24, 12]. Other related approaches add explicit communication channels between agents so that agents can share their policy parameters or gradient updates [7, 23]. These methods are typically concerned with jointly training a collection of autonomous agents. In contrast, our work focuses

---

on training agents that interact with humans, who are not under our direct control (and thus cannot access their policies), and often behave suboptimally or irrationally.
RL with humans. In recent years, increased attention has been directed toward designing agents that interact with humans. Harly works in this setting train agents that play against humans in competitive games, such as Go [3], Poker [5], and Diplomacy [4], and aim to exceed the skill level of their human opponents. Such methods typically rely on computing best responses against (near-) optimal agent. However, in many realistic tasks, modeling the human as near-optimal is insufficient, such as in collaborative or social settings [6]. We focus on tasks where modeling and improving suboptimal human behavior is important for success. Related to our work, Carroll et al. [6] investigated coordination with humans in the Overcooked environment. However, we deviate from this in several substantial ways. First, we focus on influence to improve suboptimal human behavior rather than simply accounting for it; we do this by considering tasks where an agent must make its human partner behave differently in order to succeed. Second, we do not assume access to an environment simulator, and must learn completely from prior human-humas interactions.
Multi-agent influence. We investigate whether we can train agents that learn to influence humans to behave more optimally. Prior works have also considered using influence to improve competition or cooperation, and have proposed both model-free and model-based approaches. Model-free approaches anticipate how other agents update their policy without explicitly learning a model of the dynamics. Notably, in competitive games, LOLA [8] and its more recent improvements [9. 21] estimate the opponent's 's policy updates, and use it to inform the policy updates of the agent in recursive manner. Model-based approaches, on the other hand, consider a dynamics model of how other agent's policies change. Existing works such as LILI do so practically by learning a low-level representation of other agents' behavior, and condition the learned agent's behavior on these representations [16, 33]. However, LILI is (1) trained and evaluated on scripted and heuristic policies, which display substantially less complex behavior than real humans, (2) require online interactions, and (3) assume behavior changes only between episodes. In this paper, we address all these limitations by proposing offline RL solely on prior human- -hmmnn interactions, but can also learn agents that adapt its influence even when human behavior changes within episodes.
3 Preliminaries
The goal in RL is to learn policy rT-ss) that maximizes the expected cumulative discounted reward in Markov decision process (MDP), which is defined by tuple (8, P, R, y, HH S, A represent state and action spaces, P(s'ls (s. a) and R(s, a) represent the dynamics and reward distribution, y E (0, 1) represents the discount factor, and H (optionally) is the horizon of the episode.
RL with hidden information. In our problem formulation, we are concerned with tasks that require interacting another agent, such as a person, whose policy and behavior is unknown. We model the other agent as having some low-dimensional latent strategy, z E Z, which determines their current behavior (e.g., goal, plan, etc.). We formulate this problem as special case of a partially observable MDP (POMDP), which we refer to as a hidden MDP (Hi-MDP). A Hi-MDP is defined by (S. Z, P. Pa R. y, H.), where 2 is the space of internal states that represent the other agent's s behavior, P(s'|s, z) additionally depends on the latent behavior, and Pa (z'|s a, z) represents the dynamics. We refer to the agent controlled by the learned policy in the Hi-MDP as the ego agent. similar MDP is studied in Xie et al. [34], but consider a that is fixed within single episode.
Offline RL Our approach employs offline RI. to learn coordination strategies from data, without requiring human interaction or simulation. In offline RL, we are given access
static dataset consisting of tuples D {(s, a, r, s') Standard model -ree offline RL algorithms are often actor- critic algorithms, which attempt to learn a Q-function Q S x A + R of policy т at all state-action pairs (s.a) a) S x A, Specifically, for a policy म, its Q-value (s. expected mean return starting from that state and action. The Q-function is the unique fixed point of the Bellman operator 8a given by: Q a) = r(s a) + >E.,. (tt_://.lls.s..aaaaa... a(aaa((()))ss)) [Q(s', a')] meaning Q' = 8 TºQ.. In the offline setting, where we are limited to interactions that appear in the dataset D. we do not have access to the true Bellman operator 8" Instead, we use the empirical Bellman operator that backs up a single sample as,,, E D. denoted as B². In an actor-critic algorithm, a separate policy π is trained to maximize the Q-value of its chosen actions. To do this, actor-critic methods switch between updating policy- deeeeneet Q-function
E, (_ "'re is its
3

---

- Qa via: Qh+1 arg minQ E (Q(s.a) B&h Q" @* a))
and improving the policy T T(ass)
by solving for: #*+1 arg max; (anil)) [2++] (s, a) Standard RI. algorithms such as
Q-learning suffers from distributional shift [17, 22] because the actions in D are derived from some likely suboptimal policy. In our paper, we conservative Q-learning (CQL) [18], which additionally penalizes Q-values on out-of-distribution actions when learning the Q-function:
= min B...D (Q(s..)) 8* Q" (s.
+o
log Σ exp(Q(s, ()) - EEs (Q(s. a)]
this work, we use CQL to accomplish cooperative tasks that involve influencing other agents in the environment. However, our work is not specific to CQL any particular offline RL method, and could well be adapted to other effective offline RL methods.
Human Influence in the Overcooked Environment
We hypothesize that offline RL can learn how to influence suboptimal human behavior, using solely a dataset of noman-human interactions. test this hypothesis on an environment that simplified version of the game Overcooked, originally proposed in Carroll et al. [6]. We choose this environment due to its popularity in human-awaan awaae RI,, and because humans are known behave suboptimally due to miscoor- dination with their partner.
Pots
Tomaloes
Dishes
Onions
In this environment, two players are placed into an environment consisting of 2D tiles, with ob- jects such as stoves, ingredients (tomatoes or onions), and obstacles, and tasked with deliv- ering many cooked dishes composed of ei- ther tomatoes or onions as possible within an episode. This involves series of sequential high-level actions to which both players can con- tribute: collecting ingredients, depositing them into cooking pots, letting the ingredients cook into soup, collecting plate, getting the soup the plate, and delivering it.
Delivery
Piak Ingrodients
Meve three pol
Plate
cooli
Figure 2: Illustration of the Overcooked environ- ment. The goal to place three ingredients (toma- toes onions) in pot, plate the soup from the pot, and deliver as many times as possible within the time limit. The ego agent is the one in the blue hat.
Each player observes egocentric view of the world that includes the layout of the kitchen and location of the other player, and at every step perform one of six actions: stand still, move (up, down, left, right). interact. The effect of interact varies based on the cell which the player is facing. To effectively complete the task, players must navigate the kitchen and interact with objects in the correct order, all while coordinating with what their partner doing. While the dynamics of the environment are relatively simple, the introduction of real humans as partners allows for complex and diverse influence strategies that cannot be found in prior works that use scripted or heuristic policies [34, 16, 33).
In this work, we consider two complementary challenges learning to influence suboptimal human behavior in this environment and propose how offline RL can address them. We describe and motivate both challenges below, and defer details of how we assess whether offline RL succeeds at each challenge in Section 5 and 6, respectively.
Challenge Deducing New Influence Strategies. While it is relatively straightforward collect numan-humal interactions in the Overcooked environment, it is comparatively hard to observe effective influence strategies. This is because in this environment, humans are passive and will often only respond to what their partner is doing, e,g. by completing the next available task, and not try to develop more effective coordination by influencing their partner's behavior. For instance, imagine that the agent knows its human partner is much better suited to plating and delivering soups, e.g.,

---

due to being closer to the delivery station. An intelligent agent should try to elicit this behavior, potentially by trying to "pass" the plate to its partner. However, because in this setting, humans do not tend to intentionally try to change what their partner is doing, this specific interaction may not appear in the data. This means that influence strategies must be deduced, by extending and generalizing the suboptimal human behaviors found in the data.
We hypothesize that naive offline RL algorithms can learn new influence strategies, even using a dataset of diverse suboptimal behavior that contains no evidence of influence, by combining, or "stitching", components of human behaviors to reason about unseen strategies. This is done by identifying new ways to reach intermediate states that will result in successfully changing their partner's s behavior, by propagating the reward signal from trajectories that contain the desired behavior through the Q- D-function for those states. Recall the example of influencing a human to plate soup. The dataset may not contain trajectories where plate is passed between humans, but instead contains ones where a human puts plate on the counter, and (potentially different) ones where the human partner picks up a nearby plate. Then, offline RL. can learn that moving plate to a counter near the human may influence them pick it up and plate the soup.
Challenge 2: Long-term Influence of Latent Strategies. When influencing humans, it is often not sufficient to simply learn an influence strategy and apply it unconditionally on the human partner. Rather than getting the human to execute particular actions, can be more desirable to achieve long-term influence by changing their underlying policy, or latent strategy. This is also true in the Overcooked environment. For example, consider the case where the agent wants the human always transfer tomatoes to the pot rather than onions. By explicitly blocking the human from reaching the onions for an extended period of time, the human may adapt its policy and decide to only pick up tomatoes in the future. Furthermore, the agent should recognize that the human's 's policy has now changed, and deem that blocking the onions any further is a waste of resources. This means that effective influence requires knowing how to achieve long-term sway on the human's 's latent strategy.
It is evident that naive offline RL cannot learn adaptive policies. But via natural and simple modification, i.e., modeling the human's 's latent strategy, and training policies conditioned on that strategy. we hypothesize that offline RL can train agents that can influence and adapt changes in the human's 's latent strategy. We believe this can be done even on data where humans overwhelmingly fail to coordinate successfully, likely due to incorrectly predicting their partner's s behavior. This is because offline RL can still use trajectories of failed coordination to identify the different behaviors that are present and how to optimally respond to them.
5 Learning Influence Strategies from Diverse Behaviors
Our goal in this section is to demonstrate that an agent trained using offline RL can deduce new strategies influence and improve human behavior, by smartly recombining parts of behaviors seen in the data. The setting that we consider in this set of experiments is one where the agent utilizes diverse prior data solve task, even when the prior data does not actually succeed at solving it. Offline RL has shown success at generalizing behaviors to new tasks in robotic manipulation [28], but not yet to improve human behavior.
We have a very simple goal: use offline RL to learn a policy that knows how to influence humans behaving suboptimally to take better actions. As described in Section 3, use CQL [18] as offline RL algorithm due to its effectiveness in reducing distribution shift. We defer implementation details, i.e., architecture and hyperparameter choices, to Appendix A.
One of the main benefits of offline RL over imitation learning is its ability to leverage diverse prior data to generalize to sol bing new tasks, even when the data does not contain any complete examples for the new tasks [28, 19, 35]. In our work, we also collect and train on multi-objective dataset, but unlike prior works, our dataset consists entirely of episodes where humans interact with other humans. Different objectives are created by changing the reward function of the environment. Such data can be used for generalization by simply relabeling the rewards to match that of the new task.
5.1 Experiment
Task description. We empirically evaluate our approach in the Overcooked domain. We consider two different layouts, which we call Asymmetric Advantages and Forced Coordination (Figure 3), that often require coordination in the form of passing ingredients to improve efficiency. The RL-controlled agent is the one with the blue hat.
5

---

Most prior works on Overcooked both train and evaluate on the standard task 16, 20]. In this work, we want to demonstrate that offline RI. can ac- complish something more complex- we can train an agent from multi-task data to solve new tasks by influenc- ing its human partner clever ways, when the prior data contains evidence of intentional and successful influence for the new task. Our new tasks more valuable to the customers than with onions, such that humans who are optimizing the standard reward (which rewards any type of soup equally) behave very suboptimally. Due this suboptimality, is important for the learned agent to influence its human partner, e.g., to pick up tomatoes instead of onions. We evaluate two tasks using the aforementioned layouts: Asymmetric Advantages (Human Deliver) offers doubled reward if the soup is delivered specifically by the human partner of the ego agent, and Forced Coordination (Tomato Bonus) offers doubled reward if the soup contains only tomatoes.
Figure 3: Layouts Asymmetric Advantages (left) and Forced Coordination (right).
modified reward function, e.g., soups with tomatoes
Data collection. We collected dataset of human humaan play where the human players were provided with one of several different instructions, in order to gather diverse dataset that illustrates variety of behaviors and human -uman interactions. In the first set of instructions, humans playing the game under the standard objective. We collected 20 human-human trajectories of length H ==1 200. In the second, humans are playing the game but of the humans is given the additional task of moving ingredients to counters for a small reward; interestingly, the other human is not aware of this objective, which induces various suboptimalities because they cannot predict their partner's actions. Under this modified objective, we collect 5 human-human trajectories of length H = 1, 200,
Baselines. We compare offline RL described in Section 5 to two baselines that also do not assume access to online interactions, nor simulators of the environment: naive behavior cloning (BC) on the dataset, and filtered BC that takes the subset of 10 trajectories that result in the highest reward. We also include a comparison to Fictitious Co-Play (FCP) that assumes an environment simulator and is common baseline in collaborative tasks [29]. FCP trains population of randomly-initialized agents via self-play.
Evaluation. We deployed each policy alongside a human player. We recruited 10 different human users for this evaluation. For each layout and method, we collected 30 trajectories length 400.
In Figure 5, we reported the mean and standard error of reward across all trajectories when solving both the standard task, as well as generaliza- tion to the new ones. We see that across all domains, offline RL outper- formed both BC and filtered BC, as well as FCP. This because both BC and filtered BC cannot exceed human- human performance, which often suboptimal due imperfect alloca- tion of tasks for coordination. The difference is exaggerated when eval- uating generalization, where human- human performance in the dataset solves different tasks. In addition, self-play approaches fail when the hu- man partner is strategically suboptimal, whereas our learned agent via offline RL can account for that by influencing the partner change their strategy.
70
Reward improvement
60
50
30
20-
Forced Coord.
Forced Coord. Tomato Bonus
Human Deliver
Figure 4: Improvement in reward (taken difference in cumulative reward between last 100 and first 100 timesteps) when a human is paired with our learned agent (via offline RL) across 30 trajectories of 400 timesteps.
Next, we ask whether this improvement in reward is solely due to the agent performing well on their part of the task, or due to actual impact human behavior. Qualitatively, in Figure 6. we show that offline RL is able to learn a sensible influencing strategy to influence the human partner to plate

---

250
200
Average Reward
150
100
BC
Filtered BC
Self-Play (FCP)
Offline RL
Asymm. Adv.
Forced Coord.
Asymm. Adv. Human Deliver
Forced Coord. Tomato Bonus
Figure 5: Average rewards over 30 trajectories of 400 timesteps of learned agents paired with real humans, with standard error across humans.
Figure 6: Trajectory where the ego agent influences their partner to plate the soup by moving plate on the counter nearby. The partner ends up dropping their onion order to plate the soup. This leads to more optimal coordination as the partner is in a more advantageous position to deliver soups.
the dish, which is crucial for success Asymmetric Advantages (Human Deliver). the visualized trajectory, the learned agent moves a plate near the human and leaves to pick up ingredients for the next soup; the human realizes this and plates the soup on the stove. We verify that this behavior does appear all in the prior data. However, components of this behavior (such as mistakenly picking up plate and subsequently placing it on a counter) appear in the prior data. In addition, in Figure 4, we show that the mean reward achieved in the last 100 timesteps greatly exceeds that of the first 100 timesteps, suggesting that coordination is improving over time between our agent and a human.
Achieving Long-term Influence of Latent Strategies
In this section, consider how offline RL learn to effectively achieve long-term influence the human's policy itself. Accomplishing this type of adaptive influence requires knowledge of what the other human intends do, their latent stralegy. contrast Section 5, we that offline RL can learn influence strategies to affect a human's 's actions that do not appear in the data, but rather that offline RL influence human's policy, while adapting changes in the policy.
trying to show
We propose an offline RL method that learns a -in-esinniional representation of the human't latent strategy, and condition the learned policy's behavior on that strategy. The method propose involves separately learning such representations, then a policy that conditions on these representations as part of its state. We describe the high-level approach but defer implementation details to Appendix A.
6.1 Learning Representations Latent Strategy
First, describe how we learn representations of latent strategy. Recall from the definition of Hi-MDPs Section that the other human's behavior follows latent dynamics pa s. a, 2). where z, z' Z representations of the human's 's latent strategy; as the representations, with d as a parametr.
d-dimensional embeddings
Additionally, we introduce h to be the agent's history of previous steps of interactions, where e E N is a tunable parameter. Namely, let r a1)........ (817, atr) denote trajectory taken by the ego agent, then we denote the history at step corresponding to state and action a, as
7

---

Figure 7: Example trajectory where the ego agent influences their partner to pick up tomatoes rather than onions by blocking off access the onions. Note that the agent should only block their partner they believe their partner is going towards the onions. Once their partner appears to be going to the tomatoes instead. the agent returns to their own task.
h4 ((s;-e ac-e)..... (8; a.-1))- Note that is easy augment offiine dataset such to now include such histories = rs..,r
We approximate the latent dynamics by learning encoder f(z h)) that uses the history of interactions to predict a distribution over representations of the human's 's latent strategy. Note that we use history instead of the human's 's previous strategy because anticipate that history has enough information to reconstruct the latter. This greatly simplifies training the encoder. In practice, the encoder can be of the form f h) = N(z (h), fx(h)), where is neural network that outputs both the d-dimensional mean of z as well as the x d covariance matrix.
Although we do not observe the ground truth representations of latent strategy. we can still learn these representations in an unsupervised way, since we know that they should predictive of future states. Recall that the dynamics of Hi-MDPs are given by P(s' [s, 2). We propose jointly training a decoder d(s' (s, a, z) that estimates the transition dynamics and reconstructs the next state, We ultimately propose the following joint training objective:
arg max Es,,,a' h-DD log d(s'' s, a, a)] EN-D (f h)) po(z))] f.d (x)((
(I)
The first term in (1) the reconstruction loss of the next state, and the second term is an information bottleneck [2] that regularizes the encoder towards prior (z(z). In our work, we use predicted distribution for the previous timestep as the prior, starting with Po (z) NNO,, I). By doing so, the inferred representations from the encoder are encouraged to be sequentially consistent across time, discouraging abrupt changes behavior. Note that Xie al. [34] propose a similar objective to model other agents in the environment, but only consider the reconstruction loss because each agent's latent strategy is fixed within an episode.
6.2 Offfine RI. Conditioned Latent Strategy
Recall the example of directing the human towards picking up tomatoes, where the agent's strategy influence the human depends how the human behaves. This naturally means that should condition the agent's 's policy on the human's 's latent strategy. We therefore train a policy parameterized :T (als,, 2), where Section 1. By doing so, the agent utilizes the learned dynamics of human strategy, and reasons how influence the human's strategy towards behaviors that better maximize reward.
[z][h) is the representation of latent strategy sampled by the encoder from
Our algorithm is modification of CQL. Namely, because the learned policy æ additionally conditions on human behavior, does the Q-function ((s, a, z). Our modified actor-critic algorithm learns Q-functions via an objective similar to CQL, but where z is sampled from the encoder, and serves as additional input the Q-function and policy.
6.3 Experiment
Task description. We again evaluate our described approach in the Over- cooked domain. We consider differ- ent set of layouts, which we call Open Asymmetric Advantages and Counter Circuit (Figure 8). In contrast to the layouts in Figure 3, these layouts al- low for players to block each other,
Figure 8: Layouts Open Asymmetric Advantages (left) and
Counter Circuit (right).
8

---

which we view an influence strategy that requires knowing the partner's 's behavior. We consider the modified task where soup that is delivered containing only tomatoes yields double the reward of other soups. We do so because this encourages influence by the ego agent on their human partner. If the partner appears to be going to pick up onions, the agent should try to redirect their focus (i.e., by blocking the path towards the onions).
Data Collection. For each layout, we collect a dataset of 20 human-human trajectories of length H = 1, 200. We inform the humans that is the ego agent about the modified objective, and let the other human know that they are missing information. This biases the ego agent to try various influence strategies, and for their partner to respond accordingly.
offline RL approach, which call latent offline RI., to several baselines. The first two are standard BC and offline RI., which simply condition on the current state and do not try to infer the human's 's behavior. The remaining baseline, which we call memory offline RI,, implicitly conditions on the human's 's behavior by conditioning on a history of states. The difference between this baseline and what we propose is that memory offline RL does not have an additional objective to learn representations of latent strategy as we propose in Section 1. Our final baseline an offline variant of the LILI algorithm [34], which we dub offline LILI. We choose LILI as baseline because they similarly model human behavior by learning latent representations. However, LILI assumes (1) access to online data and an environment simulator and (2) that changes in behavior solely happen between episode. Thereofre, we modify LILI to match our problem statement as follows: (1) we discard the recollection of data steps in the LILI algorithm, and (2) we re-estimate the human strategy every timesteps within an episode.
Baselines. We compare our proposed
Evaluation. We deployed each policy alongside human player, using the same 10 human users for the evaluation in Section 5.1. We collected on 30 trajectories of length // = 400.
In Figure 10 (left), we report the mean and standard error of reward across all trajectories both layouts when the learned agents are paired with human partners. We notice that taking into account hu- man behavior in some way is beneficial, as both memory offline RL and latent offtine RL greatly outperformed naive BC and offline RL. Surprisingly, our approach (latent offline RI.) noticeably improved upon memory offline RI., even though both amount of information. We hypothesize that disentangling the learn- ing of human behavior and optimal influence leads more effective training when the policies are neural networks. Our method also improves greatly over offline LILI due to modeling more nuanced transitions in behavior.
60
Reward Improvement
50
40
given the same
30
20
20.
Open Asymm. Adv. Counter Circ.
Figure 9: Improvement in re- ward (taken as difference in cu- mulative reward between last 100 and first 100 timesteps) when human is paired with our agent (latent offline RL).
In addition, we provide evidence that when agents successful, it is due to the human partner changing their behavior. In Figure 10 (right), evaluate all the considered agents when paired with a scripted policy that unconditionally chooses the suboptimal action, e.g. picking up onions when tomatoes fluenced to do otherwise. We see that since all evaluated approaches roughly perform the same, we rule out the possibility that agents learned via our proposed approach are solving tasks more successfully in isolation. Then, in Figure 9, we show that the mean reward our learned agent achieves with a human partner in the last 100 timesteps greatly exceeds that of the first 100 timesteps; since our learned agent does not change behavior unless their partner does, this shows that the human partner was successfully influenced towards more optimal coordination. Qualitatively, in Figure 7, we show that our proposed latent offfine RL learns an adaptive influencing strategy to influence the human partner to pick up tomatoes in the Open Asymmetric Advantages layout. In the trajectory, the learned agent blocks access to the onions, only until it infers that the human is not going for them.
superior, and cannot be in-
Discussion
Humans often behave suboptimally, either due to systematic biases or incomplete knowledge of the environment. In this paper, we investigate whether offline RL on human-human interactions is capable of learning policies that can influence humans toward more desirable behavior, First, we showed that given dataset with enough variability in human behaviors, existing model-free offline

---

250
140
200
Aewrage faweed
100
60
Offinne
Offline LIU
20
Kemory Offiine RL Labent Offfine
Figure 10: Average rewards over 30 trajectories H 400 timesteps of learned agents paired with real humans (left) and with scripted policy that will always pick up onions (right). Note that the scripted policy cannot be influenced.
RL algorithms can learn intentionally perform new influence strategies that are not explicitly performed by the humans in the dataset. However, humans will often adapt their policy based on past experiences, meaning agents need recognize how the human currently behaving, or their latent strategy, and then adapt influence strategies to account for changes in behavior. Therefore, we also proposed an offline RL algorithm that estimates and conditions on latent strategy, and show that can successfully influence humans even when their latent strategy changes.
Limitations and future work. We focus our evaluation on the Overcooked game, because it has interpretable rules and dynamics, while capturing the fundamental challenges to learning human influence. However, is still substantially simpler than real-world applications that require collab- oration with humans. An important direction of further investigation is to investigate whether our findings hold in more realistic domains (e.g.. dialogue systems).
In this work, we separately consider two challenges of human influence: learning unseen influence strategies, and how to achieve long-term influence that co-adapts with changes in human policies. remains to be seen offline RL. can solve both challenges simulataneously. in one end-to-ent to-end approach from diverse human-humme interactions, learn new strategies to influence human actions, then extend these strategies (potentially in simulation or online) to achieve long-term influence.
Ethical implications. Perhaps most importantly, the notion of influence and "improvement" of human behavior is a double-edged sword. On the one hand, humans behavior is suboptimal and people might lack knowledge that agents have, and agents serve implicitly coaches that help people arrive at better strategies. On the other hand, when agents are themselves lacking knowledge (especially of human values and preferences), influencing people to perform "better" under the wrong objective or wrong world state can very much hinder their performance. This painfully obvious in recommender systems, where influence with the wrong objective leads to much wasted time social medial platforms and, worse, polarization and other negative societal effects. Endowing the agent with proper uncertainty is crucial next step for this line of work.
References
[1] Mohammad Mehdi Afsar, Trafford Crump, and Behrouz H. Far. Reinforcement learning based recommender systems: A survey, CoRR, abs/2101.06286 2021.
[2] Alexander Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 1rXiv:/6/2.004/0/ 2016.
[3] DeepMind AlphaStar. Mastering the real-time strategy game starcraft ii. URL: hups://deepmind -tae-starciifi-ii.
con/blag/appaastr-mmstennnn ceatttteerinn--el---e-
[4] Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath. Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science. 2022.
[5] Noam Brown and Tuomas Sandholm. Superhuman for multiplayer poker. Science, 365 (8456))885-890, 2019.
10

---

[6] Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca D. Dragan. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, 2019.
[7] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neurul Information Processing Systems, 2016.
[8] Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learnigg awareness. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2017.
[9] Jakob N. Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte-carlo estimator. In International Conference on Machine Learning, 2018.
[10] Till Grüne-Yanonn Models of temporal discounting 1937-2000: An interdisciplinary exchange between economics and psychology. Science in context, 2015.
[11] Pablo Hermandez-Leal Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. CoRR, 2017.
[12] Shariq Iqbal and Fei Sha. Actor-atiention-critte for multi-agent reinforcement learning. In International Conference on Machine Learning, 2018.
[13] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, Abs/1771.09986, 2017.
[14] Maxim Egorov Jayesh K. Gupta and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, 2017.
[15] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651 651-673, 2018.
[16] Dong-Ki Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan How. Influencing long-term behavior in multiagent reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[17] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. 2019. URL attp://aaxxvv http://axxii.oorga////ee//////// 00949/0099/9
[18] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine, Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 2020.
[19] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should prefer offline reinforcement learning over behavioral cloning? In International Conference on Learning Representations (ICLR), 2022.
[20] Cassidy Laidlaw and Anca Dragan. The boltzmann policy distribution: Accounting for system- atic suboptimality in human models. In International Conference on Learning Representations (ICLR), 2022.
[21] Alistair Letcher, Jakob N. Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In International Conference on Learning Representations (ICLR), 2019.
[22] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv.2005.01645 2020.
11

---

[23] Dylan P. Losey, Mengxi Li, Jeannette Bohg, and Dorsa Sadigh. Learning from my partner's s actions: Roles decentralized robot teams. In Conference on Robot Learning (CoRL), 2019.
[24] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In Advances in Neural Information Processing Systems, 2017.
[25] Dorsa Sadigh, Shankar Sastry, Sanjit A. Seshia, and Anca D. Dragan. Planning for autonomous cars that leverage effects human actions. In Robotics: Science and Systems, 2016.
[26] Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study, Machine learning. 84(1-2):109-136, 2011.
[27] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 -39,, 2017.
[28] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang. Aviral Kumar, and Sergey Levine. Cog: Connecting new skills past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
[29] DJ Strouse, Kevin R. McKee, Matt M. Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In Advances in Neural Information Processing Systems, 2021.
[30] Suzanne C. Thompson. Illusions of control: How we overestimate our personal influence. Current Directions in Psychological Science, 1999.
[31] Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. Al Magazine, 2012.
[32] L. Wang, Wei Zhang, Xiaofeng He, and H. Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2018
[33] Woodrow 7. Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Inftuencing towards stable multi-agent interactions. In Conference on Robot Learning, 2021.
[34] Annie Xie, Dylan P. Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sudigh. Learning latent representations to influence multi-agent interaction. In Conference on Robot Learning, 2020.
[35] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pages 2561 1-25635, 2022.
[36] Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. CoRR, abs/191 1. 10635555.. 2019.
12


Reviews:
[{"review_id": "pgPTKH0Ihs", "reviewer": "Reviewer_RC1m", "summary": "This paper proposes a novel framework to empower an intelligent agent with the capability to effectively influence suboptimal human behavior during interactions. The primary objectives revolve around tackling two key challenges: (1) deducing a new strategy to influence human action and (2) learning to influence the human's long-term latent strategy. To evaluate the proposed framework's efficacy, a set of collaborative tasks (i.e., Overcooked environment) are employed. The experimental results demonstrate that the agent can successfully learn to influence human behavior based on a human-human interaction dataset, even though no instances of successful influence were present.", "questions": "1. When the agent tries to influence human behavior, is it possible that the human gets annoyed, particularly if the influence becomes excessively insistent, dominating, or aggressive? Such a scenario could potentially lead to negative reactions from the human and undermine the effectiveness of the interaction.\n\n2. In the provided example, the authors mention the human goes to plate when he sees the plate is right next to him. This situation seems to assume that the human has a myopic perspective. I am wondering whether the proposed approach is generalizable if the human suboptimal behavior can be attributed to other various possible reasons.\n\n3. The agent's objective is to elicit the expected human behavior. However, the agent may generate positive action (e.g., pass the plate) or negative action (e.g., block the way). These actions may significantly impact human perception and trust. How can you ensure that the agent influences human behavior effectively without causing harm to the team?\n\n4. Does the agent's active influence imply a leadership role, as it takes on a guiding role in shaping the collaborative decision-making process?", "limitations": "I appreciate the authors for the limitation and ethical implications section in the paper. Additionally, given the paper's objective of enabling the agent to influence human behavior and long-term strategies, it is essential to consider potential impacts on human trust and the possibility of unintentionally biasing the human towards a worse policy. These factors warrant serious consideration to ensure that the proposed approach fosters responsible and ethical human-AI interactions. Addressing these concerns would contribute to a more comprehensive understanding of the potential implications of the proposed method.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Generally, I find this paper to be well-motivated, accompanied by clear and easily understandable examples. The organization and presentation of the content are well-structured, making it easy to read and comprehend. The focus of the paper lies in addressing the critical issue of enabling intelligent agent to influence suboptimal human behavior which is expected to be a topic of significant interest in the human-AI interaction research field. The proposed approach to employing offline reinforcement learning to tackle the problem is innovative such that it successfully extends the application of offline reinforcement learning in a novel context. The evaluation section is comprehensive and thorough, providing compelling evidence that supports the efficacy of the proposed method.", "weaknesses": "First, for the example in line 66, if we consider the real world scenarios, such behavior (i.e., repeatedly blocking the human from doing something) without providing any explanation would largely affect the human trust in the robot. This example appears to assume that the human will naturally respond by adjusting their strategy to avoid that ingredient in the future. However, this implicit assumption is quite strong and may not hold true in real-world situations.\n\nAs the authors mention in the last section, the experiments were conducted on simple simulated game-like environments rather than real world scenarios. It would be intriguing to see how the proposed approach performs when applied to more complex real-world tasks involving human-AI interactions.\n\nFurthermore, when we talk about human-AI interaction, it includes both collaborative and competitive interactions. However, the scope of the paper is focused solely on collaborative settings, without investigating or evaluating the approach in competitive scenarios. Expanding the research to encompass competitive interactions may provide a more comprehensive understanding of the proposed method's versatility and potential applications.\n\nFinally, in the evaluation section, the authors compare the performance between BC and offline RL methods. I feel that the paper could be further strengthened by incorporating a comparison with prior works that approximate human decision-making models and subsequently learn to influence human behavior using the learned model. This broader comparison would enhance the paper's context and shed light on the relative advantages and contributions of the proposed approach in the context of existing research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We agree with you regarding expanding the scale and complexity of our evaluation. We view our work as a first-step in showing that influence can be done purely offline (without any environment or human simulators), and aim to show more compelling examples of this in the future. You also raised some interesting points that we aim to address below:\n\n**Assume that humans will naturally respond by adjusting their strategy**\n\nNote that our algorithm for influencing human strategies/policies (such as blocking an ingredient to change the person’s role in the future as well, as in the example the reviewer mentions) will only learn what appears in the data – these changes are only assumed to stick if they stick in the data, i.e. that latent transition (or similar) is observed. Of course, Offline RL can then stitch transitions in novel ways, but it will not assume a latent state / strategy transition is possible unless the data shows it. The version of our algorithm that influences human actions through state changes does not assume long-term changes in the human’s policy (but this can be unrealistic sometimes, which is why accounting for strategy change is actually needed). That said, we certainly agree that this needs to be tried out in more realistic settings. We are excited about that, and at the same time point out that this is a step far and beyond prior influence via RL work which has not studied real humans at all.\n\n\n**Expanding to competitive interactions**\n\nThough competitive settings are prevalent and interesting to study, we believe they are less relevant for influence because agents can do well by assuming the human is optimal and computing a best-response (see, for instance [cite the rohin-micah paper]). However, we agree that even in competitive settings, an agent can potentially learn to exploit a suboptimal agent. But our goal from the start has been to show how we can improve human suboptimal behavior, rather than exploit it. \n\n**Incorporating a comparison with model-based works**\n\nIn this work, we considered methods that were purely offline (can only use human-human data) and did not have access to an environment or human simulator. We believe that this limiting setting is important to study to find approaches that work in complex, real-world settings. \n\nIn the broader domain of human-AI collaboration, there are human-aware RL methods, which learn a human simulator from data and compute a best response via RL in an environment simulator [cite]. However, the topic of _influence_ has not been studied with these methods. Our work can be seen as paving the way towards studying influence in these settings where access to the environment model can be assumed, and our results show promise for these types of methods as well. In fact, the techniques from Offline RL that protect against distribution shift might very well apply to that setting as well, where driving the human model OOD could lead to poor real world performance. \n\nThere also exist approaches that use self-play, thus circumventing requiring a human simulator and only an environment one (which is more reasonable to obtain in practice).  We believe that self-play approaches fail when the human partner is strategically suboptimal, and the learned agent can account for that by influencing the partner to change their strategy. In Figure 4 (of the attached file), we reported results for an agent learned via self-play (FCP), and see that it does not perform as well as ours when the human has a misaligned objective. \n\n\n**Answers to questions**\n\nThe reviewer also raised questions that we aim to answer below:\n\n1. Whether or not the human gets annoyed is an interesting consideration. We have not measured the users’ subjective perception in the study, but can report that whatever that perception was, it did not negatively affect the effectiveness of the interaction – we see that the collaborative task performance goes up. That said, the reviewer is 100% right that this could get annoying, and further work should examine tools to assess online whether influence is working on the individual, and perhaps adapt the type and amount of influence to the user’s response and even preferences. .  \n\n2. The myopia point is a fascinating one, thank you for bringing this up. It does seem that real people tend to be reactive in this way, and our algorithm can leverage that. But no, it does not seem that the method would be limited to myopic bias. For instance, if people have partial information about the state and that’s why they take suboptimal actions, the agent might figure out that putting certain objects in their field of view might make the user aware of them and therefore change their actions. Or more speculatively, imagine a language interaction where the agent tries to influence people to donate to a charity – there biases like a propensity to anecdotal evidence might occur, and our method would be able to pick up the fact that people behave differently once they hear such anecdotes.  Like in the first question, one might expect the effects to show up in the task performance, which the agent is optimizing. The counterargument is that not annoying users should be a first order citizen in the optimization objective – we certainly agree and think it would be useful to study reward learning applied to this type of “influence”-related preferences.\n\n3. Yes, it is assumed that the human is suboptimal and the agent can help them overcome this, which implies the agent has better knowledge or computation than the human, putting them indeed in a leadership role. We believe when this is not true, it is more effective to learn AI agents that assist with the human’s strategy, which is an orthogonal paradigm that has rich prior literature. \n"}, {"review_id": "mlAl3eibtW", "reviewer": "Reviewer_rRJq", "summary": "This work aims to provide some evidence that offline reinforcement learning techniques can be used in the field of human-agent collaboration to influence or improve the behavior and underlying strategies of humans. The authors first verified that agents trained by CQL can influence the behavior of human players in some scenarios. Then, by simply modifying the CQL by conditioning on human latent strategy representations, the agents can learn to adapt to changes in human behaviors.", "questions": "See the revision recommendations in the \"Weaknesses\" section.", "limitations": "As discussed above, this work lacks the comparison of SOTA methods, and the user studies are not sufficient, which would limit its reliability. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The research purpose of this work is an important topic in the field of AI, how to improve human performance in human-agent collaboration. The author's attempt to verify the effectiveness of offline reinforcement learning techniques in human-agent collaboration scenarios is commendable.", "weaknesses": "1. After carefully reviewing the manuscript, I personally think the current content (methods and experimental results) does not effectively demonstrate the thesis that offline RL can learn to guide humans toward better performance by combining human latent policy representations. I agree that the agent may learn to adapt to changes in human behavior through offline RL, but it remains unclear how offline RL can tangibly influence and improve human performance. Furthermore, there is a wealth of research in the field of human-agent collaboration, such as Du, et al.[1] and Alamdari, et al.[2], focus on developing assistive agents to improve human performance. It would be advantageous to discuss this research in the paper to provide a more comprehensive understanding of the field. In the experimental part, the existing experimental results (improved team rewards & few examples) cannot support this assertion either. I suggest that the authors provide some objective metrics of human participant performance to make the conclusion more solid.\n\n2. User Study is crucial in human-agent collaboration research, and I’d suggest the authors provide additional information, including:\n     - Ethical review: Did participants provide informed consent for their involvement in this research and for the use of their data toward this project?  Were they fully informed about the purpose of the research, did they confirm their approval for involvement, were they told about how to withdraw their data?\n     - Test settings: What is the proficiency of the participants in the game? Were they all novices or professionals? Whether to provide a standard test specification or guide before the test to ensure the consistency of the test purpose?\n\n3. Recent research in the field of human-agent interaction has highlighted the significance of evaluating agents using both objective and subjective metrics. See Strouse, et al.[3], and McKee, et al.[4]. Incorporating subjective metrics into the evaluation of agents can provide a more holistic understanding of their impact on human performance and well-being. For example, did participants prefer playing with the trained agent over other baseline agents? During the collaboration, did the participants perceive their actions to be influenced by the agent's behaviors? If yes, to what extent? Did all participants perform better as a result?\n\n4. This work attempts to verify the effectiveness of existing offline RL techniques in some human-agent collaboration scenarios. Due to the lack of comparisons with state-of-the-art methods in the field of human-agent collaboration, such as Strouse, et al.[3], Yu, et al.[5], etc., it remains unclear where the boundary of offline RL in human-agent collaboration is.\n\n5. There are some unevidenced and unsubstantiated claims in the manuscript. For example,\n\t- In line 208, \"It is evident that naive offline RL cannot learn adaptive policies.\" \n\t- In line 179, \"humans are passive and will often only respond to what their partner is doing\"\n\n6. Minor Concerns:\n\t- Many references are not uniformly formatted, such as [27] \"nature\" and \"Science\" ,  whether conference abbreviations, e.g.: \"(ICLR)\", are reserved.\n\t- In line 369,  \"Thereofre\" should be \"Therefore\"\n\t- In line 412, \"simulataneously\" should be \"simultaneously\"\n\n[1] Du, et al. Ave: Assistance via empowerment. 2020.\n\n[2] Alamdari, et al. Be considerate: Avoiding negative side effects in reinforcement learning. 2022.\n\n[3] Strouse, et al. Collaborating with Humans without Human Data. 2021.\n\n[4] McKee, et al. Warmth and competence in human-agent cooperation. 2022.\n\n[5] Yu, et al. Learning zero-shot cooperation with humans, assuming humans are biased. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. The main concern raised in the review seems to center around the evidence that supports our conclusion. We hope to remedy this by providing additional evaluations that further reinforce our hypothesis that the human does improve their behavior during evaluation trials (implying successful influence), and offer some interpretability regarding the latent representations. We go into these additions in more detail below, and are happy to compute any others that you may suggest. You also asked for more details regarding the evaluation and discussion with related works that we offer below and will include in our updated paper. Please let us know if our response fully addresses the issues in your review.\n\n**Current content does not effectively demonstrate influence**\n\nWe understand your viewpoint that the improved performance may not be due to successful influence. Therefore, we provide the following metrics to make our conclusion more solid. In our opinion, these are the most reasonable metrics we could compute using our current evaluation data:\n\n1. In addition to showing the cumulative reward obtained, we show the difference in cumulative reward obtained between the first and last 100 timesteps of each evaluation. As shown in Figure 1 (of the attached file), the reward obtained towards the end of evaluation is much higher than in the beginning. Since the learned agent only changes behavior when the human partner does, this means that the human must have adapted their behavior over the course of evaluation.\n \n2. To confirm that the learned agent is actually trying to influence its partner, rather than solve the task itself, we evaluate all the considered approaches against a scripted policy that always performs the suboptimal action we would expect from a naive human (such as grabbing onions when tomatoes yield higher reward). In Figure 2, we see that the learned agent using our approach actually performs worse than naive offline RL. This means that our learned agent cannot be solving the task more successfully independently, but rather is trying to influence its human partner. \n\n3. To further support our hypothesis that using latent representations of strategy helps performance, we additionally show that the learned latent strategies map to different high-level actions. In Figure 3, we plot several latent strategies inferred from trajectories during evaluation (using PCA to reduce to 2 dimensions). We then colored the points that correspond to trajectories where the human partner goes to get tomatoes as red, and ones where the human goes to get onions as green. The plot shows that the red and green points are prominently in different clusters. \n\nIf you have ideas for other metrics that we can compute to support our conclusion further, we are happy to include them in the updated paper. Your suggestion to include user feedback regarding which learning agent they preferred is very useful. However, this requires redoing our evaluation, which we could not do in the rebuttal but would be happy to do in the future if you feel it would strengthen our evidence. \n\n**Discussion with other related works**\n\nThank you for bringing up these additional papers. We will include a discussion of them in our updated paper. Though the papers also look at coordination with humans, we view our paper as tackling a different problem, in that we consider influence rather than assistance. In the latter, the learned agent will always try to support the human’s intentions or strategy, whereas in our work, the learned agent tries to change the human partner’s strategy if it is suboptimal. Therefore, the methods proposed in the works you referenced would not be applicable in our setting, where the humans are not only suboptimal due to unfamiliarity with controls, but also due to having the wrong idea of what high-level action to take. \n\n**Additional details on user study**\n\nThank you for raising this important concern. We provide the additional details below and will include them in the updated paper. \n\nEthical review: Our user study is IRB-approved. The participants did provide consent and were informed about the high-level goal (of influence during coordination tasks) but not of the methodology that we used or our specific hypothesis (that pure offline RL can achieve successful influence). Per our IRB protocol, there is no option to withdraw the unidentified data. \n\nTest settings: The participants are not familiar with our specific game setting, though our controls are fairly intuitive and we provide detailed instructions on how to play. We found that the participants quickly adapted to the unfamiliar controls and learned to pilot their characters effectively. Therefore, much of the suboptimality came from not knowing the optimal high-level action to take. \n\n**Lack of comparisons with state-of-the-art methods**\n\nIn this work, we considered methods that were purely offline (can only use human-human data) and did not have access to an environment or human simulator. We also specifically study the setting of influence in human-AI collaboration. In this setting, we believe that the Offline LILI baseline is the current state-of-the-art approach. \n\nHowever, we do think that the self-play approaches you referenced, though they assume an environment simulator, are interesting to study as they are commonly used to solve general collaboration tasks. We believe that self-play approaches fail when the human partner is strategically suboptimal, and the learned agent can account for that by influencing the partner to change their strategy. Though Yu et al. consider biased humans, they do not consider that human behavior can change within a trajectory, thus not allowing for influence like we do. In Figure 4, we reported results for an agent learned via self-play (FCP), and see that it does not perform as well as ours when the human has a misaligned objective. "}, {"review_id": "XHpbkixAvL", "reviewer": "Reviewer_q3SJ", "summary": "This paper presents an investigation into the application of offline RL on a dataset of human-human interactions to develop a policy capable of influencing human behaviors towards desired outcomes. The authors demonstrate that the learned policy not only affects immediate behavior but also influences the long-term strategies and preferences of the human partner. Experimental results provide strong evidence for the effectiveness of offline RL in learning to influence humans and achieving desirable rewards.", "questions": "1. The authors state that the dataset contains no evidence of influence. However, how can one differentiate between behaviors resulting from mistakes and those intentionally performed by the human player to influence their partner?\n\n2. The data collection process outlined in Section 5 involves various approaches. What impact does the dataset have on the learned policy? Would the learned policy differ significantly if only the first or second set of instructions were used for training?\n\n3. Section 6 claims that the learned policy can induce long-term influence on the latent strategies of the human partner. Besides the average episode reward, is there any additional evidence supporting the assertion that the learned policy genuinely influences humans in the long term? The case study presented in Figure 6 and Lines 390-393 demonstrates short-term behavior influence, where the agent blocks humans from picking up onions to encourage tomato selection. My question is whether this influence is truly long-term, meaning, would the human player exhibit a stronger preference for picking tomatoes after repeated instances of being blocked from picking onions by the learned agent?\n\nClarity issues:\n1. In Line 208, the term \"naive offline RL\" is not immediately clear. Please provide a more precise explanation.\n\n2. Regarding Figure 5 and Figure 6, it would be better to mark which chef is controlled by the human player and which chef is controlled by the learned policy.\n\n", "limitations": "This work currently focuses on relatively simple environments. It would be promising to scale the method to more complicated systems.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow. \n\n2. The utilization of offline RL on an existing dataset of human interactions is a straightforward yet effective approach.\n\n3. The problem of learning to influence human behaviors from existing datasets is both interesting and important for the field", "weaknesses": "1. The experiment solely employs offline RL methods as baselines. It would be beneficial to compare offline RL methods with approaches that leverage both offline data and online interactions. For example, one potential baseline could involve first acquiring a learned human model via behavior cloning and subsequently learning an adaptive policy to cooperate with the learned human model in the simulator, as demonstrated in [1].\n\n2. The evaluation appears to be limited to assessments involving human participants. It would be valuable to evaluate the learned agent against a learned human model or scripted policies, as demonstrated in [1] [2].\n\n3. The claim that there is no evidence of influence between humans, coupled with the assertion that the learned policy can induce long-term changes in human strategy, may be overly conclusive and requires further substantiation.\n\n\nReference:\n1. Strouse, D. J., et al. \"Collaborating with humans without human data.\" Advances in Neural Information Processing Systems 34 (2021): 14502-14515.\n2. Yu, Chao, et al. \"Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. You raised some interesting additional evaluations to strengthen our paper that we have added and discussed below. Please let us know if our response fully addresses the issues in your review.\n\n\n**Compare with approaches that leverage both offline data and online interactions**\n\nIn this work, we considered methods that were purely offline (can only use human-human data) and did not have access to an environment or human simulator. We believe that this limiting setting is important to study to find approaches that work in complex, real-world settings. \n\nHowever, we agree that many existing approaches in the broader domain of human-AI collaboration use self-play, thus circumventing requiring a human simulator (though they still require a simulator of the environment/game, which is often easier to obtain).  We believe that self-play approaches fail when the human partner is strategically suboptimal, and the learned agent can account for that by influencing the partner to change their strategy.  Though Yu et al. consider biased humans, they do not consider that human behavior can change within a trajectory, thus not allowing for influence like we do. In Figure 4 (of the attached file), we reported results for an agent learned via self-play (FCP), and see that it does not perform as well as ours when the human has a misaligned objective. \n\n\n**Evaluate the learned agent against a learned human model or scripted policies**\n\nWe believe that it is important to evaluate our work against real humans because they exhibit nuances that traditional simulators typically do not emulate. Namely, real humans are able to adapt their behavior, which allows for such behavior to be influenced by changes in state or the actions of other agents. However, we see that evaluating against simulated, stationary policies can still be useful to show that our approach actually works due to influence. To rule out that our improved performance is simply due to solving the task better independently, we evaluate all the considered approaches against a scripted policy that always performs the suboptimal action (such as grabbing onions when tomatoes yield higher reward). In Figure 2, we see that the learned agent using our approach actually performs worse than naive offline RL. This means that our learned agent cannot be solving the task more successfully independently, but rather is trying to influence its human partner. \n\n\n**Claims require further substantiation**\n\nFor the claim that there is no evidence of influence between humans, we are not referring to influence of any kind, but specifically influence that assists in solving the new generalization task. Specifically, in the new task where one of the humans receives higher reward for plating, one form of influence (that offline RL learns) is to pass the plate to the human to encourage them to plate. We can confirm that this influence strategy does not appear in the offline data at all by simply inspecting all the trajectories (as there are only 25).\n\nFor the claim that there is long term influence, we provide the following quantitative evidence. We show the difference in cumulative reward obtained between the first and last 100 timesteps of each evaluation. As shown in Figure 1, the reward obtained towards the end of evaluation is much higher than in the beginning. Since the learned agent only changes behavior when the human partner does, this means that the human must have adapted their behavior over the course of evaluation i.e. the human knows to pick tomatoes over onions.\n\n\n**Answers to questions**\n\nThe reviewer also raised questions that we aim to answer below:\n\n1. We hope to have clarified what we meant by no evidence of influence above. Regarding accidental vs intentional influence, we believe that both would result in the same actions being observed in the data i.e. passing a plate looks the same even if done accidentally. Because we did not observe such actions in the dataset, we can confidently say that no accidental or intentional influence appeared.\n \n2. This is an important point, as offline RL cannot come up with entirely new strategies from scratch. Our hypothesis is that the dataset must demonstrate a reasonable range of behaviors such that such behaviors can be stitched to produce a new strategy. In the example of passing a plate, offline RL cannot learn to do this if there is no movement of objects at all in the dataset, but can if there is evidence of humans moving to and picking up objects from shared tables. We ensured that our dataset had these components. \n\n3. We hope that our additional evaluation shows evidence of long-term influence. If you have others in mind, we would be happy to compute and include them in the updated paper. \n"}, {"review_id": "Pwj4jp3j5Z", "reviewer": "Reviewer_hp68", "summary": "This paper presents an investigation for learning to influence suboptimal human opponents in agent-human interactions. It claims that by performing offline RL on human-human interaction data, an agent can learn to influence its opponent's actions and latent strategies, even if there is no explicit information of influence in the data. In the first set of experiments, the authors showed that CQL achieved higher test returns when paring with human opponents, and in the second set of experiments, they showed that decoding latent representation for states performed better than two other opponent modeling technique.", "questions": "1. Since this paper considers a multi-agent setting, I wonder how the MDP formulation in sec 3 covers the information of the opponents. In particular, do you consider the opponent's actions as part of the states or part of the actions?\n\n2. Improvements in test return can be explained by multiple factors. To explicitly show that offline RL shapes human opponents' actions or strategies, more explicit evidence is necessary. For example, what about reporting the action frequencies of humans?", "limitations": "Yes, they have.", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written. Materials are organized in a way that facilitates reading, and the connection to existing literature is clearly stated.\n2. Learning to influence human opponents, especially those suboptimal ones, is an important problem for human-agent interactions.\n3. The idea of learning influence strategies from human-human data is interesting and intuitive.", "weaknesses": "1. There lacks explicit empirical evidence to support the claims made in this paper. \n\nThe authors claimed that: (a) agents can learn to influence their opponents' actions using offline RL (the CQL algorithm), and (b) agents can learn to influence their opponents' latent strategies. However, there is no direct quantitative evidence of the increase in such influence. The reported improvement in test return can be explained by many factors. For example, rather than influencing its opponent, an agent may proactively solve the task itself. The only evidence is four states of a trajectory, yet it is not sufficiently annotated for readers.\n\n2. The presentation for modeling latent strategies is confusing.\n\nThis paper does not offer a tangible description of the \"latent strategies\". From eq (1), the \"latent strategies\" seem to be vectors that improve the prediction of the next states. But eq (1) seems not characterize the relationship between these vectors and the actions of the opponent. So I agree that they are latent vectors that improve policy learning, but I cannot see why they are strategies of human opponents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. The main concerns in your review seem to center around the evidence to support our conclusions. We hope to remedy this by providing additional evaluations that further support our hypothesis that the human does improve their behavior during evaluation (implying successful influence) and that the learned agent is not simply more adept at solving the task independently. We go into these additions in more detail below, and are happy to compute any others that you may suggest. Please let us know if these additions fully address the issues in your review.\n\n**No direct quantitative evidence of influence**\n\nYou raised a good point that the improved performance may not be due to influence. To remedy this, we report the following additional metrics on the evaluation runs:\n\n1. In addition to showing the cumulative reward obtained, we will show the difference in cumulative reward obtained between the first and last 100 timesteps of each evaluation. As shown in Figure 1 (of the attached file), the reward obtained towards the end of evaluation is much higher than in the beginning. Since the learned agent only changes behavior when the human partner does, this means that the human must have adapted their behavior over the course of evaluation.\n\n2. To rule out that the learned agent is not simply “solv[ing] the task itself”, we evaluate all the considered approaches against a scripted policy that always performs the suboptimal action we would expect from a naive human (such as grabbing onions when tomatoes yield higher reward). In Figure 2, we see that the learned agent using our approach actually performs worse than naive offline RL. This means that our learned agent cannot be solving the task more successfully independently, but rather is trying to influence its human partner. \n\nIf you have ideas for other metrics that we can compute to support our conclusion further, we are happy to include them in the updated paper.\n\n**Characterize the relationship between latent strategies and the actions of the opponent**\n\nWe agree that this would be informative. We attempt to do so in Figure 3, where we plot several latent strategies inferred from trajectories during evaluation (using PCA to reduce to 2 dimensions). We then colored the points that correspond to trajectories where the human partner goes to get tomatoes as red, and ones where the human goes to get onions as green. The plot shows that the red and green points are prominently in different clusters, showing that the latent strategies correspond to different high-level actions taken by the human. \n\n**Answers to questions**\n\nYou also raised questions that we aim to answer below:\n\n1. In this work, we consider the partner’s actions as part of the state, since they are observed by the learning agent. The partner’s intentions/strategy, however, are unobserved, making our MDP a special case of a POMDP. We will clarify this in the updated paper.\n\n2. We agree that more evidence is necessary. We propose adding the following metrics described above, and shown in the attached file. We are happy to include any other metrics you might be helpful to support our conclusion.\n"}]

Author rebuttal:
Based on reviewer feedback, we have performed multiple additional evaluations. We reference each of them in our individual responses to each reviewer, but also provide an overview of the new results below. 

In Figure 1, we compute the reward improvements of our proposed method across all the experimental layouts. This improvement is computed as the difference between the reward accumulated in the last 100 timesteps, and the first 100. By showing a noticeably positive improvement, we aim to show that our method is successfully changing the behavior of its human partner via influence.

In Figure 2, we evaluate our proposed method (along with all baselines) against a scripted policy that will always perform the suboptimal high-level action, i.e. pick up onions when tomatoes yield higher reward. Here, we see that our proposed approach actually performs worse than baselines. This is to show that the reward improvement is actually due to influence, and not our learned agent solving the tasks more successfully independently. 

In Figure 3, we visualize the learned latent strategies (as 2-dimensional embeddings obtained via PCA). Here, we show that different high-level actions map to different points in the space. This is to provide some interpretability on the learned latent strategies. 

In Figure 4, we add an evaluation of an agent learned via self-play (specifically Fictitious Co-play). Because the self-play approaches often do not account for suboptimal partners, the approach performs much worse than ours in the generalization tasks where the human partner is guaranteed to be strategically suboptimal. 

Meta-review: This paper investigates how to leverage offline reinforcement learning on a dataset of human-human interactions to learn to influence human behavior toward desired outcomes. Human-subject studies are conducted to evaluate the proposed approach.

This is a borderline paper. All reviewers agree that the paper is addressing a timely and relevant question. The approach of using offline reinforcement learning in human-human interactions is novel and practical. The main debates are on whether the current set of experimental results sufficiently supports the claims, e.g., whether the approach indeed "influences" humans towards more optimal behavior. The authors have provided additional results to support the claims though there are differing opinions on whether more comprehensive experiments are needed. 

Overall, considering the importance of the research question and the novelty of the approach, I tend to recommend accepting the paper. However, I'd recommend the authors take the reviewers' comments into account and provide more discussion on the implications and limitations of the results and how they relate to the claims of the paper.
Decision: Accept (poster)