Influencing Towards Stable Multi-Agent Interactions
Authors: Woodrow Zhouyuan Wang
Andy Shih
Annie Xie
Dorsa Sadigh
Venue: CoRL2021 Oral (2021)
Keywords: multi-agent interactions
human-robot interaction
non-stationarity

Abstract: Learning in multi-agent environments is difficult due to the non-stationarity introduced by an opponent's or partner's changing behaviors. Instead of reactively adapting to the other agent's (opponent or partner) behavior, we propose an algorithm to proactively influence the other agent's strategy to stabilize -- which can restrain the non-stationarity caused by the other agent. We learn a low-dimensional latent representation of the other agent's strategy and the dynamics of how the latent strategy evolves with respect to our robot's behavior. With this learned dynamics model, we can define an unsupervised stability reward to train our robot to deliberately influence the other agent to stabilize towards a single strategy. We demonstrate the effectiveness of stabilizing in improving efficiency of maximizing the task reward in a variety of simulated environments, including autonomous driving, emergent communication, and robotic manipulation.

Influencing Towards Stable Multi-Agent Interactions
Woodrow Z. Wang, Andy Shih, Annie Xie, Dorsa Sadigh Department of Computer Science, Stanford University (wwang153, andyshih, anniexie, dorsa]@stant oodd edu
Abstract: Learning in multi-agent environments is difficult due to the non- stationarity introduced by opponent's or partner's changing behaviors. Instead of reactively adapting to the other agent's 's (opponent or partner) behavior, we pro- pose algorithm to proactively influence the other agent's 's strategy stabilize which can restrain the non-stationarity caused by the other agent. We learn low- dimensional latent representation of the other agent's strategy and the dynamics of how the latent strategy evolves with respect to our robot's 's behavior. With this learned dynamics model, our robot to deliberately influence the other agent to stabilize towards a single strategy. We demonstrate the effectiveness of stabilizing in improving efficiency of maximizing the task reward in a variety of simulated environments, includ- ing autonomous driving, emergent communication, and robotic manipulation. We show qualitative results on our website.
define an unsupervised stability reward to train
Keywords: multi-agen interactions, human-robot interaction, non-stationarity
1 Introduction
Reinforcement learning has shown impressive performance in recent years [1, 2], but much of the focus has been single-agent environments. Many real-world scenarios, on the other hand, involve coordinating with multiple agents whose behavior may change over time, making the task non- stationary. For example, when playing volleyball, our teammate may play more aggressively over time if they are hitting the ball well, so we may need to step aside and set the ball for them more often. Many techniques in multi-agent reinforcement learning (MARL) have been proposed for dealing with such non-stationarity in the other agent's 's strategies [3].
One set of approaches reactively adapts to the other agent's change in strategy, for example by simply forgetting past experiences and training on more recent data [4, 5, 6. 7, 8]. Other works try to learn model of the other agent and take into account how the robot's 's actions affect the other agent's strategies over time [9, 10, 11. 12, 13]. With knowledge of the other agent's 's strategy dynamics, we can take actions to influence the other agent in way that maximizes
long-term reward [10, 13].
Although influencing other agents to maximize reward is a useful approach principle, it can be challenging in practice. Indeed, if we have access to a faithful model of the other agent's strategy dynamics, then we should choose actions that carefully nudge their high-level strategy to maximize our long-term reward. However, learning and leveraging such a model of the other agent's 's strategy dynamics can be impractical. For one, even just estimating the other agent's current strategy is already difficult with limited data, let alone learning the dynamics between strategies. On top of that, the robot may have to figure out a best response for a large number of opponent strategies, making the full learning process extremely sample inefficient.
Instead, we should take into account the difficulty of learning for our ego agent. That is, we would still like to influence the other agent in useful way, but in a manner that minimizes the burden of the ego agent's ' learning algorithm. Our key insight is that we can stabilize the other agent's strategy in a multi-agent environment to reduce non-stationarity and facilitate learning best response to the other agent. Our approach is a specific form of influencing learning how to stabilize that
We refer to the other agent interchangeably as the opponent, regardless of whether they are competitive or cooperative [3]. Also, we interchangeably refer to our robot as the ego agent.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

net
net
court
court
Robot (ego agent)
Partner (other agent)
both digging for the ball
stepping back stabilize
Figure I: Coordination with a partner can he critical to succeeding in task. In this volleyball game, the red and green agents both can either dig for the hall or step hack for their partner. (I.fft) A partner that indecisively switches between digging and staying back can be extremely difficult to coordinate with, potentially costing the match. (Right) By deliberately stepping back, the red agent can stabilize the partner's strategy to dig for the ball, developing convention that allows for easier learning of how win the volleyball match.
aims to reduce the non-stationarity of the other agent, and turn makes learning easier for the ego agent. Intuitively, stabilizing the other agent's strategy allows avoid learning about complex high-level strategy dynamics, and focus our efforts on maximizing reward against a fixed strategy.
As an example of stabilizing the other agent, imagine again that we are playing beach volleyball with a teammate. When the ball comes perfectly between our teammate and us, not obvious which of us should dig for the ball (see Fig. 1 (left)). In principle, it is possible to carefully model the dynamics of how our teammate's strategy changes based on our actions and precisely predict who should get the next ball that falls in between us. However, it can take many interactions before we learn a good model our opponents, and constantly switching roles between digging for the ball and staying back can make learning difficult. Instead, we can stabilize our teammate by making our intentions clear: we actively take a step away from the ball so our teammate knows to dig for the ball, as in Fig. (right). This simplifies the learning process by reducing the need to learn complicated strategy dynamics, and making the task more stationary.
In particular, our method combines the robot's task reward with stabilizing reward, where stability measured with respect to the other agent's strategies learned in an unsupervised way. The task reward pushes the robot to do well in the environment, while the stabilizing reward aims to make the learning process easy and sample efficient. We demonstrate the strength of our approach in 7 different tasks, including autonomous driving, emergent communication task, and simulated robot environments.
Our main contributions in this paper are as follows:
We propose augmenting our robot's reward function with stability reward in order to explicitly encourage the robot to influence the other agent towards a stable strategy.
We demonstrate that the stability reward can be defined as an unsupervised reward using learned representations of the other agent's 's strategy.
We perform experiments in simulated environments demonstrating that stabilizing the other agent's strategy improves our robot's ability to coordinate and maximize the task reward.
2
Related Work
Multi-Agent Reinforcement Learning. Learning in multi-agent environments can be extremely difficult when the other agents have non-stationary behavior [14, 15, 16]. Hermandez-Leal et al. [3] provide taxonomy of existing methods that attempt deal with this non-stationarity issue in multi-agent environments. Some works seek to forget about previous experiences with old opponent strategies and improve adaptation by only training on more recent data [4. 5, 6, 7, 17]. Further, centralized training has shown promise in multi-agent environments [18, 19]. Some works design explicit communication channels in which agents can share their policy parameters or gradient up- dates [20, 21]. However, these methods require that there exists some centralized method in which agents can communicate or share parameters. We are mainly interested in the decentralized setting where we train an ego agent to interact with other agents that we do not have control over.

---

Oppoment Modeling, Instead of ignoring the opponent's 's changing strategies, prior opponent mod- eling works design agents that estimate their opponent's policies or parameter updates [22, 9, 23, 11, 12, 24, 25] in potentially recursive manner. Peer-rewarding has also emerged as method of encouraging cooperative behavior [26, 27, 16], but requires the assumption that an agent's ss reward function can be externally modified. Several works recognize that the ego agent's 's policy can directly induce change in the opponent's 's policies [13, 28, 29, 30J. In [10], the ego agent learns not only a representation of the opponents" strategies, but also learns a dynamics model for how the opponent's s strategies change conditioned on the ego agent's 's behavior.
Representation Learning of Latent Strategies. Representation learning is critical to the tractabil- ity of opponent modeling. may be computationally inefficient to learn all opponent intentions or explicitly learn the parameters of the opponent's policies. Instead, we recognize that there is often some underlying structure in the opponent policy space, which can be captured by low-dimensional latent strategy. Prior works learn latent context variables to help distinguish between separate tasks, which could correspond to different opponents [31, 32, 33, 34, 35]. Representation learning has been shown to be useful in developing a dynamics model of the opponent's policy in multi-agent environments [36, 10]. In our work, we show that can be beneficial to cluster latent strategy repre- sentations into a discrete latent space [37, 38] to identify shared structure among similar strategies.
Stability and Surprise in Reinforcement Learning. To mitigate the difficulty of learning in a non- stationary environment, we seek to stabilize the opponent's 's strategy - or minimize the surprise in the opponent's 's strategy. Recent works in reinforcement learning seek to do the opposite of designing maximum entropy methods to improve exploration [39, 40, 41J. Stability in the opponent's strategy can reduce the non-stationarity in the environment and improve efficiency of learning a best response [42]. Surprise minimizing reinforcement learning (SMiRL) trains an agent to seek predictable and repeatable states to counteract the environment's entropy [43]. SMiRL considers a long horizon notion of stability in which the state should be of high likelihood with respect to the distribution of states encountered so far. Our work is similar in principle to SMiRL; however, we consider a pairwise notion of stability between consecutive latent strategies, as we recognize that an opponent that unpredictably switches between a few strategies is highly non-stationary, yet each individual strategy can be high likelihood with respect to the overall distribution of previously seen strategies.
3 Problem Statement
We consider two-player multi-agent settings, where there is an ego agent that we have control over and another agent that we do not have direct control over. Since we only control the ego agent, can consider our problem formulation in terms of a single-agent Markov decision process where the opponent policy is absorbed as part of the environment transitions and rewards. Non-stationarity in the opponent policy then corresponds to non-stationarity in the environment transitions and rewards.
Hidden Strategies. The agents repeatedly interact with each other across multiple interactions, with each interaction lasting multiple timesteps. We assume that the opponent policy z is fixed within each interaction but can change across interactions. We consider environments with no hidden states, so that if the ego agent can observe the exact opponent strategy, then we can formulate the ego agent's 's task as a fully observable MDP. However, since the ego agent does not directly observe the opponent's ' strategy, we consider a Hidden Parameter MDP (HiP-MDP).
Hidden Parameter Markov Decision Process. We formulate the interactions as variant of HiP-MDP, which a tuple M (S, TT,,,,R R. H. [44], which includes an additional T that governs the dynamics of the opponent's strategies. We let z E 2 be the opponent strategy for the j-th interaction. At each timestep f. the ego agent is in state S; E S. The ego agent takes action d; E A and the environment transitions to St41 with probability T(s+41| (st at, 2²). The ego agent receives the task reward Russ (st, at,z²), ).). The interaction ends after timesteps and y is the discount factor. Both the transition and reward function are unknown. During interaction j. the ego agent observes the trajectory 7J = {(11, 01, Rusk ($1, a1. )²))... (SH, an Reck (8 an, 2²)) Notice that the ego agent does not observe the opponent's strategy 2 in the trajectory, which makes learning the value of an observation and action difficult - the reward of an observation and action can change significantly as the opponent's 's strategy changes across interactions.
We consider the setting where the opponent's is change in strategy is governed by the trajectory of the previous interaction. At the end of interaction j. the opponent's 's strategy transitions to g/+1 based
3

---

|2²,-)) r²). Namely, the ego agent's behavior during interaction j can directly influence the opponent's 's strategy for interaction j + 1. Next, we define a notion of stability between interactions and j + 1 as follows:
on strategy dynamics
Definition 1 Let 2²,
20 Z be two opponent strategies. The strategies 202 are pairwise e-stable with respect to a metric space (2,d) if and only if d(z x3) <.
Stabilizing Opponent Strategies by Influencing Across Repeated Interactions. Over repeated in- teractions, the ego agent explores HiP-MDP M over a sequence of opponent strategies (2 2,,2" .......). The ego agent's objective is to maximize the cumulative discounted sum of rewards over all inter- actions, shown in Eq. 1. The ego agent's ability to maximize the reward within each interaction is heavily dependent their ability to estimate the opponent strategy 2 for that interaction. Without knowing the opponent's 's strategy, the ego agent will have significant difficulty taking an action that is best response the opponent's 's actions. It can be prohibitive learn model of all the oppo- nent's s strategies and form a general policy for the ego agent that can react to any opponent strategy; however, if the ego agent can stabilize the opponent strategy
fixed point, i.e., 3z such that T= - T)) > 0 if and only if d(=², 2 < 6, Vj E [K...N} (where N is the total number of interactions), then the ego agent can more easily learn best response the opponent strategy in a stationary environment.
With fixed across interactions, the HiP-MDP is reduced fully observable MDP parameterized by z. At high level, we are encouraging the agent to learn how to explore the HiP-MDP in a subset of the state space, where the hidden opponent strategy is kept stable between interactions. Thus, the ego agent is effectively learning in a fully observable MDP, given that the fixed - but unknown opponent strategy 2 is not changing. In terms of the exploration-exploitttio tradeoff, stabilizing the opponent strategy would reduce the ego agent's 's required exploration in order to find a best response to the opponent's fixed strategy.
We note that it is not always easy or possible to stabilize the opponent's 's strategy. Stabilizing requires that the dynamics of the opponent's 's strategy are defined such that the ego agent can form an inter- action where T= r²) > 0 and only if 2 and 2 are pairwise e-stable. In general, as long as there is a fixed point 2 where the optimal value function satisfies V (s1, (*), 2 V ("1s 2) for any other E 2, then stabilizing should not hinder the task reward in the long run.
Stable Influencing of Latent Intent (SILI)
Learning Latent Strategies
Stable Influencing
g 2-2
encoder Eg
Rd
encoder Eg +
decoder D R
Ratable
policy
encoder Eo
staelor, a -11)
Reask
Figure 2: Stable Influencing of Latent Intent (SILI) (Left) We leam dynamics model of the opponent's ' strategies conditioned on the ego agent's past trajectory. The latent strategy is learned in unsupervised way by jointly learning a decoder to predict the state transitions and task rewards. (Right) combine stability reward with the task reward to train our policy using RL. The stability reward is unsupervised and defined by minimizing the pairwise distance between the previous two predicted latent strategies.
In this section, we present our approach of influencing the latent strategy of our opponent towards a stable strategy. At a high level, the ego agent learns an encoder that predicts the opponent's 's next latent strategy given the previous trajectory. The ego agent then conditions its policy for the next interaction on the predicted latent strategies of the opponent. But, since we never observe the ground truth latent strategies, we must learn the latent strategy representations in unsupervised way. To do so, we follow the setup [10], where we jointly learn a decoder that reconstructs the task rewards and transitions for the next interaction using the predicted latent strategy. Using this predicted latent

---

strategy, we can measure the stability of the opponent's 's latent strategy, and optimize for a weighted combination of the task reward and stability reward.
Learned Latent Strategies. First, we describe the representation learning component of our method shown in Fig. 2 (left). Recall that assume that the latent strategy dynamics follow 122 r)). We can approximate these latent strategy dynamics by learning an encoder & that maps trajectories rJー to a latent representation of dimension & (chosen as a hyperparam- eter). Here, model the encoder f ()'-)) instead of [(2212>-1, -'="), as in [10]. This simplification, which often works well in practice, assumes that the previous trajectory already has enough information to reconstruct 2
Although we do not observe the ground truth latent strategies, we can still learn them in an un- supervised way, since we know that they should be predictive of the task reward and transitions. Thus, we jointly train decoder that predicts the task reward and transitions from the encoded latent representation. In particular, for interaction j and timestep L, the decoder Di, takes as input states and actions s², af as well as the latent strategy embedding 2², and reconstructs the next state 8/41 and task reward r To train the encoder and decoder, use the following maximum-likelihod ( (5(+1 1/ss,, ai, -'-1).
objective: max Σ3 Et log Pog
0.
Reinforcement Learning with Stable Latent Strategies. As the opponent's 's strategy changes, the ego agent's policy should change in response. Naturally, we should condition the ego agent's policy on the opponent's latent strategies. When considering the goal of keeping the opponent's strategy pairwise stable, it is important that the ego agent conditions their policy on at least the most recent two predicted latent strategies since the stability reward depends on the distance between the consecutive latent strategies, as shown Fig. 2 (right). Concretely, the ego agent's policy is defined as (a(a st, x², 1²) where z = (T(
We have defined the environment reward to be Rask. We then define the stability reward to be the distance between consecutive predicted opponent strategies, where d can be any generic distance d(2),22'1)
metric over the latent space Z as follows: Raable
Here, we abuse notation and use z to refer to the prediction of the opponent's s strategy since we never observe the ground truth. Finally, our new reward function is defined as a weighted average of these objectives, parameterized by stability weight 3: Restal = (1 3) Resk + B Rstable With 3 1, the ego agent's entire objective is to stabilize the opponent strategy, and with / 0, the ego agent greedily maximizes the environment reward.
Optimizing Rewards Across Interactions. Our ego agent seeks to influence the opponent towards desirable latent strategies. With the encoder &g. the ego agent can deliberately take actions in tra- jectory 7 in order to influence the opponent to stable zi+1.
To learn how to influence across repeated interactions, we train our ego agent's policy Ty to maxi- mize rewards across the repeated interactions
7
Σ Ra a, 21, 241
mAx
where Ph, is the trajectory distribution induced by Te under the transition function T(s= Ur, =²). With the total reward including the stability reward, the ego agent learns to take interactions such that the opponent follows a stable latent strategy so the ego agent can easily learn best response to in turn maximize the task reward.
Modeling Latent Strategies. We examine three methods to model and measure distances between the changing latent strategies: continuous latent variables, discrete latent variables, and partial ob- servations of the opponent's ' strategy. The ego agent never directly observes the opponent's 's strategy.
I) Continuous Latent Variables. In the most general case, the latent strategy space is allowed to be continuous. This enables the latent strategy space represent an infinite number of possible opponent strategies. With the continuous latent variables Z, we use the Euclidean norm for the stability reward: Raaalle -d(2', - =1|=> -
2) Discrete Latent Variables. In this setting, we constrain the latent strategy space to be discrete categorical variables by using the Gumbel-Softmi distribution [37, 38). Intuitively, we cluster the
5

---

continuous latent variable representations to filter the noise of measuring distance in the latent space. Further details of the discretization method can be found in Appendix A. With these categorical latent variables 2, we use the discrete distance metric for the stability reward:
2 # 2 = 23
Ratuble = - d(22 2011)
(2)
0
We found empirically that discrete latent variables improve the consistency of the stability reward signal compared to continuous representations, despite the decrease in expressivity and granularity provided by the distance metric. Intuitively. it is casier to identify whether or not a strategy is similar enough be clustered with another strategy than is exactly identify the distance between predicted strategies in a latent space.
3) Partial Observations of Changing Strategy. In this setting, assume that the ego agent is able to observe if their opponent's ss strategy changes between timesteps, so no representation learning is necessary to stabilize. We shed the assumption that the latent strategies fixed within an interac- tion, so we denote the latent strategy at a timestep I as z,. Thus, the stabilizing reward can be defined as Ratable G3 - 1[ zt-1]; which indicates whether the opponent strategy has changed. This corresponds to settings where it may be easy to estimate when an opponent's strategy has changed, but it may be difficult to estimate their exact strategy [12].
Experiments
We perform experiments in range of environments across different latent strategy space settings: continuous latent variables, discrete latent variables, and partial strategy observations. We compare approach, Stable Influencing of Latent Intent (SILI) to various multi-agent learning baselines with implementation details described in Appendix A:
Oracle. Oracle is given the true opponent's ' strategy as an observation.
Learning and Inftuencing Latent Intent (LILI) [10]. LILI combines soft actor-critic (SAC) [45] with a representation learning component to model the opponent's s strategy dynamics. LIII does not explicitly optimize for stabilizing the opponent's strategy.
Surprise Minimizing RL (SMiRL) [43]. SMiRL optimizes for different notion of stability, where the reward is proportional to the likelihood of state given the history of previously visited states.
Stable. We refer to Stable as the variant of SILI with 1, only optimizing for stability.
SAC [45]. SAC does not model the opponent's 's strategy and is the base RL algorithm for SILI.
5.1 Simulated Environments
The environments and trajectories of stabilizing behavior are shown in the left column of Fig. 3. Across all figures, the ego agent's ' trajectory drawn in blue, and grey corresponds to the opponent.
Circle Point Mass. In this environment, the ego agent is trying to get close to the opponent as possible in a 2D plane, inspired by pursuit-evasion games [46]. The opponent moves between loca- tions along the circumference of a circle. The ego agent never observes the true location (strategy) of the opponent. If the ego agent ends an interaction inside the circle, the opponent jumps coun- terclockwise to the next target location. If the ego agent ends an interaction outside the circle, the opponent stays at the red target location for the next interaction. We examine four variants: Circle (3 Goals), Circle (8 Goals), Circle (Continuous), Circle (Unequal). In Circle (Continuous), there infinite possible opponent strategics, so we use continuous latent variables. In Circle (Unequal), the ego agent begins closer to two unstable opponent goals, but there is stable goal farther away.
Driving, A fast ego agent is attempting to pass slow opponent driver [10]. There are lanes and a road hazard upcoming in the center lane, so both the ego agent and opponent need to merge to new lane. the ego agent merges to the left lane before the red line (giving the opponent enough reaction time), then the opponent merges to the right lane during the next interaction, understanding the convention of faster vehicles passing on the left. Otherwise, the opponent will aggressively try to cut off the ego agent by merging into the lane that the ego agent previously passed in.

---

Seayer-Reach. The opponent can choose between three goals on table with their intent hidden from the robot. The ego agent the Sawyer robot that is trying move their end effector as close as possible to the opponent's chosen goal in 3D space without ever directly observing the opponent's strategy [47]. the end-effector ends the interaction above fixed plane on the 7-axis, the opponent's strategy stays fixed. Otherwise, the opponent's s strategy changes. Semantically, we consider a robot server trying to place food on a dish for a human and needing to move its arm away from the dish by the end of the interaction to avoid intimidating the human retrieving the food.
Detour Speaker-Listeng In this environment, the agents must communicate to reach a goal, which has been popular setting to explore emergent communication [18]. The speaker (opponent) does move and observes the true goal of the listener (ego agent). The ego agent cannot speak, but must navigate to the correct goal. The speaker utters a message to refer to the goal, which the ego agent then observes. If the ego agent goes near the speaker (within some radius), the speaker follows the same communication strategy during the next interaction. Otherwise, the speaker chooses a random new strategy, or mapping of goal landmarks to communications, Critically, in this environment, the opponent's s strategy is the communication strategy, not the true goal location.
Oracle
SIL (Ours)
LILI
SMIRI.
Stable
SAC
Circle (3 Goals)
Task Reward
Stability Reward
0.0
-1.0
Circle Goals)
1000
1500
2000
2500
3000
500
10000
1500
2000
2,500
3000
0.0
0.5
200
4.00
600
000
1.000
1200
200
400
600
800
1000
1200
Circle (Unequal)
0.01
-0.5
-1.0
200
430
600
6000
1000
1200
0.0
Driving
passing
-1.0
0.5
-1.5
500
1000
1500
2000
2500
500
10000
1500
2000
2500
Sawyor- Reach
-10
-20
-30
200
400
600
800
1000
1200
Speaker- Listener Circle (CrcleCCnttinnons)
0.0
-0.2
200
400
600
500
1000
1200
7:50
600
1000
1200
-3.0
-0.4
-0.6
100.0
0.88
250
500
750
1000
1250
1500
750
1.000
1250
1500
Interactions
Figure 3: Task and stability rewards across all environments. SILI achieves comparable performance to Oracle. Compared the other baselines, SILI significantly outperforms them in task reward (left plots) by learning to stabilize the opponent's strategy (right plots).

---

5.2 Results
Across all environments, without observations of the opponent's 's strategy, SILI achieves a compa- rable task reward to Oracle, as seen in Fig. 3. Further analysis is available in the Appendix and at https://sites. google. www./viww/ttaalleeeoeeeeooe
with qualitative analysis of trajectories.
Continuous Latent Variables. We perform an experiment with continuous latent variables in the Circle (Continuous) environment (row 6, Fig. 3). Both SILI and Oracle learn to stabilize in this environment, although Oracle incurs various dips in stability reward at little to no cost to the task reward because Oracle knows the true opponent's strategy. While these results are promising, we observe in other environments that continuous latent variables and a smooth distance metric over the latent space lead to more noisy rewards, which hinder the ego agent's ability to learn effectively.
Discrete Latent Variables. We further show our results in a variety of environments with discrete latent variables. In Circle (3 Goals), SILI achieves the highest reward along with Oracle, but Oracle does not require stabilizing the opponent's strategy (row 1, Fig. 3). In Circle (8 Goals), we demon- strate that with more opponent strategies, SILI still learns stabilize and outperforms the baselines (row 2, Fig. 3), where we set the latent dimension hyperparameter to be an overestimate of the true number of opponent strategies. In Circle (Unequal), SILI avoids trying to optimize for nearby, un- stable opponent strategies, whereas the other baselines do not learn to stabilize since they greedily try to track the nearby, unstable opponent strategies (row 3. Fig. 3). In Driving, the task reward is based on the number of collisions with the opponent, and our method is the only one that learns to safely avoid all collisions (row 4, Fig. 3). In Sawyer-Readd SILI and Oracle achieve comparable performance, but again, Oracle does not need to learn to stabilize (row 5, Fig. 3).
Partial Observations of Changing Strategy. To test method in setting without representation learning, the ego agent observes whether the opponent's strategy has changed between timesteps. We perform this experiment in the Detour Speaker-Listene environment, where the assumption that the opponent's strategy is kept fixed throughout an interaction is broken (row 7, Fig. 3).
SILI learns to stabilize and significantly outperforms all other baselines in task reward, including Oracle. In this environment, although Oracle can observe the true opponent strategy, the Oracle still needs to learn the association of the opponent's strategy and opponent's communication with the true goal. Further, the communication and strategy can change frequently between timesteps while the true goal stays constant, making learning difficult. The baseline Stable learns to stabilize but does not optimize for the task reward at all. The rest of the basclines learn to move towards the centroid of the goal locations since they cannot accurately decipher the opponent's is strategy.
Notice that around interaction 250, SILI learns to stabilize, which requires moving in a detour away from the true goals, but SILI maintains the stability and learns to maximize task reward well. On the contrary, LILI temporarily learns to stabilize at around interaction 1250, but incurs a corresponding decrease in task reward, which discourages LILI from maintaining the stable opponent strategy.
6 Discussion
Summary. We propose a framework for learning in multi-agent environments that learns the dy- namics of the opponent's strategies in a latent space. We then leverage this latent dynamics model to design an unsupervised stability reward, which can be augmented with the task reward to explicitly encourage the ego agent learn how stabilize and then more easily maximize the task reward.
Limitations and Future Work. In this work, we define a notion of pairwise stability that considers consecutive changes in the strategy. However, it may be interesting to consider smoother, longer horizon stability metrics, such as average change over a window of time. We compare our method to SMiRL, which optimizes for probabilistic notion of state stability relative to states in experience replay buffer, but show that SMiRL often does not learn to stabilize in a consistent manner. We plan to compare these different metrics of stability and their effects in future work. In addition, our experiments were mainly done in simulation. In the future, we plan to consider robot-robot and human-robot interactions, where we required learn much more complex dynamics that capture the non-stationarity of humans who adapt their strategies over repeated interactions. Stabilizing a human's 's strategy can greatly improve the efficiency in which robot learns the best way to perform a task, but the exact method in which to stabilize human behavior should be further explored.
8

---

Acknowledgments
We would like to acknowledge NSF grants #1941722, #2006388, and #2125511, Office of Naval Research, Air Force Office of Scientific Research, and DARPA for their support.
References
[1] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Proceedings of the 12th International Confer- ence on Neural Information Processing Systems, NIPS'99, page 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. In NIPS 2013 Deep Learning Workshop, 2013.
[3] Hernandez-Leal, M. Kaisers, T. Baarslag, and E. M. de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint 1rXivv1707..0m 183,, 2017.
[4] J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. S. Torr, P. Kohli, and S. Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In D. Precup and Y. W. Teh, editors, Proceedings the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1146-1155. PMLR, 06-11 Aug 2017. URL attp://prooceeding.. mlr. pptss//v0/foeesrerrr/// htmm.
[5] M. Bowling and M. Veloso. Rational and convergent learning in stochastic games. In Proceed- ings of the 17th International Joint Conference on Artificial Intelligence Volume 2, UCAF01, page 1021-1026, San Francisco, CA, USA. 2001, Morgan Kaufmann Publishers Inc. ISBN 1558608122 125.
[6] M. Bowling and M. Veloso. Multiagent learning using a variable learning rate. Artif. Intell., 136(2)2115-55, Apr. 2002. ISSN 0004-3702. doi: 100060000000--7772000002222 URL attps://doi.org:
[7] E. M. de Cote, A. Lazaric, and M. Restelli. Learning to cooperate in multi-agent social dilemmas. Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS '06, page 783-785, New York, NY, USA, 2006. Associ- ation for Computing Machinery. ISBN 1595933034. 1o1:10.1145/11666 160770.. URL https://dol org/10. 1145/1160633 1160770.
[8] S. Abdallah and V. Lesser. Non-linear dynamics in multiagent reinforcement learning algo- rithms. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems Volume 3, AAMAS '08, page 1321-1324, Richland, SC, 2008. Interna- tional Foundation for Autonomous Agents and Multiagent Systems. ISBN 9780981738123.
[9] Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and Mordatch. Learning with opponent-learning awareness. AAMAS '18, page 122-130, Richland, SC, 2018. Interna- tional Foundation for Autonomous Agents and Multiagent Systems.
[10] A. Xie, D. Losey, R. Tolsma, C. Finn, and Sadigh. Learning latent representations to influence multi-agent interaction. In Proceedings of the 4th Conference on Robot Learning (CoRL), 11 2020.
[11] B. C. da Silva, E. W. Basso, A. L. C. Bazzan, and P. M. Engel. Dealing with non-stationary environments using context detection. In Proceedings of the 23rd International Conference on Machine Learning, ICMI, '06, page 217-224, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 11110.1145/11440 143844. 1111443444.14377. //àoi.org/10. 1145/1143844 1143872.
URL https:
[12] Hernandez-Leal, Y. Zhan, M. Taylor, Sucar, and Munoz de Cote. Detecting switches against non-stationary opponents. In Proceedings of the 16th Conference on Au- tonomous Agents and MultiAgent Systems, AAMAS '17, page 920-921, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.

---

[13] D. Sadigh, S. S. Sastry, S. A. Seshia. and A. D. Dragan, Planning for autonomous cars that leverage effects on human actions. In Proceedings of Roboties: Science and Systems (RSS), June 2016. 10011001500//SSS2211.11
[14] K. Tuyls and G. Weiss. Multiagent learning: Basics, challenges, and prospects. Al Magazine, 33(3):41, Sep. 2012. doi: 10::10.. v333. 2426. URLhttps://ojs. aaai org/ //dddexx. hhtp/ammaazzine//rreee htt.g//aaaanerrrriiel///////////
[15] M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In In Pro- ceedings of the Tenth International Conference on Machine Learning, pages 330-337, Morgan Kaufmann, 1993.
[16] W. Z. Wang, M. Beliaev, E. Biyik, D. A. Lazar. R. Pedarsani, and D. Sadigh. Emergent prosociality multi-agent games through gifting. CoRR, abs/2105.06593, 2021. URL https: // //arxivv org/ abs abs/2105. 06593.
[17] M. Carroll, R. Shah, M. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019. ISSN 1049-5258.
[18] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competityy environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6382-6393, Red Hook,
NY, USA, 2017. Curran Associates Inc. ISBN 978151086096
[19] S. Iqbal and F. Sha. Coordinated exploration via intrinsic rewards for multi-agent reinforce- ment learning. arXiv preprint arXiv: 2019.
[20] Foerster, I. A. Assael, N. de Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Cur- Associates, Inc., 2016. URL http:tts::/p/ooceediie nwwriss..ipiccoo. wwwuurpss.ccppppee//00//. cc/ppaper//216/ff1le// pdf.
[21] D. P. Losey, M. Li. J. Bohg, and D. Sadigh. Learning from my partner's 's actions: Roles in decentralized robot teams. In Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
[22] J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi- agent policy gradients. In Thirty-Seconi AAAI Conference Artificial Intelligence, 2018.
[23] H. He, J. Boyd-Graber, K. Kwok, and H. Daumé, III. Opponent modeling in deep reinforce- ment learning. M. Balcan and K. Q Weinberger, editors, Proceedings of The 33rd Inter- national Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1804-1813, New York, New York, USA, 20-22 Jun 2016. PMLR URL http::/prooe http://prroceedings mlr press/v48/hele/ html.
[24] D. Sadigh, N. Landolfi, S. S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for cars that co- ordinate with people: Leveraging effects on human actions for planning and active information gathering over human internal state. Autonomous Robots (AURO), 42(7): (477):4455-1266 1426, October 2018. ISSN 1573-7527. 100 10100000100000008880
[25] Z. Zhu, E. Biyik, and D. Sadigh. Multi-agent safe planning with gaussian processes. In Pro- ceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 2020.
[26] A. Lupu and D. Precup. Gifting in multi-agent reinforcement learning. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, page 789-797, Richland, SC, 2020, International Foundation for Autonomous Agents and Multiagent Sys- tems. ISBN 978 1450375118
[27] J. Yang, A. Li, M. Farajtabar, P. Sunehag. E. Hughes, and H. Zha. Learning to incentivize other learning agents. CoRR, abs/2006.06051 2020. URI. www.https//arxiv..or http://arvvvooora///////// 06051.
10

---

[28] K. Ndousse, D. Eck, S. Levine, and N. Jaques. Learning social learning. 2020. URL https: //arxiv. wwtp:///oo.bbb....oooooo 00581.
[29] H. de Weerd, R. Verbrugge, and B. Verheij. How much does it help to know what she knows you know? an agent-based simulation study. Artificial Intelligence, 199-200:67- 92. 2013. ISSN 0004-3702. doi:https:/ddoi..rgg orgg 100...........tttt221111....... //www. www.sccienedirect.oommmmmm www.sciiocceeeeeeecccccccceeoeee httpa/////.rtttee//.////////1110
URL https:
[30] F. B. von der Osten, M. Kirley, and T. Miller. The minds of many: Opponent modeling in a stochastic game. In Proceedings of the Twenty-Sixth International Joint Conference on Ar- rificial Intelligence, IJCAI-17, pages 3845-3851, 2017. 460:10022963//icai2211775
URL
https:/ http:///doooooo.../// 24963 http:iiacaa...1111/.... 2017/537.
[31] M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. NIPS'15, Cambridge, MA, USA, 2015. MIT Press.
[32] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In K. Chaudhuri and R. Salakhut- dinov, editors, Proceedings of the 36th International Conference Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pages 5331-5340. PMLR, 09-15 Jun 2019. URL http: httt:://proceedingg. mll apees/vv7/reaas w97/rakeelyl9..... htm1...1111...1
[33] R. Julian, E. Heiden, Z. He, H. Zhang, S. Schaal, J. Lim, G. Sukhatme, and K. Hausman. Scaling simulation-to-real transfer by learning composable robot skills. In J. Xiao, T. Kröger, and O. Khatib, editors, Proceedings of the 2018 International Symposium on Experimental Robotics, pages 267-279, Cham, 2020. Springer International Publishing. ISBN 978-3-030- 33950-0.
[34] Y.-M. De Hauwere, Vrancx, and A. Nowé. Learning auent state space representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1 Volume AAMAS '00, page 715-722, Richland, SC, 2010. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9780982657119.
[35] G. Tesauro. Extending q-learning to general adaptive multi-agent systems. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, vol- ume 16. MIT Press, 2004. URL https::/procceeddeeeeeee .eaggs..eurip... ht/ppper/2003/fil// e 77155c111bbc55971111 htt.a//c777..660000010166664-400
pdf.
[36] A. Grover, M. Al-Shedivat, J. Gupta, Y. Burda, and H. Edwards. Learning policy representa- tions in multiagent systems. In J. Dy and A. Krause, editors, Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1802-1811, PMLR, 10-15 Jul 2018. URL http://proceedings mlr. press/v80/ panss//80/groveri1 prover18a. html.
[37] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmant 2017. URL https://arxii http//aaaa.raaarr///////raaaaaee wwtg/bs/////.ree 01144.
[38] C. J. Maddison, A. Mnih, and Y. W. Teh. The Concrete Distribution: A Continuous Relax- ation of Discrete Random Variables. In International Conference on Learning Representations, 2017.
[39] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. AAAL088. AAAI Press, 2008. ISBN 9781577353683.
[40] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learn ing Volume 70, ICML' 17, page 2778-2787. JMLR.org, 2017.
[41] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying casnt-based exploration and intrinsic motivation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 16, page 1479-1487, Red Hook, NY, USA, 2016. Curran Associates Inc ISBN 978 9781510838819.
11

---

[42] K. Byl and R. Tedrake. Metastable walking machines. The International Journal of Robotics Research, 280));1400-104,, 2009. 661:100117777788544499944444 URL https: /ttps://doi. /doi. erg/10. 1177 /011//2227366499444466
[43] G. Berseth, D. Geng, C. Devin, N. Rhinehart, C. Finn, D. Jayaraamn,, and S. Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. In International Con- ference Learning Representattons, 2021. URL https: https://openrevien www/forummiii. cPZDyoDlox1.
[44] F. Deshi-Velez and G. D. Konidaris. Hidden parameter markov decision processes: A semi- parametric regression approach for discovering latent task parametrizations. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, 2016.
[45] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum en- tropy deep reinforcement learning with a stochastic actor. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Pro- ceedings of Machine Leurning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL htttp:/prooceeding: mlr. www..p.ee//v..... wwwpppeeevv00 Satass/v80/haar hat80/haaanooa188... html.
[46] R. Vidal, O. Shakernia, H. Kim, D. Shim, and S. Sastry. Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation. IEEE Transactions on Robotics and Automation, 18(5):662 669, 2002. 100.1..1100//RA22000.0000000
[47] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.01 5400, 2016. URL http:/tp//arzt: org/ /eee//6660000 01540.
[48] E. J. Gumbel. The Maxima of the Mean Largest Value and of the Range. The Annals of Mathematical Statistics, 25(1):76 84, 1954. //doi.org/10 10. 1214/
URL https:
[49] C. J. Maddison, D. Tarlow, and T. Minka. A sampling, 2015.
[50] A. Shih, A. Sawhney, J. Kondic, S. Ermon, and D. Sadigh. On the critical role of conventions in adaptive human-ai collaboration. 9th International Conference on Learning Representations (ICLR), 5 2021.
12


Reviews:
[{"review_id": "j6z785yDax5", "reviewer": "Reviewer_whdT", "summary": "This paper tackles the problem that other agents adapt/change their strategies/behaviors when interacting, resulting in a non-stationary policy. The authors propose to learn a policy that influences the other agents to not change their policy (i.e., to influence the other agents' strategy to stabilize) by penalizing the learning agent with a penalty function of the other agent's distance between the previous and current latent strategies. Hence, the main contributions of this work are:\n1 - A stability reward function influencing the ego-agent to learn a policy minimizing the other agent's changes in their policy/behavior\n2 - A strategy allowing to learn the stability reward function in an unsupervised manner\n\nMoreover, this work presents simulation results demonstrating the claimed contributions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, this paper is very well-written and tackles an important problem to the research community. The contributions and proposed work's motivation are clearly written, the problem statement is well-defined, and the presented results support the work claims.\n\nThe main strengths of this paper are:\n1 - The idea of rewarding the learning agent to stabilize the other's strategies when interacting.\n2 - The proposed method allowing to learn the stability reward in an unsupervised manner.\n\nThe main weaknesses of this work are:\n1 - No video displaying the qualitative results of the proposed method. Adding visual qualitative results can improve the significance of this work and can be important to make a good qualitative evaluation of the method. The results presented in the appendix do not a whole episode.\n2 - The code used for the experiments is not provided. Unfortunately, it is more and more important to release the code used to generate the presented results and allow others to reproduce presented results to avoid people getting articles with false claims published. Open-sourcing the code used for the proposed method will strengthen the contribution of this work and make future works cite it as a reliable reference.\n3 - The experiments are still very simple and far from a real robotic application scenario.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "To summarize, I think this paper tackles an important problem and the proposed method has some novelty. Moreover, the paper is very well-written. However, adding more complex experiments and qualitative results showing the learned policy in action would definitely increase this work contribution. Still, as a conference paper, I think it is in a good state to be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Clarity issues:\n1 - How is the opponent strategy defined? Is z^j a sequence of actions over the rollout j?\n2 - In line 145, in the definition of fixed-point strategy, how is \\epsilon defined? What are the conditions on \\epsilon for the definition to hold?\n\nNotation problems:\n1 - Line 157, the V function is not defined. What is V?\n2 - Line 145 and182, N is not defined. \n\nThe related works section is missing an important reference and it is important to highlight how this work differs from the following references:\n1 - Yang, Jiacheng et al. “Learning to Incentivize Other Learning Agents.” ArXiv abs/2006.06051 (2020)\n\nPossible improvements:\n1 - Ablation study and discussion on the impact of beta in the learned policy's performance.\n2 -  How is the action space defined for each experiment? In some experiments, it is not clear.\n3 - Adding qualitative results in the paper or attach a video clearly showing the learned policy in action.\n\n"}}, {"review_id": "VMg86KMBXMr", "reviewer": "Reviewer_naZQ", "summary": "This paper proposes a novel reinforcement learning framework for multi-robot coordination. In this framework, the authors use a modified reward function that, in addition to maximizing long-term discounted reward, works to stabilize the other agents' policies to facilitate learning in the ego agent. They show through detailed experiments that this augmented reward function improves the agent's ability to coordinate and maximize the task reward.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and tackles a  relevant problem. The authors give a thorough description of their approach as well as experiments validating their claims. In addition, the authors provide clear discussion sections which summarize the results and discuss limitations.\n\nHowever, the single figure used to display the results of comparing the proposed method across various environments is not descriptive enough to give a complete understanding of the framework's performance. More information on the results of these experiments would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and tackles a well-studied but\nstill relevant problem. The authors provide a good\nliterature review and clearly explain their architecture,\npresenting some experiments to support their claim.  I believe it is suitable for CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": " The single figure used to display the results of comparing the proposed method across various environments is not descriptive enough to give a complete understanding of the framework's performance. More information on the results of these experiments would be helpful.\n \nMinor comments\n1) Real-world on page one should be hyphenated\n2) Us and our teammate should be our teammate and us on page 2"}}, {"review_id": "6HLzLmw_oec", "reviewer": "Reviewer_mhit", "summary": "In a two-player game, an ego-agent's reward can be augmented to encourage it to take actions that will stabilize the partner's strategy. The stability of the partner's strategy is defined by the change in a latent parameter that the ego-agent learns via unsupervised learning from the partner's previous trajectories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\nThe idea of stability as presented in the paper is very clear. The implementation of the stability reward is also very intuitive and straightforward.\nThe authors provide an extensive review of related work in multi-agent learning. The comparison of stability and surprise is especially interesting. \nThe authors provide experimental results on several tasks that demonstrates that the SILI approach exceeds state of the performance of surprise-based algorithms.\n\nWeaknesses & Questions:\nThe authors formalize a two-player game as a Hidden Parameter MDP, and leverage the assumptions of a static opponent strategy transition function. The application of this work to two-player games is poorly motivated.\nI'm worried that this approach would have limited applicability in real human-robot interaction, and especially for competitive or adversarial tasks, in which an opponent may adapt ttheir policy or strategy to disguise their actions and win. \nMore discussion is needed about types of tasks in which it is impossible to stabilize the opponent's strategy.\n\nGiven strategy transition functions that are fixed, we can instead look at trajectories of N>>1 episodes in sequence. This new task may be considered an MDPs since the opponent's $T^z$ is stationary. One valuable baseline would be to apply SAC or standard RL algorithms to episodes that are composed of 10-100 interactions. \n\nThe authors may want to avoid the use of the term \"opponent\" since no adversarial tasks are used. Even in the Circle Point Mass task, the \"opponent\" has a stationary strategy and has no ability to adapt it to avoid capture.\n\nFor the Sawyer-Reach task, is the opponent's strategy the choice of goal?\n \nThe Detour Speaker-Listener task is the most interesting because the speaker's policy changes randomly if the ego-agent fails. What I'm surprised by is the inability of the Oracle to match the performance of SILI. This be due to the representation of the communication strategy provided to the agent. \n\nWere there any challenges with tuning the relative magnitudes of the rewards, especially for not purely cooperative tasks?\nDid the authors face any problems of convergence to sub-optimal policies? Seems like the stability reward could discourage exploration. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors apply the Hidden Parameter MDP formalism to two-player games, which is an extremely restrictive assumption as compared to the space of all two-player games. But given this problem formulation, the proposed algorithm is compelling and may be useful for partially observable control tasks. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The authors should add a discussion of the limitation of the Hidden Parameter MDP formalism when applied to multi-agent tasks.\n\nThe authors should provide more detail about the dynamics of the opponent strategy for the given tasks. \n\nThe authors should consider avoiding the use of the word opponent, since most of the tasks presented are cooperative. "}}, {"review_id": "07VE34HjswL", "reviewer": "Reviewer_CoBB", "summary": "This paper proposes to encourage agents to stabilize other agents thereby rendering the environment stationary and easier to learn. The underlying approach is to use an added reward function that encourages the agent to stabilize the other agent's strategy.\nExperiments in various domains (e.g., driving, sawyer-reach, speaker-listener) suggest that the proposed SILI works well. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the whole, the paper proposes a simple yet effective idea for stabilizing interactions between agents. To my knowledge, SILI is novel and the results are relevant for the robotics community, especially those in multi-agent learning and human-robot interaction. The paper is well-written on the whole (up to the points below). Extensive experiments in various domains validate the main claims and shows SILI works well against the baselines and unmodified SAC. \n\nMy only major concern is regarding the learnt latent strategy space $\\mathcal{Z}$. The entire approach hinges upon a well-structured latent space, but the paper is vague about how exactly $\\mathcal{Z}$ is learnt. Specifically, the authors state the approach is unsupervised but how are the data/trajectories/strategies sampled? It seems crucial that $\\mathcal{Z}$ is sufficiently \"smooth\" and that nearby latent representations correspond to similar strategies; this can be challenging since only noisy trajectories are observed? \n\nIn more complex scenarios, it may be very difficult to learn a stabilizing strategy, especially with humans or sophisticated artificial agents. Again, this is contingent upon a good latent space and the inherent complexity of the other agent. While I think this paper is a step in the right direction, it's difficult to see if it will scale up.  \n\nAs a minor comment, what stabilizing strategies were learnt in the various domains? I think qualitative descriptions would really help to increase the impact of the work. In particular, how do they differ to the non-stabilizing strategies? I understand that space is an issue but Fig 4 in the appendix should be in the main paper. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I lean towards acceptance. This work nicely executes a first-step towards more effective and stable interactions. I hope the authors can address my question above about the learnt latent strategy space.  \n\n### Post Response \nThanks for the response about the latent strategy space and for adding the qualitative results. Overall, I remain in favor of accepting this work. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The major issue for me was about how the latent space $\\mathcal{Z}$ is learnt. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this work tackles an important problem in multi-agent domain. The idea proposed in this paper is also interesting together with the promising results in experiments. Reviewers raised consistent comments that this paper is well-written and organized, and the novelty also makes sense. However, reviewers also care about a lot, such as releasing the video results and source codes. The experimental setting is a little bit simplified, which sounds like still far away from the real world scenarios. Authors should also explain more details about the learning procedure.

During the rebuttal period, authors have replied most of questions from reviewers. Most of reviewers think this is a nice work and recommend the acceptance. 
Decision: Accept (Oral)