Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games
Authors: Ziyi Liu
Guo Xian
Yongchun Fang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning
risk-sensitive reinforcement learning
general-sum games
generalization

Abstract: While various multi-agent reinforcement learning methods have been proposed in cooperative settings, few works investigate how self-interested learning agents achieve mutual coordination in decentralized general-sum games and generalize pre-trained policies to non-cooperative opponents during execution. In this paper, we present a generalizable and sample efficient algorithm for multi-agent coordination in decentralized general-sum games without any access to other agents' rewards or observations. Specifically, we first learn the distributions over the return of individuals and estimate a dynamic risk-seeking bonus to encourage agents to discover risky coordination strategies. Furthermore, to avoid overfitting opponents' coordination strategies during training, we propose an auxiliary opponent modeling task so that agents can infer their opponents' type and dynamically alter corresponding strategies during execution. Empirically, we show that agents trained via our method can achieve mutual coordination during training and avoid being exploited by non-cooperative opponents during execution, which outperforms other baseline methods and reaches the state-of-the-art.


Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games
Anonymous Author(s)
Affiliation
Address
email
Abstract
While various multi-agent reinforcement learning methods have been proposed in cooperative settings, few works investigate how self-interested learning agents achieve mutual coordination in decentralized general-sum games and generalize pre-trained policies to non-cooperative opponents during execution. In this paper, we present a generalizable and sample efficient algorithm for multi-agent coor- dination in decentralized general-sum games without any access to other agents' rewards or observations. Specifically, we first learn the distributions over the return of individuals and estimate dynamic risk-seeking bonus to encourage agents to discover risky coordination strategies. Furthermore, to avoid overfitting opponents' coordination strategies during training, we propose an auxiliary opponent modeling task so that agents can infer their opponents' type and dynamically alter corre- sponding strategies during execution. Empirically, we show that agents trained via our method can achieve mutual coordination during training and avoid being exploited by non-cooperative opponents during execution, which outperforms other baseline methods and reaches the state-of-the aftt
2
3
10
11
12
13
Introduction
16
17
Inspired by advances in deep reinforcement learning (DRL) 13 many researchers recently focus on utilizing DRL methods tackle multi-agent problems 46 However, most of these works either consider the fully cooperative multi-agent reinforcement learning (MARL) settings [7-11] general- sum games but make restrictive assumptions about opponents 14,14, e.g., either stationary [13 or altruistic 15 16. Considering future applications of MARL, such as self-driving cars 17 and
18
20
21
22
human-robot interactions 18, multiple learning agents optimize their own rewards independently in
23 general-sum games where win-win outcomes are only achieved through coordination which often coupled with risk 19, pre-trained policies should generalize to con-coopeea opponents during execution.
24
12 20 ("Risk" refers the uncertainty of future outcomes (1)). and their
25
26
To achieve coordination alongside other learning agents and generalize learned policies to non- cooperative opponents, the agent must be willing to undertake a certain amount of risk and identify the opponents' type efficiently. One set of approaches use explicit reward shaping to force agents to coordinate 22 16 [5], which can be viewed as an approach to shape the risk degree of coordination
27
28
29
30
strategies. To learn generalizable policies, [15] 20 propose to train an adaptive agent with population-
31
based training methods. Other works either treat the other agents stationary [3 23 24 20
or
directly access to opponent's policy parameters 22
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do distribute.

---

33
By contrast, we are interested in a less restrictive setting where we do not assume access opponents' rewards, observations, or policy parameters, instead, each agent can infer other agents' current strategies from the past behaviors of other agents. In this paper, one key insight that learning from opponent's past behaviors allows the agent to infer the opponent's type and dynamically alter his strategy between different modes, e.g., either cooperate or compete, during execution. Moreover, non-stationary, decision-makinn over the agent's return
34
35
36
37
given that the other learning agents distributions enables the agent to tackle uncertainties resulting from other agents' behaviors and alter his risk preference, i.e., from risk-neutral to risk-seeking, to discover coordination strategies. Motivated by the analysis above, we propose GRSP, Generalizable Risk-Sensitive MARL algorithm and our contributions are summarized as follows:
41
42
Leading mutual coordination in decentralized general-sum games. We estimate dynamic risk- seeking bonus using complete distortion risk measure Wang's Transform (WT) 25 to encourage agents discover risky cooperative strategies. The risk-secking bonus only affects the action selection procedure instead of shaping environment rewards and decreases throughout training, leading to an unbiased policy.
45
48
47
48
Generalizing pre-trained policies non-cooperative opponents during execution. Policies learned independently can overfit the other agents' policies during training, failing to sufficiently generalize during execution 26). We further propose to train each learning agent with two objectives: standard Quantile Regression objective (27 [8] and supervised agent modeling objective, which models the behaviors of opponent, applied on intermediate representation of the value network. The auxiliary opponent modeling task allows the policy to be influenced by opponent's past behaviors, forcing the intermediate representation to adapt to the new opponent.
51
52
59
54
Evaluating in multi-agent settings. We evaluate GRSP in four different Markov games: Monster- Escalation [6]. Iterated Prisoners' Dilemma (12 20 and Iterated Stag Hunt Compared with several baseline methods, including MADDPG 30), MAPPO [AC and LOLA(2), GRSP learns substantially faster, achieves mutual coordina- tion during training and can generalize to the non-cooperative opponent during execution, which
56
Hunt 15 29,,
57
(ISH) 19 15
58
LIAM 13
59
60
outperforms other baseline methods and reaches the
2
Related Work
61
Risk-sensitive RL. Risk-sensitive policies, which depend upon more than mean of the outcomes, enable agents to handle the intrinsic uncertainty arising from the stochasticity of the environment. In MARL, the intrinsic uncertainties are amplified due to the non-stationarity and partial observability created by other agents that change their policies during the learning procedure 33 35 Distributional
63
64
65
66
R 28 provides a new perspective for optimizing policy under different risk preferences within a unified framework 21 37]. With distributions of return, is able to approximate value function under different risk measures, such Conditional Value at Risk (CVaR) (38] [3] and 2 and thus produce risk-averse or risk- seek-ng policies. Qiu et al. [ propose RMIX with the CVaR measure as rivr-everse policies. Similar ideas are proposed in D4PG 40) and DFACH4]. In contrast with these works that focus on the fully cooperative settings and do not consider generalization, this paper
87
70
71
72 proposes the first algorithm that leverages risk-seeking policies to achieve coordination strategies in general-sum games and generalizable to non-cooperative opponents during testing phase.
73
Generalization across different opponents. Many real world scenarios require agents to adapt to different opponents during execution. However, most of existing works focus on learning fixed and team-dependent policy in fully cooperative setting 42
75
78
10] which can not generalize to slightly altered environments or new opponents. Other works either use a population- based training method to train an adaptive agent( [5]], or adapt to different opponents under the Tit-for-Tat Our work closely related to test-time training methods 44 45 However, they
77
78
79 principle
80 focus on image recognition or single agent policy adaption. Ad hoc teamwork[46 47 also requires
agents to generalize new teams, but they focus cooperative games and has different concerns with us.

---

83
Opponent modeling. Our approach to learning generalizable policies can be viewed as a kind 84 of opponent modeling m These approaches either model intention 49 50 assume an
85 assignment of [[]] or exploit opponent learning dynamics 12 52 Our approach is similar
86 to policy reconstruction methods[50) which make explicit predictions about opponent's actions. However, instead of predicting the opponent's future actions, learn from opponent's past behaviors update the belief, i.e., parameters of value network, of the opponent's type.
87
3 Preliminaries
89
Stochastic games. In this work, consider multiple self-interested learning agents interact with each other. We model the problem as Partially-Observall Stochastic Game (POSG) [53] 544, which consists of N agents, a state space S describing the possible configurations of all agents, set of actions A A and a set of observations O ,....., ON for each agent. At each time step, each agent i receives his own observation o² E O', and selects an action E A'' based on a stochastic policy n' O¹ A' [0, 1], which results in joint action vector a. The environment then transitions new state s' based the transition function P(s [s,a). a). Each agent obtains rewards function of the state and his action R S X A' + R. The initial states are determined by distribution p S + [0, 1]. We treat the reward "function" R of each agent as random variable to emphasize its stochasticity, and use Z (s, a') Σ 7 (sz, ai) to denote the random variable of the cumulative discounted rewards where So 8, AA a² , is discount factor and T is the time horizon.
91
92
93
94
95
96
98
99
102
Distorted expectation. Distorted expectation is risk weighted expectation of value distribution under specific distortion function[55]. A function [0, 1] + [0. 1] is distortion function if is non- nee-escegg and satisfies g(0) 0 and g(1) = 1[6). The distorted expectation of under g is defined as v = Jo Fz (T) Su I'(r) (r)dT, (r)dT, where Fz is the quantile function at T E [0, 1] for the random variable Z. We introduce two common distortion functions as follow:
103
104
105
106
CVaR is the expectation of the lower upper tail of the value distribution, corresponding risr-averse or risk-seeking policy respectively. Its distortion function is g(r) min(r/a, 1) (risk-averse) max (0, 1 -)/a) (risk-seeking), a E (0, 1) denotes confidence level.
WT is proposed by Wang 25 (T) = ( (T) ( A) where & is the distribution standard normal. The parameter À called the market price of risk and reflects systematic risk. À > 0 for risk-averse and for risk-seeking.
110
111
112
113
CVaRa, assigns O-value to all percentiles below the a or above 1 - a significance level which leads to erroneous decisions in some cases (56). Instead, WT is a complete distortion risk measure and ensures using all the information in the original loss distribution which makes training much more stable, and we will empirically demonstrate it in Sec.
114
115
116
4 Methods
118
In this section, we describe our proposed GRSP method. We first introduce the risk-seeking bonus used to encourage agents to discover coordination strategies in Sec. 4.1 and then propose the auxiliary
opponent modeling task to learn generalizable policies in Sec.
4.2
Finally, provide the details of
test-time policy adaptation under different opponents in Sec. 4.3
122
4.1
Risk-Seeking Bonus
123 In this section, we first provide an illustrative example for the insight behind risk-seeking bonus and
124 then describe its details. Consider a two-player 10 steps Sequential matrix game Stag Hunt, where
each player should decide whether to hunt stag (S) hunt hare (H) in each round. If both agents choose they will receive the highest payoff 2. However, if one agent defects, he will receive
3

---

127
descent reward for eating the hare alone while the other agent with an S action will suffer from big loss 100. both agents choose H they will receive payoff 1.
128
129
Even state of the art RL algorithms fail discover the "risky" cooperation strategies [13 16 [9] One important reason is that the expected, i.c., nist-neutral, Q value ig- nores the complete distribution information, especially the upper and lower tail information when the learned distribu- tion is asymmetric. Another reason is that when the risk is high, i.e., a high loss for being betrayed, the probability of finding the S-S (Cooperation) strategy via policy gradient
130
60
40
Quance Dotiibucion
20
132
133
20
134
-10
135
136
137
very low 15].
Noutral
WT Wiighted
138
Therefore, we adopt the distributional RL method to model the whole distribution of Q value. Fig left part shows the quantile distribution of cooperation and defection of risk risk-neutral neutral policy learned by QR-DQN[28). The mean
Figure 1: Quantile value distribution of cooperation and defection in Sequential Stag Hunt weighted by WT compared with risk- policy.
139
140
141
142
value of defection is higher than that of cooperation, but the quantile value distribution of cooperation has longer upper tail which means that it has higher potential payoff.
143
144
We propose to use WT distortion function to reweight the expectation of quantile distribution. By following (28). we first represent the return distribution of each agent i with policy n² by uniform
145
146
mix of M supporting quantiles:
Zí (0' a') = M Σ (or
where d denotes a Dirac Delta functions at x E IR, and each og an estimation of the quantile corresponding to the quantile fractions f = Tk-1 +7h with The = k
147
148
M
for O
S h & M. The state-action
149 value Q (o', a²) can then be approximated by En= 0f (o',a').
150
Furthermore, the risk-seeking bonus for agent i is defined as:
(Zz')) = f. (T)d
1 M
M Og
where 9 (T) is the derivatives of WT distortion function [0, 1]. and À controls the risk-seeking level. Fig right part shows the WT weighted quantile distribution in which the upper quantile values are multiplied by bigger weights and lower quantile values are multiplied by smaller weights to encourage agents to adopt risky coordination strategies.
152
153
154
155
naive approach exploration would be use the variance of the estimated distribution as a bonus. shows that the exploration bonus from truncated variance outperforms bonus from the variance. Specifically, the Right Truncated Variance tells about lower tail variability and the Left Truncated Variance tells about upper tail variability. For instantiating optimism in the face of uncertainty, the upper tail variability is more relevant than the lower tail, especially if the estimated distribution is asymmetric. So we adopt the Left Truncated Variance of quantile distribution to further leverage the intrinsic uncertainty for efficient exploration. The left truncated variance is defined as
57
158
159
160
161
M
0
1
Σ 1=
-0;)) 0,)² -0)=0))²
2M
162 and analysed in (57 The index starts from the median, i.e., M/2, rather than the mean due to its well-
163 known statistical robustness (58 59 We anneal the two exploration bonuses dynamically so that in
164 the end we produce unbiased policies. The anneal coefficients are defined as Grj = c.y logt = 1,
which is the parametric uncertainty decay rate (60), and сj is constant factor. This approach leads to

---

Training....
Testing
Eplobar)
%1.==
Gradient
Eplof)
Dej
Ausilary Opponent Modaling Taak
Gradient
Edlof
Dvr
Audilary Opponent Modaling Task
E
Eglo!)
Dva
Huber Quantie Regression Loes
Eg
V/a
bonus
Gradient
bonus
Gradient
Figure 2: Left: Diagram of GRSP architecture during training. Outputs of & are fed into Dv, and De., so features are shared between policy and auxiliary opponent modeling. The prediction head De. outputs other agents' actions. Right: Test- Dest-Time policy adaptation. The agent can not receive environment rewards during testing, so we only optimize the auxiliary opponent modeling objective.
186
choosing the action such that
arg max a E'EAA' (o) on c2) (d + cay (ot at)
167 These quantile estimates are trained using the Huber 61 quantile regression loss. The loss of the quantile value network of each agent at time step is then given by
188
M
M
af =
1
Σ
Pix (5t)
x
(5)
M
A=
j=0
where 0 - +03 (0i+1 T² (oi+1) 0% (of, a)), and pt (x) - l&v-
I 1xx
o0}
fa(x)
where I
170
is the indicator function and L, (I) is the Huber loss:
(x) = 1222 (x (|z) -1x)-1a)
T otherwise
(6)
[x)
4.2
Auxiliary Opponent Modeling Task
In order alter the agent's 's strategies under different opponents, share parameters between policy and auxiliary opponent modeling task. Specifically, we split the Q value network into two parts: feature extractor & and decision maker Dv... The parameters of the Q value network Qa: for agent i are sequentially divided into o' and vit. i.e., 0² = (ø', wh)). The auxiliary opponent modeling task shares a common feature extractor Eg. with the value network. We can update the parameters of Eg. during execution using gradients from the auxiliary opponent modeling task, such that Tg can generalize different opponents. The supervised prediction head and its specific parameters are Dey
178
174
175
176
177
178
179
with dg. The details our network architecture are shown in Fig. 2
During training, the agent i can collect set of transitions f(o]. of:1, a, 1) where a, indicates the joint actions of other agents except at time step 1. We use the embeddings of agent i's observations
182
of and 01+1 to predict the joint actions a," i.c., the D& is multi-head neural network whose outputs
183 are multiple soft-max distributions over the discrete action space or predicted continuous actions
184 of each other agent, and the objective function of the auxiliary opponent modeling task can be
185
formulated as
L (of, a (000 N
Σ j=1.f$1
(dl, (Ear (ot) Egt (o)=
(7)
5

---

186
where E(-) is the cross-entropy loss function for discrete actions or mean squared error for continuous actions. The strategies of opponents will change constantly during the procedure of multi-agent exploration and thus various strategies will emerge. The agent can leverage them to gain some
187
188
189
experience about how to make best response by jointly optimizing the auxiliary opponent
190
modeling task and quantile value distribution. The joint training problem is therefore
min 1 1
I (o), o v.. + C a
(8)
191
4.3 Test-Time Policy Adaptation under Different Opponents
192
During testing time, we can not optimize J anymore since the reward unavailable, but we assume the agent can observe actions made by his opponents during execution, then we can continue optimizing J to update the parameters of feature extractor & Learning from opponents" past behaviors at test time makes the agent generalize his policy to different opponents efficiently, The can be formulated as
193
min of 06
Experiments
197
In this section, we empirically evaluate our method on four multi-agent environments. In sec. 5.1 we introduce the four environments we use for experiments and training settings. In sec. 5.2 we compare
199
200
the performance of GRSP with other baselines. In sec. 5.3 we evaluate the generalization ability of
201
GRSP under different opponents during execution. The ablations are studied in sec. 5.4 Further
202
understanding of GRSP is presented in sec. 5.5 More details can be found in Appendix C
203
5.1
Environment Setup
204
Repeated games. We consider two kinds of repeated matrix games: Iterated Stag Hunt (ISH) and Iterated Prisoners' Dilemma (IPD). Both of them consist two agents and a constant episode length of At each time step, the agents can choose either cooperation or defection. both agents choose to cooperate simultaneously. they both get bonus of 2. However, if a single agent choose cooperate, he gets penalty of -10 in ISH and -1 in IPD, and the other agent get bonus of I and 3, respectively. If both agents choose to defection, they get bonus of in ISH and 0 in IPD. The optimal strategy in ISH and IPD is to cooperate at each time step, and the highest global payoffs of two agents are 40, i.e., 20 for each of them.
205
206 10 time steps [2 [9] 19
207
208
210
212
grid-world, consisting of two agents, two apples and one monster. The apples are static while the monster keeps moving towards its closest agent. When single agent meets the monster, he gets penalty of -10. If two agents catch the mon- ster together, they both get a bonus of 5. If a single agent meets apple, he get a bonus of 2. Whenever an apple is eaten or the monster meets an agent, the entity will respawn randomly. The optimal strategy, i.c., both agents move towards and catch the monster, risky coordination strategy since an agent will receive a penalty if the other agent deceives.
Monster-Hunt.
The environment
213
Apple
216
217
218
Aguszt
219
Figure 3: Monster-Humi
220
Escalation. Escalation is 5 x grid-world with sparse rewards, con- sisting of two agents and a static light. If both agents step on the light simultaneously, they receive bonus of 1, and then the light moves to
222
223
Aguett
224 a random adjacent grid. only one agent steps on the light, he gets a penalty of 1.57. where denotes the latest consecutive cooperation steps, and the light will respawn randomly. To maximize their individ-
Ligg
225
228
ual payoffs and global rewards, agents must coordinate stay together and step the light grid. For each integer L, there is corresponding
Figure 4: Escalation.

---

coordination strategy where each agent follows the light for L steps then
230
simultaneously stop coordination.
231
Training. We carry out our experiments on one NVIDIA RTX 3080 Ti and Intel i9- 9190K.
GRSP (ours)
MADOPG
MAIPO
IAC
UAM
LOLA
Optimal
terated Stag Hunt
Iterated in scnero Diicmma
Aueray Coba Seeoots.
50
206
43k
62c
82t
100k
20k
40c
20k
22k
100k
Time Stops
Stapo
Figure 5: Mean evaluation returns for GRSP, MADDPG, MAPPO, IAC, LIAM and LOLA on two repeated matrix games. The average global rewards equal to 40 means that all agents have learned coordination strategy, i.e., cooperating at each time step.
5.2
Evaluation of Returns
In this subsection, we evaluate all methods on four multi-agent environments and use 5 different random seeds to train each method. We pause training every 50 episodes and run 30 independent episodes with each agent performing greedy action selection to evaluate the average performance of each method.
234
235
236
237
5.2.1
Iterated Games
238
Fig. S shows the average global rewards, i.e., the summation of all agents' average returns, of all methods evaluated during training in ISH and IPD environments. The shadowed part represents 95% confidence interval. The average global rewards equal to 40 means that all agents have learned coordination strategy, i.e., cooperating at each time step. We can find that agents trained with our method can achieve mutual coordination in sample efficient way in two repeated matrix games with high risk while other methods only converge to safe non-cooperative strategies though some of them have much more restrictive assumptions.
242
243
244
245
5.2.2
Grid-Worlds
We further show the effectiveness of GRSP in two grid-world games, Monster-Hunt and Escalation[ both of which have high payoff but risky cooperation strategies for agents to converge
to. Fig. 6 shows that, compared with other baseline methods, GRSP constantly and significantly outperform baselines with higher sample efficiency over the whole training process both in global
249
250
rewards and agent's 's individual rewards. Specifically, in Hoonseer-Hunt GRSP agents efficiently find one of the risky cooperation strategies where two agents stay together and wait for the monster. Furthermore, the policies learned by each agent are very stable and neither would like deviate from the cooperative strategy. However, other baseline methods only converge safe non-cooperative strategies and get low payoff due to their poor exploration. seems that LOLA can not learn useful strategies in more complex environments. In Escalation, GRSP outperforms other baselines significantly and both agents have achieved coordination in a decentralized paradigm.
251
252
256
257
5.3
Generalization Study
258
This subsection investigates how well the pre-trained GRSP agent can generalize to different oppo- nents, i.e., cooperation or defection, during execution. The cooperative opponents are trained by
259
GRSP method while the non-cooperative opponents trained by MADDPG. During evaluation, random seeds of four environments different from that during training, and hyperparameters
7

---

GRSP (ours)
MADDPG
MAIYO
IAC
LAMM
LOLA
Monsten Globel
Monsser-fure Agent
Moosteentar Age't
May Broonds
20
10
-20
asciaation Claival
Escolation Agiint
Escolation Agiitt
Mo:0 7100015
10
1934
1004
2014
1054
Figure 6: Mean evaluation returns for GRSP, MADDPG, MAPPO, IAC, LIAM and LOLA on Monster-Hunt and Escalation. Global rewards are summation of both agents rewards.
Table I: Mean evaluation return of GRSP with and without auxiliary opponent modeling task on four multi-agent encironments.
Oppo: Coop(Defect)
ISH
IPD
M-H
Escalation 9.45( -..545)
GRSP-No-Aom GRSP-Aom
20( 1 100)
20(-5)
20.62(-15.03)
20(0.65)
1.08)
(..36(122.77
11.3
282
of the GRSP are same and fixed between different opponent types. Furthermore, the pre-trained coordinated agents not access the rewards to update their policies anymore and they must utilize the auxiliary opponent modeling task to force them to adapt to different opponents. The network details and hyperparameters can be found in Appendix B
265
286
Table I shows the mean evaluation return of GRSP agent with and without the auxiliary opponent modeling task on four multi-agent environments when interacting with different opponents. All returns averaged 100 episodes. The performance of the GRSP-Aon agent that utilizes the auxiliary opponent modeling task adapt different opponents outperforms that of the GRSP-No- Aom agent significantly, especially when interacting with non-cooperative opponents. Specifically, the GRSP-Aom agent using history behaviors of its opponents to update its policy can learn to alter its strategy from coordination to not when encountering a non-cooperative opponent. The empirical results further demonstrate that policies learned independently can overfit to the other agents' policies during training, and our auxiliary opponent modeling task provides method tackle this problem.
267
272
273
274
275
5.4
Ablations
In this subsection. we perform an ablation study examine the components of GRSP better understand our method. GRSP is based on QR-DQN and has three components: the risk-seeking exploration bonus, the left truncated variance (Tv) and the auxiliary opponent modeling task (Aom). We design and evaluate six different ablations of GRSP in two grid-world environments, as show in Fig. The performance of GRSP-No-Aom which we ablate the Aom module and retain all other 280
278
277
281
features of our method a little lower than that GRSP but has a much higher variance, indicating
282
that learning from opponent's 's behaviors can stable training and improve performance. Moreover, completely decentralized method whose training without any opponent information, and the ablation results of GRSP-No-Aom show that our risk-seeking bonus essential
283
the GRSP-No-Aom
284
for agents achieve mutual coordination in general-sum games. We observe that ablating the left truncated variance module leads to significantly lower return than the GRSP in the Escalation

---

GRSP (curs)
GRSP No Acm
GRSP No TV
GRSP No Reiseskingoous
GRSP CVBR
1DR DQN
Morstee- Punt
Fsceation
20
-00
324
1208
1501
2018
356
2018
130+
200R
31655
Figure 7: Mean evaluation return of GRSP compared with other ablation methods in two grid-world multi-agent environments.
287
but no difference in the Monster-Hunt. Furthermore, ablating the risk-seeking bonus increases the training variance, leads to slower convergence and perform worse than the GRSP. It is noteworthy that the Escalation is sparse reward and bard-exploration multi-agen environment since two decentralized agents can get reward only if they navigate to and step the light simultaneously and constantly. These two ablations indicate that the exploration ability of left truncated variance is important to our method and the risk-seeking bonus can encourage agents to coordinate with each other stably and converge to high-risky cooperation strategies efficiently. We also implement our risk-seeking bonus by CVaR instead of WT, and the results are shown as GRSP- CVaR. The GRSP-CVaR performs worse than our method and has higher training variance. Finally, we ablate all components of the GRSP and use e-greedy policy for exploration which leads to the IQR-DQN algorithm. As shown in Fig. 7 IQR-DQN can not learn effective policies in the Monster-Hunt and perform badly in the Escalation.
288
281
292
293
294
295
298
299
5.5
Understanding GRSP
300
The action whose value distribution has long upper tail means that taking this action may receive higher potential payoffs. However, its mean value may be lower than other actions since its distribution has longer lower tail, as shown Fig. [ Neutral-Coop, indicating higher risk. So agents with the expected RL method will not select this action. In GRSP, the risk-seeking exploration bonus encourages agents pay more attention actions whose distribution has longer upper tail. So
301
302
303
304
agents with GRSP method will be less likely defect their opponents since defects bring lower future returns, more likely to coordinate with other agents, and more tolerant of the risk. Furthermore, the auxiliary opponent modeling task can alter the agent's 's strategy from cooperation to defection if it pairs with a non-cooperative opponent. Empirically, the two components can constitute a kind of cquilibrium strategies, e.g., tit-for-tat 20), between agents.
306
307
308
309
Discussion
310
311
While various MARL methods have been proposed in cooperative settings, few works investigate how self-interested learning agents can achieve mutual coordination which is coupled with risk in decentralized general-sum games and generalize learned policies to non-cooperative opponents during execution. In this paper, we present GRSP, a novel decentralized MARL algorithm with estimated risk-seeking bonus and auxiliary opponent modeling task. Empirically, show that agents trained via GRSP can not only achieve mutual coordination during training with high sample efficiency but generalize learned policies to non-cooperative opponents during execution, while other baseline methods can not.
Conclusion.
312
313
314
315
317
318
319
Limitations and future work. The risk-seeking bonus in GRSP estimated using WT distorted expectation and its risk-sensitive level a hyperparameter that can not dynamically change throughout training. Developing method that can adjust agents' risk-sensitive levels dynamically by utilizing their observation, rewards, opponents' information is the direction of our future work.
320

---

References
323
324
[1]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
325
326
327
[2] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 2015.
328
329
330
[3] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354 -59, 2017.
331
332
333
[4] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
334
335
[5] Arambam James Singh, Akshat Kumar, and Hoong Chuin Lau. Hierarchical multiagent reinforcement learning for maritime traffic management. 2020.
337
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 191.. 26668,, 2019.
339
340
[7] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
343
[8] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages
346
347
PMLR, 2018.
[9] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
349
350
[10] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent l-leanning arXiv preprint trXiv:2008.01066 2020.
352
353
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies forcooperative reinforcement learning agents. Advances in Neural Information Processing Systems, 34, 2021.
355
[12] Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent- learning awareness. arXiv preprint 1799:467.. 2017.
359
[13] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
382
[14] Richard Mcaling and Jonathan L Shapiro. Opponent modeling by iopectation-maximizzae and sequence prediction in simplified poker. IEEE Transactions on Computational Intelligence and Al in Games, 9(1::11-24, 2015.
385
[15] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2103.04:: 2021.
10

---

388
[16]
Alexander Peysakhovich and Adam Lerer. Prosocial learning agents solve generalized stag hunts better than selfish ones. arXiv preprint arXiv: (oxXxx:7099002000 770.0286999 2017.
369
370
[17] Behrad Toghi, Rodolfo Valiente, Dorsa Sadigh, Ramtin Pedarsani, and Yaser Fallah. Social coordination and altruism in autonomous driving. arXiv preprint arXiv:2/07.00200, 2021.
372
[18]
Hirokazu Shirado and Nicholas A Christakis. Locally noisy autonomous agents improve global
373
human coordination in network experiments. Nature, 545(7654)3370- 374, 2017.
374
[19] Woodrow Wang, Mark Beliaev, Erdem BIyık, Daniel A Lazar, Ramtin Pedarsani, and Dorsa Sadigh. Emergent prosociality in multi-agent games through gifting. arXiv preprint arXiv:2105.06593, 2021.
377
[20] Weixun Wang. Jianye Hao, Yixi Wang, and Matthew Taylor. Towards cooperation sequential prisoner's dilemmas: a deep multiagent reinforcement learning approach. arXiv preprint arXiv:1803.00162, 2018.
378
379
[21] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018.
382
383
[22] Ardi Tampuu, Tambet Matüisnn Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12()):e0172395, 2017.
386
[23] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International conference on machine learning, pages 1802-1811. PMLR, 2018.
387
388
389
[24]
Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10823 2020.
390
391
[25]
Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
392
393
[26]
Mare Lanctot, Vinicius Zambaldi. Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified thee-the approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
394
395
396
[27] Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, pages 33-50, 1978.
398
[28)
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
399
400
[29]
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via neward-switching policy optimization. In Deep RL Workshop NeurIPS 2021, 2021.
402
403
[30] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competinnn environments. Advances in neural information processing systems, 30, 2017.
404
408
[31] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arX1v:2103.01955, 2021.
407
408
[32] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33: 10707-10717, 2020.
11

---

[33] Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on Robot Learning, pages 1132-1143. PMLR, 2022.
412
413
414
(34) Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Deal- ing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv:1906.04737, 2019.
415
416
417
[35] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv:1707.0918,, 2017. 2017.
418
419
420
[36]
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
421
422
423
[37] Jared Markowitz, Ryan Gardner, Ashley Llorens, Raman Arora, and I-Jeng Wang. A risk- sensitive policy gradient method. 2021.
424
[38] R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of banking & finance, 26(7):1443-144) 2002.
426
[39] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: evar optimization approach. Advances in neural information processing systems, 28, 2015.
429
[40] Wael Farag. Multi-agent reinforcement learning using the deep distributed distributional deterministic policy gradients algorithm. In 2020 International Conference Innovation and Intelligence for Informatics, Computing and Technologies (3ICT), pages 1 1-6. IEEE, 2020.
432
[41] Wei-Fang Sun, Cheng- Kuang Lee, and Chun- Yi Lee. Dfac framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In International Confer- ence on Machine Learning. pages 9945-9995 PMLR, 2021.
435
[42] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint 17X.2:100..05296
438
439
2017.
[43] Alexander Peysakhovich and Adam Lerer. Consequentiaalist conditional cooperation in social dilemmas with imperfect information. arXiv preprint arXiv: 2017.
[44] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, pages 9228.9244. PMLR, 2020.
444
[45] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment, arXiv preprint 2oXiv:0077.04309 2020.
447
[46] Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Twenty-Fourth AAAI Conference Artificial Intelligence, 2010.
450
[47] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.08531, 2020.
453
[48] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
12

---

458
[49] Zhikun Wang, Katharina Mülling, Mare Peter Deisenroth, Heni Ben Amor, David Vogt, Bern- hard Schölkopf, and Jan Peters. Probabilistic movement modeling for intention inference in human-robo interaction. The International Journal of Robotics Research, 32(7):841-858, 2013.
457
458
459
460
[50]
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pages 4257-4266. PMLR, 2018.
461
462
[51] Dylan P Losey, Mengxi Li, Jeannette Bohg, and Dorsa Sadigh. Learning from my partner's actions: Roles in decentralized robot teams. In Conference on robot learning, pages 752-765. PMLR, 2020.
484
485
[52]
Chongjie Zhang and Victor Lesser. Multi-agent learning with policy prediction. In Twenty-fourth AAAI conference on artificial intelligence, 2010.
467
468
[53] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100, 1953.
470
[54] Eric Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI, volume pages 709-715, 2004.
471
472
[55]
Julia L Wirch and Mary R Hardy. Distortion risk measures: Coherence and stochastic dominance. In International congress on insurance: Mathematics and economics, pages 15-17, 2001.
473
(56)
Alejandro Balbás, José Garrido, and Silvia Mayoral. Properties of distortion risk measures. Methodology and Computing in Applied Probability, 11(3):355-39,, 2009.
475
[57] Borislav Mavrin, Hengshuai Yao. Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference machine learning, pages 4424 4434. PMLR, 2019.
478
479
[58] Peter Huber. Robust statistics. In International encyclopedia of statistical science, pages 1248 1251. Springer, 2011.
481
[59]
Peter Rousseeuw, Frank R Hampel, Elvezio M Ronchetti, and Werner A Stahel. Robust statistics: the approach based on influence functions. John Wiley Sons, 2011.
482
483
[60] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives,
484
154) 2001.
485
[61] Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
486
Checklist
487
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The contribution of this paper can be found both in abstract and introduction.
492
(b) Did you describe the limitations of your work? [Yes] The limitations of our work can be found in S
484
(c) Did you discuss any potential negative societal impacts of your work? [No] We will discuss the potential negative societal impacts of our work in the camera-ready version.
495
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
13

---

499
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our work does not include theoretical results.
500
(b) Did you include complete proofs of all theoretical results? [N/A] Our work does include theoretical results.
503
3. If you ran experiments..
504
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Our implemen- tation details can be found in Appendix B and we have open source our codes and models which can be found by URL B.1
505
508
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The training details can be found in Sec S and Appendix B
509
510
(c) Did you report enror bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We have visualized all of our experiments with error bars.
511
512
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] The total amount of compute and the type of resources used can be found in Sec
515
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes] Codes of all baseline methods used in this work are open-sourced and we have cited the creators in Sec | and Sec
519
520
(b) Did you mention the license of the assets? [Yes] The license of the assets are MIT licenses and the mention can be found in Appendix B
522
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
523
We have open source our codes and models which can be found by a URL
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We do not use existing data and the codes used by us are open open-sourced and follow the MIT license.
528
527
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We do not existing data and the codes used by are open-sourced and follow the MIT license.
528
5. If you used crowdsoureing conducted research with human subjects...
531
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Crowdsourcing conducted research with human subjects used in our work.
532
534
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Crowdsourcing or conducted research with human subjects are not used in our work.
535
536
537
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Crowdsourcing or conducted research with human subjects are not used in our work.
538
539
14


Reviews:
[{"review_id": "M6PCWk4DHe6", "reviewer": "Reviewer_metE", "summary": "This paper presents a generalizable and sample-efficient algorithm for multi-agent coordination in decentralized general-sum games without any access to other agents’ rewards or observations.  It first learns the distributions over the return of individuals and estimates a dynamic risk-seeking bonus to encourage agents to discover risky coordination strategies. Then it proposes an auxiliary opponent modeling task so that agents can infer their opponents’ type and dynamically alter corresponding strategies during execution. Empirical studies show that the trained agents outperform other baseline methods via achieving mutual coordination during training and avoiding being exploited by non-cooperative opponents during execution.", "questions": "After reading the paper, I have some questions:\n\n1. Sec. 4.2 is not clear. A figure showing the architecture is needed.\n\n2. Why left truncated variance is used here? Why did the index start from M/2? \n\n3. Why c_tj is used here?  Why combing Eqn. 4 and Eqn. 5 in Eqn. 6 is reasonable? It lacks the motivation\n\n4. LIAM is actually a single-agent RL method, how did you conduct the experiments in MARL scenarios?\n\n5. Did you compare your method with LOLA [3]?\n\n[3] Learning with Opponent-Learning Awareness\n", "limitations": "Please see the above comments.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Risk-sensitive learning in zero-sum games is important topic in multi-agent systems. This paper applies risk-sensitive learning in zero-sum games. It uses the well-received distributional RL to learn return distributions and uses distribution distortion. The opponent learning was used to avoid overfitting opponents’ coordination strategies during training.\n\nWeaknesses: \n1. This paper combines many well-studies methods and applies them in zero-sum game scenario. The methodological contributions are limited.\n\n2. The evaluation scenarios are mainly on 2-agent cases, which is simple for didactic studies. However, it is hard to see if the proposed method can also perform well on complex 3+ agent scenarios. For example, social dilemma and zero-sum games scenarios in melting pot [1].\n\n3. The writing is not good. Sec. 4.2 is not clear. It is not easy to follow. A figure showing the architecture is needed.\n\n4. The Monster-hunt and Escalation scenarios were modified in [2], which is not cited in this paper.\n\n[1] Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot\n\n[2] DISCOVERING DIVERSE MULTI-AGENT STRATEGIC BEHAVIOR VIA REWARD RANDOMIZATION\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FgEud9uUcYI", "reviewer": "Reviewer_DtFh", "summary": "The paper introduces GRSP (Generalizable Risk-Sensitive MARL algorithm), a method for training agents in mixed-games to converge to cooperative nash-equilibrium. The method's impact is in its applicability to much more relaxed situations than other MARL algorithms (usually Centralised Training Decentralised Execution). The key insights for this method are two fold: \n\n1) cooperative equilibrium are usually “high-risk” so simple reward maximisation rarely converges to this.\n\n2) instead of requiring full access to co-players during training, simply being provided their action is suffice for opponent modelling. This is closer to independent learning than existing methods.\n \nThe method is then applied in two evaluation protocols: Evaluation Of Returns during training (can they converge to the desired equilibrium) and Generalisation to Novel Co-players at test-time. In both situations compelling arguments are made to show GRSP efficacy. Finally ablations are applied to distinguish how important the “high-risk” vs “opponent-modelling” insights and associated components are.\n", "questions": "\nI’d like more understanding of the training dynamic:\n\n\n1. How does using the distribution RL effect sample efficiency?\n2. How does risk-sensitive approaches effect exploration? \n\n", "limitations": "None to mention :) \n", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n1.  Clean presentation\n2.  Good experiment protocol\n3. Very good breath of environments\n4. Solid theoretical justification\n5. Original work.\n6. Clearly explains intuitions for method.\n\n\nWeaknesses;\n1) Very little time is given to explain training models, i’m unsure how sample efficient this method is compared to others. \n2) Generalisation Study Results could be better displayed. The opponent shaping literature has much nicer payoff graphs for showing how much an agent has cooperated with another -> these could help explain your findings.\n\n3) I have little understanding of the distortion methods discussed would be useful. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D5HCzOrkU0x", "reviewer": "Reviewer_DkS2", "summary": "This paper suggests learning risk-seeking policies in general-sum games in order to promote cooperation. The authors seek the strategy using distributional RL and risk-seeking weighting. Empirical evaluation on a range of domains shows that this algorithm finds the cooperative outcomes where other multi-agent RL methods do not, even though some of them focus on opponent modeling and coordination.", "questions": "Why did you not compare to the existing algorithms for learning cooperative outcomes in general sum games?\n\nWhat are the main reasons why the proposed algorithm finds the cooperation and the existing ones fain in some of the specific domains?", "limitations": "There is not obvious potential negative societal impact. The limitations are IMO not explored sufficiently well. It is not clear when the proposed algorithm starts failing.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Positives*\n \nI like the idea of using risk-seeking behavior for creating coordination and I like the use of distributional RL for implementing it. I also like the empirical evaluation, which uses a sufficiently wide range of domains and clearly shows the superiority of the proposed approach over the chosen baselines.\n\n*Major concerns*\n\nMy main concerns are lack of theoretical grounding and depth of understanding conveyed by the experimental results.\n\nLearning in games is a relatively old and well studied field with many solution concepts and algorithms to achieve them. This paper presents an intuitive solution to promote cooperation, but does not analyze at all what the proposed algorithm is supposed to converge to theoretically. Are there any guarantees on the behavior of the algorithm even in simple matrix games? Is there some relation between the parameters for risk seeking and the gap between cooperative and non-cooperative equilibria of the game? Does the algorithm converge to correlated equilibria as many other learning dynamics under some conditions or even that is not guaranteed?\n\nI believe that theoretical understanding of algorithms is important, but not absolutely necessary for a good paper, if the empirical evaluation chooses the right baselines and provides an in-depth analysis of why and when the proposed algorithm is superior. Since there are many papers dealing with trying to learn cooperative outcomes in general sum games (such as [A,B]) I consider the choice of baselines insufficient. Furthermore, the experiments do not provide any insights into when and why the proposed method performs better. The ablation study is definitely a step in the right direction, but I still do not know when to expect the algorithm to work well and when not.\n\n[A] Littman ML. Friend-or-foe Q-learning in general-sum games. InICML 2001 Jun 28 (Vol. 1, pp. 322-328).\n[B] Crandall JW, Goodrich MA. Learning to compete, compromise, and cooperate in repeated general-sum games. InProceedings of the 22nd international conference on Machine learning 2005 Aug 7 (pp. 161-168).\n\nTo summarize, the overall idea of the paper is interesting to me, but the thoroughness of setting in the related work and the depth of the analysis is slightly below the bar I would expect from a NeurIPS publication.\n\n*Minor suggestions*\n\nLines 31-32 claim that training a population of strategy profiles is infeasible in real world settings, which does not seem to be true with algorithms, such as AplhaStar.\n\nIt is not clear what exactly are the error areas in the graphs. Confidence intervals? Variance? STD?\n\nSome formulations are hard to follow, such as L196-197, L236, ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8hlgyAgOZfX", "reviewer": "Reviewer_ZHUo", "summary": "This paper considers the problem of coordination in the more realistic general-sum game settings, by applying techniques in distributional reinforcement learning which allows the agents to leverage risk measures to guide the exploration toward risky cooperative strategies. Besides, an auxiliary opponent modeling task is used to improve the generalizability in face of non-cooperative players.\nThe method is compared with previous works in four tasks, and the empirical results demonstrate that the proposed method could improve the coordination ability of agents even without access to other agents' rewards, and quickly adapt to non-cooperative agents by fine-tuning the feature extractor.", "questions": "1. Since long-term fine tuning is needed at test phase to achieve generalization, why keep $\\psi^i_a$ fixed and not tuning anymore?\n2. In line 285, you claim that the cooperative agents and non-cooperative agents are trained by GRSP and MADDPG, respectively. Can you explain why these two different algorithms can produce these two desired strategies?\n3. The proposed method seems to work well on the 5 by 5 grid-world environments, which left me wondering how well it performs on other popular Markov game environments. For example, social dilemma extends the prisoner's dilemma into multi-step settings, and the results on it can be more convincing.\n4. Can you explain why your method can converge to the highest global reward in IPD. This is very unintuitive because it is not an equilibrium, implying that self-interest agents would deviate from it and try to defect at last. Your method does not take the rewards of other agents into consideration and the learned policy is self-interest even though you have incorporated risk-seeking bonus.", "limitations": "1. Experiments in this work consists of \"Comparison of Returns\" only. More comprehensive demonstration experiments can better explain why the method works.\n2. Generalization of the method used in this paper requires long-term fine tuning, while a lot of works can tackle ad-hoc teamwork efficiently right now. The application value of this work is limited.\n3. During test time, the adaptation needs access to the full action history of all other agents, and fine-tune the feature extractor on it. This can be a very restrictive assumption and limits its use in practice.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper takes a look into a valuable point. Coordination is often studied in the fully cooperative setting, but general-sum game is more realistic. This paper considers an important but less studied problem. Also, generalization is a major but often overlooked issue in current MARL methods.\n2. The proposed method applies the risk-seeking bonus in distributional RL to solve the exploration problem in multi-agent tasks, which is well-motivated and has the potential to improve the coordination ability.\n3. The method used in this article combines risk-sensitive learning and agent modeling, both of which are promising techniques to handle the non-stationarity caused by environmental and agent level changes in multi-agent settings.\n\nWeaknesses:\n\n1. The most important part method is too short compared with the preliminaries and experiments. It is unclear to me how the feature extractor $\\mathcal{E}$ and the decision maker $D$ work, more details are needed to introduce how your framework functions as a whole.\n2. 1. There have been some works considering risk-sensitive or distributional Q-functions in MARL, such as [DFAC](https://arxiv.org/abs/2102.07936) and [D4PG](https://ieeexplore.ieee.org/document/9311945). This part of the method in this paper seems to be an implementation of risk-sensitive Q-function upon independent Q-learning (IQL), thus having rather restricted innovations.\n3. The risk-seeking bonus results in optimistic behavior during training, this is beneficial when training with cooperative agents, but on the other side, which can also be exploited by non-cooperative agents and thus fail to learn a generalizable cooperative policy.\n4. The paper claims to have generalization ability across different opponents during execution, but the way of achieving this requires long-term fine tuning during test phase (even just a part of the network will be tuned). I think such level of generalization should be expressed as \"policy transferring\" for clarity in order not to confuse others to think of zero-shot coordination (ZSC) or ad-hoc teamwork (AHT).\n\nMinors:\n1. The equation 7 makes it hard to understand what is the output of the function $D$. Besides, the parameters to be optimized on the LHS are $\\phi^i$ and $\\psi^i_s$, but $\\phi^i$ disappears on the RHS.\n2. The loss term $\\mathcal{J}$ in equation 8 is not defined. I know that you have introduced it in the preliminaries, but please define it again under your framework according to your parameterization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject