Two Heads are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning
Authors: Jiahui Li
Kun Kuang
Baoxiang Wang
Xingchen Li
Fei Wu
Jun Xiao
Long Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-agent reinforcement learning
influence-based exploration

Abstract: Exploration strategy plays an important role in reinforcement learning, especially in sparse-reward tasks. In cooperative multi-agent reinforcement learning~(MARL), designing a suitable exploration strategy is much more challenging due to the large state space and the complex interaction among agents. Currently, mainstream exploration methods in MARL either contribute to exploring the unfamiliar states which are large and sparse, or measuring the interaction among agents with high computational costs. We found an interesting phenomenon that different kinds of exploration plays a different role in different MARL scenarios, and choosing a suitable one is often more effective than designing an exquisite algorithm. In this paper, we propose a exploration method that incorporate the \underline{C}uri\underline{O}sity-based and \underline{IN}fluence-based exploration~(COIN) which is simple but effective in various situations. First, COIN measures the influence of each agent on the other agents based on mutual information theory and designs it as intrinsic rewards which are applied to each individual value function. Moreover, COIN computes the curiosity-based intrinsic rewards via prediction errors which are added to the extrinsic reward. For integrating the two kinds of intrinsic rewards, COIN utilizes a novel framework in which they complement each other and lead to a sufficient and effective exploration on cooperative MARL tasks. We perform extensive experiments on different challenging benchmarks, and results across different scenarios show the superiority of our method.

Two Heads are Better Than One:
A Simple Exploration Framework for Efficient Multi-A geet Reinforcement Learning
Jiahui Li', Kun Kuang'; Baoxiang Wang?.,, Xingchen Li", Fei Wu', Jun Xiao', Long Chen
ZZejiaag University, "The Chinese University of Hong Kong, Shenzhen Shenzhen Institute of Artificial Intelligence and Robotics for Society, "HKUST
{jiahuil, kunkuang) @zju. edu. cn, bxiangvang@cuhk, edu. cn, xingchenl@zju edu. en, {wufei, juffii,jux))0er zju. edu. cn, longchen@ust. .k
Abstract
Exploration strategy plays an important role in reinforcement learning, especially in sparse-reward tasks. In cooperative multi-agent reinforcement learning (MARL), designing suitable exploration strategy is much more challenging due to the large state space and the complex interaction among agents. Currently, mainstream exploration methods in MARL either contribute to exploring the unfamiliar states which are large and sparse, or measuring the interaction among agents with high computational costs. We found interesting phenomenon that different kinds exploration plays a different role in different MARL scenarios, and choosing suitable one is often more effective than designing exquisite algorithm. In this paper, we propose exploration method that incorporate the CuriOsity- -based and INfluence-based exploration (COIN) which is simple but effective in various situations. First, COIN measures the influence of each agent on the other agents based on mutual information theory and designs as intrinsic rewards which are applied to each individual value function. Moreover, COIN computes the curiosity-based intrinsic rewards via prediction errors which are added the extrinsic reward. For integrating the two kinds of intrinsic rewards, COIN utilizes novel framework which they complement each other and lead sufficient and effective exploration on cooperative MARL tasks. We perform extensive experiments on different challenging benchmarks, and results across different scenarios show the superiority of our method.
Introduction
Multi-Agent Reinforcement Learning (MARL) has been widely used to solve numerous complex real-world problems, such traffic control [38], autonomous robotics control [17, 26], game AI [25], and network routing [40], etc.. Compared with single-agent reinforcement learning, the joint state and action space grows exponentially with the number of agents. To overcome this challenge, the Centralized Training with Decentralized Execution (CTDE) [20, 12] paradigm is proposed, where the joint value function is estimated in the central critic during training, and the local agents execute actions according to their own policies.
Based the CTDE paradigm, numerous deep MARL methods have been proposed [27, 32, 35, 15). The most commonly used method for exploration e-greedy. However, these methods may perform poorly in complex scenarios [13, 29] with large number of agents and a vast action space. The main issue is that the e-greedy exploration strategy requires significant number of attempts to become
Coorespongding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

familiar with the environment, potentially leading to sub-optimal results. Therefore, an effective exploration strategy is crucial.
Currently, exploration strategies in MARL can be broadly classified into two main categories. The first category includes curiosity-based exploration methods [41, 6]. These methods focus quantifying the unfamiliarity of incoming states and encourage agents to explore a wide range of states that they have not or rarely experienced. The second category comprises influence-based exploration methods [9. 33, 39]. These methods aim to promote agents to adopt actions that can influence other agents and achieve better coordination. However, mainstream methods often prioritize the design of sophisticated algorithms within one specific category, which may not perform well in certain scenarios. On the one hand, the joint state space can be excessively large, presenting a challenge for suriosity-based methods to enable agents to experience all unseen states effectively. On the other hand, meaningful interactions among agents can be infrequent, making it difficult for influence-basee methods to identify and optimize these interactions. Fortunately, we have discovered that these two types of methods can complement each other, and even a simple combination of them often leads to satisfactory results.
In this paper, we propose an effective and compute-efficient exploration method named COIN, which and INfluence- baaeed exploration. To be specific, we define the influence
incorporates CuriOsity- of one agent on another as the mutual information between its actions and the trajectory of its peer in the next time-step. Our derivation allows us to reformulate the problem of maximizing mutual information into an optimization problem with a tractable lower bound by introducing a variational posterior estimator. This formulation enables us to quantify the impact of each agent's 's action on the others and use the influence degree as an intrinsic reward to promote exploration. The influence-based intrinsic reward is added to the individual value functions before being sent to the central mixer. Furthermore, this influence-based exploration strategy is compatible with curiosity-based methods, which encourage the agents to explore unfamiliar states. To reduce the exploration state space, we propose an additional method that effectively utilizes the prediction errors of local observations of each agent and the global states to measure the model's uncertainty towards the environment. This uncertainty can be used as an intrinsic reward, which can be added to the extrinsic global reward provided by the environment. Moreover, based on these two types of intrinsic rewards, we have designed a Multi-Agent Reinforcement Learning (MARL) framework where they complement each other, resulting in thorough and efficient exploration without incurring significant computational costs.
Our main contribution can be summarized as follows: (i) We propose straightforward exploration method based on mutual information that enables the quantification of each agent's 's influence on the others. (ii) We introduce a simple yet effective approach to measure the model's unfamiliarity with future states, resulting in curiosity-driven exploration. (iii) To achieve effective exploration in MARL settings, we design framework in which the two kinds of methods complement each other and act as intrinsic rewards which are compute-efficien (iv) The experiments on three benchmarks StarCraft II [29], MACO [34], and Google Football [13] show the superiority and effectiveness of our COIN.
2 Related Work
Curiosity-based Exploration Methods. Exploration algorithms focus encouraging agents to visit a wide range of different states within the environment to avoid getting trapped in sub-optimal results. Among the various branches of exploration methods, curiosity- based approaches [22, 28] have gained significant popularity. One common approach involves designing intrinsic rewards that measure the degree of unfamiliarity of states. Some researchers utilize pseudo-state counts [2, 21, 30] to calculate such rewards, while others [24, 4]. develop models to predict target states and employ the prediction errors as rewards. For instance, Bai et al. [1] propose the utilization of multiple models and compute intrinsic rewards based on the standard deviation of predictions, especially in complex environments. Zheng et al. [41] extends the application of curiosity- based exploration to MARL settings and employs the prediction errors of individual Q-functions as intrinsic rewards. Similarly, Yang et al. [39] designs intrinsic rewards by considering the variance of an ensemble predicting model that incorporates both local observations and global states.
Influence-based Methods in MARL. The influence-based exploration strategy specifically de- signed for MARL problems, where the focus is on the complex coordination and interactions among

---

agents, In this context, mutual information serves an effective and commonly used tool. Jaques al. [9] introduces the concept of "social influence", represented as (M affls)). quantify how an agent's action can influence the action of another agent. This measure captures the interplay and dependencies between agents, providing valuable insights into their collaborative behavior. Simi- larly, Wang et al. [33] draws inspiration from counterfactual thinking and proposes two exploration techniques: EITI and EDTI. Both of them aim to encourage agents to influence each other's 's actions. In EITI, mutual information MI(o)+ : of. a' a²) is also utilized as a formulation to incentivize agents to exert influence on one another, fostering cooperative exploration and coordination.
These influence-based exploration strategies leverage mutual information the interdependencies among agents and promote collaborative exploration in MARL.settingg In this paper, we also propose an influence-based intrinsic reward MI(T, af) based on mutual information which is simple but efficient. The discussions of the superiority of our method compared with the others are left in Appendix.
capture
The two exploration strategies mentioned above focus on different aspects, with many studies emphasizing one while neglecting the other. One hybrid method called CIExplore [39] incorporates both strategies by designing them as intrinsic rewards, which is the closest approach to our work. However, CIExplore has shown to be unstable and ineffective in complex MARL scenarios.
Hybrid Exploration Methods in MARL.
In approach, COIN, not only ensure compute efficiency in the designed framework for calculating intrinsic rewards, but we also leverage the complementary nature the two kinds of rewards achieve sufficient and effective exploration, ultimately enhancing the model capabilities in complex MARL scenarios.
Preliminaries
Dec-POMDPs. The exploration method we discussed in this paper focuses on the settings of decentralized partially observable Markov decision process (Dec-POMIDP) [19, 3, 5, 23], which can always be described as tuple:
G G
where N represents the set of agents with W N. At each time-step, each agent i W selects and executes an action 0; E A to compose a joint action a E AAN. The environment produces the global state S according the transition function P(s'|s, a) S x A S. Meanwhile, the environment provides global reward shared by all agents according to the function r(s, a): : S N + IR. Moreover, each agent gains the partial observation E O according the observation function (?s,;) i) and learns its local policy x (a.(T)) T x A + [0, 1] conditions on the its local trajectory 7, T. The goal of all agents is maximize the discounted cumulative return E where is discount factor.
Method
In this section, we introduce two simple yet effective intrinsic rewards for influence and curiosity- based exploration. Following that, we present framework, as depicted in Fig. 1, that integrates these two strategies to facilitate exploration. Finally, we provide a brief overview of the loss function and offer implementation details, ensuring both effectiveness and computational efficiency in the exploration process.
4.1 Influence-based Intrinsic Reward
Compared with single-agent reinforcement learning, complex coordination plays important role MARL. As result, it essential to incentivize agents to explore states or execute actions that can influence each other. Mutual information [9, 14, 33 a commonly used tool for quantifying the influence between pairwise agents. Following previous works, we define the influence of the i-th agent on the other agents at time-step é as the sum of the mutual information between the trajectory

---

Dp
Mixing Netwark
Influence Estimator
SUM
Curiosity Estimator
NM
Vilar
My
IE
Of
IE
2
Q
QN
Awy
+++
of
1+
M 1G
fo
Agentí
Agent
a,a
AG dly
(0 !
(of.
(0f. at
Figure I: The training framework of COIN, Two additional estimators are set up for computing the intrinsic rewards, where the local observation-action history of all agents are the inputs.
of the other agents and the action taken by the i-th agent:
p(T E+1 lab) Dlo
Σ MI f+1, aI(+ = Σ (oft)) - H(T F+
-
la
6+1
E
Σ
p(Tk*) (+1)
KEWW
KEN
REN
[+1 where 7
represents the trajectory of the k-th agent time-step i 1, and HT)) represents the entropy term.
If we want to promote the influence among agents, we need to maximize ExE MI(T f+1 :a)). Meanwhile, we can represent y(Tk E+1) and p(rk (af) the following equations:
p(TE-1) = p(o)) II IPaaths)p 1
(1)
p p(olla)) Il
plo Irk ( o aj
ah, at).
(2)
t=0
Therefore, the mutual information can be rewritten as:
p(ogla!)
plaf I af) log
t
plo 1T at)
Σ MI(+E+),0 ;af) E( KEA
Σ flog
Σ log
H..
kENVi p(o2)
p(a +
plot it 1 ag))
We that Eq. (3) is divided into three terms. Term ! is determined by the environment, and hence can be ignored. Term 2 computes the information difference of the k-th agent's 's action selection when the action at time-step of the i-th agent is given. Notice the fact that the action of the i-th agent at particular time can not influence any actions, observations, or trajectories of the other agents the previous time. Thus, the term 2 can also be ignored. Term 3 measures the changes in
the target observation of the k-th agent. This term can be simplified log p(of*"/nf (n
p(o (r(
Therefore, we conclude that optimizing the mutual information Eq. (1) equivalent optimize:
plob 1 af, at)
E
log
plo 1+1
kEN

---

According to the methods of variational inference [31, 14], we can optimize a tractable evidence lower bound of the above expectation by introducing variational posterior estimator. Hence, have the following equation:
pl 1T al. a.) 1+1
Bllog 20 ok 1 a/p)
Ellog
pl
where de represents the variational posterior estimator parameterized by Φ, and we omit the summa- tion notation for convenience.
For encouraging the interaction among agents and promoting exploration, we design an influence- based intrinsic reward for each agent. The intrinsic reward for agent at time-step i is computed as:
Tin
Σ
[3, log go ( ape at) - log p( + 1 d4)],
(5)
where Bg is scaling factor to guarantee rf > 0.
4.2 Curiosity-based Intrinsic Reward
Curiosity-based exploration is widely adopted approach in single-agent reinforcement learning, with the objective of encouraging agents to explore unfamiliar states. Typically, this is achieved by computing intrinsic rewards based on prediction errors [24, 4] or variances [1] of future states, rewards, value functions. In the context of MARL, curiosity- -aased exploration remains essential. However, the prediction of joint states becomes unreliable due the exponential increase in state space with the number of agents, and the scarcity of meaningful states. Fortunately, the local observations of each agent can serve measure of curiosity. For simplicity, we use the prediction error of both the global state and the local observations to calculate the curiosity-basee intrinsic reward in the partially observable environment as:
N Σ 1-1
- db1)² + (6 fal"", a²))²
(6)
where for (T a²) and fa(T", agent and global state at time-step i + and a', T represent the joint action, and trajectories of all agents at time-step T.
(.r',a)) a²) represent the predicting network to estimate the observation of each
4.3 Implementation of the Estimators
Our goal is to develop two effective estimators for computing intrinsic rewards well as make them compute--fficient For the influence estimator, we model (o (o | af. at) by a variational posterior estimator, which is updated via loss:
L Σ Σ
(ox i+1 REM
- I af,
Meanwhile, the p(o Th at) is approximated by a prediction network Po updated via loss:
In = Σ Σ
(oft1 - po(o (011 a
(8)
KENY
For the curiosity estimator with parameters yp, we update it via loss:
Tup
1
(o i+1 (T ( a²)) + (6 -,(T", a²)3
99)
M
5

---

Notice that the summation symbol can become an obstacle when computing intrinsic rewards and losses efficiently. In order to accelerate the training procedure, we have implemented the two estimators in a compute-effiient manner, as illustrated in Fig. 2.
N
9, P4
h2
In the influence estimator, Eq. (7) requires us to propa- gate to for N - -1 iterations in order to obtain the intrin- sic reward for single agent. To improve efficiency, we have designed a batch implementation for ExEAT al- lowing us to estimate the observations of all other agents simultaneously in single forward propagation. Likewise, p(-IT,, ax) also adopts the same trick.
Ay. 41
:
hx
dx
Influence Estimator
Curiosity Estimator
Figure 2: Implementations of the two
estimators for computing the summation symbol in single forward propagation.
As for the curiosity estimator, all for have the same inputs. To streamline this process, we implement them using a single neural network that outputs a concatenated tensor.
To further alleviate the computational burden, we adopt the parameter-sharing technique, and all agents share the same network in the influence estimator. Similarly, for curiosity-basee intrinsic reward, f. and fo share the same encoder to get the rep- resentations. Inspired by the approach presented in [15], represent the trajectory of the agent using the hidden state of a recurrent neural network. Consequently, we have hf = T and h r². Mcanwhile, for the learning more stable, we compute the intrinsic rewards in Eq. (5) and Eq. (6) via target networks &' ny and sf which are copied from & J7 and yp periodically.
4.4 Incorporation of the Two Heads
We would like to emphasize the effectiveness of combining these two methods in a straightforward manner, rather than designing a intricate algorithm that solely focuses on one aspect. There are two reasons for this: (i) Curiosity- intrinsic rewards designed to encourage exploration of unfamiliar states, while influence-
intrinsic rewards aim to promote coordination among agents. These rewards operate on different dimensions and play distinct roles in various scenarios. (ii) Either of the methods may not work in certain training time-steps, as no method can guarantee constant positive influence among agents, and the meaningful unfamiliar joint state is too sparse. Some recent works [33, 39] attempt to use both kinds of methods, however, concerns regarding their effectiveness and computational efficiency remain which lead the unsatisfactory results in complicated scenarios.
In this subsection, we will introduce how to incorporate them under the paradigm of CTDE. For simplicity, we omit the superscript of time-step in this subsection. For the influence-based intrinsic rewards, we apply it to the individual local value functions of each agent to promote the interaction. For i-th agent it is computed as:
=
Qi
Binf Pinfet
(10)
where Q; is the local value function, and Biny is the scale factor. Q. will then be sent to the central critic Mixer get the joint value function:
Qint = Misser(Qi,...,..)
(1))
Notice that the intrinsic rewards are incorporated before the value functions are sent the central mixer. This is done to ensure that the intrinsic rewards can be scaled by the network appropriately. Meanwhile, it is worth mentioning that the intrinsic rewards are not applied to the target network. The reasons for this decision will be discussed in the Appendix.
The curiosity- based intrinsic rewards directly added to the extrinsic rewards, and subsequently, the agents and the central mixer are updated using TD-loss:
Lg (リ) Qiot), r + Beur Teur + * D'ot::
(12)
(13)
where Viot represents the target network, Bour represents the scale factor.
We show the algorithm of our COIN in Alogrithm 1.

---

Algorithm Inftuence and Curiosity-basee Exploration
Initialize: Local agents and central critic with its target networks 0', max episode length T, networks of the estimator for computing intrinsic rewards ø, 1 and with its target networks ø', and epl.
1: for each training episode do
2: while global state * terminal and time step & T do
3:
f= for each agent do
4:
5:
Compute the local value function Qi and get the hidden state h
6:
Select action af and execute
7:
end for
8:
Execute the joint action (at; "i: 1--9 a, Get reward r2+1 and next state 8c+-1
9:
10:
end while
11:
Add episode to replay buffer
Collate episodes in the buffer into single batch
13:
for t 1 to T do
Compute the influence- based intrinsic reward via Eq. (5)
15:
Compute the Curiosity-basee intrinsic reward via Eq. (6)
Add the influence- based to individual value functions via Eq. (10)
17:
Compute the joint value function via Eq. (5)
Compute the targets y via Fq. (13)
19:
Update A by minimizing the TD-loss in Eq. (12)
Update o, 7) and yo by minimizing the loss Eq. (7).Eq. (8) and Eq. (9)
21:
Update 0' 0, o' Φ. r/ = m, sp' yp periodically
22:
end for
23:
end for
Experiments
Our experiments aim to answer the following questions: (1) Q1: the exploration strategy important in MARL? (2) Q2: How does each kind of exploration strategy affect the model performance in different scenarios?
(3) Q3: Can influence-based exploration cooperate with auriosity-based
exploration?
We performed experiments on three popular MARL benchmarks: StarCraft II micro management challenge (SMAC)[29), MACO[36], and Google Research Football (GRF) [13], with different scenarios. The details of each benchmark will be introduced in the Appendix. We adopted two popular baselines, QMIX [27] and QPLEX [32], and applied our exploration method to them. To further demonstrate the effectiveness our exploration method, we chose the curiosity- exploration method EMC [41], the influence-hased exploration method EITT [33], and the hybrid method ClExplore [39] as strong baselines. should be mentioned that ETTI and CIExplore are implemented based the actor-critic framework in the original paper. For fair comparison, implemented EITI and CIExplore based on QMIX under the same framework as us. Meanwhile, since we aim to verify the effectiveness of different exploration strategies, we do not use the special episode buffer in EMC, as well as the other tricks in all methods. All the curves in the figures of this section are the mean of 10 random seeds, which have been smoothed, and the shaded areas represent the standard deviation. The experiment settings are also provided the Appendix.
5.1 Performance on several benchmarks (Q1)
Performance on StarCraft II benchmark. We first conduct experiments on the most popular environment, StarCraft II micro management challenge. StarCraft II is real-time strategy game. We set the difficulty of game at the "very difficult" level. This benchmark consists of various maps with different levels of challenges, "casy", hhard", and "super hard". We seleet three representative
7

---

Tixt Win Buso
0.0
0.6
0.0
Tee
044
0.4
0.4
0.2
0.2
0.2
2
2
Tanestups (mi))
Timessgs (mill)
Timestags (mil)
(a) v65s_vs_3s6z
(b) 6h_ vs_8z
(c) Corridor
Figure 3: Performance of our exploration method on three scenarios in StarCraft II benchmark.
60
Text Transmttel
Tex Win Rate
60
0.8
0.6
Teee
0.4
0.4
20
0.5
0.2
0.4
0.8
0.2
0.4
0.6
0.8
Timesteps (nil)
Timesaps (mil)
(mil))
(a) Aloha
(b) Gather
(c) Hallway
Figure 4: Performance of our exploration method on three scenarios in MACO benchmark.
and super hard scenarios, 3s5z_vs vy wy_3s6z, corridor, and 6h_vs vz_8z to carry out experiments², and the results are depicted in Fig. 3. Our method shows a significant improvement over the baselines in all maps, highlighting the importance of the exploration strategy in MARL. Furthermore. the superior results compared to other exploration baselines demonstrate the effectiveness of our COIN method. We also conducted experiments with sparse reward setting, and the results are provided in the Appendix.
Performance on MACO benchmark. Next, we evaluated our method in the MACO benchmark, which features relatively sparse rewards. MACO incorporates several coordination tasks from various multi-agent learning literature and enhances their difficulty. We conducted experiments on three MACO scenarios: Aloha, Halbway, and Gather. These scenarios require different levels of agent coordination. The results of our experiments are presented in Fig. 4.
In the Aloha scenario [8, 18], the agents strive to achieve as many successful transmissions as possible within limited number of time steps. None of the other exploration methods proved effective in this scenario. Our COIN approach improved the two baselines by nearly 20 mean transmissions. The results of all methods were peculiar, with an initial burst followed by a drop and then plateaued poor results. Initially, the agents attempted exploration, resulting in more successful transmissions. However, the punishment mechanism in this setting was overly severe, causing the agents to adopt conservative strategies. In the Gather scenario, the task is relatively simple, with agents aiming to choose the same action to achieve common goals set at the beginning of each episode. Although exploration did not significantly improve the performance of the basclines, our COIN approach still achieved a faster convergence speed. There was a failure case known as the "noisy TV" in the Gather scenario. Choosing the goal set at the beginning of the episode resulted in higher global reward, while not choosing this goal led to a relatively lower reward. It evident that QPLEX failed
We do not carry out the experiment of EMC on corridor because it takes too much computational resources which is out of the max memory of our GPU (GTX 2080 Ti).
8

---

and produced suboptimal results, and our framework did not provide significant improvement. The Hallway scenario [34] involved agents divided into several groups, where agents within the same group aimed to reach the same state simultaneously, while different groups aimed to reach different states. Our COIN approach succeeded with QPLEX, resulting in a 60% improvement mean win rates. However, applying any exploration method to QMIX showed little improvement, which can be considered another failure case.
Performance on GRF benchmark. In the GRF game, players need to coordinate with each other to organize offense and score goals. The rewards in GRF are sparse, with agents only receiving rewards at the end of the game. Therefore, exploration plays vital role in guiding the agents to discover winning strate- gies.
UMIX+Durr
QMIX
0.8
OPLEX-Ours
OPLEX
0.6
EMC
We compared the performance of several baselines on the GRF benchmark, as shown in Fig. 5. The curve representing our COIN approach demonstrates the ap- plication of our exploration framework on QMIX. Our approach outperforms in the scenario "academy 3 vs 1 with keeper". QMIX and QPLEX fail to perform well due to the sparse rewards. EMC applies a curiosity- based exploration strategy to QPLEX but results in a decrease in model performance. In contrast, our COIN approach combines influence-based exploration and curiosity-based based exploration, outperforming all base- lines with an improvement of nearly 0.2 mean scores per episode.
CIExplore
Test
0.4
0.2
1
2
Timesteps (mil)
Figure Experiment results on GRF bench- mark in academy 3 vs I with keeper.
5.2 Ablations and Interesting Findings (Q2, Q3)
qpt
Tea Vin Bate
The Win Bate
0.
0.6
0.6
0.4
0.4
0.9
0.2
0.2
0.2
2
2
Timesteps (mil)
Timesoep (mil)
Timestepe (mi))
(a) academy 3 vs with keeper
(b) 3s52_ vs 3562
(c) Corridor
Figure 6: Ablation study of our COIN on GRF and SMAC benchmark.
To understand how influence-based exploration and curiosity- -base exploration work in different scenarios, we conducted ablation studies. We selected three representative scenarios to showcase these results, as depicted in Fig. 6. The dotted line represents applying only the infuencc--bsed intrinsic reward on QMIX, while the dashed line represents applying only the curiosity- intrinsic reward. Interestingly, we found that different exploration strategies play different roles different scenarios. When only influence- based or curiosity-ba based exploration was conducted in the academy 3 vs / with keeper scenario, the model did not benefit from the exploration strategy. Instead, the performance decreased by nearly 0.5 mean scores. To understand this finding, we consulted literature and watched replays, and find that this phenomenon arises from the parameter sharing technique. According to Li et al. [14], shared parameters among agents can lead to similar behaviors, hindering the model's 's ability to learn successful policies in challenging tasks. Our influence-based intrinsic reward promotes agents to influence each other, bringing diversity to their behaviors. However, focusing solely on influence

---

among agents neglects crucial patterns, such as cutting to attract the enemy defender, resulting in poor outcomes. Fortunately. our proposed method succeeded by striking balance between influence and curiosity. This interesting result also explains why EMC decreases the baseline performance in this scenario, as shown in Fig. 5. In the scenario 3s5z_vs_3aal s__066,, influence-based exploration plays a significant role, while curiosity-basee winning strategy involves one the allied units kiting several enemies to the corner of the map and sacrificing itself, thus not requiring much exposure unknown states. In the corridor scenario, both exploration strategies improve the baseline performance significantly. However, curiosity-bases exploration achieves even greater improvement and reaches the best performance.
exploration harms the performance. This is because the learned
In all scenarios, the best performance is achieved when both kinds of exploration strategies are incorporated. Therefore, we conclude that different scenarios require different exploration strategies, and when we are unsure which one will be effective, incorporating both is often a good choice. For more experimental results, please refer to the Appendix.
5.3 The Chosen hyperparameters
In Section 4, we introduced several hyperparameters for computing intrinsic rewards. To ensure stability and guarantee that the Satuence-baseee intrinsic reward is larger than zero, we set Bg in Eq. (5) to 0.5 for all scenarios in all benchmarks. Additionally, Binf and Beur are scaling factors used to adapt the two types of intrinsic rewards to the environment. Given that the importance of the two types of exploration may vary at different time steps, determining which one will dominate is challenging. To address this, we combine the two types of intrinsic rewards in a straightforward manner. When designing the hyperparameters, we follow two principles: (i) ensuring that both types of intrinsic rewards are on the same order of magnitude, and (ii) scaling the intrinsic rewards to be approximately 100 to 1000 times lower than the maximum extrinsic reward at the beginning of the training stage. Details of all the hyperparameters are listed in Appendix.
6 Conclusion
In this paper. we explore an interesting phenomenon where different types of exploration play varying roles in different scenarios of MARL. However, most current studies tend to focus on one type of exploration and overlook the other. In light of this, we propose a framework called COIN that effectively combines and complements these two types of exploration strategies. Firstly, COIN measures the influence of each agent on the others using mutual information theory, and assigns it as intrinsic rewards to individual value functions. Additionally, COIN calculates curiosity-based -bsee intrinsic rewards by utilizing prediction errors, which are then added to the extrinsic reward. By integrating these two types of intrinsic rewards, COIN creates novel framework where they complement each other, ultimately leading to more efficient and effective exploration Both of these types of intrinsic rewards are computationally efficient within our designed framework, thus enabling effective exploration. Our experiment results on various benchmarks demonstrate the superiority of our proposed method.
cooperative MARL tasks.
Limitations and future work. Though the incorporation of two kinds of exploration is useful, our method relies on scaling hyperparameters and cannot automatically weigh the two kinds of strategies. In future work, we will investigate how to discover the causal relations among agents. With these potential interdependencies, we can assign appropriate weights each agent and avoid computing the MI for unnecessary agent pairs.
Acknowledgements
This work was supported by the National Key Research & Development Project of China (2021ZD01 10700), the National Natural Science Foundation of China (62006207. 62037001, U19B2043, 61976185, 1866, the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-JU--SIAS-010)) and the Fundamental Research Funds for the Central Univer- sities (226-2023-00048 by The National Natural Science Foundation of China (62106213 and 721 50002) and Shenzhen Science and Technology Program (RCBS2021060) 104356063 and JCYJ202 10220000022000110000 1032).
226-2022-00051, 226-2022-00142) Baoxiang Wang is partially supported
10

---

References
[1] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivee offline reinforcement learning. In International Conference on Learning Representations, 2021.
[2] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27()): 119-840,20 2002.
[4] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018.
[5] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multia- gent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-177 2008.
[6] Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning indi- vidual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[7] Jayesh K Gupta, Maxim Egoroy, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66 83, 2017.
[8] Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI, volume 4, pages 709-715, 2004.
[9] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning. pages 3040-3049. PMLR, 2019.
[10] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi-agent rein- forcement learning with subgoals generated from experience replay buffer. In International Conference Machine Learning, pages 10041-10052. PMLR, 2022.
[11] Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pages
PMLR, 2021.
[12] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82 94, 2016.
[13] Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajkac, Olivier Bachem, Lasse Espcholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501 -510- 2020.
[14] Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Infor- mation Processing Systems, 34:3991-40033 2021.
[15] Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Changjie Fan, Fei Wu, and Jun Xiao. Deconfounded value decomposition for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12843-12856. PMLR, 2022.
[16] Pengyi Li, Hongyao Tang, Tianpei Yang, Xiaotian Hao, Tong Sang, Yan Zheng, Jianye Hao, Matthew E Taylor, Wenyuan Tao, and Zhen Wang. Pmic: Improving multi-agent reinforcement learning with progressive mutual information collaboration. In International Conference on Machine Learning, pages 12979-12997. PMLR, 2022.
11

---

[17] Timothy P Lillierap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 2016.
[18] Frans Oliehoek. Value-based planning for teams of agents in stochastic partially observable envirouments. Amsterdam University Press, 2010.
[19] Frans A Olichock and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[20] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
[21] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
[22] Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, page 6, 2009.
[23] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 443-451, 2018
[24] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by supervsue prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[25] Peng Peng, Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent indirectionallyccoorinate nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv: 1103.0006 10069, 2017.
[26] Sarvapali D Ramchurn, Alessandro Farinelli, Kathryn S Macarthur, and Nicholas R Jennings. Decentralized coordination in robocup rescue. The Computer Journal, 1467-1414 2010.
[27] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304, 2018.
[28] Tabish Rashid, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. International Conference on Learning Representations, 2019.
[29] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. 2019.
[30] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[31] Martin J Wainwright, Michael Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-305, 2008
[32] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent geleanning In International Conference on Learning Representationss 2021.
[33] Tonghan Wang, Jianhao Wang. Yi Wu, and Chongjic Zhang. Influence-based multi-agent exploration. In International Conference Learning Representations, 2019.
[34] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly de- composable value functions via communication minimization. In International Conference on Learning Representations, 2020.
12

---

[35] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. 2021.
[36] Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. arXiv preprint arXiv:2106.02886 2021.
[37] Ermo Wei and Sean Luke. Lenient learning in independdentlearmer stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914 29--2955,, 2016.
[38] Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. Hierarchically and cooperatively learning traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 669-677, 2021.
[39] Huanhuan Yang. Dianxi Shi, Chenran Zhao. Guojun Xie, and Shaowu Yang. Ciexplore: Curiosity and influence-based exploration in multi-agent cooperative scenarios with sparse rewards. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2321 2330, 2021.
[40] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. sensors, 15(5):10026-1044,, 2015.
[41] Lulu Zheng, Jiarui Chen, Jianhao Wang. Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang, Episodic multi-agent reinforcement learning with curiosity- driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
13

---

A Appendix
A.1 Details of the benchmarks
Multi-agent coordination challenge benchmark. Wang et al. [36] collect six coordination tasks from various multi-agent learning literature to create the Multi-Agent Coordination Challenge (MACO) benchmark. For our experiments, we have selected three tasks from this benchmark.
The Aloha coordination task [8, 18] consists of 2 5 array with 10 agents. The objective messages and maximize the backlog, which has a maximum value of 5. However, if two adjacent agents send messages simultaneously. a collision occurs, and the messages need to be resent. For each successful transmission, positive global reward of 0.1 is gained, while a collision results in a negative global reward of -10. Each unit initially has backlog, and at each time step, there is probability of 0.6 for a new packet arrive, provided that the maximum backlog has not been reached yet. Gather [37], there are 5 agents whose objective to collectively choose the same action in order to achieve common goals. These goals are established at the beginning of each episode. If all agents successfully choose the goal, the entire team receives a reward of 10. On the other hand, none of the agents chooses the goal, reward of 5 is given to the team. Hallway [34] involves 12 agents organized into multiple groups. Each agent randomly assigned to a state within each chain. In this task, if agents belonging the same group reach the same state at the same time, they receive a global reward. However, if more than one group reaches the same state, a global punishment is given.
send
StarCraft II micro management benchmark. The StarCraft II micromanagement challenge [29] involves agents controlling allied armies and collaborating engage with enemy forces controlled by the game engine. This benchmark includes different maps that pose varying levels of challenges, categorized as "casy", "hard" and "super hard". For our experiments, we have chosen three represen- tative super hard scenarios: 3s5z vs ___z,, corridor, and 6h vs 1r. The game AI difficulty is set to "very difficult" level. Table provides an introduction to these selected maps.
Table I: Experiment maps of StarCraft II micro management benchmark.
Map Name
Ally Units
Eenemy Units
Agent Type
Difficulty
3s52_ vs v8_3s6z
3 Stalkers, 5 Zealots
3 Stalkers, 6 Zealots
Heterogeneous
Super hard
6h_vs_8z
Hydraliks
Zealots
Homogeneous
Super hard
corridor
Zealots
24 Zerglings
Homogeneous
Super hard
Google research football benchmark. In the Google Research Football (GRF) game [13], players required to coordinate with each other organize offensive plays and score goals. In this game, agents control the players on the left side, excluding the goalkeeper, while the players on the right side are controlled by rule-based bots using hand-coded built-in AL. Each player can observe the positions and movement directions of all players, as well as the football. They have a discrete action space that includes options for moving, sliding, shooting, and passing. Soccer players must determine the optimal timing for shooting by combining dribbling and passing techniques. In this game environment, rewards are only provided at the end of the game, with reward of 100 for scoring a goal and a penalty of 11 for losing the game.
A.2 Experiment settings
We list the experiment settings and hyperparameters in Table The two important scale factors Bour and Biny in MACO depends on the scenarios. We set Bour = Bmn = 0.05 in aloha, Beur = Binf = 0.01 in gather, and Bour = Benf = 0.0005 in hallway.
A.3 Comparison with the other exploration methods
In this paper, we employ both maaeencc-based and curiosity- exploration strategies. In compar- ison to other methods, we utilize a simple yet effective approach to implement these two types of exploration. In this subsection, we will discuss the differences among these methods.
14

---

Table 2: Hyperparameters and experiment settings
Settings
StarCraft
MACO
GRF
Batch size
32
32
Training episodes
Smil
1mil(2mil for aloha)
Smil
Agent optimiser
RMSProp
RMSProp
Adam
Estimator optimiser
Adam
Adam
Adam
Agent learning rate
0,0005
0.0005
0,0005
Estimator learning rate
0,0005
0.0005
0.0005
Target central critic update interval
200 episodes
200 episodes
200 episodes
Bour
0.0005
0.05/0.01/0.0005
0.005
B
0.0005
0.05/0.01/0.0005
0.005
8
0.5
0.5
0.5
Mutual information and influence-based exploration. Mutual information is commonly utilized concept in existing works for multi-agent reinforcement learning (MARL) problems. MOA [9] was the first propose using MI(a) af|s') as intrinsic rewards. Another formu- lation called EITI [33] is given by MI(o,-1;of of of oj. a/)) Similarly, PMIC [16] proposes (a;a;s) as a collaboration criterion. Additionally, Jiang and Lu propose M1(1;0) O), while Li al. [14] propose MI(T; I), where represents the agent's ID. The main differences among these methods lie in their intentions. The first three methods aim to promote influence and collaboration among agents, while the last two aim enhance diversity among the agents. In this paper, we also incorporate mutual information (T a)) to quantify the influence of one agent's action on another agent. While the basic motivation aligns with MOA and EITI, our proposed approach is (a) more reasonable and (b) easier compute. The reasons for these distinctions are as follows: (a) More reasonable: Intuitively, an agent can only influence the future behavior of other agents. Therefore, we consider MI(a;; dsss')) as less reasonable. Additionally, the crucial factor that influences other agents is the action taken, rather than the current states or observations. Consequently, MI ++1, af) is more reasonable than M ((o,tt,, of at ol. of). Specifically, in the computation of the i-th influence on j.
Figure 7: Markov Decision Process in MARL.
(I(T) aa) = log plaft
whereas MI(ott1, ;o;, oflo), = log plo,+ 1o.. p(o) (03.a))
Refer-
p(r, T)
encing the Markov decision process in Fig. 7, in p(of+1 (o), at, at). of becomes the noise factor in predicting of which can potentially hinder accurate influence estimation in certain scenarios. For instance, let the optimal policy <aab lo;, st) be and operation, and let the observation function g(o)]-) be an dlement-wise zor operation for all inputs. of should be 0 when of = 1 and s' 0. but introducing of into g would result in of 1. (b) Easier compute: We have designed framework where the intrinsic reward at each time step can be computed in a single forward propaga-
tion. This approach is straightforward to implement and does not impose a significant computational burden.
Prediction network and curiosity-based exploration. Curiosity-based exploration methods, widely employed in reinforcement learning [22, 28], typically involve designing intrinsic rewards to measure the degree of unfamiliarity associated with states. A common approach entails setting up one or several networks to predict information about the next time-step, with the instability of these predictions serving as the intrinsic reward. For instance, Pathak et al. [24] and Burda et al.
[4] consider the prediction error of target states as intrinsic rewards. Bai al. [1] use the standard
deviation of target Q Q-function predictions as the intrinsic reward. Zheng et al. [41] utilize the pre- diction errors of individual Q-functions as intrinsic rewards multi-agent reinforcement learning (MARL) problems. Similarly, Yang et al. [39] design intrinsic rewards based on the variance of an ensemble predictive model incorporating local observations and global states. our approach, we adopt similar strategy to the aforementioned methods. We use the prediction errors of next
15

---

states and observations as intrinsic rewards. Since the observations and global states are provided by the environment, we consider them to be more reliable supervision information compared to value functions. Moreover, these prediction errors can capture the unfamiliarity from two perspectives: local observations for individual unfamiliarity and global states for joint unfamiliarity. Consequently, this approach is straightforward yet effective in MARL settings.
A.4 Two heads are better than one
One key highlight of this paper is the effectiveness of incorporating both methods rather than solely focusing on a single aspect through an elegant algorithm. To support our viewpoint, we evaluate different exploration strategies on the baselines [27, 32] and present the results on the MACO benchmark in Fig. 8 amd Fig 9. In the figure, the dashed line represents solely applying the curiosity- based intrinsic reward, while the dotted line represents solely applying the influence-based intrinsic reward. It is worth noting that either exploration strategy may fail in certain scenarios. However, by incorporating both methods, we consistently achieve the best results. We also perform experiments on the StarCraft II benchmark with sparse reward setting, and the results can is show in Fig. 10, in which MASER [10] is goal based exploration method.
OMRAGTUMIHCC QMA46TTM
QUAEE
OM
Tea Whe 1lan
0.2
Timseapp (e())
Theaarpp (ni))
(a) Aloha
(b) Gather
(c) Hallway
Figure 8: Performance of different kinds of exploration strategies on MACO benchmark.
OPLEXiESiiur
OPLESHIT
OPLEX
OPLDWAOU
00
0.0
M
85
35
02
0.
06
0.2
Thmaman tech)
Themaaga (xil)
(a) Aloha
(b) Gather
(c) Hallway
Figure 9: Ablations of our method when applying COIN to QPLEX.
OUS
QMX
MASER
2.3
033
2.4
0.5
1.4
2.2
0.2
5.6
4.5
Tras
Thansage (440
Figure 10: Results on StarCraft II sparse setting with map 3m(left) and 2s3z(right), where the rewards only come from death, killed or terminated.
16


Reviews:
[{"review_id": "DUICJTF8jT", "reviewer": "Reviewer_PR9o", "summary": "This paper looks at two different exploration strategies within Multiagent Reinforcement Learning. The two exploration strategies are curiosity-based and influenced based, where the latter is unique to MARL and takes into account the mutual information of one agents actions on the trajectory of another agent. The study then looks at combining these two forms of exploration and studies the effects of the individual intrinsic rewards as well as their combined effects on three distinct environments. A number of ablations are performed during these studies.", "questions": "I believe that the weaknesses highlight the questions that I have, in particular surrounding the hyperparameters and a more detailed investigation of the statistics.", "limitations": "While limitations around the hyperparameters have been noted, these do not give sufficient response to the points raised above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper introduces a novel algorithm which is able to explore in a MARL setting using two different additional intrinsic reward signals. It seems that such an exploration strategy is effective in the examples given at improving performance. The algorithm is relatively clean and the explanation is clear and precise. Exploration strategies are clearly a very important topic within the realm of computationally costly settings and so such an improvement is clearly important.", "weaknesses": "The language in the paper needs some significant work. I would recommend passing the paper through an LLM for checking for spelling and grammar throughout, as there are a fair number of such issues.\n\nThe lack of clear exploration in the hyperparameter space is a certainly an important limitation of this work. While the choice of hyperparameters is motivated, it is not clear that this has actually been studied, and in particular where the mixture of two intrinsic rewards is being used together, it seems that such an investigation would be very important to have a deeper understanding of their effects. Although an ablation study has been performed, this is really a binary investigation, whereas the continuum of mixings between the rewards would be most interesting.\n\nAlthough there are a fair number of plots showing the learning trajectory on the test score, no summary statistics have been included. It would be useful to have a more detailed, quantitative set of measures of improvement, in particular over the hyperparameter sweeps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The lack of clear exploration in the hyperparameter space. The continuum of mixings between the rewards would be most interesting. Quantitative set of measures of the hyperparameter sweeps.\n\n**A1:** The importance of two kinds of exploration is various in different time steps. Thus, it is very hard to decide which kind of strategy will dominate.\nIn this work, We follow two principles to design the hyper-parameters: 1). Both two kinds of intrinsic rewards are on the same order of magnitude. 2). The two scaled intrinsic rewards are about 100 to 1000 times lower than the maximum extrinsic reward at the beginning of the training stage.\nWe choose different $\\beta_{inf}$ and $\\beta_{cur}$ for different scenarios, which are shown in Appendix.\n\nFurthermore, it is worth noting that different scenarios require different exploration strategies, when it is hard to decide which one to use, implementing both in a simple and efficient way be a good choice.\n\n**Q2:** Checking for spelling and grammar throughout.\n\n**A2:** Thanks for your careful reviews and suggestions again. We will carefully proofread our paper and revise all typos."}, {"review_id": "fJMiwLPBKF", "reviewer": "Reviewer_a6y3", "summary": "In this paper the authors propose a new framework to improve exploration in MARL. The method COIN improves exploration by combining the concepts in the literature of curiosity and influence.", "questions": "1. In line 46 the authors state that their method is compute-efficient. Is there any evidence or proof for this?\n2. in equation 7, does it mean that the only difference to predict state and observations is that to predict observations the agents use only the trajectories from a single time step? If so, is this accurate if we dont consider the previous values, since MARL is a non stationary sequential problem?\n3. I have some questions about the influence of agents on others: does influence-based exploration mean that the agents are going to explore states where each agent has more influence on the actions of the others? If this is the case, how do they measure this influence? From my understanding, this is measured by summing the mutual information between all trajectories and the action of each agent individually to see the influence of each agent on the others; but does this guarantees that influence on the others is always good influence? i.e., can there be cases where we have bad influence and so these states shouldn't be explored?\n4. In line 156: \"Curiosity-based intrinsic rewards aim to courage the unfamiliar state, and influence-based intrinsic rewards aim to promote coordination.\" While curiosity based methods clearly directly tackle exploration problems, can we say the same from influence-based methods? Following the same logic, we could also say that almost any method in MARL is a method that tackles specifically exploration problems since most of them promote coordination.\n5. the authors start the paper by mentioning that exploration is important in sparse reward environments; in this sense, have they tried to run their method on starcraft with sparse rewards too? Why choosing only normal stracraft environments in this case?\n6. in section 5.3 it is unclear what the authors end up picking for $\\beta_{inf}$ and $\\beta_{cur}$: do they get the same weight, i.e., the same value?\n7. there are also other methods that work towards efficient exploration, such as []. These methods, for example, use sub-goals instead of curiosity, which is also a common benchmark for some exploration-based methods. Have the authors considered comparing against some of these too?\n\n[1] https://arxiv.org/abs/2206.10607", "limitations": "Please see above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a method that combines in a framework two popular concepts in the reinforcement learning literature, curiosity and influence, aiming to improve exploration. The authors analyse how the different components of their method affects their approach. A detailed description of how the method works is also provided and it is then tested in a set of different environments.", "weaknesses": "- \"Li et al. [12] point that shared parameters among agents induce similar behaviors of players which makes the model fail to learn successful policies on this challenging task. Our influence-based intrinsic reward promotes the agents to affect each other and hence brings diversity.\" - this is a problem caused by the famous parameter sharing paradigm, that the authors also adopt in COIN; i feel that this is a big claim that I believes it needs further evidence.\n\nMinor:\n- line 156: \"courage\"->\"encourage\"\n- line 96-97: \"it shows unstable and innefective\" poorly phrased\n- l 144: \"varriance\" -> variance\n- l 154: shouldnt it also be $\\tau^~t$?\n- line 283: plays -> play\n- description of the dec pomdp in section 3 needs to be reviewed: for example, the observation model is missing and $\\mathcal{O}$ represents the observation function; $\\mathcal{N}$ inside the tuple G should be the number of agents and not the set; among others\n\nPlease find my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** ''Our influence-based intrinsic reward promotes the agents to affect each other and hence brings diversity.\" needs further evidence.\n\n**A1:** The parameter-sharing paradigm leads to the same behavior of different agents in Google Research Football(GRF). After watching the replay, we find that for the models without adding the influence-based intrinsic reward, different agents tend to perform the same action (same as Li et al.[1]). Meanwhile, influence-based rewards encourage each agent to influence the others' actions in the future, and hence can bring diversity.\n\n**Q2:** Is there any evidence or proof for compute-efficient?\n\n**A2:** COIN is compute-efficient for two main reasons:\n(1) First, we adopt simple formulations of both MI for influence-based intrinsic rewards and prediction errors for curiosity-based intrinsic rewards.\n(2) Both the MI and prediction errors can be estimated by neural networks, and the summation symbol in Eq.(5) and Eq.(6) can be computed in a single forward propagation.\nWe will provide more discussions and emphasize them in the revision.\n\n**Q3:** Is the influence are measured by summing the mutual information?  Does this  that influence on the others is always good influence?\n\n**A3:** Yes, we define the influence of agent A on agent B as the MI, which means how agent A's current action will influence the future trajectory of agent B. And the summation of all the MI of agent A to its peers becomes the influence-based intrinsic reward of agent A.\n\nOur method can not guarantee the influence is always useful. Indeed, no method can guarantee the influence among agents is always good. \nRecent literature deems that promoting the interaction is often more useful to find meaningful states than exploring blindly.\nMeanwhile, what we want to highlight is that ''two heads are better than one'' which encourages the readers to try different kinds of strategies simultaneously rather than focusing on one single strategy.\n\n**Q4:** Curiosity based methods clearly directly tackle exploration problems. Following the same logic, we could also say that almost any method in MARL is a method that tackles specifically exploration problems since most of them promote coordination.\n\n**A4:** Curiosity-based methods encourage the agents to experience the unseen joint states. \nHowever, in the MARL setting, meaningful states are too sparse and hard to find.\nTherefore, how to find potentially meaningful states within acceptable explore time steps is a problem.\nDifferent kinds of exploration strategies follow different assumptions, and the influence-based methods assume that the states where the agents interact with each other are more likely to be meaningful.\nWe find that the MARL scenarios are often complicated and only relying on one kind of strategy may not get the desired effect.\n\n**Q5:** Have tried to run on starcraft with sparse rewards ? Have the authors considered comparing against some methods with subgoals[1]?\n\nWe perform experiments on sparse rewards, and compare our method with MASER[1], which is a goal-oriented method.\nThe results are left in Fig. (3) of the affiliated PDF .\nFig. (3) depicts that our method outperform MASER in 3m and 2s3z.\nOur results are slightly inconsistent with those reported in [1]. The reasons may come from the game version, random seed, et al.\nAnd we will perform more experiments to justity the superiority of our method.\n\n[1] https://arxiv.org/abs/2206.10607\n\n**Q6:** It is unclear what the authors end up picking for the hyperparameters?\n\n**A6:**  We mainly follow two principles to design the hyper-parameters: 1) The two kinds of intrinsic rewards are on the same order of magnitude. 2) The two scaled intrinsic rewards are about 100 to 1000 times lower than the maximum extrinsic reward at the beginning of the training.\nWe choose different $\\beta_{inf}$ and $\\beta_{cur}$ for different scenarios which are shown in Appendix."}, {"review_id": "Zvf2xrnEkB", "reviewer": "Reviewer_dhms", "summary": "This paper investigates exploration methods in multi-agent reinforcement learning (MARL). Specifically, among the two main families of exploration methods, namely curiosity-based methods and influence-based methods, the paper reports that they could serve complementary roles to each other. Subsequently, the paper proposes a method that takes advantage of both types of exploration methods, demonstrating its effectiveness in 3 common MARL benchmarks. ", "questions": "1.\tThe results of QMIX + ours in Aloha seem to be quite odd given the early burst and drop with the plateaued bad results. Any intuition or explanation for the phenomenon?\n2.\tCan you elaborate more on the possible reasons behind the performance discrepancies between the results of QMIX and QPLEX when adding your method?\n3.\tDo you have the results for QPLEX+ours in GRF? This could strengthen the results \n4.\tGiven the drastic difference between QMIX and QPLEX when combined with the proposed exploration method, do the ablation results apply to QPLEX too\n5.\tCan you elaborate on how causal relationships could help in balancing the two terms?\n6.\tMore of an open question, Could this have any failure cases specific to the multi-agent setting like the noisy TV issues in the single-agent case?\n7.\tWould these results apply in fully decentralized MARL methods without sharing parameters?\n", "limitations": "One limitation is discussed. No negative societal impact was discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tMARL exploration is an open and important problem for the field. The proposal of combining the two families of exploration methods in MARL is novel and the highlight of their complementary role is significant. \n2.\tThe experimental results overall show the effectiveness of the proposed method against strong baselines\n3.\tThe paper is well-written with the method well-explained.\n", "weaknesses": "1.\tThere are some inconsistencies in results across base algorithms of the proposed methods in different environments. Some additional ablation results could improve this (will be mentioned in the question section)\n2.\tThe mentioned inconsistencies are not well explained, making the results less convincing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The results of QMIX + ours in Aloha seem to be quite odd given the early burst and drop with the plateaued bad results. Any intuition or explanation for the phenomenon?\n\n**A1:** In Aloha, messages collided when two adjacent agents send messages simultaneously.\nEach successful transmission will gain a positive global reward (i.e., 0.1) and the collision will gain a negative global reward (i.e., -10). \nMeanwhile, the agents can not communicate with each other and do not know whether the action will cause a collision.\nIn the beginning, they try to explore and hence get more successful transmissions. However, they gain more punishment at the same time.\nThe severe punishment mechanism makes the agents adopt conservative strategies. \nThat is the reason why the curve looks like this.\n\n**Q2:**  Can you elaborate more on the possible reasons behind the performance discrepancies between the results of QMIX and QPLEX when adding your method?\n\n**A2:** In most scenarios, COIN can improve the performance of both QMIX and QPLEX. \nThe discrepancies between the results may come from environmental uncertainty and the method itself.\nThe MARL scenarios are much more complicated than single-agent RL, thus, it is hard to say which method will definitely outperform the other.\nEven the same method with different random seeds can get different performance with a huge gap.\nHence, it is a normal phenomenon where discrepancies exist, but it is hard for us to provide specific reasons for them.\n\n*Q3:** Do you have the results for QPLEX+ours in GRF? Do the ablation results apply to QPLEX too?\n\n*A3:** Our method improves QPLEX in GRF scenario, and the results are shown in Fig. (2) of the the affiliated file.\nWe show the ablations of our method when apply on QPLEX in the affiliated file Fig. (1).\nOur method can improve QPLEX in different content in different scenarios except for a failure case ''Gather''.\n\n*Q4:** Can you elaborate on how causal relationships could help in balancing the two terms?\n*A4:** Given the causal relationships, we can know the potential interdependence among agents.\nWe can assign lower $\\beta_{inf}$ to the agents that do not rely on the others and higher $\\beta_{inf}$ to the agents that most agents rely on. Meanwhile, we can simplify the computation of Eq.(3) to avoid computing the unnecessary MI.\n\n**Q5:** Could this have any failure cases specific to the multi-agent setting like the noisy TV issues in the single-agent case?\n\n**A5:** There is a failure case like ``noisy TV'' in the Gather scenario.\n\nA higher global reward of 10 is received if all agents choose the goal set at the beginning of the episode,\nand a relatively lower reward of 5 is received if no agents choose this goal.\n\nObviously, QPLEX fails and falls into suboptimal results and our framework can not bring too much improvement.\n\n**Q6:** Would these results apply in fully decentralized MARL methods without sharing parameters?\n\nThe experiments can also be applied in methods without sharing parameters, and the results could be even better.\nIn some scenarios, the sharing parameters paradigm brings challenges and is harmful to MARL training. For example, in GRF, the sharing parameter paradigm leads to the same behavior of all agents at the beginning of the training procedure and hampers the possibility of the agents' cooperation. \n"}, {"review_id": "fGlwHOZeh7", "reviewer": "Reviewer_DahA", "summary": "This paper propose a novel and compute-efficient exploration method: COIN, which can incorporate Curiosity-based and Influence-based exploration. For influence-based exploration，COIN quantify how each agent’s action affect the other agents and use the influence degree as the intrinsic reward to promote the exploration. For curiosity-based exploration，COIN uses the prediction errors of local observations of each agent and the global states to measure the uncertainty of the model to the environment. The experiments on three benchmarks StarCraft II, MACO, and Google Football show the superiority and effectiveness of COIN. ", "questions": "(1) In Equ.(1) and (2), the trajectory τkt should include state and action. So what is the meaning of p({a_k^t|\\tau}_k^t\\ )?\n\n(2) Influence-based intrinsic reward is higher. This indicates that the action is more encouraging. Thus, In Equ.(8), why is“-”instead of“+”？\n\n(3) Comparing with Fig.2, In Equ.(7), why is the input different between them ? \n\n(4) The description of hyperparameters in the experiment can not be found.\n\n(5) In Fig.5(a), why is the performance of Qmix better than Qmix+Inf and Qmix+Cur?\n\n(6) Is the exploration framework of this paper attached to other MARL algorithms? How well?\n", "limitations": "In section 6, the authors talk about some limitations about this paper. At present, the potential negative social impact of this work has not been identified.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) This work designs a MARL framework which combines two kinds of main exploration methods and achieve sufficient and efficient exploration without bringing many computational costs.\n\n(2) The exploration method proposed in this paper can be easily applied to other multi-agent methods.\n\n(3) The overall idea of the paper is clear and easy to understand.\n", "weaknesses": "(1) To be honest，it lacks a certain degree of novelty. As is well known, the methods used in this paper like Qmix, MI and Prediction Error are all previous work. We just do some combination work.\n\n(2) It should explain how to extend the method in this paper to other MARL methods.\n\n(3) The existing experimental results seem insufficient，mainly reflected in the following aspects:\n\nMany result curves do not show the final convergence state, but rather the intermediate state;\n\nLack of quantitative forms of expression, such as reward values or case study.\n\n(4) The main body of this paper is clear to understand, here is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** It lacks a certain degree of novelty, and just do some combination of previous work such as MI and prediction error.\n\n**A1:** Off course, MI and prediction error are commonly used in RL.\n**In this paper, our main motivation is to highlight that different kinds of exploration play a different role in different scenarios. And we provide a simple but efficient way to combine two kinds of exploration strategies.**\nCompared with the related works, our proposed MI is (a) more reasonable and (b) easier to compute.\nMore details are left in Appendix.\n\n**Q2:** Is the exploration framework of this paper attached to other MARL algorithms? How well?\n**A2:** Our framework is suitable for most of the MARL algorithms. Our method does not need to modify the baseline model, but only set up additional modules for computing the intrinsic rewards. These modules can be trained simultaneously with the original model.\n\n**Q3:** Many result curves do not show the final convergence state.  Lack of quantitative forms of expression, such as reward values or case study.\n\n**A3:** All randoms seeds of all baselines converge at the end in this paper. \n(1) Different random seeds may induce different outcomes with a large gap. (2) We add the smooth weight while drawing the curve.\nThese are probably the reasons that it ``looks not to converge''.\n\n**Q4:** what is the meaning of $p(a_k^t|{\\tau}_k^t)$?\n\n**A4:** ${\\tau}_k^t$ is defined as $({o}_k^0, {a}_k^0,...,{o}_k^t)$ in this paper (see Preliminaries) which do not contains the actions in the current time-step. Hence, $p(a_k^t|{\\tau}_k^t)$ means the probability to take action $a_k^t$ when given the current observations and history observation-action pairs.\n\n**Q5:** In Equ.(8), why is“-”instead of“+”?\n\n**A5:** We subtract (“-”) the influence-based intrinsic reward from $Q_i$ and get a pessimistic \\~$Q_i$ to encourage the agents to influence each other during training. \nConsidering the simple case of $y=r+\\gamma*Q'-(Q-r_{inf})$.\nAbstract $r_{inf}$ from $Q$ is equivalent to add $r_{inf}$ to $r$.\nWe apply the influence-based intrinsic rewards before each $Q_{i}$ is sent to the central critic because it can be scaled along with the ${Q}_{i}$.\n\n** Q6:**  In Equ.(7), why is the input different between them?\n\n**A6:** Thanks for pointing out that. The inputs are the same, and we will rectify these typos in the revision.\n\n**Q7:** In Fig.5(a), why is the performance of Qmix better than Qmix+Inf and Qmix+Cur?\n\n**A7:** Fig.5(a) is an interesting finding. After watching the replay, we can provide an intuitive explanation. In this GRF scenario, each agent is initialized at the same location on the map. Hence, they need to explore the whole map in acceptable training time steps. Meanwhile, influence-based exploration will not promote such behavior, but only focus on the interactions among agents. This leads to too many unnecessary passes and makes the performance even worse than $\\epsilon$-greedy.\nThe sharing parameter paradigm leads to the same behavior of agents at the beginning of the training procedure. However, curiosity-based exploration will not promote the agents to influence each other and can not bring diversity for agents, which makes QMIX+Cur get worse.\n\n"}]

Author rebuttal:
Thanks for your careful reviews and discerning comments. We will explain all the mentioned questions in the following, and some new experiments are added to the affiliated file.

Meta-review: Its insights are not too surprising, and the algorithmic ingredients are not novel, but the results are useful on their own terms, specifically in how they demonstrate the benefits of qualitatively different exploration types in different circumstances. Also, while “pure” method papers tend to be easier to review, I find this to be an empirically well-executed hybridisation work and hence give it the benefit of the doubt.
Decision: Accept (poster)