Title: Two Heads are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-agent reinforcement learning

influence-based exploration
Abstract: Exploration strategy plays an important role in reinforcement learning, especially in sparse-reward tasks. In cooperative multi-agent reinforcement learning~(MARL), designing a suitable exploration strategy is much more challenging due to the large state space and the complex interaction among agents. Currently, mainstream exploration methods in MARL either contribute to exploring the unfamiliar states which are large and sparse, or measuring the interaction among agents with high computational costs. We found an interesting phenomenon that different kinds of exploration plays a different role in different MARL scenarios, and choosing a suitable one is often more effective than designing an exquisite algorithm. In this paper, we propose a exploration method that incorporate the \underline{C}uri\underline{O}sity-based and \underline{IN}fluence-based exploration~(COIN) which is simple but effective in various situations. First, COIN measures the influence of each agent on the other agents based on mutual information theory and designs it as intrinsic rewards which are applied to each individual value function. Moreover, COIN computes the curiosity-based intrinsic rewards via prediction errors which are added to the extrinsic reward. For integrating the two kinds of intrinsic rewards, COIN utilizes a novel framework in which they complement each other and lead to a sufficient and effective exploration on cooperative MARL tasks. We perform extensive experiments on different challenging benchmarks, and results across different scenarios show the superiority of our method.

Two Heads are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning

Abstract

Exploration strategy plays an important role in reinforcement learning, especially in sparse-reward tasks. In cooperative multi-agent reinforcement learning (MARL), designing a suitable exploration strategy is much more challenging due to the large state space and the complex interaction among agents. Currently, mainstream exploration methods in MARL either contribute to exploring the unfamiliar states which are large and sparse, or measuring the interaction among agents with high computational costs. We found an interesting phenomenon that different kinds of exploration plays a different role in different MARL scenarios, and choosing a suitable one is often more effective than designing an exquisite algorithm. In this paper, we propose a exploration method that incorporate the CuriOsity-based and INfluence-based exploration (COIN) which is simple but effective in various situations. First, COIN measures the influence of each agent on the other agents based on mutual information theory and designs it as intrinsic rewards which are applied to each individual value function. Moreover, COIN computes the curiosity-based intrinsic rewards via prediction errors which are added to the extrinsic reward. For integrating the two kinds of intrinsic rewards, COIN utilizes a novel framework in which they complement each other and lead to a sufficient and effective exploration on cooperative MARL tasks. We perform extensive experiments on different challenging benchmarks, and results across different scenarios show the superiority of our method.

1 Introduction

Multi-Agent Reinforcement Learning (MARL) has been widely used to solve numerous complex real-world problems, such as traffic control [38], autonomous robotics control [17, 26], game AI [25], and network routing [40], etc .. Compared with single-agent reinforcement learning, the joint state and action space grows exponentially with the number of agents. To overcome this challenge, the Centralized Training with Decentralized Execution (CTDE) [20, 12] paradigm is proposed, where the joint value function is estimated in the central critic during training, and the local agents execute actions according to their own policies.

Based on the CTDE paradigm, numerous deep MARL methods have been proposed [27, 32, 35, 15]. The most commonly used method for exploration is ϵ -greedy. However, these methods may perform poorly in complex scenarios [13, 29] with a large number of agents and a vast action space. The main issue is that the ϵ -greedy exploration strategy requires a significant number of attempts to become

familiar with the environment, potentially leading to sub-optimal results. Therefore, an effective exploration strategy is crucial.

Currently, exploration strategies in MARL can be broadly classified into two main categories. The first category includes curiosity-based exploration methods [41, 6]. These methods focus on quantifying the unfamiliarity of incoming states and encourage agents to explore a wide range of states that they have not or rarely experienced. The second category comprises influence-based exploration methods [9, 33, 39]. These methods aim to promote agents to adopt actions that can influence other agents and achieve better coordination. However, mainstream methods often prioritize the design of sophisticated algorithms within one specific category, which may not perform well in certain scenarios. On the one hand, the joint state space can be excessively large, presenting a challenge for curiosity-based methods to enable agents to experience all unseen states effectively. On the other hand, meaningful interactions among agents can be infrequent, making it difficult for influence-based methods to identify and optimize these interactions. Fortunately, we have discovered that these two types of methods can complement each other, and even a simple combination of them often leads to satisfactory results.

In this paper, we propose an effective and compute-efficient exploration method named COIN, which incorporates CuriOsity-based and INfluence-based exploration. To be specific, we define the influence of one agent on another as the mutual information between its actions and the trajectory of its peer in the next time-step. Our derivation allows us to reformulate the problem of maximizing mutual information into an optimization problem with a tractable lower bound by introducing a variational posterior estimator. This formulation enables us to quantify the impact of each agent's action on the others and use the influence degree as an intrinsic reward to promote exploration. The influence-based intrinsic reward is added to the individual value functions before being sent to the central mixer. Furthermore, this influence-based exploration strategy is compatible with curiosity-based methods, which encourage the agents to explore unfamiliar states. To reduce the exploration state space, we propose an additional method that effectively utilizes the prediction errors of local observations of each agent and the global states to measure the model's uncertainty towards the environment. This uncertainty can be used as an intrinsic reward, which can be added to the extrinsic global reward provided by the environment. Moreover, based on these two types of intrinsic rewards, we have designed a Multi-Agent Reinforcement Learning (MARL) framework where they complement each other, resulting in thorough and efficient exploration without incurring significant computational costs.

Our main contribution can be summarized as follows: (i) We propose a straightforward exploration method based on mutual information that enables the quantification of each agent's influence on the others. (ii) We introduce a simple yet effective approach to measure the model's unfamiliarity with future states, resulting in curiosity-driven exploration. (iii) To achieve effective exploration in MARL settings, we design a framework in which the two kinds of methods complement each other and act as intrinsic rewards which are compute-efficient. (iv) The experiments on three benchmarks StarCraft II [29], MACO [34], and Google Football [13] show the superiority and effectiveness of our COIN.

2 Related Work

Curiosity-based Exploration Methods. Exploration algorithms focus on encouraging agents to visit a wide range of different states within the environment to avoid getting trapped in sub-optimal results. Among the various branches of exploration methods, curiosity-based approaches [22, 28] have gained significant popularity. One common approach involves designing intrinsic rewards that measure the degree of unfamiliarity of states. Some researchers utilize pseudo-state counts [2, 21, 30] to calculate such rewards, while others [24, 4]. develop models to predict target states and employ the prediction errors as rewards. For instance, Bai et al. [1] propose the utilization of multiple models and compute intrinsic rewards based on the standard deviation of predictions, especially in complex environments. Zheng et al. [41] extends the application of curiosity-based exploration to MARL settings and employs the prediction errors of individual Q-functions as intrinsic rewards. Similarly, Yang et al. [39] designs intrinsic rewards by considering the variance of an ensemble predicting model that incorporates both local observations and global states.

Influence-based Methods in MARL. The influence-based exploration strategy is specifically designed for MARL problems, where the focus is on the complex coordination and interactions among

agents. In this context, mutual information serves as an effective and commonly used tool. Jaques et al. [9] introduces the concept of 'social influence", represented as MI ( a t i ; a t j | s t ) , to quantify how an agent's action can influence the action of another agent. This measure captures the interplay and dependencies between agents, providing valuable insights into their collaborative behavior. Similarly,Wang et al. [33] draws inspiration from counterfactual thinking and proposes two exploration techniques: EITI and EDTI. Both of them aim to encourage agents to influence each other's actions. In EITI, mutual information MI ( o t +1 j ; o t i , a t i | o t j , a t j ) is also utilized as a formulation to incentivize agents to exert influence on one another, fostering cooperative exploration and coordination.

These influence-based exploration strategies leverage mutual information as a means to capture the interdependencies among agents and promote collaborative exploration in MARL settings. In this paper, we also propose an influence-based intrinsic reward MI ( τ t +1 j ; a t i ) based on mutual information which is simple but efficient. The discussions of the superiority of our method compared with the others are left in Appendix.

Hybrid Exploration Methods in MARL. The two exploration strategies mentioned above focus on different aspects, with many studies emphasizing one while neglecting the other. One hybrid method called CIExplore [39] incorporates both strategies by designing them as intrinsic rewards, which is the closest approach to our work. However, CIExplore has shown to be unstable and ineffective in complex MARL scenarios.

In our approach, COIN, we not only ensure compute efficiency in the designed framework for calculating intrinsic rewards, but we also leverage the complementary nature of the two kinds of rewards to achieve sufficient and effective exploration, ultimately enhancing the model capabilities in complex MARL scenarios.

3 Preliminaries

Dec-POMDPs. The exploration method we discussed in this paper focuses on the settings of decentralized partially observable Markov decision process (Dec-POMDP) [19, 3, 5, 7, 23], which can always be described as a tuple:

where N represents the set of agents with |N| = N . At each time-step, each agent i ∈ N selects and executes an action a i ∈ A to compose a joint action a ∈ A N . The environment produces the global state s ∈ S according to the transition function P ( s ′ | s, a ) : S × A N → S . Meanwhile, the environment provides a global reward shared by all agents according to the function r ( s, a ) : S × A N → R . Moreover, each agent gains the partial observation o ∈ O according to the observation function Ω( s, i ) and learns its local policy π i ( a i | τ i ) : T × A → [0 , 1] conditions on the its local trajectory τ i ∈ T . The goal of all agents is to maximize the discounted cumulative return ∑ ∞ i =0 γ i r i , where γ is a discount factor.

4 Method

In this section, we introduce two simple yet effective intrinsic rewards for influence and curiositybased exploration. Following that, we present a framework, as depicted in Fig. 1, that integrates these two strategies to facilitate exploration. Finally, we provide a brief overview of the loss function and offer implementation details, ensuring both effectiveness and computational efficiency in the exploration process.

4.1 Influence-based Intrinsic Reward

Compared with single-agent reinforcement learning, complex coordination plays an important role in MARL. As a result, it is essential to incentivize agents to explore states or execute actions that can influence each other. Mutual information [9, 14, 33]is a commonly used tool for quantifying the influence between pairwise agents. Following previous works, we define the influence of the i -th agent on the other agents at time-step ˜ t as the sum of the mutual information between the trajectory

…

…

�

𝑸𝑸

𝒕𝒕𝒕𝒕𝒕𝒕

Influence

Estimator

𝒓𝒓

𝒄𝒄𝒄𝒄𝒓𝒓

𝒕𝒕

𝓛𝓛

𝜽𝜽

𝓛𝓛

𝝋𝝋

𝓛𝓛

𝝓𝝓

…

…

𝐌𝐌𝐌𝐌𝐌𝐌

𝒇𝒇

𝒔𝒔

𝒉𝒉

,

𝒂𝒂

𝒔𝒔

𝒕𝒕+𝟏𝟏

𝒇𝒇

𝒕𝒕

𝐌𝐌𝐌𝐌𝐌𝐌

𝒕𝒕

𝟏𝟏

𝒕𝒕+𝟏𝟏

…

𝒇𝒇

𝒕𝒕

𝒉𝒉

𝑵𝑵

𝒕𝒕

,

𝒂𝒂

𝑵𝑵

𝒕𝒕

𝐌𝐌𝐌𝐌𝐌𝐌

𝐀𝐀𝐀𝐀𝐌𝐌

𝒕𝒕

𝑵𝑵

𝒕𝒕+𝟏𝟏

+

Curiosity

Estimator

𝒂𝒂

𝒊𝒊

𝒕𝒕

𝒒𝒒

𝝓𝝓

𝒉𝒉

𝒌𝒌

𝒕𝒕

,

𝒂𝒂

𝒌𝒌

𝒕𝒕

𝒑𝒑

𝒉𝒉

𝒌𝒌

𝒕𝒕

,

𝒂𝒂

𝒌𝒌

𝒕𝒕

×

𝜷𝜷

𝒔𝒔

𝐌𝐌𝐒𝐒𝐌𝐌

𝒌𝒌 ∈ 𝑵𝑵

\

𝒊𝒊

𝒓𝒓

𝒊𝒊

𝒕𝒕

𝒉𝒉

𝟏𝟏

𝒕𝒕

,

𝒂𝒂

𝟏𝟏

𝒕𝒕

…

-

𝒓𝒓

𝟏𝟏

𝒕𝒕

(

𝒕𝒕

𝟏𝟏

𝒕𝒕

,

𝒂𝒂

𝟏𝟏

𝒕𝒕-𝟏𝟏

)

𝑸𝑸

𝟏𝟏

-

Agent

𝟏𝟏

�

𝑸𝑸

𝟏𝟏

IE

𝒉𝒉

𝟏𝟏

𝒕𝒕

𝒂𝒂

𝟏𝟏

𝒕𝒕

×

𝜷𝜷

𝒓𝒓

𝒊𝒊

𝒕𝒕

(

𝒕𝒕

𝒊𝒊

𝒕𝒕

,

𝒂𝒂

𝒊𝒊

𝒕𝒕-𝟏𝟏

)

𝑸𝑸

𝒊𝒊

-

Agent

𝒊𝒊

�

𝑸𝑸

𝒊𝒊

IE

𝒉𝒉

𝒊𝒊

𝒕𝒕

𝒂𝒂

𝒊𝒊

𝒕𝒕

×

𝜷𝜷

𝒓𝒓

𝑵𝑵

𝒕𝒕

(

𝒕𝒕

𝑵𝑵

𝒕𝒕

,

𝒂𝒂

𝑵𝑵

𝒕𝒕-𝟏𝟏

)

𝑸𝑸

𝑵𝑵

-

Agent

𝑵𝑵

�

𝑸𝑸

𝑵𝑵

IE

𝒉𝒉

𝑵𝑵

𝒕𝒕

𝒂𝒂

𝑵𝑵

𝒕𝒕

×

𝜷𝜷

of the other agents and the action taken by the i -th agent:

where τ ˜ t +1 k represents the trajectory of the k -th agent at time-step ˜ t +1 , and H ( · ) represents the entropy term.

If we want to promote the influence among agents, we need to maximize ∑ k ∈N\ i MI ( τ ˜ t +1 k ; a ˜ t i ) . Meanwhile, we can represent p ( τ ˜ t +1 k ) and p ( τ ˜ t +1 k | a ˜ t i ) as the following equations:

Therefore, the mutual information can be rewritten as:

We can see that Eq. (3) is divided into three terms. Term 1 ⃝ is determined by the environment, and hence can be ignored. Term 2 ⃝ computes the information difference of the k -th agent's action selection when the action at time-step ˜ t of the i -th agent is given. Notice the fact that the action of the i -th agent at a particular time can not influence any actions, observations, or trajectories of the other agents at the previous time. Thus, the term 2 ⃝ can also be ignored. Term 3 ⃝ measures the changes in the target observation of the k -th agent. This term can be simplified as log p ( o ˜ t +1 k | τ ˜ t k ,a ˜ t k ,a ˜ t i ) p ( o ˜ t +1 k | τ ˜ t k ,a ˜ t k ) .

Therefore, we conclude that optimizing the mutual information at Eq. (1) is equivalent to optimize:

According to the methods of variational inference [31, 14], we can optimize a tractable evidence lower bound of the above expectation by introducing a variational posterior estimator. Hence, we have the following equation:

where q ϕ represents the variational posterior estimator parameterized by ϕ , and we omit the summation notation for convenience.

For encouraging the interaction among agents and promoting exploration, we design an influencebased intrinsic reward for each agent. The intrinsic reward for agent i at time-step ˜ t is computed as:

where β s is a scaling factor to guarantee r ˜ t inf i > 0 .

4.2 Curiosity-based Intrinsic Reward

Curiosity-based exploration is a widely adopted approach in single-agent reinforcement learning, with the objective of encouraging agents to explore unfamiliar states. Typically, this is achieved by computing intrinsic rewards based on prediction errors [24, 4] or variances [1] of future states, rewards, or value functions. In the context of MARL, curiosity-based exploration remains essential. However, the prediction of joint states becomes unreliable due to the exponential increase in state space with the number of agents, and the scarcity of meaningful states. Fortunately, the local observations of each agent can serve as a measure of curiosity. For simplicity, we use the prediction error of both the global state and the local observations to calculate the curiosity-based intrinsic reward in the partially observable environment as:

where f o i ( τ ˜ t , a ˜ t ) and f s ( τ ˜ t , a ˜ t ) represent the predicting network to estimate the observation of each agent and global state at time-step ˜ t +1 , and a ˜ t , τ ˜ t represent the joint action, and trajectories of all agents at time-step ˜ t .

4.3 Implementation of the Estimators

Our goal is to develop two effective estimators for computing intrinsic rewards as well as make them compute-efficient. For the influence estimator, we model q ϕ ( o ˜ t +1 k | τ ˜ t k , a ˜ t k , a ˜ t i ) by a variational posterior estimator, which is updated via loss:

Meanwhile, the p ( o ˜ t +1 k | τ ˜ t k , a ˜ t k ) is approximated by a prediction network p η updated via loss:

For the curiosity estimator with parameters φ , we update it via loss:

Notice that the summation symbol can become an obstacle when computing intrinsic rewards and losses efficiently. In order to accelerate the training procedure, we have implemented the two estimators in a compute-efficient manner, as illustrated in Fig. 2.

In the influence estimator, Eq. (7) requires us to propagate q ϕ for N -1 iterations in order to obtain the intrinsic reward for a single agent. To improve efficiency, we have designed a batch implementation for ∑ k ∈N\ i , allowing us to estimate the observations of all other agents simultaneously in a single forward propagation. Likewise, p ( ·| τ k , a k ) also adopts the same trick.

As for the curiosity estimator, all f o i have the same inputs. To streamline this process, we implement them using a single neural network that outputs a concatenated tensor.

To further alleviate the computational burden, we adopt the parameter-sharing technique, and all agents share the same network in the influence estimator. Similarly, for curiosity-based intrinsic reward, f s and f o share the same encoder g to get the representations. Inspired by the approach presented in [15], we represent the trajectory of the agent using the hidden state of a recurrent neural network. Consequently, we have h ˜ t i = τ ˜ t i and h ˜ t = τ ˜ t . Meanwhile, for the learning more stable, we compute the intrinsic rewards in Eq. (5) and Eq. (6) via target networks ϕ ′ , η ′ and φ ′ which are copied from ϕ , η and φ periodically.

4.4 Incorporation of the Two Heads

We would like to emphasize the effectiveness of combining these two methods in a straightforward manner, rather than designing a intricate algorithm that solely focuses on one aspect. There are two reasons for this: (i) Curiosity-based intrinsic rewards are designed to encourage exploration of unfamiliar states, while influence-based intrinsic rewards aim to promote coordination among agents. These rewards operate on different dimensions and play distinct roles in various scenarios. (ii) Either of the methods may not work in certain training time-steps, as no method can guarantee constant positive influence among agents, and the meaningful unfamiliar joint state is too sparse. Some recent works [33, 39] attempt to use both kinds of methods, however, concerns regarding their effectiveness and computational efficiency remain which lead the unsatisfactory results in complicated scenarios.

In this subsection, we will introduce how to incorporate them under the paradigm of CTDE. For simplicity, we omit the superscript of time-step in this subsection. For the influence-based intrinsic rewards, we apply it to the individual local value functions of each agent to promote the interaction. For i -th agent it is computed as:

where Q i is the local value function, and β inf is the scale factor. ˜ Q i will then be sent to the central critic Mixer to get the joint value function:

Notice that the intrinsic rewards are incorporated before the value functions are sent to the central mixer. This is done to ensure that the intrinsic rewards can be scaled by the network appropriately. Meanwhile, it is worth mentioning that the intrinsic rewards are not applied to the target network. The reasons for this decision will be discussed in the Appendix.

The curiosity-based intrinsic rewards are directly added to the extrinsic rewards, and subsequently, the agents and the central mixer are updated using TD-loss:

where Q ′ tot represents the target network, β cur represents the scale factor.

We show the algorithm of our COIN in Alogrithm 1.

𝑞

!

/

𝑝

"

ℎ

!

, 𝑎

!

𝑁 - 1

ℎ

"

, 𝑎

"

ℎ

#

, 𝑎

#

!

𝑎

$

𝑎

$

𝑎

$

!

𝑜

!

, … , 𝑜

#

𝑁 - 1

ℎ

!

𝑎

!

ℎ

"

𝑎

"

ℎ

#

𝑎

#

!

𝑔

𝑓

!

𝑓

"

Influence Estimator

Curiosity Estimator

𝑠

𝑜

!

𝑜

"

𝑜

#

!

Algorithm 1 Influence and Curiosity-based Exploration

Initialize: Local agents and central critic θ with its target networks θ ′ , max episode length T , networks of the estimator for computing intrinsic rewards ϕ , η and φ with its target networks ϕ ′ , η ′ and φ ′ .

̸

5 Experiments

Our experiments aim to answer the following questions: (1) Q1: Is the exploration strategy important in MARL? (2) Q2: How does each kind of exploration strategy affect the model performance in different scenarios? (3) Q3: Can influence-based exploration cooperate with curiosity-based exploration?

We performed experiments on three popular MARL benchmarks: StarCraft II micro management challenge (SMAC)[29], MACO[36], and Google Research Football (GRF) [13], with different scenarios. The details of each benchmark will be introduced in the Appendix. We adopted two popular baselines, QMIX [27] and QPLEX [32], and applied our exploration method to them. To further demonstrate the effectiveness of our exploration method, we chose the curiosity-based exploration method EMC [41], the influence-based exploration method EITI [33], and the hybrid method CIExplore [39] as strong baselines. It should be mentioned that EITI and CIExplore are implemented based on the actor-critic framework in the original paper. For a fair comparison, we implemented EITI and CIExplore based on QMIX under the same framework as us. Meanwhile, since we aim to verify the effectiveness of different exploration strategies, we do not use the special episode buffer in EMC, as well as the other tricks in all methods. All the curves in the figures of this section are the mean of 10 random seeds, which have been smoothed, and the shaded areas represent the standard deviation. The experiment settings are also provided in the Appendix.

5.1 Performance on several benchmarks (Q1)

Performance on StarCraft II benchmark. We first conduct experiments on the most popular environment, StarCraft II micro management challenge. StarCraft II is a real-time strategy game. We set the difficulty of game AI at the 'very difficult' level. This benchmark consists of various maps with different levels of challenges, 'easy', 'hard', and 'super hard'. We select three representative

QMIX+Ours

QPLEX+Ours

QPLEX

EITI

CIExplore

EMC

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(a) 3s5z_vs_3s6z

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(b) 6h_vs_8z

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(c) Corridor

QMIX+Ours

QPLEX+Ours

QPLEX

EITI

CIExplore

EMC

0

0.5

1

1.5

2

0

20

40

60

Timesteps (mil)

Test Transmitted

(a) Aloha

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(b) Gather

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(c) Hallway

and super hard scenarios, 3s5z_vs_3s6z , corridor , and 6h_vs_8z to carry out experiments 2 , and the results are depicted in Fig. 3. Our method shows a significant improvement over the baselines in all maps, highlighting the importance of the exploration strategy in MARL. Furthermore, the superior results compared to other exploration baselines demonstrate the effectiveness of our COIN method. We also conducted experiments with sparse reward setting, and the results are provided in the Appendix.

Performance on MACO benchmark. Next, we evaluated our method in the MACO benchmark, which features relatively sparse rewards. MACO incorporates several coordination tasks from various multi-agent learning literature and enhances their difficulty. We conducted experiments on three MACO scenarios: Aloha , Hallway , and Gather . These scenarios require different levels of agent coordination. The results of our experiments are presented in Fig. 4.

In the Aloha scenario [8, 18], the agents strive to achieve as many successful transmissions as possible within a limited number of time steps. None of the other exploration methods proved effective in this scenario. Our COIN approach improved the two baselines by nearly 20 mean transmissions. The results of all methods were peculiar, with an initial burst followed by a drop and then plateaued poor results. Initially, the agents attempted exploration, resulting in more successful transmissions. However, the punishment mechanism in this setting was overly severe, causing the agents to adopt conservative strategies. In the Gather scenario, the task is relatively simple, with agents aiming to choose the same action to achieve common goals set at the beginning of each episode. Although exploration did not significantly improve the performance of the baselines, our COIN approach still achieved a faster convergence speed. There was a failure case known as the "noisy TV" in the Gather scenario. Choosing the goal set at the beginning of the episode resulted in a higher global reward, while not choosing this goal led to a relatively lower reward. It is evident that QPLEX failed

and produced suboptimal results, and our framework did not provide significant improvement. The Hallway scenario [34] involved agents divided into several groups, where agents within the same group aimed to reach the same state simultaneously, while different groups aimed to reach different states. Our COIN approach succeeded with QPLEX, resulting in a 60% improvement in mean win rates. However, applying any exploration method to QMIX showed little improvement, which can be considered another failure case.

Performance on GRF benchmark. In the GRF game, players need to coordinate with each other to organize offense and score goals. The rewards in GRF are sparse, with agents only receiving rewards at the end of the game. Therefore, exploration plays a vital role in guiding the agents to discover winning strategies.

We compared the performance of several baselines on the GRF benchmark, as shown in Fig. 5. The curve representing our COIN approach demonstrates the application of our exploration framework on QMIX. Our approach outperforms in the scenario 'academy 3 vs 1 with keeper". QMIX and QPLEX fail to perform well due to the sparse rewards. EMC applies a curiositybased exploration strategy to QPLEX but results in a decrease in model performance. In contrast, our COIN approach combines influence-based exploration and curiosity-based exploration, outperforming all baselines with an improvement of nearly 0.2 mean scores per episode.

5.2 Ablations and Interesting Findings (Q2, Q3)

QMIX+Inf+Cur

QMIX+Inf

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Score

(a) academy 3 vs 1 with keeper

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(b) 3s5z_vs_3s6z

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(c) Corridor

To understand how influence-based exploration and curiosity-based exploration work in different scenarios, we conducted ablation studies. We selected three representative scenarios to showcase these results, as depicted in Fig. 6. The dotted line represents applying only the influence-based intrinsic reward on QMIX, while the dashed line represents applying only the curiosity-based intrinsic reward. Interestingly, we found that different exploration strategies play different roles in different scenarios. When only influence-based or curiosity-based exploration was conducted in the academy 3 vs 1 with keeper scenario, the model did not benefit from the exploration strategy. Instead, the performance decreased by nearly 0.5 mean scores. To understand this finding, we consulted literature and watched replays, and find that this phenomenon arises from the parameter sharing technique. According to Li et al. [14], shared parameters among agents can lead to similar behaviors, hindering the model's ability to learn successful policies in challenging tasks. Our influence-based intrinsic reward promotes agents to influence each other, bringing diversity to their behaviors. However, focusing solely on influence

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Score

among agents neglects crucial patterns, such as cutting to attract the enemy defender, resulting in poor outcomes. Fortunately, our proposed method succeeded by striking a balance between influence and curiosity. This interesting result also explains why EMC decreases the baseline performance in this scenario, as shown in Fig. 5. In the scenario 3s5z_vs_3s6z , influence-based exploration plays a significant role, while curiosity-based exploration harms the performance. This is because the learned winning strategy involves one of the allied units kiting several enemies to the corner of the map and sacrificing itself, thus not requiring much exposure to unknown states. In the corridor scenario, both exploration strategies improve the baseline performance significantly. However, curiosity-based exploration achieves even greater improvement and reaches the best performance.

In all scenarios, the best performance is achieved when both kinds of exploration strategies are incorporated. Therefore, we conclude that different scenarios require different exploration strategies, and when we are unsure which one will be effective, incorporating both is often a good choice. For more experimental results, please refer to the Appendix.

5.3 The Chosen hyperparameters

In Section 4, we introduced several hyperparameters for computing intrinsic rewards. To ensure stability and guarantee that the influence-based intrinsic reward is larger than zero, we set β s in Eq. (5) to 0.5 for all scenarios in all benchmarks. Additionally, β inf and β cur are scaling factors used to adapt the two types of intrinsic rewards to the environment. Given that the importance of the two types of exploration may vary at different time steps, determining which one will dominate is challenging. To address this, we combine the two types of intrinsic rewards in a straightforward manner. When designing the hyperparameters, we follow two principles: (i) ensuring that both types of intrinsic rewards are on the same order of magnitude, and (ii) scaling the intrinsic rewards to be approximately 100 to 1000 times lower than the maximum extrinsic reward at the beginning of the training stage. Details of all the hyperparameters are listed in Appendix.

6 Conclusion

In this paper, we explore an interesting phenomenon where different types of exploration play varying roles in different scenarios of MARL. However, most current studies tend to focus on one type of exploration and overlook the other. In light of this, we propose a framework called COIN that effectively combines and complements these two types of exploration strategies. Firstly, COIN measures the influence of each agent on the others using mutual information theory, and assigns it as intrinsic rewards to individual value functions. Additionally, COIN calculates curiosity-based intrinsic rewards by utilizing prediction errors, which are then added to the extrinsic reward. By integrating these two types of intrinsic rewards, COIN creates a novel framework where they complement each other, ultimately leading to more efficient and effective exploration in cooperative MARL tasks. Both of these types of intrinsic rewards are computationally efficient within our designed framework, thus enabling effective exploration. Our experiment results on various benchmarks demonstrate the superiority of our proposed method.

Limitations and future work. Though the incorporation of two kinds of exploration is useful, our method relies on scaling hyperparameters and cannot automatically weigh the two kinds of strategies. In future work, we will investigate how to discover the causal relations among agents. With these potential interdependencies, we can assign appropriate weights to each agent and avoid computing the MI for unnecessary agent pairs.

References

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations , 2021.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research , 27(4): 819-840, 2002.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 38(2):156-172, 2008.

Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems , pages 66-83, 2017.

Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI , volume 4, pages 709-715, 2004.

Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning , pages 3040-3049. PMLR, 2019.

Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning , pages 10041-10052. PMLR, 2022.

Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning , pages 4992-5001. PMLR, 2021.

Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing , 190:82-94, 2016.

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Michał Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4501-4510, 2020.

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:3991-4002, 2021.

Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Changjie Fan, Fei Wu, and Jun Xiao. Deconfounded value decomposition for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 12843-12856. PMLR, 2022.

Pengyi Li, Hongyao Tang, Tianpei Yang, Xiaotian Hao, Tong Sang, Yan Zheng, Jianye Hao, Matthew E Taylor, Wenyuan Tao, and Zhen Wang. Pmic: Improving multi-agent reinforcement learning with progressive mutual information collaboration. In International Conference on Machine Learning , pages 12979-12997. PMLR, 2022.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations , 2016.

Frans Oliehoek. Value-based planning for teams of agents in stochastic partially observable environments . Amsterdam University Press, 2010.

Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs . Springer, 2016.

Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research , 32:289-353, 2008.

Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning , pages 2721-2730. PMLR, 2017.

Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics , page 6, 2009.

Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 443-451, 2018.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069 , 2017.

Sarvapali D Ramchurn, Alessandro Farinelli, Kathryn S Macarthur, and Nicholas R Jennings. Decentralized coordination in robocup rescue. The Computer Journal , 53(9):1447-1461, 2010.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4295-4304, 2018.

Tabish Rashid, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations , 2019.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. 2019.

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems , 30, 2017.

Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning , 1(1-2):1-305, 2008.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations , 2021.

Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In International Conference on Learning Representations , 2019.

Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. In International Conference on Learning Representations , 2020.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. 2021.

Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. arXiv preprint arXiv:2106.02886 , 2021.

Ermo Wei and Sean Luke. Lenient learning in independent-learner stochastic cooperative games. The Journal of Machine Learning Research , 17(1):2914-2955, 2016.

Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. Hierarchically and cooperatively learning traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 669-677, 2021.

Huanhuan Yang, Dianxi Shi, Chenran Zhao, Guojun Xie, and Shaowu Yang. Ciexplore: Curiosity and influence-based exploration in multi-agent cooperative scenarios with sparse rewards. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2321-2330, 2021.

Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. sensors , 15(5):10026-10047, 2015.

Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiositydriven exploration. Advances in Neural Information Processing Systems , 34:3757-3769, 2021.

A Appendix

A.1 Details of the benchmarks

Multi-agent coordination challenge benchmark. Wang et al. [36] collect six coordination tasks from various multi-agent learning literature to create the Multi-Agent Coordination Challenge (MACO) benchmark. For our experiments, we have selected three tasks from this benchmark.

The Aloha coordination task [8, 18] consists of a 2 × 5 array with 10 agents. The objective is to send messages and maximize the backlog, which has a maximum value of 5. However, if two adjacent agents send messages simultaneously, a collision occurs, and the messages need to be resent. For each successful transmission, a positive global reward of 0.1 is gained, while a collision results in a negative global reward of -10. Each unit initially has 1 backlog, and at each time step, there is a probability of 0.6 for a new packet to arrive, provided that the maximum backlog has not been reached yet. In Gather [37], there are 5 agents whose objective is to collectively choose the same action in order to achieve common goals. These goals are established at the beginning of each episode. If all agents successfully choose the goal, the entire team receives a reward of 10. On the other hand, if none of the agents chooses the goal, a reward of 5 is given to the team. Hallway [34] involves 12 agents organized into multiple groups. Each agent is randomly assigned to a state within each chain. In this task, if agents belonging to the same group reach the same state at the same time, they receive a global reward. However, if more than one group reaches the same state, a global punishment is given.

StarCraft II micro management benchmark. The StarCraft II micromanagement challenge [29] involves agents controlling allied armies and collaborating to engage with enemy forces controlled by the game engine. This benchmark includes different maps that pose varying levels of challenges, categorized as 'easy', 'hard' and 'super hard'. For our experiments, we have chosen three representative super hard scenarios: 3s5z_vs_3s6z , corridor , and 6h_vs_8z . The game AI difficulty is set to 'very difficult" level. Table 1 provides an introduction to these selected maps.

Google research football benchmark. In the Google Research Football (GRF) game [13], players are required to coordinate with each other to organize offensive plays and score goals. In this game, agents control the players on the left side, excluding the goalkeeper, while the players on the right side are controlled by rule-based bots using hand-coded built-in AI. Each player can observe the positions and movement directions of all players, as well as the football. They have a discrete action space that includes options for moving, sliding, shooting, and passing. Soccer players must determine the optimal timing for shooting by combining dribbling and passing techniques. In this game environment, rewards are only provided at the end of the game, with a reward of 100 for scoring a goal and a penalty of -1 for losing the game.

A.2 Experiment settings

We list the experiment settings and hyperparameters in Table 2. The two important scale factors β cur and β inf in MACO depends on the scenarios. We set β cur = β inf = 0 . 05 in aloha , β cur = β inf = 0 . 01 in gather , and β cur = β inf = 0 . 0005 in hallway .

A.3 Comparison with the other exploration methods

In this paper, we employ both influence-based and curiosity-based exploration strategies. In comparison to other methods, we utilize a simple yet effective approach to implement these two types of exploration. In this subsection, we will discuss the differences among these methods.

Mutual information and influence-based exploration. Mutual information is a commonly utilized concept in existing works for multi-agent reinforcement learning (MARL) problems. MOA [9] was the first to propose using MI ( a t i ; a t j | s t ) as intrinsic rewards. Another formulation called EITI [33] is given by MI ( o t +1 j ; o t i , a t i | o t j , a t j ) . Similarly, PMIC [16] proposes MI ( a ; s ) as a collaboration criterion. Additionally, Jiang and Lu [11] propose MI ( I ; O ) , while Li et al. [14] propose MI ( τ ; I ) , where I represents the agent's ID. The main differences among these methods lie in their intentions. The first three methods aim to promote influence and collaboration among agents, while the last two aim to enhance diversity among the agents.

In this paper, we also incorporate mutual information MI ( τ t +1 j ; a t i ) to quantify the influence of one agent's action on another agent. While the basic motivation aligns with MOA and EITI, our proposed approach is (a) more reasonable and (b) easier to compute. The reasons for these distinctions are as follows: (a) More reasonable: Intuitively, an agent can only influence the future behavior of other agents. Therefore, we consider MI ( a t i ; a t j | s t ) as less reasonable. Additionally, the crucial factor that influences other agents is the action taken, rather than the current states or observations. Consequently, MI ( τ t +1 j ; a t i ) is more reasonable than MI ( o t +1 j ; o t i , a t i | o t j , o t j ) . Specifically, in the computation of the i -th influence on j ,

𝑠𝑠

𝑡𝑡

𝑜𝑜

𝑡𝑡+1

𝐸𝐸

𝐸𝐸

𝑎𝑎

𝑖𝑖

𝑡𝑡

𝑜𝑜1

𝑡𝑡

𝑜𝑜

𝑛𝑛

𝑡𝑡

…

…

𝑜𝑜

𝑖𝑖

𝑡𝑡

𝑜𝑜

𝑡𝑡

𝑜𝑜1

𝑡𝑡+1

𝑜𝑜

𝑛𝑛

𝑡𝑡+1

…

…

𝑜𝑜

𝑗𝑗

𝑡𝑡+1

encing the Markov decision process in Fig. 7, in p ( o t +1 j | o t i , a t i , o t j , a t j ) , o t i becomes the noise factor in predicting o t +1 j , which can potentially hinder accurate influence estimation in certain scenarios. For instance, let the optimal policy π ( a t i | o t i , s t i ) be an and operation, and let the observation function g ( o t j |· ) be an element-wise xor operation for all inputs. o t +1 j should be 0 when o t i = 1 and s t = 0 , but introducing o t i = 1 into g would result in o t +1 j = 1 . (b) Easier to compute: We have designed a framework where the intrinsic reward at each time step can be computed in a single forward propagation. This approach is straightforward to implement and does not impose a significant computational burden.

Prediction network and curiosity-based exploration. Curiosity-based exploration methods, widely employed in reinforcement learning [22, 28], typically involve designing intrinsic rewards to measure the degree of unfamiliarity associated with states. A common approach entails setting up one or several networks to predict information about the next time-step, with the instability of these predictions serving as the intrinsic reward. For instance, Pathak et al. [24] and Burda et al. [4] consider the prediction error of target states as intrinsic rewards. Bai et al. [1] use the standard deviation of target Q-function predictions as the intrinsic reward. Zheng et al. [41] utilize the prediction errors of individual Q-functions as intrinsic rewards in multi-agent reinforcement learning (MARL) problems. Similarly, Yang et al. [39] design intrinsic rewards based on the variance of an ensemble predictive model incorporating local observations and global states. In our approach, we adopt a similar strategy to the aforementioned methods. We use the prediction errors of next

states and observations as intrinsic rewards. Since the observations and global states are provided by the environment, we consider them to be more reliable supervision information compared to value functions. Moreover, these prediction errors can capture the unfamiliarity from two perspectives: local observations for individual unfamiliarity and global states for joint unfamiliarity. Consequently, this approach is straightforward yet effective in MARL settings.

A.4 Two heads are better than one

One key highlight of this paper is the effectiveness of incorporating both methods rather than solely focusing on a single aspect through an elegant algorithm. To support our viewpoint, we evaluate different exploration strategies on the baselines [27, 32] and present the results on the MACO benchmark in Fig. 8 amd Fig 9. In the figure, the dashed line represents solely applying the curiositybased intrinsic reward, while the dotted line represents solely applying the influence-based intrinsic reward. It is worth noting that either exploration strategy may fail in certain scenarios. However, by incorporating both methods, we consistently achieve the best results. We also perform experiments on the StarCraft II benchmark with sparse reward setting, and the results can is show in Fig. 10, in which MASER [10] is a goal based exploration method.

1

QMIX+Inf+Cur

QMIX+Inf

QMIX+EITI(Inf)+Cur

QMIX+EITI(Inf)

0

0.5

1

1.5

2

0

20

40

60

Timesteps (mil)

Test Transmitted

(a) Aloha

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(b) Gather

0

0.2

0.4

0.6

0.8

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(c) Hallway

QPLEX+Cur

QPLEX+Inf

QPLEX+inf+cur

QPLEX

0

0.5

1

1.5

2

0

20

40

60

Timesteps (mil)

Test Transmitted

(a) Aloha

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(b) Gather

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

(c) Hallway

QMIX+Ours

MASER

0

0.5

1

1.5

2

2.5

3

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

0

0.5

1

1.5

2

2.5

3

0

0.2

0.4

0.6

0.8

1

Timesteps (mil)

Test Win Rate

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a novel and compute-efficient exploration method: COIN, which can incorporate Curiosity-based and Influence-based exploration. For influence-based exploration，COIN quantify how each agent’s action affect the other agents and use the influence degree as the intrinsic reward to promote the exploration. For curiosity-based exploration，COIN uses the prediction errors of local observations of each agent and the global states to measure the uncertainty of the model to the environment. The experiments on three benchmarks StarCraft II, MACO, and Google Football show the superiority and effectiveness of COIN.\n\nSTRENGTHS:\n(1) This work designs a MARL framework which combines two kinds of main exploration methods and achieve sufficient and efficient exploration without bringing many computational costs.\n\n(2) The exploration method proposed in this paper can be easily applied to other multi-agent methods.\n\n(3) The overall idea of the paper is clear and easy to understand.\n\nWEAKNESSES:\n(1) To be honest，it lacks a certain degree of novelty. As is well known, the methods used in this paper like Qmix, MI and Prediction Error are all previous work. We just do some combination work.\n\n(2) It should explain how to extend the method in this paper to other MARL methods.\n\n(3) The existing experimental results seem insufficient，mainly reflected in the following aspects:\n\nMany result curves do not show the final convergence state, but rather the intermediate state;\n\nLack of quantitative forms of expression, such as reward values or case study.\n\n(4) The main body of this paper is clear to understand, here is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n\nQUESTIONS:\n(1) In Equ.(1) and (2), the trajectory τkt should include state and action. So what is the meaning of p({a_k^t|\\tau}_k^t\\ )?\n\n(2) Influence-based intrinsic reward is higher. This indicates that the action is more encouraging. Thus, In Equ.(8), why is“-”instead of“+”？\n\n(3) Comparing with Fig.2, In Equ.(7), why is the input different between them ? \n\n(4) The description of hyperparameters in the experiment can not be found.\n\n(5) In Fig.5(a), why is the performance of Qmix better than Qmix+Inf and Qmix+Cur?\n\n(6) Is the exploration framework of this paper attached to other MARL algorithms? How well?\n\nLIMITATIONS:\nIn section 6, the authors talk about some limitations about this paper. At present, the potential negative social impact of this work has not been identified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates exploration methods in multi-agent reinforcement learning (MARL). Specifically, among the two main families of exploration methods, namely curiosity-based methods and influence-based methods, the paper reports that they could serve complementary roles to each other. Subsequently, the paper proposes a method that takes advantage of both types of exploration methods, demonstrating its effectiveness in 3 common MARL benchmarks.\n\nSTRENGTHS:\n1.\tMARL exploration is an open and important problem for the field. The proposal of combining the two families of exploration methods in MARL is novel and the highlight of their complementary role is significant. \n2.\tThe experimental results overall show the effectiveness of the proposed method against strong baselines\n3.\tThe paper is well-written with the method well-explained.\n\nWEAKNESSES:\n1.\tThere are some inconsistencies in results across base algorithms of the proposed methods in different environments. Some additional ablation results could improve this (will be mentioned in the question section)\n2.\tThe mentioned inconsistencies are not well explained, making the results less convincing.\n\nQUESTIONS:\n1.\tThe results of QMIX + ours in Aloha seem to be quite odd given the early burst and drop with the plateaued bad results. Any intuition or explanation for the phenomenon?\n2.\tCan you elaborate more on the possible reasons behind the performance discrepancies between the results of QMIX and QPLEX when adding your method?\n3.\tDo you have the results for QPLEX+ours in GRF? This could strengthen the results \n4.\tGiven the drastic difference between QMIX and QPLEX when combined with the proposed exploration method, do the ablation results apply to QPLEX too\n5.\tCan you elaborate on how causal relationships could help in balancing the two terms?\n6.\tMore of an open question, Could this have any failure cases specific to the multi-agent setting like the noisy TV issues in the single-agent case?\n7.\tWould these results apply in fully decentralized MARL methods without sharing parameters?\n\nLIMITATIONS:\nOne limitation is discussed. No negative societal impact was discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose a new framework to improve exploration in MARL. The method COIN improves exploration by combining the concepts in the literature of curiosity and influence.\n\nSTRENGTHS:\nThis paper proposes a method that combines in a framework two popular concepts in the reinforcement learning literature, curiosity and influence, aiming to improve exploration. The authors analyse how the different components of their method affects their approach. A detailed description of how the method works is also provided and it is then tested in a set of different environments.\n\nWEAKNESSES:\n- \"Li et al. [12] point that shared parameters among agents induce similar behaviors of players which makes the model fail to learn successful policies on this challenging task. Our influence-based intrinsic reward promotes the agents to affect each other and hence brings diversity.\" - this is a problem caused by the famous parameter sharing paradigm, that the authors also adopt in COIN; i feel that this is a big claim that I believes it needs further evidence.\n\nMinor:\n- line 156: \"courage\"->\"encourage\"\n- line 96-97: \"it shows unstable and innefective\" poorly phrased\n- l 144: \"varriance\" -> variance\n- l 154: shouldnt it also be $\\tau^~t$?\n- line 283: plays -> play\n- description of the dec pomdp in section 3 needs to be reviewed: for example, the observation model is missing and $\\mathcal{O}$ represents the observation function; $\\mathcal{N}$ inside the tuple G should be the number of agents and not the set; among others\n\nPlease find my questions below.\n\nQUESTIONS:\n1. In line 46 the authors state that their method is compute-efficient. Is there any evidence or proof for this?\n2. in equation 7, does it mean that the only difference to predict state and observations is that to predict observations the agents use only the trajectories from a single time step? If so, is this accurate if we dont consider the previous values, since MARL is a non stationary sequential problem?\n3. I have some questions about the influence of agents on others: does influence-based exploration mean that the agents are going to explore states where each agent has more influence on the actions of the others? If this is the case, how do they measure this influence? From my understanding, this is measured by summing the mutual information between all trajectories and the action of each agent individually to see the influence of each agent on the others; but does this guarantees that influence on the others is always good influence? i.e., can there be cases where we have bad influence and so these states shouldn't be explored?\n4. In line 156: \"Curiosity-based intrinsic rewards aim to courage the unfamiliar state, and influence-based intrinsic rewards aim to promote coordination.\" While curiosity based methods clearly directly tackle exploration problems, can we say the same from influence-based methods? Following the same logic, we could also say that almost any method in MARL is a method that tackles specifically exploration problems since most of them promote coordination.\n5. the authors start the paper by mentioning that exploration is important in sparse reward environments; in this sense, have they tried to run their method on starcraft with sparse rewards too? Why choosing only normal stracraft environments in this case?\n6. in section 5.3 it is unclear what the authors end up picking for $\\beta_{inf}$ and $\\beta_{cur}$: do they get the same weight, i.e., the same value?\n7. there are also other methods that work towards efficient exploration, such as []. These methods, for example, use sub-goals instead of curiosity, which is also a common benchmark for some exploration-based methods. Have the authors considered comparing against some of these too?\n\n[1] https://arxiv.org/abs/2206.10607\n\nLIMITATIONS:\nPlease see above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at two different exploration strategies within Multiagent Reinforcement Learning. The two exploration strategies are curiosity-based and influenced based, where the latter is unique to MARL and takes into account the mutual information of one agents actions on the trajectory of another agent. The study then looks at combining these two forms of exploration and studies the effects of the individual intrinsic rewards as well as their combined effects on three distinct environments. A number of ablations are performed during these studies.\n\nSTRENGTHS:\nThe paper introduces a novel algorithm which is able to explore in a MARL setting using two different additional intrinsic reward signals. It seems that such an exploration strategy is effective in the examples given at improving performance. The algorithm is relatively clean and the explanation is clear and precise. Exploration strategies are clearly a very important topic within the realm of computationally costly settings and so such an improvement is clearly important.\n\nWEAKNESSES:\nThe language in the paper needs some significant work. I would recommend passing the paper through an LLM for checking for spelling and grammar throughout, as there are a fair number of such issues.\n\nThe lack of clear exploration in the hyperparameter space is a certainly an important limitation of this work. While the choice of hyperparameters is motivated, it is not clear that this has actually been studied, and in particular where the mixture of two intrinsic rewards is being used together, it seems that such an investigation would be very important to have a deeper understanding of their effects. Although an ablation study has been performed, this is really a binary investigation, whereas the continuum of mixings between the rewards would be most interesting.\n\nAlthough there are a fair number of plots showing the learning trajectory on the test score, no summary statistics have been included. It would be useful to have a more detailed, quantitative set of measures of improvement, in particular over the hyperparameter sweeps.\n\nQUESTIONS:\nI believe that the weaknesses highlight the questions that I have, in particular surrounding the hyperparameters and a more detailed investigation of the statistics.\n\nLIMITATIONS:\nWhile limitations around the hyperparameters have been noted, these do not give sufficient response to the points raised above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}