SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The curiosity-based intrinsic rewards are directly added to the extrinsic rewards, and subsequently, the agents and the central mixer are updated using TD-loss:
$$
L _ {\theta} = \left(y - Q _ {t o t}\right), \tag {12}
$$
$$
y = r + \beta_ {c u r} * r _ {c u r} + \gamma * Q _ {t o t} ^ {\prime}, \tag {13}
$$
where  $Q_{tot}^{\prime}$  represents the target network,  $\beta_{cur}$  represents the scale factor.
We show the algorithm of our COIN in Algorithm 1.
Algorithm 1 Influence and Curiosity-based Exploration
Initialize: Local agents and central critic  $\theta$  with its target networks  $\theta^{\prime}$  , max episode length  $T$    
networks of the estimator for computing intrinsic rewards  $\phi$ $\eta$  and  $\varphi$  with its target networks  $\phi^{\prime}$ $\eta^{\prime}$  and  $\varphi^{\prime}$  .   
1: for each training episode do   
2: while global state  $s\neq$  terminal and time step  $t <   T$  do   
3:  $t = t + 1$    
4: for each agent i do   
5: Compute the local value function  $Q_{i}$  and get the hidden state  $h_t^i$    
6: Select action  $a_{t}^{i}$  and execute   
7: end for   
8: Execute the joint action  $\pmb {a} = (a_t^1,a_t^2,\dots,a_t^n)$    
9: Get reward  $r_{t + 1}$  and next state  $s_{t + 1}$    
10: end while   
11: Add episode to replay buffer   
12: Collate episodes in the buffer into a single batch   
13: for  $t = 1$  to  $T$  do   
14: Compute the influence-based intrinsic reward via Eq. (5)   
15: Compute the Curiosity-based intrinsic reward via Eq. (6)   
16: Add the influence-based to individual value functions via Eq. (10)   
17: Compute the joint value function via Eq. (5)   
18: Compute the targets y via Eq. (13)   
19: Update  $\theta$  by minimizing the TD-loss in Eq. (12)   
20: Update  $\phi ,\eta$  and  $\varphi$  by minimizing the loss in Eq. (7),Eq. (8) and Eq. (9)   
21: Update  $\theta^{\prime} = \theta$ $\phi^{\prime} = \phi$ $\eta^{\prime} = \eta$ $\varphi^{\prime} = \varphi$  periodically   
22: end for   
23: end for
Our experiments aim to answer the following questions: (1) Q1: Is the exploration strategy important in MARL? (2) Q2: How does each kind of exploration strategy affect the model performance in different scenarios? (3) Q3: Can influence-based exploration cooperate with curiosity-based exploration?
We performed experiments on three popular MARL benchmarks: StarCraft II micro management challenge (SMAC)[29], MACO[36], and Google Research Football (GRF) [13], with different scenarios. The details of each benchmark will be introduced in the Appendix. We adopted two popular baselines, QMIX [27] and QPLEX [32], and applied our exploration method to them. To further demonstrate the effectiveness of our exploration method, we chose the curiosity-based exploration method EMC [41], the influence-based exploration method EITI [33], and the hybrid method CIExplore [39] as strong baselines. It should be mentioned that EITI and CIExplore are implemented based on the actor-critic framework in the original paper. For a fair comparison, we implemented EITI and CIExplore based on QMIX under the same framework as us. Meanwhile, since we aim to verify the effectiveness of different exploration strategies, we do not use the special episode buffer in EMC, as well as the other tricks in all methods. All the curves in the figures of this section are the mean of 10 random seeds, which have been smoothed, and the shaded areas represent the standard deviation. The experiment settings are also provided in the Appendix.
5.1 Performance on several benchmarks (Q1)
Performance on StarCraft II benchmark. We first conduct experiments on the most popular environment, StarCraft II micro management challenge. StarCraft II is a real-time strategy game. We set the difficulty of game AI at the "very difficult" level. This benchmark consists of various maps with different levels of challenges, "easy", 'hard', and "super hard". We select three representative
(a) 3s5z_vs_3s6z
(b) 6h_vs_8z
(c) Corridor
Figure 3: Performance of our exploration method on three scenarios in StarCraft II benchmark.
(a) Aloha
Figure 4: Performance of our exploration method on three scenarios in MACO benchmark.
(b) Gather
(c) Hallway
and super hard scenarios,  $3s5z\_ vs\_ 3s6z$ , corridor, and  $6h\_ vs\_ 8z$  to carry out experiments<sup>2</sup>, and the results are depicted in Fig. 3. Our method shows a significant improvement over the baselines in all maps, highlighting the importance of the exploration strategy in MARL. Furthermore, the superior results compared to other exploration baselines demonstrate the effectiveness of our COIN method. We also conducted experiments with sparse reward setting, and the results are provided in the Appendix.
Performance on MACO benchmark. Next, we evaluated our method in the MACO benchmark, which features relatively sparse rewards. MACO incorporates several coordination tasks from various multi-agent learning literature and enhances their difficulty. We conducted experiments on three MACO scenarios: Aloha, Hallway, and Gather. These scenarios require different levels of agent coordination. The results of our experiments are presented in Fig. 4.
In the Aloha scenario [8, 18], the agents strive to achieve as many successful transmissions as possible within a limited number of time steps. None of the other exploration methods proved effective in this scenario. Our COIN approach improved the two baselines by nearly 20 mean transmissions. The results of all methods were peculiar, with an initial burst followed by a drop and then plateaued poor results. Initially, the agents attempted exploration, resulting in more successful transmissions. However, the punishment mechanism in this setting was overly severe, causing the agents to adopt conservative strategies. In the Gather scenario, the task is relatively simple, with agents aiming to choose the same action to achieve common goals set at the beginning of each episode. Although exploration did not significantly improve the performance of the baselines, our COIN approach still achieved a faster convergence speed. There was a failure case known as the "noisy TV" in the Gather scenario. Choosing the goal set at the beginning of the episode resulted in a higher global reward, while not choosing this goal led to a relatively lower reward. It is evident that QPLEX failed
2We do not carry out the experiment of EMC on corridor because it takes too much computational resources which is out of the max memory of our GPU (GTX 2080 Ti).
and produced suboptimal results, and our framework did not provide significant improvement. The Hallway scenario [34] involved agents divided into several groups, where agents within the same group aimed to reach the same state simultaneously, while different groups aimed to reach different states. Our COIN approach succeeded with QPLEX, resulting in a  $60\%$  improvement in mean win rates. However, applying any exploration method to QMIX showed little improvement, which can be considered another failure case.
Performance on GRF benchmark. In the GRF game, players need to coordinate with each other to organize offense and score goals. The rewards in GRF are sparse, with agents only receiving rewards at the end of the game. Therefore, exploration plays a vital role in guiding the agents to discover winning strategies.
We compared the performance of several baselines on the GRF benchmark, as shown in Fig. 5. The curve representing our COIN approach demonstrates the application of our exploration framework on QMIX. Our approach outperforms in the scenario "academy 3 vs 1 with keeper". QMIX and QPLEX fail to perform well due to the sparse rewards. EMC applies a curiosity-based exploration strategy to QPLEX but results in a decrease in model performance. In contrast, our COIN approach combines influence-based exploration and curiosity-based exploration, outperforming all baselines with an improvement of nearly 0.2 mean scores per episode.
Figure 5: Experiment results on GRF benchmark in academy 3 vs 1 with keeper.
5.2 Ablations and Interesting Findings (Q2, Q3)
(a) academy 3 vs 1 with keeper
(b) 3s5z_vs_3s6z
(c) Corridor
Figure 6: Ablation study of our COIN on GRF and SMAC benchmark.
To understand how influence-based exploration and curiosity-based exploration work in different scenarios, we conducted ablation studies. We selected three representative scenarios to showcase these results, as depicted in Fig. 6. The dotted line represents applying only the influence-based intrinsic reward on QMIX, while the dashed line represents applying only the curiosity-based intrinsic reward. Interestingly, we found that different exploration strategies play different roles in different scenarios. When only influence-based or curiosity-based exploration was conducted in the academy  $3 \text{ vs } 1$  with keeper scenario, the model did not benefit from the exploration strategy. Instead, the performance decreased by nearly 0.5 mean scores. To understand this finding, we consulted literature and watched replays, and find that this phenomenon arises from the parameter sharing technique. According to Li et al. [14], shared parameters among agents can lead to similar behaviors, hindering the model's ability to learn successful policies in challenging tasks. Our influence-based intrinsic reward promotes agents to influence each other, bringing diversity to their behaviors. However, focusing solely on influence
among agents neglects crucial patterns, such as cutting to attract the enemy defender, resulting in poor outcomes. Fortunately, our proposed method succeeded by striking a balance between influence and curiosity. This interesting result also explains why EMC decreases the baseline performance in this scenario, as shown in Fig. 5. In the scenario  $3s5z\_ vs\_ 3s6z$ , influence-based exploration plays a significant role, while curiosity-based exploration harms the performance. This is because the learned winning strategy involves one of the allied units kiting several enemies to the corner of the map and sacrificing itself, thus not requiring much exposure to unknown states. In the corridor scenario, both exploration strategies improve the baseline performance significantly. However, curiosity-based exploration achieves even greater improvement and reaches the best performance.
In all scenarios, the best performance is achieved when both kinds of exploration strategies are incorporated. Therefore, we conclude that different scenarios require different exploration strategies, and when we are unsure which one will be effective, incorporating both is often a good choice. For more experimental results, please refer to the Appendix.
5.3 The Chosen hyperparameters
In Section 4, we introduced several hyperparameters for computing intrinsic rewards. To ensure stability and guarantee that the influence-based intrinsic reward is larger than zero, we set  $\beta_{s}$  in Eq. (5) to 0.5 for all scenarios in all benchmarks. Additionally,  $\beta_{inf}$  and  $\beta_{cur}$  are scaling factors used to adapt the two types of intrinsic rewards to the environment. Given that the importance of the two types of exploration may vary at different time steps, determining which one will dominate is challenging. To address this, we combine the two types of intrinsic rewards in a straightforward manner. When designing the hyperparameters, we follow two principles: (i) ensuring that both types of intrinsic rewards are on the same order of magnitude, and (ii) scaling the intrinsic rewards to be approximately 100 to 1000 times lower than the maximum extrinsic reward at the beginning of the training stage. Details of all the hyperparameters are listed in Appendix.
In this paper, we explore an interesting phenomenon where different types of exploration play varying roles in different scenarios of MARL. However, most current studies tend to focus on one type of exploration and overlook the other. In light of this, we propose a framework called COIN that effectively combines and complements these two types of exploration strategies. Firstly, COIN measures the influence of each agent on the others using mutual information theory, and assigns it as intrinsic rewards to individual value functions. Additionally, COIN calculates curiosity-based intrinsic rewards by utilizing prediction errors, which are then added to the extrinsic reward. By integrating these two types of intrinsic rewards, COIN creates a novel framework where they complement each other, ultimately leading to more efficient and effective exploration in cooperative MARL tasks. Both of these types of intrinsic rewards are computationally efficient within our designed framework, thus enabling effective exploration. Our experiment results on various benchmarks demonstrate the superiority of our proposed method.
Limitations and future work. Though the incorporation of two kinds of exploration is useful, our method relies on scaling hyperparameters and cannot automatically weigh the two kinds of strategies. In future work, we will investigate how to discover the causal relations among agents. With these potential interdependencies, we can assign appropriate weights to each agent and avoid computing the MI for unnecessary agent pairs.
This work was supported by the National Key Research & Development Project of China (2021ZD0110700), the National Natural Science Foundation of China (62006207, 62037001, U19B2043, 61976185), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010), and the Fundamental Research Funds for the Central Universities (226-2023-00048, 226-2022-00051, 226-2022-00142). Baoxiang Wang is partially supported by The National Natural Science Foundation of China (62106213 and 72150002) and Shenzhen Science and Technology Program (RCBS20210609104356063 and JCYJ20210324120011032).
[35] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. 2021.
[36] Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. arXiv preprint arXiv:2106.02886, 2021.
[37] Ermo Wei and Sean Luke. Lenient learning in independent-learner stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914–2955, 2016.
[38] Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. Hierarchically and cooperatively learning traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 669-677, 2021.
[39] Huanhuan Yang, Dianxi Shi, Chenran Zhao, Guojun Xie, and Shaowu Yang. Ciexplore: Curiosity and influence-based exploration in multi-agent cooperative scenarios with sparse rewards. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2321-2330, 2021.
[40] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. sensors, 15(5):10026-10047, 2015.
[41] Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
A.1 Details of the benchmarks
Multi-agent coordination challenge benchmark. Wang et al. [36] collect six coordination tasks from various multi-agent learning literature to create the Multi-Agent Coordination Challenge (MACO) benchmark. For our experiments, we have selected three tasks from this benchmark.
The Aloha coordination task [8, 18] consists of a  $2 \times 5$  array with 10 agents. The objective is to send messages and maximize the backlog, which has a maximum value of 5. However, if two adjacent agents send messages simultaneously, a collision occurs, and the messages need to be resent. For each successful transmission, a positive global reward of 0.1 is gained, while a collision results in a negative global reward of -10. Each unit initially has 1 backlog, and at each time step, there is a probability of 0.6 for a new packet to arrive, provided that the maximum backlog has not been reached yet. In Gather [37], there are 5 agents whose objective is to collectively choose the same action in order to achieve common goals. These goals are established at the beginning of each episode. If all agents successfully choose the goal, the entire team receives a reward of 10. On the other hand, if none of the agents chooses the goal, a reward of 5 is given to the team. Hallway [34] involves 12 agents organized into multiple groups. Each agent is randomly assigned to a state within each chain. In this task, if agents belonging to the same group reach the same state at the same time, they receive a global reward. However, if more than one group reaches the same state, a global punishment is given.
StarCraft II micro management benchmark. The StarCraft II micromanagement challenge [29] involves agents controlling allied armies and collaborating to engage with enemy forces controlled by the game engine. This benchmark includes different maps that pose varying levels of challenges, categorized as "easy", "hard" and "super hard". For our experiments, we have chosen three representative super hard scenarios:  $3s5z\_ vs\_ 3s6z$ , corridor, and  $6h\_ vs\_ 8z$ . The game AI difficulty is set to "very difficult" level. Table 1 provides an introduction to these selected maps.
Table 1: Experiment maps of StarCraft II micro management benchmark.
Map Name	Ally Units	Eenemy Units	Agent Type	Difficulty
3s5z_vs_3s6z	3 Stalkers, 5 Zealots	3 Stalkers, 6 Zealots	Heterogeneous	Super hard
6h_vs_8z	6 Hydraliks	8 Zealots	Homogeneous	Super hard
corridor	6 Zealots	24 Zerglings	Homogeneous	Super hard
Google research football benchmark. In the Google Research Football (GRF) game [13], players are required to coordinate with each other to organize offensive plays and score goals. In this game, agents control the players on the left side, excluding the goalkeeper, while the players on the right side are controlled by rule-based bots using hand-coded built-in AI. Each player can observe the positions and movement directions of all players, as well as the football. They have a discrete action space that includes options for moving, sliding, shooting, and passing. Soccer players must determine the optimal timing for shooting by combining dribbling and passing techniques. In this game environment, rewards are only provided at the end of the game, with a reward of 100 for scoring a goal and a penalty of -1 for losing the game.
A.2 Experiment settings
We list the experiment settings and hyperparameters in Table 2. The two important scale factors  $\beta_{cur}$  and  $\beta_{inf}$  in MACO depends on the scenarios. We set  $\beta_{cur} = \beta_{inf} = 0.05$  in aloha,  $\beta_{cur} = \beta_{inf} = 0.01$  in gather, and  $\beta_{cur} = \beta_{inf} = 0.0005$  in hallway.
A.3 Comparison with the other exploration methods
In this paper, we employ both influence-based and curiosity-based exploration strategies. In comparison to other methods, we utilize a simple yet effective approach to implement these two types of exploration. In this subsection, we will discuss the differences among these methods.
Table 2: Hyperparameters and experiment settings
Settings	StarCraft II	MACO	GRF
Batch size	32	32	32
Training episodes	5mil	1mil(2mil for aloha)	5mil
Agent optimiser	RMSProp	RMSProp	Adam
Estimator optimiser	Adam	Adam	Adam
Agent learning rate	0.0005	0.0005	0.0005
Estimator learning rate	0.0005	0.0005	0.0005
Target central critic update interval	200 episodes	200 episodes	200 episodes
βcur	0.0005	0.05/0.01/0.0005	0.005
βinf	0.0005	0.05/0.01/0.0005	0.005
β	0.5	0.5	0.5
Mutual information and influence-based exploration. Mutual information is a commonly utilized concept in existing works for multi-agent reinforcement learning (MARL) problems. MOA [9] was the first to propose using  $MI(a_i^t; a_j^t | s^t)$  as intrinsic rewards. Another formulation called EITI [33] is given by  $MI(o_j^{t+1}; o_i^t, a_i^t | o_j^t, a_j^t)$ . Similarly, PMIC [16] proposes  $MI(a; s)$  as a collaboration criterion. Additionally, Jiang and Lu [11] propose  $MI(I; O)$ , while Li et al. [14] propose  $MI(\tau; I)$ , where  $I$  represents the agent's ID. The main differences among these methods lie in their intentions. The first three methods aim to promote influence and collaboration among agents, while the last two aim to enhance diversity among the agents.
In this paper, we also incorporate mutual information  $MI(\tau_j^{t + 1};a_i^t)$  to quantify the influence of one agent's action on another agent. While the basic motivation aligns with MOA and EITI, our proposed approach is (a) more reasonable and (b) easier to compute. The reasons for these distinctions are as follows: (a) More reasonable: Intuitively, an agent can only influence the future behavior of other agents. Therefore, we consider  $MI(a_{i}^{t};a_{j}^{t}|s^{t})$  as less reasonable. Additionally, the crucial factor that influences other agents is the action taken, rather than the current states or observations. Consequently,  $MI(\tau_j^{t + 1};a_i^t)$  is more reasonable than  $MI(o_j^{t + 1};o_i^t,o_i^t|o_j^t,o_j^t)$ . Specifically, in the computation of the  $i$ -th influence on  $j$ ,
Figure 7: Markov Decision Process in MARL.
$MI(\tau_j^{t + 1};a_i^t) = \log \frac{p(\tau_j^{t + 1}|a_i^t)}{p(\tau_j^{t + 1})}$ , whereas  $MI(o_{j}^{t + 1};o_{i}^{t},a_{i}^{t}|o_{j}^{t},a_{j}^{t}) = \log \frac{p(o_{j}^{t + 1}|o_{i}^{t},a_{i}^{t},o_{j}^{t},a_{j}^{t})}{p(o_{j}^{t + 1}|o_{j}^{t},a_{j}^{t})}$ . Referencing the Markov decision process in Fig. 7, in  $p(o_j^{t + 1}|o_i^t,o_i^t,o_j^t,o_j^t)$ ,  $o_i^t$  becomes the noise factor in predicting  $o_j^{t + 1}$ , which can potentially hinder accurate influence estimation in certain scenarios. For instance, let the optimal policy  $\pi (a_i^t |o_i^t,s_i^t)$  be an and operation, and let the observation function  $g(o_j^t |\cdot)$  be an element-wise  $x$  or operation for all inputs.  $o_j^{t + 1}$  should be 0 when  $o_i^t = 1$  and  $s^t = 0$ , but introducing  $o_i^t = 1$  into  $g$  would result in  $o_j^{t + 1} = 1$ . (b) Easier to compute: We have designed a framework where the intrinsic reward at each time step can be computed in a single forward propagation. This approach is straightforward to implement and does not impose a significant computational burden.
Prediction network and curiosity-based exploration. Curiosity-based exploration methods, widely employed in reinforcement learning [22, 28], typically involve designing intrinsic rewards to measure the degree of unfamiliarity associated with states. A common approach entails setting up one or several networks to predict information about the next time-step, with the instability of these predictions serving as the intrinsic reward. For instance, Pathak et al. [24] and Burda et al. [4] consider the prediction error of target states as intrinsic rewards. Bai et al. [1] use the standard deviation of target Q-function predictions as the intrinsic reward. Zheng et al. [41] utilize the prediction errors of individual Q-functions as intrinsic rewards in multi-agent reinforcement learning (MARL) problems. Similarly, Yang et al. [39] design intrinsic rewards based on the variance of an ensemble predictive model incorporating local observations and global states. In our approach, we adopt a similar strategy to the aforementioned methods. We use the prediction errors of next
states and observations as intrinsic rewards. Since the observations and global states are provided by the environment, we consider them to be more reliable supervision information compared to value functions. Moreover, these prediction errors can capture the unfamiliarity from two perspectives: local observations for individual unfamiliarity and global states for joint unfamiliarity. Consequently, this approach is straightforward yet effective in MARL settings.
A.4 Two heads are better than one