SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiahui Li $^{1}$ , Kun Kuang $^{1*}$ , Baoxiang Wang $^{2,3}$ , Xingchen Li $^{1}$ , Fei Wu $^{1}$ , Jun Xiao $^{1}$ , Long Chen $^{4}$
<sup>1</sup>Zhejiang University, <sup>2</sup>The Chinese University of Hong Kong, Shenzhen
3Shenzhen Institute of Artificial Intelligence and Robotics for Society, 4HKUST
{jiahuil,kunkuang}@zju.edu.cn,bxiangwang@cuhk.edu.cn,
xingchenl@zju.edu.cn, {wufei,junx}@cs.zju.edu.cn, longchen@ust.hk
Exploration strategy plays an important role in reinforcement learning, especially in sparse-reward tasks. In cooperative multi-agent reinforcement learning (MARL), designing a suitable exploration strategy is much more challenging due to the large state space and the complex interaction among agents. Currently, mainstream exploration methods in MARL either contribute to exploring the unfamiliar states which are large and sparse, or measuring the interaction among agents with high computational costs. We found an interesting phenomenon that different kinds of exploration plays a different role in different MARL scenarios, and choosing a suitable one is often more effective than designing an exquisite algorithm. In this paper, we propose a exploration method that incorporate the CuriOsity-based and INfluence-based exploration (COIN) which is simple but effective in various situations. First, COIN measures the influence of each agent on the other agents based on mutual information theory and designs it as intrinsic rewards which are applied to each individual value function. Moreover, COIN computes the curiosity-based intrinsic rewards via prediction errors which are added to the extrinsic reward. For integrating the two kinds of intrinsic rewards, COIN utilizes a novel framework in which they complement each other and lead to a sufficient and effective exploration on cooperative MARL tasks. We perform extensive experiments on different challenging benchmarks, and results across different scenarios show the superiority of our method.
Multi-Agent Reinforcement Learning (MARL) has been widely used to solve numerous complex real-world problems, such as traffic control [38], autonomous robotics control [17, 26], game AI [25], and network routing [40], etc.. Compared with single-agent reinforcement learning, the joint state and action space grows exponentially with the number of agents. To overcome this challenge, the Centralized Training with Decentralized Execution (CTDE) [20, 12] paradigm is proposed, where the joint value function is estimated in the central critic during training, and the local agents execute actions according to their own policies.
Based on the CTDE paradigm, numerous deep MARL methods have been proposed [27, 32, 35, 15]. The most commonly used method for exploration is  $\epsilon$ -greedy. However, these methods may perform poorly in complex scenarios [13, 29] with a large number of agents and a vast action space. The main issue is that the  $\epsilon$ -greedy exploration strategy requires a significant number of attempts to become
familiar with the environment, potentially leading to sub-optimal results. Therefore, an effective exploration strategy is crucial.
Currently, exploration strategies in MARL can be broadly classified into two main categories. The first category includes curiosity-based exploration methods [41, 6]. These methods focus on quantifying the unfamiliarity of incoming states and encourage agents to explore a wide range of states that they have not or rarely experienced. The second category comprises influence-based exploration methods [9, 33, 39]. These methods aim to promote agents to adopt actions that can influence other agents and achieve better coordination. However, mainstream methods often prioritize the design of sophisticated algorithms within one specific category, which may not perform well in certain scenarios. On the one hand, the joint state space can be excessively large, presenting a challenge for curiosity-based methods to enable agents to experience all unseen states effectively. On the other hand, meaningful interactions among agents can be infrequent, making it difficult for influence-based methods to identify and optimize these interactions. Fortunately, we have discovered that these two types of methods can complement each other, and even a simple combination of them often leads to satisfactory results.
In this paper, we propose an effective and compute-efficient exploration method named COIN, which incorporates CuriOsity-based and INfluence-based exploration. To be specific, we define the influence of one agent on another as the mutual information between its actions and the trajectory of its peer in the next time-step. Our derivation allows us to reformulate the problem of maximizing mutual information into an optimization problem with a tractable lower bound by introducing a variational posterior estimator. This formulation enables us to quantify the impact of each agent's action on the others and use the influence degree as an intrinsic reward to promote exploration. The influence-based intrinsic reward is added to the individual value functions before being sent to the central mixer. Furthermore, this influence-based exploration strategy is compatible with curiosity-based methods, which encourage the agents to explore unfamiliar states. To reduce the exploration state space, we propose an additional method that effectively utilizes the prediction errors of local observations of each agent and the global states to measure the model's uncertainty towards the environment. This uncertainty can be used as an intrinsic reward, which can be added to the extrinsic global reward provided by the environment. Moreover, based on these two types of intrinsic rewards, we have designed a Multi-Agent Reinforcement Learning (MARL) framework where they complement each other, resulting in thorough and efficient exploration without incurring significant computational costs.
Our main contribution can be summarized as follows: (i) We propose a straightforward exploration method based on mutual information that enables the quantification of each agent's influence on the others. (ii) We introduce a simple yet effective approach to measure the model's unfamiliarity with future states, resulting in curiosity-driven exploration. (iii) To achieve effective exploration in MARL settings, we design a framework in which the two kinds of methods complement each other and act as intrinsic rewards which are compute-efficient. (iv) The experiments on three benchmarks StarCraft II [29], MACO [34], and Google Football [13] show the superiority and effectiveness of our COIN.
Curiosity-based Exploration Methods. Exploration algorithms focus on encouraging agents to visit a wide range of different states within the environment to avoid getting trapped in sub-optimal results. Among the various branches of exploration methods, curiosity-based approaches [22, 28] have gained significant popularity. One common approach involves designing intrinsic rewards that measure the degree of unfamiliarity of states. Some researchers utilize pseudo-state counts [2, 21, 30] to calculate such rewards, while others [24, 4]. develop models to predict target states and employ the prediction errors as rewards. For instance, Bai et al. [1] propose the utilization of multiple models and compute intrinsic rewards based on the standard deviation of predictions, especially in complex environments. Zheng et al. [41] extends the application of curiosity-based exploration to MARL settings and employs the prediction errors of individual Q-functions as intrinsic rewards. Similarly, Yang et al. [39] designs intrinsic rewards by considering the variance of an ensemble predicting model that incorporates both local observations and global states.
Influence-based Methods in MARL. The influence-based exploration strategy is specifically designed for MARL problems, where the focus is on the complex coordination and interactions among
agents. In this context, mutual information serves as an effective and commonly used tool. Jaques et al. [9] introduces the concept of "social influence", represented as  $MI(a_{i}^{t};a_{j}^{t}|s^{t})$ , to quantify how an agent's action can influence the action of another agent. This measure captures the interplay and dependencies between agents, providing valuable insights into their collaborative behavior. Similarly, Wang et al. [33] draws inspiration from counterfactual thinking and proposes two exploration techniques: EITI and EDTI. Both of them aim to encourage agents to influence each other's actions. In EITI, mutual information  $MI(o_{j}^{t + 1};o_{i}^{t},a_{i}^{t}|o_{j}^{t},a_{j}^{t})$  is also utilized as a formulation to incentivize agents to exert influence on one another, fostering cooperative exploration and coordination.
These influence-based exploration strategies leverage mutual information as a means to capture the interdependencies among agents and promote collaborative exploration in MARL settings. In this paper, we also propose an influence-based intrinsic reward  $MI(\tau_j^{t + 1};a_i^t)$  based on mutual information which is simple but efficient. The discussions of the superiority of our method compared with the others are left in Appendix.
Hybrid Exploration Methods in MARL. The two exploration strategies mentioned above focus on different aspects, with many studies emphasizing one while neglecting the other. One hybrid method called CIExplore [39] incorporates both strategies by designing them as intrinsic rewards, which is the closest approach to our work. However, CIExplore has shown to be unstable and ineffective in complex MARL scenarios.
In our approach, COIN, we not only ensure compute efficiency in the designed framework for calculating intrinsic rewards, but we also leverage the complementary nature of the two kinds of rewards to achieve sufficient and effective exploration, ultimately enhancing the model capabilities in complex MARL scenarios.
Dec-POMDPs. The exploration method we discussed in this paper focuses on the settings of decentralized partially observable Markov decision process (Dec-POMDP) [19, 3, 5, 7, 23], which can always be described as a tuple:
$$
G = <   \mathcal {N}, \mathcal {S}, \mathcal {A}, \mathcal {P}, r, \mathcal {O}, \Omega , \gamma >,
$$
where  $\mathcal{N}$  represents the set of agents with  $|\mathcal{N}| = N$ . At each time-step, each agent  $i \in \mathcal{N}$  selects and executes an action  $a_i \in \mathcal{A}$  to compose a joint action  $\pmb{a} \in \mathcal{A}^N$ . The environment produces the global state  $s \in S$  according to the transition function  $\mathcal{P}(s'|s, \pmb{a}) : S \times \mathcal{A}^N \to S$ . Meanwhile, the environment provides a global reward shared by all agents according to the function  $r(s, \pmb{a}) : S \times \mathcal{A}^N \to \mathbb{R}$ . Moreover, each agent gains the partial observation  $o \in \mathcal{O}$  according to the observation function  $\Omega(s, i)$  and learns its local policy  $\pi^i(a_i | \tau_i) : \mathcal{T} \times \mathcal{A} \to [0, 1]$  conditions on the its local trajectory  $\tau_i \in \mathcal{T}$ . The goal of all agents is to maximize the discounted cumulative return  $\sum_{i=0}^{\infty} \gamma^i r_i$ , where  $\gamma$  is a discount factor.
In this section, we introduce two simple yet effective intrinsic rewards for influence and curiosity-based exploration. Following that, we present a framework, as depicted in Fig. 1, that integrates these two strategies to facilitate exploration. Finally, we provide a brief overview of the loss function and offer implementation details, ensuring both effectiveness and computational efficiency in the exploration process.
4.1 Influence-based Intrinsic Reward
Compared with single-agent reinforcement learning, complex coordination plays an important role in MARL. As a result, it is essential to incentivize agents to explore states or execute actions that can influence each other. Mutual information [9, 14, 33] is a commonly used tool for quantifying the influence between pairwise agents. Following previous works, we define the influence of the  $i$ -th agent on the other agents at time-step  $\tilde{t}$  as the sum of the mutual information between the trajectory
Figure 1: The training framework of COIN. Two additional estimators are set up for computing the intrinsic rewards, where the local observation-action history of all agents are the inputs.
of the other agents and the action taken by the  $i$ -th agent:
$$
\sum_ {k \in \mathcal {N} \backslash i} M I (\tau_ {k} ^ {\tilde {t} + 1}; a _ {i} ^ {\tilde {t}}) = \sum_ {k \in \mathcal {N} \backslash i} H (\tau_ {k} ^ {\tilde {t} + 1}) - H (\tau_ {k} ^ {\tilde {t} + 1} | a _ {i} ^ {\tilde {t} + 1}) = \mathbb {E} \sum_ {k \in \mathcal {N} \backslash i} [ \log \frac {p (\tau_ {k} ^ {\tilde {t} + 1} | a _ {i} ^ {\tilde {t}})}{p (\tau_ {k} ^ {\tilde {t} + 1})} ],
$$
where  $\tau_k^{\tilde{t} +1}$  represents the trajectory of the  $k$ -th agent at time-step  $\tilde{t} +1$ , and  $H(\cdot)$  represents the entropy term.
If we want to promote the influence among agents, we need to maximize  $\sum_{k\in \mathcal{N}\setminus i}MI(\tau_k^{\tilde{t} +1};a_i^{\tilde{t}})$ . Meanwhile, we can represent  $p(\tau_k^{\tilde{t} +1})$  and  $p(\tau_k^{\tilde{t} +1}|a_i^{\tilde{t}})$  as the following equations:
$$
p \left(\tau_ {k} ^ {\tilde {t} + 1}\right) = p \left(o _ {k} ^ {0}\right) \prod_ {t = 0} ^ {\tilde {t}} p \left(a _ {k} ^ {t} \mid \tau_ {k} ^ {t}\right) p \left(o _ {k} ^ {t + 1} \mid \tau_ {k} ^ {t}, a _ {k} ^ {t}\right), \tag {1}
$$
$$
p \left(\tau_ {k} ^ {\tilde {t} + 1} \mid a _ {i} ^ {\tilde {t}}\right) = p \left(o _ {k} ^ {0} \mid a _ {i} ^ {\tilde {t}}\right) \prod_ {t = 0} ^ {\tilde {t}} p \left(a _ {k} ^ {t} \mid \tau_ {k} ^ {t}, a _ {i} ^ {\tilde {t}}\right) p \left(o _ {k} ^ {t + 1} \mid \tau_ {k} ^ {t}, a _ {k} ^ {t}, a _ {i} ^ {\tilde {t}}\right). \tag {2}
$$
Therefore, the mutual information can be rewritten as:
$$
\sum_ {k \in \mathcal {N} \backslash i} M I \left(\tau_ {k} ^ {\tilde {t} + 1}; a _ {i} ^ {\tilde {t}}\right) = \mathbb {E} [ \sum_ {k \in \mathcal {N} \backslash i} \{\underbrace {\log \frac {p \left(o _ {k} ^ {0} \mid a _ {i} ^ {\tilde {t}}\right)}{p \left(o _ {k} ^ {0}\right)}} _ {①} + \underbrace {\sum_ {t = 0} ^ {\tilde {t}} \log \frac {p \left(a _ {k} ^ {t} \mid \tau_ {k} ^ {t} , a _ {i} ^ {\tilde {t}}\right)}{p \left(a _ {k} ^ {t} \mid \tau_ {k} ^ {t}\right)}} _ {②} + \underbrace {\sum_ {t = 0} ^ {\tilde {t}} \log \frac {p \left(o _ {k} ^ {t + 1} \mid \tau_ {k} ^ {t} , a _ {k} ^ {t} , a _ {i} ^ {\tilde {t}}\right)}{p \left(o _ {k} ^ {t + 1} \mid \tau_ {k} ^ {t} , a _ {k} ^ {t}\right)}} _ {③} \}. \tag {3}
$$
We can see that Eq. (3) is divided into three terms. Term ① is determined by the environment, and hence can be ignored. Term ② computes the information difference of the  $k$ -th agent's action selection when the action at time-step  $\tilde{t}$  of the  $i$ -th agent is given. Notice the fact that the action of the  $i$ -th agent at a particular time can not influence any actions, observations, or trajectories of the other agents at the previous time. Thus, the term ② can also be ignored. Term ③ measures the changes in the target observation of the  $k$ -th agent. This term can be simplified as  $\log \frac{p(o_k^{\tilde{t} + 1}|\tau_k^{\tilde{t}},a_k^{\tilde{t}},a_i^{\tilde{t}})}{p(o_k^{\tilde{t} + 1}|\tau_k^{\tilde{t}},a_k^{\tilde{t}})}$
Therefore, we conclude that optimizing the mutual information at Eq. (1) is equivalent to optimize:
$$
\mathbb {E} [ \sum_ {k \in \mathcal {N} \backslash i} \log \frac {p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a _ {k} ^ {\tilde {t}} , a _ {i} ^ {\tilde {t}}\right)}{p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a _ {k} ^ {\tilde {t}}\right)} ].
$$
According to the methods of variational inference [31, 14], we can optimize a tractable evidence lower bound of the above expectation by introducing a variational posterior estimator. Hence, we have the following equation:
$$
\mathbb {E} \left[ \log \frac {p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a _ {k} ^ {\tilde {t}} , a _ {i} ^ {\tilde {t}}\right)}{p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a _ {k} ^ {\tilde {t}}\right)} \right] \geq \mathbb {E} \left[ \log \frac {q _ {\phi} \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a _ {k} ^ {\tilde {t}} , a _ {i} ^ {\tilde {t}}\right)}{p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}} , a ^ {\tilde {t}} _ {k}\right)} \right], \tag {4}
$$
where  $q_{\phi}$  represents the variational posterior estimator parameterized by  $\phi$ , and we omit the summation notation for convenience.
For encouraging the interaction among agents and promoting exploration, we design an influence-based intrinsic reward for each agent. The intrinsic reward for agent  $i$  at time-step  $\tilde{t}$  is computed as:
$$
r _ {i n f _ {i}} ^ {\tilde {t}} = \sum_ {k \in \mathcal {N} \backslash i} \left[ \beta_ {s} \log q _ {\phi} \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}}, a _ {k} ^ {\tilde {t}}, a _ {i} ^ {\tilde {t}}\right) - \log p \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}}, a _ {k} ^ {\tilde {t}}\right) \right], \tag {5}
$$
where  $\beta_{s}$  is a scaling factor to guarantee  $r_{inf_i}^t >0$
4.2 Curiosity-based Intrinsic Reward
Curiosity-based exploration is a widely adopted approach in single-agent reinforcement learning, with the objective of encouraging agents to explore unfamiliar states. Typically, this is achieved by computing intrinsic rewards based on prediction errors [24, 4] or variances [1] of future states, rewards, or value functions. In the context of MARL, curiosity-based exploration remains essential. However, the prediction of joint states becomes unreliable due to the exponential increase in state space with the number of agents, and the scarcity of meaningful states. Fortunately, the local observations of each agent can serve as a measure of curiosity. For simplicity, we use the prediction error of both the global state and the local observations to calculate the curiosity-based intrinsic reward in the partially observable environment as:
$$
r _ {c u r} ^ {\tilde {t}} = \frac {1}{| \mathcal {N} |} \sum_ {i = 1} ^ {\mathcal {N}} \left(o _ {i} ^ {\tilde {t} + 1} - f _ {o _ {i}} \left(\boldsymbol {\tau} ^ {\tilde {t}}, \boldsymbol {a} ^ {\tilde {t}}\right)\right) ^ {2} + \left(s ^ {\tilde {t} + 1} - f _ {s} \left(\boldsymbol {\tau} ^ {\tilde {t}}, \boldsymbol {a} ^ {\tilde {t}}\right)\right) ^ {2}, \tag {6}
$$
where  $f_{o_i}(\pmb{\tau}^{\tilde{t}}, \pmb{a}^{\tilde{t}})$  and  $f_s(\pmb{\tau}^{\tilde{t}}, \pmb{a}^{\tilde{t}})$  represent the predicting network to estimate the observation of each agent and global state at time-step  $\tilde{t} + 1$ , and  $\pmb{a}^{\tilde{t}}, \pmb{\tau}^{\tilde{t}}$  represent the joint action, and trajectories of all agents at time-step  $\tilde{t}$ .
4.3 Implementation of the Estimators
Our goal is to develop two effective estimators for computing intrinsic rewards as well as make them compute-efficient. For the influence estimator, we model  $q_{\phi}(o_k^{\tilde{t} +1}|\tau_k^{\tilde{t}},a_k^{\tilde{t}},a_i^{\tilde{t}})$  by a variational posterior estimator, which is updated via loss:
$$
L _ {\phi} = \sum_ {i \in \mathcal {N}} \sum_ {k \in \mathcal {N} \backslash i} \left(o _ {k} ^ {\tilde {t} + 1} - q _ {\phi} \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}}, a _ {k} ^ {\tilde {t}}, a _ {i} ^ {\tilde {t}}\right)\right) ^ {2}. \tag {7}
$$
Meanwhile, the  $p(o_k^{\tilde{t} + 1} | \tau_k^{\tilde{t}}, a_k^{\tilde{t}})$  is approximated by a prediction network  $p_\eta$  updated via loss:
$$
L _ {\eta} = \sum_ {i \in \mathcal {N}} \sum_ {k \in \mathcal {N} \backslash i} \left(o _ {k} ^ {\tilde {t} + 1} - p _ {\eta} \left(o _ {k} ^ {\tilde {t} + 1} \mid \tau_ {k} ^ {\tilde {t}}, a _ {k} ^ {\tilde {t}}\right)\right) ^ {2}. \tag {8}
$$
For the curiosity estimator with parameters  $\varphi$ , we update it via loss:
$$
L _ {\varphi} = \frac {1}{| \mathcal {N} |} \sum_ {i = 1} ^ {\mathcal {N}} \left(o _ {i} ^ {\tilde {t} + 1} - f _ {o _ {i}} \left(\boldsymbol {\tau} ^ {\tilde {t}}, \boldsymbol {a} ^ {\tilde {t}}\right)\right) ^ {2} + \left(s ^ {\tilde {t} + 1} - f _ {s} \left(\boldsymbol {\tau} ^ {\tilde {t}}, \boldsymbol {a} ^ {\tilde {t}}\right)\right) ^ {2}. \tag {9}
$$
Notice that the summation symbol can become an obstacle when computing intrinsic rewards and losses efficiently. In order to accelerate the training procedure, we have implemented the two estimators in a compute-efficient manner, as illustrated in Fig. 2.
In the influence estimator, Eq. (7) requires us to propagate  $q_{\phi}$  for  $N - 1$  iterations in order to obtain the intrinsic reward for a single agent. To improve efficiency, we have designed a batch implementation for  $\sum_{k \in \mathcal{N} \setminus i}$ , allowing us to estimate the observations of all other agents simultaneously in a single forward propagation. Likewise,  $p(\cdot | \tau_k, a_k)$  also adopts the same trick.
As for the curiosity estimator, all  $f_{o_i}$  have the same inputs. To streamline this process, we implement them using a single neural network that outputs a concatenated tensor.
Figure 2: Implementations of the two estimators for computing the summation symbol in a single forward propagation.
To further alleviate the computational burden, we adopt
the parameter-sharing technique, and all agents share the same network in the influence estimator. Similarly, for curiosity-based intrinsic reward,  $f_{s}$  and  $f_{o}$  share the same encoder  $g$  to get the representations. Inspired by the approach presented in [15], we represent the trajectory of the agent using the hidden state of a recurrent neural network. Consequently, we have  $h_i^t = \tau_i^t$  and  $\pmb{h}^{\tilde{t}} = \pmb{\tau}^{\tilde{t}}$ . Meanwhile, for the learning more stable, we compute the intrinsic rewards in Eq. (5) and Eq. (6) via target networks  $\phi', \eta'$  and  $\varphi'$  which are copied from  $\phi, \eta$  and  $\varphi$  periodically.
4.4 Incorporation of the Two Heads
We would like to emphasize the effectiveness of combining these two methods in a straightforward manner, rather than designing a intricate algorithm that solely focuses on one aspect. There are two reasons for this: (i) Curiosity-based intrinsic rewards are designed to encourage exploration of unfamiliar states, while influence-based intrinsic rewards aim to promote coordination among agents. These rewards operate on different dimensions and play distinct roles in various scenarios. (ii) Either of the methods may not work in certain training time-steps, as no method can guarantee constant positive influence among agents, and the meaningful unfamiliar joint state is too sparse. Some recent works [33, 39] attempt to use both kinds of methods, however, concerns regarding their effectiveness and computational efficiency remain which lead the unsatisfactory results in complicated scenarios.
In this subsection, we will introduce how to incorporate them under the paradigm of CTDE. For simplicity, we omit the superscript of time-step in this subsection. For the influence-based intrinsic rewards, we apply it to the individual local value functions of each agent to promote the interaction. For  $i$ -th agent it is computed as:
$$
\tilde {Q} _ {i} = Q _ {i} - \beta_ {i n f} * r _ {i n f _ {i}}, \tag {10}
$$
where  $Q_{i}$  is the local value function, and  $\beta_{inf}$  is the scale factor.  $\tilde{Q}_i$  will then be sent to the central critic Mixer to get the joint value function:
$$
\tilde {Q _ {t o t}} = \operatorname {M i x e r} \left(\tilde {Q} _ {1}, \dots , \tilde {Q} _ {N}\right). \tag {11}
$$
Notice that the intrinsic rewards are incorporated before the value functions are sent to the central mixer. This is done to ensure that the intrinsic rewards can be scaled by the network appropriately. Meanwhile, it is worth mentioning that the intrinsic rewards are not applied to the target network. The reasons for this decision will be discussed in the Appendix.