Efficient Exploration in Multi-Agent Reinforcement Learning via Farsighted Self-Direction
Authors: Tiancheng Lao
Xudong Guo
Mengge Liu
Junjie Yu
Yi Liu
Wenhui Fan
Venue: Accepted by TMLR ()

Abstract: Multi-agent reinforcement learning faces greater challenges with efficient exploration compared to single-agent counterparts, primarily due to the exponential growth in state and action spaces. Methods based on intrinsic rewards have been proven to enhance exploration efficiency in multi-agent scenarios effectively. However, these methods are plagued by instability during training and biases in exploration direction. To address these challenges, we propose Farsighted Self-Direction (FSD), a novel model-free method that utilizes a long-term exploration bonus to achieve coordinated exploration. Since prediction error against individual Q-values indicates a potential bonus for committed exploration, it is taken into account in action selection to directly guide the coordinated exploration. Further, we also use clipped double Q-learning to reduce noise in prediction error. We validate the method on didactic examples and demonstrate the outperformance of our method on challenging StarCraft II micromanagement tasks.

Published in Transactions on Machine Learning Research (04/2025)
Efficient Exploration in Multi-Agent Reinforcement Learning via Farsighted Self-Direse
Tiancheng Lao Department of Automation Tsinghua Universily, Beijing, China
mr.229maill.tsinghuua.o
Xudong Guo
amd200maii.ttiinghheee cdd...r
Department of Automation Tsinghua University, Beijing, China
Mengge Liu
aong222@mails.tsininan edu.cr
Department of Automation Tsinghua University, Beijing, China
Junjie Yu Department Automation Tsinghua Universily, Beijing, Chvna
Mij280mail..siinghuua
Yi Liu
yilku Wttiinguusaaa...m edu..cr
Department of Automation
Tsinghua University, Beijing, China
Wenhui Fan
Janwentui@Ussinghuee edu. cn
Department of Automation
Tsinghua University, Beijing, China
Reviewed on OpenReview: https: // openreview. net/ forum? id= NUV8THrLZC
Abstract
Multi-agent reinforcement learning faces greater challenges with efficient exploration com- pared to single-agent counterparts, primarily due to the exponential growth state and action spaces. Methods basod on intrinsic rewards have been proven to enhance exploration efficiency in multi-agent scenarios effectively. However, these methods are plagued by in- stability during training and biases in exploration direction. To address these challenges, propose Farsighted Self-Direction (FSD), novel model-free method that utilizes long- term exploration bonus to achieve coordinated exploration. Since prediction error against individual Q-values indicates a potential bonus for commit exploration, it is taken into ac- count in action selection to directly guide the coordinated exploration. Further, we also use clipped double Q-learning to reduce noise the prediction error. We validate the method on didactic examples and demonstrate the outperformance of our method on challenging StarCraft TT micromanagement tasks.
1 Introduction
Currently, multi-agent reinforcement learning (MARL) has been applied in multiple fields, such as au- tonomous driving (Wu et al., 2019) and traffic control Zhang et al., 2019). In the field games, where ample training data more readily available, it has demonstrated top-level performance in StarCraft II (Vinyals al. 2019). Quake III Arena (Jaderberg et al. 2019), and Dota 2 (Berner al. 2019). To facilitate the practical implementation of MARL, ralizal training with
powerful potential, reaching human

---

Published in Transactions on Machine Learning Research (04/2025)
decentralized execution (CTDE) has been widely adopted MARL paradigm. It leverages global infor- mation during training, alleviating the non-stationarity problem (Gronauer & Diepold 2022 while agents only rely on local observation history during execution, enabling distributed deployment. Under the CTDE framework, value decomposition methods have been continuously explored and refined (Sunchag et al.) 2018
Rashid et al. 2018 Wang et al. 2021b Xu el al. 2023), demonstrating excellent performance in nurnerous tasks.
Despite the current success, it has been found that with the simple e-greedy exploration strategy, these methods are sample-ineffiiint to solve complex tasks, especially those that require coordinated and persistent exploration (Wang et al.) 2021b; Zheng et al.) 2021). In the single-agent domain, there have been plenty of extensive studies, including uncertainty-orientel methods (Janz et al.) 2019 Lee et al. 2021) and intrinsic
motivation-orien oriented methods (Mohamed & Rezende 2015 Pathak et al. 2017 Fox et al. 2018 Burda et al,
2019 Badia et al. ) p Unfortunately, these methods cannot be directly applied to multi-agent systems due to the exponential growth of state and action spaces, partial observability, and the requirement for coordinated exploration Zheng al. 2021 Jiang et. al.) 2024).
To address the limitation, the intrinsic reward is designed as the prediction error of the individual Q- values (Zheng et al. 2021 Zhang& & Yu 2023) for value decomposition methods, promoting efficient coor- dinated exploration. To avoid excessive coupling between intrinsic reward generation and policy learning, these curiosity-driven methods typically employ a wewpolee-eepplor--reeeeeiii
structure where the controller learns the policy used to interact with the enviromment and the explorer is only used during training to gen- erate the intrinsic reward. The prediction error between the explorer and the predictor is used as the intrinsic reward for the controller to enhance exploration. Despite the impressive performance in the experiments, they still have two main drawbacks: (1) instability during training. The intrinsic reward varies as the train- ing progresses; thereby, agents face non-stationary environmental dynamics during training, which increases the risk of training instability. (2) biases in exploration direction. The explorer learns solely using extrinsic rewards, while the controller uses extrinsic and intrinsic rewards. Within such a framework, the strategies of the controller and the explorer inevitably differ, and using the intrinsic rewards generated by the explorer to guide the controller can lead to biased, even wrong, exploration direction. Just as when we seek advice from others, their suggestions often do not suit us best due to differences in values other factors.
Thereby, propose a novel multi-agent exploration framework named Farsighted Sell-Direction (FSD). The first distinction from previous curiosity- methods is that we forgo the explorer. The framework contains only two network components: the controller Q and the predictor Q. We use the prediction error against the controller's individual Q-values by the predictor Q to measure the potential exploration bonus. By eliminating the intermediary explorer, FSD avoids the exploration direction bias resulting from the strategy discrepancy. Furthermore, we only use extrinsic rewards to update the parameter of the controller, circumventing the instability caused by time-varying intrinsic rewards. The usage of prediction error leads to our second distinction: we do not treat prediction error an immediate intrinsic reward. Instead, we consider it as long-term potential exploration gain, referred to as Qi"t. Qie is added with controller Q-values, together dominating action selection. Intuitively, as training progresses, both the controller Q and the predictor Q will gradually converge, causing the prediction error to become small enough (mainly dependent on the fluctuation of the controller Q). Thus, it can be reasonably assumed that the prediction error represents the current estimation of the upper bound of the benefits that can be obtained through exploration in subsequent training.
FSD uses the prediction error to directly bias the action selection, which requires our prediction error to accurately reflect the novelty of the state-action pair and the strength of the agents' interdependence in the current state. Similar to the approach used in the Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al. 2018), we introduce clipped double Q-learning to mitigate the overestimation problem of the controller Q thereby reducing the noise in Qiae caused by the original Q function approximation error.
We validate our method on didactic examples used in EMC Zheng et al. 2021. J. By illustrating the distribu- tion of agents' positions at different times during training, we demonstrate that our method can effectively guide agents to explore states with greater potential value. Subsequently, we evaluated FSD Predator Prey (Rashid et al. 2020) and several challenging StarCraft II micromanagement tasks (Samvelyan et al.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Empirical results show that FSD achieves state-of-the-art performance compared to several widely adopted baseline methods. Notably, unlike previous curiosity-driven methods, FSD does not use an ex- plorer, thereby saving a considerable amount of computational resources. Additionally, we demonstrate the effectiveness of Qirt and clipped double Q-learning separately through ablation studies.
2019
Contributions. (1) We introduce FSD, novel approach that utilizes the prediction error against the controller Q the long-term exploration bonus achieve farsighted exploration and forgoes the explorer Q to conserve computational resources and achieve self-directed exploration. (2) We provide an intuitive analysis of the prediction error and demonstrate the soundness of FSD along with clipped double Q-learning, (3) We show that FSD can be scamlessly combined with value decomposition approaches to enhance their performance, conducting experiments on Simultaneous Arrival, Predator Prey, and several challenging tasks in the StarCraft II micromanagementt benchmark.
2 Background
Decentralized Partially Observable Markov Decision Process. In practice, the full state of the environment is often unavailable to agents, and agents can only have observations of the environment. In such cases, a cooperative multi-agent task can be modeled as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016). Dec-POMDP can be described by tuple R. n, > where I represents the set of n agents, S is the global state space and A is the action space. At cach time step, agent i E observes o; E Ω determined by observation function (s,,t) and ils action-observation history will be updated as Ti E = (Ω x A)" x f2. Then agent selects an action a, according to its policy Tyy which aims to maximize the long-term team reward jointly. After the joint action [a.]=_1 E A = A" has been conducted, all agents receive the feedback from the environment, which includes shared reward state determined by transition probability function P(s"|s, a). The objective of agents is to find the joint policy π that maximize the expectation of discounted cunulative team reward E 7 = so, TT), or simply denoted as the joint value function V*(s), where y E [0,1] is the discount factor.
R(s, a) and new agent-specific observation of O(s', .)). s' is the new
Centralized Training with Decentralized Execution (CTDE). In this paradigm, global informa- tion is utilized during training, guiding agents in learning superior strategies while circumventing the non- stationarity problem inherent in independent learning, thus allowing for the convenient use of replay buffer to enhance sample efficiency. During execution, agents' policies rely solely on local action-observation history. Except for decentralized actor, centralized critic approaches based on actor-critic framework (Lwwe et al. 2017 Yu et al. 2022). value decomposit ion approaches (Sunehag el al. 2018 Rashid et al. 2018 Son el al.
2019 Wang et al.] 2021b) are the mainstream methods for the CTDE paradigm, where the global Q value is docomposed into combination of local Q-values:
Q a) ((Qi(1))1) .... Qu(Tn,an))
(1)
Qtot is trained to minimize the expected TD error:
(()) = BT.TTED (r + max (T', Quo Qf.c((T. (Ta)))
(2)
where D represents the replay buffer and 0 is the paramcter for the target network. 0- is regularly updated by 0, serving a latent and stable follower of 0. After training, the execution policy is derived from Qi-
VDN (Sunehag et al. 2018) is the earliest method based on value decomposition, which simply factorizes Qiot into the sum of individual Q-values. QMIX (Rashid et al. 2018) highlights that, for value decomposition approaches, it is essential to ensure the following consistency between Qtot and individual Q;:
arg max,, (T(T)
arg max QIoe(T, =
arg max.., Qn, an)
which is later denoted as the IGM principle (oon et al. 2019). Consequently, QMIX introduces a hypernet- work f as eq. and ensures that of 2 0, vi I. As classic method, QMTX usually provides stable and aq.
3

---

Published in Transactions on Machine Learning Research (04/2025)
satisfactory performance. Subsequently, Weighted-QMIX (Rashid et al. 2020 introduces weighting into Q- value learning and places more importance on better joint actions to address the limitation of monotonicity. More algorithms have been proposed to enhance representation expressiveness, such as QTRAN (Son et al. 2019), QPLEX (Wang et al. 2021b) and DAVE (Xu et al. 2023).
3 Related Work
Single-agent exploration. In the single-agent domain, how to achieve effective exploration has been ex- tensively studied. Recent literature (Aubret et al. 2019 Hao al. 2024) has already provided a comprehen- sive summary of the Lopic, especially on intrinsic motivatioon-oriented methods, also known as curiosity-driven methods (Zheng et al., 2021). Intuitively, the novelty of certain states can be measured by the visit times to those states, and intrinsic rewards can be calculated accordingly. To extend these methods to large state space, various methods with pseudo-count have been introduced (Bellemare et al.] 2016: Fox et al. 2018) where DORA (Fox et al., 2018) suggests leveraging the pseudo-count as the long-term exploratory value. RND (Burda et al.) 2019 on the other hand, assesses state novelty through the prediction error of the next state embedding, which is obtained from a randomly initialized, fixed network and predicted by another network with the same structure but learnable. Moreover, VTMTM (Mohamed & Rezende, 2015) encour- ages agents to explore states with higher empowerment, calculated using the mutual information between action sequences and states. VIME (Houthooft et al. 2016) also utilizes information theory, encouraging agents to maximize information gain about the environment dynamics with each step, thus satisfying the agent's 's curiosity about the environment. ICM (Pathak et al.] 2017) uses an inverse model for better state embedding, allowing the forward model to be less affected by the 'noisy TV' problem (Burda et al.] 2019) when predicting the next state. Recently, NGU (Badia al. 2020 leverages both episode and inter-episode intrinsic rewards, achieving better performance in long-sequence problems.
Multi-agent exploration. Unlike exploration in the single-agent setting, efficient coordinated exploration is crucial in multi-agent environment. Jaques et al.] (2019) proposes calculating intrinsic rewards for agents' chosen action by assessing ils influence on other agents' action selection. Subsequently, EITI and EDTI (Wang et al. 2020) incorporate agents' interaction values regularization terms in policy optimization, where EITI encourages exploration of states that influence other agents' transition dynamics while EDTI focuses on states that affect other agents' decision-making. IE (Liu et al.) 2024) leverages transformer models to identify interaction states and guides agents to start exploration from these high-influence states. MACE (Jiang et al., 2024g is a method under the distributed learning setting with limited communication, Iacilitating coordinated exploration by sharing local novelty and hindsight influence estimation.
On the other hand, MAVEN [Mahajan et al. 2019) introduces latent variable learn hierarchical policies, enabling committed exploration through diverse cxploration strategies. CDS [Li et al. 2021 encourages agents to exhibit diverse exploratory behaviors by incorporating mutual information between agents a regularization term. EMAX (Schäfer et al. 2024] explicitly uses ensemble value functions to estimate the variance among Q-values and follows the UCB (Auer ct al. 2002) exploration policy.
EMC Zheeng el al. 2021 is the first to use the prediction error of individual Q-values as intrinsic rewards. Since value decomposition methods achieve implicit credit assignment Wann et al. (021aa), the prediction error of the individual Q-value function is also influenced by other agents; thereby, agents are encouraged this, EXPODE (Zhang & Yu 2023) uses
to visit states where careful coordination is required. Building twin QMIX and intricately designs the intrinsic reward as the prediction error against the higher Q-value. structure used in both EMC and EXPODE inevitably leads to
However, the controller-explooer--eeee deviation in the controller's exploration direction.
4 Method
This section introduces FSD, a novel model-free method for efficient multi-agent exploration. FSD takes the prediction error against individual controller Q-values as Qi"t, which is combined with controller Q-values to determine action selection during sample collection. First, we describe the method in detail; then, we
4

---

Published in Transactions on Machine Learning Research (04/2025)
Excecution
Training
Ptot
T, T
environment
Mixing
min Pror k=12
Controllers 0
P2 Q
TD Target
Gradients
ai
Controller
Twin QMIX
Action Selector Q 0
Predictor
Optimization Target O min o (T/+)
Pairwise distance
Prediction error !
Precictors Q
aQii
Gradients
MSE
Figure An overview of FSD's framework
analyze the sources of prediction error to verify the significance of Qid and why clipped double Q-learning can help reduce noise in prediction error.
4.1 Farsighted Self- Daff-Direction
The prediction error against individual Q-values contains not only information about local exploration but also global exploration status due to implicit credit assignment (Zheng et al. 2021 Wang et al. 2021a] Thus, we use the prediction error against individual Q-values as the signal to guide agents in coordinated exploration. As shown in Fig. T FSD contains two network components, namely the controller and the predictor, where the controller represents the strategy of agent interaction with the environment, and the predictor is trained to approximate the coutroller. The prediction error between the predictor and the controller serves as the signal for directing coordinated exploration.
Additionally, FSD considers the prediction error potential long-term exploration gain Q."t in farsighted manner rather Ihan immediale reward gained from exploration. Over the long lerm, the controller and the predictor gradually converge after sufficient exploration, and the prediction error will eventually become a value small enough. Thus, to some extent, the prediction error represents the long-term exploration bomus that can be obtained from exploration starting from the current stale-action pair. In the action selection phase, FSD calculates Q(a((Ta))) :), which, along with the controller (T, guides the action selection process:
(T.)) (T..) + aQ (Ta.),
(4)
where is usod control the magnitude of exploration. Subsoquently, e-groddy strategy basod on Q determines the final selected action.
Let the predictor be denoted by Q Qiat is caleulated as the distance between the controller and the predictor:
(T" = (f. Q (T..))
(5)
larger (T, a.) with specific a indicates more insufficient visits to the state- action pair or stronger interdependence among agents existing in the next state after is taken, which makes the action more worth taking and the next state more worth exploring. With Qine biasing agents into coordinated exploration, more valuable experience is collected, and agents are likely to achieve a better policy.
As analyzed in section 4.2
QMIX Rashid al. 2018 a classic method performing well across multitude of tasks, selected the backbone. It's worth noting that our method be embedded into any value decomposition method.
5

---

Published in Transactions on Machine Learning Research (04/2025)
As shown in eq. directly the pointwise prediction error, which requires that prediction error accurately reflects the information for coordinated exploration. The inherent over-estimation problem in Q-eaaninn can lead to fluctuations in the controller Q-values, which constitute a significant source of noise in the prodiction error. To reduce noise, FSD introduces clipped double Q-lcarning similar to TD3 (Fujimoto et al. 2018); mitigating the over-estimation problem by laking the smaller targel Q to form the TD larget.
Specifically, the controller includes two networks with the same structure but different initializations, Q and Q, where 0 learns the policy to interact with the environment (referred to as Q in eq. 1q.4andd and eq. [
while Q is only used during training. Correspondingly, two target networks, Q and Q%, are used stabilize training, and the smaller target Q-value is used to calculate the one-step TD target:
0 = arg max 2iol (T',
k 1,2
(6)
(controller(0))
r + min k==1,
Qio (T', .44))
Q (T, a)
k 1,2
(7)
t + (1- E) EdET:
k == 1,2
(8)
The predictor has the same network structure with the local Of and is trained concurrently with the controller to approach it:
Σ (1))))))
min of o 1=12
Spredictor (Vb)
(9)
4.2 Reduce Noise Prediction Error
The prediction error of the individual controller Q-values stems from two main sources:
Moving largel. As the optimization larget for the predictor, the individual controller Q-values are continually changing along with the advancement of the training, which makes the target difficult to approach.
Error in approaching a faxed largel. Even if the individual controller Q-values were fixed, there would still be an approximation error between the predictor and the fixed target.
Since both the predictor and controller share the same local Q; network structure, the error in approaching fixed target primarily depends on how frequently particular state-action pair (or its neighboring spaces) is visited, which indicates the novelty of the state-action pair (s,a) and usually the novelty of the next state s'.
The flutuuttion in the individual controller Q-values can be further broken down
into three major components:
Infrequent visits a particulor state-action pair. Few visits to some state- action pair make the estimation of its value unstable.
2
Strong interdependence among agents in some states. strong interdepen- dence lies among agents some states, the estimation of individual Q-values will be highly susceptible to the actions chosen by other agents during their exploration.
Figure 2: Strong interdependence lies between agents in the state.
Overestimation in Q-learning. has been known that Q-value is casily overestimated in training, and such positive bias becomes more pronounced error accumulates (Liang et al. 2022

---

Published in Transactions on Machine Learning Research (04/2025)
To better illustrate the second component, let's consider the scenario shown in Fig. two agents in red are required to arrive at the blue target simultaneously, and any individual arrival would be punished. If Agent 1 moves up, the reward may be positive if Agent 2 moves right, or negative otherwise. On the other hand, if Agent 1 takes other actions, the reward may be negative if Agent 2 moves right, or zero otherwise. Compared to the other actions, the move-up action of Agent. the other agent. And the fluctuation of the controller Q1 (up) is likely larger, which leads to a higher Qi" (TI, aup), guiding the agent to actively try to reach the target. Besides, Qi also guides the agents toward similar states with strong agent interdependence.
the current state is more influenced by
We aim for agents to explore novel states or interaction states where strong interdependence lies. From the analysis above, we show that among the listed factors contributing to the prediction error of individual controller Q-values, only the fluctuations caused by Q-value over estimattin constitute noise. Therefore, akin to TD3, we employ clipped double Q-learning to significantly mitigate the problem of Q-value overestimation, thereby effectively reducing noise in prediction error.
EXPODE (Zhag & Yu 2023) also uses twin QMIX, however, EXPODE primarily aims to leverage the min- max operator to construct better intrinsic rewards, with the max operator employed in generating intrinsic rewards. In contrast, our approach solely focuses on reducing noise in prediction error, and we simply use the prediction error between the predictor Q and the smaller target controller Q-value, which is consistent. with the update of controller Q in TD target formation.
5 Results
In this section, we conduct a set of empirical experiments and demonstrate the performance of FSD in Simul- tancous Arrival (Zheng et al., 2021), Prodator Prey (Rashid et al.) 2020), and StarCraft II micromanagement (Samvelyan et al. 2019). QMIX the default backbone if not specified. Experiments are conducted on eight NVIDIA RTX 4090s, with training time ranging from half an hour to 10 hours, depending on the complexity of the task and the number of agents involved. Baselines are trained using their open-source codes, with some results derived from the open-source results of EXPODE (Zhang & Yu 2023 As for evaluation, action is selected greedily based the controller Q All experiments have been repeated for five runs over different random seeds. In the ablation study section, we separately showcase the effectiveness of Qint and clipped double Q-learning.
5.1 Simultaneous Arrival
Simultaneous Arrival is introduced by Zheng et al. (2021) as a didactic example. In this task, two agents are separated from the middle, and they can only observe each other when they are near their corresponding targets. A positive reward is given only if both agents arrive at their corresponding largets simultaneously, while a negative reward -p is given if only one agent reaches its target. Otherwise, agents receive no reward, In our experiments, set p to a relatively high value of 3. The success of the task requires a high level of coordinated exploration from the agents, and its grid-world environment also facilitates visualization.
It has been validated that EMC-VDN performs better than its counterparts in this task, so our comparison is conducted among methods using VDN as the backbone. The tested methods include FSD-VDN, FSD-sgl- VDN denoting no clipped double Q-learning is used, FSD-woin- HiD-woin-VDN denoting no Qtat is used, EXPODE VDN, EMC-VDN, and VDN.
As illustrated in Fig. both FSD-VDN and EMC-VDN achieve success in this task. However, EXPODE- VDN and FSD-sgl-V VDN frequently fail in this task despite occasional successes. Similar to VDN, FSD- voin-VDN consistently fails to complete the task. These results affirm the necessity of Qint for FSD in the Simultaneous Arrival task. Notably, the standard setting of the scenario lacks randomness, meaning that during evaluation, the win rate is either 0 or 1 for a fixed policy, and the curves essentially represent the proportion of wins across five different runs.
Besides, as shown in Fig. we have plotted the heat map of the frequency of visitation to various locations by agents at different times during the training of FSD-VDN. In the very early stages of training, the agents
7

---

Published in Transactions on Machine Learning Research (04/2025)
FSD-VDN
FSD sgl-sg/ivi VON
Test Win Rate
FSD-woin-VDN
0.8
EXPODE-VON
EMC-VDN
0.8
VON
0.4
100000
200000
300000
400000
500000
Steps
Figure 3: The performance on Simultaneous Arrival
15k
60k
150k
100
D40
Figure 4: The visitation frequency heat map Simultaneous Arrival during the training of FSN-VDD
broadly explore their surrounding enviromment, while in this case, each agent primarily moves around the starting point, the upper left and lower right corners of the map, respectively. Directed by Qint, agents gradually move closer to the center of the map, where the targets are located. By the 150k time steps, the agents are almost always successful in completing the task, reaching the targets on row simultaneously. The heat map shows that the agent on the right. side visits the area near its larget more frequently than the agent on the left, indicating that it has learned strategy wait for the other agent before reaching the target.
5.2 Predator Prey
Predator Prey challenging, partially observable multi- agent game where predators must cooperate capture prey and receive rewards, with penalties imposed for in- dividual captures. This scenario features a larger num- ber of agents (eight) and includes mobile prey, mak- ing it more challenging than the simultancous arrival task. We conduct comparisons of the FSD against several bascline algorithms, including EXPODE, EMC, Weight- QMIX (Rashid et al. 2020 QMIX, and VDN.
Predator Prey
25
Test Test_reweer reward
0
FSD-VDN FSD-QMIX
-25
EXPODE-VDN
EXPODE-OMIX OMIX
-50
VDN
QMIX
As shown in Fig. 5 FSD-VDN and FSD-QMIX outper- form other basclines in learning efficiency. EXPODE- VDN and EMC-VDN also succeed in the task and demon- strate strong performance. It is worth noting that, in con- trast FSD-QMIX, EXPODE-QMIX fails in the task and tends toward local optimum where predators do not capture prey at all, which indicates method
-75
EMC-VDN
CW_ QMIX
OW_OMIX
-100
0.0
0.2
0.4
0.6
0.8
1.0
Steps
1e6
Figure The performance on Predator Prey
8

---

Published in Transactions on Machine Learning Research (04/2025)
better enhance coordinated exploration in this task. Among other bascline methods, Weighted-QMIX is noted for its ability to focus on learning the value of Q(,, a") where a' arg max, Q(s, a), thus also finds the optimal policy. VDN also succeeds in the task, albeit much less efficiently. QMIX, which is based on the monotonicity assumption, fails in the task due to its non-monotonic reward structure.
5.3
SMAC Benchmark
FSD
EXPODE
CDS
QMXX
GPLEX
EVC
VON
DW QMIX
OW OMIX
1.0
Bh_ys_8z
MMM2
5m VS_BVmmmm
0.8
Teel wit rale
0.6
Test win rale
0.8
0.5
0.4
0.4
0.4
0.2
0.2
0.0
0.5
1.5
Steps 2.0
Sleps
Steps
Figure 6: Results on SMAC benchmark
StarCraft II Micromanagement (SMAC) is a commonly used benchmark in the field of multi-agent reinforce- ment learning (Wang et al. 2021b Zheng al. 2021 Zhang & Yu! 2023). In SMAC, agents trained by the evaluated methods have only partial observation of the environment, while their opponents are driven by the built-in AI. We tested FSD on two super hard maps: 6h _s_ yz and MMM2, and one hard map: 5m_vs 6m. Baseline algorithuss include EXPODE, EMC, CDS, QMIX, QPLEX, VDN and Weighted-QMMn As illus- trated in Fig. FSD has achieved stale-of-the-art performance.
CDS fosters better exploration by encouraging diversity among agents. EMC, with the compact expressive capabilities of QPLEX, utilizes intrinsic rewards and episodic memory to direct coordinated exploration and make the best use of experience, demonstrating strong performance in the SMAC benchmark. EXPODE further improves performance through the design of intrinsic rewards and enhanced learning of the predictor network.
In 6h vs 82, FSD exhibits the fastest learning rate, and 5m vs 6m, it achieves the highest win rate, while in MMM2, FSD achieves performance indistinguishable that of EXPODE, FSD leverages the prediction error against the controller Q to directly formalize Qi and reduce the noise in Q."tt through the implementation of clipped double Q-learning, thereby more accurately directing agents in coordinated exploration.
5.4 Ablation Study
Effect of iiae and clipped double Q-lcarning. We conduct ablation studics on Prodator Prey and the super hard 6h vs 8z and MMM2 maps of SMAC specifically assess the impacts of Qi and clipped double Q-learning. Note that removing these two components reduces algorithm QMIX. FSD without Qiae is denoted as FSD-wo-in, and FSD without clipped double Q-learning is denoted as FSD-sgl.
Fig. a shows Ihat on Predator Prey, FSD-wo-in does not show improvement over QMIX and similarly fails on the task. In contrast, FSD-sgl succeeds in the task and achieves indistinguishable performance from FSD. This suggests that, in relatively simpler scenarios where the overostimation issue is less severe and also less noise in a clipped double Q-learning does not have a significant impact. On the other hand, Qiatt can help find the optimal policy by directly enhancing coordinated exploration.
9

---

Published in Transactions on Machine Learning Research (04/2025)
FSD
FSD-wo-in
FSD
20
0.8
FSD-agl
0.8
FSD-sg)
QMIX
QMIX
Test win rate
0.8
0.6
-20
Testt.
0.4
-40
-60
FSD-wo-in FSD-sg
-80
GMIX
0.0
0.5
1.0
2.0
Steps
Stepe
(o) Predator Pray
(b) MMM2
(c) 6in _a______ _r_
Figure 7: Ablation study on the two major components
Prodator Pray
MMM2
1.0
4pha-1
apha=3.1
0.8
4pha-2.01
apha=0
rate
apphe11
slpta-0.1
Test
alpta-0
apph=0 01
0.4
-40
0.2
-90
0.0
02
0.4
0.8
0.00
0.25
0.50
0.75
1.00 Staps
1.25
1.50
1.75
2,00
Steps
1e8
Figure 8:
Ablation study on the coefficient term
It can be observed from Fig. (c) that clipped double Q-learning crucial for agents to learn effective strategies on these two maps due to the complex environments where over-estimation issues can destabilize training and introduce much noise in Qiat QiTt also plays an important role, as FSD-sgl shows performance improvements over QMIX on both maps. Additionally, with higher learning efficiency, FSD achieves better and more stable performance compared to FSD-wo-in.
Across both the complex SMAC tasks and the simpler Predator Prey task, combining clipped double Q- learning with qin maximizes the effect of Qine, more accurately directing agents in coordinated exploration and achieving high performance stably.
Effect of exploration coefficient a. We also analyze the effect of the exploration coefficient a in eq. 4 on performance using FSD-sgIVDDN and FSD in the Predator Prey and the MMM2 map of SMAC, respectively. Fig. shows that, in general, a coefficient between 0.01 and 0.1 can effectively improve exploration efficiency. In the relatively simple Predator- Predator-Prey scenario, where coordinated exploration crucial, a can take a larger value, such 1, to further enhance exploration. However, on the more complex MMM2 map, setting a leads to excessive exploration by the agents, which ultimately reduces their learning efficiency.
6 Conclusion
This paper introduces FSD, a simple yet effective method for enhancing efficient multi-agent exploration through farsighted self-diroction. FSD leverages long-term exploration bonus, tailored for the agent, as a directional signal, which denoised using clipped double Q-learning. With this efficient exploration method, our approach has outperformed wide range of widely adopted baseline methods the challenging SMAC
10

---

Published in Transactions on Machine Learning Research (04/2025)
II benchmark. The limitation of work lies in the lack of robustness against fluctuation in prediction error. For future work, we may consider using methods that adaptively adjust exploration magnitude to enhance robustness. Additionally, we will evaluate our method on other influential benchmarks or different application scenarios.
References
Arthur Aubret, Lactitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforcement learning, 2019.
Peter Auer, Nicolò Bianch,, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., 47(2-3):235 256, 2002.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohhan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Ticleman, Martín Arjovsky, Alexander Pritzel, Androw Bolt, et al. Never give up: Learning directed exploration stralegies. In 8th International Conference on Learning Representations, 2020.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Newral Information Processing Systems, pp. 1471-1479, 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning, 2019. URL https://aaxiv org/ htt///as///////// 1912.066680
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distil- lation. In 7th International Conference on Learning Representations, 2019.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. DORA the explorer: Directed outreaching reinforce- ment action-selection. In 6th International Conference on Learning Representations, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Proceedings of the 35th International Conference Machine Learning, pp. 1582-1591.
PMLR, 2018. URL http://prroceedings... mlr httpppppreee//////iiaaoooooa..a. html
Sven Gronaner and Klaus Diepold. Multi-agent deep reinforcement learnigg a survey. Artif. Intell. Rea., 55 (2):895 943, feb 2022. ISSN 0269-2821. 0007/s10462-021-099988
doi: 10. 10.1007/1110662-01--999999 URL https://doi. org/10.
Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Trans. Newral Networks Learn. Syst., 35(7):8762 8782, 2024.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109- 1117, 2016.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859 865, 2019. doi: (1l11///ccence.aattee 1126/science.
David Janz, Jiri Hron, Przemyslaw Mazur, Katja Hofmann, José Miguel Hernándoz-Lobato, and Sebastian Tschiatschck. Successor uncertainties: Exploration and uncertainty in temporal difference learning. In Advances in Newral Information Processing Systems, pp. 1509-4518.. 2019.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Gülgehre, Pedro A. Ortega, DJ Strouse, Jocl Z. Leibo, and Nando de Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 3040 3049. PMLR, 2019.
Haobin Jiang, Ziluo Ding, and Zongqing Lu. Settling decentralized multi-ngent coordinated exploration by novelty sharing, In Thirty-Eio Eighth AAAI Conference on Artificial Intelligence, pp. 17444 17452. AAAI Press, 2024.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Newral Information Processing Systems, pp. 3991 4002, 2021.
Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. Reducing variance in temporal-differenee value estimation via ensemble of deep networks. In International Conference on Machine Learning, pp. 13285 13301. PMLR, 2022.
Zeyang Liu, Lipeng Wan, Xinrui Yang, Zhuoran Chen, Xingyu Chen, and Xuguang Lan. Imagine, initialize, and explore: An effective exploration method in multi-agent reinforcement Icarning. In Thirty-Eiee Eighth AAAI Conference on Artificial Intelligence, pp. 17487 17495, 2024.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-complliive environments. In Advances in Neurul Information Pro- cessing Systems, pp. 6379 6390, 2017. URL
66597505744411a86466661111. aatrec1199aaadec--stttttt...... html
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7611 7622, 2019.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically mo- tivated reinforcement learning. In Advances in Newral Information Processing Systems, pp. 2125 2133, 2015.
Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016. ISBN 978-3-319-28927-4. doi: 10.1007/966-33319-22222 URL 0ttp://///o...0/1111000000
28929-8
Deepak Pathak, Pulkit Agrawal, Alexci A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, pp.
2778-2787. PMLR, 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pp. 4292-4301. PMLR, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding mono- tonic value function factorisation for deep multi-agent reinforcement learning. In Advances in Neu- ral Information Processing Systems, 2020. URL httpss:/prroceeiings.c 742277aabbe02222222177777776-363
wwwpoeedddinnn.uuisssoo.o nourips. www.nnws.....pprr/////// ccc/pappr/222//aasll
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Wiut, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp.
International Foundation for Autonomous Agents and Multiagent Systems, 2019. URI. http://dl. acm. http:/ttattioo..mmid-3-3333333
12

---

Published in Transactions on Machine Learning Research (04/2025)
Lukas Schäfer, Oliver Slumbers, Stephen McAleer, Yali Du, Stefano V. Albrecht, and David Mguni. Ensemble value functions for efficient exploration in multi-agent reinforcement learning, 2024.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to fac- torize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonncrat, Joel Z. Leibo, Karl Tuyls, et al. Valoo-dccomposition networks for cooperative multi-agent. learning based on team reward. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp.
2018.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350 354, 2019. doi: 10.1038 14008//515566001-112222 URL https:///oi. wtt.:/o................ 1ttp://100000//1111--011------11
Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards understanding cooperative multi-agent q-learning with value factorization. In Advances in Neural Information Pro- cessing Systems, pp. 29142 29155, 2021a. URL httpp///ppooeeennnn.rrrr....pppp
13f 3 wat.eeabbb-bbstratttoeoa Abstract.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjic Zhang. QPLEX: duplex ducling multi-agent q-learning. In 9th International Conference on Learning Representation, 2021b.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevicw.nct, 2020. URL
Yuanyuan Wu, Haipeng Chen, and Feng Zhu. Del-aim: Decentralized coordination learning of autonomous interscction management for connected and automated vehicles. Transportation Research Part C: Emerging Technologies, 103:246-280 260, 2019.
Zhiwei Xu, Bin Zhang, Dapeng Li, Guangehong Zhou, Zeren Zhang, and Guoliang Fan. Dual self-awareness value decomposition framework without individual global max for cooperative MARL. In Advances in Newrul Information Processing Systems, 2023.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems, 2022.
Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment. for large scale city traffic scenario. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 19-17, 2019, pp. 8620-3624 ACM, 2019.
Yucong Zhang and Chao Yu. EXPODE: exploiting policy discrepancy for efficient exploration in multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiogent Systems, pp. 58 66. ACM, 2023. do1/10.5555/355
doi: 10.5555 10.5555/3545994 05555/3555544555866111 URL https://dll acm. org/
3598619
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, pp. 3757-3769, 2021.
13


Reviews:
[{"review_id": "hhMWCInRaL", "reviewer": "Reviewer_PcAA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method presented is intuitive, simple to understand and builds upon existing curiosity-based methods that use prediction error to guide exploration. \n2. Results presented on 2 popular MARL domains show that the method not only outperforms existing baselines, but is also more computationally efficient than some of the existing exploration methods.\n\nWeaknesses:\n\n1. The writing has scope for further improvement. The introduction is overly technical and dives into details that are better described in the Method section. \n2. The novelty of the work is limited. One of the main contributions of the work is applying clipped double Q-Learning, which is an existing method that is already known to stabilize training. The primary contribution is the bonus based on prediction error, however the contribution of this component in improving performance is not very significant. \n3. Some open questions remain, and answering them could improve the strength of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a method for co-operative multi-agent exploration that uses prediction error in Q-values as a signal to guide exploration. Furthermore, the work incorporates a clipped double Q-Learning objective to reduce instability in the prediction error. Results show that the presented framework, referred to as Far-Sighted Direction (FSD), improves performance on popular MARL benchmarks such as Starcraft (SMAC) and Predator-Prey. Ablations show that both components (prediction bonus and double Q-Learning) contribute to the success of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The introduction dives too quickly into finer technical details and is difficult to follow. The overview of the proposed method should be simplified and some of the content should be moved to the Method section.  I think the controller and predictor terminologies need to be properly defined in the methods section. “Controller” and “Predictor” are not standard terminology and the section begins of talking about individual controller Q-values, without introducing proper notation for it.  [Strengthen the work] \n\n2.  Is it possible to show results with larger number of agents? It seems that this approach might not scale well if multiple agents are learning in parallel due to increased non-stationarity. [Strengthen]\n\n3. Why is method restricted to a Q-Mix type setup? If N individual agents are learning in an environment with a shared reward structure and individual Q-networks, shouldn’t it be possible to train using the method described in this paper? [Critical to answer] \n\n4.  From the ablation experiments, it  seems that FSD-wo-int also performs extremely well and is only marginally below FSD. However, FSD-wo-int is simply using clipped double Q-Learning. This suggests that main gains of the experiments come from clipped double Q-Learning (which is arguably well-known) while the intrinsic reward only has a small contribution (even though it is the main focus of this paper). The authors should feel free to refute this as it is based on my understanding of the paper. [Critical] \n\n5. Is the predictor network initialized to be the same as the controller? If not, is there a warm-up period without intrinsic rewards that could be useful to stabilize the Q-values because random initialization might lead to assignment of extremely noisy intrinsic rewards. [Strengthen] \n\n6. The work mentions that considerable time is being saved by foregoing an explorer. Can the authors roughly estimate the increase in computational efficiency (through a metric such as wall clock time)? Even though the explorer is removed, the addition of a predictor along with maintaining 2 versions of the network might increase training time. [Critical to support claims] \n\n7. It would be useful to have brief descriptions of the baseline algorithms, the current plots are not easy to decipher. \n\n8.  How often is the predictor network updated? Updating too fast might reduce the signal while updating too slowly might lead to staleness in exploration. Is this controlled by some hyperparameter? [Strengthen understanding]", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nbJ3oqTR0Z", "reviewer": "Reviewer_z7h1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper provides a comprehensive background on the topic.\n\n- The idea of using prediction error to encourage exploration is interesting.\n\n- Illustrative examples are included to aid in understanding the method.\n\n- Recent baselines are incorporated in the comparison, ensuring a fair evaluation.\n\n**Weaknesses**:\n\n1. Unclear Method Introduction\n\nThe concept of \"prediction error,\" which is central to the proposed approach, is not formally defined.\nThe definition of $Q_i^{int}$ should be introduced earlier, given its frequent usage. The phrase \"a long-term potential exploration gain\" is vague. The paper should explicitly define $Q_i^{int}$, for example, as the expected return of intrinsic rewards.\n\n2. Lack of Clarity in Justification\n\nIt is unclear why the proposed method works. The explanation in Section 4.2 is not fully consistent with the actual method.\nThe paper categorizes errors of individual controllers as \"moving target\" errors and errors in \"approaching a fixed target,\" but it is not evident how $Q_i^{int}$ captures these errors.\nSection 4.2 states that greater fluctuation in the controller leads to a higher $Q_i^{int}$, thereby promoting exploration. However, this is not apparent from Equations (5), (7), and (9). The paper should provide additional evidence on how reward fluctuations influence the controller, the predictor, and ultimately $Q_i^{int}$.\n\n3. Lack of Detailed Comparison with EXPODE\n\nThe proposed method bears strong similarities to EXPODE, but the paper does not provide a clear distinction between the two.\nWhat specific limitations of EXPODE does FSD aim to address?\nIs there a strong motivation that justifies the development of FSD instead of EXPODE?\n\n4. Limited Relevance to MARL\n\nThe proposed method does not appear to be specifically tailored for MARL.\nWhile it can be integrated into value factorization methods, the method itself lacks specialized design elements for MARL.\n\n5. Unclear Evidence on Exploration Efficiency\n\nThe performance improvements may primarily stem from clipped double Q-learning rather than the proposed exploration method.\nThe experiments do not sufficiently demonstrate that the efficiency gains result from improved exploration.\nThe paper could include additional experiments to analyze how the coefficient $\\alpha$ influences performance and whether $Q_i^{int}$ is actually larger in less-visited states with fluctuating values.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method, Farsighted Self-Direction (FSD), to promote exploration in MARL. The proposed approach utilizes prediction error as a long-term exploration bonus, facilitating self-directed exploration. When integrated with value decomposition methods, FSD enhances performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical Issues:\n\n1. Provide a more thorough analysis of why and how the proposed method works, specifically how prediction error is captured by $Q_i^{int}$ and how this contributes to improved performance.\n\n2. Include a more detailed comparison with EXPODE, clarifying the motivation behind FSD and the specific limitations of EXPODE that it addresses.\n\nAdditional Suggestions:\n\n1. Address the other weaknesses outlined above.\n\n2. Strengthen the theoretical and empirical justifications for the proposed method.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cqlwxk9AHd", "reviewer": "Reviewer_uMaH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of achieving improved and coordinated exploration in MARL is an important one, and worth attention. The proposed idea is quite simple and general, and I like the fact that it lifts the need for an explicit explorer in the intrinsic-motivation framework. The methodology seems sound and well motivated, although the main point on the method convergence is only provided as an intuition, rather than being formally proved. The experimental results show indeed the good performance of the proposed framework, although their interpretation is not always as clear as it should, and the results analysis is a bit poor. In general, the paper is quite easy to read and follow, but notation is not always extremely clear, and this renders the understanding of some aspects a bit difficult sometimes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel method for multi-agent coordinated exploration based a simpler way of computing the intrinsic-motivation reward to guide it. This farsighted self-directed motivation removes the need for an explorer in the algorithm, instead only leaving the exploitation behaviours and the predictor one, that is used to compute the error with the former that acts as the exploration incentive. The algorithm uses double-clipped $Q$-learning as a building component, and can be combined with different value-decomposition methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In the Introduction, you say that in intrinsic-motivated exploration, the exploiter is used to interact with the environment, while the explorer is used to provide the intrinsic reward (and thus the exploration strategy). Why then you say a bit later than the explorer learns from the extrinsic (real) reward, while the exploiter learns from both? Shouldn't it be the other way around? Perhaps I misunderstood how this is supposed to work? Moreover, what does it mean that the explorer uses a different predictor? That is is using a different policy? How does this come into play? In any case, I suggest you carefully rephrase this paragraph to be clearer and more explicit, especially as it is then useful to understand the differences with your proposed method, underpinned in the paragraph below.\n\n- In the green box in Figure 1, how can you compute $\\min_{k=1,2}Q_{tot}^{\\phi'_k}$ on local histories $\\tau'_i$? The pairwise distance is between $Q^{\\phi}_i$ and the corresponding $Q^{\\psi}_i$ from Equation (5), so what have the joint target networks to $Q^{\\phi'_k}\\_{tot}$ do here? Perhaps these should be be $Q^{\\phi'_k}_i$ (as it seems from Equation (9) indeed)?\n\n- The paragraph before Equation (6) is a bit confusing: what is double here? Only the mixed (joint) $Q$-function $Q_{tot}^{\\phi}$ or also the agents' local $Q$-networks $Q_i^{\\phi}$? From Equation (9) it seems that also the agents' local $Q$-networks are double, but this is not clear neither from the text nor from Figure 1 (where indeed there is a single set of local $Q$-networks in the blue box, but clearly two mixing processes coming from them, leading to two different joint $Q$-functions). I suggest you clarify these aspects a bit more in details, and adjust notations and representations everyone to be consistent with that.\n\n- In the Simultaneous Arrival experiment, FSD-woin-VDN basically results in clipped double VDN no?\n\n- Figure 5 is really difficult to parse correctly, and differentiate between lines with similar colours (like FSD-sgl-VDN and QPLEX for example) is hard, especially because many of them are behaving similarly and overlap quite often. I suggest to change the colour scheme for the lines to make them more easily identifiable, and perhaps to remove the less interesting algorithms to its improve readability.\n\n- The fact that (FSD-/EXPODE-)QMIX behaves worse than its VDN variants in Predator Prey is interesting: usually QMIX has an higher expressive power than VDN (additivity is indeed a special case of monotonicity used in QMIX's constraint), but it seems to not be able to learn at all in this setting. Do you have any insight on why this may be the case (especially more so, why your more complex FSD-QMIX or EXPODE-QMIX are outperformed by a much simpler standard VDN)?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After considering the reviews and the authors’ rebuttal, I recommend acceptance of this paper. All reviewers now lean toward acceptance, acknowledging that the proposed method—Farsighted Self-Direction (FSD)—is intuitive, performs well empirically, and removes the need for an explicit explorer in MARL.

While initial concerns were raised about the clarity of the method and the attribution of performance gains to clipped double Q-learning, the authors’ responses and additional experiments helped address these points to a satisfactory degree.

For the final revision, the authors should aim to clarify terminology and improve notation throughout the paper. Some figures, particularly Figure 5, would benefit from better readability, and a more thorough comparison with prior work, such as EXPODE, would further strengthen the contribution.
Decision: Accept (poster)