Year: 2020

Abstract:
We explore stochastic-based policy solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factorization methods are susceptible to the relative overgeneralization, where finding a suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology. To resolve this issue, we propose a novel factorization method for cooperative MARL, named FSV, which learns to factorize the joint soft value function into individual ones for decentralized execution. Theoretical analysis shows that FSV solves a rich class of factorization tasks. Our experiments for the well-known tasks of the Non-Monotonic Matrix game and the Max of Two Quadratics game show that FSV converges to optima in the joint action space in the discrete and continuous tasks by local searching. We evaluate FSV on a challenging set of StarCraft II micromanagement tasks, and show that FSV significantly outperforms existing factorization multi-agent reinforcement learning methods.

heading: 1 INTRODUCTION
text: Cooperative multi-agent reinforcement learning (MARL) aims to instill in agents policies that maximize the team reward accumulated over time (Panait & Luke (2005); Busoniu et al. (2008); Tuyls & Weiss (2012)), which has great potential to address complex real-world problems, such as coordinating autonomous cars (Cao et al. (2013)). Considering the measurement and communication limitations in practical problems, cooperative MARL faces the partial observability challenge. That is, each agent chooses actions just based on its local observations.
Centralized training with decentralized execution (CTDE) (Oliehoek et al. (2011)) is a common paradigm to address the partial observability, where agents’ policies are trained with access to global information in a centralized way and executed only based on local observations in a decentralized way, such as the MADDPG (Lowe (2017)) and COMA (Foerster et al. (2017)). However, the size of the joint state-action space of the centralized value function grows exponentially as the number of agents increases, which is known as the scalibility challenge.
Value function factorization methods have been an increasingly popular paradigm for solving the scalability in CTDE by satisfying the Individual-Global-Max (IGM) where the optimal joint action selection should be consistent with the optimal individual action selections. Three representative examples of value function factorization methods include VDN (Sunehag et al. (2017)), QMIX (Rashid et al. (2018)), and QTRAN (Son et al. (2019)). All these methods are -greedy policies, where VDN and QMIX give sufficient but unnecessary conditions for IGM by additivity and monotonicity structures respectively, and the QTRAN formulates the IGM as an optimization problem with linear constraints.
Although these methods have witnessed some success in some tasks, they all face relative overgeneralization, where agents may stick into a suboptimal Nash Equilibrium. In fact, relative overgeneralization is a grave pathology arising which occurs when a suboptimal Nash Equilibrium in the joint space of action priors to an optimal Nash Equilibrium since each agent’s action in the suboptimal equilibrium is a better choice (Wei & Luke (2016)). The non-monotonic matrix game is a simple discrete example. Both VDN and QMIX fail to learn the optimal policy in the non-monotonic
matrix due to their structure limitation. Although QTRAN expresses the complete value function representation ability in the non-monotonic matrix, its full expressive ability decreases in the complex tasks due to the computationally intractable constraints relaxing with tractable L2 penalties. Besides, QTRAN sacrifices the tractability in continuous action space. Therefore, in discrete and continuous tasks, achieving effective scalability while avoiding relative overgeneralization remains an open problem for cooperative MARL.
To address this challenge, this paper presents a new definition of factorizable tasks called IGO (Individual-Global-Optimal) which introduces the consistency of joint optimal stochastic policies and individual optimal stochastic policies. Theoretical analysis shows that IGO degenerates into IGM if the policy is greedy, which represents the generality of IGO. Under the IGO, this paper proposes a novel factorization solution for MARL, named FSV, which learns to factorize soft value function into individual ones for decentralized execution enabling efficient learning and exploration through maximum entropy reinforcement learning. To our best knowledge, FSV is the first multiagent algorithm with stochastic policies using the idea of factorization, and theoretical analysis shows that FSV solves a rich class of tasks.
We evaluate the performance of FSV in both discrete and continuous problems proposed by Son et al. (2019); Wei et al. (2018) and a range of unit micromanagement benchmark tasks in StarCraft II. The Non-Monotonic Matrix game shows that FSV has full expression ability in the discrete task, and the Max of Two Quadratics game shows that FSV is the first factorization algorithm that avoids the relative overgeneralization to converge to optima in the continuous task. On more challenging StarCraft II tasks, due to the high representation ability and exploration efficiency of FSV, it significantly outperforms other baselines, SMAC (Samvelyan et al. (2019)).

heading: 2 PRELIMINARIES

heading: 2.1 DEC-POMDP AND CTDE
text: A fully cooperative multi-agent task can be described as a Dec-POMDP defined by a tuple G = 〈S,U ,P, r,Z,O,N , γ〉, where s ∈ S is the global state of the environment. Each agent i ∈ N choose an action ui ∈ U at each time step, forming a joint action u ∈ UN . This causes a transition to the next state according to the state transition function P(s′|s, u) : S×UN ×S → [0, 1] and reward function r(s, u) : S × UN → R shared by all agents. γ ∈ [0, 1] is a discount factor. Each agent has individual, partial observation z ∈ Z according to observation function O(s, i) : S × N → Z . Each agent also has an action-observation history τi ∈ T : (Z × U)∗, on which it conditions a stochastic policy πi(ui|τi) : T × U → [0, 1]. The joint policy π has a joint action-value function Qπ(st, ut) = Est+1:∞,ut+1:∞ [ ∑∞ k=0 γ
krt+k|st, ut]. Centralized Training with Decentralized Execution (CTDE) is a common paradigm of cooperative MARL tasks. Through centralized training,the action-observation histories of all agents and the full state can be made accessible to all agents. This allows agents to learn and construct individual action-value functions correctly while selecting actions based on its own local action-observation history at execution time .

heading: 2.2 VDN, QMIX AND QTRAN
text: An important concept for factorizable tasks is IGM which asserts that the joint action-value function Qtot : T N × UN → R and individual action-value functions [Qi : T × U → R]Ni=1 satisfies
arg max u Qtot(τ, u) = (arg max u1 Q1(τ1, u1), ..., arg max uN QN (τN , uN )) (1)
To this end, VDN and QMIX give sufficient conditions for the IGM by additivity and monotonicity structures, respectively, as following:
Qtot(τ, u) = N∑ i=1 Qi(τi, ui) and ∂Qtot(τ,u) ∂Qi(τi,ui) > 0,∀i ∈ N (2)
However, there exist tasks whose joint action-value functions do not meet the said conditions, where VDN and QMIX fail to construct individual action-value function correctly. QTRAN uses a linear constraint between individual and joint action values to guarantee the optimal decentralisation.
To avoid the intractability, QTRAN relax these constraints using two L2 penalties. However, this relaxation may violate the IGM and it has poor performance on multiple multi-agent cooperative benchmarks as reported recently.

heading: 2.3 THE RELATIVE OVERGENERALIZATION PROBLEM
text: Relative overgeneralization occurs when a sub-optimal Nash Equilibrium (e.g. N in Fig. 1) in joint action space is preferred over an optimal Nash Equilibrium (e.g. M in Fig. 1) because each agent’s action in the suboptimal equilibrium is a better choice when matched with arbitrary actions from the collaborating agents. Specifically, as shown in Figure 1, where two agents with one-dimensional bounded action (or three actions in discrete action space) try to cooperate and find the optimal joint action, the action B (or C) is often preferred by most algorithms as mentioned in (Son et al. (2019) and Wei et al. (2018)) due to their structure limitation and lack of exploration.

heading: 3 METHOD
text: In this section, we will first introduce the IGO (Individual-Global-Optimal), a new definition of factorizable MARL tasks with stochastic policies. Theoretical analysis shows that IGO degenerates into IGM if the policy is greedy. With the energy-based policy, the structure between joint and individual action values of IGO can be explicitly constructed, which is a novel factorization stochastic-based policy solution we proposed, named FSV. Specifically, FSV realizes IGO using an efficient linear structure and learns stochastic policies through maximum entropy reinforcement learning.

heading: 3.1 INDIVIDUAL GLOBAL OPTIMAL
text: In the CTDE paradigm, each agent i ∈ N chooses an action based on a stochastic policy πi(ui|τi) at the same time step. The joint policy πtot(u|τ) = ∏N i=1 πi(ui|τi) describes the probability of taking joint actions u on joint observation history τ . If each agent adopts its optimal policy while the joint policy is exactly the optimum, the task itself can achieve global optimum through local optimum, which naturally motivates us to consider the factorizable tasks with stochastic policy as following:
Definition 1 For a joint optimal policy π∗tot(u|τ) : T N × UN → [0, 1], if there exists individual optimal policies [π∗i (ui|τi) : T × U → [0, 1]]Ni=1, such that the following holds
π∗tot(u|τ) = N∏ i=1 π∗i (ui|τi) (3)
then, we say that [πi] satisfy IGO for πtot
As specified above, IGO requires the consistency of joint optimal policy and individual optimal policies rather than the actions in IGM, but it degenerates into IGM if policies are greedy. That is to say, IGO is more generality than IGM.

heading: 3.2 FSV
text: In this work, we take the energy-based policies as joint and individual optimal policy respectively,
π∗tot(u|τ) = exp( 1
α (Qtot(τ, u)− Vtot(τ))) (4)
π∗i (ui|τi) = exp( 1
αi (Qi(τi, ui)− Vi(τi))) (5) where α, αi are temperature parameters, Vtot(τ) = α log ∫ UN
exp( 1αQtot(τ, u))du and Vi(τi) = αi log ∫ U exp( 1αiQi(τi, u))du are partition functions.
The benefit of using energy-based policy is that it is a very general class of distributions that can represent complex, multi-modal behaviors Haarnoja et al. (2017). Moreover, energy-based policies can easily degenerate into greedy policies as α, αi anneals.
To learn this decentralized energy-based policy, we extend the maximum entropy reinforcement learning framework for the multi-agent setting, which we’ll describe in the next. Another benefit of considering the stochastic policy with explicit function class for factorizable tasks through IGO is that the architecture between joint and individual action values can be easily constructed through its constrains on policies with specific meanings as follows.
Theorem 1 If the task satisfies IGO, with energy-based optimal policy, the joint action value Qtot can be factorized by individual action values [Qi]Ni=1 as following:
Qtot(τ, u) = N∑ i=1 λ∗i [Qi(τi, ui)− Vi(τi)] + Vtot(τ) (6)
where λ∗i = α/αi.
Theorem 1 gives the decomposition structure like VDN—the joint value is a linear combination of individual values weighted by λ∗i > 0. However, the function class defined by Eq(6), which should only concern the task itself, is related to and limited by the distributions of policy. Although energybased distribution is very general which has the representation ability of most tasks, to establish the correct architecture between joint and individual Q-values and enable stable learning, we need to extend the function class into any distributions. The key idea is that we approximate the weight vector λi directly as α, αi is zero instead of annealing αi during training process. This extends the function class and will at least guarantee IGM constraint when α, αi is zero .
Theorem 2 When α, αi → 0, the function class defined by IGM is equivalent to the following
Qtot(τ, u) = N∑ i=1 λi(τ, u)[Qi(τi, ui)− Vi(τi)] + Vtot(τ) (7)
where λi(τ, u) = lim α,αi→0
λ∗i .
Note that λi is now a function of observations and actions due to the relaxation. Eq(7) allows us to use a simple linear structure to train joint and individual action values efficiently and guarantee the correct estimation of optimal Q-values. We’ll describe it in experiment.
Then, we introduce the maximum entropy reinforcement learning in CTDE setting which is an directly extension of soft actor-critic (q-learning).
The standard reinforcement learning tries to maximum the expected return ∑ tEπ[rt], while the maximum entropy objective generalizes the standard objective by augmenting it with an entropy term, such that the optimal policy additionally aims to maximize its entropy at each visited state
πMaxEnt = arg max π ∑ t Eπ[rt + αH(π(·|st))] (8)
where α is the temperature parameter that determines the relative importance of the entropy term versus the reward, and thus controls the stochasticity of the optimal policy (Haarnoja et al. (2017)). We can extend it into cooperative multi-agent tasks by directly considering the joint policy πtot(u|τ) and defining the soft joint action-value function as following:
Qtot(τt, ut) = r(τt, ut) + Eτt+1,...[ ∞∑ k=1 γk(rt+k + αH(π ∗ tot(·|τt+k))] (9)
then the joint optimal policy for Eq(8) is given by Eq(4) (Haarnoja et al. (2017)). Note that we don’t start considering decentralized policies, the joint Q-function should satisfy the soft Bellman equation:
Q∗tot(τt, ut) = rt + Eτt+1 [V ∗ tot(τt+1)] (10)
And we can update the joint Q functions in centralized training through soft Q-iteration:
Qtot(τt, ut)← rt + Eτt+1 [Vtot(τt+1)] (11)
It’s natural to take the similar energy-based distribution as individual optimal policies π∗i in Eq(5) which allows us to update the individual policies through soft policy-iteration:
πnewi = arg min π′∈ ∏ DKL(π′(·|τ)||π∗i (·|τ)) (12)

heading: 3.3 ARCHITECTURE
text: In this section, we present a novel MARL framework named FSV, which incorporates the idea in a simple and efficient architecture through Eq(7) with multi-agent maximum entropy reinforcement learning. FSV can be applied both in continuous action space and also in discrete action space as a simplification.
Figure 2 shows the overall learning framework, which consists of two parts:(i) individual parts for each agent i, which represents Qi, Vi and πi (ii)incorporation part that composes Qi, Vi to Qtot.
Individual parts for each agent i has three networks: (i)individual Q network takes its own action and observation history τi, ui as input and produces action-values Qi(τi, ui) as output.(ii)individual value network takes its own observation history τi as input and produces Vi(τi) as output.(iii)individual policy network takes its own observation history τi as input and produces a distribution (e.g. mean and standard deviation of Gaussian distribution) for sample actions.
Incorporation part composes Qi, Vi to Qtot through linear combination. Specifically, it sums up [Qi − Vi]Ni=1 with coefficients λi and uses a one-layer hyper-network to efficiently approximate the high-dimensional partition function as following:
Vtot(τ) = N∑ i=1 wi(τ)Vi(τi) + b(τ) (13)
where wi, b is a positive weight and bias respectively. To enable efficient learning, we adopt a multi-head attention structure to estimate the weight vector:
λi(τ, u) = H∑ h=1 λi,h(τ, u) (14)
where H is the number of attention heads and λi,h is defined by
λi,h ∝ exp(eTuWTk,hWq,hes) (15)
where eu and es is obtained by two-layer embedding transformation for u and s. The joint action value function Qtot is updated through soft Q-iteration:
JθQtot = E(τt,ut)∼D[Qtot(τt, ut)− Q̂(τt, ut)] 2 (16)
where Q̂(τt, ut) = r(τt, ut) + γEτt+1∼D,ut+1∼π[Qtot(τt+1, ut+1)− α log πtot(ut+1|τt+1)]. The individual value network is trained by minimize
JφiVi = Eτi∼D[Vi(τi)− (Eui [Qi(τi, ui)− α log πi(ui|τi)])] 2 (17)
The policy network of each agent is trained by minimizing the expected KL-divergence
Jψiπi = Eτi∼D,ui∼πi [α log πi(ui|τi)−Qi(τi, ui)] (18)
For discrete action space, it’s convenient to simplify this framework to Q-learning. Specifically, we directly compute the individual value function Vi = αi log ∑ exp( 1αiQi(τi, ·)) instead of updating the value network, and action distributions are directly produced by Eq(5) instead of the policy network.

heading: 4 RELATED WORK
text: There are many early works with maximum entropy principle such as Todorov (2010) and Levine & Koltun (2013) use it in policy search in linear dynamics and Kappen (2005) and A. Theodorou et al. (2010) use it in path integral control in general dynamics. Recent off policy methods (Haarnoja et al. (2017);Haarnoja et al. (2018b);Haarnoja et al. (2018a)) have been proposed to learn an energy-based policy efficiently through the maximum entropy objective which is adopted in our framework. Value function factorization methods start from VDN (Sunehag et al. (2017)), and is extended by QMIX (Rashid et al. (2018)) and QTRAN (Son et al. (2019)). Other methods such as QATTEN (Yang et al. (2020)) and MAVEN (Mahajan et al. (2019)) go a step further on architecture and exploration. Our method are a member of them but out of the deterministic policy
Current methods adopt different ideas to solve the relative overgeneralization problem. Wei et al. (2018) conduct multi-agent soft Q learning for better exploration. Wen et al. (2019) uses probabilistic recursive reasoning to model the opponents, Yu et al. (2019) adopts inverse reinforcement learning to avoid this problem through right demonstrations, Tian et al. (2019) derives a variational lower bound of the likelihood of achieving the optimality for modeling the opponents. However, none of them adopt value function factorization like FSV which means they suffer the scalability problem.

heading: 5 EXPERIMENTS
text: In this section, we first consider two simple examples proposed by prior work (Son et al. (2019),Wei et al. (2018)) to demonstrate the optimality and convergence of FSV in discrete and continuous action space respectively. And we evaluate the performance in a challenging set of cooperative StarCraft II maps from the SMAC benchmark (Samvelyan et al. (2019)).

heading: 5.1 MATRIX GAME
text: The matrix game is proposed by QTRAN Son et al. (2019), where two agents with three actions and shared reward as illustrated in Table1, should learn to cooperate to find the optimal joint action (A,A). This is a simple example of the relative overgeneralization problem, where the sub-optimal action B,C has higher expected return in exploration process. We train all algorithms through a full exploration (i.e., = 1 in -greedy) conducted over 20,000 steps while FSV is trained by annealing α from 1 to α0. To demonstrate the expressive ability related to temperature parameter α, we set α0 = 1, 0.1, 0.01 respectively. As shown in Table3, QMIX fails to represent the optimal joint action
Table 1: Payoff of matrix game
u1 u2 A B C
A 8 -12 -12 B -12 0 0 C -12 0 0
Table 2: QTRAN-alt
Q1 Q2 3.3 0.1 0.1
4.7 8.0 -12.0 -12.0 -0.1 -12.0 0.0 0.0 -0.1 -12.0 0.0 0.0
Table 3: QMIX
Q1 Q2 -5.6 0.1 0.1
-6.6 -8.1 -8.1 -8.1 0.2 -8.1 0.0 0.0 0.1 -8.1 0.0 0.0
Table 4: FSV, α0 = 0.01
Q1 Q2 3.3 -0.7 -0.0
4.7 8.0 -12.0 -12.0 0.7 -12.0 0.0 0.0 0.7 -12.0 0.0 0.0
Table 5: FSV, α0 = 0.1
Q1 Q2 7.2 -0.5 -0.5
0.9 8.0 -11.8 -11.7 0.5 -11.8 -0.0 0.0 0.5 -11.8 -0.0 0.0
Table 6: FSV, α0 = 1
Q1 Q2 0.3 0.1 0.1
7.7 8.0 -9.4 -4.9 0.0 3.8 0.1 -0.0 -0.1 3.8 -0.1 -0.0
value and the optimal action due to the limitation of additivity and monotonicity structures while FSV and QTRAN successfully represent all the joint action values. In addition, even if α is not annealed to very small, FSV correctly approximated the optimal joint action values because we directly estimate λ when α and αi tend to 0, which relaxes the constraints of the function class to guarantee the correct structure during the training process.

heading: 5.2 MAX OF TWO QUADRATICS GAME
text: We use The Max of Two Quadractics game (Wei et al. (2018)), which is a simple single state continuous game for two agents, to demonstrate the performance of current algorithms in the relative overgeneralization problem. Each agent has one dimensional bounded action with shared reward as following  f1 = h1 × [−(u1−x1s1 ) 2 − (u2−y1s1 ) 2] f2 = h2 × [−(u1−x2s2 ) 2 − (u2−y2s2 )
2] + c r(u1, u2) = max(f1, f2)
(19)
where u1,u2 are the actions from agent 1 and agent 2 respectively, h1 = 0.8, h2 = 1, s1 = 3, s2 − 1, x1 = −5, x2 = 5, y1 = −5, y2 = 5, c = 10. The reward function is shown as Fig 3(a). Although this game is very simple, the gradient points to the sub-optimal solution at (x1, y1) over almost all the action space which will fox the policy-based method. And for value function factorization methods, this task requires non-monotonic structures to correctly represent the optimal joint Q-values through individual Q values. We extend QMIX and VDN to actor-critic framework (like DDPG) while QTRAN is not applicable in continuous action space due to its requirement of max operations on Q-values.
Fig 3(b) is the training result averaged over 20 experiment runs and Table 7 gives a more detailed result, where MADDPG and QMIX happened to find the optimal actions due to random initialization twice. VDN never find the optimal actions and even fails to find the sub-optimal 4 times. These
results indicate that, a more explorative policy and correct estimation of Q-values are both needed to overcome the relative overgeneralization problem. Using a centralized critic like MADDPG to guide the decentralized actors will mislead the policy gradients because it averages the Q-values based on others’ policies (?). Using individual Q-values to guide actors requires the full expressive ability of factorizable tasks where QMIX and VDN fail to estimate individual Q-values correctly due to the structural limitation as shown in Sec5.1 and QTRAN losts its tractability for continuous tasks. To enable better exploration in joint action space, Wei et al. (2018) adopt multi-agent soft q-learning to avoid the relative overgeneralization problem, but it still uses a centralized critic which suffers scalability and it’s very sensitive to how the temperature parameter anneals. It’s clear that, FSV utilizes value function factorization method to get correct estimation of individual Q-values and carries exploration with a more explorative energy-based policy can achieve 100% success rate.

heading: 5.3 STARCRAFT II
text: We choose a challenging set of cooperative StarCraft II maps from the SMAC benchmark (Samvelyan et al. (2019)). Our evaluation procedure is similar to Samvelyan et al. (2019), where the training process is paused every 100000 time steps to run 32 evaluation episodes with decentralised greedy action selection. We compare FSV with VDN, QMIX and QTRAN on several SMAC maps. Here we present the results for Easy map 2s3z, Hard map 3s vs 5z and Super Hard map MMM2, which is classified by Samvelyan et al. Fig 4 shows the test win rate averaged over 5 experiment runs for the different algorithms on the maps. FSV achieves state-of-the-art due to the high representation ability and exploration efficiency

heading: 6 CONCLUSION
text: In this paper, we proposed a new definition of factorizable tasks with stochastic policies named IGO. Then we introduced FSV, a novel MARL algorithm under IGO, which learns to factorize soft value function into individual ones for decentralized execution enabling efficient learning and exploration through maximum entropy reinforcement learning. As immediate future work, we aim to develop a theoretical analysis for FSV as a policy-based method. We would also like to explore the committed exploration like Mahajan et al. (2019) in continuous space due to the miscoordination caused by energy-based policy (Wei & Luke (2016)).

heading: 7 APPENDIX

heading: 7.1 PROOF

heading: 7.1.1 RELATIONSHIP BETWEEN IGO AND IGM
text: If the joint and individual optimal policies are greedy:
πi(ui|τi) =
{ 1, ui = arg max
ui
Qi(τi, ui)
0, otherwise (20)
π(u|τ) =
{ 1, u = arg max
u Q(τ, u)
0, otherwise (21)
Then IGO gives that u = arg max u Q(τ, u) if and only if ui = arg max ui Qi(τi, ui) for any i which is equivalent to IGM.

heading: 7.1.2 PROOF OF THEOREM1
text: Theorem 1 If the task satisfies IGO, with energy-based optimal policy, the joint action value Qtot can be factorized by individual action values [Qi]Ni=1 as following:
Qtot(τ, u) = N∑ i=1 λ∗i [Qi(τi, ui)− Vi(τi)] + Vtot(τ) (22)
where λ∗i = α/αi.
Proof. Considering Eq(4) and Eq(5), IGO can be reformulated as:
exp( 1
α (Qtot(τ, u)− Vtot(τ))) = N∏ i=1 exp( 1 αi (Qi(τi, ui)− Vi(τi))) (23)
This gives:
Qtot(τ, u) = N∑ i=1 α αi [Qi(τi, ui)− Vi(τi)] + Vtot(τ) (24)
which is Theorem 1.

heading: 7.1.3 PROOF OF THEOREM2
text: Theorem 2 When α, αi → 0, the function class defined by IGM is equivalent to the following
Qtot(τ, u) = N∑ i=1 λi(τ, u)[Qi(τi, ui)− Vi(τi)] + Vtot(τ) (25)
where λi(τ, u) = lim α,αi→0
λ∗i .
Proof. IGM⇒Eq(7): It’s clear that Eq(7) can always hold if λi is well constructed. Here we give one way to construct λi which meanwhile explains how we extend function class limited by energy-based policy.
Denote πi, πtot to be the current policy and π∗i , π ∗ tot to be the optimal policy. We can always take individual optimal policies during the process of approaching greedy, thus
πi(ui|τi) = exp( 1
αi (Qi(τi, ui)− Vi(τi))) =
{ 1− , ui = arg max
ui
Qi(τi, ui)
, otherwise (26)
where is a small parameter. Then, the joint policy is given by: πtot(u|τ) = {
(1− )n, u = [ui]ni=1 1− (1− )n, otherwise (27)
Considering IGM, u = arg max u
Qtot(τ, u). Then πtot = π∗tot = exp( 1 α (Qtot(τ, u)− Vtot(τ))) and
α, αi is a function of observations and actions:
αi(τi, ui) = Qi(τi, ui)− Vi(τi)
log(1− ) , ui = arg max
ui
Qi(τi, ui) (28)
α(τ, u) = Qtot(τ, u)− Vtot(τ)
n log(1− ) , u = arg max u Qtot(τ, u) (29)
Thus, λi is given by:
λi(τ, u) = Qtot(τ, u)− Vtot(τ) n(Qi(τi, ui)− Vi(τi))
(30)
This makes the Eq(7) permanent. In particular, if sampled action u is exactly current arg max u Qtot, then Qtot = Vtot and Qi = Vi when α, αi → 0 and λi can be set to 1. Thus we extend the function class and Eq(7) always holds for any action.
IGM⇐Eq(7): denote u∗i = arg max
ui
Qi(τi, ui) and u∗ = arg max u Q(τ, u). Remember
Vtot(τ) = α log ∫ UN exp( Qtot(τ, u) α )du ≥ Qtot(τ, u) (31)
Vi(τi) = αi log ∫ U exp( Qi(τi, ui) αi )du ≥ Qi(τi, ui) (32)
Therefore, Qtot = Vtot and Qi = Vi if and only if u = u∗ and ui = u∗i , respectively. Considering Eq(7) and λi > 0, u = u∗ if and only if ui = u∗i , that complete the proof.

heading: 7.2 IMPLEMENTATION DETAILS
text: In discrete tasks, We follow the PyMARL Samvelyan et al. (2019) implementation of VDN, QMIX and QTRAN, where the hyper-parameters of are the same in SMAC Samvelyan et al. (2019). We illustrate the special hyper-parameters of FSV in Table 8 and others are the default settings in PyMARL. In discrete tasks, we extend VDN and QMIX to the actor-critic framework. Specifically, we add an actor for individual agents to maximise the Q-values from their own critic like DDPG. We illustrate the hyper-parameters of these algorithm as well as FSV in Table 8. For stability, we reformulate the Eq(7) as following:
Qtot(τ, u) = N∑ i=1 λi(τi, ui)(Qi(τi, ui)− Vi(τi)) + Vtot(τ)− N∑ i=1 Qi(τi, ui) + N∑ i=1 Qi(τi, ui) (33)
Then we stop all the gradients except for λi and the last term Qi. Considering incorrect weight λi will cause incorrect αi at the beginning of training thus obstruct the exploration, we use αi = α , which means we ignore the KL-divergence between current policy and optimal policy. The only temperature parameter α is updated through annealing or Automating Entropy Adjustment in Haarnoja et al. (2018b) as following:
Jα = Eu∼π[−α log π(u|τ)− αH̄] (34)
where the H̄ is the target entropy.

Review summaries:
This paper proposes a novel MARL framework named FSV, which incorporates the idea of energy-based policies and an efficient linear decomposition architecture in the joint action-value function with multi-agent maximum entropy reinforcement learning. Besides, the authors propose the IGO, which extends the IGM in stochastic policy cases. FSV suits in both the discrete and continuous action space scenarios. Experiments conducted on two simple examples with discrete and continuous action settings show that FSV could overcome the relative overgeneralization problem with the proper temperature setting. Furthermore, FSV in the challenging SMAC benchmark outperforms VDN, QMIX, and QTRAN in three scenarios.

The paper proposes a Q-factorization method by assuming an energy-based policies model. Q-functions are formulated as soft value functions with the energy parameters, and this adoption renders the function factorization more flexible compared to existing ones. The proposed solution applies to continuous-action tasks, a feat left unconquered by some of the existing methods. Authors exhibit that FSV outperforms others in various environments characterized by local optima.