FSV: Learning to Factorize Soft Value Function for Cooperative Multi-Agent Reinforcement Learning
Authors: Yueheng Li
Tianhao Zhang
Chen Wang
Jinan Sun
Shikun Zhang
Guangming Xie
Venue:  (2021)
Keywords: cooperative MARL
value function factorization
stochastic policy
continuous tasks

Abstract: We explore energy-based solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factorization methods are susceptible to the relative overgeneralization, where finding a suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology. To resolve this issue, we propose a novel factorization method for cooperative MARL, named FSV, which learns to factorize the joint soft value function into individual ones for decentralized execution. Theoretical analysis shows that FSV solves a rich class of factorization tasks. Our experiment for the well-known task of the Max of Two Quadratics game shows that FSV fully converges to global optima in the joint action space in the continuous tasks by local searching in the joint action space. We evaluate FSV on a challenging set of StarCraft II micromanagement tasks, and show that FSV significantly outperforms existing factorization multi-agent reinforcement learning methods.

Under review as a conference paper at ICLR 2021
FSV: LEARNING TO FACTORIZE SOFT VALUE FUNC- TION FOR COOPERATIVE MULTI-AGENT REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
We explore stochastic-based policy solutions for cooperative multi-agent rein- forcement learning (MARL) using the idea of function factorization centralized training with decentralized execution (CTDE). Existing CTDE based factoriza- tion methods are susceptible to the relative overgeneralization, where finding a suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology. To resolve this issue, we propose novel factorization method for cooperative MARL, named FSV, which Icarns to factorize the joint soft value function into individual ones for decentralized execution. Theoretical analysis shows that FSV solves rich class of factorization tasks. Our experiments for the well-known tasks of the Non-Monotonic Matrix game and the Max of Two Quadratics game show that FSV converges optima the joint action space in the discrete and contin- uous tasks by local searching. We evaluate FSV on challenging set of StarCraft II micromanagement tasks, and show that FSV significantly outperforms existing factorization multi-agen reinforcement learning methods.
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) aims to instill in agents policies that max- imize the team reward accumulated over time (Panait & Luke (2005); Busoniu al.] (2008): [Tuyls) & Weiss (2012), which has great potential to address complex real-world problems, such coor- dinating autonomous cars (Cao et al. (2013). Considering the measurement and communication limitations in practical problems, cooperative MARL faces the partial observability challenge. That is, each agent chooses actions just based on its local observations.
Centralized training with decentralized execution (CTDE) (Oliehock al. (201T) is common paradigm to address the partial observability, where agents' policies are trained with access to global information in centralized way and executed only based local observations in decentralized way, such as the MADDPG (Lowe (2017)) and COMA (Foerster et al.] (2017)). However, the size of the joint state-action space of the centralized value function grows exponentially the number of agents increases, which is known as the scalibility challenge.
Value function factorization methods have been an increasingly popular paradigm for solving the scalability in CTDE by satisfying the Adiivddal--Global-aae (IGM) where the optimal joint action selection should be consistent with the optimal individual action selections. Three representative examples of value function factorization methods include VDN (Sunehag et al. (2017). QMIX (Rashid et al. (2018). and QTRAN (Son et al. (2019)). All these methods are e-greedy policies, where VDN and QMIX give sufficient but unnecessary conditions for IGM by additivity and mono- tonicity structures respectively, and the QTRAN formulates the IGM as an optimization problem with linear constraints.
Although these methods have witnessed some success in some tasks, they all face relative overgen- eralization, where agents may stick into a suboptimal Nash Equilibrium. In fact, relative overgener- alization is a grave pathology arising which occurs when a suboptimal Nash Equilibrium in the joint space of action priors to optimal Nash Equilibrium since each agent's action in the suboptimal equilibrium is a better choice (Wei & Luke (2016)). The non- monotonoo matrix game is a sim- ple discrete example. Both VDN and QMIX fail to learn the optimal policy in the non-monotonic

---

Under review as a conference paper at ICLR 2021
matrix due to their structure limitation. Although QTRAN expresses the complete value function representation ability in the non-monotonic matrix, its full expressive ability decreases in the com- plex tasks due to the computationally intractable constraints relaxing with tractable L2 penalties. Besides, QTRAN sacrifices the tractability in continuous action space. Therefore, in discrete and continuous tasks, achieving effective scalability while avoiding relative overgeneralization remains an open problem for cooperative MARL.
To address this challenge, this paper presents new definition of factorizable tasks called IGO Uhddiiddull-GoobllOOPP -Otiiaaa-- which introduces the consistency of joint optimal stochastic policies and individual optimal stochastic policies. Theoretical analysis shows that IGO degenerates into IGM if the policy is greedy. which represents the generality of IGO. Under the IGO, this paper proposes novel Factorization solution for MARL,, named FSV, which learns to factorize soft value function into individual ones for decentralized execution enabling efficient learning and exploration through maximum entropy reinforcement learning. To our best knowledge, FSV the first multi- agent algorithm with stochastic policies using the idea of factorization, and theoretical analysis shows that FSV solves a rich class of tasks.
We evaluate the performance of FSV in both discrete and continuous problems proposed by Son jet al. (2019;; Wei et al. 2018) and a range of unit micromanagement benchmark tasks in StarCraft II. The Non-Monotonic Matrix game shows that FSV has full expression ability in the discrete task, and the Max of Two Quadratics game shows that FSV is the first factorization algorithm that avoids the relative overgeneralization
converge to optima in the continuous task. On more challeng- ing StarCraft II tasks, due to the high representation ability and exploration efficiency of FSV, it significantly outperforms other baselines, SMAC (Samvelyan al. (2019)).
2 PRELIMINARIES
2.1 DEC-POMDP AND CTDE
A fully cooperative multi-agent task can be described as a Dec-POMDP defined by a tuple g (S,U,P, rr, Z, O,NV, where S is the global state of the environment. Each agent i E / choose an action u, E ll at each time step, forming a joint action u E UN, This causes a transition to the next state according to the state transition function P(s'Is, u) S uV x - [0, 1] and reward function r(s, u) S x 4N + R shared by all agents. 7 E [0. 1] is a discount factor. Each agent has individual, partial observation = Z according to observation function O(s, S x N 2. Each agent also has an coserviobservation history T, E T : (Z x U)", on which it conditions a stochastic policy <u(uTTT)) T U [0, 1]. The joint policy has joint action-value function k= T uz).
Q*(ss. (st. u.) M444100,M4110 E
Centralized Training with Decentralized Execution (CTDE) is a common paradigm of cooperative MARL tasks. Through centralized training,the full state can made accessible to all agents. This allows agents to learn and construct individual action-value functions correctly while selecting actions based on its own local action-observation history at execution time.
histories of all agents and the
2.2 VDN, QMIX AND QTRAN
An important concept for factorizable tasks is IGM which asserts that the joint aation-value function
Qiot: T x UN + and individual action-valus value functions [Q, T x U - R satisfies
arg maxQror (T, u) = (arg maxQ1 (Ti u1), ..., arg maxQ un)) ux
(1)
d1
To this end, VDN and QMIX give sufficient conditions for the IGM by additivity and monotonicity structures, respectively, as following:
Qrae(T, u) Σ Ei(T,, and (Quor(T, > V aQ u:)
(2)
However, there exist tasks whose joint action-valt value functions do not meet the said conditions. where VDN and QMIX fail to construct individual action-valie function correctly. QTRAN uses a lin- constraint between individual and joint action values to guarantee the optimal decentralisation.
2

---

Under review as a conference paper at ICLR 2021
To avoid the intractability, QTRAN relax these constraints using two L2 penalties. However, this relaxation may violate the IGM and it has poor performance on multiple multi-agent cooperative benchmarks as reported recently.
2.3 THE RELATIVE OVERGENERALIZATTON PROBLEM
Relative overgeneralization occurs when a sub-optimal Nash Equilibrium (e.g. N Fig. 1) in joint action space is preferred over optimal Nash Equilibrium (e.g. M in Fig. 1) because each agent's action the suboptimal equilibrium a better choice when matched with arbitrary actions from the collaborating agents. Specifically, shown in Figure where bounded action (or three actions in discrete action space) try to cooperate and find the optimal joint action, the action B (or C)) is often preferred by most algorithms as mentioned in (Son et al. (2019) and Wei et al. (2018) due to their structure limitation and lack of exploration.
agents with one-dimensional
A
C
A
8
-12
B
-12
0
-10
C
-12
0
(R)
-10
(b)
Figure 1: The relative overgeneralization in discrete (a) and continuous (b) action space
3
METHOD
In this section, we will first introduce the IGO (Individual-Gion -ai----Golaa-llaa -aabba--pttiaa)..... a new definition of fac- torizable MARL tasks with stochastic policies. Theoretical analysis shows that IGO degenerates into IGM the policy is greedy. With the energy-based -based policy, the structure between joint and individual action values of IGO can be explicitly constructed, which is novel factorization stochastic- -bsed policy solution we proposed, named FSV. Specifically, FSV realizes IGO using an efficient linear structure and learns stochastic policies through maximum entropy reinforcement learning.
3.1 INDIVIDUAL GLOBAL OPTIMAL
In the CTDE paradigm, each agent E N chooses an action based on a stochastic policy me Te(ug(T)) at the same time step. The joint policy Teoe (ulr) = I (u:u/T;;) describes the probability of taking joint actions u on joint observation history т. If cach agent adopts its optimal policy while the joint policy is exactly the optimum, the task itself can achieve global optimum through local optimum, which naturally motivates us to consider the factorizable tasks with stochastic policy as following:
Definition For joint optimal policy Tt (u(r) TN x UN - 10, 1}, if there exists individual [0, i=11 such that the following holds
optimal policies [x]] (wg(TT) T
Tioe (u|T)=
II m (udTT)
then, we say that [T:] satisfy IGO for Ttot
As specified above, IGO requires the consistency of joint optimal policy and individual optimal policies rather than the actions in IGM, but degenerates into IGM if policies are greedy. That is to say, IGO is more generality than IGM.
3.2 FSV
In this work, we take the energy- based policies as joint and individual optimal policy respectively,
Tim (u|T) = exp( (Qtoe(T, u) Vioe(T)))
(4)

---

Under review as a conference paper at ICLR 2021
(u,l)) expl - (Qi(Ti, Vi(T,))) ai
(5)
where a, aj are temperature parameters, Vio((T) log SuN exp( Qtoe(T, u))du and Vi(Ts) ag log So erp( oe (T du are partition functions.
The benefit of using energy-basee policy is that it is a very general class of distributions that can represent complex, multi-modal behaviors Haarnoja et al. (2017). Moreover, energy-based policies can easily degenerate into greedy policies as a, ay anneals.
To learn this decentralized energy-based policy, we extend the maximum entropy reinforcement learning framework for the multi-agent setting, which we'll describe the next. Another benefit of considering the stochastic policy with explicit function class for factorizable tasks through IGO is that the architecture between joint and individual action values can be easily constructed through its constrains on policies with specific meanings as follows.
Theorem If the task satisfies IGO, with energy-baser optimal policy, the joint action value Quot N as following:
can be factorized by individual action values [Qu]
QT, - Σ XOQ(T, Vi(T:)) - Veoe(T)
(6)
where X; =a/44
Theorem gives the decomposition structure like VDN-the joint value is linear combination of individual values weighted by X > O. However, the function class defined by Eq which should only concern the task itself, related to and limited by the distributions of policy. Although energy- based distribution is very general which has the representation ability of most tasks, to establish the correct architecture between joint and individual Q-values and enable stable learning, we need to extend the function class into any distributions. The key idea is that we approximate the weight vector X directly as a, ag is zero instead of annealing as during training process. This extends the function class and will at least guarantee IGM constraint when a, is zero
Theorem When a₁ + O, the function class defined by IGM is equivalent the following
Qr = Σ AeT, uQQitt) V3(T:)] Vio((T)
where \r(T, u) lim X;
o,,,,
Note that A; is now a function of observations and actions due to the relaxation. Eq allows to use a simple linear structure to train joint and individual action values efficiently and guarantee the correct estimation of optimal Q-values. We'll describe it in experiment.
Then, we introduce the maximum entropy reinforcement learning in CTDE setting which is an directly extension of soft actor-critic (q-learning).
The standard reinforcement learning tries to maximum the expected return E, Ex [re], while the maximum entropy objective generalizes the standard objective by augmenting with an entropy term, such that the optimal policy additionally aims to maximize its entropy at each visited state
Mar = arg max
Exlre (aH
where a the temperature parameter that determines the relative importance of the entropy term versus the reward, and thus controls the stochasticity of the optimal policy (Haarnoja et al. (201))
We can extend into cooperative multi-agent tasks by directly considering the joint policy Teat u T) and defining the soft joint action-value function as following:
Qroe 44) = r(Te, ue) + ET......
y (rets aH (1744)))]
(9)
4

---

Under review as a conference paper at ICLR 2021
then the joint optimal policy for Eq(8) is given by Eq (Haarojja al. (2017)). Note that
don't start considering decentralized policies, the joint O-fucction should satisfy the soft Bellman equation:
(Io u4) re + Bri43 [Toe(T++2))
(10)
And we can update the joint functions in centralized training through soft Q-iteration:
(Te, u4) + rt Er (Vior(Te+1))
(11)
It's natural take the similar energy-based distribution as individual optimal policies nं in 5 which allows us to update the individual policies through soft policy-iteration:
arg minD (f
(12)
r'ell
3.3 ARCHITECTURE
In this section, we present a novel MARL framework named FSV, which incorporates the idea in simple and efficient architecture through Eq(7) with multi-agent maximum entropy reinforcement learning. FSV can be applied both in continuous action space and also in discrete action space as simplification.
14
Qut
Que
Incorporation part
V network
product
network work
$
Viee
Q
Vn
Qx
Medgltted sam
ML.
Policy nełwork
Agent
A gent
Individual part
En
10.12
[Va
Figure 2: FSV network architecture
Figure 2 shows the overall learning framework, which consists of two parts:(i) individual parts for each agent i, which represents Q,, V and n. (i)jincopporation part that composes Q. Vi to Qiot-
Individual parts for each agent i has three networks: (i)individual Q network takes its own action and observation history ug as input and produces action-values Qi(T;, u:) as out- put. (i)iiddivii value network takes its own observation history : as input and produces Vi(Ti) as output. (ii)iinividual policy network takes its own observation history т, input and produces distribution (e.g. mean and standard deviation of Gaussian distribution) for sample actions.
Incorporation part composes Qi, Vi to Qtat through linear combination. Specifically, it sums up IQ; VIl with coefficients and uses one-layer hyper-network efficiently approximate the high-dimensional partition function as following:
Viot (T) = Σ (T))))()))+(())))
(13)
where w,, b is a positive weight and bias respectively. To enable efficient learning, we adopt a multi-head attention structure estimate the weight vector:
H
>e(T, u) = Σ (T,
(14)
5

---

Under review as a conference paper at ICLR 2021
where H is the number of attention heads and Ar,h is defined by
A exp(e, W W g.,)
(15)
where eu and is obtained by two-layer embedding transformation for and 8. The joint action value function Qiot updated through soft Q-iteration:
u.) ((T,, we)]
(16)
Qeat
where Q(Te, uz) = r(Te, un) 7 I D,D 1w (oo "(+1) log Toot (4444)))411]
The individual value network is trained by minimize
M D(V(((((TT)) (ilT, u.) a log (u,Tr)))
(17)
The policy network of each agent is trained by minimizing the expected KI diveereeee
1 Er,,D,,urr,, (a log n. (u. Qi( w,]]
(18)
For discrete action space, it's convenient to simplify this framework to Q-learning. Specifically, we directly compute the individual value function V ag log Σ exp( J, (T.,.))) instead of updating the value network, and action distributions directly produced by Eq(5) instead of the policy network.
4 RELATED WORK
There are many early works with maximum entropy principle such as Todorov 2010) and Levine & Koltun (2033) it in policy search in linear dynamics and Kappen (005) and A. Theodorou al. (2010) use it in path integral control in general dynamics. Recent off policy methods (Haarnoja et al. 2017 Haarnoja al. (2018b) (0188)/Haame Haarnoja et al. (2018a)) have been proposed learn an energy-based policy efficiently through the maximum entropy objective which is adopted in our framework. Value function factorization methods start from VDN (Sunehag et al. (2017)) and is extended by QMIX Rashid et al. (2018)) and QTRAN (Son et al. (019H). Other methods such as QATTEN (Yang et al. 2020)) and MAVEN (Mahajan et al. (2019) go a step further on architecture and exploration. Our method are member of them but out of the deterministic policy
Current methods adopt different ideas to solve the relative overgeneralization problem. Wei et al. (2018) conduct multi-agent soft learning for better exploration. Wen et al. (2019) uses proba- bilistic recursive reasoning to model the opponents, Yu et al. (2019) adopts inverse reinforcement learning to avoid this problem through right demonstrations, Tian et al. (2019) derives variational lower bound of the likelihood of achieving the optimality for modeling the opponents. However, none of them adopt value function factorization like FSV which means they suffer the scalability problem.
5 EXPERIMENTS
In this section, we first consider two simple examples proposed by prior work (Son et al. (2019) Wei et al.] (2018)) demonstrate the optimality and convergence of FSV in discrete and continuous action space respectively. And we evaluate the performance a challenging set of cooperative StarCraft II maps from the SMAC benchmark (Samvelyan al. (2019)).
5.1 MATRIX GAME
The matrix game is proposed by QTRAN Son et al. (2019), where two agents with three actions and shared reward as illustrated in Table] should learn to cooperate to find the optimal joint action (A, A). This is simple example of the relative overgeneralization problem, where the sub-optimal action B, C has higher expected return in exploration process. We train all algorithms through full exploration (i.e.. from to ag. To demonstrate the expressive ability related to temperature parameter a, we set og = 0.1. 0.01 respectively. As shown in Table QMIX fails to represent the optimal joint action
in e-greedy) conducted over 20,000 steps while FSV is trained by annealing

---

Under review as a conference paper at ICLR 2021
Table 1: Payoff of matrix game
Table 2: QTRAN-alt
Table 3: QMIX
W2
A
B
C
Q2
3.3
0.1
0.1
Q2
-5.6
0.1
0.1
41
Q1
Q1
A
8
-12
-12
4.7
8.0
-12.0
-12.0
-6.6
-8.1
-8.1
-8.1
B
-12
-0.1
-12.0
0.0
0.0
0.2
-8.1
0.0
0.0
C
-12
0
-0.1
-12.0
0.0
0.0
0.1
-8.1
0.0
0.0
Table 4: FSV, og = 0.01
Table 5: FSV, og = 0.1
Table 6: FSV, og = 1
Q2
Q2
Q2
3.3
-0.7
-0.0
7.2
-0.5
-0.5
0.3
0.1
0.1
Q
Q1
Q1
8.0
-12.0
-12.0
8.0
-11.8
-11.7
7.7
-9,4
.4.9
0.7
-12.0
0.0
0.0
0.5
-11.8
-0.0
0.0
0.0
3,8
0.1
-0.0
0.7
-2.00
0.0
0.0
0.5
-11.8
-0.0
0.0
-0.1
3.8
-0.1
-0.0
value and the optimal action due to the limitation of additivity and monotonicity structures while FSV and QTRAN successfully represent all the joint action values. In addition, even if a is not annealed to very small, FSV correctly approximated the optimal joint action values because we directly estimate A when a and a; tend to which relaxes the constraints of the function class to guarantee the correct structure during the training process.
5.2 MAX OF TWo QUADRATICS GAME
We use The Max of Two Quadractics game (Wei et al. (2018), which a simple single state con- tinuous game for two agents, to demonstrate the performance of current algorithms in the relative overgeneralization problem. Each agent has one dimensional bounded action with shared reward as following
fi = h1 [- 2 (" )21 f== h2 [- (1- 22
59 (22- r(u,, u2) max f2)
82 +
(19)
where U1 are the actions from agent and agent 2 respectively, h1 0.8, h2 1, 81 3, $2 1, Ø1 = -5, Ø2 = 5, 8/1 = -5 3/2 = 5, c = 10. The reward function shown as Fig 3(a). Although this game is very simple, the gradient points the sub-optimal solution at (x1, over almost all the action space which will fox the policy-based method. And for value function factorization methods, this task requires non-monotonic structures to correctly represent the optimal joint Q-values through individual Q values. We extend QMIX and VDN to actor-critic framework (like DDPG) while QTRAN is not applicable in continuous action space due to its requirement of max operations on -aluue.
NADOFO
optimie
0.2
-0.25
Sub oprimuz
-1.33
10
25
30
apiiade (8)
(a)
(b)
Figure 3: Max of Two Quadratics ga)eww)) function, (b)average reward for FSV,VDN.QMIX and MADDPG
Fig 3(b) is the training result averaged over 20 experiment runs and Table gives more detailed result, where MADDPG and QMIX happened to find the optimal actions due to random initialization twice. VDN never find the optimal actions and even fails to find the sub-optimal 4 times. These
7

---

Under review as a conference paper at ICLR 2021
Table 7: training result for Max of Two Quadratics game
opt
sub-opt
other
FSV
20
0
0
MADDPG
2
18
0
QMIX
2
18
VDN
0
16
4
results indicate that, more explorative policy and correct estimation of Q-values are both needed to overcome the relative overgeneralization problem. Using centralized critic like MADDPG to guide the decentralized actors will mislead the policy gradients because averages the Q-values based on others' policies (?). Using individual Q-values to guide actors requires the full expressive ability of factorizable tasks where QMIX and VDN fail to estimate individual Q-values correctly due to the structural limitation as shown in Sec5.1 and QTRAN losts its tractability for continuous [
tasks. To enable better exploration in joint action space, |eei et al. (2018) adopt multi-agent soft q-learning to avoid the relative overgeneralization problem, but it still uses centralized critic which suffers scalability and it's very sensitive to how the temperature parameter anneals. It's clear that, FSV utilizes value function factorization method to get correct estimation of individual Q-values and carries exploration with a more explorative energy- based policy can achieve 100% success rate.
5.3 STARCRAFT II
We choose a challenging set of cooperative StarCraft II maps from the SMAC benchmark (Samvelyan al. (2019). Our evaluation procedure is similar to (Samvllyan et al. (2019). where the training process is paused every 100000 time steps to run 32 evaluation episodes with decentralised greedy action selection. We compare FSV with VDN, QMIX and QTRAN on several SMAC maps. Here we present the results for Easy map 2s3z, Hard map 3s_us_5 _52 and Super Hard map M M2, which is classified by Samvelyan al. Fig 4 shows the test win rate averaged over 5 experiment runs for the different algorithms on the maps. FSV achieves state-of-the- due to the high representation ability and exploration efficiency
Drant
0TAW
(TAM
0.
<gfonde
33
<ghodd
CajEasy 2x
(b]lard 5z
(c)Super Hard MMME
Figure 4: test win rate of FSV, VDN, QMIX and QTRAN
CONCLUSION
In this paper, we proposed new definition of factorizable tasks with stochastic policies named IGO. Then we introduced FSV, a novel MARL algorithm under IGO, which learns to factorize soft value function into individual ones for decentralized execution enabling efficient learning and exploration through maximum entropy reinforcement learning. As immediate future work, we aim to develop a theoretical analysis for FSV a policy-based based method. We would also like to explore the committed exploration like Mahajan et al. (2019) in continuous space due to the miscoordination caused by energy-based policy (Wei & Luke (2016)).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Evangelos A. Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach reinforcement learning. Journal of Machine Learning Research, 11(11::1177-311, 2010.
L. Busoniu, R. Babuska, and B. De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems Man Cyhernetics Part C, (..38)))16661777 2008.
Y. Cao, W. Yu, W. Ren, and G. Chen. An overview recent progress the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics, 9()):277-43, 2013.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.
Counterfactual multi-agent policy gradients. CoRR, abs/1705.08922, 2017. URL http://
arxiv.org/ abs////b//aa 705. 08926
Tuomas Haarojj, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with
deep energy-based policies. CoRR, abs/1702.08165. 2017. URL htllhtttp:/axxiv.orgg org/abs/
1702.08155
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs//801.011 2018a. URL
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05905, 2018b. URL http:::/arx....rgg///////
1812.05900
H Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statis- tical Mechanics: Theory and Experiment, 1011(11:P110111-11000 Nov 2005. ISSN 1742- 5468. doi: 72008//742-54//0005///11/1
URL http://dx doi wwg.r0.g/1000 100///0..88////0
1742 54/64466/2200 /144602220511/1111
Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimiza- tion. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems 26, pp. 207- 215. Curran Associates, Inc., 2013. URL htttp:/ www. http:ersssiisssceppppeeeeeeeeeee cctppperrcpppp////eepppp 5178-vars 7778-variational. wtt.::/.......o.....aaaaaaaaaaaa
pdf
Ryan Lowe. Multi-agent actor-critic for mixed cooperative-comperiine environments. 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent
variational exploration. 2019.
Frans A. Oliehoek, Matthijs T. Spaan, and Nikos A. Vlassis. Optimal and approximate v-value functions for decentralized pomdps. CoRR, abs/1 111.0062, 2011. URL Rtllatttp//axxx..oom
abs/1111. 0062
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous
Agents and Multi-Agent Systems, 11(3):387-434, November 2005. ISSN 1387-2532. doi: 10.
URL org. 11t.0.0000011100 005 2631-2222222
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR,
2019, URL http://arxiv.
org/abs/1902 04043
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. 2019.

---

Under review as a conference paper at ICLR 2021
Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czameeki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, and Karl and Tuyls. Value- decomposition networks for cooperative multi-agent learning. 2017.
Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. CoRR, abs/1905.08087, 2019. URL http:
Emanuel Todorov. Policy gradients in linearlysolvable mdps. Advances in Neural Information Processing Systems, pp. 2298-2306, 2010.
Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. Ai Magazine, 33(3):41-52, 2012.
Ermo Wei and Sean Luke. Lenient learning in independent-leare stochastic cooperative games.
Journal of Machine Learning Research, 17(84):1-42, 2016. URL. http:: htmmmmmlrrooo
papers/v17/15-417 .tmml
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. CoRR, abs/1804.09817, 2018.
//ap:://rrree .rgg abs/aaas//1/ 09817
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. CoRR, abs/1901.09207, 2019. URL http::/arxiv.
org/abs/1901. 1a.bb//1901.00 09207
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. 2020.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learn- ing. 2019.
APPENDIX
7.1 PROOF
7.1.1 RELATIONSHIP BETWEEN IGO AND IGM
If the joint and individual optimal policies greedy:
1, u, arg max u:)
T (u:(T:)
(20)
0, otherwise
1,
arg magga((TTe))
x(ul7)
(21)
0. otherwise
Then IGO gives that arg argmaa((T. u) if and only if u; arg magga u.) for any which is equivalent to IGM.
7.1.2 PROOF OF THEOREM
Theorem 1 If the task satisfies IGO, with energy-based optimal policy, the joint action value Qtot can be factorized by individual action values [Q:1m=, N following:
N
Quoe(T, = Σ A[Q((T. - Vs(T.)] - Vioe(T)
(22)
where X a/oi.
10

---

Under review as a conference paper at ICLR 2021
Proof. Considering Eq and Eq(5), IGO can be reformulated as:
exp( 1 (Qror u) Vior(T)))
N
1
II
exp (Qi(T, Vi(T.)))
(23)
This gives:
N
u)) = Σ
a
Qq((T, u.) Vi(T:)] + Vioe(T)
(24)
which is Theorem 1
7.1.3 PROOF THEOREM2
Theorem 2 When a, a; - O, the function class defined by IGM is equivalent to the following
Qtor(T, Σ u((ii)) u.) V.(T:)] Vioe(T)
(25)
where Ae(T, u) lim x-- a,a00
Proof. IGM-EE
It's clear that 7) always hold if A is well constructed. Here we give way to construct A which meanwhile explains how we extend function class limited by energy-based policy.
Denote T; tot be the current policy and x;, Tfot be the optimal policy. We can always take individual optimal policies during the process of approaching greedy, thus
(u, = exp( aj
(Q,(To u,) Vi(Ti)))
6,
Us arg magQi((Ts uz)
", otherwise
(26)
C,
where is small parameter. Then, the joint policy is given by:
toot ttor (w/T) {
(1 - c)n
== [u,]
-1-
otherwisc
(27)
Considering IGM, = arg max trot (T, u). Then Teat Tiot (Qrn - Vena Veo((T))) and a, az is function of observations and actions:
(T, u4) = Qi(Ti, u.) Vi(TT)
U, = arg maxQ; u.)
(28)
log(1 c)
a(T,u) Qtoe(T, u) VIoe(T) n log(11 ()
" arg max max
(29)
Thus, A; is given by:
>e(T, u) Qioe u) (T)
(30)
n(Qu(T;, u,) Vi(T:))
This makes the Eq permanent. In particular, if sampled action u exactly current arg maxQ
then Q Via and Qi V, when a; - and A; can be set to 1. Thus we extend the function
class and Eq always holds for any action.
IGM Eq
denote up arg maxQi u;) and v² = arg maxQ u). Remember
Qtor (T, )d 2 Qroe(T, u)
VTne(T = a log
expl
(31)
UN
Qi(Ti, Ui) Un) )du 2 Qi(T,, uz) ai Therefore, Q = Viat and Q; = V₁ if and only if u = u² and u; = u, respectively. Considering
/ exp(
Vi(TT)) = a; log
(32)
Eq 7 and > 0, u² if and only if u/ that complete the proof.
11

---

Under review as a conference paper at ICLR 2021
Table 8: typer-parameters
settings
discrete
continuous
FSV
layer number of À
2
unit number hidden layer layer number of w, b unit number of hidden layer in w,
64
64
0
64
layer number of actor unit number of hidden layer actor
0
learning rate of actor a decay scheme
0
3e - 4
linear decay from to 0.01
Automate Entropy Adjustment
QMIX and VDN layer number of actor unit number of hidden layer in actor learning rate of actor
0
0
0
3e-4
7.2 IMPLEMENTATION DETAILS
In discrete tasks, We follow the PyMARL Samvelyan al. (2019) implementation of VDN, QMIX and QTRAN, where the paream-prrmeterrs of are the same in SMAC Samvelyan et al. (2019). We illustrate the special hyper-parameters of FSV Table g and others are the default settings in Py- MARL. In discrete tasks, we extend VDN and QMIX to the actor-critic framework. Specifically, we an actor for individual agents to maximise the Q-values from their own critic like DDPG.
We illustrate the hyper-parameters of these algorithm as well FSV in Table g For stability, we reformulate the Eq(7) as following:
N Qroe(T, u) Σ A(T,, (i) vi(Tz)) + Vio((T) Σ Q. u.) u))+ + Σ Qi(T. u,) (33)
N
Then we stop all the gradients except for X, and the last term Q. Considering incorrect weight A will cause incorrect at the beginning of training thus obstruct the exploration, we use aş = a which means we ignore the Kl.-divergence between current policy and optimal policy. The only temperature parameter is updated through annealing or Automating Entropy Adjustment in Haarnoja et al. (2018b) as following:
Ja = Eura alg - aH)
(34)
where the H is the target entropy.
12


Reviews:
[{"review_id": "oZG3QrAvOdU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel MARL framework named FSV, which incorporates the idea of energy-based policies and an efficient linear decomposition architecture in the joint action-value function with multi-agent maximum entropy reinforcement learning. Besides, the authors propose the IGO, which extends the IGM in stochastic policy cases. FSV suits in both the discrete and continuous action space scenarios. Experiments conducted on two simple examples with discrete and continuous action settings show that FSV could overcome the relative overgeneralization problem with the proper temperature setting. Furthermore, FSV in the challenging SMAC benchmark outperforms VDN, QMIX, and QTRAN in three scenarios.\n\nOverall, this paper is well-organized and easy to read. The authors present interesting ideas that combine the energy-based policy and maximum entropy reinforcement learning into the centralized Q-value mixing network to obtain a better expression ability than VDN and QMIX and overcome the relative overgeneralization problem.\n\nThere are some questions.\n\nQ1: Does the linear decomposition of Qtot of Qi and Vi limit the representation ability of FSV? It seems that FSV cannot represent the non-linear formation of Qtot and Qi.\n\nQ2: In Section 5.1, it is better to show the estimated \\lambda_i’s performance compared with the different \\alpha_0 settings.\n\nQ3: In Section 5.3, there lacks analysis of FSV and other methods. Especially, the ablation of FSV should be considered. In these three scenarios, which part (soft RL, critic’s structure, or others) contributes to FSV most and improves its performance steadily?\n\nThere are some typos.\nIn section 5.2, “on others’ policies (?).”\nIn section 5.3, “exploration efficiency” -> “exploration efficiency.”\n\nAs pointed out by other reviewers, this paper shares much similarity with QPLEX but without reference and discussions, thus I would be inclined to reduce the score as well.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TYd-nZDu860", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors use a method in soft RL style and a QPLEX factorization to propose the first multi-agent value function decomposition (VFD) method for stochastic policies. I appreciate the efforts to extend VFD to a larger function class.\n\nHowever, I am confused about the similarity to QPLEX [https://arxiv.org/abs/2008.01062v1 ], which was published several months ago. One of the main contributions of the paper is the value function factorization stated in Theorem 2. However, it is identical to QPLEX, including notations. Specifically, Eq. 7 and 13 of FSV are the same as Eq. 8, 10, and 12 in QPLEX, including notations.\n\nMoreover, some statements in the paper and details of the figures are very similar to those in the QPLEX paper.  Eq. 33 of FSV is the same as Eq. 50 of QPLEX. These equations describe an implementation detail where QPLEX and FSV stop gradients at the same variables. Stopping gradients is reasonable in the context of QPLEX, and the reason for using this trick is well-motivated there. By contrast, why this trick is necessary for FSV remains largely unclear. I encourage the authors to explain why they stop gradients here in the context of soft reinforcement learning.\n\nIn summary, the similarity including theorems, equations, and notations (and even figures). Despite these similarities, the authors did not cite QPLEX, which has been online for several months before FSV is published.\n\nAdditionally, about the integration of QPLEX and soft Q-learning, I also have some concerns. The definition of soft value functions depends on the specific selection of the temperature parameter. In this paper, the temperature parameters are end-to-end learned by minimizing the temporal-difference error of Q-learning. Could the authors explain why this makes sense in the framework of soft Q-learning? The counterpart in QPLEX ($\\lambda$ in Eq. 10) is end-to-end learned to ensure the rich expressivity of QPLEX so that it can represent the complete IGM function class. This design is well-motivated in the original QPLEX paper.\n\n\n** Minor points\nSome of the claims in the paper need refinements. For example, in Para. 2 of the introduction, \"Centralized training with decentralized execution (CTDE) (Oliehoek et al. (2011)) is a common paradigm to address the partial observability\". This claim is not solid. Partial observability can still induce miscoordination in decentralized execution [Wang et al. ICLR 2020, https://openreview.net/forum?id=HJx-3grYDB]. That is to say, the framework of CTDE itself cannot solve the problem of partial observability. It is better to say something like \"CTDE with communication.\" I also find the claim \"Value function factorization methods have been an increasingly popular paradigm for solving the scalability in CTDE\" not convincing. In fact, QMIX, which is a VFD method, generally can not work very well in tasks with more than 20 agents.\n\nTypos: Last paragraph in the introduction: \"it significantly outperforms other baselines, SMAC (Samvelyan et al. (2019)).\" SMAC is not an algorithm.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S_uHz2VaL8N", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a new method for learning factored value functions in cooperative multi-agent reinforcement learning. The approach uses energy-based policies to generate this factorization. The method is presented and experiments are given for smaller domains as well as starcraft. \n\nThe idea of learning factored value functions is promising for learning separate value functions for each agent that allow them to learn in a centralized manner and execute in a decentralized manner (centralized training and decentralized execution). Several methods have been proposed along these lines, but as the paper points out, they have limitations that makes them perform poorly in some problems. \n\nThe proposed approach in this paper has some promising experimental results, but there are questions about the novelty and significance of the method. Furthermore, evaluating these contributions is difficult due to the lack of clear details in the paper. \n\nIn particular, the details of the approach itself in 3 are not clear. Starting with Definition 1, it seems like IGO is using an optimal *centralized* policy. Is this what is meant? If so, why is this needed (as opposed to an optimal decentralized policy). It will typically be impossible to achieve a centralized policy with decentralized information. Furthermore, the energy-based policies are defined in 3.2, but 'key' ideas such as approximating the weight vector aren't fully explained making the exact approach hard to determine. Also, it is beneficial that the current theorems and proofs are included, but the lack of sufficient detail makes it hard to parse and evaluate them. \n\nThere are also similar max entropy approaches, such as the paper below. \n\nIqbal, S. & Sha, F.. (2019). Actor-Attention-Critic for Multi-Agent Reinforcement Learning. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:2961-2970\n\nAs well as other factorized methods, such as the papers below (which are admittedly new). \n\nWeighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Tabish Rashid, Gregory Farquhar, Bei Peng, Shimon Whiteson. NeurIPS 2020.\n\nde Witt, Christian Schroeder, et al. \"Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control.\" arXiv preprint arXiv:2003.06709 (2020).\n\nThe paper should discuss how the proposed method is an improvement over this other work and have a more comprehensive related work section. \n\nThe experiments are promising, but the relevant related work is not included and there isn't sufficient detail describing how the methods were run and discussing the results. In terms of comparisons, the paper should also need to compare with non-factored state-of-the-art methods. It is, of course, natural to compare with other factored methods, but what matters is general state-of-the-art performance of the domains. \n\nAs noted, the clarity and writing of the paper should be improved. Beyond the examples above, some other instances are below. \n\n- If the reader doesn't already understand the relative overgeneralization problem, Section 2.3 probably isn't sufficient. Figure 1 is helpful, but it should be described in the text to make the issue clear. \n\n- The connection between the overgeneralization problem and factored representations isn't completely clear. Factored representations have problems because they typically cannot represent the optimal value function (or policy). That is a separate issue than getting stuck in a local optimum (which can happen with any type of method).", "extra_scores": {}, "extra_text": {}}, {"review_id": "AuIqMS0dAbA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is concerned with the problem of cooperative multi-agent reinforcement learning for CTDE scenario which is well studied in recent literature. The authors propose a factorisation method based on soft value functions. I found that the paper is extremely poorly written which makes it very difficult to understand the overall method. The presentation is also quite arbitrary with discussion around results that seem unnecessary. There is little novelty as most of the paper borrows from SAC paper by Harnooja et al, albeit with gross errors in copying. Here are some of the  major issues:\n\n1. The authors discuss the IGO decentralisability, however what is the relation between IGO optimal policy and soft policy when the former is not representable by latter?\n\n2. How does the local soft policy iteration guarantee joint policy improvement? \n\n3. Why is the * being arbitrarily switched in sec 3.2? What does eq 12 even imply? isn't the KL minimiser $\\pi_i^*$ itself? Where is $\\Pi$ defined?\n\n4. How did eq 18 come about?\n\n5. The paper is full of unbacked blanket statements like: \" Although energy based distribution is very general which has the representation ability of most tasks,\" \"Our method are a member of them but out of the deterministic policy\" etc.\n\n6. There are many unintelligible sentences like: \"we need to extend the function class into any distributions\", \"IGO is more generality than IGM\", \"The individual value network is trained by minimize\", \"relative overgeneralization, where finding a\nsuboptimal Nash Equilibrium, which is a well-known game-theoretic pathology\" etc.\n\n7. In proof for Theorem 2, $\\epsilon$-greedy eq 26 cannot be matched by a soft policy in general, thus the rest of the proof cant follow without corrections.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "77WkftOwD3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a Q-factorization method by assuming an energy-based policies model. Q-functions are formulated as soft value functions with the energy parameters, and this adoption renders the function factorization more flexible compared to existing ones. The proposed solution applies to continuous-action tasks, a feat left unconquered by some of the existing methods. Authors exhibit that FSV outperforms others in various environments characterized by local optima.\n\nStrengths:\n\n+ The formulation of Q-functions as soft functions, despite appearing simple, shows some effectiveness in a number of MARL tasks.\n\n+ The network architecture is intuitive.\n\n\nMajor Concerns:\n\n- Neither energy-based policies nor soft value functions is an original contribution of this work. True, the authors do not claim so. But the reviewer is left unsure as to what then the primary contribution of the paper would be.\n\n- The method generalizes IGM to IGO but in doing so, foregoes the simplicity of the IGM condition. The reviewer would then expect to be met with a somewhat strong guarantee, but is instead presented with approximations on \\lambda_i. It is not clear from the paper how much insightful value the method has, when its criticism of a previous work (QTRAN) was based on intractability but the FSV method itself still relies on approximations. It would seem as though QTRAN and FSV each chose different paths to approximate different components of an MARL training scheme - the former takes may stronger assumption on the value functions while the latter takes assumptions on the nature of value functions being parametrized by approximated weights.\n\n- The effectiveness of the proposed method is not yet well-accounted for. Issues are raised, but little explanation (or any attempt thereof) is provided. For example, the reviewer would have very much liked to gain an understanding of the relevance between IGO and its ability to alleviate relative overgeneralization. How does taking on greedy policies (which makes IGO collapse into IGM) make MARL agents more prone to overgeneralize with respect to each other? What kinds of findings would the authors present? What evidence could support those findings? The evaluation, while illustrating great performance gaps, needs a careful redesign so as to construct solid grounds for the soft value function factorization under IGO to be \"explainably\" better than existing works.\n\n- The paper could be better positioned. The Related Works section could be put to better use to clearly distinguish two very different lines of research: value function factorizing MARL works and maximum entropy principle.\n\n- There needs to be some justification about multi-head attention being used to \"enable efficient learning\" in Section 3.3. The reviewer is left hanging as to why and how such a choice was made.\n\n\nMinor Concerns:\n\n* A few parts of the paper were difficult to follow. For example, there is an unfinished sentence in Related Works. In Section 2.1, there is an incomplete clause beginning with \"the reward function [...] shared by all agents\". Under Theorem 1, \"any distributions\" --> \"any distribution\". Also, what is meant by \"correct architecture\" in that same paragraph?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the paper presents some interesting ideas, in general the reviewers agree that the paper lacks clear results and is not an easy read. The paper proposes a factorisation of value functions, a topic that has received quite some attention in the literature (e.g. QPLEX), and it seems that their is not sufficient innovation in the proposed method in the paper. There are also a number of claims in the paper (e.g. partial observability etc.) with which some of the reviewers disagree, and should be discussed more carefully in a revised version of the article, that all in all seems to need more work.
Decision: Reject