ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Authors: Siqi SHEN
Mengwei Qiu
Jun Liu
Weiquan Liu
Yongquan Fu
Xinwang Liu
Cheng Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Agent Reinforcement Learning
Value Factorization
residual Q

Abstract: The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.

ResQ: A Residual Q Function-baseed Approach for Multi-Agent Reinforcement Learning Value Factorization
Siqi Shen', Mengwei , Liu', Weiquan Liu', Yongquan Fu"; Xinwang Liui, Cheng Wang
Fujian Key Lab of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University, China SSchool of Computer, National University of Defense Technology, China (siqishen, wqliu, owang @xmu edu, en, (yongquanf, xinwanggiiii @nudt edu. cn (mengweigiu, junliu millij @suu. xmu, edu. cn
Abstract
The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, propose, ResQ, MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some cate-action value pairs from joint which transformed as the sum main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments matrix games, the predator-prey and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.
value function,
Introduction
Many real-world tasks involve multiple agents acting together, such robot control [1]. Such tasks can be modelled as Multi-Agent Reinforcement Learning (MARL) problems where a group of agents must cooperate achieve a common goal. MARL has attracted great research interest because of its social and economic impact. However, MARL is highly stochastic and hard to learn due to the partial-observabiiity [2] and changing policies of agents. To address these issues, many approaches adopt the Centralized Training with Decentralized Execution paradigm (CTDE) [3].
In the CTDE paradigm, value factorization approaches [4] are widely adopted due to their remarkable performance and sample efficiency. A joint state-action value function is factorized into per-agent utilities Q;, and each agent i acts greedily according Q; To enable effective CTDE, it is critical to ensure the Individual-Global--ee Max (IGM) principle [5] that the optimal joint action should be equivalent to the collection of each agent's greedy actions. VDN [6] and QMIX [4] are popular value factorization methods, which satisfy the IGM principle.
Most value factorization approaches focus on the deterministic state-action value functions. Albeit MARL is highly stochastic, the full distribution of per-agent utilities and the multi-agent system overlooked and represented only expected value of the full distribution. Such distributional information could be beneficial for policy learning. DMIX and DDN [7] extend QMIX and VDN with distributional RL. They can factorize stochastic joint state-action value pairs into stochastic per-agent principle [7]. However, because
utilities, and satisfy the distributional individual
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Agent 2
A
C
A
C
A
8
C
U:
Agent
+
A
-12
0
0
0
-12
0
0
-12
7.9
0
0
C
-12
7.9
Qjt
Qiol
Qr
Figure I: Motivating Example of ResQ. The reward istate-action value function) of a one-step game, denoted as Qj:(u1, (2).. The game consists of two agents; each has three actions (A, B. and C). The optimal policy is to choose action A for both agents. This is non-monotonic matrix. If agent chooses action A, the reward vector for agent 2 becomes [8. -12, -12]. monotonically increases from right to the left. agent chooses action B/C, the rewards ([-12, 0, 0J and [-12, 0, 7.9]) for agent 2 monotonically increase from the left to the right. The direction of increment is different between the first and the second/third action.
of the representation limitations of QMIX and VDN, DMIX and DDN cannot factorize distributional value functions whose mean value non-monotonic.
To address the representation limitations, QTRAN [5], WQMIX [8], and QPLEX [9] have been pro- posed. However, QTRAN and QPLEX are sample-inefficient, and WQMIX has high approximation errors to non-optimal actions. These methods may lead to incorrect estimations of optimal values. Moreover, they satisfy only the IGM principle. Achieving the IGM and the DIGM principles without representation limitations remains an open challenge.
We address these challenges through, ResQ, residual Q function- based approach. Given hard-to- factorize joint state-action value function 25 (c.g.. shown in Fig. 1). the key insight of our work described as follows. We could obtain casy-to-factorize state-action value function, the main function Qtor, if we can mask out some state-action value pairs from Qjt- If the main function obtain the optimal policy of Q5e through
Qtot shares the same optimal policy with Qjt, then the factorization of Qoo by just using some simple mixers (c.g., [6]). After the masking, Q₁z is divided into value functions: main function Qtot and a residual function Q,- Q, the masked state-action value pairs from Qjt-
Besides mean-vales value RL (e.g.. 25c). the idea of ResQ be used with distributional RL [10] well. ResQ can factorize stochastic joint acatinaction value functions Zi into per-agent stochastic utilities Z; with mask and residual function Z,. We have shown that ResQ satisfies both the IGM and the DIGM principles without representation limitations.
For evaluation, we conduct extensive experiments one-step matrix games, the StarCraft II MARL tasks [11], and predator-pre tasks. The experimental results show that ResQ can obtain better results than multiple competitive value factorization methods, and the ablation study shows that through the use of residual functions for multiple factorization methods, ResQ can improve the performance of these methods. The source code of ResQ can be visited at https://github.co https/hww..ooommm-------th-
2 Background
2.1 Dec-POMDPs
We consider cooperative Multi-Agent Reinforcement Learning (MARL) scenarios which can be modelled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [12] defined G for agents. G (S, h P.r, ( { n, Y). S denotes the set of states and U4 the set of actions available agent and we consider discrete actions only. A joint action of all agents is defined as u' E 2 := 21 X x Ur, At discrete time step t and state s', after the joint action is issued, the next state s1+1 E of the environment is drawn from transition function st+ ~ P(-|s", u²). All the agents receive reward r' after the state transition happens. partially observable environment that agent i observes part of the environment of O which is drawn from of o- (-(s)). y is the discount factor. Each agent i maintains an

---

action- cetton-observatton history т; = (O, U,)*, and acts according to policy (T( We denote E T := Ti x ... X To as the joint-observation history. The learning objective to find the optimal policy π =< "h. ................................ nn
2.2 Value Function Factorization
The Individual-Global-ee (IGM) principle proposed in [5] is important realize the factorization of MARL state-action value function. It is defined as follows.
Definition (IGM), For a joint state-action value function Qi: : T x U + R. where T E T is joint action-observation history, if there exist individual state-action functionns [Q: T, xux + 1==1 such that the following conditions are satisfied
arg max u) (arg max QI(T1, U1), arg max Qn (Tas Ua)),
(1)
u
then, [Qi) satisfy IGM for Qjt under T. We can state that Qj((T, u) is factorized by Qo u,) ) In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a exists
QMIX is widely used monotonically increasing mixer function which proposes the following sufficient conditions for IGM:
(Monotonicity) aQ w)/0/ u.) 2 0, Vi E.
(2)
There are tasks whose joint state-action value function cannot be expressed well by monotonic increasing conditions, as shown in Fig.
2.3 Distributional RL
Distributional RL models the stochastic return of state-action pair through (TT, u) explicitly. They models full return distribution Z(T, u) instead of Q u.). The distribution of return be ap- proximated through a categorical distribution [13] or a quantile function [10]. Implicit Quantile Function (IQN) [10] models the stochastic value function [ u) as quantile function F-1 1/ ww), where F-1 is the generalized inverse cumulative distribution function (CDF), w E [0, 1] is a quantile sample. IQN defined distributional Bellman operator, and use it to updates its Z(T, u). After applying the distributional Bellman operator on Z(T, u). its resulting Z(T', u' remains in the same distribution as (TT, u). The loss Pt We, for temporal-diffeet -difference error d, duii,e;; is defined as Huber quantile regression loss, where W/; and Wi are two quantiles. During execution, the action with the largest expected return arg max, E[Z(T, (T, u)] is chosen.
MARL is highly stochastic, and distributional RL could be used to deal with the stochasticity of MARL. [7] proposes the Distributional IndviiuallGlooal-Maa (DIGM) principle, which is defined as follows.
Definition (DIGM), Given a set of stochastic individual stale-action value function [Z((T,, us) 4,)) ) and a stochastic join! state-action value function ZTx(T, u.), if the following conditions are satisfied
arg max E Zze(T, u)] = (arg max E[Zi(T,, u1)], ..., arg max E, ZZ (Tas Wn))),
21
then, [Z.(TE, u))l I satisfy DIGM for Zju under T. We can state that Zze (T, u) is distributionally factorized by [Zi(T,, u;)l
3 Related Work
Value factorization approaches are widely adopted in MARL [14]. VDN [6] factorizes the value function Qtot as the sum of per-agents' utility Qi QMIX [4] supports monotonic relationships among Qi and Quat- During execution, each agent acts greedily according to Q. RMIX [15] learns return distribution of each agent, and integrates risk-sensitive RL with QMIX. [7] proposes DMIX and
I represents to T. where T denotes the time step. means that the history could be short or long. For example, Ti could E Of x Uf x Of Uf and could Of x U.. where the superscript represent time, and the subscript the index of agent.
3

---

DDN, which extend QMIX and VDN with distributional RL through mean-shape decomposition. They satisfy the DIGM principle but suffer from the representation limitation of QMIX and VDN. They cannot model stochastic value functions whose mean-value are non-monotonic QAtten [16] and REFIL [17] adopt the attention mechanisms to focus on certain agents/scenarios when factorizing value function. QRelation [18] considers the relationships among agents for value factorization. However, they cannot represent non-monotonic value functions well.
To address the representational limitations, WQMIX [8] prioritizes the estimation of the optimal state- action value. It assigns higher weights to the optimal joint actions when minimizing approximation errors. WQMIX can be viewed as special version of ResQ. It pays attention to the optimal actions via masking out all the sub-optimal actions. The 22 of WQMIX consists of all the sub-optimal state- action pairs, and the Qoot models only the optimal ones. However, it assigns high learning priorities to Qtot, which puts the learning of sub-optimal state-action pairs and Qjt in trouble. Moreover, difficult to assign proper weights to obtain satisfactory performance.
QTRAN [5] and +TRAN+ [19] transform the joint state-action value function Q5r into an easy- fo-factorize one through a series of linear constraints. QTRAN learns an approximated function Qiean (T, u. = - (T uz) + (T)(T) to approximate the optimal policy of Q₃+- They can be viewed as using Qtot = ΣN (Ta u,) to model the optimal actions, and constraining Qtran 2 Q5t for sub- sub-optimal actions. QTRAN use soft constraints (MSE losses) to implement IGM which could
lead to the violation of IGM.
RQN [20] extends QTRAN by adding an individual correction factor for each utility function to compute an adjusted utility function. RQN can be viewed special case of ResQ. It uses the sum of per-agent utility as the main function Qtot. and the sum of individual correction factors as the residual function Q,.
QPlex decomposes O5 as the sum of a value function Viot and a non-positive advantage function Atot- The role of Acot is similar to the main function Qtol used in ResQ. QPlex obtains agents' policies from the factorization of Atot. which must be S 0, but ResQ does not have such restrictions. Although QPlex does not suffer from representation limitation issues, it may struggle to learn good policy in complex tasks [8].
MAVEN [21] deals with the inefficient exploration problem in MARL. Inefficient exploration problems could interact with the representation limitation problems. It adopts QMIX as its value factorization method. CDS [22] improves the effective cooperation of agents with efficient exploration. Different from them, ResQ focus on value factorization, and can be used in MAVEN and CDS.
COMNet [23]. DIAL [24], and GraphComm [25] study communication among agents. DCG [26] uses graphs for coordination. COPA [27] and NDQ [28] factorize the value function with communi- cation. The communication method adopted by them could be used with ResQ to improve MARL performance. There exist various MARL actor-critic methods, such as MADDPG [29], MAAC [30],
and COMA [31]. ResQ is value-based method. Other methods exist. For example, MAPPO [32] combines MARL. with PPO [33]. UPDeT [34] proposes MARL policy decoupling method. ResQ is orthogonal to them.
4 Method
Motivating Example Fig. 1 shows an example of a non-monotonic payoff matrix, which is hard- fo-factorized. Agent 1 has different action-value orderings, which depend on the action of Agent Moreover, the second-best state-action value 7.9 is only slightly lower than the optimal value 8, which causes difficulties for methods with representation limitations or overestimations.
Given a hard-to-factorize state-action value function Qüt. the key insight ResQ is that if we can mask out some state-action value pairs from 2it, then we could obtain an easy-to- to-ytto-fctorize value function Qiot- As shown in Fig. 1, if we mask out those action value pairs colored in red, then the main function Qiot can be factorized easily by mixers such as QAtten or QMIX. ResQ seeks to find a mask w, a main functions Qrot- and a residual function Q, which satisfy the following equation.
QTt(T, u) = Wroe (T, 2) Q tox (T, u) + wy((T (.T.u)Qraa (T,
where masks Wsat (T, u), WT(T, u) E (0, 1). The main Qiot shares the same greedy optimal policy as Qj.- Qeoe (T, u) is factorized into per-agent utilities Qi, and each agent selects its action greedily

---

with respect to Qi- There are many possibilities to construct the masks Whot and Why, in this work, we set thot be constant 1, and focus on w,.
We describe how ResQ factorizes an expected-value joint state-action value function Qjt into per- agent utilities in Sec. 4.1, and how ResQ decomposes stochastic joint state-action value function Zy: into stochastic individual utilities in Sec. 4.2.
4.1 Residual Q
For a given joint action- observation history т, consider factorizable joint state-action value function Qj((T, u) that can expressed in the form as (5). Let ;i = arg maxu; Qi uz), ú = | Theorem states the sufficient conditions for state-action value functions (Qi uz) M that satisfy the IGM principle for Qzt-
Theorem 1. joint state-action function
Qu((T,)) Qtoc(T, u)) + wr ())(T)
(5)
factorized by [Q, uz) { if QT, u) S O. Qtol(T, u) and u;) satisfy the
monotonicity conditions (2), and
(6a)
Wr(T, u)
u + T.
(6b)
The proof of Theorem shown the appendix. In Theorem 1, we have shown that for any factorizable value function Qyt that can be expressed as (5), we can find main function Qtoe that shares the same optimal policy as 2jc- In Theorem 2, we show that for any joint state-action value function Q, we can find a joint state-action value function Qj that can be expressed as (5), and it has the same optimal policy as Q-
Theorem 2. For any joint state-action function Q(T, u). we can find Qje(T, u) = Qco((T, WT, QT u) that
arg max Q(T, u) arg max u))(T
Q(T,u) Que(T, u) Vu + a
(8)
Qior u) monotonically increases with QT u, 1)) wr(TT u) satisfies (6), and Qr(T, u) $ 0.
The proof of Theorem and other theorems are provided in the appendix. From Theorem 2, for any joint aatte-aciion function Q(T, u). we can find Qj u) that shares the same optimal policy as Q. Moreover, Q3 can be expressed as (5). Combining Theorem and 2, for any Q(T, ((), we can find [Q,(T, u;) 2 satisfy the IGM principle for Q(T, u) through residual function Qr(T, u) < 0, main function Qtoe(T, u), and a mask function w, w,(T, u) E f0, 1). The policies (Qi(T, uz) f can be obtained through factorizing Qtor (T, u) using mixers such QMIX [4]. Furthermore, Formula (8) in Theorem 2 shows that Qje(T, u. can track Q(T, u) closely Vu o u. Thus, ResQ can model Q values precisely.
4.2 Distributional Residual Q
One promising way to deal with the MARL stochasticity to model the return of state-action pairs as distributional values [10]. Researchers [7] have proposed distributional MARL approach that satisfies the DIGM principle. However, it cannot model stochastic joint value functions whose value-expectation are non-monotonic. ResQ can address this issue through using Zamin(T, u) the main function and Zr as the residual function, where Zamiz (T, u) the factorization function of DMIX DDN [7], and Zdmizz Z, are both stochastic value functions. Formula (9) in Theorem ?? describes how Zze is decomposed.
Theorem 3. A stochastic joint state-action function
Zze(T, u) Zamiz (T, u) + (T, u),r(T u)
(9)
is factorized by [Z,(T,, u, M,,, if Z,(T, u) < O and wr u) 0 when u û. otherwise
1. = arg maxu; E u;) á [Tg 1= Zamin (amina(Tuu (T, = Zmean (T,u), u) + Zahape u E. E[ZZshape( (T, u)] = 0. Q: = E[Z,(T,, u:)]. Zmnan (T, u.) is monotonic increasing function with respect to Q.
5

---

Qje
Z
Z, Qr
masking
50
Main Function
Residual Function
FF
Residual Mixer
GRU
GRU
WC+
FF
COS
Zz
Z
Q1
Qg
Qx
10
0 u
0 14
(a)
(b)
(c)
Figure 2: The architecture of ResQ. (a): the main and the residual function, (b); agent utility, (c): stochastic agent utility
where Zmeen (T, u) models the mean of Zdmia- In DMIX [7], is implemented using QMIX, and in DDN, it is implemented VDN. 7shape models the variation of Zdmiz- Theorem 3 shows that ResQ can remove the representation limitations of DMIX.
Besides freeing [7] from representation limitations, ResQ can factorize a stochastic value function
Zje into individual stochastic utilities Z; with linear main function Zror, a residual function Z,, and mask function w,. Theorem states the sufficient condition for [Zi(T, uz) wi) satify the DIGM principle.
Theorem 4. A stochastic joint state-action function
Zze(T, u) Zzoo(T, u) + (T.(T,))Z,TT.
(10)
is factorized by [T, ", if (T, S 0, Zro (T, u) = E.m ( u:) k, 2 and (T, u) = 0 when u = ú, otherwise 1, where u = [u Th arg max., E|Z, u:)l.
The proofs of Theorem and 4 are provided in the appendix. Theorem 4 states that ResQ can find individual stochastic utilities satisfying the DIGM principle for Zz.. And it does not limit the representation capacity of stochastic joint state-action function Zzc.
In (10), the main function Zest is positive weighted sum of Z; instead of monotonic mixing individual utilities as in (5). This is because not all monotonic increasing functions can be used as the main function. Some monotonic increasing function (e.g., Ziot = =' Zf) could lead to incorrect estimation of the optimal actions (please refer to the Appendix for an example), they cannot satisfy the DIGM principle.
4.3 Neural Networks
For any state-action value function Qjt, if can construct the main function Qiot. the mask w,, and the residual function Q, as (5), then we can recover the optimal policy for Qj+- Similarly, for stochastic function Zya, we can recover the optimal actions for Zj through the main function Zoot. the residual function Z, and the mask as (10). The neural network of ResQ is depicted in Fig. 2.
Main Function For Qjt. Qtor (T, 24) can be any monotonically increasing function. The most well known monotonic increasing function of per-agent utility [Q.] QMIX, VDN, and QAtten. In default, ResQ uses QMIX as Qtot- For the stochastic case, the stochastic main function Zeot (T, u) can be a positive weighted sum of individual utilities ZT(Ti, w:). (Toe(T, EN Z wa) k 2 0. Specifically, we model k; by the attention mechanism [16].
Residual Function The residual function Q,(T, u) is a feed-forward network that takes action- observation history т. agents' utilities Q., and action [u.]F" as input. Unlike Qtot- Q, is not constrained to be monotonic or positive weighted-sum of utilities, thus allowing it to model larger class of functions. For [Q.] Qr is a simple feed-forward network which takes [Q,] as inputs, and the negative absolute function is the activation function of the last neural network layer to ensure
Q 0. For the stochastic case, the residual function Z, is modeled E D Ze

---

Table 1: Studied Algorithms
Valuc type
Algorithms
Expected value
ResQ, QMIX [4], CW QMIX 18), OW QMIX 181, REFILL 1171. QTran [5], QPlex 191, QAtten (16), VDN [6]
Stochastic value
ResZ, DAtten, DMIX [7]. DDN [7]
Mask Function The mask (T, u)) requires knowing the maximal joint actions for the joint state- action value function. However, it is intractable to obtain the optimal actions without searching over the entire action space. For practical implementation, we use approximation, and w, is defined as follows.
0 u ú
(1la)
Wrp (T, u)
1
u +
(11b)
where ū arg max, Qtoe(T, u) for the expectation case. For the stochastic case, arg max, E [T,
Loss For the expected-valle value case, the loss L of ResQ consists of two temporal losses: L" and _ L, = L' + T L' >1 I==1 (y* Qj((T,, wa,,,,,,,)) 0))2 where y² r +7 (Q ú, 0-), [ N U, = arg max Qi(T, Ut). 8 are the parameters of target network. I measure the distance between U3 and Qiot + w. w, It is defined as L [(₁, -Qu--Qu Quo wr Q,) For the distributional case, L, = T,a T,ast I, and T,aJt are similar to IT" and T respectively. 1 and T, use the pair-wise sampled temporal difference error instead of the standard temporal error, and the Huber quantile regression loss [10] rather than the mean square error loss.
Evaluation
We evaluate the performance of ResQ on one-step matrix games, the StarCraft II Multi-Agent Challenge benchmark (SMAC) [11], and predator-prey with multiple algorithms. The experimental results show that ResQ can obtain better resulty than state-of-the-art methods, and it satisfies both the IGM and DIGM principle together without representation limitation. The ablation study shows that ResQ can improve the performance of multiple value factorization methods through using the residual function. Please refer to the appendix for detailed setup and more results.
5.1 Experimental Setup
The studied algorithms are listed in Table 1. ResZ is the distributional version of ResQ factorized according to (10). DAtten, designed by us, integrates distributional RL [10] and the attention mechanism. Other algorithms are configured with their default parameters, and the number of rollouts is one. Each experiment is repeated at least times with different seeds. The configuration of ResQ follows WQMIX [8].
5.2 Matrix Game
We study the representation power of multiple methods for matrix game (depicted in Fig. I and Table 2). All the algorithms are ran through a full exploration 50,000 steps. This setting guarantees the exploration of all possible actions.
for e-greedy conducted over
Table 2 depicts the Q,₁ of multiple methods. As it observed in the table, only ResQ/ResZ, QTran, and CW QMIX can obtain the optimal policy, DMIX, DDN, and QPiex learn the second-best policy, and OW QMIX learns a wrong policy. Albeit CW QMIX can find the optimal action, its learned Qj has high approximation error, as it focuses on the optimal actions only. The detailed factorization results are depictd in Table ?? in the appendix. Further, we modify the matrix into distributional matrix by adding a nomral distribution (mean O, std=1) value into all action-value, The payoff matrix and the reconstructed Q values are shown in Table and QTran can find the optimal policy while most of the algorithms cannot. These results demonstrate the ability of ResQ to factorize difficult state-action value functions with low approximation errors without representation limitations.
the appendix. We find that ResQ/ResZ
7

---

Table 2: Payoff matrix of one-slep matrix game and reconstructed value function approximate the optimal policy. Boldface means greedy actions. Red color indicates wrongly estimated optimal actions, whereas blue color represents the opposite.
22
A
22
0.108 (A) -0.300 (B) 0.106 (C)
21
7/2
0.82(A)
-0.77(83)
0.77(C)
A
8
-12
-12
0.108(A)
8,03
-12.00
-11.99
0.82(A)
7,96
-12.37
-12.37
12
-0.300LB)
-12.00
0.00
-0.77(B)
-12.13
-0.27
-0.38
-12
7.9
106(C)
-12.00
0.00
7.87
0.77(C)
-12.22
-0.27
7.86
(a) Game Payoff matrix.
(b) ResQ: Q1, Q2 Q3
(c) ResZ: E Zior) 11
[2
Q2
-6.07AA)
-0.07(B)
0.04(C)
Q2
-6.00A)
-0.23(B)
1.45(C)
Q2
3,88A)
0.15(B)
3.46CC)
Q1
Q-
-6.0M(A)
-10.88
-9.99
-9.93
-6.70(A)
-13.40
-6.94
-5.25
3.27(A)
8,00
4,67
7,98
-0.0TC
-9.92
-0.20
0.16
BB
-6.93
-0.47
1.22
0.15(B)
4,88
(.04(C)
-9.85
0.15
7,81
1.45(C)
-5.25
1.22
2.91
3.26(C)
7,99
4.65
7.97
(d) DMIX: 11 Q2 Qsa
(e) DDN: Q1, Q2 Qze
(1) QTran: Q1 Q2 Q
Q2
Q/2
Q2
0.07(A)
150(B)
0.08(C)
0.17(A)
25.72(B)
25.74(C)
-0.03(A
50.79(B)
0.26(C)
(.07AA)
15.7
-3.72
0.34
0.17(A)
8,00
-3.04
-5.04
0.221A)
6.07
-0.87
6.86
B]
-2.62
12.66
350B
-3.04
-3.04
-5.04
30.32(B)
-0.87
-0.16
(.08C))
-1.20
12.44
15.83
-3.04
0.04(C)
-0.87
6.29
QPlex: Q1, Q2, Qit
(h) CW QMIX: Q1, Q2, Qs
(i) OW QMIX: Q1, Q2 Qzt
MAYA
Ic3552
2004
30mm
vt. 540g
Bm, 10_.999mm
27mLvs_30m
Figure 3: The Test Win Rate the SMAC benchmarks. The horizontal axis shows the training time step.
5.3 StarCraft II Multi-Agent Challenge (SMAC)
In SMAC [11], two teams of agents fight against each other. For the SMAC tasks, we train each algorithm for 1 to million steps, and for every 10,000 steps, the learned policies are evaluated. The test win rate, which measures the average win ratio of the agents controlled by MARL algorithms for each 32 test episodes, is reported.
As shown Figure (a), ResQ and ResZ achieve the best performance, and QMIX has the third-best results in the MMM2 scenario. The performance improvement over QMIX comes from the fact that ResQ/ResZ. can model better the non-monotonic state-action function of the MMM2 scenario. For the MMM scenario, ResO/ResZ has the best test win rate.

---

3056
Figure 4: The return of Predator Prey with punishment: p = (Left), p -2 (Middle) and p (Right).
The results for the 3s_vs_5y tasks are depicted in Fig. 3 (c). ResQ can obtain close-to-optimg performance after I million steps, while others cannot. For the bane_ vs_bane scenario, in Fig. (d), ResQ and ResZ. can learn the optimal policy in less than 0.2 million steps, while the other methods cannot. The results indicate that through masking state-action value pair from Zjt ResQ/ResZ can learn the optimal policy in short time, and the performance is stable. Albeit QPlex and DMIX can obtain close-to-optimal performance at a much later time, but their performance is unstable. The results for CW/OW QMIX for bane vs bane and 27m 30m are not shown due to out-of-memorry eгror on an NVIDIA Geforce RTX 3090 GPU.
ResQ/ResZ can obtain the close-to-optime performance after 0.25 million steps in the 2s _s_vs Isc scenario, shown in Fig. 3 (e). Although the other methods can learn co-e-toloptimall policy with more training steps, their performance is not stable. For the 1c3s5z scenario, depicted in Figure 2 (f), ResQ/ResZ are among the perffrrmnn algorithms.
In the 2c_vs_ vs_64zg scenario, 2 Colossi agents fights against 64 enemy agents, It has large action space, and the reward of each action varies rather than is constant in other scenarios. As it is depicted in Fig. 3 (g), ResZ can obtain the best performance. This indicates ResZ's ability to model distributional reward. The results for 8m vs 9m and 27m 303 are depicted Fig. 3 (h) and (f), respectively. ResQ and ResZ are the second-bes best performing algorithms. DMIX is good at scenarios which consist of homogeneous agents (c.g., 8m vs 9m), but it under-performs for scenarios with heterogeneous agents (e.g., MMM2, 3s_ vs 52), where agent interactions are complex.
5.4 Predator Prey
In the predator prey environment [26], agents hunt preys in 10 10 grid world. Each agent has 6 actions: moves in either 4 directions, stands still, and catches prey. If two adjacent agents execute simultaneously the catch action to a nearby prey, reward = 10 given to the agents, but failed catch by a single agent is punished by a reward p < 0. The more negative p is, the higher level of coordination is needed for the agents. Following [8]. the e of the e-greedy strategy for QPlex, ResQ. and CW/OW QMIX is gradually annealed from to 0.05 within 1 million steps.
The test return for three punishments: p p 22 and p -4 are shown in Fig. 4. For the most easy scenario performance of CW/OW QMIX drops in the middle of training. For the difficult p 22 case, ResQ can obtain the best returns. CW QMIX and OW QMIX have the second and third best scores QTRAN and QMIX can not obtain any positive returns. For p = -4, ResQ can obtain the best returns in the end. These results indicate ResQ's ability for challenging cooperation scenarios.
O, ResQ, QMIX, and QPlex can learn the optimal policies quickly. The
5.5 Ablation Study
We study the impact of different implementations for the main function and the residual function. and Starcraft versions the SMAC benchmark. We find that ResQ can improve the performance of multiple value factorization approaches through the use of the residual function Qr-
Main Functions In SMAC, QMIX is used as the implementation of Qtot- For ResQ, the results of using QMIX, QAtten, and VDN as the main function Qtot are depicted as ResQ, Qtot-QAtten Qtot-VDN in Fig. 5 (a) and (b). As is shown, for the MMM2 and 8m_vs_ vs_ vs_9m scenarios, ResQ, Qtot- QAtten, and Qtot-VDN perform better than QMIX, QAtten, and VDN, respectively. This indicates that ResQ can improve the performance of value factorization approaches (the main function) through the use of residual functions and mask. Moreover, we find that QMIX the preferred main function for the SMAC benchmark.

---

HVVN2
Ber yT, 9m
2004
Tand
Figure 5: Impacts of the main functions (a). (b). (c), the residual functions (d) (e), and StarCraft versions (I)
For the distributional case, shown in Fig. 5 (c). The results of using DAtten, DDN, and DMIX as the main function Zeot are depicted as ResZ, LDo,-DDN, and 7tot-DMIX. DAtten is designed by us; it integrates IQN [10] with QAtten. ResZ, Ztot-DDN, and 7tot-DMIX perform better than DAtten, DDN, and DMIX, respectively. This indicates that ResZ can improve the performance of stochastic value factorization methods through residual functtions.
Residual Functions The results of using different mixers (factorization methods) for the residual functions shown in the Fig. (d) and (e). The performance of using a feed-forward network (FF) as Q, (denoted as ResQ) is similar to that of using advanced mixers such as QMIX (Qr-QMIX) and VDN (Qr- VDN), and higher than that of QAtten (Qr- Qrtten) In ResQ, we do not restrict the function classes of Q,. This suggests that using a simple feed-forward network may be sufficient for Q,- For the distributional case, the performance of using DAtten, FF, DDN, DMIX is depicted as ResZ. Zr-FF, Zr-DDN Z--DMIX, As it is depicted in Fig. 5 (e), we find that using DAtten as the residual function Z, is preferred.
The Starcraft II version used in this work is SC2.4.6.2.662332 (2.4.6 for short), which is the same version used as QMIX [4] and WQMIX [8]. Some methods (e.g..[7)) uses the version SC2.4. 10. We compare ResQ with Pymar12_ _MIXX [14), which is a fine-tuned QMIX, across two versions. Figure 5 (f) depicts the results for different methods. The version of StarCraft is written inside the parentheses after the name of each method. For example, ResQ (2.4.6) denotes runing ResQ against SC2.4.6. As it is shows that ResQ/ResZ perform better than QMIX and Pymar12 QMIX for both two versions in the MMM2 scenario. As Pymarl2_QQea QMIX performs better than most algorithms in most case for SC2.4.10 [14), this suggests that ResQ/ResZ may perform better than other algorithms in SC2.4.10.
6 Conclusion
In this work, we propose, ResQ, residual function-based approach for Multi-Agent Reinforcement Learning (MARL) value function factorization. ResQ recovers the optimal policy for any joint state-action value function by masking state-action value pairs from the value function. We show that ResQ can satisfy the individual-glod -ooa-lgloll-mmaa -mx (IGM) principle and the distributional IGM principle without representation limitations. Through extensive experiments on multiple MARL tasks, we show that ResQ can obtain promising results.
A Acknowledgement This work was partially supported by the National Natural Science Foundation of China (61872376, 61972409), by open fund of PDL (WDZC202115501 13), by the China Post- doctoral Science Foundation (No.2021 M690094); the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform (N0:5502//CQXT221100033 We would like thank the anonymous reviewers for their valuable comments.
10

---

References
[1] Pablo Hernande7-Leal, Bilal Kartal, and Matthew Taylor. Is multiagent deep reinforcement learning the answer or the question? A brief survey. CoRR, abs/1810.05587, 2018.
[2] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In ICMI., pages 2681-200 -6900, 2017.
[3] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos A. Vlassis. Optimal and approximate q-value functions for decentralized pomdps. J. Artif. Intell. Res., 32:289-353, 2008.
[4] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018
[5] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, 2019.
[6] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo. Karl Tuyls, and Thore Graepel. delue-decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
[7] Wei-Fang Sun, Cheng-Kuang Lee, and Chun- Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In /CML, 2021.
[8] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurIPS, 2020.
[9] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multi-agent q-learning. In ICLR, 2021.
[10] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, ICML, volume 80, pages 1104-1113 2018.
[11] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In AAMAS, pages 2186-2188, 2019.
[12] Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. 2016.
[13] Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforce- ment learning. In Doina Precup and Yee Whye Tch, editors, ICML, volume 70, pages 449-458, 2017.
[14] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning, 2021.
[15] Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. RMIX: learning risk-sensitive policies for cooperative reinforcement learning agents. In NeurIPS, pages 23049-23062, 2021.
[16) Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. CoRR, abs/2002.03939, 2020.
[17] Shariq lqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In ICML, 2021.
11

---

[18] Siqi Shen, Jun Liu, Mengwei Qiu, Weiquan Liu, Cheng Wang, Yongquan Fu, Qinglin Wang, and Peng Qiao. Qrelation: an agent relation-basee approach for multi-agent reinforcement learning value function factorization. In ICASSP, 2022.
[19] Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran++: Improved value tramsformation for cooperative multi-agent reinforcement learning, 2020.
[20] Rafael Pina, Varuna De Silva, Joosep Hook, and Ahmet Kondoz. Residual q-networks for value function factorizing in multi-agent reinforcement learning, CoRR, volume abs/2205.15245. 2022.
[21] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi- agent variational exploration. In NeurIPS, pages 7611-7622, 2019. 2019.
[22] Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang. Qianchuan Zhao, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In NewrIPS, 2021.
[23] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. NeurIPS, 2016.
[24] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. NeurlPS, pages 2137-2145, 2016.
[25] Siqi Shen, Yongquan Fu, Huayou Su, Hengyue Pan, Peng Qiao, Yong Dou, and Cheng Wang. Graphcomm: graph neural network based method for multi-agent reinforcement learning. In ICASSP, 2021.
[26] Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. JCML, 2020.
[27] Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach- player multi-agent reinforcement learning for dynamic team composition. In ICML, 2021.
[28] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decom- posable value functions via communication minimization. ICLR, 2020.
[29] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-comeetiivee environments. In NeurlPS, pages 6379-6390, 2017.
[30] Shariq Iqbal and Fei Sha. Actor-atiention-critte for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, 2019.
[31] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.
[32] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of MAPPO in cooperative, multi-agent games. CoRR, abs/2103.01955, 2021.
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.00337, 2017.
[34] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. In ICLR, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] The complete proofs are included in the appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes] It provided the supplemental material, and the code can be viewed at https: wwt.:/oooommmm--/ymm
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] The standard errors are shown.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Each experiment is ran with NVIDIA 3090 GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use some of the code from Pymarl, Pymar12, and DFAC
(b) Did you mention the license of the assets? [Yes] It is source code released by the authors of papers. Their code licensed under the Apache License v2.0.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] The source code of this work is included in the supplemental material, and it will be open-sourced.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "i4Rba3HUE5w", "reviewer": "Reviewer_rgjE", "summary": "This paper extends distributional QMIX with a residual value function. Distributional QMIX (DMIX) an multi-agent reinforcement learning (MARL) algorithm using distributional Q-function to tackle stochasticity in MARL. DMIX decomposes the joint Q-value function of multiple agents into several sub value functions and aggregate them with a so-called mixer network. The limitation of DMIX is that each sub Q-function needs to be monotonically increasing with respect to the joint Q-function (i.e., the derivative of the joint Q-value w.r.t., the sub Q-value must be non-negative. This paper removes this limitation by introducing a residual Q-function in the factorization. The experimental results show that the proposed method improves the performance over the baselines.", "questions": "- Could the author comment on a concurrent work [1]?\n- Do the boldsymbols stand for vectors? If so, why $Q_{jt}$ takes vector of actions as inputs in Equation (1) while taking scalar actions as inputs in Equation (4)?\n- As the proposed method uses more parameters (because of residual network) than DMIX, did the author try comparing with DMIX in a larger architecture?\n\n[1] Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning, https://arxiv.org/abs/2205.15245", "limitations": "I encourage the author to think of the limitation of representation limits of residual decomposition. For example, could this residual decomposition be able to account all kinds of payoff functions?", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The use of residual decomposition is not new, yet this is the first time being used in factorizing value function in MARL. As such, I acknowledge the originality. \n\nQuality and clarity:\nThe figure 1 keeps being cited in the paper to illustrate the representation limitations and difficulty of modeling the value function. Yet, I don't understand how this makes value function representations difficult. I would like to see further explanation from the author. \n\nSignificance:\nThis work lifted the monotonicity assumption in QMIX framework. It is remarkable to my knowledge since non-monotonic payoff functions could exist in some applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K4HgraAXO6P", "reviewer": "Reviewer_dP9n", "summary": "The paper proposed ResQ, a value factorization method for converting the joint Q function to a monotonic and residual part. Theoretical results show that the proposed method can satisfy the IGM condition, and experimental results show that it performs well in the matrix game, SMAC, and Predator Prey.", "questions": "- ResQ factorize the joint Q into the monotonic part and residual part, whereas Qtran factorize the joint Q into the monotonic part and residual part as well. The difference between ResQ and Qtran is that Qtran does not explicitly learn a residual function; instead, it employs an equality and an inequality which work similar to the coefficient w in ResQ. So, what is the benefit of learning the residual function? Does it aid in the reduction of approximation errors or the improvement of sample efficiency?\n- Both Qtran and ResQ learns a joint Q network and trains individual Qs through another MSE loss. One of the main issues of Qtran is that the IGM is not guaranteed when the loss is not well minimized, which may lead to poor results in complex tasks. Is ResQ affect by the same problem? By the way, in the submitted code, Q_r is trained using loss max(Q_r, 0), which differs from that described in the paper. \n- In matrix game, why is the Q_jt of ResQ(table 2(b)) and Qtran (table 2(f)) so different from the payoff matrix? They are expected to be the same as the payoff matrix (like the Q_jt in Qplex) because they are both trained using MSE(reward,Q_jt) with no expressive constraints. Moreover, I believe that training the network to distinguish 0.001 is inapproproate. Because of the learning rate and stochastic gradient descent, the values are likely to differ by more than 0.001 each iteration after convergence.\n", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written clearly in general, but it could be better organized to bridge the expectational and distributional parts. The paper investigates an important issue of value factorization methods, and the experiments appear to be adequate. The main weakness of this paper, in my opinion, is that the factorization and loss, both of which are similar to Qtran, have no better theoretical guarantees than Qtran and Qplex. The paper does not go into detail on how ResQ achieves \"low approximation errors and high sample efficiency.\" It is unclear why ResQ performs better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E_mg3UDz7uP", "reviewer": "Reviewer_HPe6", "summary": "Post rebuttal:\n> I am updating my score post rebuttal based on the responses provided by the authors. They have answered almost all my questions satisfactorily. My increased score is contingent on the authors making the appropriate modifications to their paper as mentioned in their rebuttal.\n\n\nThis paper extends the action-value function factorization based approaches developed for centralized training and decentralized execution in cooperative multi-agent reinforcement learning problems. The main contribution is decomposing the action-value function into a factorizable component and a residual (formed by masking some state-action value pairs), such that the factorizable component alone is sufficient for each agent to independently choose their optimal actions, which also leads to joint optimality for all agents. The authors develop this method for both expected value and stochastic value (distributional) RL. The main challenges addressed by this method are improved representation capability, sample efficiency, and approximation error. The authors analyze their method theoretically and empirically through experiments on matrix games, predatory-prey and StarCraft benchmarks.", "questions": "Major comments:\n1. Can the authors provide a detailed comparison with QTRAN and QPLEX which also appear to have a residual decomposition, where the residual is a value function? Is *ResQ* a generalization of QTRAN and QPLEX? What are the benefits of *ResQ* over these algorithms other than generalization?\n2. I did not understand why *ResQ* can track joint state-action value pairs more closely than other algorithms. Can the authors please elaborate this point?\n3. The authors mention that they restrict attention to discrete actions. Is this a fundamental limitation of the approach? What needs to be done to extend this work to continuous action spaces?\n4. Can the authors explain why the joint state-action value function of the task shown in Fig.1 cannot be expressed well by monotonic increasing functions (Ref. Line 81, Lines 133-134)? This might be obvious, but one or two sentences explaining this would be helpful.\n5. In the Related Work section, especially in the first paragraph, the authors list several related works. I think, it would be more useful to present these works in context to the work of the present paper instead of just listing them.\n6. In Lines 124-126, the authors state: \"QTRAN over-estimates sub-optimal actions, which may lead to non-optimal decisions, whereas ResQ can estimate correctly non-optimal state-action values.\" Isn't it shown in the QTRAN paper that under appropriate factorization, QTRAN recovers the joint optimal policy, despite this over-estimation? Can the authors clarify and elaborate on this comment in the paper? This is again mentioned in Lines 135-137, but a more detailed explanation would be helpful.\n7.  In Line 459, in the Appendix, the authors write that \"Proof. This Lemma was proved as Theorem 4 of [8].\" for the Proof of Lemma 1. I could not find this theorem in [8]. As this Lemma is crucial for proving Theorem 2, I could not follow that proof as well. Can the authors provide an updated reference or am I missing something here?\n8. In Fig.2, what is the input to the residual mixer in part (a)?\n9. Doesn't approximating the mask function with the current best action ($\\tilde u$) instead of the best action ($\\bar u$) make the algorithm dependent on the initialization? Does the error introduced by this approximation go to 0 as the iterations progress? Is there any theoretical or empirical evidence for this?\n10. Why ResZ was not tried for the predator-prey example?\n11. Why is ResQ slower to learn in the p=-4 case in the predator prey example shown in Fig. 4?\n12. What is the motivation behind factorizing the residual function $Q_r$ in the ablation study in Sec 5.5?\n\nMinor comments:\n1. In Line 67, the equation is written as $s_{t+1} \\sim P(\\cdot|s^t, \\mathbfit{s^t})$. Shouldn't this be $s_{t+1} \\sim P(\\cdot|s^t, \\mathbfit{u^t})$?\n2. Can the $*$ superscript be defined in Line 70 for ease of comprehension for readers?\n3. Shouldn't the policy in Line 70 be $\\pi_i(u_i|\\tau_i)$ instead of $\\pi_i(u|\\tau_i)$?\n4. In line 71, shouldn't the joint policy be $\\pi = \\langle \\pi_1, \\dots, \\pi_n \\rangle$ instead of $\\pi = \\langle \\pi_1, \\dots, \\pi_N \\rangle$?\n5. The joint action-observation history is defined as $\\tau^N$ in line 70 and as $\\mathcal{T}^N$ in line 75. Can this be made consistent throughout? \n6. In line 76, consider replacing 'histories' with 'history'.\n7. In Line 76, shouldn't it be $Q_i: \\mathcal{T}_i \\times \\mathcal{U}_i \\to \\mathbb{R}$ rather than $Q_i: \\mathcal{T} \\times \\mathcal{U} \\to \\mathbb{R}$?\n8. In general *argmax* is a set operation and hence the output is a set of candidates. It would be good if the authors add a line stating that in this paper, they assume *argmax* is either unique or the ties are broken in a consistent manner so that equation (1) holds.\n9. In equation (1), consider replacing $N$ with $n$ for consistency. I think $n$ is the index of the last agent, and $N$ is a superscript that denotes the set of all agents. I think this confusion appears in multiple locations in the paper (such as lines 78, 180, 181, 484 etc.). It would be good to rectify this.\n10. It would be good to define $\\mathcal{N}$ or $N$ as the set of agents as it is used explicitly in equation (2), and then this notation can be used consistently in the paper.\n11. I could not understand this sentence in lines 88-90: \"IQN updates its value distribution through a distributional Bellman operator, which ensures the value distribution follow the same distribution.\" Can this be rephrased?\n12. I think the symbols should be $w_i, w_j$ instead of $w_1, w_2$ in Line 91.\n13. In Line 134, it should be \"factorize\" instead of \"factorized\".\n14. In Lines 147, 148, it should be \"joint\" instead of \"jointed\".\n15. In Line 151, $\\tau$ is referred to as an observation, where in fact, it is a joint action-observation history.\n16. In Line 237, shouldn't it be \"Table 2\" instead of \"Table 1a\"?\n17. In Line 454, shouldn't \"From (2)\" be after equation (5) and not equation (4)? And the reason for (4) should be $\\bar u$ maximizes $Q_{tot}$?\n18. $Q_r(\\tau, u)$ is written in Line 497. Shouldn't it be $Z_r(\\tau, u)$?\n19. In Line 524, there is a typo: \"matrxi\" -> \"matrix\".\n\n\n\n\n\n\n", "limitations": "I think the authors need to add more discussion around the limitation of their approach. Specifically, can all action-value functions be factorized using the *ResQ* approach? They address this partially via Theorem 2, which seems to suggest that all action-value functions can be factorized using *ResQ*. If not, under what conditions does it fail? A discussion on this point would be useful.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written. The original contributions are highlighted clearly.\n2. The *ResQ* insight is a neat and useful one and I think it extends the applicability of factorization based approaches to a larger set of problems.\n3. The authors provide theoretical analyses and also perform a detailed experimental study establishing the utility of their approach and improvement over the current best approaches.\n\nWeaknesses:\n1. Most of the theoretical concepts used in the paper such as IGM, DIGM have been already established in literature (and duly acknowledged by the authors). Furthermore, as far as I understand, the residual decomposition technique is very similar to QTRAN, QPLEX and the main difference, as far as I can see, is that it generalizes the decomposition proposed in QTRAN, QPLEX. As a result, most of the proofs in the current paper are also similar to QTRAN. Is my understanding correct?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "12bUQHN_nmJ", "reviewer": "Reviewer_Ea2R", "summary": "Authors introduce a constraint-free residual function into the state-action value function factorization construct, thereby allowing the joint (main, residual) function to be able to express a larger class of functions than with the main functions alone. The proposed algorithm is tested against a rich set of baselines and benchmarks and shows substantial improvements.", "questions": "Q1: continued from the Strengths and Weaknesses: how much complexity has been added with the ResQ function approximator NN?\n\nQ2: How would ResQ be positioned in the literature against, for example, MAVEN? MAVEN had its take on the different classes of state-action joint value function factorization with the corresponding algorithms published at the time. How would ResQ's take differ in terms of representational complexity?\n\nQ3: Under what circumstances do the approximations for the mask functions fail? What assumptions, if any, should hold in order for the approximations to remain valid? How strong would you say are those assumptions?", "limitations": "Please do include some analyses on the computational costs of ResQ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is easy to follow, and the claims made in the abstract are adequately addressed, with relevant theoretical setup, literature review, and experiment results analyses. Findings from the experiments indicate that the proposed algorithm works well and explainable MARL settings such as the matrix game make the better performance attributable indeed back to the motivating example.\n\nOne potential shortcoming of the paper could be the lack of insights provided on the computational costs incurred in calculating the residual functions. As ResQ requires fully connected neural networks to estimate the residual functions, one training step of ResQ would probably be more expensive than that of other baselines. Some analyses into this matter would be a nice addition to the draft.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept