Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning
Authors: Yitian Hong
Yaochu Jin
Yang Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Individual Global Max
Cooperative Multi Agent Reinforcement Learning
Value Decomposition
Imitation Learning
Data Aggregation

Abstract: In cooperative multi-agent reinforcement learning, centralized training and decentralized execution (CTDE) has achieved remarkable success. Individual Global Max (IGM) decomposition, which is an important element of CTDE, measures the consistency between local and joint policies. The majority of IGM-based research focuses on how to establish this consistent relationship, but little attention has been paid to examining IGM's potential flaws. In this work, we reveal that the IGM condition is a lossy decomposition, and the error of lossy decomposition will accumulated in hypernetwork-based methods. To address the above issue, we propose to adopt an imitation learning strategy to separate the lossy decomposition from Bellman iterations, thereby avoiding error accumulation. The proposed strategy is theoretically proved and empirically verified on the StarCraft Multi-Agent Challenge benchmark problem with zero sight view. The results also confirm that the proposed method outperforms state-of-the-art IGM-based approaches.

Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning
Yitian Hong
Yaochu Jin
East China University of Science and Technology ythong13140mail ecust. edu. en
Bielefeld University
yaochu. jonchu.iin@uii
in wewoboiiiiielleeed.d..oo
Yang Tang*
East China University of Science and Technology yangtangdecust, edu. cn
Abstract
cooperative multi-agent reinforcement learning, centralized training and decen- tralized execution (CTDE) has achieved remarkable success. Individual Global Max (IGM) decomposition, which an important element CTDE, measures the consistency between local and joint policies. The majority of IGM-based research focuses on how to establish this consistent relationship, but little attention has been paid to examining IGM's potential flaws. In this work, reveal that the IGM condition is a lossy decomposition, and the error of lossy decomposition will accumulated in hypernetwork-basee methods. To address the above issue, we propose to adopt an imitation learning strategy to separate the lossy decomposition from Bellman iterations, thereby avoiding
accumulation. The proposed strat- egy is theoretically proved and empirically verified on the StarCraft Multi-Agent Challenge benchmark problem with zero sight view. The results also confirm that the proposed method outperforms statc-of-the-arr IGM-based approaches.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has been proposed for multi-agent col- laborations to accomplish many challenging tasks M 2 MARL often relies on decentralized structures because of constraints in communication and observation commonly seen in applications [5]. Using additional information during the training process a popular paradigm for decentralized MARL, known centralized training [7.
The value decomposition (VD) method 8 as one of the centralized training and decentralized execution (CTDE) paradigm, decomposes the joint-action value into multiple individual-action values and has achieved the state-of-the-an performance StarCraft Multi-Agent Challenge (SMAC) [9] In the CTDE paradigm, Individual Global Max (IGM) is an important principle in VD for efficiently facilitating centralized training for decentralized execution.
IGM was proposed in QMIX [10), a popular VD method, which uses hypernetwork (MIX net- work) structure with additional environmental information decompose the joint-action value into
"Coresponding author
This work was supported by the National Natural Science Foundation of China (Basic Science Center Program: 61988101), Natural Science Foundation of China (62136003, 62233005), Programme Introducing Talents of Discipline to Universities (the 111 Project) under Grant B17017 and Shanghai Al Lab. Y. Jin is supported by Alexander von Humboldt Professorship for Al endowed by the German Federal Ministry of Education and Research.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

celviduull-cttion values. However, the structure of the mixing network in QMIX assumes that the joint-action value and individual- action values are strictly monotonic, which is, however, only a sufficient condition for IGM. To achieve frer-ffree VD, however, a necessary and sufficient condition for IGM is required. Following the idea of QMIX, series of research has been reported to improve the performance of QMIX by constructing more sophisticated mixing network structures. For in- stance, Yang et al. [I] present a framework, called Qatten, which introduces an attention model into the mixing network to accelerate the training process. QPLEX, proposed by Wang et al. (12). is constructed using a duplex dueling network that satisfies the necessary and sufficient condition for IGM. DMIX [3] integrates value function factorization methods into distributed reinforcement learning for highly stochastic environments. Surprisingly, recent results show that QMIX with fine-tuned hyperparameters and normalization outperforms many other recently developed methods on the SMAC benchmark problem [1]].
To the best of our knowledge, not much work has been dedicated to examining potential defects of IGM. In this paper, we prove that the IGM cannot equivalently transform actions from global state dependence to local observation dependence. In other words, the decomposition from global action value to individual action values is lossy. Furthermore, we point out that the error will accumulate in the reinforcement learning training process. As a result, the error of lossy decomposition can be amplified, which greatly limits the use of IGM decomposition. Hence, hypernetwork- -based VD methods suffer from significant performance degradation.
To address the aforementioned lossy decomposition problem, this paper proposes a novel training paradigm, called DAgger-based IGM (IGM-DA). Since lossy decomposition is unavoidable due to limited perception, this work aims to prevent the lossy decomposition error from being accumulated, considering that error accumulation mainly occurs in the reinforcement learning training process. To this end, we propose a novel training paradigm consisting an IGM decomposition training process and an imitation learning training process. The former trains global observation MARL agent as an expert, whilst the latter uses an imitation learning technique, namely DAgger [3,, to decompose the trained result in the former process relying on global observation into the latter that replies on local observation only.
In the next section, we theoretically prove that the lossy decomposition error accumulates in hypernetwork- based VD methods and that error accumulation can be avoided using the proposed IGM-DA. To empirically validate the effectiveness of IGM-DA in mitigating the influence of the lossy decomposition, we investigate the performance change when the perception range in SMAC varies. Ablation studies and additional analysis confirm the importance of the proposed IGM-DA by showing that the performance significantly enhanced when IGM-DA embedded in QMIX, QPLEX, and DMIX, three state-of-the-art IGM algorithms.
2 Analysis
As shown in previous work [16] 17 the cooperative MARL problem can be modeled as a Decentral- ized Partially Observable Markov Decision Processes (DEC-POMDPs). DEC-POMDPs are defined by a tuple of (s, Z, O,T, U, U,P,r 2), where E S denotes the current state of the environment. At time instant f, each agent E N = (1,..., n takes actions u, E U facilitate cooperation. All these actions form a joint action set u E U L P(s'|s. u) S x U x S + [0, 1] represents the state transition after the agents take the joint action. r(s, u) by all agents and 7 E [0, 1) is the discount factor.
U + R denotes the reward shared
In partial observation setting, each agent can only perceive local information of environment E 7 according to the observation function (s, i) : S x N - Z. The action-observation history for each agent is T' E T (Z x U). According action-observation history, each agent conditions its own strategy T' (u' ( : [0. 1]. Based on the joint strategy म, the joint-action value finction (st uz) = E Es. tooe. 1o [E >h >rr+t u.] can be established. Furthermore, finding the optimal strategy can be reduced to finding the optimal joint action-valie function Q"
2.1 Individual Global Max (IGM)
In the centralized training process, each agent can access additional global information for strategy training. On the other hand, during the decentralized exccution process, each agent only access

---

its own local action- obboon-tberrvtion history r¹ for decision making. This paradigm, known as CTDE, is widely used cooperative MARL. IGM is an important principle for realizing CTDE in value-based MARI methods, which can be represented as follows:
arg max q1 (T', u²)
arg max (8, u)=
aug max On (T",u")
where the individual agent utility represented by qe N. The IGM principle affirms the consistency of global and local action selection, well as the factorization relationship between the joint-action- valuue function Qtot and the local-action-van value function Qi-
Assumption 2.1 IGM decomposition is realized by introducing the IGM principle into hypernetwork construction. shown in Figure IGM MIX represents meticulously designed network enabling to approximately learn the joint-action value Quot relying on global observation from the action-values of individual agents qi based on local observations. In this case, the learned Qrot is denoted by IGM(g). (n)))..))))..
The hypernetwork in IGM based decomposition is currently one of the most popular structures, for example, in AVD- Net [18], MAVEN 19. and ROMA [20]. All these methods integrate the IGM consistency condition, which is achieved through the delicate design of the IGM MIX network, into the network structure, thereby eliminating possible inconsisten- cies between the local maximum action-value and the global maximum joint- aeint-action value.
IGM Hypereetwerk
o'teaad)
(c.a))
To update the joint-action value, the Bellman equation 21 is
Figure The hypernetwork IGM decomposition, where each agent holds local-observation input.
introduced. Then, the joint-action value function is reformu- lated as follows:
Viat u.) = IGM (qi (Tt ut), In (T uf))
= ymax (IGM(q (7 "/41), (n(1: (2411))
m++1
Benefiting from the hypemetwork of the IGM dccomposition, the global maximum joint-action- value function IGM (91, ... Qn) can be ensured by maximizing the local action-value function [g:(Ti41 ut4 1) Thus, a new iterative equation of the local action-value obtained as follows:
IGM u4)......(ff..... u()) r 7IGM qu (T 1441),... wf
max qn (T/t u(+1))
This way, each agent participates in the estimation of joint-action-value to improve the efficiency of exploration. Therefore, the hypernetwork in IGM makes possible extend the action-value iteration equation from a global state and joint action selection to local scenarios and individual action selections.
2.2
Defects of IGM
2.2.1
The Lossy Decomposition
The IGM factorization consists of the following two steps:
arg max Q1 (8, u
arg max Qsoe(s, u)
arg max Qn (8, u²) k
(4)
arg max Q1 u²)
arg max dn ( u²) nll
arg max Qn u)
arg max On (7 u²)
3

---

Qi(s, u') represents the local action-value based on the global state rather than the local observation. The first line in equation posed into multiple mecal-action-valuee (i w²). Thus, individual action selection does not need to rely on the policy of others. This also makes each individual capable of independent exploration, which is the fundamental goal of the VD method. The second line in equation A that for local action selection, the global state-based acction-vauue is converted into local observation-bast based.. There- fore, the decentralized execution process can rely on individual local observations only. Compared | indiicates two roles of the IGM decomposition: decoupling interdepen-
4 Aindicates that for the same state, state joint-acction--alltee Quat u is decom-
with equation I equation dence between actions of different agents and converting actions from global state dependency into local observation dependency. In this work, we focus on the second part, i.c. how to more accurately learn the local action-valuce based on the action-valus of individual agents with global observation.
Existing work on hypernetwork based VD, such as QMIX [Q], focuses on increasing the hypernet- work learning potential by introducing glohal states into the hypernetwork, without paying much attention to the dependence of actions on global state. In the following, we will show that action-valle based on the global state cannot be perfectly decomposed into action-values based on local observa- tion only. In other words, the decomposition based on IGM is lossy when the local observation is insufficient. The definition of insufficient observation and lossy decomposition is given as follows:
Definition (Insufficiens Observation). Local observation T is an insufficient observation of global state 8, if there a case where global state changes while local observation T does not change.
Definition 2 (Lossy Decomposition). For any individual uction-value functions based local observa- tion [g:(T' xd) : XxX U - [R);= The decomposition from Qrat (8, u) into [q' (pr,(""")))))) uu))== is lossy, if 3 S, 7 E T,, s.l. arg max Qtoc(s, u. # jarg max (((T, u²" =²)]
u
This paper also gives the proposition of the existence of lossy decomposition and a proof is given in Appendix A
Proposition 1 (Existence of Lossy Decomposition). Let T be an insufficient observation of global state s defined in [ then 3 Qto((s, u) such that the decomposition from Quals, u) into [g'(r', u²) )'))== is lossy.
As consequence, for the same observation 75, the agents cannot distinguish whether the environ- mental state outside their sensing range has changed or not. Because the agents based on global information s, individual agents select the action based only on their local observations T', resulting in incorrect action selection (lossy decomposition).
joint action selection of the
To mitigate the negative impact of partial observations, global information can be introduced in the hypernetwork during training progress (CTDE). As shown in Figure [ the global information s is considered an embedded input in the hypernetwork-baasee method to prevent its direct impact on action selection. Therefore, the action selection module based on local observation can be separated more conveniently. Namely, the global information is kept from influencing the individual action selection, resulting in a lossy decomposition. In addition to the proof in Appendix [A] the influence of lossy decomposition on the performance will be further shown in discussing the experimental results.
2.2.2 Error Accumulations
The sensing limit requires that the learned strategies are based only on local information. Therefore, in most cases, lossy decomposition in IGM is inevitable. In addition to lossy decomposition, we find that IGM also suffers from the problem of error accumulation during the training process.
Because IGM is a lossy decomposition, equation 2 can be rewritten by:
Qia u.) z IGM (T), u!)..., (T" (TT",)")) u.)).
(5)
where the approximately equal symbol signifies the existence of lossy decomposition. Similarly, equation can be revised as follows:
IGM (gn (Td, u!), (7 (x)))) x rr +++71GMMM (max qı (T u(42),.., max gn (T u(+1)).
(max
(6)

---

local
I.. with
Decomposition
Lawrrer trajcttry
((s,a()
III. Hysemsweek of Exper opent
Offpoliy
According to praviem muthods, sush DMIX QJEX, UMI
Update Expert
MLF
Learning (DAgger)
(S
Experi spent ghibal
Exppit Trauedory
Hy pornetw of
(5.M)
Leamnar Iroectory
Supervoad kammn
IGM Hyperneterk
Update Learner
Learee apen: with local otservation
2 k,
Leatnst Iipectory
Jeamer agant polky
Dxpesager
Experagee ppemaagmen
(5,4()
B5,44)
(a) The proposed IGM-DA framework.
(b) The detailed network structure.
The proposed IGM-DA framework and detailed network structure. (a) The value decompo- sition part (the upper part) trains an individual expert agent with global state: the imitation learning part (the lower part) trains an individual learner agent with local observation through supervised learning. (b) The detailed network structure of learner agent, expert agent, and hypernetwork of expert agent.
Figure 2:
In the following, we will show that errors resulting from the lossy decomposition will accumulate in the iterative training. Let error denote the error generated by lossy decomposition in IGM, errorother the remaining emror caused e.g., by noisy input information and limited network learning ability, and (Tror(() = Q - Q the total error in the training process, where Q is the true action value, and Q is the calculated action value. Then we have the following Proposition.
Proposition 2 (IGM with error accumulation).
According to assumption [. the IGM decomposition is implemented by incorporating the IGM principle into the hypernetwork construction. Then Qtot is updated by the Bellman equation, thus updating (T; w according to equation 5
The total error can be expressed by:
done
Error(Qiat (8t, ux)
> Jerrorded + error other (i))-
Error accumulation is proved in Appendix A In the following, we introduce imitation learning based on the DAgger structure into IGM, called IGM-DA, to avoid error accumulation.
3 Method
3.1 Error-accumulation-free IGM
Due to the accumulation in the training process, the total error of the whole system may become larger and larger through the training process. To resolve this problem, we propose an IGM-DA training paradigm prevent error accumulation.
In the first stage, as shown in Figure 2(a) we obtain individual expert agents based on the global state through RL training of the IGM hypemetwork (the upper part). In the second stage, train the

---

individual learner agent based on local observation by means of imitation learning (the lower part). This way, lossy decomposition separated from the iterative training process so that occurs the second stage only.
Figure 2(b) shows the detailed network structure of the learner agent, expert agent, and hypernetwork of expert agent. In the hypernetwork, the IGM MIX acts as a connection between action-values of individual expert agents [g: (si, u ")]]E= with global state (instead of (q'( wf with local observation) and joint-action-value Qrot. Thus, equation rewritten by:
Qiot u,) IGM (5), u/) (.),...... 4n (86; u,)),
(8)
Equation also corresponds to the first line in equation Because both the left and right sides of the equations dependent global state, lossy decomposition caused by insufficient observation does not exist.
Proposition 3 (IGM Integrated imitation learning without error accumulation).
If ret is updated by the Bellman equation, thus updating (s), ut) according to equation g then [g;(T), ut) ut)l== is obtained from [g:(s); ut)l by supervised imitation learning. Thus, equation
7 can be rewritten as follows:
done
Error(Qiot (st, un)
Σ
y i-t (errorotherr errordeo(t).
By comparing equation and equation 9 decomposition can be avoided. The proof of Proposition 3 is provided in Appendix A
we can see that error accumulation resulting from lossy
3.2 Imitation Learning and Integrated DAgger
As shown in the Figure we use imitation learning to avoid error accumulation. Common imitation learning, represented by Behavioral Cloning [22 and DAgger [23], aims to learn strategies from expert experience. Different from traditional imitation learning, experts in the proposed IGM-DA are real experts but an intermediate learning result of RL. The main difference between expert agents and learner agents lies in the fact that expert agents can observe the whole state of the environment, while learner agents can only observe within limited range. Note, however, that Behavioral Cloning is not well suited for the present work since in Behavioral Cloning the virtual experts modify their strategies through reinforcement learning based on the data collected by themselves. which remain to be global information based. By contrast, the virtual experts in DAgger can constantly modify their strategies through reinforcement learning based on the data collected by the learners. This way, the policies learned by the virtual experts based on global information can be adapted to the local observations.
Recall that expert strategies can observe global information, while learners can only observe local information. Although have already theoretically analyzed that the integrated imitation learning structure in Figure 2 can prevent the eтror accumulation, we still need to find a proper decomposition in the imitation learning stage reduce the error. Similar Proposition [ proposition 4 can be
obtained:
Proposition 4 (Existence of Lossy Decomposition 2). , u² u²) such that the decomposition from Ig' u² )) to
Let be an insufficient observation of global
state as defined in [ Then 3 (s'
lg' u² } is lossy.
It is worth noting that q'(r', u²)) represents the strategy based on local observation, and Pa (u'ls) represents the expert strategy based on global observation learned:
P=(w((s) {
if ue arg max o' (s, w')
(10)
if u of arg max q'(s, u²)
In order to find the optimal decomposition, we need to define the optimal decomposition from (s',, wf to (a (7*, =²)] To this end, we introduce the Bayesian expected loss 24 in lossy imitation learning.

---

Proposition 3 (action-value after lossy imitation learning). Suppose we have k samples that satisfy local observation т. Then the optimal action-valas after imitation learning will be:
d'(T' 1/k (
(11)
Proposition is proved in Appendix A loss 2 1/k P
The loss function of the supervised imitation learning is
Experimental Results
To rigorously investigate the performance of the proposed learning strategy, adopt the StarCraft benchmark task as the test problem. Partial observations of the environment are reflected in the limited sensing ranges of the agents, resulting in lossy decomposition. In this section, first show the existence of lossy decomposition before comparing the proposed framework with state-of-the-an baselines, including QMIX, QPLEX, and DMIX. To clearly show the robustness of the proposed framework, we set the vision range of the agents in the environment to O. In this case, each agent only knows its own attributes and optional actions. Several ablation studies are also given by comparing the proposed framework with DAgger. The implementation details and experimental settings can all algorithms under
be found in Appendices C and D For fair evaluations, the comparison as well as the optimizers, are the same, and the experimental results are presented with the average performance with 25-75% percentile. Moreover, the presented curves are smoothed by moving average filter with its window size being set to 5 for better visualization. The code of the proposed algorithm can be downloaded at
4.1
Lossy decomposition
Consistent action selection.
1.0
target, mean
sight view
action value
expert
0.2
sight view
Median
Inconsistent action selection.
0.4
action value
TimeSteps
action
(a) Win rate comparison.
(b) Comparison of action values (sight view O).
Figure 3: Verification of the existence of lossy compression. Because multiple global states may correspond to the same local observation, the action value under local observation is the weighted average of the action values of multiple global states.
In order verify the existence of lossy decomposition in StarCraft II, first train group of agents in relatively large sensing range, which is called sight-view hereafter (sight-view 9), and then use the agent with sight-view represents the average winning rate of a set of strategies trained under sight-view 9, Other curves plot the imitation results of the trained strategy (expert action value) under different settings. sight view denotes the sensing (viewing) range of the learner agents is 9. Similarly, sight view 0 denotes the viewing range Figure (b) shows the distribution comparison of action values for sight view Because of the insufficient observation, the distribution of action value will have errors that lead to the final wrong action selection. Compared the experimental results of different fields of vision, can be concluded that the strategy learned in sight view 9 depends on additional information beyond sight view
imitate the trained policies. As shown in Figure (a). target mean

---

4.2 Robustness to zero sight view
The environments in SMAC are divided into three difficulty levels: Easy, Hard, and Super Hard. In this work, the scope of the agents' vision is limited to O, which poses additional diffi- culty to the environment. We choose six of these environments for performance evaluation: (a) 3sSy(Fasy), (b) 5m_vs_ "s_6mmHard), (c) MMMe(Super Hard), (d) 8m(Fasy), (e) 3s (s__x((ard),, and (f) 8m vs_ sas_ssssa_allle It worth noting that because of the increased difficulty, the win rate of all test algorithms may be 0 in Super Hard environment. In order to better show the differences between algorithms, we do not pay much attention to the Super Hard environments.
8m_vs________
In Figure we use -DA to represent variants of the original method combined with the IGM-DA framework. We use coarse curves for the IGM-DA results make easier to distinguish. We find that IGM-DA outperforms the other compared algorithms. The detailed results of Figure are listed Table I The best performing algorithms in different environments are as follows: 3s5z (qmix- 8m (dmix-DA): 3s vs_5z (qmix-DA);
aree
DA); 5m_ vs 6_ (dmix-DA); MMM2 Mumix-DA.qplex-D.. 8m_vs_ ss_v__99 99m (qmix-DA). The average success rate of the proposed approach improved by about 20% over that of the compared algorithms. Figure ((c) shows the robustness of our method for different sight views.
Table Average win rate of SMAC challenges in sight view 0
3s5z
5m_vs_6m
MMM2
8m
3s_vs_5z
8m _m__v__mm Avg.Score
qmix
90.2
36.4
26.2
98.3
1.9
33.9
47.8
qmix-DA
93.3
50.6
55.4
99.2
43.9
70.8
68.9
aplex
87.0
5.0
0.0
98.7
18.9
36.2
41.0
qplex-DA
88.3
38.9
55.4
99.6
31.6
64.8
63.1
dmix
76.2
58.7
0.0
99.4
3.3
64.7
50.4
dmix-DA
91.7
59.0
46.2
99.8
40.6
64.4
67.0
gTta-AAmmuma
411a
galex Edlpurt
Mosuan Wen P00"
qaler
3s57.
(h) 5m vm_6m.
(c) MMM2.
quta
Vo0 on WN Putto
Cioomn dris
236
Timestips
110
18
6.0
000
Timestope
657
Tin1300033
(d) 8m.
(e) 3s. vs S2.
(1) 8m vs. 9m.
Figure 4: Results of QMIX, QPLEX and DMIX with or without integrated DAgger in six environ- ments, showing that integrated DAgger can significantly increase the median win rate in 0 sight view.
4.3 Ablation Studies
Figure (a) and (b) plots the comparative results of different imitation learning structures, where refers DAgger, and BC refers Behavior Cloning- detailed introduction of imitation
learning is given in Appendix B As shown in Figure 5(a) when combined with imitation learning,

---

the algorithm's 's performance can be significantly improved. The results in Figure 5(b) confirm the limitation of Behavior Cloning. We can clearly see that there huge gap between Behavioral Cloning and DAgger algorithm in the Super Hard MMM2 environments.
cmbl-CC
smleeC
O-amlk
WtL---mm-DDizuuuu DAzzurs
signt L-7-mxxx
right 1-mmbeD8/oum)
ThraGliags
ThraGiops
TimeSlaps
(a) 5m_vs vs_
(b) MMM2.
(c) 5m_vs___ve 6m.
Figure 5: (a) and (b) :essuts comparing Behavioral Cloning and DAgger. (c) Our algorithm is robust in different sight views.
5
Related Work
The proposed IGM-DA aims to enhance the robustness of the value decomposition method in the presence of partial observation when no communications between the agents are allowed. case communications are possible, the information loss might be properly compensated. For example, Foerster al. (25] proposed deep reinforcement learning for communication topology learning, where agents can use information from others to stabilize training. Along the same line, two information-theoree regularizers between value function factorization learning and communication learning were proposed in (26). which reduces the amount of required communication without sacrificing the performance. Chen al. 27 considers the problem of inefficient sampling caused by frequent changes in communication channels, and proposes accelerate communication learning by integrating centralized learning and knowledge distillation. Although Chen et al. also use fully centralized training, which is similar to this work, the motivations and assumptions completely different. The reader may refer to (28) for more research on communication based learning. Moreover, studies on the algorithmic property of the VD methods from different perspectives have also been reported. Through a large number of experiments in one-shot (i.e., non-sequential) problems, Castellini et al. [29] visualize the expression ability of various MARL methods. Factorized Multi- Agent Fitted Q-Iteration was proposed by Wang et al. [30] to analyze the cooperative MARL based on value decomposition. Under the IGM conditions, Huang al. [3] take into account the sub-team coordination problem to achieve more efficient collaborations.
6 Conclusion and Future Work
In this paper, we have pointed out that IGM decomposition is lossy decomposition, and that the error resulting from the lossy decomposition may accumulate in the training process. The accumulated error may seriously degrade the performance of VD-based algorithms. To tackle the above problems, we have proposed IGM-DA, which integrates imitation learning into IGM decomposition. We show theoretically and empirically that proposed framework can prevent error accumulation by introducing imitation learning into the training process, making it possible to adapt the learned policies to the local information, thereby avoiding error accumulation.
The proposed work is able to achieve the best performance improvement in extreme environments of zero vision. If the sight range becomes larger, the improvement will be less significant (refer to the experimental results shown in Appendix [F). In addition, this paper focuses on one class of value decomposition methods, namely hypernetwork with local observation. Other methods, such as QTRAN (32) based on loss design are beyond the scope of this paper. Finally, we have adopted a classical imitation learning technique for avoiding error accumulation. Therefore, an immediate future work is to integrate STOA imitation learning technologies with multiple value decomposition methods. Furthermore, we plan to extend the imitation learning structure to partially observed single-agent reinforcement learning algorithms.

---

References
[1] Jinming Ma and Feng Wu. Feudal multi-agent deep reinforcement learning for traffic signal control. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, pages 816-8886888888 824, 2020.
[2] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim Green. Multi-agent reinforcement learning for active voltage control power distribution networks. Advances in Neural Information Processing Systems, 34:3271 1-224,,, 2021.
[3] Yuchao Zhu, Haipeng Yao, Tianle Mai, Wenji He, Ni Zhang. and Mohsen Guizani. Multi-agent reinforeement learning aided service function chain deployment for internet of things. IEEE Internet of Things Journal, 2022.
[4] Liang Yu, Yi Sun, Zhanbo Xu, Chao Shen, Dong Yue, Tao Jiang, and Xiaohong Guan. Multi- agent deep reinforcement learning for hvac control in commercial buildings. IEEE Transactions on Smart Grid, 12(1):407-419, 2020.
[5] Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. Learning correlated communication topology in multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 456-464, 2021.
[6] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[7] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
[8] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decoopositionn networks for cooperative multi-agent learning based on team reward. In Proceedings of the 1 17th International Conference on Autonomous Agents and MultiAgent Systems, 2018.
[9] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agent and MultiAgent Systems, pages 2186-2188. 2188, 2019.
[10] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the International Conference on Machine Learning. pages 0295-4304, PMLR, 2018.
[11] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03999, 2020.
[12] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. (QPLEX): Duplex dueling multi-agent leaarning. In Proceedings of the International Conference on Learning Representations, 2021.
[13] Wei-Fang Sun, Cheng Kuang Lee, and Chun- Yi Lee. Dfac framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In Proceedings of the International Conference Machine Learning, pages 9945-9954, PMLR, 2021.
[14] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wa, and SW Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning, arXiv preprint arXiv:2102.03479 2021.
10

---

[15] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- lional conference on artificial intelligence and statistics, pages 627-635 JMLR Workshop and Conference Proceedings, 2011.
[16] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[17] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: selective overview of theories and algorithms. Handbook of Reinforcement Leurning and Control, pages 321-384, 2021.
[18] Yuanxin Zhang, Huimin Ma, and Yu Wang. Avd-net: Attention value decomposition network for deep multi-agent reinforcement learning. In Proceedings of the 25th International Conference on Pattern Recognition, pages 7810-7816. IEEE, 2021.
[19] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Proceedings of the Advances in Neural Information Processing Systems, volume 32, 2019.
[20] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, pages 9876-9886. PMLR, 2020.
[21] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[22] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intel- ligence 15, Intelligent Agents (St. Catherine's 's College, Oxford, July 1995/, pages 103-129, 1999.
[23] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[24] Jun Shao. Monte carlo approximations in bayesian decision theory. Journal of the American Statistical Association, 84(407):727-732, 1989.
[25] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Proceedings of the Advances in neural information processing systems, volume 29, 2016.
[26] Tonghan Wang. Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly de- composable value functions via communication minimization. In International Conference on Learning Representations, 2019.
[27] Gang Chen. A new framework for multi-agent reinforcement learning centralized training and exploration with decentralized execution via policy distillation. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1801-1803, 2020.
[28] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. JEEE transactions on cybernetics, 50(9):382633)) 3839, 2020.
[29] Jacopo Castellini, Frans Olichock, Rahul Savani, and Shimon Whiteson. The representational capacity of action-value networks for multi-agent reinforcement learning. In AAMAS 2019: The 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1862- 1 1864. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2019.
[30] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards un- derstanding cooperative multi-agent q-learning with value factorization. Advances Neural Information Processing Systems, 34:29142 142-29155, 2021.

---

[31] Wenhan Huang, Kai Li, Kun Shao, Tianze Zhou, Jun Luo, Dongge Wang, Hangyu Mao, Jianye Hao, Jun Wang, and Xiaotic Deng. Multiagent q-learnng with sub-team coordination. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1630-1632, 2022.
[32] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the Interaational Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] The code and instructions to reproduce the results given in github repository: wamoooooxiaoii//eaooo wwtpaooooooooooyaaa____am
https://github.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "oVkjssIRrhg", "reviewer": "Reviewer_So3D", "summary": "This work analyzes the current IGM decomposition framework in CTDE problem, especially the lossy in value decomposition and the accumulation of lossy in the training process. Based on the analysis, this work proposes a method combined with imitation learning to solve the lossy between global observation and local observation of Q (s, U) = \\sum Q(s_i, u_i) in value decomposition.\nThe method part includes an expert agent trained using off-policy algorithm with global observation and a learner agent trained using supervised algorithm with local observation,  through alternately training, the training lossy of learner is alleviated.\n", "questions": "In CTDE, the decentralized execution is constrained with partial observation, the difference between global obs and local obs is determined by the partial obs environment, this problem can not be solved by imitating an expert. I'm not sure it's always good to imitate global state expert, especially, when the learner agent needs to execute independently.\n\nI think some previous works have proposed some methods like using a CTCE teacher and a CTDE student, what's the difference when comparing this paper to these works?", "limitations": "This paper analyses the value decomposition error problem, but the method of this paper is of limited help in solving this problem.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is well organized and well written and the analysis/method part is very detailed, and the definition is more formal.\nIt is a relatively novel perspective to integrate imitation learning into the framework of CTDE  to alleviate value decomposition error.\n\nWeakness:\nThe flaws of IQM-based method is widely studied. This paper has spent a lot of space on analysis, but most of the analysis is not deep enough and I think the analysis result is not innovative enough.\nThe experiment part is insufficient. The setting of the experiment is zero sight SMAC, which is obviously more conducive to the setting of this method, I think adding some experiments with normal sight in SMAC will be better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "leWzAEVRrty", "reviewer": "Reviewer_yNmi", "summary": "This paper rethinks a popular principle in cooperative MARL called Individual Global Max (IGM) decomposition, discovers the limitation of the IGM condition, and proposes an imitation learning strategy (IGM-DA) to address this problem. Due to partial observation, local observation may not provide enough information. Thus, the IGM condition will be a lossy decomposition. This paper trains an expert using a global state during centralized training and distils a decentralized agent using imitation learning. Empirical results show that IGM-DA can outperform state-of-the-art IGM-based approaches with zero sight view.", "questions": "For example, there are two landmarks to the left and right of the agents. Agents should go to the winning landmark. The probability of whether the left and right landmarks are winning landmark is equal. The local observations of each agent cannot see the location of the winning landmark. In this case, local history does not have enough information to make the right decisions. IGM condition is a lossy decomposition due to partial observation and can IGM-DA address this problem?", "limitations": "The main concerns are listed in the WEAKNESSES. The authors should discuss the examples presented in the reviewer's Question to justify IGM-DA.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reviewer will list the main strengths and weaknesses as follows:\n\nSTRENGTHS\n\nIGM is a very popular principle in the value-based MARL algorithm. This paper rethinks the limitation of the IGM principle, which is very interesting and novel. Due to the partial observation, IGM is hard to keep the consistency between local and global greedy action selection. This paper proposes a teacher-student model to utilize the global information during centralized training to address this problem, which is an appropriate way to handle it.\n\nWEAKNESSES\n\n1) To address partial observation of IGM conditions, there is another way to enrich the observation of each agent using inter-agent communication [1]. By discussing this branch of related work, the position of this paper would become clearer.\n\n2) Proposition 5 indicates that when multiple samples satisfy the same local observation $\\tau$, imitation learning will take the average of these samples. The problem is that if the policies on these samples are completely different, imitation learning may lead to a bad policy, i.e., the policy distillation will fail due to insufficient information. The authors should discuss this case to justify the soundness of IGM-DA.\n\n[1] Wang T, Wang J, Zheng C, et al. Learning nearly decomposable value functions via communication minimization[J]. ICLR 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZIN0S4yVz-a", "reviewer": "Reviewer_bd75", "summary": "This paper analyzes the error accumulation in multi-agent Q-learning and focuses on the error caused by partial observability. Regarding this problem, this paper integrates a DAgger module to distill the full-observability policy to the partial-observability policy.", "questions": "Major concerns:\n\n1. What is the **formal definition** of $error$-terms in proposition 2? Does it mean Bellman errors? What is $Error(Q)$? These error terms are the core concepts of this paper and should be defined in formal mathematical formulas.\n\n2. There are no experiment results supporting the error accumulation is reduced from Eq.(7) to Eq.(9) in practice. It raises a concern that the performance improvement may come from any unmentioned implementation details.\n\n3. The loss function of \"supervised imitation learning\" should be included in the main text. I even cannot find it in the appendix.\n\n4. What is the purpose of proposition 5. I do not understand what \"satisfy local observation\" means in proposition 5 and what is \"the optimal action-value after imitation learning\".\n\nMinor:\n\n1. Discount factor is missing in Eq.(7).\n\n2. In Eq.(10), please use the indicator function to represent the one-hot probability.", "limitations": "The discussions on background literature and related work are limited. I do not penalize my evaluation by this point, but I strongly encourage authors to include a related work section in the paper.\n\nAs this paper is titled by \"rethinking IGM\", it would be better to connect with other papers studying the algorithmic property of VD methods (maybe in other aspects), e.g., [1-3].\n\n[1] Castellini et al. \"The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning\" AAMAS 2019.\n\n[2] Wang et al. \"Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization\" NeurIPS 2021.\n\n[3] Huang et al. \"Multiagent Q-learning with Sub-Team Coordination\" AAMAS 2022.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This paper studies an important problem and the idea is novel and makes sense.\n\nWeakness: Although I can roughly capture the main idea of this paper, the notation and definition of many core concepts are not clear (see questions). The writing quality is far from a conference publication.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ATGI8cVL8Y2", "reviewer": "Reviewer_7jPS", "summary": "This paper provides a new perspective on the study of Q-value decomposition methods in the field of multi-agent reinforcement learning. In order to maintain decentralized execution while avoiding learning non-stationarity during the training phase, centralized training with decentralized execution paradigm is widely studied. The authors pinpoint a gap between local and global observation, which leads to the fact that local action-value functions cannot represent the right Q values. Such a representational error may accumulate temporally. The authors use a supervised learning method (with the \"label\" being action values conditioned on global observations) to solve this problem and found it can improve the performance of multiple MARL methods.", "questions": "(1) It is not surprising that local Q functions conditioned on local information are not accurate enough. Can the authors give a bound to $Error_{dec}$? Perhaps by quantifying the ratio of lossy information in $\\tau$. (It is fine if this is too difficult, but the reviewer will fight for the acceptance of this paper if the author can make it and change Figure 3.)\n\n(2) Figure 3 does not give many insights. It should be replaced. Please demonstrate the gap between Q functions depending on local and global information. \n\n(3) Please show the influence of the sight range (changing from 0 to, say, 5). For now, a zero sight range may be too extreme. ", "limitations": "The reviewer thinks the paper has made many interesting findings and does not see obvious negative societal impacts", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Significance\n\nThe paper discusses an interesting issue in CTDE value-based multi-agent learning methods that are ignored in the previous literature. As far as the reviewer is concerned, the paper is interesting, and the contribution may inspire many MARL researchers.\n\n### Quality\n(1) The major concern of the reviewer is that the experimental results cannot fully support the analysis in the previous sections. Figure 3 does not make sense. A smaller sight range must hurt the performance, but it may simply be because they have little information about the other agents and enemies. It cannot be guaranteed that the discussed issue in this paper leads to this performance gap.\n\n(2) Another drawback of the paper is that some results are not surprising. For example, it is well-known that Q functions conditioned on local observations are less accurate than those based on global information. And such an error will certainly accumulate temporally. It would be much more interesting if the authors could bound this error.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept