PAC: Assisted Value Factorization with Counterfactual Predictions in Multi-Agent Reinforcement Learning
Authors: Hanhan Zhou
Tian Lan
Vaneet Aggarwal
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
Multi-agent reinforcement learning

Abstract: Multi-agent reinforcement learning (MARL) has witnessed significant progress with the development of value function factorization methods. It allows optimizing a joint action-value function through the maximization of factorized per-agent utilities. In this paper, we show that in partially observable MARL problems, an agent's ordering over its own actions could impose concurrent constraints (across different states) on the representable function class, causing significant estimation errors during training. We tackle this limitation and propose PAC, a new framework leveraging Assistive information generated from Counterfactual Predictions of optimal joint action selection, which enable explicit assistance to value function factorization through a novel counterfactual loss. A variational inference-based information encoding method is developed to collect and encode the counterfactual predictions from an estimated baseline. To enable decentralized execution, we also derive factorized per-agent policies inspired by a maximum-entropy MARL framework. We evaluate the proposed PAC on multi-agent predator-prey and a set of StarCraft II micromanagement tasks. Empirical results demonstrate improved results of PAC over state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms on all benchmarks.

PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning
Hanhan Zhou The George Washington University hanhan@gwu. edu
Tian Lan
The George Washington University tlan@gwu. edu
Vaneet Aggarwal
Purdue University
vaneet&purdue. edu
Abstract
Multi-agent reinforcement learning (MARL) has witnessed significant progress with the development of value function factorization methods. It allows optimizing joint action-value function through the maximization of factorized per-agent utilities. In this paper, we show that in partially observable MARL problems, an agent's ordering over its own actions could impose concurrent constraints (across different states) on the representable function class, causing significant estimation errors during training. We tackle this limitation and propose PAC, a new framework leveraging Assistive information generated from Counterfactual Predictions of optimal joint action selection, which enable explicit assistance to value function factorization through novel counterfactual loss. A variational inference-based information encoding method developed to collect and encode the counterfactual predictions from estimated bascline. To enable decentralized execution, we also derive factorized per- per-agent policies inspired by a maximum-entrop MARL framework. We evaluate the proposed PAC multi-agent predator-pre -rey and set of StarCraft II micromanagement tasks. Empirical results demonstrate improved results of PAC over state-of-the-ar value-based and policy- -bsed multi-agent reinforcement learning algorithms on all benchmarks.
Introduction
Many real-world reinforcement learning (RL) problems, such as autonomous vehicle coordination [1] and network packet delivery [2], often involve coordination among multiple entities and naturally formulated as multi-agent reinforcement learning (MARL). Factorization- based methods have greatly progressed in dealing with the exponentially growing joint state-action space in MARL. Under the notion of Centralized Training and Decentralized Execution (CTDE), algorithms like VDN [3] and QMIX [4] learn centralized joint action-value function Qtot through monotonic factorization into local per-agent value functions so that Quat can be maximized as long as each per-agent value function maximized by local action selection. Even conditioned
joint state information, monotonic mixing network for Qiot is shown to restrict the representable function class. Despite efforts to mitigate this, e.g., QTRAN [5] and WQMIX [6], in practice, they empirically perform poorly in complex MARL environments with partial observability [7, 8, 9].
In this paper, show that in partially observable MARL problems (as exemplified by multi-state matrix game), an agent's 's ordering over its own actions could impose concurrent constraints on the representable action-value Qio in different states. This restriction causes large estimation errors of Qiat during training. It cannot be addressed by existing methods, e.g., adding a state-value correction
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

term (like QTRAN [5]) or introducing importance weights on dominant state-actions (like WQMIX [6]). It aggravates the relative over-generaie werrgeneralizzttonn problem [10] - when fully decomposed, per-agent value functions only depend on partial observations and local actions. It renders optimal decentralized policies unlearnable when the employed value function does not have enough representational ability. Solving tasks that require significant coordination in partially observable problems remains key challenge.
The key insight of this paper is that an accurate factorization in partially observable MARL problems requires improved representation of the value functions, which is crucial to supporting the learning of optimal decentralized policies. We propose a novel architecture, denoted as PAC, for assisted value factorization with counterfactual predictions. It leverages a counterfactual baseline that marginalizes out an agent's s potential optimal action while keeping all other agents' actions fixed. The counterfactual predictions of potential optimal actions enable (i) training assistive information that generated using the variational inference method to expand the representational ability of value functions, and (ii) directly optimizing the factorization of Qtot through new counterfactual loss. We note that in contrast to communication-base PAC aims to directly improve per-agent value functions in factorization. Minimizing our proposed counterfactual loss together with an information bottleneck loss ensures that such assistive information is relevant, optimal, and succinet for value function factorization in PAC.
methods like NDQ [11]. the use of assistive information in
Relying on the accuracy of assisted factorization, we further decouple the decision decision-making of local value function (through separate policy networks) from value function networks, which allows PAC to maintain full CTDE despite the use of assistive information during training and also enables the maximization of entropy to encourage exploration. The accuracy of assisted factorization makes PAC outperform policy-based methods with factorization like DOP [12] and FOP [13]), especially on difficult tasks that require more coordination among the agents since sub-optimality from one agent's 's policy might propagate and aggravate the training of other agents through the centralized critic [12, 14]. Our key contributions are summarized as follows:
We propose a novel method, PAC, the first framework for value function factorization by providing iitiational-enodde counterfactual predictions assistive information to facilitate per-agent value function estimation.
The counterfactual predictions be efficiently computed from feed-forward baseline Q* and based on a local search to facilitate direct optimization of Qior factorization through a new counterfactual loss.
PAC decouples individual agents' policy networks from value function networks and thus maintains fully decentralized execution while enjoying the benefits of assisted value function factorization. It also leads to an entropy maximization MARL for more effective exploration.
We demonstrate the effectiveness of PAC and show that PAC significantly outperforms both state- of-the-art value-based and policy-based multi-agent reinforcement learning algorithms on the StarCraft II micromanagement challenge in terms of better performance and faster convergence.
2
Model and Background
Model: Consider a fully cooperative multi- aulti- task as decentralized partially observable Markov decision process (DEC-POMDP) [15], given by a tuple G = SI, U, P. r, Z,O. n, 7), where I= {1, 2, ... ..n] .n}
is the finite set of agents. The state is given as s S, from which each agent draws its own observation from the observation function 0; E (s, i) S x A-0. At each timestamp 4, each agent i choose an action u, E U, composing a joint action selection u. A shared reward is then given as r a)= XU-RR with the next state of each agent is s' with transition probability function (s'lss u): S U + [0, 1]. Each agent has an actionnobservattin history T, E T = (0 x U'* from its limited local observations. Then the overall objective is to find a ajointt policy = (T.,.... .... Tn) which corresponds to the joint action-value function Q(+,, u.) = E[ (t. ue]. that is used to maximize the joint policy function V(T, u) = Ega.se 20 D 7'ri|s0 8, Ug u, т| and E [0, 1) is the discount factor. Quantities in bold denote a joint quantities across all agents, and quantity with super script denote a quantity specifically belong to agent i.
Following the paradigm of CTDE, VDN [3] and QMIX [4] are popu- lar and representative methods for value function decomposition which learn centralized action value Qtot through value decomposition assuming its additivity and monotonicity. VDN, per- agent sum of the local value is used to calculate the action value Qiot Ea=1 Qa(Ta, ua). In
Value Decomposition:

---

QMIX, monotonic mixing function of each agents' local utilities is proposed as Q (T, u, s; fo (s, (TI,))))) ai)...,,
(n)),,nn)), where "Q* (T..)
(T.. O, Vi N. The monotonic mixing function is able to ensure the global optimal Q yields the same result the set of individual optimal Q where fo is approximated by the monotonic mixing network parameterized by 8. The weights are generated by a separate hyper-network that conditions on the global state, where its monotonicity is ensured by performing absolution function on generated parameters for non-negative mixing weights. Then QMIX is trained in way like DQN [3] with goal to minimise the squared TD error on minibatch of b samples from the replay buffer as E
u, s; 0) - y where ), r is the global reward and 6' is the parameters of the target
yo =r + max,' Q (T', u', :: (),, network whose parameters are periodically copied from B for training stabilization.
3 Existing Limitations in Partially Observable Multi-state Problems
State 1
State 2
State
State 2
0
C
01
B
0,3
-12
-2.6
0.4
-1.3
-2.2
-2
-2
A
-0.2
0.1
-1.0
-1.0
-0.2
0.4
-1.0
-1.0
0
-2
0.3
-09
-1.0
0.3
1.4
-0.9
-1.0
0
C
0
-2.6
110
1.0
-1.0
-2.6
-1.0
-1.0
-1.0
(a) Payoff of 2-state matrix game
(b) QMIX: O2 Qg Awt
State
State 2
State
1.0
0.2
0.2
0.0
0.5
0.5
0.7
-2.0
-2.3
0.6
-2.1
-2.0
1.0
4,0
4.2
-12
1.0
-1.2
0.1
0.1
0.7
4.0
-2.1
-2.1
-1.2
-18
-2.5
-2.5
0.0
0.2
1.6
-1.6
0.0
-1.6
-122
-1.2
-200
2.1
-244
-2.4
1.6
4.0
2.0
-2.0
0.0
-.2
1.6
-1.6
0.0
-1.
-122
-1.2
2.1
21
-24
-2.4
-1.8
2.1
-2.5
-2.5
(c) WQMIX: Q2 Og Qur
(d) PAC (ours): Q2 Qg Owr
Figure 1: In partially observable multi-state games, Qeot is limited to a restricted function class imposing concurrent inequality constraints on Qtot in different states. causes large factorization error and thus erroneous computation of argmax (Qtor). PAC successfully addresses this problem by leveraging assistive information trained using counterfactual predictions, with the direct goal of achieving better value function factorization.
The limitations of monotonic value function factorization have been identified via single-state matrix games, which inspired new algorithms like QTRAN [5] and Weighted-QMIX [6]. In this paper, show in multi-state problems with partial observability, that one agent's ordering over its own actions could impose concurrent inequality constraints on the joint action-value Qtot in different states, resulting in restrictive function representations of Qtot with large estimate error.
states with 0.5 transition probabilities between them and two payoff matrices shown in Figure 1(a). Suppose that agent has the same partial observation 01 in states s!)) and s!2), Then, its per-agent value function (., .1) computed from partial observation o₁ are also the same in both states. Due to the monotonicity of the mixing network (even though it is provided with complete joint state information), for any un and u' with ordering gi (M1, 71) 2 qı (u', without loss of generality, we must simultaneously have Qior (u1, u2
Consider a Markov decision process (MDP) consisting
s()) 2 Quo U2 s()) and Quor(up, u2 222) 2 Qtorlu', u2, s(2) ) for any action us
of agent 2 in both states. Representing Qto: on this restricted function class results in significant error QMIX as shown Figure 1(b). Although Weighted-QMIX introduces an importance weighting on the dominant state-actions of this game, it only improves the approximation in state and yet causes even higher error in state 2 (in Figure I(c)). This exactly because of the inequality restrictions simultancously imposed on both states, limiting the representational ability of the value functions in partially observable problems.
Clearly, additional information is needed to facilitate successful factorization in these partially observable multi-state problems. It also worth noticing that even with agent-wise communications, NDQ [11] also fails the task since the communication messages and related loss functions are not designed to drive better factorization. It underscores the importance of making effective use of the right information for successful factorization. We put the results from other methods in Appendix A.3. Our proposed PAC (in Figure 1(d) addresses this issue by leveraging assistive information trained by novel notion of counterfactual predictions. More preciscly, counterfactual predictions of potentially
3

---

optimal agent actions are readily computed from an unrestricted, feed-forward baseline Q", Training per-agent value functions using new counterfactual assistance loss leads to substantially improved estimate Qnn and correct (argmax Qtot) in different states. We compare PAC with a number of state-of-the-ar value-based and policy-based methods with function factorizations and demonstrate its performance in challenging partially observable tasks that require significant coordination.
4 Methods
To overcome the limitations of existing Value Decomposition methods discussed in Sec. 3, in this section, we introduce the idea of assisted optimal joint policy factorization and propose such method under the multi-agent soft actor-critic framework.
Framework overview. Fig. 2 shows the architecture of the learning framework. There are three main components in PAC : (1) weighted Q utilizing per-agent local critics (1,,,, Ta, m,). where m, serves as assistive information aiding value factorization, (2) an unrestricted joint action estimator Q', which serves as a baseline estimator of Q" and allows the computation of counterfactual predictions from quick local search, and (3) assisted information generating module, which is able utilize the deep variational bottleneck method to encode the counterfactual optimal joint action selection. In addition TD errors for Qtot and Q", the PAC system is trained using three more loss functions: counterfactual assistance loss I.C.A for optimized value function factorization, information bottleneck loss LL for succinet and effective assistive information generation, and local policy loss LLP for training factorized agent policy with entropy maximization.
One Lass Mis uit (A.K.M) y>>
Ocor (Fp u+)
Qtot
Info Bottleneck Loss
Mixing Network
St
qE(ME, m..
qi(u,, th m; .. (MG, xf. m)
(R), -... (2F)
Agent
Agent
Agent
m
0
VAE
of
gfcs Ti. m
Q
0' @ (s.r.9) ->0
Local Po Icy
Local Critic
02
(0G) QTT)
T6. = argmax F. .. Mo, St.
Local Policy Loss L,p
Counterfactual Assistance Loss Lçn
Figure 2: The overall architecture of PAC. With the help of assistive information, m. counterfactual predictions iz - which are obtained through an approximation of optimal Q" and an efficient local search are used to directly train the per-agent value functions g.(-, ,/. m, with respect to a new counterfactual assistance loss. It results in significantly improved value function factorization for partially-observable MARL problems.
Generating Counterfactual Predictions One of the key insights underlying this method is that the optimal joint action selection u" from the centralized Q* can be used as a direct information assisting Qeot explicitly. Although the complexity of computing u* argmax, argmax,E[("" EEQ"(s, r, -)] is impractical as it grows in O() however, it is possible to compute local estimation of ú as a; argmax (s T, u...). which reduces the computational complexity from exponential to linear level of O(n UI).
However, even with u' provided, in what manner it can benefit value function factorization is not researched. Directly feeding this to the policy or critic network is counterproductive as the neural network can easily learn that this specific input is the key to reducing the TD-error while it is not helpful to explore or train the local policy and the training process might stall. Using an extra loss function, e.g. cross-entropy, an effort to train the network to make decisions that are close to u²

---

sounds promising, however, it only quantifies the error without actually guiding the training since it may trap the policy in local sub-optimum and further limits the exploration process.
Inspired by difference rewards [16] and counterfactual baseline [17] for policy gradients, we propose counterfactual assistance loss. For each agent we can use an advantage function that compares the i" from Q2 and u from Qtot to counterfactual baseline that marginalizes out the agent's potential optimal action which relegates ig while keeping all other agents' action u -a fixed, a counterfactual assistance loss is proposed as:
CC σ) = (uggr(u)17))
Σ E/v-
iq(0,, oz, m.) - Tlu,, oz)q(u,, m,)l
(1)
This looks similar to calculating the counterfactual advantage as used in COMA, however, in COMA such counterfactual advantage is calculated for centralized critic which is computationally expensive and unstable which limits its performance, while our counterfactual assistance loss providing direct guidance in training the policy network towards the direction of ut. We later show in ablation studies that this loss contributing to the performance improvements.
Generating Assistive Information Additionally, we show û" can be encoded and then used as input for local critics qi (u,, Tap m4), as assistive information aiding value factorization. As previous works suggests [11, 18]. we consider the information bottleneck method [19], with the Markov chain o-m o-m-ü" during encoding. To be specific, we regard the internal representation of the intermediate layer as stochastic encoding m; of input source or, with the goal to learn an encoding that is maximally informative about the result úØ. Formally, the objective for each agent i can be written as:
32 Σ [Te... (a::m. m_j) - -Bee... (m4j01] f=1
JIB (f m))
my is an instance of random variable of assitive information that is drawn from multivariate Gaussian distribution N ((m((o;; 0m), I)). O. is the parameters of encoder J' and I is an identity matrix, a Lagrange multiplier parameter 3 2 O is used to control the trade-off between the encoding the necessary information and reaching maximal compression. Yet this does not lead a learnable model, since this mutual information / is intractable. With the help of variational approximation, specifically, deep variational information bottleneck [20], we are able parameterize this model using a neural network. We then derive and optimize a variational lower bound of such objective as
LIB (0m) = Eo.-D.m, ~m |- H. ((ppp(((j)))) gs (0710z, m)] + 3DKL (p(m: (m))))) (3)
where H is the entropy operator, DRL. denotes Kullback-Leibler divergence operator and (m:) is variational posterior estimator of p(m:) with parameters Φ Using the loss above a message encoder fm with parameters 0m is trained to generate information m, ~ (f (o fm), that is useful for decision making. Compared to [11, 18] that encodes the general state information and other agents' action selections communication messages, which can not reduce the uncertainties in action-valle vallue functions; using the encoded û" as assistance information can provide an explicit direction toward better individual value estimation and thus joint value factorization. Detailed derivations and proofs can be found in Appendix A.
Factorized Policy Iteration with Entropy Maximization We leverage factorized policies to maintain decentralized execution PAC despite the use of assistive information for training. Recent works have shown that Boltzmann exploration policy iteration is guaranteed to improve the policy and converge to optimal with unlimited iterations and full policy cvaluation[21], within MARL domain it can be defined as: (TT) E, E [r (st, uz) + aH (н (-[s:)) where denotes the temperature parameter that is used to adjust the balance between maximizing the entropy for a better exploration and maximizing the expected reward. We present one possible method of expanding this to MARL problems, to achieve decentralized policies in PAC and to encourage efficient exploration.
Several recent works are proposed to expand actor-critic or soft-actor-critte in to factorization based MARL methods [13, 12, 14], they all follow a centralized critic with decentralized actors (CCDA) framework. In this work, we train the actors in a centralized but factorized way. Unlike [14] that reuses the local utility network for both actor or critic or [13] which consists of soft V-network and soft Q D-network for local policy net, we use separate network as policy networks and propose
5

---

centralized but factorized soft policy iteration with factorized Maxiimum-Entroppe trained as:
(LP(T) = Ep [a log π (ug(Te) - Qioe (st. TE, ut m,)l
(4)
- mix (st, E.z [o" (T's "/, m;) - a' log (u]lr()))
where qmit is the monotonic value decomposition network with u, (o())) and D is the replay buffer used to sample training data.
As noted in previous research, choosing the temperature term in soft-actor-critic is non-trivial as it can vary in unpredictable way when the policy becomes better as the training continues [13]. Instead of finding each individual temperature using approximation functions, since we use a centralized but factored policy, we consider one global temperature that is automatically adjusted by considering it as constrained optimization problem as parameter that is trained independently with loss:
a(a) = E |-a' log क (ue (sc) - aHHo,,
(5)
where Ho a fixed entropy term so the temperature term a generally decreasing such that the degree of exploration is reduced as the training proceeds[21].
Training Process So far have discussed the components of our method, formulate the new scalable RL algorithm, we now explain the implementation and the centralized training process for deep RL under (marked as local policy in Fig.1 is enabled so that full CTDE is maintained.
In the decentralized execution phase, only the policy (agent) network
Despite that monotonicity constraints limitations to the expressive power of the mixing network, recent research [14, 22] demonstrated that the IGM principle [5] as equivalent to joint greed actions and individual greedy actions is crucial, it greatly promotes the sample efficiency; meanwhile most designs with only one unrestricted joint action-value function show poor empirical performance(5, 23], and thus we follow the design of WQMIX and keep both Qiot network and Q* network.
The Q architecture (Green part in Fig. 1) used as the estimator for Q" from unrestricted functions, where its mixing network is a feed-forward network that takes its local utilities. While the Qtor module (blue part in Fig. ) looks similar to QMIX and many other factorized methods, there is significant difference, for its local estimator is provided with assistive information and the local value feeding to the network q(u, Tr, m.) is selected based on local policy, rather than taking argmax qe(, ; Téy m:). Then Qtoe and Q" are trained with the objective to reduce their respective loss
(0) = Σ (s,
(,, -.))
(6)
LQuar (0) = E"m(s urs8 u) T, u, m) yz)²
where ú, = argmax (:, (:, Tr, m²), 34 = + T', argmax Qior ú', s s':0) with 0 being the parameters of the target network that are periodically updated to stabilize the training. u, (o,). and w(s, u) is the weighting function' to ensure the weighted projection can recover the correct maximal joint action value function for any Qí6). Note the action selection for loss function in method is different from the original design of QMIX and WQMIX. Apart from the independently updated entropy term a, all other components (including the message encoder) end-to-end manner with the objective to minimize the weighted sum of all losses proposed above, including the counterfactual loss and information bottleneck loss as
trained in an
(()) LLP + LC + LIB + f + LQuet
Detailed derivations can be found in Appendix A.1
5 Experiments
In this section, we compare the results with several state-of-the-art MARL methods on Predator-Prey [24] and selected StarCraft Multi-Agent Challenge (SMAC) [8] scenarios as benchmarks. More details on implementation, experiment settings, and hyperparameters are included in Appendix A.3. Code is available
In this work we follow the weighting function design of OW-QMIX in WQMIX [6]. as = 1 if Quoe(s, T, m) ys <0, otherwise.

---

5.1 Cooperative Predator-Prey
We first consider Predator-Prey [24], which is a involves 8 agents in cooperation predators catch built-in-Al controlled units prey 10x10 grid. Only when two or more predator agents surround and capture the same prey at the same time, it is considered successful capture. We consider two settings: simpler task without failed capture punishment and a harder task with a punishment reward of -2 when a capture attempt fails. This is test the baseline algorithms on relative over-generalization and monotonicity constraint limitations. More details about this environment can be found in Appendix A.3.2.
multi-agent environment that
40
INGVIO
VIAG
-100
-158
T(9k))
7.00
(a) PP without mastur penalty
(b) PP with mis mistcapture penalty
Figure 3: Results on Predator- Prey benchmark
As shown in Fig, 3a, for the easy task value-based method like QMIX, WQMIX can learn a policy that reaches the highest reward. However, in this environment, we see the performance gap between state-of-the-arr value-based and policy-based methods since policy-based methods might suffer from relative over-generalle wvergenerralizatoon problem [9. 25, 26] or making a poor trade-off in joint policy representation. Our work as an actor-critic method can match the highest performance, although it takes a slightly longer time to converge, potentially due to the training of the variational inference models.
On the other hand, for the harder task that requires significant coordination among agents as shown in Fig. 3b, increased exploration and joint representation capabilities are required to finish the task which makes WQMIX and PAC the only two methods capable of learning usable policy. Intuitively, when joint actions from uncoordinated decision-making occur more than coordinated ones, the penalty term will then dominate the average return from the environment and further the value estimation of each agent's 's local utility. We attribute these performance improvements to the use of entropy maximization which promotes more exploration attempts and the higher coordination abilities from assistive information.
5.2 SMAC
We then consider the SMAC as the second benchmark, wherein each agent controls a unit cooperat- ing with other friendly units in combat against the game's built-in Al-controlled units. The combat can be symmetric (same units for both parties) or asymmetric. Since it is shown that most state-of- the-art algorithms perform really well on easy and medium maps, which limits the demonstration of clear comparison and the potential improvements, we begin our test six maps including two hard maps (5m_vs_6m) f6,, 3s_vs_50 and four super-hard maps (MMM2, 27m_vs_30m, corridor). Selected maps are classified as hard or super hard due to (i) very large action space like 27m-vs-30m, (ii) requiring advanced exploration strategies like corridor (iii) requiring a higher level of coordination between the agents like 6h_vs_ vs_ vs_8z etc. We use the same default environment setting for all benchmark algorithms throughout the test. Each baseline algorithm is trained with 4 random seeds and evaluated every 10k training steps with 32 testing episodes. Details of the environment setup and hyper-paramett settings are listed in Appendix A.3.3. We choose state-of-the-art MARL algorithms as baseline including decomposed actor critic method: FOP [13] and DOP [12], decom- posed policy gradient method: VDAC [14], decomposed value based method: QMIX [4], QPLEX [9]
6h_vs_8z
II this paper all SMAC experiments are carried out utilizing the latest SC2. 4. 10, performance is always not comparable across versions. We implemented our algorithm based on an open- sourced codebase [22] and acquired the results of QMIX and WQMIX from it.
7

---

PAC
RAC
PAC
QMX
GHX
GMIX
K30
NDQ
WOMIX
VOMI
WOMX
VOAC
VOAC
POP
POP
POP
DCP
(a) 5m vs. 6m (hard)
(b) 3s vs 5z (hard)
(c) MMM2 (super hard)
NDQ
WOME
WOME
VOAD
VDAC
FDP
DOP
1.00
23
110
(d) 27m__mm _ap___s__0mmm (super hard)
(c) 66_ Sh__s__r (super hard)
(f) corridor (super hard)
Figure 4: Results on SMAC benchmark
and WQMIX [6] and communication enabled value-based method: NDQ [11]. Results shown in Fig.4. Overall, our method achieves the highest win rate compared to the baseline algorithms in terms of higher performance faster convergence, especially those maps that require more exploration and agents' coordination. As previous research demonstrated, there a gap between SOTA value-based method and policy gradient methods, as the performance for most of them is limited on those maps that require extensive exploration techniques. Specifically, on 3s_vs___ vs_ ___z and vh_ve___ va_v____. 8z, PAC able train a usable policy that outperforms all baseline algorithms. On corridor PAC and the selected two value-based methods are able to learn model, with our method converging faster with slightly better performance, while policy-based methods suffer from this map as it requires more exploration to find the specific trick in winning this challenging scenario. Finally on relatively casier maps, although most baseline algorithms hold relatively close performance, the value-based and policy-based method performance gap still exists. Although FOP recently claimed to be the first multi-agent actor-critic method that outperforms state-of-the-art value-based methods on SMAC, we empirically found its limits when the chosen environment is substantially complicated and harder. We especially observe that our method as an actor-critic method has over-performed SOTA value-based MARL methods and brought significant improvements for actor-critic MARL.
5.3 Ablation Studies
We conduct ablation experiments to validate the effective- ness and contribution of each core component introduced in PAC on MMM2 scenario in SMAC. Namely, in Fig.5 we consider verifying the effect of (1) optimization of temperature term policy I(a) by fixing
PAC PAC_ Flxed _n
PAC no_ _nfo
PAC, CF Joes
PAC disabdd
0.5
PAC_Ne_o
PAC_fixeda, (2) assisted information loss LIB by disable as PAC no info, (3) counterfactual assistance loss LC by replace it with a simple cross-entropy loss as PAC_CE Loss, (4) disable both L1 and LCA while substituting all a* with â as PAC_ _i__diabled and (5) further remove the Q* structure from PAC disabled as PAC NN "" In this way, overall we can observe PAC outperforms ablated versions, especially by large margin compared PAC disabled which indicates that general encoding of the state infor-
Figure 5: Ablations results comparing PAC and its hyperparameter-tun -uunee ab- lated versions SMAC map MMM2
In this section refer WQMIX ow-qmix it shows general better performance than ew-qmix.

---

mation without explicit direction for training is not helpful and may also harm the training.Althougg PAC_CE Loss later acquires competitive results, it takes a longer time, potentially due to the cross-entropy loss in this case trapped the training in local sub-optimum. By fixing the entropy term a 0.5 seems to be a balance point for maximizing the reward while promoting the exploration, yet it brings a performance drop, indicating the importance of updating the entropy term with its loss previously shown. PAC_No cCC_No_Q" suffers from the most significant performance drop after most core components are removed from the original design. Such results validate how each component is crucial for achieving performance through experiments.
5.4 Limitations
Although the empirical results of PAC demonstrated improvements over both SOTA value-based and policy-based methods, at this stage, no strict convergence guarantee to the optimum is provided due to the scope of research. We also observe higher variance in terms of the performance of PAC than value-based methods, this might be because of the entropy-mand maximization penalty. Also, we follow the tradition of parameter sharing among the agents, thus the role assignment of formulating distinctive behaving agents, which another important topic was not considered.
Related Works
Cooperative multi-agent decision-maker often suffers from exponential joint state and action spaces. Multiple approaches including independent Q-leaming and mean-field games have been considered in the literature, while they do not perform well in challenging tasks or require homogeneous agents [14]. A paradigm of centralized training and decentralized execution (CTDE) has been proposed for scalable decision-makinn the joint value function. Some of the key CTDE approaches include value function decomposition and multi-agent policy gradient methods.
[27]. QPLEX [9] takes a duplex dueling network architecture to factorize
Policy Gradient methods are considered have more stable convergence compared to value-based methods [28, 29, 13] and can be extended to continuous action problems easily. A representative multi-agent policy gradient method is COMA [17], which utilizes a centralized critic module for estimating the counterfactual advantage of an individual agent. DOP [12] uses factorized policy gradients with architecture similar to Qatten[30]. However, as pointed out in [31, 23], multi-agent policy-based methods like MADDPG[32] are still outperformed by value-based methods StarCraft multi-agent challenge (SMAC) [8].
Decomposed actor-critic methods, which combine value function decomposition and policy gradient methods with the use of decomposed critics rather than centralized critics, are introduced to guide policy gradients. VDAC [14] combined the structure of actor-critic and QMIX for the joint state-value function estimation, while DOP [33] directly uses a network similar to Qatten [30] for policy gradients with off-policy tree backup and on- on-policy TD. The authors of [33] pointed out that decomposed critics are limited by restricted expressive capability and thus cannot guarantee the convergence of
global optima; even though the individual policies may converge to local optima [13]. Extensions of the monotonic mixing function have also been considered, e.g., QTRAN [5] and weighted QMIX [6]. But solving tasks that require significant coordination remains a key challenge.
Another related topic is representational learning in reinforcement learning. A VAE-based forward model is proposed in [34] to learn the state representations in the environment. [35] considers a model to learn Gaussian embedding representations of different tasks during meta-testing, The authors in [36] proposed a recurrent VAE model which encodes the observation and action history and learns a variational distribution of the task. NDQ [11] encodes the state information as communication messages between agents. [37] use an inference model to represent the decision maasinnn of the opponents. RODE [38] uses an action encoder to learn the action representations in restricting the role action spaces for a reduced policy search space. MAR [39] learns the metarepresentation for generalization problems. Unlike previous work, our method focuses on learning information from counterfactual predictions that are explicitly assistive for local estimation and efficient factorization.

---

7 Conclusions
In this paper, we propose PAC, multi- framework utilizing extra state information as assistance for a better value function factorization, under a centralized but factored soft-actor critic setting. With the newly proposed counterfactual optimal joint action selection used for training and encoding assistance information, empirically we show our method not only matches or outperforms both -ate-offtheeart policy-based and value-based MARL algorithms on selected benchmarks but also bridges the gap between the two. Future work will also explore more effective ways to formulate and utilize extra state information to accelerate the training and tackle tasks in more complicated environments.
References
[1] Yeping Hu, Alireza Nakhaci, Masayoshi Tomizuka, and Kikuo Fujimura. Interaction- aware decision making with adaptive strategies under merging scenarios. In 2019 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 151-158. IEEE, 2019.
[2] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. sensors, 15(5):10026-00477 10047, 2015.
[3] Peter Sunehag. Guy Lever, Audrunas Gruslys, Wojciech Marian Czameeki, Vinicius Zambaldi, Max Jaderberg. Mare Lanctot, Nicolas Sonnerat, Joel 7. Leibo, Karl Tuyls, al. Value-decompossitonn networks 2017.
for cooperative multi-agent learning. arXiv preprint /706.
[4] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295 4304, PMLR, 2018.
[5] Kyunghwan Son, Daewoo Kim, Wan Kang, David Earl Hostallero, and Yung Yi. Qtram: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMIR, 2019.
[6] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. 2020.
[7] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. arXiv preprint arXiv: arXiv:/9/0.0000 /9/0.0743, 2019.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr Jakob Foerster, and Shimon Whiteson. The starcraft aueti-agent challenge. arXiv preprint arXiv:1902.04040 2019.
[9] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjic Zhang. Qplex: Duplex ducling multi-agent g-lcarming. arXiv preprint arXiv:2008.01062 2020.
[10] Jacopo Castellini, Frans A Oliehoek, Rahul Savani, and Shimon Whiteson. The representational capacity of action-value networks for multi-agent reinforcement learning, arXiv preprint arXiv:1902.074991 07447, 2019.
[11] Tonghan Wang. Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. arXiv preprint prXiv:/910.055 2019.
[12] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
[13] Tianhao Zhang, Yueheng Li. Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entt -entrnpp multi-agent reinforcement learning. International Conference on Machine Learning, pages 12491-12500. PMLR, 2021.
[14] Jianyu Su, Stephen Adams, and Peter Beling. Value-decomposition multi-agent actor-critics. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages I 11352-1 13300, 2021.
[15] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[16] Scott Proper and Kagan Tumer. Modeling difference rewards for multiagent learning. In AAMAS, pages 1397-1398, 2012.
10

---

[17] Jakob Foerster. Gregory Farquhar. Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[18] Hanhan Zhou, Tian Lan, and Vaneet Aggarwal. Value functions factorization with latent state information sharing in decentralized multi-agent policy gradients. arXiv preprint arXiv:2201.0/247, 2022.
[19] Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0000057, 2000.
[20] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 2016.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[22] Jian Hu, Siyang Jiang. Seth Austin Harding. Haibin Wu, and Shih wei Liao. Rit: Rethinking the importance of implementation tricks in multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479) 2021.
[23] Kyunghwan Son, Sungsoo Ahn, Roben Delos Reycs, Jinwoo Shin, and Yung Yi. Qtran++: Improved value transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2006.1201 12010, 2020.
[24] Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In International Conference on Machine Learning, pages 980-991. PMLR, 2020.
[25] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pages -9411 PMLR. 2021.
[26] Liviu Panait, Sean Luke, and R Paul Wiegand. Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions on Evolutionary Computation, 10(6):29-645,, 2006.
[27] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentral- ized planning. Neurocomputing. 190:82 94, 2016.
[28] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. Springer, 2017.
[29] Bei Peng. Tabish Rashid. Christian Schroeder Witt, Pierre-Alexandre Kamienny, Philip Tor. Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34, 2021.
[30] Yaodong Yang. Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.0332 039399 2020.
[31] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Comparative evaluation of multi-agent deep reinforcement learning algorithms. arXiv preprint arXiv: 2006.0869, 2020.
[32] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbecl, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettine environments. Neural Information Processing Systems (NIPS), 2017.
[33] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjic Zhang. Off-policy multi-agent decomposed policy gradients. arXiv preprint arXiv:2007. 12322, 2020.
[34] David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1rXiv:1803.1011 2018.
[35] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International conference machine learning, pages 1802-1811. PMLR, 2018.
[36] Maximilian Igl, Luisa Zintgraf, Tuan Anh Lee, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pages 2117-2126. PMLR, 2018
[37] Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi- agent systems, arXiv preprint arXiv22001.1022, 2020.

---

[38] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles decompose multi-agent tasks. arXiv preprint arXi::201000523,, 2020.
[39] Shenao Zhang, Li Shen, and Lei Han. Learning meta representations for agents in multiageem reinforcement learning. arXiv preprint arXiv:2108. 12988, 2021.
[40] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1r12::3312.6114
2013.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] All code, sample data and running instructions are provided in supplementary files for reproducing the results
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training details are provided in Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We use random seed and all algorithms are evaluated in different runs
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] The details of experiment settings are provided in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We use Apache License for our code
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] We use and modify open-sourced code under Apache License (c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Our data does not contain such contents.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "gJrCt70mnzR", "reviewer": "Reviewer_i5it", "summary": "This paper leverages Assistive information by Counterfactual Predictions of optimal joint action selection, which use a novel counterfactual loss to assist value function factorization. The paper develops a variational inference-based information encoding method to collect and encode the counterfactual predictions from an estimated baseline with the Markov chain o-m-ˆu∗.", "questions": "On page 3, line 106, the paper says that there are the same $$ in different states $ s $. Can you explain why?\n\nWhether the conditional mutual information I(u_j;m_i|o_j,m_-j) given in formula (2) still conforms to the original theory.\n", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "originality：\nThis paper uses a deep variational bottleneck to encode the counterfactual optimal joint action selection. our loss functions are set up to assist in the value decomposition between agents. This method is novel. The training assistive information is generated using the variational inference method to expand the representational ability of value functions.\n\nquality, clarity:\nThe article is well written. Some symbols need to be clearly defined.\n\nsignificance:\nThis paper introduces the information bottleneck into the multi-agent cooperative task, which is of significance to the application of the information bottleneck method in this field.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zdej_fpYnP", "reviewer": "Reviewer_QNBi", "summary": "This paper aims to improve the joint-action Q-values learnt in a cooperative MARL setting under the paradigm of CTDE. The authors aim to do so by first identifying the shortcomings of existing methods, QMIX and WQMIX in particular, under significant partial observability. To remedy the errors, during the centralised training they allow for agents to communicate and explicitly encourage them to send messages that improve the learning of the optimal actions for each agent.", "questions": "1. Could you expand on the derivation on Eqn 4 (more details on specific questions are in the weaknesses section).\n\n2. It seems that NDQ performs terribly in your SMAC experiments, is that expected?\n\n3. What weighting function are you using for the WQMIX baseline and your algorithm in Eqn 7?\n\n### Minor:\n\nAbstract - Monotonicity isn't required for maximisation (e.g. QPLEX).\n\nLine 191 - The justification for using the maximum entropy framework is a little misleading. It can help exploration in practise, but to prove it converges the paper uses full policy evaluation and iteration which is generally not feasible in a practical setting.\n\nFootnote 1 pg6 - I don't think the built-in AI is really considered harder than the prev version. The strength of the units has changed (colossus in particular), which makes some maps much easier.\n", "limitations": "-", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strs\n\nSection 3 is good for motivating the central issue the paper aims to address. It could be made clearer that these are the joint-action $Q$-values learnt during the centralised training where communication *is* allowed for PAC (but not for the other methods).\n\n\n## Weaknesses\n\nNo mention of QPLEX in this paper at all (except for a reference related to over-generalisaiton). At the very least it should be in the related works section.\n\nSending messages to be maximally informative about which action to take is a nice idea. However, to really drive this point home there should be an ablation where this part of the loss is turned off (apologies if there already is, it was unclear in Section 5.3).\n\nI didn't follow the derivation of Eqn 4 in the Appendix. What is $k^i(s)$? Did you mean to have the expectation be over $\\pi^i$ for the first line? \nAre you missing a summation over all possible joint-actions in the 3rd line?\nWhy expand the expectation going from line 2 to 3, when you just bring it back in line 5?\nHow do you get to the last line from the penultimate one? The expectation changes from $\\pi$ to $\\pi^i$ and you also introduce $q^{mix}$ and drop the $b(s)$.\n\nThe results look fine, but I would have liked to have seen another environment that exhibited the same kind of issues as Fig 1.\nFrom the results it appears that the issues due to partial observability aren't that much of a concern in these benchmarks.\nIs that the expected outcome? What kind of settings would significantly benefit from your approach (in the sense that something like WQMIX fails completely but yours doesn't)?\n\nThe weighting function you are using for the WQMIX baseline and your algorithm in Eqn 7 should be described.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UmKwG1M-Zwq", "reviewer": "Reviewer_wLBP", "summary": "This paper aims at addressing the estimation error caused by partial observation in value decomposition methods. A new framework leverages assistive information generated from counterfactual predictions of optimal joint action selection, which guides the Q values of each agent. Experiments show that the proposed method overperforms many baselines including QMIX.", "questions": "1. Authors mention QTRAN in the paper but do not compare it in the experiments and Figure 1. Is there any reason?\n2. One existing work [1] is missing, which is the state-of-art method in this domain. I think it can also address the problem shown in Figure 1 since it has complete expressiveness.\n\n[1] QPLEX: Duplex Dueling Multi-Agent Q-Learning. Wang et al. 2020", "limitations": "null", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: 1. novel method to use counterfactual loss in value decomposition methods. 2. Experiments based on StarCraft II show the improvement of the proposed method. 3. Good writing.\nWeaknesses: 1. Some existing works are not compared. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pd_e8z4U3Ko", "reviewer": "Reviewer_cYxS", "summary": "The paper proposes that in partially observable MARL problems, the agent's ordering of its own action may impose concurrency constraints on the representation function, resulting in estimation error at training time. To address this problem, the authors propose that PAC uses counterfactuals selected from the best joint action. The auxiliary information generated in the prediction is explicitly aided by a new counterfactual loss on the decomposition of the value function. Variational inference-based information encoding is also proposed to collect and encode the counterfactual predictions from the estimated baseline. To achieve distributed decision making, thus deriving a factorization strategy for each agent inspired by the maximum entropy MARL framework, and the benefits of the algorithm are experimentally illustrated.", "questions": "* Is it possible to design further ablation experiments to illustrate the specific effects of the different parts? Instead of just performance, I am curious about the actual impact of the different designs in the experiments.\n* Are the hyperparameters used in the ablation experiments the same parameters used in the training? Was fine tuning of the hyperparameters done after the ablation? Intuitively, if the original training hyperparameters were used, then removing a certain part would definitely affect the training performance, so further hyperparameter tuning would be needed.\n* The experimental results in the paper do not look too good, does it come from the detailed differences in code implementation? The difference in code implementation has been widely concerned in the ma field, so I suggest the authors use a more fair codebase for comparison, e.g., https://github.com/hijkzzz/pymarl2", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n* paper writing is clear and better understood\n* The motivation is clear and easy to understand\n* The introduction of matrix game example is very intuitive\n* Many comparison methods in the experimental part\n* There are ablation experiments to illustrate the effect of different parts on performance\n\nWeaknesses：\n* The ablation section just tests the impact of different parts on the performance, without further analysis of what is the specific manifestation of the impact of each part?\n* The authors claim that their method is SOTA, but the performance of the experiment is puzzling, for example, the performance on smac environment does not look good. Are there some differences in code implementation details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept