An Adaptive Entropy-Regularization Framework for Multi-Agent Reinforcement Learning
Authors: Woojun Kim
Youngchul Sung
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Agent Reinforcement Learning
Entropy Regularization
Exploration-Exploitation Tradeoff

Abstract: In this paper, we propose an adaptive entropy-regularization framework (ADER) for multi-agent reinforcement learning (RL) to learn the adequate amount of exploration for each agent based on the degree of required exploration. In order to handle instability arising from updating multiple entropy temperature parameters for multiple agents, we disentangle the soft value function into two types: one for pure reward and the other for entropy. By applying multi-agent value factorization to the disentangled value function of pure reward, we obtain a relevant metric to assess the necessary degree of exploration for each agent. Based on this metric, we propose the ADER algorithm based on maximum entropy RL, which controls the necessary level of exploration across agents over time by learning the proper target entropy for each agent. Experimental results show that the proposed scheme significantly outperforms current state-of-the-art multi-agent RL algorithms. 

Under review as a conference paper at ICLR 2023
AN ADAPTIVE ENTROPY-REGGLAAIZATION FRAME- WORK FOR MULTI-AGENT REINFORCEMENT LEARN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose an adaptive entropy-regularizate framework (ADER) for multi-agent reinforcement learning (RL) to learn the adequate amount of exploration for each agent based on the degree of required exploration. order to handle instability arising from updating multiple entropy temperature parameters for multiple agents, we disentangle the soft value function into two types: one for pure reward and the other for entropy. By applying multi-agent value factorization to the disentangled value function of pure reward, we obtain a relevant metric to assess the necessary degree of exploration for each agent. Based on this metric, we propose the ADER algorithm based on maximum entropy RL, which controls the necessary level of exploration across agents over time by learning the proper target entropy for each agent. Experimental results show that the proposed scheme significantly outperforms current saute-of-the-eee multi-agent RL algorithms.
INTRODUCTION
RL is one of the most notable approaches to solving crcissonnmakingg problems such as robot control (Hester et al.) 2012 Ebert et al. 2018), traffic light control (Wei et al. 2018: Wu et al.) 2020) and games (Mnih al. 2015 Silver al. 2017). The goal of RL is find optimal policy that maximizes expected return. To guarantee convergence of model-free RL., the assumption that each element in the joint state-action space should be visited infinitely often required (Sutton & Barto 2018), but this is impractical due to large state and/or action spaces in real-world problems. Thus, effective exploration has been core problem in RL. In practical real-world problems, however, the given time for learning is limited and thus the learner should exploit its own policy based on its experiences so far. Hence, the learner should balance exploration and exploitation in the dimension of time and this is called exploitation trade-off' becomes more challenging in multi-agent RL (MARL) because the state-action space grows exponentially the number of agents increases. Furthermore, the necessity and benefit of exploration can be different across agents and even one agent's 's exploration can hinder other agents' exploitation. Thus, the balance of exploration and exploitation across multiple agents should also be considered for MARI. in addition to that across the time dimension. We refer to this problem as multi-agent ppploration-exploitatioo trade-off. Although there exist many algorithms for better exploration in MARL (Mahajan et al.) 2019 Kim et al.) 2020 Liu et al. 202 Zhang et al. 2021). trade-off has not been investigated much yet.
trade-off in RL. The problem of exploration-
the research on multi-agent exploration-exploiaatee
In this paper, we propose a new framework based on entropy regularization for adaptive exploration trade-off. The proposed framework
in MARL handle the multi-agent pnlloration-epllottatiin allocates different target entropy across agents and across time based on our newly-proposed metric for the benefit of further exploration for each agent. To implement the proposed framework, we adopt the method of disentanglement between exploration and exploitation (Beyer et al.) 2019: Han & Sung. 202I) to decompose the joint soft value function into two types: one for the return and the other for the entropy sum. This disentanglement alleviates instability which can occur due to the updates of the temperature parameters. It also enables applying value factorization to return and entropy separately since the contribution to the reward can be different from that to the entropy from an agent's 's perspective. Based on this disentanglement, we propose a metric for the desired level of exploration for each agent, based the partial derivative of the joint value function of pure

---

Under review as a conference paper at ICLR 2023
return with respect to (W.Ft.) policy action entropy. The intuition behind this choice is clear for entropy-based exploration: Agents with higher gradient of joint pure-return value w.r.t. their action entropy should increase their target action entropy resulting in higher exploration level in order to contribute more to pure return. Under the constraint of total target entropy sum across all agents, which we will impose, the target entropy of agents with lower gradient of joint pure-return value w.r.t. their action entropy will then be reduced and inclined to exploitation rather than exploration. Thus, multi-agent xxporation-exploitaiie
trade-off can be achieved. The experiments demonstrate the effectiveness of the proposed framework for multi-agent
-ettiiot-expoiiationn trade-off.
2 BACKGROUND
Basic setup We consider decentralized partially observable MDP (Dec-POMDP), which describes a fully cooperative multi-agent task (Olichoek & Amato 2016). Dec-POMMP is defined by a tuple < NN, {A,;, P, {2,), o, 2, where N = -1,22 N is the set of agents. At time step 1, Agent i E N makes its own observation of E 22 according to the observation function (ss, i): 8 - Ω (st, + of, where E S is the global state at time step t. Agent selects action aj E A,, forming joint action a₂ = =aa az.
,.p J. The joint action yields the next global
state S according the transition probability P(-|st,as) az) and joint reward r(sr, ot). Each agent i has an observation-action to maximize the return E E >'r.- We consider the framework of centralized training with decentralized execution (CTDE), where decentralized policies are trained with additional information including the global state in a centralized way during the training phase (Oliehoek et al. 2008).
history T' E (S2, x A,)" and trains its decentralized policy π (a'(r")
Value Factorization It is difficult to learn the joint action-value function, which is defined as QTT(S, T, a) = E r'r, T, "r, a) due to the problem of the curse of dimensionality as the number of agents increases. For efficient learning of the joint action-value function, value fuctorization techniques have been proposed to factorize it into individual action-value functions Qi(T', .a),, 1, .N. One representative example is QMIX, which introduces monotonic constraint between the joint action-value function and the individual action-value function. The joint action-value function in QMIX is expressed as
8QrT(B,T,a)) (Q((ria') 20, ViE, (1)
QuT(S, fmir (a), (.T ()
where fiir is mixing network which combines the individual action-values into the joint action-
value based on the global state. To satisfy the monotonic constraint /0Q; 2 0, the mixing network is restricted to have positive weights. There exist other value-based MARL algorithms with
value factorization (Son et al. 2019 Wang et al. 2020a). Actor-critic based MARL algorithms also considered value factorization to learn the centralized critic (Peng et al., 2021
Su al. 2021).
Maximum Entropy RL and Entropy Regularization Maximum entropy RL. aims to promote exploration by finding an optimal policy that maximizes the sum of cumulative reward and entropy
(Haarnoja et al. 2017 2018a). The objective function of maximum entropy RL is given by
J MawEna (T) = Ex
( 1=0
(2)
where H(.) is the entropy function and is the temperature parameter which determines the importance of the entropy compared to the reward. Soft actor-critic (SAC) is an off-policy actor-critic algorithm which efficiently solves the maximum entropy RL problem (2) based on soft policy iteration, which consists of soft policy evaluation and soft policy improvement. For this, the soft Q function is defined the sum of the total reward and the future entropy, i.e.,
+ ΣN a-((T(-(s:))) In the soft policy evaluation step, for a fixed policy x, the soft Q function is estimated with convergence guarantee by repeat- edly applying the soft Bellman backup operator T to estimate function Q S x A + R. and the soft Bellman backup operator is given by Teac a.) = Ti + EE.,, [V(++1)), where V(st) Eo,~z (Q(sz, az)-e log π (a, ss,)]. In the soft policy improvement step, the policy is updated using the evaluated soft Q function as follows: Tnew arg max, En,o, 'QTaid (#1, as) a log v [s))]. By iterating the soft policy evaluation and soft policy improvement, called the soft policy iteration,
Q QT((s. : Er
2

---

Under review as a conference paper at ICLR 2023
SAC converges to optimal policy that maximizes 22) within the considered policy class in the case of finite MDPs. SAC also works effectively for large MDPs with function approximation.
One issue with SAC is the adjustment of the hyperparameter in ② which control the relative importance of the entropy with respect to the reward. The magnitude of the reward depends not only on tasks but also on the policy which improves over time during the training phase. Thus, (Haanojj et al. 2018b) proposed a method to adjust the temperature parameter a over time to guarantee the minimum average entropy at each time step. For this, they reformulated the maximum entropy RL as the following regulay-regularized optimization:
(ToTT) = Eorr Σ
s.t. B(ss,a))m. (Te 2 Ho
(3)
where Ho is the target entropy. Here, to optimize the objective 3). the technique of dynamic pro-
gramming is used, i.e., maXser EE>" max=, B[rr]] + max EI ri]] Starting
from time step T, we obtain the optimal policy Th.r and o applying the backward recursion. That is, we begin with the constrained optimization at time step T, given by
max B[rT] s.l. (or,,r))- [- (og(TT((arlsr))) 2 Ho
(4)
aT
and convert the problem into the Lagrangian dual problem follows:
min max E(rr- - Oer log TT (ar sr)) ]] Ho = min E| -or log (ar(()) - Ho).
(5)
Here, the optimal temperature parameter a at time step T, which corresponds to the Lagrangian multiplier, is obtained by solving the problem 5 Then, the backward recursion can be applied to obtain optimal a at time step /. hased on the Lagrange dual problem:
of
arg min Eu,~x; a- log nī (aelst) HoHo
(6)
where nे is the maximum entropy policy at time step l. Here, by minimizing the loss function J (a). a is updated to increase (or decrease) if the entropy of policy is lower (or higher) than the target entropy. In the mafnite-horizon case, the discount factor is included and z is replaced with the current approximate maximum entropy solution by SAC. In this way, the soft policy iteration of SAC is combined with the a adjustment based on the loss function J(a defined in (6) This algorithm effectively handles the reward magnitude change over time during training (Haarnoja et al.) (2018b).
Hence, one needs set only the target entropy Ho for each task and then a is automatically adjusted over time for the target entropy.
Related Works Here, we mainly focus on the entropy-based MARL. Other related works regarding multi-agent exploration are provided in Appendix E. There exist previous works on cutropy-based Whou al. (2020) proposed an actor-critic algorithm, named LICA, which learns implicit credit assignment and regularizes the action entropy by dynamically controlling the magnitude of the gradient regarding entropy to address the high sensitivity of the temperature parameter caused by the curvature of derivative of entropy. LICA allows multiple agents to perform consistent level of exploration. However, LICA does not maximize the cumulative sum of entropy but regularize the action entropy. Zhang et al. (202T) proposed an cntropy-rreularinn MARL algorithm, named FOP, which introduces a constraint that the mntrroy--reulaate into the product of the optimal individual policies. FOP introduced weight network to determine individual temperature parameters. /Zhang et al. (202T) considered individual temperature parameters for updating policy, but in practice, they used the same value (for all agents) which is annealed during training for the temperature parameters. This encourages multiple agents to focus on exploration the beginning of training, which considers exploration-exploitttio only in time dimension in a heuristic way.
optimal joint policy is decomposed
key point is that the aforementioned algorithms maximize or regularize the entropy of the policies to encourage the same level of exploration across the agents. Such exploration is still useful for several benchmark tasks but cannot handle the multi-agent exploration-exploiaaiii in the previous methods, the joint soft Q-function defined as the total sum of return and entropy is directly factorized by value decomposition, and hence the return is not separated from the entropy in the Qavuue.. From the perspective of one agent, however, the contribution to the reward and that to the entropy can be different. What we actually need to assess the goodness of policy is the return estimate, which is difficult to obtain by such unseparated factorization.
trade-off. Furthermore,
3

---

Under review as a conference paper at ICLR 2023
3 METHODOLOGY
In order to address the aforementioned problems, we propose an Aapptive Regulaazaaal framework (ADER), which can balance exploration and exploitation across multiple agents by learning the target entropy for each agent.
1.0
3.1 MOTIVATION
0.5
The convergence of model-free RL requires the assumption that all state-action pairs should be visited infinitely often, and this neces- sitates exploration (Sutton Barto, 2018). In practice, however, the number of time steps during which an agent can interact with the environment is limited. Thus, a balance between exploration and exploitation in the dimension of time is crucial for high perfor- mance in RL. Furthermore, in the case of MARL, a balance between exploration and exploitation in the dimension of agents should be considered. This is because 1) the degree of necessity and benefit of exploration can be different across multiple agents and 2) one agent's ss exploration can hinder other agents' exploitation, resulting in the situation that simultaneous exploration of multiple agents can make learning unstable. We refer to this problem as multi-agent multi-agent apploration-cxpliiaaiiii trade-off, we need to control the amount of exploration of each agent adaptively and learn this amount across agents (i.e., agent dimension) and over time (i.e., time dimension). In the case of entropy-basee exploration, we should allocate higher target entropy values to the agents who need more exploration or have larger benefit from exploration and allocate lower target entropy values to the agents who need more exploitation or have less benefit from exploration. In order to see the necessity of such adaptive ingloration-exploiaaae
0.0
0
-1
-1.00
-0.5
0.0
0.5
1.0
Figure 1: Reward surface in the considered matrix game. a1 and az correspond to x-axis and y-axis, respectively.
trade-off. To handle the problem of
trade-off control in MARL, let us consider a modified continuous cooperative matrix game (Peng et al. 202I). The considered game consists of two agents: each agent has nee-imensional continuous action a' which is bounded in 1]. The shared reward is determined by the joint action, and the reward surface is given in Fig. [ As seen in Fig. [ there is connected narrow path from the origin (0, 0) to (0.6, 0.55), consisting of two subpaths: one from 0) to (0.6, 0) and the other from (0.6, 0) to (0.6, 0.55). There is a circle with center at (0.6, 0.6) and radius 0.05. The reward gradually increases only along the path as the position approaches the center of the circle and the maximum reward is 5. There is penalty if the joint action yields the position outside the path or the circle, and the penalty value increases as the outside position is farther from the origin (0, 0). The agents start from the origin with initial action pair a = (0, 0) and want to learn to reach the circle along the path. Even if this game is stateless, exploration for action space is required to find the action (0.6, 0.6). One can think that one can find the optimal joint action once the action near the circle is selected. However, the action starting with (0,0) cannot jump to (0.6, 0.6) since we use function approximators for the policies and train them based on stochastic gradient descent. The action should be trained to reach the circle along the two subpaths. In the beginning, to go through the first subpath, a2 (i.e., y-axis movement) should not fluctuate from 0 and a should be trained to increase upto 0.6. In this phase, if a2 explores too much, the positive reward is rarely obtained. Then, ai is not trained to increase upto 0.6 because of the penalty. Once the joint action is trained to (0.6, 0). on the other hand, the necessity of exploration changed. In this phase, 01 should keep its action at 0.6, whereas de should be trained increase upto 0.55. As seen in this example, it is important to control the tradc-off between exploitation and exploration across multiple agents. In addition, we should update the trade-off over time because the required tra can change during the learning process. As we will see in Section 4 method that retains the same or different-but-constaan level of exploration across all agents fails to learn in this continuous cooperative matrix game. Thus, we need framework that can adaptively learn appropriate levels of exploration for all agents over time, considering the time-varying multi-agent exploration-explootttio trade-off.
3.2 ADAPTIVE ENTROPY- REGULARIZED MARL
We now propose our ADER framework enabling adaptive exploration capturing the multi-agent iitooration-eploittttt trade-off. One can adopt the entropy constrained objective defined in (3) and extend it to multi-agent systems. A simple extension is to maximize the team reward while keeping
4

---

Under review as a conference paper at ICLR 2023
the average entropy of each agent above the same target entropy. For the sake of convenience, call this scheme simple entropy- handle the multi-agent reapaarreuulliizatiinn for MARL -SER-MARL.. However, SER-MARL cannot -epppattonnxlloitatttinn trade-off because the amounts of exploration for all agents are the same. One can also consider different but fixed target entropies for multiple agents. However, this case cannot handle the time-varying behavior of multi-agent exploitation-expllratto trade-off, discussed in the previous subsection with Fig. [ exploration-exllotttion trade-off, we consider the following optimization problem:
Thus, to incorporate the multi-agent
20
N
max ExT π
Σ g'r
S.t. Ro [- [-log( 2 Hs, Vi E N and Σ
H, = Ho: (7)
where (IT1 N,, Hi is the target entropy of Agent i, and Ho is the total sum of all target entropies. The key point here is that we fix the target entropy sum as Ho but each Hi is adaptive and learned. The total entropy budget Ho is shared by all agents. When some agents' target entropy values are high for more exploration, the target entropy values of other agents should be low, leading to more exploitation, due the fixed total entropy budget. Thus, the poploittiionexploraaaeee across agents (i.e., agent dimension) can be captured. The main challenge is how to learn individual target entropy values H1, HN over time (i.e., time dimension) as the learning progresses.
trade-off
We postpone the presentation of our method of learning the individual target entropy values to Section 3.4 Here, we consider how to solve the problem 7 when Hi.... Hy are determined. In order to solve the problem 7 can simply extend the method in (Haarnoja et al., 2018b) to the MARL case. That is, one can first consider a imite-horizon case with terminal time step T, apply approximate dynamic programming and the Lagrange multiplier method, obtain the update formula time step t, and then relax to the infinite-horiza case by introducing the discount factor, as in (Haarnoja et al.) 2018b). For this, the joint soft Q-function Q (St; Te, a+) can be defined as
N
Q Te, at) == rt. + E
Σ 1=++1
Σ a H((()))))
(8)
and then this joint soft Q-function is estimated based on the following Bellman backup operator: T T=QUT(TT:. az) Ti + BBr..n [V(*++1, Te+1)], where V St, Tx)
ExinT Qur (5t (ge, Te, a1) E log (aa) r+) However, optimizing the objective based on the joint soft Q-function in g and the corresponding Bellman operator 7 has several limitations. First, the estimation of the joint soft Q-function can be unstable due to the changing {a in (8) as the determined target entropy values are updated over time. Second, we cannot apply value factorization to return and entropy separately because the joint soft Q-function defined (8) estimates only the sum of return and entropy. For single agent, the contribution to the global reward may be different from that to the total entropy. Thus, learning to decompose the entropy can prevent the mixing network from learning decompose the global reward. Furthermore, due to the inseparability of reward and entropy, it is difficult to pinpoint each agent's contribution sensitivity to the global reward itself, which is used for assessing the necessity and benefit of more exploration.
3.3 DISENTANGLED EXPLORATION AND EXPLOITATION
To address the aforementioned problems and facilitate the acquisition of metric for the degree of required exploration for each agent in MARL, we disentangle the return from the entropy by decomposing the joint soft Q-function into two types of Q-functions: One for reward and the other for entropy. That is, the joint soft Q-function is decomposed as QST(Te, a+) = QF,(Te, at) + EN' a'Q"lii (Te, at), where (T, az) and Q (Tt, 0z) are the joint action value function for reward and the joint action value function for the entropy of Agent i's policy, respectively, given by
( Te, at) = = ret ET
Σ
r
and
(9)
l
Qi Te, at) ETLLN
Σ rt ((
iEN.
(10)
1=
5

---

Under review as a conference paper at ICLR 2023
The action value functions (st, Tr. at) and Hid (st, Te, ae) can be estimated based on their corresponding Bellman backup operators, defined by
T Tr, ac) := r + 7B [V (s,, Te+1)l,
THe H (sc, Te, := VJ ' T++1)
(st.
(11)
where (st, Tt) = Ea. (s. Tr, at)] and VJr (St, TE) Ea. Qf (sc. Te,
at) aa log (aj]r])) are the joint value functions regarding reward and entropy, respectively.
Proposition! 1 The disentangled Bellman operators Ti and THis are contractions.
Proof: See Appendix A.
Now we apply value decomposition using mixing network (Rashid et al. 2018) to represent each of the disentangled joint action-value and value functions as a mixture of individual value functions. For instance, the joint value function for reward Vfr (s, 7) is decomposed as V (s, ( (s, ()*() (+>)). where V?R(7*) is the individual value function of Agent and f is the mixing network for the joint value function for reward. Similarly, we apply value decomposition
and mixing networks to ( az) and H.i (Te, az), i W.
Based on the disentangled joint soft Q-functions, the optimal policy and the temperature parameters can be obtained as functions of "i....., HNy by using similar technique to that in (Haarnoja et al.) 2018b) based on dynamic programming and Lagrange multiplier. That is, we first consider the finite-
horizon case and apply dynamic programming with backward recursion: max
E
n)))) Biggoe)om [- log (T(((())))) 2 Hi. Vl
InaX B[ri]
max
E
m1l.
(12)
k-2+1
We can obtain the optimal policy and the temperature parameters by recursively solving the dual problem from the last time step T by using the technique of Lagrange multiplier. At time step 4, the optimal policy is obtained for given temperature parameters, and the optimal temperature parameters computed based the obtained optimal policy follows:
N Hail (st, Te, a:) i
π.. = arg max E Bay m
(st, a1) at) +
Σ
a
at) - log n (a((l))))
(13)
1=1
(6)
0j" arg min E Bay
a log n. (atlr))
aHHl VieW.
(14)
af
In the infinite-horizon case, 13 and 14 provide the update formulae at time step t, and the optimal policy replaced with the current approximate multi-agent maximum-entrope entropy solution, which can be obtained by extending SAC to MARL. Note that maximizing the term (a) in (I3) corresponds to the ultimate goal of MARL,, i.e., the expected return. On the other hand, maximizing the term (b) in (13) corresponds to enhancing exploration of Agent i.
3.4 LEARNING INDIVIDUAL TARGET ENTROPY VALUES
In the formulation the amount of exploration for Agent is controlled by the target entropy H. N under the sum constraint Ej=1 H,, = No. In this subsection, we describe how to determine the target entropy for each agent over time. First, we represent the target entropy of Agent as H, = -3 x Ho to satisfy the entropy sum constraint. Then, we need to learn 8 over time t. Considering the fact that the ultimate goal is to maximize the return and this is captured by the value function of return V by disentanglement, we adopt the partial derivative NVBR//HH( t to assess the benefit of increasing the target entropy H. of Agent for more exploration at time Note that R /0H((() denotes the change in the joint pure-return value w.r.t. the differential increase in Agent i's policy action entropy. Suppose that aV / > DVR/ ( for two agents/ i and j. Then, we update two policies oे and के to two new policies so that the entropy of each of the two policies is increased by the same amount AH, then Agent contributes more the
N with 1i 3
time

---

Under review as a conference paper at ICLR 2023
pure return than Agent j. Then, under the total entropy sum constraint, the targe entropy of Agent i should be assigned higher than that of Agent j for higher return. Furthermore, when this quantity for certain agent is largely negative, increasing the target entropy for this agent can decrease the joint (return) value significantly, which implies that exploration of this agent can hinder other agents' allocate higher (or lower) target entropy agents whose 0v Jr. (m()( is larger (or smaller) than that of other agents. With the proposed metric, in the case of Ho 2 0, we set the coefficients B,, i 1, N for determining the individual target entropy values H., i 1, .N as follows: B 31
exploitation. Therefore,
Bi,
AN|
( 8, [++()
aV (3 (H(()((T))
av ( [ (1)))
Softmax
(15)
The relative required level of exploration across agents can change as the learning process and this is captured in these partial derivatives. We compute the partial derivative for 15 in continuous and discrete action cases as follows:
(e.T)
OVR JT (s, 0H (H(T((1T))
Gaussian policy for continuous action
(16)
( (r') 8H(#))
Categorical policy for discrete action
(F)
where o' is the standard deviation of Agent i's Gaussian policy. In the case of Gaussian policy for continuous action, the partial derivative @v /0H(((x')) is obtained as the partial derivative w.r.t. the log of the standard deviation of Gaussian policy based on the fact that the entropy of Gaussian random variable with variance of is log( V2Teo,). This can be done by adopting the reparameterization trick. On the other hand, it is difficult to directly obtain the partial derivative in the discrete- action case based on the categorical policy. For this, we use the chain rule to compute the partial derivative
OV (T) as shown in (16), where we numerically compute (H((T)) AH(=))) For numerical
(r") (v"((')
computation, we first update the policy in the direction of maximizing entropy and then compute the
changes of (T") and (T[)) to obtain the approximation. That is, the approximation is given by
(v." (r' (r by updating ां T" in direction of maximizing H( detailed
[H(()) explanation of computation of the metric is provided in Appendix B.1
H(x
During the training phase, we continuously compute 15 from the samples in the replay buffer and set the target entropy values. Instead of using the computed values directly, we apply exponential moving average (EMA) filtering for smoothing. The exponential moving average filter prevents the target entropy from changing abruptly. More concretely, if the partial derivative /V////H(( has a large variance over the samples the replay buffer, the computed metric can fluctuate whenever the transitions are sampled. This causes instability in learning. and thus the EMA filter can prevent the instability by smoothing the value. The output of EMA filter BEMA F3E
3 is
computed recursively as
BEMA + (1 (1--))3 3) + EB
(17)
where ,3 is given in 15 and & [0, 1]. Thus, the target entropy is given by H4 = BEMA x Ho.
Finally, the procedure of ADER is composed of the policy evaluation based on the Bellman operators and Proposition I, the policy update for policy and temperature parameters [3) and (14), and the
target entropy update in [55 and 17 The detailed implementation is provided in Appendix B.
4 EXPERIMENTS
In this section, we provide numerical results and ablation studies. We first present the result on the continuous matrix game described in Sec. 3.T and then results including sparse StarCraft II micromanagement (SMAC) tasks (Samvelyan et al. 2019).
Continuous Cooperative Matrix Game As mentioned in Sec 3.1 the goal of this environment is to learn two actions aj and d2 that the position (a1, starting from (0, 0) reach the target circle along a narrow path, as shown in Fig. [ The maximum reward is obtained if the position reaches the center of the circle. We compare ADER with four baselines. One is SER-MARL with the same target entropy for all agents. The second is SER-MARI with different-but botssontt target entropy values for two agents (SER-DCE). Here, set higher target entropy for an than d2. The
7

---

Under review as a conference paper at ICLR 2023
Phase
Phase
0.6
Target Entropy
Reversed ADER
0.3
-0.8
FOP
0.4
SER MARL
SFI-DGE
0.3
-1.0
-1.2
0.1
-1.4
0
10
20
Time Steps (1E4)
Time Steps (1E4)
(a) Averaged test return
(b) Target entropy
Figure 2: (a) The performance of ADER and the baselines on the considered matrix game (The performance of the baseline marked with yellow box is enlarged and displayed, and the black dotted line denotes the time when the position reaches the junction of the two subpaths) and (b) The learned target entropy values during the training.
third is Reversed ADER, which reversely uses the proposed metric // for the level of required exploration. The fourth is FOP, which an entropy-regulare MARL algorithm.
Fig. 2 a(a) shows the performance of ADER and the baselines averaged over random seeds. It is seen that the considered baselines fail learn to reach the target circle, whereas ADER successfully learns to reach the circle. Here, the sestrent-buu-consttae target entropy values of SER-DCE are fixed as (H1, H2) (-0.7, 1.3), which are the maximum entropy values in ADER. It is observed that SER-DCE performs slightly better than SER-MARI but cannot learn the task with time-varying ppploration-xxpliiatain
trade-off. Fig. (b)) shows the target entropy values Hi and H2 for 01 and 02 respectively, which are learned with the proposed metric during training, and shows how ADER learns to reach the target circle based on adaptive exploration. The black dotted lines in Figs. Maa and (b) denote the time when the position reaches the junction of the two subpaths. Before the dotted line (phase 1), ADER learns so that the target entropy of 01 increases whereas the target entropy of a2 decreases. So, Agent I and Agent 2 trained so to focus exploration and exploitation, respectively, After the black dotted line (phase 2), the learning behaviors of target entropy values of ( and 02 reversed that Agent now does exploitation and Agent does exploration. That is, the trade-off of exploitation and exploration is changed across the two agents. In the considered game, ADER successfully learns the time- vareing trade-off of multi-agent exploration-exploitaionn by learning appropriate target entropies for all agents.
multi-agent
Continuous Action Tasks We evalu- ated ADER on two complex continuous action tasks: multi-agent HalfCheetah (Peng al., (202T) and heterogeneous predator-prey (H-PP). The multi-agent HalfCheetah divides the body into dis- joint sub-graphs and each sub-graph cor- responds to agent. We used 6 H-HalfCheetah, which consists of six agents with one action dimension. Next, the H-PP consists of three agents, where the maximum speeds of an agent and other agents are different. In both envi- ronments, each agent has a different role to achieve the common goal and thus the multi-agent exploration-exxllitaaioo tradeoff should be considered. Here, we used two baselines: SER-MARL and FACMAC Peng et al. (202T). Fig. 3 showing the performances of ADER and the baselines averaged over random seeds, ADER outperforms the considered baselines.
3000
ADER RACMAC
600
Test Return
Test Return
400
1000
200
Time Steps
Time Steps
(a) HalfCheetah(6 1)
(b) H-PP
Figure 3: Comparison of ADER with SER-MARL and FACMAC multi-agent HalfCheetah and H-PP
We also evaluated ADER on the StarcraftII micromanagement benchmark (SMAC) environment (Samvelyan et al. 2019). To make the problem more difficult, we modified the SMAC environment to be sparse. The considered sparse reward setting consisted of dead reward and time- time-penalty reward. The dead reward was given only when an ally or an enemy died. Unlike the original reward in SMAC which gives the hit-po damage dealt reward, multiple agents did not
Starcraft II
8

---

Under review as a conference paper at ICLR 2023
Tes Win Rate Dk0
1.0
AZEX
1.0
0.
06
0.6
0.4
0.4
0.4
0.2
NAPPO
VOAC
0.
0.2
55
1.0
155
20
1.0
2.0
0.5
0.
0A
0.5 0.
Time Steps (M)
Time Sleps (M)
Time Steps IM:
Time Sueps (M)
(a) MMM2
(b) 8m vs 9m
(c) lc3s5z
(c) 3m
Figure 4: Average test win rate on the SMAC maps. More results are provided in Appendix D.
receive a reward for damaging the enemy immediately in our sparse reward setting. We compared
ADER with six offte----thearr baselines: DOP (Wang al., 2020b), FACMAC (Peng al. (02T). FOP (Zhang et al. (021), LICA (Zhou et al. 2020), QMIX (Rashid et al. 2018). VDAC(Su et al.
2021) and MAPPO (Yu et al.) 2021). For evaluation. we conducted experiments on the different SMAC maps with 5 different random seeds. Fig. 4 shows the performance of ADER and the considered seven baselines on the modified SMAC environment. It is seen that ADER significantly outperforms other baselines in terms of training speed and final performance. Especially in the hard tasks with imbalance between allies and enemies such as MMM2, and 8m vs 9m, is difficult to obtain reward due to the simultaneous exploration of multiple agents. Thus, consideration of multi-agent eetoorationeepllittiiiie effectively achieves this goal.
trade-off is required to solve the task, and seems that ADER
We additionally provide several experiments on the original SMAC tasks and Google Research Football (GRF) task in Appendix D.
Ablation Study We provide an anal- ysis of learning target entropy in the continuous cooperative matrix game. Through the analysis, we the changing target entropy affects
1.0
Test Win Rate (%)
ADER
how
0.8
ADERIW/O
ADERIÉ DJ ADERIW/O
0.6
0.6
the learning as seen in Fig. [ In
0.4
SER-MARL
0.4
addition, we conducted an ablation study on the key factors of ADER in
0.0
the SMAC environment. First, we compared ADER with SER-MARL. As in the continuous action tasks, Fig- shows that ADER outperforms SER-MARL. From the result, it is seen that consideration of the multi- agent expploratio-eexplotttaii trade-
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps (M)
Time Steps (M)
(a) MMM2
(b) 8m vs 9m
Figure Ablation study: Disentangled exploration (DE), EMA filter (5 0), SER-MARL (fixed target entropy) and
off yields better performance. Sec-
the monotonic constraint (MC)
ond, we compared ADER with and without the EMA filter. As seen in Fig. it seems that the EMA filter enhances the stability of ADER. Third, we conducted an experiment to access the effectiveness of disentangling exploration and exploitation. We implemented ADER based one critic which estimates the sum of return and entropy. As seen in Fig. 5 using two types of value functions yields better performance. Lastly, we compare ADER with and without the monotonic constraint to show the necessity of the monotonic constraint. It is seen that enforcing the constraint improves performance. We provided the training details for all considered environments and further ablation studies in Appendix C and D, respectively.
5 CONCLUSION
We have proposed the ADER framework for MARL to handle multi-agent exploration-exxloitatio trade-off. The proposed method based on entropy regularization with learning proper target entropy values across agents over time by using newly-proposed metric to measure the relative benefit of more exploration for each agent. Numerical results on various tasks including the sparse SMAC environment show that ADER can properly handle time-varying multi-agent exploration-exploitae trade-off effectively and outperforms other state-of-the-ant baselines. Furthermore. we expect the key ideas of ADER can be applied other exploration methods for MARL such intrinsic motivation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, el. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Lucas Beyer, Damien Vincent, Olivier Teboul, Sylvain Gelly, Matthieu Geist, and Olivier Pietquin. Mulex: Disentangling exploitation from exploration in deep rl. arXiv preprint arXiv: 1907. 000.08 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint 1910:7791.
2019.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00066. 2018
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, et al. Noisy networks for exploration. International Conference on Learning Representations, 2018.
Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal
value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 3930-3941. PMLR, 2021.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1eXivv/801.01290 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.059090 2018b.
Scungyul Han and Youngchul Sung. A max max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Todd Hester, Michael Quinlan, and Peter Stone. Rtmba: A real-time model-based reinforcement learning architecture for robot control. In 2012 IEEE International Conference on Robotics and Automation, pp. 85 90. IEEE, 2012.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. A maximum mutual information framework for multi-agent reinforcement learning. arXiv preprint 2Xiv:2006.02732. 2020.
Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for
multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 6826-68366 PMLR, 2021a.
Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 6826-6836. PMLR, 2021b.
10

---

Under review as a conference paper at ICLR 2023
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
Frans A Oliehoek and Christopher Amato. A concise introduction to POMMP Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353. 2008.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count- exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34, 2021.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.00 1902.04043, 2019.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 350(7676):354223 359, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Jianyu Su, Stephen Adams, and Peter A Beling. Value-decomposition multi-agent actor-critics. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11352 11360, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent geearnnnn.. In International Conference on Learning Representations, 2020a.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration.
In International Conference on Learning Representations, 2019.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-
agent decomposed policy gradients. In International Conference on Learning Representations, 2020b.
Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learning approach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2496-2505, 2018.
Tong Wu, Pan Zhou, Kai Liu, Yali Yuan, Xiumin Wang. Huawei Huang, and Dapeng Oliver Wu. Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Transactions Vehicular Technology, 69(8):8243 8256, 2020.
11

---

Under review as a conference paper at ICLR 2023
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 12491-12500. PMLR, 2021.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33:11853-111 11864, 2020.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX A: PROOFS
Proposition 2 The decomposed soft Bellman operators Tí and Ti, are contractions.
Proof:
The action value functions ( ax) and Q (Te, ax) can be estimated based on their corresponding Bellman backup operators, defined by
TEOTT Te. a= ri + 7B (eta))
where
(18)
(st Te) = E (fr Te, a.)
H. (st, Tz. a+) 7B [ (st+1, T++1]
where
(19)
VJ (st, Tp) Q thi (tt Te, at) a' log log
Here, VJ (st, Te) and VJl (s2, Te) are the joint value functions regarding reward and entropy, respectively.
First, let consider the decomposed Bellman operator regarding reward, Tg. For the sake of simplicity, we abbreviate (Q H.i V V as (QR, Q.,,, VR, vH.)). From (18 we have
az) r2 + 7 a041 (0((5) 1041+11611:011 Te41:0r11).
(20)
Then, we have
T TØl (1
= =1(r+ +y Σ
7(4H1174)0((H1 T411 Tz, az) g(+1)
T441
9ct1
- (r++7
Σ
7(0411774))(6))177441 Te, az) qiti)l
9641 T4-1 9111
=ih
Σ
7(4411744)06441774ll ai) (gh41 g Mo
9441 act1
lh
Σ
T(a44l7742)p(sL_ T44i/s, Te, at)llo14444 d
7141
act1
< 57llg/41 12
of
[ff( Te,
e.s.aa and oi [Q Tr, at) T A)"
since
E S, a
(i) x A)"
Σ 114
((0041l/4411p(se. T441 Te, abo s Thus, the operator T is --
4441
contraction.
Next, let us consider the decomposed Bellman operator regarding entropy, TH From 19 we have
Ti Te, at) 7B ". (et1,0441) a' log πt (a (
(21)
13

---

Under review as a conference paper at ICLR 2023
Then, have
(TH,
- Thi 12
= =1(
Σ
a(a441/741))(04 1841/80, Tr, ar) (de+1 - a² log (4(4((411))))))
1141
(7
x(0441/T41))(0eH) 1441 Te, a+) (gê+1 a' log n (12 *041,7+41
Σ
a:41
=hh
Σ
7(4441177+))1044 T441/s4. a,) ($b+1 12+1))
4441:744
1/
Σ
T(a+41774410(ee Tet1 le: Tr, at)llllll - 1l
4111-7011
$r114/41 1111l
2
for of
[Qf Te, at)
" a and of [o Tt, a+) T((AAA)
since
E 8.00 E (52 A)"
EΣ T441
T4il a:) I 1. Thus, the operator TH is a 7
9111
contraction.
14

---

Under review as a conference paper at ICLR 2023
B APPENDIX B: DETAILED IMPLEMENTATION
Here, we describe the implementation of ADER for discrete action tasks based on SAC-discrete (Christodoulou 2019). The learning process consists of the update of both temperature parameters and target entropies and the approximation of multi-agent maximum entropy solution, which consists of the update of the joint policy and the critics. do this, we first approximate the policies ( Ji= the joint action value functions Q/r.en and Q7T, IT. by using deep neural networks with parameters. {or J Oz and {0
First, the joint policy is updated based on Eq. (12) and the loss function given by
N
I(Ø)= = E ( (.
Σ
(log ty. (ailr)) - Qui Jr (8t, Tı, ac))
- -iT T1 (x)
(B.1)
where { is the parameter for the joint policy. Next, the joint action value functions trained based on the disentangled Bellman operators defined in Eq. (10) and the loss functions are given by
I(OOn) Be
( (8t, Te, Te, - (r+ /VJT, (8t+1, Tt+1)))
2
(B.2)
I thi (8 ae, -V - ST (8441;
I(OHH,) Be 2
(B.3)
where VR and V.l. are defined follows:
JT,0n
JT,
V JT (st, Tx.) = [ (st, Te, at)
(B.4)
HH JT.Bn. (st. TC) [ (st, Te, ar) - a" log ग (aflTT))
(B.5)
Note that 0R and 0H. are obtained based the EMA of the parameters of the joint action-value functions. Although the definitions the state value functions are given by (B.4) and (B.5), we do not this definition to compute the state value functions. This is because the marginalization over joint action becomes complex as the number of agents increases. For the practical computation of VR and V do the following for reduced complexity. We first marginalize the individual Q-function V
R of all agents to the mixing network (TN)). Here, mix is learned such that yVR follows the definition by the TD loss eq. (B.2) and the Bellman equation. In addition, we share the mixing network for Q for all i E A and inject the one-hot vector which denotes the agent index to handle the scalability.
based on individual action to get V.R. Then, we feed V/R.
Jmig to obtain the joint state value function as VR 7) = P. (s, (7t)
V.R
vVR
We update the temperature parameters based on Eq. (13) and the loss function is given by
L(a*) ET,,D, (1))) [-a' log n. (a:l)t)) - a a'Hi]. Vi N.
(B.6)
Finally, we update the target entropy of each agent. For Ho 2 we set the coefficients 8 for determining the individual target entropy H4 as B [3,,.... ,,,
aV (V, E
OVR (
ov (8 LOH( (1I-
Softmax
(B.7)
LOH(m) (-171))
where the computation of (s.r) [HG[(1))
is explained in Section B.1
15

---

Under review as a conference paper at ICLR 2023
Note that we change the sign of the elements in Eq. (B.7) if Hu 0 to satisfy the core idea of ADER, which assigns a high target entropy to the agent whose benefit to the joint value small.
In addition, before the softmax layer, normalize the elements in Eq. (B.7). Based the coefficients, the target entropy is given by H3 = 3.MA x Ho where 3.MA is computed recursively
BEMA t (1 3)BBMA 5B
(B.8)
OV (a.T) (H(
B.1 COMPUTATION OF THE METRIC
We adopted an actor-critic structure for our algorithm. Hence, for each agent we have a separate actor, ie., policy in both continuous- action and discrete discrete-action cases, as seen in Figures G Gand which show the overall structure for continuous-action and discrete-action cases, respectively. The computation of the partial derivative 8V (- in Eq. (B.7) depends on the overall structure, especially on the structure of the individual critic network.
First, consider the continuous-action case. In this case, used a Gaussian policy for each agent. Then, the policy neural network of Agent i with trainable parameter 6* takes trajectory T. as input and generates the mcan 2² and the log variance log of as output, as shown in Figure 6 Based on these outputs and the reparameterization trick, the action of Agent i generated as a' eexp(log a¹) ZZ, where Zt is
with zero mean and identity covariance matrix, i.e., Z N(0, 7). The action at and trajectory x¹ are applied as input to both return and entropy critic networks for Agent i, as seen in Figure 6 Now, focus on the return critic network of Agent i, which is relevant to the computation of our metric. The return critic of Agent i generates the local Q-value Qf((r) (²'r)"
All local Q-values ( .... QN (T a² from all agents are applied as input to the mixing network for global return value QTT. as seen in Figure Due to the connected tensor structure in Figure 6 at the time of learning, the gradient of Q4 with respect to log can be computed by deep learning libraries such as Pytorch. Note that log o¹ is simply a scaled version of the Gaussian policy entropy. So, can just obtain this value 0Q log o¹ from deep learning libraries. Furthermore, VJJ can be obtained by sampling multiple a'ss from the same policy xí. computing the corresponding multiple Q Q-values and taking the average over the multiple a' samples. However, we simplify this step and just use TQ log of as our estimate for the metric 8V (a.r) Indeed, many algorithms (H((
use single-sample average for obtaining expectations for algorithm simplicity.
Second, consider the discrete-action case. In this case, we again use an actor-critic structure for our algorithm. The structure of the critic network of Agent in the discrete-action case is different from that in the continuous action case. Whereas the critic network takes the trajectory r' and the action a' as input, and generates ( uses the DQN structure (Mnih et al. 2015), which takes the trajectory 7 as input and generates Q (7', ((4)) ) as output in the discrete-action case. In the discrete-action case,
(T'' a²) in the continuous-actio
case, the critic network typically
all Q (7*, f action over a finite action set A {aI,... , J. and the policy is described by a categorical distribution p² = pí. ... M over A for each state (or trajectory). Hence, our actor, i.e, policy n for Agent / is a deep neural network which takes the observation -' as input and generates probability vector p pi. Pla]] as output. Here, let us denote the policy deep neural network parameter by H¹ and denote the policy कं by Thes showing the current parameter explicitly. Then, using the output p' = [pi..... Pla of the policy network and the output (F)) of of the critic network, we compute the local return value
1A - - (((()))f())))))
(22)
Then, all local return values VR
V& are fed to the mixing network for global return
value V.R. seen in Figure
16

---

Under review as a conference paper at ICLR 2023
In this discrete-are action case, the policy entropy is given by H( (+|T)) = =-E pj pj. On the contrary to the continuous-action case which the policy entropy log of is an explicit node value in the overall structure and hence the output v gradient with respect to the node log of is directly available, in the discrete-actio action case there is no node corresponding to the value H(T (-17")
- -E Pj pj Hence, the gradient JH(
av is not readily available from the architecture. Note
that we only have nodes for pi: ... A|A| in the architecture, but the gradient of V with respect to
p} is not a Furthermore, it is not easy to compute
av from
av
j 1, [A| with Dp}
TH(
Pj = for general cardinality [A].
BV To circumvent this difficulty and compute the metric "M(m)))) we exploit the policy network parameter
0¹ and numerical computation. When the current policy network parameter is 0', we have the corresponding policy network output pi, .PLAl Then, consider the temporary scalar objective function H(Tg:)) for the policy network. We can compute the gradient of H(TTg)) with respect the
policy parameter 0* Let us denote this gradient by 0H(())))) which is the direction of 0' for maximum
80-
8*H(*).) where is policy entropy increase. Then, we update the policy parameter as 0 6 8 a0
a positive stepsize. Then, for the updated policy Th.. we compute the corresponding pi,". " P All
Using these updated probability values, compute the local value V. by using eq.
22 Using the
(v)
(r H( H(
values before and after the update, we compute AR(T))
av 1T av can be computed based on the chain rule. That is, we have Now, the metric (H(
87((
(v.7) x (r') Here, the first term R (e.7) is available from deep learning libraries since (V)"
(R(()))))
VJT and V are nodes of the learning architecture. The second term UH( can be approximated
(v in the above.
by AH(
(H((g))) Note that the policy update 6" = 0¹ + 80 for the actual learning update.
is only for computation of the metric. It is not done
B.2 OVERALL ARCHITECTURE AND ALGORITIMM PSEUDOCODE
We summarize the proposed algorithm in Algorithm and illustrate the overall architecture of the proposed ADER in Figures 6 and
Algorithm 1 ADpptive Entropy- Regulaaiangullizzation for multi-agent reinforcement learning (ADER)
Initialize parameters (d), ) N On. (0n.: | 0n. ( N
Generate trajectory 7 by interacting with the environment by using the joint policy π and store in the replay memory
for episode = 2, do
Generate a trajectory by using the joint policy æ and store r in the replay memory for each gradient step do
Sample a minibatch from D
Update (ox)) by minimizing the loss function Eq. (B.1)
Update OR. HHH,)) J.W. by minimizing the loss functions Eq. (B.2) and Eq. (B.3)
Update at by minimizing the loss function Eq. (B.6)
Update (H,,ee N. by computing Eq, (B.7) and Eq. (B.8)
Update 0R and 1 by EMA based on 0g and (0
end for
end for
17

---

Under review as a conference paper at ICLR 2023
U - V (s) (s,,,a)))
Ha Q
5t-
Mixing Network (reward)
Mixing Network (entropy)
(S 1)
Q
Qt Hi
Q
Critica
Critica
Critich 1
CriticH
N
Compute the metric
(v.7))
al = 1 to x NO, 1)
log (r*)
Reprrmmcciiaaion tick
log
log N n
Actor
Actor N
Figure 6: Overall architecture of ADER in continuous action cases
of Mixing Network (Q, reward)
V
OF
Mixing Network (V, reward)
Mixing Network (Q, entropy)
Mixing Network (V, entropy)
(s,.1)
(3...)
Critica
Critic N
Criticy
Criticy N
Compute
Actor
Actor
Figure 7: Overall architecture of ADER in discrete action cases
18

---

Under review as a conference paper at ICLR 2023
APPENDIX C: TRAINING DETAILS
We compute the joint value function as (s) - (s, this, as similar in ZZang et al. 2021). we first obtaïn the local value functions as (T" E. aa)] and then input the obtained local value functions into the mixing network. For discrete action environments, mixing network is trained to minimize the TD error of Q It works well as the reviewer in the experimental results. For continuous action environments, we use two mixing networks for VL. and Q which are trained separately in SAC (Haarnoja et al. 2018aj. In addition, need N mixing networks for Q To handle the scalability, we share the mixing network for Q for all i E N and inject the one-hot vector which denotes the agent index as QMIX shares the local Q-functions with one parameterized neural network.
((""")).....(((""))) To compute
share the mixing network for both V and Q and thus the
Predator1
Predator2
Landmarks
Predator3
Preys
(a) 6 1-HalfCheetah
(b) H-PP
Figure 8: Considered continuous action tasks
CL. Environment Details
Multi-agent HalfCheetah We considered the multi-agent HalfCheetah introduced in (Peng et al. 202T). As illustrated in Fig. (a), the multi-agent HalfCheetah divides the body into disjoint sub-graphs and each sub-graph corresponds to an agent. We used 6 x 1-HalfCheetah, which consists of six agents with one action dimension. We set the maximum graph distance 1, where & denotes the distance each agent can observe. We set the maximum episode length as Tmaz 1000.
Heterogeneous Predator-Prey (H-PP) We modified the continuous predator-prey environment considered in (Peng et al.) 2021 to be heterogeneous. As illustrated in Fig. (b), the considered heterogencous predator-prey consists of three predator agents, where the maximum speeds of an agent (v- (2/aaz 1.0) and other agents (v = 0.75) are different, three preys with the maximum speed (ur) (umaz = 1.25) is faster than all predators and the landmarks. The preys move away from the nearest predator implemented in (Peng et al.) (021)) and thus the predators should be trained to pick one prey and catch the prey together. Each agent observes the relative positions of the other predators and the landmarks within view range and the relative positions and velocities of the prey within view range. The reward +10 is given when one of the predators collides with the prey. We set the maximum episode length as T Tmaz= 50.
We evaluated ADER on the StarcraftII micromanagement benchmark (SMAC) environment (Samvelyan al. 2019). To make the problem more difficult, we modified the SMAC environment to be sparse. The considered sparse reward setting consists of death reward and time-penalty reward. The time cenee-pealty reward -0.1 and the death reward is given +10 and -1 when one enemy dies and one ally dies, respectively. Additionally, the dead reward is given +200 if all enemies die.
Starcraft II
C2. Training Details and Hyperparameters
We implemented ADER based on (Samvelyan et al. 2019 Peng et al. 2021 Zhang et al. 2021] and conducted the experiments server with Intel(R) Xeon(R) Gold 6240R CPU @ 2,40GHz and
19

---

Under review as a conference paper at ICLR 2023
Nvidia Titan xp GPUs. Each experiment took about 12 to 24 hours. We used the implementations of the considered baselines provided by the authors.
Multi-agent HalfCheetah In the multi-agent halfcheetah environment, the architecture of the policies and critics for ADER follows (Peng et al. 202T). We use an MLP with hidden layers which have 400 and 300 hidden units and ReLU activation functions. The final layer uses tanh activation function to bound the action as in (Haarnoja a et al. (018a). We also use the reparameterization trick for the policy as in (Haarnoja et al. 2018a). The replay buffer stores up to 10 transitions and 100 transitions are uniformly sampled for training. As in (Haarnoja et al. 2018b), we set the sum of target entropy as
Ho = N x (-dim(AA) = 6 x (-1) = -6,
where N is the number of agents. We set the hyperparameter for EMA filter as Ç = 0.9 and initialize the temperature parameters anntt e- for all E E.
Heterogeneous Predator-Prey In the heterogeneous predator-prey environment, the architecture of the policies and critics for ADER follows (Peng et al. 2021]. To parameterize the policy, we use a deep neural network which consists of a fully-connected layer, GRU and a fully-connected layer which have 64 dimensional hidden units. The final layer uses tanh activation function bound the action. Next, for the critic network, we use a MLP with 2 hidden layers which have 64 hidden units and ReLU activation function. The replay buffer stores up to 5000 episodes and 32 episodes are uniformly sampled for training. As in (Haarnoja et al. 2018b), we set the sum of target entropy as
Ho N x (-dim(A)) = 3 x (-2) = -6
We set the hyperparameter for EMA filter as ç = 0.9 and initialize the temperature parameters as anit e for all i E ...
Starcraft II For parameterization of the policy we use a deep neural network which consists of fully-connected layer, GRU and fully-connected layer which have 64 dimensional hidden units. For the critic networks we use MLP with 2 hidden layers which have 64 hidden units and ReLU activation function. The replay buffer stores up to 5000 episodes and 32 episodes sampled for training. For the considered maps in SMAC, we use different hyperparameters. We set the sum of target entropy based on the maximum entropy, which be achieved if the policy is uniform distribution, as
uniformly
Ho = N X H X Kyatio = N x log(dim(A)) x kratio-
The values of kration 5, and initial temperature parameter for each map are summarized Table
Table Hyperparameters for the considered SMAC environment
MAP
kyatio
Ic.355z
0.05
0.9
e 3
3m
0.1
0.9
e
3s5z
0.05
0.9
e
3s 3z
0.1
0.9
e
MMM2
0.1
0.9
2.6
8m 9m
0.1
0.9
e
In all the considered environments, we apply the value factorization technique proposed in (Rashid) al. 2018). The architecture of the mixing network for ADER, which follows (Rashid et al. 20118). takes the output of individual critics as input and outputs the joint action value function. The weights of the mixing network are produced by the hypernetwork which takes the global state as input. The hypernetwork consists of a MLP with single hidden layer and an ELU activation function. Due to the ELU activation function, the weights of the mixing network are non-negative and this achieves the monotonic constraint in (Rashid et al. 2018). We expect that ADER can use other value factorization technique to yield better performance.
20

---

Under review as a conference paper at ICLR 2023
APPENDIX D: FURTHER EXPERIMENTS
Experiments on the original SMAC environments
We here provide the experiments the original SMAC environments. We compared ADER with three baselines including FACMAC (Peng et al. 202T), FOP (Zhang et al. 2021) and QMIX (Rashid al. 2018). For all the considered maps, ADER outperforms the baselines, shown in Fig. Thus, the proposed adaptive entropy reguopy-rgullarization method performs well in both original and sparse SMAC environments.
1.0
Test Win (%6
0.8
FACMAC
0.8
0.6
QMIX
0.6
0.4
0.2
0.2
0.0
1.5
0.0
0.5
1.0
1.5
2.0
Time Steps (M)
Time Steps (M)
Time Steps (M)
(a) MMM2
(b) 8m vs 9m
(c) 3855z
Figure Average test win rate on the original SMAC maps.
Experiments on google research football environment
We evaluated ADER on the google research football (GRF) environment, which is known hard exploration tasks. We consider one scenario in GRF named Academy vs with keeper. In this environment, the agents receive reward only when they succeed in scoring, which requires hard exploration. Thus, is difficult to obtain the reward if all agents focus on exploration simultaneously.
We compared ADER with four baselines: QMIX, FOP, FACMAC, and SER-MARL. Fig. 10 shows the performance of ADER and the baselines, and the y-axis in Fig. B.2 denotes the median winning rate over 7 random seed. seen Fig. 10 that ADER outperforms the baselines significantly. Since ADER handles multi-agent exploration-explotaaiee time, ADER performs better than SER-MARL, which keeps the same level of exploration across agents.
trade-off across multiple agents and over
Experiments on the modified SMAC environments
Fig. 11 shows the performance of ADER and the considered seven baselines on the modified SMAC environment. It is seen that ADER outperforms all the considered basclines. Especially, on the hard tasks shown in Fig. 11, ADER significantly outperforms other baselines in terms of training speed and final performance. This is because those hard maps require high-qualing adaptive exploration across agents over time. In the maps 35 3z, the stalkers (ally) should attack zealot (enemy) many times and thus the considered reward is rarely obtained. In addition, since the stalker is ranged attacker whereas the zealot is a melee attacker, the stalker should trained to attack the zealot at a distance while avoiding the zealot. For this reason, if all stalkers focus exploration simultancously, they hardly remove the zealot, which leads to failure in solving the task. Similarly, in the hard tasks with imbalance between allies and enemies such MMM2, and 8m 9m, it is difficult to obtain reward due to the simultaneous exploration of multiple agents. Thus, consideration of multi-agent exploration-exploitaaee trade-off is required to solve the task, and it seems that ADER effectively achieves this goal.
21

---

Under review as a conference paper at ICLR 2023
Academy with Keeper
QMIX
ADER
FOP
FACMAC
SER SER-MARL
Test Win Rate (%)
0.6
0.4
0.2
1M
2M
3M
4M
Time Steps
(a) Academy 3 vs with keeper
Figure 10: Median test winning rate on Academy 3 vs with keeper
1.0
Test Win Rate (96)
0.8
0.8
DOP
0.6-
0.6
FOP
0.6
QMIX
VOAC
02
0.2
MAPPO
00
0.0
0.0
1.0
1.5
0.0
0.2
0.4
0.6
0.8
0.0
0.5
1.5
2.0
Time Steps
Time Steps (M)
Time Steps (M)
(a) vs 3z
(b) Ic355z
(c) MMM2
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.0
0.5
1.0
1.5
2.0
2.0
Time Steps (M)
Time Steps (M)
Time Steps (M)
(d) 8m 9m
(e) 3m
(f) 3x5z
Figure 11: Average test win rate on the sparse SMAC maps.
APPENDIX E: FURTHER RELATED WORKS
For effective exploration in single-agent RL, several approaches such as maximum entropy/entropy regularization (Haarnoja et 2017 2018a), intrinsic motivation (Chentanez et al.) 2004 Badia et al, 2019, Burda et al. 20188) parameter noise (Plappert et al. count-based exploration (Ostrovski et al. 2017 Bellemare et al. 2016) have been considered. Also in MARL, exploration has been actively studied in various ways. MAVEN introduced latent variable and maximized the mutual information between the latent variable and the trajectories to solve the poor exploration of QMIX caused by the representational constraint (Mahajan et al.) 2019). Wang et al. (2019) proposes a coordinated exploration strategy by considering the interaction between agents. Liu al. (202Tb) proposes efficient coordinated exploration method based
et al. 2018) and
22

---

Under review as a conference paper at ICLR 2023
restricted space selection encourage multiple agents to explore worthy state space. Zheng ct al. (2021) extends the intrinsic motivation- boadea exploration method to MARL and utilizes the episodic memory which stores highly rewarded episodes to boost learning. Gupta et al. (2021) promotes joint exploration by learning different tasks simultaneously based on multi-agent universal successor features to address the problem of relative overgeneralizatton. The aforementioned methods successfully improve exploration in MARL. However, to the best of our knowledge, none of the works address the multi-agent exploration-eploottaton tradeoff, which is the main motivation of this paper.
23

---

Under review as a conference paper at ICLR 2023
APPENDIX F: LIMITATION
In this paper, we only considered a fully cooperative setting where multiple agents share the global reward and showed that the proposed method successfully addresses the multi-agent exploration- exploitation tradeoff in such setting, However, the metric to measure the benefit of exploration can differ in other MARL settings such mixed cooperative-competitve settings. Thus, we believe finding the metric in other MARL settings can be good research direction.
24


Reviews:
[{"review_id": "f8lmUCwMFi", "reviewer": "Reviewer_mfnN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an MARL algorithm to adpatively handle the entropy regularization in multi-agent RL. In this method, the level of exploration of each agent is controlled by its time-varying target entropy, which severs as a constraint in the optimization problem. To determine a proper target entropy for each agent, the authors propose a partial derivative-based metric to evaluate the benefit of exploration and then use this term to decide the target entropy for each agent. Empirically, in a customized SMAC environment, their method is demonstrated to be better than other value-decomposition methods.\n", "summary_of_the_review": "This paper studies adaptive entropy regularization in MARL, and provides a naive yet reasonable solution for this problem. However, the presentation of this paper can be substantially improved, and some technical details about the computation of partial derivatives have to be further explained.  In short, I will not recommend acceptance of this paper, but I will be happy to reconsider it if the authors can address all those problems.\n", "strength_and_weaknesses": "### Strengths\n\n1. This paper studies adaptive entropy regularization, which is important in entroy-regularized MARL and well-demonstrated in the motivation section.\n\n2. The authors provide a comprehensive literature review of entropy-based MARL algorithms.\n\n3. A variety of baselines is included in their SMAC experiments. \n\n### Weaknesses\n\n1. **Standard SMAC maps and more baselines should be included.** In the experiment section, MAPPO should also be included as a baseline, as it is one of the SOTA MARL algorithms. Also, experiments on the *standard* (unmodified) SMAC environment should be included, as it is a standard of all MARL papers. Moreover, for the matrix game, FOP should also be included as a baselines. As far as I understand, FOP uses a weight network to determine individual temperature parameters in their tasks except SMAC. That said, adaptive entropy regularization over agents and time is indeed considered in FOP. \n\n2. **The writing can be largely improved.** Some paragraphs are too long, e.g., Section 1 is one paragraph, and Section 3.1 is one paragraph. Some references to equations are misused in section 3.4. The authors should refer to where those equations appear in the main text, not where they are in the appendix. For example, eq B.7 should be eq 15 in the main text.\n\n4. **The presentation of the proposed method can be improved.** The overall workflow of this algorithm is complicated and unclear. An algorithm should be included to make it clear. Figure 6 does not provide a clear illustration. \n\n### Questions\n\n1. Two mixer networks are used in this method. The update of Q-mixer and $Q_i$ is common and easy to understand. However, It seems two methods are mentioned for computing $V_{jt}$. One is using V-mixer $V_{jt} = f_{mix}(s, V_1, \\cdots V_n)$ and another is using expectation $V_{jt} = E[Q_{jt}]$. Do these two methods result in the same value? Or do you use those two values in different places?\n\n2. As stated in eq 13, the policy update is achieved in a joint form using joint value function $Q_{jt}$, so the $Q_i$ will only be used for the computation of target entropy? If so, why do we need a mixer network for $Q_{jt}$, as we can directly parameterize $Q_{jt}$ via a single neural network? The learning of the proposed method is unclear in many ways, though I may misunderstand some parts. \n\n3. As mentioned in Section 3.4, the partial derivative $\\frac{\\partial V_{jt}^{R}}{\\partial H(\\pi^i)}$ is used to determine the target entropy of agent $i$. For the categorical policy of discrete action space, as stated in eq 16, $\\frac{\\partial V_{jt}^{R}}{\\partial H(\\pi^i)} = \\frac{\\partial V_{jt}^{R}}{\\partial V_i^R} \\times \\frac{\\partial V_i^R}{\\partial H(\\pi^i)}$. The second term is approximated by $\\frac{\\Delta V_i^R(\\tau^i)}{\\Delta \\mathcal{H}(\\pi^i_t)}$. However, it is still unclear to me how it is computed numerically. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OP22Ykrg2_", "reviewer": "Reviewer_dyHK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an adaptive entropy-regularization framework (ADER) to address the multi-agent exploration-exploitation trade-off problem in MARL. The key insight of ADER is that the amount of exploration every agent needs to perform is different and can change over time, so we need to adaptively control the amount of exploration each agent conducts and learn this amount across agents and over time. To achieve this, ADER adaptively learn an individual target entropy for each agent over time (to control the amount of exploration for each agent), assuming a fixed total entropy budget. And it uses the change in the joint pure-return value w.r.t. the change in agent $i$'s policy entropy to estimate the benefits of increasing the target entropy of agent $i$.  In addition, in ADER, the exploration and exploitation is disentangled by disentangling the return from the entropy (i.e., the joint soft Q-function is decomposed into one joint Q-function for reward and one joint Q-function for the entropy). ", "summary_of_the_review": "I like the main idea of ADER. It seems to provide a simple yet effective way to better coordinate the level of exploration across different agents over time in cooperative tasks. I think the authors do a very good job at explaining the main intuitions of the proposed method. For instance, the motivation example is quite nice. It clearly explains how one agent's exploration can change over time and how it can hinder other agents' exploitation, thus showing the necessity of adaptively controlling the trade-off between exploitation and exploration across multiple agents. \n\nMy only (small) concern of the paper is most of the cooperative tasks tested (e.g., predator-prey and SMAC) in the experiments were different from the original settings. They were specifically designed to be difficult for the agents to gain positive rewards under simultaneous exploration (or similar levels of exploration). While I understand this might be because exploration is not a big problem in these original tasks, it seems that the authors could have evaluated their method on MARL benchmark like Google Research Football (with the original setting), which is known to have hard exploration issues. Also, it'd be nice to know if ADER could perform well in the original SMAC tasks, such that we could know if continuously learning the target entropy for each agent could hurt performance in cooperative tasks that do not require adaptive exploration across agents. \n\nSome minor comments/suggestions:\n- I think the introduction could be improved some. For example, the introduction says \"This disentanglement alleviates instability which can occur due to the updates of the temperature parameters and enables applying value factorization to return and entropy separately.\" It suddenly mentions value function factorization, but does not explain the motivation/necessity for using it. \n- The related work section mentions that LICA does not maximize the cumulative sum of entropy but regularize the action entropy. Why maximizing the cumulative sum of entropy can be better than regularizing the action entropy? This could be explained some. \n- The ablation study section mentions \"we compare ADER with and without the monotonic constraint to show the necessity of the monotonic constraint.\" For ADER without the monotonic constraint, how does it learn the joint $Q$ or $V$? Does it learn a centralized critic that directly conditions on the global state and joint action of agents? Also, I do not understand why it is necessary to use a *monotonic* value factorization. ADER learns both actors and critics and only the actors are needed during execution. So one can also use a *nonmonotonic* value function factorization. ", "strength_and_weaknesses": "- Strengths:\n\t- The proposed framework is well motivated and seems to work well in a variety of complex cooperative tasks that require adaptive exploration across agents over time.\n\t- The main idea of adaptively learning an individual target entropy for each agent over time to better balance the exploration and exploitation in the dimension of agents is very interesting and seems novel to me. \n\t- The paper is well-written and easy to follow overall. \n- Weaknesses:\n\t- It is not clear how the proposed method performs in cooperative tasks that do not require adaptive exploration across agents over time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3EILcfDnSF", "reviewer": "Reviewer_5qy9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve the exploration-exploitation tradeoff problem in the context of multi-agent reinforcement learning. While there have been a myriad of works on exploration-exploitation tradeoff on single age reinforcement learning, there are not many on the multi agent RL. This work proposes an adaptive entropy-regularization framework that learns adequate amount of exploration for each agent. To this end, this work proposes to decompose the joint soft value function into pure return and entropy sum. This disentanglements enable a more stable while updating the temperature parameters. This work focuses entropy-based MARL. ", "summary_of_the_review": "I would give marginally above the acceptance threshold. It would be good if the authors could answer my questions. There might be some issues that I didn’t catch, and if other reviewers have raised issues, I’m happy to discuss.  ", "strength_and_weaknesses": "- The strength of the paper comes from the idea that, while previous works encourage same level of exploration across agents, this work proposes to differentiate the level of exploration across agents in multi-agent RL setting. \n- Another strength comes from the core idea of this work: joint soft value function decomposition / separated factorization. \n- The motivation part 3.1 sounds convincing to me; one agent’s exploration can hinder other agent’s exploitation, resulting that simultaneous exploration of multiple agents can make learning unstable. Need a framework that can adaptively learn proper levels of exploration for each agent. \n- Experiments are well done, not extensive though. \n\nQuestions. \n\nQ1. Question about ADER performance shown in Figure 2a. It seems that ADER outperforms other methods like SER-DCE, SER-MARL, but there is a point where ADER’s performance suddenly jumps up in the middle. Is there any explanation on why this happens? \n\nQ2. In Appendix B, could you give me more justification on setting the coefficient beta_i? Especially, line B.7, beta_i are defined as softmax of expectation of \\partial V^R_{JT}(s,\\tau) / \\partial H (pi (|))) ? Could you give us more detailed explanations on it? And can you explain why it is difficult to directly obtain the partial derivative in discrete-action case, and using chain rule is justified? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject