Adaptive Learning Rates for Multi-Agent Reinforcement Learning
Authors: Jiechuan Jiang
Zongqing Lu
Venue:  (2021)

Abstract: In multi-agent reinforcement learning (MARL), the learning rates of actors and critic are mostly hand-tuned and fixed. This not only requires heavy tuning but more importantly limits the learning. With adaptive learning rates according to gradient patterns, some optimizers have been proposed for general optimizations, which however do not take into consideration the characteristics of MARL. In this paper, we propose AdaMa to bring adaptive learning rates to cooperative MARL. AdaMa evaluates the contribution of actors' updates to the improvement of Q-value and adaptively updates the learning rates of actors to the direction of maximally improving the Q-value. AdaMa could also dynamically balance the learning rates between the critic and actors according to their varying effects on the learning. Moreover, AdaMa can incorporate the second-order approximation to capture the contribution of pairwise actors' updates and thus more accurately updates the learning rates of actors. Empirically, we show that AdaMa could accelerate the learning and improve the performance in a variety of multi-agent scenarios, and the visualizations of learning rates during training clearly explain how and why AdaMa works.

Under review as a conference paper at ICLR 2021
ADAPTIVE LEARNING RATES FOR MULTI-AGENT RE- INFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-agen reinforcement learning (MARL), the learning rates of actors and critic are mostly hand-tuned and fixed. This not only requires heavy tuning but more importantly limits the learning. With adaptive learning rates according gradient patterns, some optimizers have been proposed for general optimizations, which however do not take into consideration the characteristics of MARL. In this paper, we propose AdaMa to bring adaptive learning rates to cooperative MARL. AdaMa evaluates the contribution of actors' updates to the improvement of Q- value and adaptively updates the learning rates of actors to the direction of maxi- mally improving the Q-value. AdaMa could also dynamically balance the learning rates between the critic and actors according to their varying effects on the learn- ing. Moreover, AdaMa can incorporate the second-order approximation capture the contribution of pairwise actors' updates and thus more accurately updates the learning rates of actors. Empirically, we show that AdaMa could accelerate the learning and improve the performance in variety of multi-agent scenarios, and the visualizations of learning rates during training clearly explain how and why AdaMa works.
INTRODUCTION
Recently, multi-agent reinforremnt learning (MARL) has been applied to decentralized cooperative systems, e.g., autonomous driving (Shalev-Shwartz et al., 2016), smart grid control (Yang et al., 2018), and traffic signal control (Wei et al., 2019). Many MARL methods (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019; Son al., 2019) have been proposed for multi-agent cooperation, which follow the paradigm of centralized training and decentralized execution. many of these methods, a centralized critic learns the joint Q-function using the information of all agents, and the decentralized actors updated towards maximizing the Q-value based on local observation.
However, in these methods, the actors are usually assigned the same learning rates, which is not optimal for maximizing the Q-value. This is because some agents might be more critical than others to improving the Q-value and thus should have higher learning rates. On the other hand, the learning rates of actors and critic are often hand-tuned and fixed, and hence require heavy tuning. More importantly, over the course of training, the effect of actors and critic on the learning varies, so the fixed learning rates will not always be the best at every learning stage. The artificial schedules, e.g., time-based decay and step decay, are pre-defined and require expert knowledge about model and problem. Some optimizers, e.g., AdaGrad (Duchi et al., 2011), could adjust the learning rate adaptively, but they are proposed for general optimization problems, not specialized for MARL.
In this paper, we propose AdaMa for adaptive learning rates in cooperative MARL. AdaMa dy- namically evaluates the contribution of actors and critic to the optimization and adaptively updates the learning rates based on their quantitative contributions. First, we examine the gain of Q-value contributed by the update of each actor. We derive the direction along which the Q-value improves the most. Thus, we can update the vector of learning rates of all actors towards the direction of maximizing the Q-value, which leads to diverse learning rates that explicitly captures the contribu- tions of actors. Second, we consider the critic and actors updated simultancously. If the critic's 's update causes a large change of Qavalu,, we should give a high learning rate to the critic since is leading the learning. However, the optimization of actors, which relies the critic, would strug-

---

Under review as a conference paper at ICLR 2021
gle with the fast-moving target. Thus, the learning rates of actors should be reduced accordingly. On the other hand, if the critic has reached a plateau, increasing the learning rates of actors could quickly improve the actors, which further generates new experiences to boost the critic's 's learning. These two processes alternate during training, promoting the overall learning. Further, by incor- porating the second-order approximation, we additionally capture the pairwise interaction between actors' updates so as to more accurately update the learning rates of actors towards maximizing the improvement of Q-value.
We evaluate AdaMa in four typical multi-agent cooperation scenarios, i.e. going logether, coop- erative navigation, predator-prey and clustering. Empirical results demonstrate that dynamically regulating the learning rates of actors and critic according to the contributions to the change of Q- value could accelerate the learning and improve the performance, which can be further enhanced by additionally considering the effect of pairwise actors' updates. The visualizations of learning rates during training clearly explain how and why AdaMa works.
RELATED WORK
MARL. We consider the formulation of decentralized partially observable Markov decision process (Dec-POMDP). There are N agents interacting with the environment. At each timestep t, each agent i receives a local observation o', takes action of, and gets shared reward r₁- The agents aim to maximize the expected return E Ei= r',,, where Y is a discount factor and T the episode time horizon. Many methods (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019; Son et al., 2019) have been proposed for Dec-POMDP, which adopt centralized learning and decentralized execution (CTDE). In many of these methods, centralized critic learns joint Q-function by minimizing the TD-error. In training, the critic is allowed to use the information of all agents. The actors, which only have access to local information, learn maximize the Q-value learned by the critic. In execution, the critic is abandoned and the actors act in a decentralized manner.
Adaptive Learning Rate. Learning rate schedules aim to reduce the learning rate during training according to a pre-defined schedule, including time-based decay, step decay, and exponential decay. The schedules have to be defined in advance and depend heavily on the type of model and problem, which requires much expert knowledge. Some optimizers, such AdaGrad (Duchi et al., 201 I), AdaDelta (Zeiler, 2012), RMSprop (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2015), provide adaptive learning rate to ease manual tuning. AdaGrad performs larger updates for more sparse parameters and smaller updates for less sparse parameters, and other methods are derived from AdaGrad. However, these methods only deal with the gradient pattern for general optimization problems, offering specialized way to boost multi-agent learning. WoLF (Bowling & Veloso, 2002) provides variable learning rates for stochastic games, but not for cooperation.
Meta Gradients for Hyperparameters. Some meta-learning methods employ hyperparameter gra- dients to tune the hyperparameter automatically. Maclaurin et al. (2015) utilized the reverse-mos mode differentiation of hyperparameters to optimize step sizes, momentum schedules, weight initializa- tion distributions, parameterized regularization schemes, and neural network architectures. Xu et al. (2018) computed the meta-gradient update the discount factor and bootstrapping parameter in reinforcement learning. OL-AUX (Lin et al., 2019) uses the meta-gradient to automate the weights of auxiliary tasks. The proposed AdaMa can also be viewed meta-gradient method for adaptive learning rates in MARL.
3 METHOD
In this section, we first introduce the single-critic version of MADDPG (Lowe al., 2017), on which we instantiate AdaMa. However, AdaMa can also be instantiated on other MARL methods, and the instantiation on MAAC (Iqbal & Sha, 2019) for discrete action space is also given in Appendix A. Then, we use the Taylor approximation to evaluate the contributions of the critic and actors' updates to the change of Q-value. Based on the derived quantitative contributions, we dynamically adjust the direction of the vector of actors' learning rates and balance the learning rates between the critic and actors. Further, we incorporate higher-order approximation to estimate the contributions more accurately.
2

---

Under review as a conference paper at ICLR 2021
3.1 SINGLE-CRITIC MADDPG
In mixed cooperation and competition, each MADDPG agent learns actor иi and critic for the local reward. However, since the agents share the reward in Dec-POMDP, we only maintain single shared critic, which takes the observation vector o and the action vector ā and outputs the Q-value, as illustrated in Figure 1. The critic parameterized by ø trained by minimizing the TD-error 5
TD error
@ a)
Critic
0
aQ(G.)) da
aN
da:
B(Ga) [Q a) - 1 where y - r+ (d, (d)).
Actor
Actor
Q is the target critic, "i is the target actor, and DD is replay buffer. Each actor ग; (parameterized by 0, is updated to max- imize the learned Q-value by gradient ascent. The gradient of 0g is
ON
0Q σ) da.
Figure 1: Single-Critic MADDPG
0a: 00;
We denote the learning rates of each actor i and the critic as la and le respectively.
3.2 ADAPTIVE I DIRECTION
First, suppose that the critic is trained and frozen, and only update the actors. By expanding the Q-function, we can estimate the gain of Q-value contributed by actors' updates by the Taylor approximation:
AQ - -00, Ad) - Q
~ Q(ō,a)= Aa; aQ(ā,a)
Oa,
Σ (0, + la. aQ(ō,a)) (0.)) aQ(ō,a) T
00;
0a:
N
8Q a) da.
8Q( T
00,
00,
Oa;
0Q(6,0 σ) oQ(ā,a) a)
00, 00,
aQ aQ T
1 00 08
Assuming the magnitude of the learning rate vector ||[|||| is a fixed small constant || the largest AQ is obtained when the direction of [, is consistent with the direction of vector aq aq T can softly update 1,, to the direction of aq aQ T to improve the Q-value:
00 Thus, we
aQ aQ T
aQ aQ T
[ = al + (1 - [T
/1l
II
00 a0
00
00
[[
T, 1
where the second line normalizes the magnitude of T, to ||Ta|] and ar is parameter that controls the
soft update. From another perspective, the update rule (1) can be seen updating ,, by gradient ascent to increase the Q-value the most, since
8AQ 8Q aq T
81_
09 89
3

---

Under review as a conference paper at ICLR 2021
3.3 ADAPTIVE AND ||[|||
In the previous section, we assume that the critic is frozen. However, in MADDPG and other MARL methods, the critic and actors are trained simultaneously. Therefore, we investigate the change of Q-value by additionally considering the critic's update:
AQ Q(o AØ, o, + Ad) Q(6,ō,ā)
Q + Σ Da ā))
N
aQ(6,ō,a) T
+ Ao
Q(6,ō,a)
da;
d6
8Q aQ
00 00 le 05 aQ
We can that AQ is contributed by the updates of both the critic and actors. In principle, the critic's learning is prioritized since the actor's s learning is determined by the improved critic. When the critic's update causes a large change of the Q-value, the critic is leading the learning, and should assign it a high learning rate. However, the optimization of actors, which relies on the cur- rent critic, would struggle with the fast-moving target. Therefore, the actors' learning rates should be reduced. On the other hand, when the critic has reached a plateau, increasing the actors' learning rates could quickly optimize the actors, which further injects new experiences into the replay buffer to boost the critic's learning. thus promoting the overall learning. The contributions of actors' up- dates are always nonnegative, but the critic's update might either increase or decrease the Qavalu. Therefore we use the absolute value a5 aQ to evaluate the contribution of critic to the change of Q-value. Based on the principles above, adaptively adjust Le and || by the update rules:
06 aQ²
Le ale + (1 - aa)l -a)aallliii ccip(p 0s 0ø 1/m,e, 1-c)
(2)
|| 1-
The hyperparameters m, I, and have intuitive interpretations and are easy to tune. controls the soft update and m controls the target value of le. The clip function and the small constant prevent the learning rate being too large too small. Therefore, AdaMa works follows: first update L.
and get || using (2), then regulate the direction and magnitude of [, according to (1).
As Liessner et al. (2019) pointed out, the actor should have a lower learning rate than the critic, and a high learning rate of actor leads performance breakdown. Also, empirically, in DDPG (Lillicrap et al., 2016) the critic's 's learning rate is set to 10 times higher than the actor's ' learning rate. However, we believe such setting only partially addresses the problem. During training, the learning rates of actor are always low, actors learn slowly and thus the learning is limited. Therefore, AdaMa decreases le and increases ||!l|| when the learning of critic reaches plateau, which could avoid the fast-moving target and speed up the learning.
3.4 SICOND-ORDER APPROXIMATION
Under the first-order Taylor approximation, the actor i's contribution AQ is only related to the change of o;, without capturing the joint effect with other agents' updates. However, when there are strong correlations between agents, the increase of the Q-value cannot be sufficiently estimated as the sum of individual contributions of each actor' update, which instead is a result of the joint update. To estimate the actors' contributions more precisely, we extend AdaMa to the second-order Taylor approximation to take pairwise agents' updates into account:
AQ Q(õ, ā Ad) Q(ā,a)
aQ(á,))
N
PQ(ā a) Da,0a;
Q( a) +
Aa
+
Aa
Aaj" T -(,,a))
Da.
í,==1
8Q a) (Q(ō,)) T
+
1
Σ
lacka;
aQ a) do, 82Q a) da aQ(ā,a)
00,
00,
00, 00, da,Da; 00, 00,
i=
4

---

Under review as a conference paper at ICLR 2021
Landmark
Prey
Center
Agent
Figure 2: Illustration of experimental scenarios: going together, cooperative navigation, predator- prey, and clustering (from left to right).
As the actors updated by the first-order gradient, still estimate Aā utiliizing the first-order approximation and compute the second-order AQ on the first-order Ad. Then, the gradient la aQ 0a," a²Q Oa aq T + 1 Σ las 00, 80, Ba,du; 80, 86,
adQ aQ aQ T 8la 89; 80; + Σド=z lay 89, 80, du,da; 80; 89;
aQ da. T 8²Q 0a, aq T
2
Similarly, la can be updated as:
JAQ 01,
AAQ Ola
= + (1-
/11
11,
[ = La
(3)
4 EXPERIMENTS
We validate AdaMa four cooperation scenarios with continuous observation space and continu- action space, which illustrated in Figure 2. In these scenarios, agents observe the relative positions of other agents, landmarks, and other items, and take two-dimensional actions E [-1, 1] physical velocity.
Going Together: the scenario, there are agents and landmark. The reward 0,, - dij. where dj is the distance from agent i to the landmark, and di is the distance between the two agents. The agents have to go to the landmark together, avoiding moving
away from each other.
Cooperative Navigation. In the scenario, there agents and corresponding landmarks. The reward is - -max, (d:), where d, is the distance from agent to the landmark The slowest agent determines the reward in this scenario.
Predator- Prey. In the scenario, 4 slower agents learn to chase faster rule-based prey. Each time one of the agents collide with the prey, the agents get reward +1.
Clustering. In the scenario, agents learn cluster together. The reward is Σ d, where de the distance from agent i to the center of agents' positions. Since the center is changing along with the agents' movements, there are strong interactions between agents.
To investigate the effectiveness of AdaMa and for ablation, we evaluate the following methods:
AdaMa adjusts 4e and ||=||| using (2), and T, according to (1).
Fixed uses grid search find the optimal combination of lg and ||[||| from 0.01 to 0.001 with step 0.001. The learning rate of each agent set to ||~a|// /N.
Adaptive In, direction sets L, and ||'a||| as that in Fixed and only adjusts the direction of l, using (1). Additionally, Adaptive I,, direction (2nd) uses the update rule (3) for the second-order approximation.
Adaptive and ||A,]| adjusts 4 and ||aa|| using (2) and sets 4, |IG
AdaGrad is an adaptive learning rate optimizer that performs larger updates for more sparse parameters and smaller updates for less sparse parameter. The initial learning rates are sets that in Fixed Ir.
Except AdaGrad, all other methods use SGD optimizer without momentum. More details about experimental settings and hyperparameters available in Appendix A.3. We trained all the models for five runs with different random seeds. All the learning curves are plotted using mean and standard deviation.
5

---

Under review as a conference paper at ICLR 2021
-02
Mean Reward
Maan Reward
-03
AdaMa
AdaMa
Adaptive fz direction
Adaptive fg direction
Adaptive & and
[[C]]]
-44
Adaptive and [K
Fixed lr
Fixed Ir
191
AdaGrad
AdaGrad
2
Episodes
(a) going together
(b) cooperative navigation
AdaMa
Adaptive f. direction
2.-
Adaptive 1, and [GII
Fived It
Moan Revard
AdaGrad
Mean
AdaMa
1.3
Adaptive direction
Adaptive and [K
Fixed
AdaGrad
Six. 10
4 10
3
3
101 Episodes
5%
Episodes
(c) predator-prey
(d) clustering
Figure 3: Learning curves in the four scenarios.
4.1 PERFORMANCE OF ADAPTIVE l DIRECTION
shown Figure 3(a) and 3(c), Adaptive T direction converges to higher reward than Fixed Ir that treats each agent as equally important. To make explicit explanation, visualize the normalized actors' learning rates /// || in Figure 4 for Appendix A.4. In going together and predator-prey. the actors' learning rates fluctuate dynamically and alternately as depicted Figure 4(a) and 4(b). An actor has much higher learning rate than other actors in different periods, meaning that the actor is critical to the learning. The direction of
and more results available in
l, is adaptive to the changing contributions during the learning, assigning higher learning rates to the actors that make more contributions to AQ. clustering, the center is determined by all agents' positions, and the actors' updates make similar contributions to AQ, leading to similar learning rates for the actors. That is the reason Adaptive T,, direction is not beneficial in this scenario. Moreover,
going Logether
(b) predator- pretoor-ppyyyy
BO M
(c) going together (2nd)
(d) cooperative navigation (2nd)
Figure Normalized actors' learning rates.

---

Under review as a conference paper at ICLR 2021
(a) going together
(b) cooperative navigation
(c) predator-prey
(d) clustering
Figure 5: le and ||C.]]-
in single-critic MADDPG, the gradient of an actor depends on the current policies of other actors. other actors are updating similar rate, the update of this actor will become unstable, since the changes of others' policies are invisible and unpredictable. In our method, the agents critical to increasing the Q-value learn fast while other agents have low learning rates, which partly attenuates the instability.
4.2 PERFORMANCE OF ADAPTIVE Le AND |Va.ll
As illustrated in Figure 3(b), 3(c), and 3(d), Adaptive le and ||%,]| learns faster than Fixed Ir. To interpret the results, we plot le and ||%]]] during the training in Figure 5 and find that le and ||[||| rise and fall alternately and periodically. When the update of the critic impacts greatly AQ, e.g., at the beginning with large TD-error, the fast-moving Q-value, which is the optimization target of actors, might cause performance breakdown if the actors also learning fast. In this situation, our method could adaptively speed up the learning of the critic and slow down the learning of actors for stability. After while, the TD-error becomes small and makes the critic reach plateau. According to the update rules (2), the learning of actors is accelerated whilst the learning rate of the critic falls, which keeps the target of actors stable and thus avoids the breakdown. The fast-improving actors generate new experiences, which change the distribution in the replay buffer and increase the TD- error. As a consequence,
learning rate of the critic rises again. Therefore, learning rates critic and actors fluctuate alternately, promoting the overall learning continuously. In going together, the alternate fluctuation is not obvious, Adaptive l. and ||T," performs worse than Fixed Ir with grid search.
Combined with the two adaptive mechanisms, AdaMa learns faster and converges higher reward than all other baselines in Figure 3(c). In other scenarios, AdaMa produces similar results to the
Mean Reward
12-
Mcan
.0a
Adaptive direction (2nd)
Adaptive direction (2nd)
Adaptive C direction
Adaptive fü direction
Fixed
Floed
2x
05.5 30
1x tel
35x 10
2x
Episodes
Episodes
(a) going together
(b) cooperative navigation
Figure Learning curves with the second-order approximation.

---

Under review as a conference paper at ICLR 2021
41
LGS
50
50
Mean Reward
Fixed
100
1x
2- 10f
3i 107
5xx 101
3.14
Episodes
Episodes
(b) curve of 8Q T
(a) learing curves with different m
36 D5
Figure 7: Learning curves and visualizations with different in Predator-Prey.
mechanism that brings the main improvement. Since Fixed Ir has to search 100 combinations, the cost is prohibitive. Despite adaptively adjusting the learning rates, AdaGrad does not show competitive performance, since it only focuses on the gradient pattern, ignoring the characteristics of MARL.
4.3 PERFORMANCE OF SECOND-ORDER APPROXIMATION
In Figure 3, the performance gain of Adaptive 1o direction is limited, which we think is attributed to that the first-order approximation is relatively rough when an actor's update affects other actors' updates. We apply the second-order approximation to Adaptive la direction (2nd) and find that achieves better results shown in Figure 6. Comparing the later episodes in Figure 4(a) and 4(c), there is larger gap between the two actors' learning rates under the second-order approximation. This accurately reflects the later training is dominated by actor, which is the reason for the higher reward. In Figure 4(d), there are obvious ups and downs in the learning rates of actors before convergence (1 1 episodes), and after that the fluctuation becomes gentle. The second-order approximation that captures the pairwise effect of agents' updates on AQ obtains more accurate update on the learning rates, which eventually leads to better performance.
4.4 TUNING HYPERPARAMETER m
The hyperparameter controls the target value of the critic's learning rate. If m is too large or too small, the learning rate will reach the boundary value el or (1 (1-e)4, e)4, which destroys the adaptability and hampers the learning process. An empirical approach for tuning is setting to be the mean 36 aq of the first K updates in a trial run. In predator-prey, we test m = 40, 50, 60, among Do 80
which 50 is the rounding mean value of 100 updates, and plot the results in Figure 7(a). The three settings show similar performance, revealing our method is robust to the hyperparameter m. Having noticed that le and ||[||| change violently in Figure
2M aq |during the training in Be de
visualize
Figure 7(b) to interpret the robustness. Since most of the time
08 aq is higher than 60 or lower Bo 06
than 40, similar learning rate patterns is observed when m is between 40 and 60, which verifies that there is high fault tolerance in m. Although Adaptive 4. and ||(,||| converges to similar reward with Fixed Ir, the former learns faster and is much easier to tune.
CONCLUSION
In this paper, we proposed AdaMa for adaptive learning rates in MARL. AdaMa adaptively updates the vector of learning rates of actors to the direction of maximally improving the Q-value. It also dynamically balances the learning rates between the critic and actors during learning. Moreover, AdaMa incorporates the higher-order approximation to more accurately update the learning rates of actors. Empirically, we show that AdaMa could accelerate the learning and improve the performance in variety of multi-agent scenarios.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2):215-251)) 250, 2002.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Jakob N Foerster, Gregory Farguhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Shariq Iqbal and Fei Sha. Actor-attention-critie for multi-agent reinforcement learning, In Interna- tional Conference on Machine Learning (ICML), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Roman Liessner, Jakob Schmitt, Ansgar Dietermann, and Bernard Bäker. Hyperparameter optimiza- tion for deep reinforcement learning in vehicle energy management. In International Conference on Agents and Artificial Intelligence (ICAART), 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference Learning Representations (ICLR), 2016.
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettivee environments. In Advances in Neural Information Processing Systems (NewrIPS), 2017.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-basee hyperparameter optimiza- tion through reversible learning, In International Conference Machine Learning (ICML), 2015.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICMI), 2018.
Shai Shalev- Shwarr-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:
2016.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2019.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. In International Conference on Information and Knowledge Management (CIKM), 2019.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems (NeurIPS), 2018.
Yaodong Yang, Jianye Hao, Mingyang Sun, Zan Wang, Changjie Fan, and Goran Strbac. Recurrent deep multiagent g-learning for autonomous brokers in smart grid. In International Joint Confer- ence on Artificial Intelligence (IJCA!). 2018
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.570. 2012.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 INSTANTIATING ADAMA ON MAAC
AdaMa could be applied to other multi- multi--aeent actor-critic methods, modified according to the gradi- ent computation. We present how to instantiate AdaMa on MAAC (Iqbal & Sha, 2019). In discrete action space, Ad does not meet the assumption of the Taylor approximation. But in practice, is feasible to learn the value function (ó, o, iT,, taking as input the action distribution я, instead of the action a₃. Since the update of the actor is different from that MADDPG, the change of Q-value is written
AQ = Q(0 AQ, O, A Q(o,ō,
v aQ(6, ō,
O)
+
Ad
O,
O6
N
blog n. A, -5-,,, [0,))
aQ o. #)
(0: + la:
+ AΦ
00,
① Olog A 0m, T 0Q a,))
08 8Q o, F)
- le 0Φ
00,
00:
0T.
06
a log क. A aQ T
06 aQ
le
09,
08;
0e 06
A; is the advantage function of agent i' proposed in MAAC. Rewritting the gradient of la; 8 log A. 8Q T
BAQ
Bl.
the AdaMa implementation MAAC is the same that on MADDPG.
00, 00;
A.2 ADAMA ALGORITHM
For completeness, we provide the AdaMa algorithm on MADDPG below.
Algorithm 1 AdaMa on MADDPG
1: Initialize critic network Φ, actor networks 0,, and target networks
Initialize the learning rates le and la Initialize replay buffer D
2: for episode= = 1, M do
3: for (. 1,... do
4: Select action at ग. (ot) + NV for each agent i
5: Execute action a;. obtain reward rt, and get new observation o/41 for each agent
6: Store transition (or, ae, Ft, ox+1) in D
7: end for
8: Sample random minibatch of transitions from DD
Adjust and || by le alc (1 a:)l clip() 06 8Q T ae B6
1/m, c, - c), [Tall = 1 1-Le la
Adjust lo by T al, + (1 - ||a1 AAQ /I DAQ
|1
La |
at. ai.
Update the critic é by Φ = Φ Le 05 where is the TD-error Dz
Update the actor 0; by 0; 0: + lu.
aQ(ā,a) da; for each agent da.
Update the target networks
9: end for
A.3 EXPERIMENTAL SETTINGS AND HYPERPARAMETERS
In each task, the experimental settings and hyperparameters summarized in Table 1. Initially, set 1, ||-|| and la; [/ in AdaMa. For exploration, add random noise the action
10

---

Under review as a conference paper at ICLR 2021
(1- (1-E)a; E.), where the uniform distribution 7j [-1, 1]. We anneal linearly from 1.0 to 0.1 over 10 episodes and keep it constant for the rest of the learning. We update the model every episode and update the target networks every 20 episodes.
Table 1: Hyperparameters
Hyperparameter
Going Together
Cooperative Navigation
Prodator-Prey
Clustering
horizon (7)
10
20
10
discount (n)
0.96
0.9
0.97
0.95
replay buffer size
5 x 10
x 10
1 X 10
1 10%
Le (grid search)
8 10-3
x 10
x 10
8 x 10
(grid search)
10-3
x 10-3
10
1 10
batch size
1024
MLP units
(64, 64)
MLP activation
ReLU
10
50
80
0.99
x 10
0.1
A.4 ADDITIONAL RESULTS
Here, we show the visualizations in the other four runs. Similar curve patterns are observed in all runs of each scenario.
Pyheddos
Figure Normalized actors' learning rates in going together

---

Under review as a conference paper at ICLR 2021
Figure 9: Normalized actors' learning rates in predator-prey
Calodes
Figure 10: Normalized actors' learning rates in going together (2nd)
Apdi
Azor Azor
Figure 11: Normalized actors' learning rates cooperative navigation (2nd)
12

---

Under review as a conference paper at ICLR 2021
801
120
Figure 12: L. and ||C/|| in going together.
wmwn Ar
mm
Figure 13: and ||[||] in cooperative navigation.
Cood
1
Figure 14: l. and [|?|||] in predator- prey.
13

---

Under review as a conference paper at ICLR 2021
Figure 15: and ||[|]] || in clustering.
14


Reviews:
[{"review_id": "wIC6Ara7fk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm called AdaMa for multi-agent reinforcement learning (MARL). Based on the contribution of the critic and actors, the algorithm adopts adaptive learning rates. Numerical experiments are provided in four cooperation scenarios to show the performance of AdaMa.\n\n\nIn my view, the paper currently falls below the bar for an ICLR publication. The detailed comments are as follows:\n\n- The key concern about this paper is the lack of rigorous analysis. The contents in Section 3 is largely based on heuristic approximations. There is no rigorous analysis, statement or proof. And there is theoretical guarantees of the performance of the algorithm with respect to the sample complexity or time complexity.\n\n- In addition, there is no comparison with state-of-the-art algorithms, which limits the contribution. It would be much better if authors can include this either in terms of theory or numerical experiments.\n\n- The paper is not well written. For example, there is no formal description of the proposed algorithm.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bz6Ay9QyJrq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies adaptive learning rate for Actor-Critic style MARL algorithm. I have several questions:\n\n1. How is the adaptive learning rate related to MARL? It seems this is just a general improvement for actor-critic methods?\n2. This is closely related to the first question. Some optimization algorithms with adaptive learning ate are mentioned in the related work section. However, it is not clear why these methods cannot solve the problem and we need new techniques. Or, what structure in MARL makes it possible to achieve further improvement? Is this possible for, say single-agent setting?\n3. I am not very familiar with the Dec-POMDP setting so I have a stupid question: shouldn't the policy a function of the state instead of the observation? Otherwise, we will need to work with the significantly larger observation space instead of the state space?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I_SigFtVNo9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\nSummary\n\nThis paper studies automatic learning rate tuning in Multi-Agent Reinforcement Learning (MARL). It proposes AdaMa, an algorithm which balances the learning rates of actors and the critic, and can also make use of the second-order information. Experiments show that AdaMa can learn agents faster. \n\n\n---\nWriting Quality\n\nThe paper is not well-written as I don't fully understand it. Notations are used without definition ($\\delta$ in Section 3.3) and inconsistent (some $Q$ has arrow above it and some doesn't) , and $\\frac{\\partial Q}{\\partial \\theta}$ has a confusing shape. Simpler symbols can also be used for readability (e.g., $\\widehat{\\|\\overrightarrow{l_a}\\|}$ can be replaced by something like $\\eta_a$). The derivation of $\\Delta Q$ can be deferred to appendix. \n\nI'd suggest the authors to make an algorithm box for the full algorithm. Equation (1), (2) and (3) might be confusing, as they're describing algorithms, not mathematical equations. \n\n\n---\nComments\n\n\nI'm not fully convinced. The figures indeed show that AdaMa uses different LR for different actor. Why is this helpful? Why should we slow down the learning of an actor if it's already good? \n\nHow does Adam, one of the most widely used optimizer, work? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "88Tt76X4Egr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n\nThe paper proposes a new algorithm for multi-agent reinforcement learning (MARL) that adaptively picks learning rates for actor and critic. Specifically, the learning rates are updated to directions maximally affecting the Q-function, and the algorithm dynamically balances the learning rates between actor and critic. In numerical studies, the authors illustrate the efficiency of their method via four toy experimental scenarios and intuitively explain the underlying mechanism.\n\n\n------------------------------------------------------------------------------------------\n\n\nPros:\n\n+ The choice of learning rates in MARL is an interesting and important issue.\n+ The paper is well written. The methodology part is clearly organized and easy to follow. The learning rate balance between actor and critic is well motivated.\n+ The numerical experiments are well designed. Each model represents a different cooperation mode. The results are well presented.\n\n\n------------------------------------------------------------------------------------------\n\nCons:\n\n- The numerical results are not satisfying. The improvement in AdaMa is incremental. In Figure 3 (a), (b) and (d), AdaMa has similar performance with Fixed lr (fixed learning rate).\n- Currently each model is trained for 5 runs. More experiments are needed for more reliable results.\n- The authors should evaluate the performance of AdaMa in more practical models in addition to these four toy examples.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ppvaLxt7GP", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed AdaMa, which can automatically use adaptive learning rates for each agent in cooperative Multi-Agent Reinforcement Learning (MARL). AdaMa calculated the learning rate of each actor and critic according to their contributions of locally increasing value functions.  Simple experiments using toy examples show that the proposed AdaMa method can improve fixed learning rate method and other heuristics.\n\nPros:\n1. The topic and idea of using adaptive learning rates to avoid hand-tuning are quite interesting and important. I think the related topics are worth investigating.\n2. The proposed AdaMa method looks reasonable to me, at least from an intuitive perspective.\n3. Experimental results also look promising.\n\nCons:\n1. The experiments look simple, and the improvements seem to be incremental rather than significant (please correct me if I misunderstood or missed something). I think more experiments on larger scale tasks are needed to make the effectiveness of the proposed method convincing.\n2. Using first- and second-order Taylor expansion to obtain the best possible learning rates seems to be a reasonable idea. However, I think some more rigorous theoretical understanding is worth pursuing to show the benefits of the proposed method in a more convincing way, e.g., the speed of convergence for fixed and adaptive learning rate methods.\n\nOverall, I found the problem and the idea important and interesting. The proposed method is intuitively reasonable and verified by small scale experiments. However, the proposed method is not convincing in terms of the lack of larger-scale experiments or theoretical results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how to deploy adaptive learning rates in multi-agent RL (MARL). In particular, the learning rates are adaptively chosen based on which directions maximally affect the Q-function, and take into account the interplay and balance between the actors and the critics. The topic is certainly of great interest when designing fast-convergent MARL algorithms. However, the reviewers point out the inadequacy and insufficiency of empirical gains in the reported experiments. Also, larger-scale experimental settings are needed in order to provide more convincing evidence about the practical benefits of the proposed scheme.   
Decision: Reject