MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning
Authors: Kefan Su
Siyuan Zhou
Chuang Gan
Xiangjun Wang
Zongqing Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning

Abstract: Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns to update their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees $\varepsilon$-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.

Under review as a conference paper at ICLR 2023
MA2QL: A MINIMALIST APPROACH TO FULLY DECEN- TRALIZED MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Decentralized learning has shown great promise for cooperative multi-agent inforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized leaninig In the paper, we tackle the con-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns update their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees s-convergence at each turn, their joint policy converges to Nash equilibrium. practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) is a well-abstracted model for a broad range of real applications, including logistics (Li et al., 2019), traffic signal control (Xu et al., 2021), power dispatch (Wang et al., 2021b), and inventory management (Feng et al., 2022). In cooperative MARL, centralized training with decentralized execution (CTDE) is a popular learning paradigm, where the information of all agents can be gathered and used training. Many CTDE methods (Lowe et al., 2017; Foerster et al., 2018; 8; Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 202 Zhang et al., 2021; Su & Lu, 2022; Li al., 2022) have been proposed and shown great potential solve cooperative multi- agent tasks.
Another paradigm is decentralized learning, where each agent learns its policy based on only local information. Decentralized learning is less investigated but desirable in many scenarios where the information of other agents is not available, and for better robustness, scalability, and security (Zhang al., 2019). However, fully decentralized learning of agent policies (i.e., without communication) is still an open challenge in cooperative MARL.
The most straightforward way for fully decentralized learning is directly applying independent learning at each agent (Tan, 1993), which however induces the well-known non-stationarity problem for all agents (Zhang et al., 2019) and may lead learning instability and non-convergent joint policy, though the performance varies as shown in empirical studies (Rashid et al., 2018; de Witt al., 2020; Papoudakis et al., 2021; Yu al., 2021).
In the paper, we directly tackle the non-stationarity problem in the simplest and fundamental way. i.e., fixing the policies of other agents while one agent learning. Following this principle, we propose multi-agent alternate Q-learning (MA2QL), a minimalist approach to fully decentralized cooperative multi-agent reinforcement learning, where agents take turns to update their policies by Q-learning. MA2QL is theoretically grounded and we prove that when each agent guarantees e-convergence at each turn, their joint policy converges to a Nash equilibrium. only requires minimal changes to independent Q-learning (IQL) (Tan, 1993; Tampuu et al., 2015) and also independent DDPG (Lillicrap et al., 2016) for continuous action, i.e., simply swapping the order of two lines of codes follows. Their major difference be highlighted as: MA2QL agents take turns to update Q-functions by Q-learning, whereas IQL agents simultaneously update Q Q-functions by Q-learning.
practice, MA2QL

---

Under review as a conference paper at ICLR 2023
IQL
MA2QL
repeat
repeat
2: all agents interact in the environment
2:
for i f 1. n do
3: for i 6 1, n do
all agents interact in the environment agent i updates by Q-learning 5: end for
3:
4:
agent i updates by Q-learning
4:
5: end for
6:
until terminate
6:
until terminate
We evaluate MA2QL didactic game empirically verify its convergence, and multi-agent particle environments (Lowe et al., 2017). multi-agent MuloCo (Peng et al., 2021), and StarCraft multi-agent challenge (Samvelyan al., 2019) verify its performance with discrete and continuous action spaces, and fully and partially observable environments. We find that MA2QL consistently outperforms IQL, despite such minimal changes. The effectiveness of MA2QL suggests that sim- pler approaches may have been left underexplored for fully decentralized cooperative multi-agent reinforcement learning.
2
BACKGROUND
2.1 PRELIMINARIES
Dec-POMDP. Decentralized partially observable Markov decision process (Dec-POMDP) is general model for cooperative MARL. Rec-POMDP is a tuple M P, r. S is the state space, is the number of agents, 7 [0, 1) is the discount factor, and = {1, (1,2...n] is the set of all agents. A = A x A₂ x ... x A, represents the joint action space where A; is the individual action space for agent i. P(s'ls, a) S x A S + [0, 1] is the transition function, and r(s, a): : S x A + R is the reward function of state and joint action a. Y is the observation space, and Oss, a) S I Y is a mapping from state observation for each agent. The objective of Dec- POMDP is to maximize J(T = Ex |Ei= ('r(st, at)], and thus we need to find the optimal joint policy x arg max ( settle the partial observable problem, history Tı E T. (Y x A:)" is often used to replace observation o, Y. Each agent i has an individual policy T:(a: m)) and the joint policy π the product of each k, Though the individual policy learned as n. (4./TT) in practice. Dec-POMDP is undecidable (Madani et al., 1999) and the analysis in partially observable environments is much harder, we will use ग. (a,[s) 5) in analysis and proof' for simplicity.
Dec-MARL. Although decentralized cooperative multi-agent reinforcement learning (Dec-MARL) has been previously investigated (Zhang et al., 2018; de Witt et al., 2020), the setting varies across these studies. In this paper, we consider Dec-MARL POMDP, where each agent learns its policy/Q-function from its own action individually without communication or parameter-sharing. Therefore, in Dec-MARL, each agent i actually learns in the environment with transition function (s'sss,, a²) = -a [P('') a, a-)] and reward function ri(s, a.) -o__:>x___r(s 0j, a=]]] where -= and a respectively denote the joint policy and joint action of all agents expect i. As other agents are also learning (i.e., . is changing), from the perspective of each individual agent, the environment non-stationary. This is the stationniny problem, the main challenge in Dec-MARL.
fully decentralized solution to Dec-
IQL. Independent Q-learning (IQL) is straightforward method for Dec-MARL, where each agent i learns a Q-function Q(,, a.) by Q-learning. However, as all agents learn simultaneously. there is no theoretical guarantee on convergence due to non-stationarity, to the best of our knowledge. In simple baseline in favor of more elaborate MARL approaches, such
practice, IQL is often taken as value-based CTDE methods (Rashid et al., 2018; Son et al., 2019), However, much less attention has been paid to IQL itself for Dec-MARL.
2.2 MULTI-AGENT ALTERNATE POLICY ITERATION
To address the stationiinn problem in Dec-MARL, a fundamental way is simply to make the environment stationary during the learning of each agent. Following this principle, we let agents learn by turns; in each turn, one agent performs policy iteration while fixing the policies of other agents. This procedure is referred to multi-agent alternate policy iteration. As illustrated in Figure 1,
2

---

Under review as a conference paper at ICLR 2023
Aulti-Agent Alternate Policy Iteration
Visn 2 Va
policy Iteration of agent round k
policy lleration of agent round
policy lterationn of agent 3 in round
policy iteration of agent round +
equilibrium
Mi (29 1
M
M
M
M of M unless sT., has converged
Q-iterations of agent round l
I Q-llerations of agent round
Q-tterations of agent 3 in round
Q-iterations of agent round +-11
Nash equilibrium
Multi-Agent Alternate Q-Iteration
1Q Q: c
Figure I: Illustration of multi-agent alternate policy iteration (upper panel) and multi-agent alternate Q-iteration (lower panel) three agents. As essentially the MDP differs at different turns of each agent, policy iteration/Q-ittratton of each agent iterates over different MDPs.
multi-agent alternate policy iteration differs from policy iteration in single-agent RL. In single-agent RL, policy iteration is performed on the same MDP. However, here, for each agent, policy iteration different round performed on different MDP. As π is fixed each turn Ps(s"|s, a;) and rs(s, a.) are stationary and we can easily have the following lemma.
Lemma (multi-agent alternate policy iteration). If all agents take turns to perform policy iteration, their joint policy sequence {T) monotonically improves and converges to Nush equilibrium.
Proof. In each turn, as the policies of other agents are fixed, the agent has the following update rule for policy evaluation,
(s, a;) t as) + P. 1Q.. (s', d()]
We can have the convergence of policy evaluation in each turn by the standard results (Sutton & Barto, 2018). Moreover, as . is fixed, it is straightforward to have
Qn, (s, a.)
a,
(2)
Then, the agent performs policy improvement by
Aig
new
(s) arg max Eny [Qmord (s, as, a-)) a,
(3)
As the policies of other agents are fixed (i.c., T old), we have
Vyota (s) E Egiiia QQeara (s, ais am)] = EEpplaB 1Qmola ai a.ل)
S E new Enoid 12 l (s, aj a) = E new Ee Exnow [Q (s, aj, a-)]
(4)
EL now 12 qulaa (s, ai, a=))] = m (r ai a-;) + 7 Viuiaa (s')l
< < V new (s).
where the first inequality is from (3). This proves that the policy improvement of agent in each turn also improves the joint policy. Thus, as agents perform policy iteration by turn, the joint policy sequence {π] improves monotonically, and {T) will converge to Nash equilibrium since no agents can improve the joint policy unilaterally at convergence.
Lemma 1 immediately indicates approach for Dec-MARL. with convergence guarantee and also tells us that if we find the optimal policy for agent i in each round & given the other agents' policies Th p. then the joint policy will obtain the largest improvement. This result can be formulated as follows,
A arg max E, [Or,.t, (s, as)a),a)) W.
(5)
,, (8)
ar (s) Vm,, Vs.
& We could obtain this T. by policy iteration with many on-policy iterations. However, such method will face the issue of sample inefficiency which may be amplified in MARL settings. We will use Q-iteration settle this problem in the next section.
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
To address the problem of multi-agent alternate policy iteration, we propose multi-agent alternate Q-iteration, which is sufficiently truncated for fast learning but still has the same theoretical guarantee. Further, based on multi-agent alternate Q-iteration, we derive multi-agent alternate Q-learning, which makes the minimal change to IQL to form simple yet effective value-basee decentralized learning method for cooperative MARL.
3.1 MULTI-AGENT ALTERNATE Q-ITERATION
Instead of policy iteration, we let agents perform Q-iteration by turns depicted in Figure Let M r denote the MDP of agent in round k, where have M 7 MM unless T- has converged, and Q (s. a;) denote the Q-function of agent i with updates in the round k. We define the Q-iteration as follows,
(st + (8, + t max (6', .d))
(6)
Then, the sequence (Q/t) converges to O with respect to the MDP M (P r and have the following lemma and corollary.
Lemma 2 (e-convergent Q-iteration). By iteratively applying Q-iteration (6) at each agent i for each turn, for any E > 0, we have
10 27 Se, when t t2 log ((1 - 7)s) log 2E)
log
where Tmax and Tmax = maxs,a r(s, I-7
Corollary 1. For any > 0, we take sufficient Q-iteration th, i.e., O Q .hhn have I1
| - Q2 02* Se Vk,
With Lemma 1, Lemmaa 2, and Corollary 1, we have the following theorem.
Theorem (multi-agent alternate Q-iteration). Suppose that Q, (s. -) has the unique maximum for all states and all agents. If all agents turn take Q-iteration to I| Q, ll S E, then their joint policy sequence {**} converges Nash equilibrium, where x (s) arg maxa; Q (s, az).
All the proofs are included in Appendix A.
Theorem 1 assumes that for each agent, Qi has the unique maximum for all states. Although this may not hold in general, in practice we can easily settle this by introducing positive random noise the reward function. Suppose the random noise bounded by ō, then can easily derive that the performance drop of optimizing environmental reward plus noise is bounded by 6/(1 7). As we can make ó arbitrarily small, the bound is tight.
3.2 MULTI-AGENT ALTERNATE Q-LEARNING
From Theorem 1, we know that if each agent guarantees e-convergence Q** in each round k, multi-agent alternate Q-iteration also guarantees Nash equilibrium of the joint policy. This immediately suggests simple, practical fully decentralized learning method, namely multi-agent alternate Q-learning (MA2QL).
For learning Q-table or Q-network, MA2QL makes minimal changes to IQL.
For learning Q-tables, all agents in turn update their Q-tables. At round & of an agent i, all agents interact the environment, and the agent updates its Q-table a few times using the collected transitions (s, 0;, s²).
For learning Q-networks, all agents in turn update their Q-networks. round of an agent i, all agents interact in the environment, and each agent j stores the collected transitions (s, aa.r.,, s') into its replay buffer, and the agent updates its Q-network using sampled mini-batchi from its replay buffer.
4

---

Under review as a conference paper at ICLR 2023
There is slight difference between learning Q-table and Q-network. Strietly following multi-agent alternate Q-iteration, Q-table is updated by transitions sampled from the current MDP. On the other hand, Q-network is updated by mini-batches sampled from the replay buffer. If the replay buffer only contains the experiences sampled from the current MDP, learning Q-network also strictly follows multi-agent altemnte Q-iteration. However, in practice, we slightly deviate from that and allow the replay buffer to contain transitions of past MDPs, following IQL (Sunchag et al., 2018; Rashid al., 2018; Papoudakis et al., 2021) for sample efficiency, the convergence may not be theoretically guaranteed though.
MA2QL and IQL can be simply summarized and highlighted as MA2QL agents take turns update Q-functions by Q-learming, whereas IQL agents simultaneously update Q-functions by Q-learning.
4 RELATED WORK
CTDE. The most popular learning paradigm in cooperative MARL is centralized training with decentralized execution (CTDE), including value decomposition and multi-agent actor-critic. For value decomposition (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang et al., 2021a), joint Q-function is learned in centralized manner and factorized into local Q-functions to enable decentralized execution. For multi-agent actor-critic, centralized critic, Q-function or V-function, learned to provide gradients for local policies (Lowe et al., 2017: Foerster et al., 2018; Yu et al., 2021). Moreover, some studies (Wang et al., 2020; Peng et al., 2021; Su & Lu, 2022) combine value decomposition and multi-agent actor-critic take advantage of both, while others rely on mexiimum-entrop RL to naturally bridge the joint Q-function and local policies (Iqbal & Sha, 2019; Zhang et al., 2021).
Decentralized learning. Another learning paradigm in cooperative MARL is decentralized learning, where the simplest way for each agent to learn independently, e.g., independent Q-learning (IQL) or independent actor-critic (IAC). These methods are usually taken simple baselines for CTDE methods. For example, IQL taken as baseline in value decomposition methods (Sunehag et al., 2018; Rashid et al., 2018), while IAC is taken 2018; Yu et al., 2021). Some study further considers decentralized learning with communication (Zhang et al., 2018; Li et al., 2020). However, parameter-sharing should be
baseline in multi-agent actor-critic (Foerster et al.,
for parameter-sharine considered as centralized training (Terry et al., 2020). More recently, IAC (i.e., independent PPO) has been empirically investigated and found remarkably effective in several cooperative MARL tasks (de Witt et al., 2020; Yu et al., 2021), including multi-agent particle environments (MPE) (Lowe al., 2017) and StarCraft multi-agent challenge (SMAC). However, as actor-critic methods follow the principle different from Q-leaning, we will not focus on IAC for comparison in the experiment. On the other hand, IQL has also been thoroughly benchmarked and its performance is close to CTDE methods in few tasks (Papoudakis et al., 2021). This sheds some light on the potential of value-based decentralized cooperative MARL. Although there are some Q-learning variants, iee. hysteretic Q-learning (Matignon et al., 2007) and lenient Q-learning (Palmer et al., 201 8), for fully decentralized learning, they both are heuristic and their empirical performance is even worse than IQL (Zhang et al., 2020).
Unlike existing work, MA2QL is theoretically grounded for fully decentralized learning and in practice makes minimal changes to IQL. In the next section, we provide the empirical comparison between MA2QL and IQL.
EXPERIMENTS
In this section, we empirically study MA2QL on a set of cooperative multi-agent tasks, including a didactic game, multi-agent particle environments (MPE) (Lowe et al., 2017), multi-agent MuJoCo (Peng et al., 2021), ), and StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019), to investigate the following questions.
1. Does MA2QI. converge and what does it converge to empirically, compared with the optimal solution and IQL.? Does the number of Q-function updates affect the convergence?
As MA2QL only makes the minimal changes to IOL, is MA2QL indeed better than IQL in both discrete and continuous action spaces, and in more complex tasks?
5

---

Under review as a conference paper at ICLR 2023
250
350
multi
musts 19 1n
MAT 10.004
3-yddter 50
2033 Cempter
P520U (0000000--53
0-
Cemplerr
100
301
533
30:0
1568
2000
120
42c
300
200
5.00
(a) Q-iteration
(b) MA2QI
(c) Q-table updates
(d) samples
Figure 2: Empirical studies of MA2QL, on the didactic game: (a) different numbers of Q-iterations performed by dynamic programming each turn; (b) learning curve of MA2QL compared with IQL and the global optimum: (c) different numbers of Q-table updates at each turn; (d) different number of sampled transitions at each turn, where x-axis is learning steps.
In all the experiments, the training of MA2QL and IQL is based on the same number of environmental steps (i.e., the same number of samples). Moreover, as the essential difference between MA2QL and IQL. is that MA2QI agents take turns to update Q-function while IQL. agents update Q-function simultaneously, for a fair comparison, the total number of Q-function updates for each agent in MA2QL set to be the same with that in IQL. For example, in a setting of agents, if IQL agents update Q-function one step (e.g., one gradient step) every environmental step, MA2QL agents update Q-function n steps every n environmental steps. For learning Q- Q-networks, the size of the replay buffer is also the same for IQL and MA2QL. Again, we do not use parameter-sharae shariig.... which should not be allowed in decentralized settings (Terry et al., 2020). More detailed experimental settings, hyperparameters, and additional results are available in Appendix B, C, and D, respectively. All results are presented using the mean and standard deviation of five random seeds.
5.1 A DIDACTIC GAME
The didactic game is a cooperative stochastic game, which is randomly generated for the reward function and transition probabilities with 30 states, 3 agents, and 5 actions for each agent. Each episode the game contains 30 timesteps. For comparison, we use dynamic programming to find the global optimal solution, denoted as OPTIMAL. For MA2QL and IQL, each agent independently learns a 30 X 5 Q-table.
First, we investigate how the number of Q-iterations empirically affects the convergence of multi- agent alternate Q-iteration, where Q-iteration is performed by dynamic programming and denoted as MA2QL-DP. As shown in Figure 2(a), can see that different numbers of Q-iterations (i.e., t t= 1,5, 10, 50) that each agent takes at each turn do not affect the convergence in the didactic game, even when & = 1. This indicates e-convergence of Q-iteration can be easily satisfied with as few as one iteration.
Next, we compare the performance of MA2QL and IQL. As illustrated in Figure 2(b), IQL converges slowly (about 2000 steps), while MA2QL converges much faster (less than 100 steps) to a better return and also approximates OPTIMAL. Once again, MA2QL and IQL use the same number of samples and Q-table updates for each agent. One may notice that the performance of MA2QL is better than MA2QL-DP. This may be attributed to sampling and exploration of MA2QL, which induces a better Nash equilibrium.
Then, we investigate MA2QL in terms of the number of Q-table updates each turn, which resembles the number of Q-iterations by learning on samples. Specifically, denoting K as the number of Q-table updates, to update an agent, we repeat K times of the process of sampling experiences and updating Q-table. This means with a larger K, agents take turns less frequently. As shown in Figure 2(c), with larger K, the learning curve is more like, which means in this game a small number K is enough for convergence at each turn. Thus, with larger K, the learning curve converges more slowly.
Last, we investigate how the number of collected transitions at each turn impacts the performance of MA2QL. As depicted in Figure 2(d), the performance of MA2QL is better with more samples. This is because, with more samples, the update of Q-learning using these samples is more like to induce a full iteration of Q-table.
In summary, as illustrated in Figure 2. the number of Q-function updates or samples may influence the performance of MA2QL.. However, in the following experiments, we tune the performance of

---

Under review as a conference paper at ICLR 2023
1emple_ppeedddd
line control
pircle_control
-25
-60
Eprode Reworts
Epiaadd
120
-10
MA2QL
47.
IOL
34
25
1.5
26
15
00
12
2.
22
1.
00
25
25
Figure 3: Learning curve of MA2QL compared with IQL in 5-agent simple spread, S-agent line control, and 7-agent circle control in MPE, where x-axis is environment steps.
HalfCheetah
Hopper
Walker2d
E2p000 Rowbrds
1.00
Eplacia
125
MA2QL
IQL
Figure 4: Learning curve of MA2QL compared with IQL in 2-agent HalfCheetah, 3-agent Hopper, and 3-agent Walker2d in multi-agent MuJoCo, where x-axis environment steps.
IQL as best as we can and correspondingly change the configuration of MA2QL as stated above for fair comparison.
5.2
MPE
MPE is a popular environment in cooperative MARL. We consider three partially observable tasks: 5-agent simple spread, 5-agent line control, and 7-agent circle control (Agarwal et al., 2020), where the action space set to discrete. Moreover, we use the sparse reward setting for these tasks, thus they are more difficult than the original ones. More details are available in Appendix B. For both IQL and MA2QL, Q-network learned by DQN (Mnih et al., 2013).
Figure 3 shows the learning curve of MA2QI. compared with IQL in these three MPE tasks. In simple spread and circle control, at the early training stage, IQI. learns faster and better than MA2QL, but eventually MA2QL converges to better joint policy than IQL. IQL always converges to worse sub-optimum than MA2QL, similar to that in the didactic game. Morcover, unlike the didactic game, simultaneous learning of IQL may also make the learning unstable even at the late training stage as in line control and circle control, where the episode rewards may decrease. On the other hand, learning by turns gradually improves the performance and converges better joint policy than IQL.
As MA2QL and IQL both use replay buffer that contains old experiences, why does MA2QL outper- form IQL? The reason is that their experiences are generated in different manners. In the Q-learning procedure for each agent i, the ideal target is y: r. (s, 0;) + ET/ ((a,,,) [max,; Qi(s', a4)) and the practical target is 5; = - (s, ax) + (- [max,; Qi(s', (,(s,,,)), where TD is the average joint policy for the experiences in the replay buffer. We then can easily obtain bound for the target that 1y; - Mil < r (-)) give an analysis from the aspect of the divergence between π and лD. MA2QL obtains experiences with only one agent learning, so the variation for the joint policy is smaller than that of IQI.. Thus, in general, the divergence between π and TD is smaller for MA2QL, which is beneficial to the learning.
where Tmax max,,a r(s, a). We can then
5.3 MULTI-AGENT MUJoCo
Multi-agent MuJoCo has become a popular environment in cooperative MARL for contimuous action space. We choose three robotic control tasks: 2-agent HalfCheetah, 3-agent Hopper, and 3-agent
7

---

Under review as a conference paper at ICLR 2023
3s_ $$_vs_44 4z
5m _i__s___ 6m
corridor
10
Rude
Rade
04
MA2QL
sdagae
stapss
Figure 5: Learning curve of MA compared with IQL. on 3s_ __ (easy), 5m vs 6m (hard) and corridor (super hard) in SMAC, where x-axis is environment steps.
Walker2d. To investigate continuous action space in both partially and fully observable environments, we configure 2-agent HalfCheetah and 3-agent Hopper as fully observable, and 3-agent Walker2d as partially observable. For both IQL and MA2QL, we use DDPG (Lillicrap et al., 2016) as the alternative to DQN to learn a Q-network and a deterministic policy for each agent to handle continuous action space.
In comparison to discrete action space, training multiple cooperative agents in continuous action space still remains challenging due the difficulty of exploration and coordination in continuous action space. Thus, the evaluation on these multi-agent MuJoCo tasks can better demonstrate the effectiveness of decentralized cooperative MARL methods. As illustrated in Figure 4, in all the tasks, we find that MA2QL consistently and significantly outperforms IQL while IQL struggles. We believe the reason is that the robotic control tasks are much more dynamic than MPE and the non-stationarity induced by simultaneous learning of IQL may be amplified, which makes it hard for agents to learn effective and cooperative policies. On the other hand, alternate learning of MA2QI. can deal with the non-stationarity and sufficiently stabilize the environment during the learning process, especially in HalfCheetah and Hopper, where MA2QL stably converges to much better performance than IQL. According to these experiments, we can verify the superiority of MA2QL over IQL in the continuous action space.
5.4 SMAC
SMAC popular partially observable environment for benchmarking cooperative MARL algorithms. SMAC has a much larger exploration space, where agents are much easy to get stuck in sub-optimal policies especially in the decentralized setting. We test our method on three representative maps for three difficulties: 3s ss_v______ 4z (easy), 5m vs 6m (hard), and corridor (super hard), where harder map has more agents. It is worth noting that we do not use any global state in the decentralized training and each agent learns on its own trajectory.
The results are shown in Figure 5.
On the map 3s_vs__ $s__v__4z,, vz_ IQL. and MA2QL both converge to the winning rate of 100%. However, on the hard and super hard map 5m vs_ 6m and corridor, MA2QL achieves stronger than IQL. Moreover, in these maps there are different numbers of agents, agents in $s_vs_4z, 5 agents in 5m__v_66____ vs_ 6m. and 6 agents in corridor, thus the results may also indicate the good scalability of MA2QL, though the difficulty of the map varies.
It is worth noting that the recent study (Papoudakis et al., 2021) shows that IQL performs well in SMAC, even close to CTDE methods like QMIX (Rashid et al., 2018). Here, we show that MA2QL can still outperform IQL. in three maps with various difficulties, which indicates that MA2QL can also tackle the non-stationarity problem and bring performance gain in more complex tasks.
5.5 STUDY ON HYPERPARAMETERS AND SCALABILITY
We further investigate the influence of the hyperparameters on MA2QL and the scalability of MA2QL. First, we study the effect of K (the number of Q-network updates at each turn) in the robotic control task: 3-agent Hopper. We consider K = [4000, 8000, 40000). As shown in Figure 6(a), when K is small, it outperforms IQL but still gets stuck in sub-optimal policies. On the contrary, if K is large, different K affects the efficiency of the learning, but not the final performance.
8

---

Under review as a conference paper at ICLR 2023
Hopper
simple_spread
HaffCheetah
MADOL 801
MAD0L 1855 00005
MA0L119
MADI 20 12, 290
(a) effect of K
(b) learning rate
(c) batch size
Figure 6: The effect of hyperparamettrs: (a) learning curves of MA2QL with different K in 3-agent Hopper, compared with IQL; (b) learning curve MA2QL compared with IQL with different learning rates in simple spread in MPE; (c) learning curve of MA2QL. compared with IQL with different batch sizes HalfCheetah in multi-agent MuloCo.
As MA2QL is essentially an add-on for IQL, the hyperparame- ters are the same for IQL and MA2QL.. In the experiments, the hyperparameters of IQL are taken from previous studies (Achiam, 2018; Hu al., 2021) for well-tuned performance, and do not tune the hyperparameters of IQL for MA2QL. To further study the effect of hyperparameters of IQL on MA2QL, we conduct additional experiments in simple spread with different learning rates and in HalfCheetah with different batch sizes. As shown in Figure 6(b) and Figure 6(c), under these different hyperparame- ters, the performance of IQL, and MA2QI. varies, but MA2QI. consistently outperforms IQL, which can be evidence of the gain of MA2QL over IQL is insensitive to the hyperparameters of IQL. The default learning rate in MPE is 0.0005 and the default batch size multi-agent MuJoCo is 100.
15 agents
MAZQL
ICL
Figure 7: Learning curve of MA2QL compared with IQL in 15-agent simple spread in MPE.
As for scalability, We demonstrate the result of 15-agent simple spread in MPE. As illustrated in Figure 7, MA2QL brings large performance gains over IQL. More agents mean the environments become more complex and unstable. IQI. is easy to get stuck by the non-stationarity problem while MA2QI. can handle it well. The results show again that alternate learning of MA2QI. can sufficiently stabilize the environment during the learning process, also indicates the good scalability of MA2QL.
CONCLUSION AND DISCUSSION
In the paper, we propose MA2QL, a simple yet effective value- fully decentralized cooperative MARI algorithm. MA2QL is theoretically grounded and requires minimal changes independent Q-larning. Empirically, we verify MA2QL. in a variety of cooperative multi-agent tasks, including a cooperative stochastic game, MPE, multi-agent MuJoco, and SMAC. The results show that, in spite of such minimal changes, MA2QL outperforms IQL in both discrete and continuous action spaces, fully and partially observable environments.
MA2QL makes minimal changes to IQL but indeed improves IQL. In practice, MA2QL can be easily realized by letting agents follow a pre-defined schedule for learning. MA2QL also has the convergence guarantee, yet is limited to Nash equilibrium in tabular cases. As a Dec-POMDP usually has many Nash equilibria, the converged performance of MA2QL may not optimal as shown the stochastic game. Nevertheless, learning the optimal joint policy in fully decentralized settings is still an open problem. In the stochastic game, we see that IQL also converges, though much slower than MA2QL. This indicates that IQL may also have the convergence guarantee under some conditions, which however is not well understood. We believe fully decentralized learning for cooperative MARL is an important and open research area. However, much less attention has been paid to decentralized learning than centralized training with decentralized execution. This work may provide some insights to further studies of decentralized learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent teams. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
Dimitri Bertsekas.
Multiagent value iteration algorithms in dynamic programming and reinforcement learning. Results Control and Optimization, 1: 1:100003, 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
Jelena Diakonikolas and Lorenzo Orecchia. Alternating randomized block coordinate descent. In International Conference on Machine Learning (ICML), 2018.
Xinyuan Fang, Tiankui Zhang, Yuanwei Liu, and Zhimin Zeng. Multi-agent cooperative alternating q-learning caching in d2d-enabled cellular networks. IEEE Global Communications Conference (GLOBECOM), 2019.
Mingxiao Feng. Guozi Liu, Li Zhao, Lei Song, Jiang Bian, Tao Qin, Wengang Zhou, Houqiang Li, and Yie-Yan Liu. Multi-agent reinforcement learning with shared resource in inventory management, 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the imple- mentation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.
Shariq Iqbal and Fei Sha. Actor-attention-critt for agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2019.
Wenhao Li, Bo Jin, Xiangfeng Wang, Junchi Yan, and Hongyuan Zha. F2a2: Flexible fully- decentralized approximate actor-critic for cooperative multi-agent reinforcement learning, arXiv preprint arXiv:2004. 1/ 1454 2020.
Xihan Li, Jia Zhang, Jiang Bian, Yunhai Tong, and Tin-Yan Liu. A cooperative multi-agent reinforce- ment learning framework for resource balancing in complex logistics network. In International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2019.
Yueheng Li, Guangming Xie, and Zongqing Lu. Difference advantage estimation for multi-agent policy gradients. In International Conference Machine Learning (ICML), 2022.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
Ryan Lowe, Yi Wu, Aviy Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi Multi-agent actor-critic for mixed cooperative-compettive environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Songtao Lu, Mingyi Hong. and Zhengdao Wang. PA-GD: On the convergence of perturbed alternating gradient descent to second-order stationary points for structured nonconvex optimization. In International Conference Machine Learning (ICML), 2019.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and Mhnite-horizonn partially observable markov decision problems. In AAAI/IAAI, 1999.
10

---

Under review as a conference paper at ICLR 2023
Laëtitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: algo- rithm for decentralized reinforcement learning in cooperative multi-agent teams. In /EEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2007.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arX1v:13/2.5602, 2013.
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In International Conference on Autonomous Agents and MultiAgent Systems, 2018.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, fierre-Alexanre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2019.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. International Conference Machine Leurning (ICML), 2019.
Kefan Su and Zongqing Lu. Eivergence-Regularize Multi-Agent Actor-Critic. In International Conference on Machine Learning (ICML), 2022.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Interna- tional Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, 2018.
Ardi Tampuu, Tambet Matüsen, Dorian Kodelja. Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint
2015.
Ming Tan. Multi-agent reinforcement learning: independent versus cooperative agents. In Interna- tional Conference on Machine Learning (ICML), 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint a3X5: 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2012.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-learnin. In International Conference on Learning Representations (ICLR), 202
Jianhong Wang, Wangkun Xu, Yunjic Gu, Wenbin Song, and Tim C Green. Multi-agent reinforce- ment learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems (NeurIPS), 2021b.

---

Under review as a conference paper at ICLR 2023
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Off Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations (ICLR), 2020.
Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. Hierarchically and cooperatively learning traffic signal control. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Haifeng Zhang, Weizhe Chen, Zeren Huang. Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang. Bi-level actor-critic for multi-agent coordination. AAAI Conference on Artificial Intelligence (AAAI), 2020.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Baar. Fully decentralized multi- agent reinforcement learning with networked agents. In International Conference on Machine Learning (ICML), 2018.
Kaiqing Zhang, Zhuoran Yang. and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint arXiv:1911.1063: 2019.
Tianhao Zhang, Yueheng Li, Chen Wang. Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropp multi-agent reinforcement learning. In International Conference Machine Learning (ICML), 2021.
12

---

Under review as a conference paper at ICLR 2023
A DETAILED PROOFS FOR THEORETICAL RESULTS
A.1 PROOF FOR LEMMA 2
Proof. From the definition of 0.*t (6), we have
|0 1 0ia1 y (maxa: 21 (5', d() maxa; P
(8', [1)6ll)
(8)
2 1. 11 S Q 0t |xx
Then for any integer m 2 1, we have
|| - -hth aoo0 - |1 - 2.ttm-1.1k1 +... 11. - -,1t 11
1 7mh
1 110, - -0%%
(9)
S
Let m + and we have
-
2 11:00 S
- Q
(10)
max (8, az) + 7B. pt (max;; (8' d])] O. (s, az)l
8,04
As all agents update by turns, have Q.tht Q for agent in the k round. Therefore, we have
where - is the number of Q-iteration
1 0 Q k-1 11 = =1 1--1 - -2 11:0 SE.
(11)
With this property, we have
(r, (s, a:) + PE [max,; 0% 0. (6', a4)] -
= |r (s, ax) yIEy m 0ºth 0.0 (5,, a/}]] Q, (s,a;) + 12 a.) S. (s,a;)
< |rt x -1 11 + 7| pt mmax Q. 0.k (8', %da) - E Ba [mxx Q k- (s', a(]l
- 1 4 (s, a.) - Q (s, a;)l s 2r max +
2 +e) + 2R 28,
(12)
1-7
where the second term in the last inequality is from 1|Q,,-=1l and (11). Finally, by combining (10) and (12), we have
1-7 11o0 S
Imax
110. 11 S 1|@, k-
t
72 (2R (2R+22) 22).
|1 - Q 1-1
(13)
We need ||Q,* - 2 1oc S e, which can be guaranteed by 2 log((1 (111--))))0)00 1-))))000 (og(2
log
A.2 PROOF FOR THEOREM 1
Proof. First, from Lemma 1, we know Q also induces a joint policy improvement, thus Q converges to Q Let m. (s) arg max,; Qi (s. az), then "²" is the joint policy of Nash equilibrium.
Then, we define A as
A min max (s, m3 (s)) (s.
(14)
5, a; (n)
From the assumption we know that A > O. We take = A and from Lemma 2, we know there exists kn such that
|0 01a011 S vk 2 ko.
(15)
13

---

Under review as a conference paper at ICLR 2023
For k Ky and any action a, * nz (s), we have
(8.T (s)
= - (10)0000000001000000000000000000
+ Q1((s, +2//55tth00000100000000000000020
(s, (16)
2
(s,
- (s, - 1 ((.
= - 4 4/3
which means .. = arg maxa, agent induces कं and all together induce πτ", which the joint policy of Nash equilibrium.
= arg max., (i. = (s). Thus, O of each
B
EXPERIMENTAL SETTINGS
B. MPE
The three tasks are built upon the origin MPE (Lowe et al., 2017) (MIT license) and Agarwal et al. (2020) (MIT license) with sparse reward, more agents, and larger space. These tasks more difficult than the original ones. In the original tasks, we found IQL and MA2QI. perform equally well, which corroborates the good performance of IQL on simple MPE tasks (Papoudakis al., 2021). Specifically, we change the dense reward setting to the sparse reward setting that will be described below. We increase the number of agents from
in the task of simple spread and enlarge the range of initial positions from [-1, 1] to [-2, 2]. A larger space would require the agents to explore more and the agents would be easier to get stuck in sub-optimum. As there no distance signal in the sparse reward setting, we add a boundary penalty into the environment. If the agents move outside the boundaries, the environment would return the negative reward of -1. The boundaries in our experiment are [-3, 3].
Simple spread. this task, the agents need find some landmarks in a continuous space. In the sparse reward setting, only if the agents cover the landmarks, the environment would return the positive reward. There is no reward after the landmarks are covered. Whether the landmark is covered by the agent is based on the distance between the landmark and the agent. We set threshold of 0.3 in our experiment. If the distance is smaller than the threshold, we consider the landmark covered by the agent. We set 5 agents and landmarks in this task. Fewer landmarks require the agents to do more exploration in the space to find the landmarks.
Line control. In this task, there 5 agents in the space. The goal of the task is that the agents need to arrange themselves a straight line. The reward depends on the number of agents in the straight line. If the straight line is formed by the agents of 5, the environment would return the reward of 5, Since the action space is discrete, we set a threshold of 15t to judge whether it is a line.
Circle control. There are agents and I landmark in the space. This task is similar the task of line control, while the agents are asked to form circle where the landmark is the center. We also set threshold of 0.15.
B.2 MULTI-AGENT MUJoCo
Multi-agent MuloCo (Peng et al., 2021) (Apache-2.0 license) is built upon the single-agent MuJoCo (Todorov al., 2012). A robot can be considered as the combination of joints and body segments. These joints and body segments in robot can be regarded as the vertices and connected edges in graph. We can divide graph into several disjoint sub-graphs and sub-graph can considered as an agent. We can design new observation and action spaces for each agent based on the division. For example, the robot of Walker2d has 6 joints in total. We can divide joints and body segments into left parts and right parts two agents.
The details about our experiment settings in multi-agen Mujoco are listed in Table 1. The configura- tion defines the number of agents and the joints of each agent. The "agent obsk" defines the number of nearest agents an agent can observe. Thus, HalfCheetah and Hopper are fully observable, while Walker2D is partially observable.
14

---

Under review as a conference paper at ICLR 2023
Table 1: The environment settings of multi-agent MuJoCo
task
configuration
agent obsk
HalfCheetah
2
1
Hopper
3 X 1
2
Walker2d
3 x
1
C TRAINING DETAILS
C.I MPE
We use DRQN with an RNN layer and MLP as the architecture of Q-network. The RNN layer composed of a GRU with the 64-dimension hidden state. We utilize ReLU non-linearities. All networks are trained with a batch size of 128 and Adam optimizer with the learning rate of 0.0005. There 16 parallel environments that collect transitions and the replay buffer with the size of 5000 contains trajectories. For exploration, we use e-greedy policy and starts from 1.0 and decreases linearly 0.1 in the first 50000 timesteps.
For MA2QL, we define K is the training steps for updating an agent at each turn. For instance, is the turn update the agent i. After training K steps, it would change the turn to update the agent i + 1. We set K as 30 in this experiment.
C.2 MULTI-AGENT MuJoCo
We use DDPG for continuous action space and the setting is similar to the one used in OpenAl Spinning Up (Achiam, 2018) (MIT license). The architecture both actor network and critic network is MLP with two hidden layers with 256-dimension hidden state. We utilize ReLU non-linearitie except the final output layer of the actor network. The activation function of the final output layer of the actor network is tanh function that can bound the actions. All networks are trained with a batch size 100 and Adam optimizer with the learning rate of 0.001. There is only one environment that collects transitions and the replay buffer contains the most recent 10 transitions. For exploration, the uncorrelated, meeanzeere Gaussian noise with o = 0.1 is used. For MA2QL, we set K as 8000. Unlike other two environments where agents learn on trajectories, in multi-agent MuJoCo, agents learns on experiences. Thus, in each turn, the agent can updated every environment step and hence K is set to be large.
C.3 SMAC
Our code is based the Hu et al. (2021) (Apache-2.0 license) and the version of SMAC is SC2.4. The architecture Q-network is DRQN with an RNN layer and MLP. The RNN layer is composed of GRU with the 64-dimension hidden state. All networks trained with batch size of 32 and Adam optimizer with the learning rate of 0.0005. All target networks are updated every 200 training steps. There are parallel environments that collect transitions and the replay buffer with the size of 5000 contains trajectories, For exploration, we use e-greedy policy and starts from 1.0 and decreases linearly to 0.05 in the first 100000 timesteps. We add the action of last timestep into the observation as the input to the network. For MA2QL, we set K as 50.
C.4 RESOURCES
We perform the whole experiment with total of ten Tesla V100 GPUs.
D ADDITIONAL RESULTS
D.1 ADDITIONAL SMAC RESULTS
We also demonstrate the results on more maps in SMAC. As shown Figure 8, IQL shows strong performance in these maps, which corroborates the good performance of IQL in SMAC (Papoudakis
15

---

Under review as a conference paper at ICLR 2023
2c 64zg
3s5z
8m
Win Palt
Witn Rate
%
MA2QL
stops
5tops
stops
Figure 8: Learning curve of MA2QL compared with IQL on 2c_va_64zg (hard), 3s5v (hard) and 8m (easy) in SMAC, where x-axis is environment steps.
HalfCheetah
Hopper
Walker2d
2010
Epkocile Phass
Eppende Reww
808
800
880
MA2QL
IQL
IPPO
50000
stops
stpps
Learning curve of MA2QL compared with IQL and IPPO in 2-agent HalfCheetah, 3-agent Hopper and 3-agent Walker2d in multi-agent MuloCo, where x-axis is environment steps.
Figure 9:
et al., 2021). MA2QL performs similarly to IQL on the maps of 3s5v and 8m, but better on the map 2c vs 64zg. Together with the results in Section 5.4, can that although IQL performs well in SMAC, MA2QL can still bring performance gain in more difficult maps, which again verifies the superiority of MA2QL over IQI..
D.2 ADDITIONAL. COMPARISON WITH IPPO IN MULTI-AGENT MUJoCo
As MA2QL derived from Q-learning and thus a value-based method, we focus on the comparison with IQL. Here provide the results additionally compared with actor-critic method, i.e., independent PPO (IPPO) (de Witt et al., 2020), the experiment of multi-agent MuJoCo. The setting and the architecture of IPPO are also similar to the used in OpenAl Spinning Up (Achiam, 2018) (MIT license). As illustrated in Figure 9, MA2QL also consistently outperforms IPPO.
D.3 ALTERNATE UPDATE OF IPPO
The principle of alternate update can also be applied to independent actor-critic methods. Here we additionally provide the empirical results of alternate update of IPPO (termed as MA2PPO) in the cooperative stochastic game and multi-agent MujoCo. For the cooperative stochastic game, as illustrated in Figure 10, MA2PPO substantially outperforms IPPO. For multi-agent MujoCo, as shown in Figure 12, MA2PPO brings performance gain in HalfCheetah and performs comparably with IPPO in Hopper and Walker2D. These results may suggest that such a simple principle can also be applied independent actor-critic methods for better performance. A more thorough investigation on independent actor-critic methods is beyond the scope of this paper and left as future work.
D.4 ALTERNATE UPDATE CTDE
Though the principle of alternate update still be applied CTDE methods, there may not be many benefits. In the CTDE setting, there are actually a number of algorithms that have convergence guarantee. Without the advantage of convergence, alterntee update will actually make the learning
16

---

Under review as a conference paper at ICLR 2023
Stochastic Game
150
mean epiiode rewards
mean opicode rowards
15
1.00
10
50
OPTIMAL
MATPRO-CTDE
IPPO
MAPPO
0
100
2000
300
400
500
0.4
1.0
step
stop
1c6
Figure 10: Learning curves of MA2PPO com- pared with IPPO in the cooperative stochastic game.
Figure 11: Learning curves of MA2PPO- CTDE compared with MAPPO on the map 3m in SMAC.
HalfCheetah
Hopper
Walker2d
Eginada Resunns
Eplcoda Rewerdte
200
200
MA2PPO
IPPO
22
staps
staps
Figure 12: Learning curves of MA2PPO compared with IPPO 2-agent HalfCheetah, 3-agent Hopper and 3-agent Walker2d in multi-agent MuJoCo, where x-axis is environment steps.
procedure slower. We modified the CTDE algorithm MAPPO (Yu et al., 2021) with alternate update (termed as MAZPPO-CTDE, and compare with MAPPO the simple map 3m in SMAC. The empirical result is illustrated in Figure 11. We can find that MA2PPO-CTDE obtains low performance while MAPPO has the performance which is close to the optimum. Though alternate update may become better after a longer training, alternate update is not quite meaningful in CTDE.
E ADDITIONAL RELATED WORK
Our method shares the same principle of alternate update with existing studies for optimization, e.g. alternating gradient descent (Lu et al., 2019) and coordinate gradient descent (Diakonikolas & Orec- chia, 2018). There is also existing work (Fang al., 2019; Bertsekas, 2020) investigating alternate update for multi-agent reinforcement learning, which however is only limited to the theoretical result of Lemma in Section 2.
F ADDITIONAL EXPERIMENTS FOR REBUTTAL
We carried out additional experiments in the didactic game, including different stochasticity levels, exploration schemes, and update orders.
First, we control the stochasticity level of the environment by introducing p_trans. For any transition, a state has the probability of p_lrans to transition to next state uniformly random, otherwise follows the original transition probability. Thus, larger p_ trans means higher level of stochasticity. As illustrated in Figure 13, MA2QL outperforms IQL all stochasticity levels.
We also investigate the performance of MA2QL under different exploration schemes, including different decaying rates and constant e. As illustrated in Figure 14, MA2QL again outperforms IQL in all these exploration schemes. Note that decay 100K means decays from to 0.02 over 100K environment steps, similarly for others. These results shows the robustness of MA2QL under various exploration schemes, even when 1.
17

---

Under review as a conference paper at ICLR 2023
150
150
150
150
300
100
100
100
50-
OPTIMAL
MAZGI-_F_ P.tee--_
0
12L, tramsm)
IQL. p transm0.22
IQL, transm0.5
IGL, j. transmDa
1.0
200
300
400
500
100
200
300
400
500
100
200
300
400
500
0 100
200
300
400
500
Figure 13: Learning curves of MA2QL compared with IQL in the didactic game under different stochasticity levels, where p trans is the probability of a state transitions to next state uniformly at random and hence larger p_ trans means higher level of stochasticity.
Stechatl: Care
Sunchalls
Sosthalk Come
Stechollt Gare
Stachellec Sume
1.5
150
350
150
OFTINAL
OFTIMAA
OFTIVAAL
OITIMAA
OFTESAL
MASC. deaay
HA22L docey, 33BK
MAZCL SRow,
HA20, coea,
100
200 300
463
506
100
260
40
220
200
350
400
500
120
200
300
408
500
200
320
400
580
siop
Rap
st0p
Figure 14: Learning curves of MA2QL compared with IQL in the didactic game under different exploration schemes, where d0oKy_100K means decays from 0.02 over 100K environment steps, similarly for others.
We further investigate the performance of MA2QL under different pre-defined update orders and random order at each round. As there are three agents in the environment, there are essentially two different pre-defined orders. Suppose that three agents
indexed from 0 to 2. The alternating update orders [0, 1,2, [1,2,0] and 12 are the same, while [0,2, [1,0,2], and [2,1,0] are the same. As illustrated in Figure 15, the performance of MA2QL is almost the same under the two orders (MA2QL is the order of [0.1.2]), which shows the robustness of MA2QL under different pre-defined orders. As for random order at each round, the performance drops but still better than IQL. One possible reason is that agents are not evenly updated due to the random order at each round, which may consequently induce a worse Nash equilibrium. However, pre-defined order better and more practical than random order that requires all agents reach an agreement of who should update at each round.
Stochastic Game
miaan aptode riviarie
100
OPTIMAL MAZQL
50-
IQL
0-
MAZQL_ MAAZQ_ffxe__one order
MAZQL_ rocoss__anommm__m order
0
100
200
300
400
500
step
Figure 15: Learning curves of MA2QL of different update orders. There are three agents in the didactic game, so there are essentially two different pre-defined update orders: [0,1,2] (MA2QL) and [0,2,1]
We also carried out the experiments in matrix game, of which the payoff matrix is illustrated in Figure 16(a). The learning curves of MA2QL, and IQL. are shown in Figure 16(b), where left is linearly decayed and right is e 1. MA2QL and MA2QL with action AA while IQL learn the suboptimal joint action CC. For both MA2QL and MA2QL with
both learn the optimal joint
= 1, the learned Q-table of two agents are the same, which is [11, 330, 0] for action [A, B, CJ respectively. For IQL, the Q-table of 01 is 2.2, 1.9, 6.9], and the Q-table of az is [-2.1 -2.7, 6.9]. For IQL with 1, the Q-table of a1 is 6-3.3 5.66, 2.3]. the Q-table of og is [-6.3, -7.7, 4.3]. Although the learned joint action in both cases is CC, the value estimate CC is less accurate for IQL with
1. Meanwhile, MA2QI. has the same accurate value estimate regardless of exploration,
18

---

Under review as a conference paper at ICLR 2023
11
A
B
C
Q1
20
30-
30
9-
B
30
7
6
C
101
[Q)_<=]
100
200
300
400
500
stop
stop
(a) payoff matrix
(b) learning curves
Figure 16: A matrix game: (a) payoff matrix; (b) learning curves of MA2QL and IQL with linearly decayed (left) and 1 (right).
MA2QL IQ
MARQI
MA2CL
Easods Ooaard
Coloude Roorr
Splicde Rowrd
5-17
2x4 Swimmer
(b) 6|2 Ant
17 1 Humanoid
Figure 17: Learning curve of MA2QL compared with IQL in 2-agent Swimmer, 6-agent Ant, and 17-agent Humanoid in multi-agent MuloCo, where x-axis is environment steps.
which corroborates the good performance of MA2QL under different exploration schemes in the didactic game.
Moreover, we carried out additional experiments in multi-agent MuJoCo, including 2 Swimmer (agent_obsker 6/2 Ant (agent_ (agent_obsk=O)), and 17 x Humanoid (full observation). The results are shown in Figure 17, and MA2QI also outperforms IQL. in these tasks.
19


Reviews:
[{"review_id": "od3_hhgUOPO", "reviewer": "Reviewer_p9BU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes multi-agent alternate Q-learning (MA2QL) to tackle the non-stationarity problem in decentralized MARL. The main difference between MA2QL and existing method IQL is that MA2QL agents take turns to update their Q-functions while IQL agents simultaneously update their Q-functions. The authors prove that, in MA2QL, when each agent guarantees $\\epsilon$-convergence at each turn, their joint policy converges to a Nash equilibrium. They empirically show that MA2QL outperforms IQL in a variety of cooperative multi-agent tasks with discrete and continuous action spaces. ", "summary_of_the_review": "My biggest concern of this paper is the specific \"fully decentralized\" multi-agent setting considered (where parameter sharing is not allowed) is not well motivated and clearly explained. It is not clear to me what the practical use cases for this setting are. I can think of some real-world scenarios where there are separate machines/robots who are not able to/not allowed to share parameters with each other. But in MA2QL, you seem to be implicitly assuming some kind of synchronization mechanism that enables the agents to reach an agreement on who should update their policy/Q-function at each learning stage. In that sense, is it really \"fully decentralized\" based on your definition?\n\nMy another main concern is the idea introduced in MA2QL is not really new as I've mentioned above. In addition, for the experiments, while I appreciate the evaluation of MA2QL on a variety of cooperative multi-agent tasks with discrete and continuous action spaces (and full/partial observability), I think comparing only against IQL is not enough. What about other independent learning baselines such as independent actor-critic, independent DDPG, and independent PPO? For instance, does MA2QL outperform independent PPO in SMAC? \n\nHere are some other comments/questions:\n- \"we do not use parameter-sharing, which should not be allowed in decentralized settings (Terry et al., 2020).\" I do not understand why parameter sharing \"should not be allowed in decentralized settings.\" \n- In Multi-Agent MuJoCo, using names MA2QL and IQL for the algorithms while DDPG is actually used are misleading. \n- In MA2QL, how do you decide the order in which each agent updates their Q-functions? Is it a random order? Do agents need to fully agree on who should update their Q-functions next?\n- For the hyperparameter K used in MA2QL, it is set to be 30 in MPE but 8000 in Multi-Agent MuJoCo. The range of values looks quite large. How do you efficiently tune this hyperparameter? \n\n[1] Lanctot, Marc, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems, 2017.", "strength_and_weaknesses": "- Strengths:\n\t- The main approach proposed is theoretically grounded, with some good experimental results.\n\t- The paper is well-written and easy to follow overall.\n- Weaknesses:\n\t- The specific \"fully decentralized\" multi-agent setting considered (where parameter sharing is not allowed) in this work is neither well motivated nor clearly explained, which is critical for justifying the significance of this work.\n\t- The main idea of  the proposed algorithm (i.e., \"fixing the policies of other agents while one agent is learning\") is not new. For instance, it has been used in existing works in competitive MARL [1].\n\t- The experiments are not sufficiently comprehensive to me (e.g., MA2QL is only compared against IQL).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SYJrumAztwO", "reviewer": "Reviewer_J8oo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors consider a fully decentralized cooperative multi-agent reinforcement learning (MARL). To alleviate the non-stationarity in decentralized MARL, this paper proposes to learn agents by letting each agent take turns updating each action-value function. Based on this simple idea, multi-agent alternate Q-learning (MA2QL) is proposed for practical implementation, and a theoretical explanation is described. Numerical results show that MA2QL can perform better than independent Q-learning in the considered cooperative MARL environments under fully decentralized settings.", "summary_of_the_review": "It is crucial to study new learning methods in fully decentralized multi-agent reinforcement learning. However, the technical novelty of the newly developed algorithm remains questionable, and there are many unclear aspects of the core idea of alternate learning. Discussions and experiments should be expanded. Thus, a major revision is necessary both theoretically and experimentally.", "strength_and_weaknesses": "While this paper is well-organized, easy to follow, and theoretically supported, there is a fundamental debate on alternate learning concepts which should be clearly explained. In addition, there is a gap between theory and practice which is not properly dealt with. Experiments should be expanded for further analysis, and finally, reproducibility cannot be verified.\n\nQ1. Fundamental discussions on alternate learning \n\nThe core concept of this paper is alternate learning. There are substantial issues regarding the alternate learning concept.\n- Imagine we consider an extremely large distributed system such as decentralized traffic control. Training each agent alternatingly requires an extreme amount of time and each agent may not adequately take action at each moment.\n- If $n$ is extremely big, each agent is not fully trained because the period is long. In fact, in the wall-clock time sense, IQL can conduct more gradient steps within the same wall-clock time to improve performance. In this sense, “a fair comparison” in this paper may be questionable.\n- This paper does not explicitly describe the effect of agent order. Can the order be random at each round, or the order should be predefined? Can the authors show any numerical study showing whether order matters or not? If $n$ is extremely large, there can be one good agent and the other bad agents, which can cause asymmetric bias. Then it can be sensitive to the order.\n- Though controversial, defining agent order may not correspond to fully decentralized learning (unless every agent agrees to predefined order “protocol” before starting training) since each agent should know whether a certain turn is “my turn” or “other’s turn.” \n- Can MA2QL apply to non-cooperative MARL settings? (i.e., Markov Game)\n\nQ2. Unclear description of ‘fair comparison’ between MA2QL and IQL\n\n- In Multi-agent Mujoco, $K=8000$. In 2-agent HalfCheetah, for example, for $2K=16000$ environment steps, each agent is trained with MA2QL. Then how IQL is fairly trained? Are the two agents simultaneously trained every 2 environment steps during $2K=16000$ environment steps?\n- I am also confused with MPE and SMAC. At agent $i$'s turn, each agent $j$ stores the $m$ transition samples ($m$ environment steps), and agent $i$ conducts $K$ training steps. In total, each of $n$ agents conducts $K$ training steps for $mn$ environment steps. \nThen how IQL is fairly trained? I think IQL agents simultaneously update Q-function one step (i.e., one gradient step) every $\\frac{mn}{K}$ environmental step so that during $mn$ environment steps, each of $n$ agents conducts $K$ training steps.\nIf my understanding is correct, what is the value of $m$ in MPE and SMAC? What is the effect of the value of $m$? Is IQL properly tuned in this setting?\n - Clarification with pictorial examples can help to understand.\n\n\nQ3. Regarding sampling issues in learning Q-network \n\nAs stated on page 5, when learning Q-network, MA2QL permits containing samples from past transitions in a single replay memory. The authors did not provide any remedy reducing the gap between theoretical on-policy learning and single replay memory usage. It would be better to try any remedy to reduce the logical gap.\n\nQ4. Multiple Nash equilibria\n\nThe main claim relies on the assumption that there is a unique Nash equilibrium, and the authors described a short remedy dealing with multiple Nash equilibria on page 4. Can we verify the remedy experimentally? For example, we may construct a matrix game with multiple Nash equilibria to check whether the proposed method can be extended in practical cases. \n\n\nQ5. Comparison with hysteretic Q-learning\n\nThe authors need to compare MA2QL with hysteretic Q-learning since hysteretic Q-learning is pre-existing baseline in fully decentralized learning. The considered environments by **Zhang et al. (2020) are different from those in this paper. \n** Zhang et al., Bi-Level Actor-Critic for Multi-Agent Coordination, AAAI 20.\n\n\nQ6. Environments\n- Can MA2QL be effective with Multi-agent Mujoco when agent_obsk=0? \n- Please explicitly write whether each environment is partially observable or not in each title in Figure 4 for readers.\n- In SMAC, is there any case where IQL fails but only MA2QL succeeds?\n\nQ7. Reproducibility\n\nThe authors did not provide their implementation codes for verification.\n\n\nNotation: $\\gamma$ is missing in the third paragraph of Section 5.2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KGbpL8RCKg", "reviewer": "Reviewer_SmxM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple modification to Independent Q-Learning to make it better address the non-stationarity issue in credit assignment. The modification involves allowing agents to take turns in updating their individual Q-tables or Q-functions. This is as opposed to synchronous updates in IQL implementations. This paper motivates this approach through theory (in the tabular value iteration and Q-learning) and experiments (across several standard benchmarks).    ", "summary_of_the_review": "I generally like the paper, but I have several concerns that I hope the authors can address in their rebuttal. ", "strength_and_weaknesses": "**Strengths:**\n\n- The idea is simple and elegant.\n\n- The idea is more realistic than standard IQL, in which updates have to be synchronized across agents. Their approach MA2QL, on the other hand, does not require synchrony and in fact is closer to what would happen in a practical physical multiagent system, where agents will receive updates at semi-fixed intervals of time without a synchronized starting time.\n\n- Improving Independent Learning is much harder than CTDE, and advances in this direction could prove more useful in real-world learning scenarios where no good simulator is available or communication during training is strictly limited.   \n\n\n**Weaknesses:**\n\n- While the paper is generally well-written, I struggled to obtain answers to many questions: partly due to the way things are explained/described, and partly due to missing analysis/experiments. Please refer to my questions in the next section to help clarify some of these concerns.  \n\n- Novelty might be limited with some related work in the context of MARL focusing on the same concept (see Appendix E for some such connections discussed by the authors, but not in great detail). I hope the authors can discuss this more.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I6Wo84pOnB", "reviewer": "Reviewer_xmYF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a modification to IQL called MA2QL in which each agent alternately updates the parameters rather than the simultaneous update. The main contribution of this paper is the theoretical analysis of this modification. In experiments, MA2QL is demonstrated to outperform IQL in several environments.", "summary_of_the_review": "This paper proposes a simple modification of IQL and give a complete theoretical analysis to show the convergence property. However, the similar theoretical results have appeared in the past several times, so the novelty of this paper faces a challenge. From the practical side, the proposed alternative udpate will lead to n times interactions with the environment which is a huge cost as the number of agents increases.", "strength_and_weaknesses": "## Strength\n1. The motivation of this paper is clear and the discussion of the related works are sufficient.\n2. The theoretical analysis of this paper is easy to follow and correct.\n3. The experiment design is comparatively good which covers variants of scenarios.\n4. The proposed algorithm is neat.\n\n## Weaknesses\n1. Simultaneous move (update) and iterative move (update) are two typical paradigms existing in both game theory and multi-agent learning. Iterative move (update), e.g. the original version of Brown's fictitious play, has been proved to converge to Nash equilirbium in potential game. The team reward game considered in this paper is a special case of the potential game. Even equipped with Markov dynamics, there is also some existing work to show the same idea as the paper proposes [1]. In addition, in the recent paper [2] the alternate update (or rollout in sequence) has also been discussed. Therefore, the theoretical contribution of this paper is minor in my view. From the multi-agent learning side, there is a past paper [3] that also shares the similar idea.\n2. As for the practical aspect, the proposed MA2QL will increase the times of the interaction with the environment to n when there are n agents. It will lead to extra cost as the number of agents increases. It is meaningless to practice in my view, unless it can exceed the state-of-the-art methods largely.\n\n\n## Reference\n[1] Hu, Ruimeng. \"Deep fictitious play for stochastic differential games.\" arXiv preprint arXiv:1903.09376 (2019).\n\n[2] Bertsekas, Dimitri. \"Multiagent reinforcement learning: Rollout and policy iteration.\" IEEE/CAA Journal of Automatica Sinica 8.2 (2021): 249-272.\n\n[3] Lauer, Martin, and Martin Riedmiller. \"An algorithm for distributed reinforcement learning in cooperative multi-agent systems.\" In Proceedings of the Seventeenth International Conference on Machine Learning. 2000.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject