I2Q: A Fully Decentralized Q-Learning Algorithm
Authors: Jiechuan Jiang
Zongqing Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-agent reinforcement learning

Abstract: Fully decentralized multi-agent reinforcement learning has shown great potentials for many real-world cooperative tasks, where the global information, \textit{e.g.}, the actions of other agents, is not accessible. Although independent Q-learning is widely used for decentralized training, the transition probabilities are non-stationary since other agents are updating policies simultaneously, which leads to non-guaranteed convergence of independent Q-learning. To deal with non-stationarity, we first introduce stationary ideal transition probabilities, on which independent Q-learning could converge to the global optimum. Further, we propose a fully decentralized method, I2Q, which performs independent Q-learning on the modeled ideal transition function to reach the global optimum. The modeling of ideal transition function in I2Q is fully decentralized and independent from the learned policies of other agents, helping I2Q be free from non-stationarity and learn the optimal policy. Empirically, we show that I2Q can achieve remarkable improvement in a variety of cooperative multi-agent tasks.

I2Q: A Fully Decentralized Q-Learning Algorithm
Jiechuan Jiang
Zongqing Lu School of Computer Science Peking University zongqing lu@pku edu. en
School of Computer Science
Peking University
j jiechuan. jinchuannjaan@ppuuu jiang@pku. edu.
Abstract
Fully decentralized multi-agent reinforcement learning has shown great potential for many real-world cooperative tasks, where the global information, e.g., the ac- tions of other agents, used for decentralized training, the transition probabilities are non-stationary since other agents are updating policies simultaneously. which leads non non-guaranteed convergence of independent Q-learning. To deal with non-stationarity. we first introduce stationary ideal transition probabilities, on which independent Q-learning could converge to the global optimum. Further, we propose a fully decentralized method, I2Q, which performs independent Q-learning on the modeled ideal tran- sition function to reach the global optimum. The modeling of ideal transition function in 12Q is fully decentralized and independent from the learned policies of other agents, helping I2Q be free from non-stationarity and learn the optimal policy. Empirically, we show that I2Q can achieve remarkable improvement in a variety of cooperative multi-agen tasks.
accessible. Although independent Q-learning widely
Introduction
Multi-agent reinforcement learning (MARL) has shown great potential in real-world applications, including UAV [20], IoT [3], and games [29]. A number of MARL methods have been proposed for training agents cooperatively maximize the cumulative shared reward, most of which follow the paradigm of centralized training and decentralized execution (CTDE), where the information of all agents is collected and used in the training phase. However, in many industrial applications where agents may belong to different companies, e.g., autonomous vehicles or robots, the actions of other agents may be accessible, so that CTDE methods cannot work. One way address this challenge is fully decentralized learning, where the agents only use local experiences without the actions of other agents in both training and execution.
Independent Q-Jearning [28, 27] one of the most straightforward decentralized methods. However, since other agents treated part of the environment, from the perspective of individual agent, the transition probabilities that also depend on the policies of other agents will change as other agents updating their policies simultaneously [7]. Whether the agent updates the individual Q-values using the off-policy experiences, which are stored in the decentralized replay buffer, or the on-policy experiences, which are collected under the latest learned policies of all agents, the transition probabilities are non-stationary, thus the convergence of independent Q-learning is not theoretically guaranteed.
To tackle the non setsstttionartt problem, we propose a novel variant of independent Q-learning. First, we introduce ideal transition probabilities for each agent, which is induced by the optimal conditional joint policy of other agents, conditioned on action of this agent. We theoretically show that if all agents independently perform Q-learning on respective ideal transition probabilities, their policies
"orresppondigg Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

converge to the optimal joint policy when there is only one optimal joint policy, Certainly, such ideal transition probabilities are unknown in advance, but they can be deliberately learned by each agent in decentralized way. We let each agent learn the QSS-value [6], the value of state and next state, which will converge on its replay buffer and be equivalent with the optimal joint Q-value in e
deterministic environments. Then, for each agent, the next state under ideal transition probabilities, given a state and an action, is the one with the highest an ideal transition function by learning QSS value and perform independent Q-learning on the ideal transition function, which guarantees the convergence to the optimal policy.
Therefore, each agent can model
The proposed method, ideal independent Q-learning (I2Q), is fully decentralized, without the information of other agents. Although the theoretical proof built on deterministic environments. 12Q can also be applied in stochastic environments. We theoretically analyze the value gap and experimentally demonstrate its effectiveness in stochastic environments. We evaluate I2Q on variety of multi-agent cooperative tasks, i.e., matrix games, MPEE-basd differential games [15], Multi-Agent MuloCo [5], and SMAC [23], covering fully and partially observable, deterministic and stochastic, discrete and continuous environments. Empirically, 12Q outperforms baselines, verifying the analyzed convergence and optimality, and the modeled ideal transition function.
Related Work
CTDE. Most recent MARL methods follow the paradigm of centralized training and decentralized execution (CTDE). Policy-based methods [15, 8, 32, 37, 25, 18] extend policy-gradient into multi- agent cases by designing different optimization objectives. Value factorization methods [26, 22, 24, 31, 21] decompose the joint value function into individual value functions according to the mndviddual-GlobalMMas condition. In these methods, the information of all agents can be accessed in a centralized way during training, and the convergence can be guaranteed. Unlike these methods, we focus on decentralized learning where global information is not available.
DTDE. When agents cannot obtain global information, they have to adopt decentralized training and decentralized execution (DTDE). The most straightforward decentralized methods are independent Q- learning [27] and independent PPO [4]. However, since all agents are updating policies simultaneously. from the perspective of each agent, the transition probabilities are non-stationary, and thus the convergence of these methods may not be guaranteed. Fingerprints [7] uses iteration number and exploration rate to alleviate the problem of obsolete experiences in replay buffer for independent Q-learning. Hysteretic-QU [16] and Lenient-QL. [17] let the agents be optimistic and attach less importance the value punishment. However, none of these variants guarantees the convergence of independent Q-learning- Some theoretical methods [2, 38) guarantee the convergence to a Nash cquilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria [35]. Moreover, these methods cannot use replay buffer and have to re-collect experiences once the policies are updated, which is not practical. Our I2Q practical method learn the optimal policy using replay buffer. Some studies [36, 11] consider decentralized learning with communication. However, our fully decentralized settings, agents cannot share any information.
3 Method
We first introduce the problem setting and analyze the non-stationarity in independent Q-learning. Then, we construct ideal transition probabilities, on which the agents are theoretically guaranteed to converge the optimal joint policy via independent Q-learning. Finally, we propose I2Q, which performs independent Q-learning on the modeled ideal transition function.
3.1 Preliminaries
There are N agents in multi-agen MDP Menv =< S, R, Penve > with the state space S and the joint action space A. each timestep, each agent performs an individual action a; and the environment transitions to the next state s' by taking the joint action a with the transition probabilities Panv (s') fs, a). For simplicity of theoretical analysis, we let all agents obtain the state s [13], though in practice each agent i may learn based on the observation O; E O. All agents get shared reward r == R R(, 8') and learn to maximize the expected return E 7 prr,, where 7 is the discount factor.
We consider the fully decentralized learning, where Menv is partially observable to each agent since

---

each agent only observes its own action instead of the joint action a. From the perspective of each agent there is an MDP M, =< A, P:: > with the individual action space A, and the transition probabilities
P.
= = (s'1s, (a-(a_)
(1)
where a-; and πh, respectively denote the joint action and the joint policy of all agents except agent i. Each agent performs independent Q-learning:
Qi(s, 0.) (s((ss)))
[-- +
(2)
In this paper, all Q values denote the optimal values Q", we refer Q" as Q for simplicity. According to (I), the transition probabilities depend on the policies of other agents. Since other agents are updating their policies continuously, P. becomes -on-stttionayy during training. Concretely, if each agent i updates Q; in an off off-policy way, using the experiences stored in the replay buffer D; which does not contain other agents' actions, the transition probabilities in the replay buffer could be seen as the ones depending on the average policy Æ- along the training process, which is non-stationary and outdated [7]. If each agent learns in an on-policy way, only using the most recent experiences, P: could be seen as the one under the latest learned policy -h which is not outdated but still non- stationary. The convergence of Q-learning on non-stationary transition probabilities not guaranteed. How solve the non-stationn non-stationarity? In the next section, we introduce ideal transition probabilities and prove that if all agents perform independent Q-learning on the ideal transition probabilities, they will converge to the optimal joint policy.
3.2 Ideal Transition Probabilities
Assume there is only one optimal joint policy T T*()) = arg max, (s,, a), where Q the optimal joint Q-function,
(ss, a) = PEr (e))s.a)
m a')
a'
The unique optimal policy π (s) must be deterministic [19]. Thus. n² can be uniquely factorized into deterministic optimal individual policies π" = (m ... ... . -Ty) For agent i, we build a deterministic optimal joint policy of other agents which is conditioned a;:
π (s, a;) arg maxQ(ax as,a; a_.
(4)
There may exist multiple Ti... and we can choose one arbitrarily. If other agents act the policy (s, a;), the transition probabilities viewed by agent i is P; (s' (s, oz) Ponv (s'|s, az, (8)
which are defined as ideal transition probabilities. Then we have the following theorem.
Theorem 1. If agent performs Q-learning on the ideal transition probabilities P, (s'1s,a,) = Penv (5'18, 0i, T- (8, ax)), its policy will converge to the optimal individual policy x/.
Proof. According to the fixed-point formulation of Q-learning, have
Qi(s, = E. (111010, r 76) of
(5)
maxQ(s,a;.d 0;, a.;) max E Ep,,, (P1sa) a_
r ymax maxQ a's o': of a";
(6)
a_
E = (s11s,a,1 r ymax maxQ(s, d/ a'i) of
where (6) is from taking maxa_, on both sides of (3), and (7) is by folding max... into Penv From (5) and (7), we see Qi(s, a;) and max._, Q(s, a, a_;) have the same stationary Bellman operator. According to the contraction mapping theorem, the Bellman operator converges to unique fixed point. Thus,
Qi(8. = maxQ(s, 0j, a.;).
a_
3

---

Then, we have:
(axQ)(s.a.) de
max maxQ ai a, Q(s TT (s)).
(8)
a, a-
By contradiction, if a # (s) but satisfies Qi a.) 2 (s( dicts that π is the unique optimal joint policy. Therefore, (5( = arg max Qi a,).
Q(s, TT it contra-
n
By extending this theorem, if all agents perform Q-learning on such ideal transition probabilities under the conditional optimal joint policy, they will converge to the respective optimal individual policies, and arrive at the optimal joint policy (Ti m2. ,T)
3.3
12Q
How to obtain the ideal transition probabiliries from the non-stationary replay buffer for each agent? We first introduce the QSS- USS-leerning [6]. Following the setting of QSS [6], assume the environment is deterministic." Each agent i learns a value function Q (s, s') using the experiences in its own replay buffer D;:
Q (s, =r+-
max T (s', EMEN((s)
where N (s') is the neighboring state set of the state (the set of all next states of s'). The value function QT has several advantages. First, it has been proven in QSS [6] that Q and Q learn equivalent values, which means
mf maxQ(s,a).
(10)
The equivalence shows that agent can independently infer the next state under the optimal joint policy according to arg max, (s, Second, as Q (8,8') decouples the action a; from the value and the implied transition probabilities: P(s'|s,*') 1, E (s(),, are always stationary, Q converges to the optimal value even on its own replay buffer, without the information of other agents. Third, since all agents act in the same environment, they collect the same state set and the same next state set in their own replay buffers. Therefore, they will converge to the same Q independently, which builds the consensus between the learned policies of agents.
Then, how to build the ideal transition function from Q As the environment Penv and the opti- mal conditional joint policy of other agents π (s, a;) are deterministic, under the ideal transition probabilities Peuv (s'ls, as, T (s, a.)). the environment from the perspective of agent i will deter- ministically transition to the next state that has the highest Q," among all neighboring states given a,;
arg max Q (s, ee.o().) where ((ss, ag) is the neighboring state set of given ag. This is proved in the proof of the following theorem.
(11)
Theorem 2. In deterministic environments, all agents will converge to the optimal policies, if each agent i performs Q-learning the transition function functtoo s = arg (ax Q,8 (8, 8").
Proof. The proof is given in Appendix A, which shows that s" arg max the ideal transition function.
Qf (s, 8') is
To model the stationary ideal transition function s"* = arg max (s, 8'), we train a neural network fs as) to predict s'* and update fi(s, a,) ).) by maximizing:
E. Ea [AQ fi(s, a:)) - (f((s, - -
(12)
The first enforces that the predicted next state has the highest Qt and the second term constrains the predicted next state to be in the set set ((s, a.). The hyperparameter À is the coefficient. Based on the transition model fs az), Q is updated by minimizing the TD-error:
Ey DrND; Tww,
[100 (s, - r Q7 M1 "
at arg max Qi(8', of
(13)
AAthoouhh the environment P..v is deterministic, from the perspective of each agent i, the viewed environ- ment P. would still be non-stationary.

---

Algorithm 1. 12Q for each agent
1: Initialize transition model fi, Q-networks Qi and Q and the target networks Q₁ and Q+e.
2: Initialize the replay buffer 'D--
3: for = 1,..., max_ piau_itteatton do
4: All agents interact in the environment and store experiences (s, az s s', r) in replay buffer D;.
5: Sample mini-batch from D,.
6: Update fi by maximizing (12).
7: Update Q," by minimizing (13).
8: Update Qi by minimizing (1 (14) or (15).
9: Update the target networks Q, and Q
10: end for
And Qi is updated by minimizing:
ExagyrD (0. - -r--r--- max QilJi(s, az). at) (r of
(14)
Q is the target network of Q. When updating Q: (14), the target value is not computed on the next state sampled from replay buffer D, but on the one predicted by the modeled ideal transition function fs, az). In (14), the reward is simplified to depending just When depends on both and s', given s and a, the next state under the ideal transition function (11) satisfies:
Q. 8*) = r(8, 8*)) + y max (,"(s", (8""
8"
-rss + max Q(s", a') + (10)
=r(s,8") + max Q,(s" (2"s"".0 ..)) (8)
of
(,(s, a,) +
(deterministic ideal transition probabilities).
Therefore, update Q, by minimizing:
Ex (Q: (s) QT fi(s,
(15)
The training procedure of 12Q is summarize in Algorithm 1, where each agent i learns f., Q and Q;- Although all the modules update simultancously, as the convergence of Q is guaranteed, the transition function fa derived from Q will be stationary in the later stage. Thus Q2 will also converge. 12Q can be applied in environments with both discrete and continuous state-action space. In continuous action space, we build I2Q on DDPG [14], where policy network (s(s) is trained by maximizing Qi (+)) as a substitute of arg max., Qi a;). ). In continuous state space, since fi(s, a4) is differentiable, (12) can be maximized by gradient ascent. In discrete state space or large state space, we can map the state space to a continuous embedding space, and apply 12Q on the embedding space. We also provide implementation without forward model fę in Appendix B.7,
3.4 Assumptions
We will further discuss the two assumptions. The prime one is that there only one optimal joint policy, but I2Q can easily solve tasks with multiple optimal joint policies. With multiple optimal actions (with the max (s a;)), if each agent arbitrarily selects one of the optimal independent actions, the joint action might not be optimal. To address this, we can set performance tolerance E and introduce a fixed randomly initialized reward function f(s, s') in the range (0, fmax], where Amax (1- 7)s. Then all agents perform 12Q on the shaped reward + f and learn the value function (s, a.) in terms of +r++ + Å. Since f is positive, Qi(s, a:) Qi(s, In Qi(s, at). the maximal contribution from f is fmax/(1
7) = E, so the minimal contribution from r is Qi(8.a) E Qs a:) e, which means that the maximal performance drop is when selecting actions according to Q. Moreover, since the reward function f(s, s') is randomly initialized, it is small probability event to find multiple optimal joint policies on the reward function r + f. Thus, if set to be small enough, 12Q could solve the task with multiple optimal joint policies.
The secondary one is deterministic environments. In stochastic environments, (10) does not hold, and Qt (s, s') can be considered the "best possible value" [6]. So s' arg max Q7 s') is
5

---

fouta
slagee
Figure 1: Illustration of the matrix games.
72.9
(a) L 3
(b) L
(c)
L 5
(d) L 6
Figure 2: Learning curves 100 random matrix games with different stage L.
the best possible transition function, rather than the ideal transition function. The difference between the learned Q: of 12Q and the true value under the ideal transition function Q: is guaranteed by the following theorem.
Theorem 3. 1|Q Qillao S Ar
1- 1- "aaaaaa.mm PPnn T (s(as))
where Ar = Tmax Tmin-
Proof. The proof is given in Appendix A.
Theorem shows that Q; of 12Q closer to the true value if the transition probability of s higher in stochastic environments. Since sample transitions from D) to update fi by maximizing (12), the second term of (12) makes the predicted next state fi a;) be close to the high-frequency next states in replay buffer D; given and a,, which means that the transition probability of the predicted next state Peny (fi(s, [s, a,, (s, a;)) would not be too small. Thus, Qi of I2Q will be close to the true value and the worst cases where the predicted next states have very small transition probabilities
can be avoided. That is the reason why 12Q can be successfully applied in stochastic environments
Experiments
In experiments, first evaluate 12Q on class of randomly generated matrix games verify our theoretical analysis. Second, we compare I2Q with a series of Q-learning variants on MPE-based [15] differential games to illustrate the convergence and optimality of 12Q on more complex tasks with continuous action. Third, we test 12Q on two popular MARL benchmarks: Multi-Agent MuJoCo [18] and SMAC [23]. The experiments cover both fully and partially observable, deterministic and stochastic, discrete and continuous environments. Since we consider the fully decentralized settings, 12Q and the baselines do not use parameter-shue
The results are presented using mean and standard deviation (std) with different random seeds. More details about hyperparameters are available in Appendix C.
4.1 Matrix Games
To support the theoretical analysis of I2Q, we perform experiments on a class of two-agent matrix games with stages. The action space of each agent so the joint action space [A] 9. The payoff tree is shown in Figure 1, where the reward r is randomly generated in the range [- 1] and fixed in each matrix game. Without loss generality, we randomly generate 100 matrix games for each stage L, train the agents for four different seeds in each game, and plot the mean normalized return (normalized by the optimal return) and std over the 100 matrix games. Since the states are countable, we adopt Q-tables instead of neural networks, where Qi table and Q: table are updating simultaneously, and use e-greedy policies. Figure shows the learning curves with different stages under 0.5. IQL cannot converge to the optimal policies and shows large std, because it is afflicted

---

12L
126
Gradto
Shawap
Shadp
Sharap
(8)
0.1
(b) 6
0.3
(c)
0.5
(d) c 0.7
Figure 3: Learning curves with different on 100 random matrix games with stage L.
A(
A
(
A(1))
A(2)
A(3)
1
-12
.12
A1
-12
-12
22
-1
-12
1
12
A
12
(a) case
(b) case
(c) curves on case
(d) curves on case
Figure 4: Learning curves on two specific one-stage matrix games.
with non-stationarity. I2Q converges the global optimum the matrix games, even when the stage grows, which confirms our theoretical analysis.
In Figure 3, we show the effect of exploration rate on the matrix games with L 4. As c increases, 12Q converges faster, while the performance IQL drops. In conventional decentralized methods like IQL, there is dilemma of exploration and exploitation. If is small, the weak exploration causes low sample efficiency and slow learning. When is large, agents act more randomly. From perspective of an individual agent, the transition probabilities will be much different from the real ones under the learned policies of other agents, which causes bad performance. That is the reason why IQL drops when increases. I2Q can avoid the dilemma, because Q7 (s, s') is independent from agent policies and converges faster with larger as it sees more states. Thus, 12Q can use large e, e.g., 0.5 0.7, to promote sample efficiency.
Moreover, we test I2Q on two one-stage matrix games proposed in QTRAN [24] and QPLEX [31] as shown in Figure 4, which are special cases of Figure 1. In case 1, IQL converges to the local optimum, and in case 2, IQL does not converge. Our 12Q can converge to the global optimum easily, in fully decentralized way.
4.2 MPE
To investigate the effectiveness of I2Q in complex continuous environments with neural network implementation, design class of MPE-based differential games, where N agents can move in range [-1, 1]. In each timestep, agent acts action a; [-1, 1]. and position agent updated 24 clip(æ; 0.1 x d, 1, 1) (i.e., the updated position is clipped to [-1, 11). The state the position vector {x1, ... ,NN) The reward function cach timestep is defined as
0.5 cos(co(tmm)) 0.5
if S m
N
if m < S 0.6 0.6 < <1 1.0 1.0 if 1.0
r 0.15 0.8)) + 0.15
m = 0013(N - 1).
N
1=0
The visualization of reward function of two-agent case is shown in Figure 5a. There is only one global optimum (1 = and = 1) but infinite sub-optima (1 = 0.8 and r = 0.3), and the region with > 0.3 is very narrow and surrounded by the region with policies in decentralized way. The episode contains 100 timesteps, and the agents' positions are randomly initialized at the beginning of each episode. To verify the scalability, we test 12Q in the settings with different agent numbers N, and train the agents for eight random seeds in each setting. The results are shown in Figure 6. IDDPG always falls into local optimum due to the non-stationarity and outdated transition probabilities. To be optimistic towards the value punishment is an important technique decentralized MARL. Although adopting the optimistic update, Hysteretic IDDPG [16] still falls into local optimum due to the non-stationarity problem. D3G [6] applies the idea
0. So is hard to learn the optimal

---

8
41.
00
-10
-10 -05
10
-10
-83
0.
(a) reward function
(b) Idcal
(c) 12Q
(d) IDDPG
Figure 5: (a): Visualization reward function differential game with two agents. (b)- Visualizations
learned values of agent in differential game with two agents, xl-axis and x2-axis: agents I and 2's positions, respectively.
DOS
Pedast:DDN DOPS
Uppaatt DDPG
Tpeastic EDPS
DO
B00
N
(b) N
(c) N
Figure 6: Learning curves on MPE-based differential games with different agent numbers N.
120
(a) N 2
(b) N 3
(c)
N==4
(d) N N=
Figure 7: Learning curves MPD-based differential games with different A.
inses
10.0
(a) E = 0.02
(b) g = 0.03
(c) = 0.04
(d) { = 0.05
Figure 8: Learning curves MPE-based differential games with different noise E.
of QSS single-agent learning. To extend D3G into decentralized MARL, we train each agent independently using D3G, without the information of other agents. However, D3G cannot learn the optimal solution either, since the forward model in D3G requires the transition probabilities P. to be deterministic. However, when learning the decentralized replay buffer, P; is the average of non-stationary transition probabilities and cannot be deterministic. Thus, D3G is not applicable to decentralized multi-agent learning. Learning the ideal transition function inferred from Q (s, 8'), 12Q can escape from local optimum and significantly outperform others. To thoroughly demonstrate the effectiveness of I2Q, Figure we visualize the learned values of agent the two-agent case. In Figure 5b, we let the agent 2 always act the optimal policy (moving to the position = 0), which means the agent performs Q-learning on the true ideal transition probabilities. Thus, Figure 5b shows the optimal values under ideal transition probabilities. In Figure 5c, both agents use I2Q. I2Q values are very similar the true ideal values, where the value of the center point highest, which means that the modeled transition function in 12Q is close to the true one. In Theorem 4 (see Appendix A), we theoretically analyze that the difference between ideal values (Figure 5b) and 12Q values (Figure 5c) is bounded by the model error. In Figure 5d, both agents use IQL., resulting in that the value of the center point = 0) the lowest and the suboptimal points (1 0.8) have the highest values.
0) is the
8

---

005 Fommscc
150%
(a) 2 X 3 HallCheetah
(c) 2 x 4 Ant Figure 9: Learning curves Multi- Agent MuloCo.
(b) 2/4 HalsCheetah
(d) 6|2 Ant
BQ Husteria
Hyskows
(a) 8m
(b) 3s vs_42
(c) 2s3z
(d) 5m VS vs_6m
Figure 10: Learning curves SMAC.
To model the ideal transition function, next states predicted by fc should have the highest T values (optimality) and be in the neighboring state set given state-action pair (referred to as neighborhood constraint). To achieve this objective, À in (12) controls the balance between optimality and neighbor- hood constraint. Figure shows the effect of A. When À is too small, the next state predicted by f; cannot maximize Q (s, s¹), and thus 12Q gets stuck at local optimum. Especially, when À = 0, l2Q degrades into popular model-based method MBPÓ [9]. When À is too large, e.g., À 10, paying much more attention to maximize (s, ), ft would generate out-of-neighhorhood states, causing that I2Q even cannot converge to local optimum. We notice that À 0.5 can generally achieve good performance in different settings of N.
Although we assume deterministic environments in theoretical analysis, 12Q can also obtain perfor- mance gain in stochastic environments. To measure the impact of stochastic environments, we add a noise to the position update: z; clip(x; + 0.1 X a, + x 2, -1, 1), where is uniform random variable in [-1, 1]. and & is a constant controlling the randomness. Figure 8 shows the performance with different & in the three-agent case. In the stochastic environments, IDDPG still converges to local optimum, while 12Q always escapes from local optimum when & = 0.02 and 0.03, free from the impact of stochastic environments. When & 0.04 and 0.05, the effect of stochasticity is nearly half of the effect of action. I2Q cannot always learn optimal policies with the strong stochasticity but still significantly outperforms IDDPG.
4.3 Multi-Agent MuJoCo and SMAC
To investigate the effectiveness of 12Q in partially observable environments, we perform experiments on Multi-Agent MuloCo [18], where each agent independently controls one or some joints of the robot and could only observe the state of its own joints and bodies (with the parameter agent obsk = =). The results are shown in Figure 9 with eight random seeds. I2Q achieves higher rewards than IDDPG, which indicates that 12Q could be extended to partially observable tasks and obtain performance gain by performing Q-learning on ideal transition probabilities instead of transition probabilities in replay buffer. In the partially observable setting, we only consider two-agent cases. When there are more agents, each agent can only control one or two joints, and the observation range too limited to learn strong policies. In Appendix B.4, provide the 6-agent Walker and 8-agent Ant experiments with the full observation setting to verify the scalability.
We test 12Q on partially observable and stochastic SMAC tasks [23] with the version SC2.4.10 including both easy and hard maps [33]. We adopt the implementation of PyMARL [23] that takes as input the trajectory of partial observations, which is high-dimensional and not differentiable on some dimensions, meaning that directly optimizing (12) hard. So the QSS model Q is not built on the trajectory of partial observations, but on the hidden state of Q; (the output of RNN layer), which is 64-dimension and differentiable, and the transition function fa predicts the next hidden state. We train the agents for four random seeds. D3G can only be used in continuous action space, do

---

not compare with it in SMAC. As shown in Figure 10, 12Q is capable of handling high- gigh-dimensional and stochastic tasks.
5 Closing Remarks
In this paper, we propose I2Q. a novel variant of independent Q-learning, to deal with the non- stationarity in decentralized cooperative MARL. 12Q models the ideal transition function, which depends on the optimal conditional joint policy of other agents and performs independent Q-learning on the modeled ideal transition function. Theoretically, prove that I2Q converges the optimal joint policy in deterministic environments. Empirically, 12Q obtains performance gain in a variety of multi-agent tasks.
We focus on the base algorithm for fully decentralized learning in cooperative MARL, as it lay the algorithmic foundation for decentralized learning. Although did not consider explicit coordination among agents, in partially observable environments coordination such as communication is desirable improve empirical performance. We envision that more methods can be further built the base algorithm. Moreover, currently, although fully decentralized learning algorithms may not empirically perform as well as CTDE methods like QMIX [22] in some benchmarks, e-g., hard maps in SMAC, we believe more efforts should be made by the MARL community on decentralized learning, considering its abundant benefits over CTDE, First, fully decentralized algorithms do require centralized learner during training, hence they are more applicable and cleaner, easier to implement [10]. Second, fully decentralized algorithms are more versatile as the individual learners are indifferent to the number of other agents, thus they have better scalability [34]. Third, fully decentralized algorithms make agents more robust to the presence of agents they were not trained with (such as humans) [30]. Therefore, fully decentralized learning is a fundamental problem in cooperative MARL, but remains be investigated.
Acknowledgments and Disclosure of Funding
This work was supported by NSF China under grant 61872009. The authors would like to thank the anonymous reviewers for their valuable comments.
References
[1] Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
[2] Natalia Akchurina. Multiagent reinforcement learning: algorithm converging to nash equilib- rium in general-sum discounted stochastic games. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pages 725-732. Citeseer, 2009.
[3] Zilong Cao, Pan Zhou, Ruixuan Li, Siqi Huang, and Dapeng Wu. Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0. IEEE Internet of Things Journal, 7(7):6201-622 6213, 2020.
[4] Christian Schroeder de Witt. Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is Independent Learning All You Need in The StarCraft Multi-Agent Challenge? arXiv preprint vrXiv:2011.09533. 2020.
[5] Christian Schroeder de Witt, Bei Peng, Pierre-Alexander Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control. arXiv preprini arXiv: arXiv:2003.06709 2020.
[6] Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q (s. s') with deep deterministic dynamics gradients. In International Conference on Machine Learning (ICML), 2020.
[7] Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising Experience Replay for Deep Mueti-Agent Reinforcement Learning. In International Conference on Machine Learning (ICML), 2017.
10

---

[8] Shariq Iqbal and Fei Sha. Actor-Attention International Conference on Machine Learning (ICML). 2019.
for Multi-Agent Reinforcement Learning, In
[9] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[10] Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 2021.
[11] Sachin Konan, Esmaeil Seraj, and Matthew Gombolay. Iterated reasoning with mutual information in cooperative and byzantine decentralized teaming. In International Conference on Learning Representations (ICLR), 2021.
[12] llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In International Conference on Learning Representations (ICLR), 2021.
[13] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. International Conference on Learning Representations (ICLR), 2022.
[14] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[15] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi- Actor-Critic for Mixed Cooperative-Compettivs Environments. Neural Information Processing Systems (NeurIPS), 2017.
[16] Lačtitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 64 69. IEEE, 2007.
[17] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient Multi- Deep Reinforcement Learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
[18] Bei Peng. Tabish Rashid, Christian Schroeder de Witt, -lexaa-Ae Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[19] Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming, 1994.
[20] Han Qie, Dianxi Shi, Tianlong Shen, Xinhai Xu, Yuan Li, and Liujing Wang. Joint optimization of multi-uav target assignment and path planning based on multi-agent reinforcement learning. IEEE access, 7:146264- 146272, 2019.
[21] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems (NeurlPS), 2020.
[22] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In International Conference on Machine Learning (ICML), 2018.
[23] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim J. Rudner, Chia-Man Hung. Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. arXiv preprint
2019.
[24] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: I Learning To Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learn ing. In International Conference on Machine Learning (ICML), 2019.
11

---

[25] Kefan Su and Zongqing Lu. Divergence-Regulaaiieed Regularized Multi-Agent Actor-Critic arXiv preprint 1rX1v:2110.00304, 2021.
[26] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward. In International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2018.
[27] Ardi Tampuu, Tambet Matüisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4)=01172395, 2017.
[28] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Interna- tional Conference on Machine Learning (ICML), 1993.
[29] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575(7782):350-334 2019.
[30] Caroline Wang, Ishan Durugkar, Elad Liebman, and Peter Stone. DM Distributed multi-agent reinforcement learning for distribution matching. arXiv preprint arXiv:2206.002: 2022.
[31] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex Dueling Multi-Agent Q-Leernnigg In International Conference on Learning Representations (ICLR), 2021.
[32] Yihan Wang. Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off- policy multi-agent decomposed policy gradients. In International Conference Learning Representations (ICI.), 2020.
[33] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
[34] Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint arXiv: 2019.
[35] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[36] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. International Conference on Machine Learning (ICML), 2018.
[37] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. FOP: Factor- izing Optimal Joint Policy of Maximum Entropy Multi-Agent Reinforcement Learning. In International Conference Machine Learning (ICML), 2021.
[38] Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Peter Glynn, and Claire Tomlin. Multi-agent online learning with imperfect information, 2018
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.4.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] As we provide a new method for decentralized MARL, do not see any potential negative societal impacts at this stage.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] We have clearly stated the assumptions in Section 3.2 and 3.3, and further discussed them in Section 3.4.
(b) Did you include complete proofs of theoretical results? [Yes] We include the proof of Theorem 1 in Section 3.2, and include the proofs of Theorem 2,3,4 in Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] We include the training details in Appendix C.
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We include the details of computation in Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL?? Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r using/curating? [N/A] We do not use any dataset in this paper.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zdaFcnUy6Fc", "reviewer": "Reviewer_zsxJ", "summary": "This paper presents I2Q, an algorithmic approach for decentralized MARL. The authors present the non-stationarity problem in this setting and propose to use \"ideal transition probabilities\" to solve it. Particularly, these are transition probabilities for which all agents are ensured to converge to an optimal solution when trained in a decentralized manner. The authors then propose to use the next state (in deterministic environments) as a representation of an action, and show that it induces an ideal transition probability, which ensures convergence to an optimal solution. They experiment on many baselines in various domains, showing the benefit of their approach.", "questions": "Most of my questions relate to things I already mentioned above.\n1. Can stochasticity be addressed with \"ideal transition probabilities\" more explicitly? How can we learn such probabilities? How would this affect training? If not possible, what are the limitations?\n2. How does approximation of $f$ affect convergence?", "limitations": "The authors discuss limitations of their work. Some of these limitations coincide with points I've already raised. As mentioned above, I believe some of these points should be addressed more thoroughly in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes an elegant solution to the non-stationarity problem of decentralized MARL. I'm not able to say if it is the first method to solve this problem, and I hope one of the other reviewers will address this. The paper is clearly written, and the presentation is great. Also, I found everything to be easy to read and follow. Finally, the experiments section seems to have chosen a wide variety of tasks, and I'm glad the authors also chose to show results on the high dimensional problem of SCII.\n\n----------------------------------------------------------------------------------------------------------\n\nThe paper doesn't have strong flaws, but there are some issues that make it a borderline paper for Neurips.\n\nFirst, the theory is not very deep. There are many questions that remain open that the authors don't address theoretically, and I think are important for a better understanding of the problem. One of these, is convergence proof of I2Q, which the authors don't really prove, but only discuss informally.\n\nSecond, I feel that the deterministic assumption in the paper is a strong one, unless carefully addressed. In favor of the authors, they do discuss this in the paper, showing a result of the value gap, and also experiments on a wide variety of tasks. Still, I believe this is not adequately addressed. A stronger result for stochastic environments should be provided. I assume there exist some \"ideal transition probabilities\" for this setting. If it is the case that such are impossible to theoretically find, then this is an important point to address in the paper. Overall, I find Theorem 3 to be a trivial result. I wish to see an approach that tackles stochasticity explicitly, and provides a tighter bound for approximation errors.\n\nThird, the fact that I2Q must learn a forward model is troubling, as model-based methods usually fail againt state of the art model-free methods on high dimensional tasks (unless latent spaces are used, such as in MuZero). The authors don't address the problem of estimating $f$ in their work. Moreover, I feel that this is not addressed fully in the experiments either.\n\nFinally, while the experiments show results on different types of environments, I find that I2Q was not compared against enough baselines. There are a lot of new baselines on MARL, and particularly I would expect the authors to compare I2Q to at least three more baselines which are considered SOTA, and not only IQL - even if they are not decentralized.\n\n----------------------------------------------------------------------------\nStrengths:\n1. A new solution for decentralized MARL\n2. Proofs to formal statements seem correct\n3. Paper is clearly written and presentation is great\n4. Experiments show a variety of interesting tasks\n\nWeaknesses:\n1. Theory is weak\n2. Stochastic environments should be addressed\n3. Forward model should be addressed theoretically and in experiments\n4. Experiments are lacking comparison to other algorithms", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "jmJUJkgbSxX", "reviewer": "Reviewer_fWps", "summary": "This paper presents an important and interesting approach on fully decentralized MARL. Fully decentralized Q-learning is highly applicable to realistic and real-world applications. The method is evaluated extensively, showing a great potential.", "questions": "- I like the idea behind this approach and I believe that this is a strong work with potential for real-world applicability. \n\n- Some important related work on fully decentralized MARL are not covered [1-2]. These works and similar prior work must be discussed to cover similarities and differences so that readers can select the proper method based on their applications. \n\n[1] Zhang, Kaiqing, et al. \"Fully decentralized multi-agent reinforcement learning with networked agents.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Konan, Sachin G., Esmaeil Seraj, and Matthew Gombolay. \"Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming.\" International Conference on Learning Representations. 2021.\n\n- The approach potentially needs to be compared against SOTA fully decentralized MARL work to evaluate its utility and feasibility.", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Theoretical analysis\n- Extensive evaluations\n- Interesting perspective to the MARL problem with potential real-world applicability\n\n## Weaknesses\n- missing out on some related work on fully decentralized MARL\n- Lack of SOTA baselines", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TsixHy2Vct9", "reviewer": "Reviewer_8HEx", "summary": "This paper proposes a new MARL algorithm under the DTDE paradigm. Specifically, the proposed algorithm (I2Q) is introduced based on the ideal transition probability (where each agent assumes that the others adopt the optimal actions for each decision) and a previous idea named QSS-Learning. Theoretical guarantee on the convergence of the proposed algorithm is provided under certain conditions. For experimental studies, the significant superiority of I2Q is demonstrated in matrix games, MPE, MA MuJoCo and SMAC.", "questions": "Questions:\n+ If we add a fixed randomly initialized reward function and the performance tolerance to the other algorithms, is there a performance improvement? \n\n\nI appreciate the authors’ reasonable introduction of QSS for DTDE.\nHowever, my major concern is the lack of novelty in the proposed algorithm. The current algorithm seems to be a simple application of the QSS-Learning in the MARL scenario, the QSS and the prediction of next state in this paper are consistent with the original QSS paper. \n\n&nbsp;\n\nSuggestions:\n\n+ I recommend the authors to combine the proposed approach with more base RL algorithms, such as PPO, SAC and TD3 to better evaluate of the generality of the method.\n\n+ The authorss should add a discussion of QSS and the differences between I2Q and QSS in the related work or background.\n", "limitations": "The rationality and limitations of the main assumptions adopted in this paper are discussed in Sec.3.4.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ This paper is clearly written.\n+ The experimental part is relatively diverse and adequate.\n\n&nbsp;\n\nCons:\n- The novelty of the proposed algorithm is limited. \n\n&nbsp;\n\n\nMinor issues and typos:\n- l.299 \"SAMC\" → \"SMAC\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept