Finite-Time Analysis of Fully Decentralized Single-Timescale Actor Critic 
Authors: qijun luo
Xiao Li
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Actor Critic
finite-time analysis

Abstract: Decentralized Actor-Critic (AC) algorithms have been widely utilized for multi-agent reinforcement learning (MARL) and have achieved remarkable success. Apart from its empirical success, the theoretical convergence property of decentralized AC algorithms is largely unexplored. The existing finite-time convergence results are derived based on either double-loop update or two-timescale step sizes rule, which is not often adopted in real implementation. In this work, we introduce a fully decentralized AC algorithm, where actor, critic, and global reward estimator are updated in an alternating manner with step sizes being of the same order, namely, we adopt the \emph{single-timescale} update. Theoretically, using linear approximation for value and reward estimation, we show that our algorithm has sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-2})$ under Markovian sampling, which matches the optimal complexity with double-loop implementation (here, $\tilde{\mathcal{O}}$ hides a log term).  The sample complexity can be improved to ${\mathcal{O}}(\epsilon^{-2})$ under the i.i.d. sampling scheme. The central to establishing our complexity results is \emph{the hidden smoothness of the optimal critic variable} we revealed. We also provide local action privacy preserving version of our algorithm and its analysis. Finally, we conduct experiments to show the superiority of our algorithm over the existing decentralized AC algorithms.

Finite-Time Analysis of Fully Decentralized Single-Timescal Actor-Critic
Anonymous Author(s)
Affiliation
Address
email
Abstract
Decentralized Actor-Critic (AC) algorithms have been widely utilized for multi- agent reinforeement learning (MARL) and have achieved remarkable success. Apart from its empirical success, the theoretical convergence property of decen- tralized AC algorithms is largely unexplored. The existing finite-time convergence results are derived based on either double-loop update two-timescale step sizes rule, which is not often adopted in real implementation. fully decentralized AC algorithm, where actor, critic, and global reward estimator are updated in an alternating manner with step sizes being of the same order, namely, update. Theoretically, using linear approximation for
this work, we introduce
adopt the value and reward estimation, we show that our algorithm has sample complexity Ōc=²?) under Markovian sampling, which matches the optimal complexity with double-loop implementation (here, Ô hides log term). The sample complexity be improved to O under the i.i.d. sampling scheme. The central establishing our complexity results is the hidden smoothness of the optimal critic variable we revealed. We also provide a local action privacy-preervving version of our algorithm and its analysis. Finally, we conduct experiments to show the superiority of our algorithm over the existing decentralized AC algorithms.
10
11
12
14
15
16
17
Introduction
19
Multi-agent reinforcement learning (MARL) [16, 30] has been very successful in various models of multi-agent systems, such as roboties [14], autonomous driving [37], Go [25], etc. MARL has been extensively explored in the past decades; see, e.g., [18, 20, 41, 26, 8, 22). These works either focus the setting where central controller is available, assuming common reward function for all agents. Among the many cooperative MARL settings, the work [42] proposes the fully decentralized MARL with networked agents. In this setting, each agent maintains a private heterogeneous reward function, and agents can only access local/neighboring information through communicating with its neighboring agents the network. Then, the objective of all agents is to jointly maximize the average long-term term reward through interacting with environment modeled by multi-agent Markov decision process (MDP). They proposed the decentralized Actor-Critis (AC) algorithm to solve this MARL problem, and showed its impressive performance. However, the theoretical convergence properties of such class of decentralized AC algorithms are largely unexplored; see [41] for a comprehensive survey. In this work, our goal is establish the strong finite-time convergence results under this fully decentralized MARL setting. We first review some recent progresses on this line of research below.
21
22
29
24
25
26
27
28
29
30
31
32
33
Related works and motivations. The first fully decentralized AC algorithm with provable con- vergence guarantce was proposed by [42], and they achieved asymptotic convergence results under two-time scale step sizes, which requires actor's step sizes to diminish in a faster scale than the critic's step sizes. The sample complexities of decentralized AC were established recently. In particular, [6]
34
35
36
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

and [11] independently propose two communication efficient decentralized AC algorithms with opti- mal sample complexity of O( log(E-") under Markovian sampling scheme. Their analysis are based on double-loop implementation. where each policy optimization step follows a nearly accurate critic optimization step (.. policy evaluation), i.e., solving the critic optimization subproblem to e-accuracy. Such a double-loop scheme requires careful tuning of two additional parammearameters, which are the batch size and inner loop size. In particular, the batch size and inner loop size need be of order ((=" and (log(s-1)) in order to achieve their sample complexity results, respectively. In practice, single-loop algorithmic framework is often utilized, where one updates the actor and critic in an alternating manner by performing only one algorithmic iteration for both of the two subproblems: see, e.g., [23, 18, 15, 39]. The work [38] proposes a new decentralized AC algorithm based on such a single-lopp alternative update. Nevertheless, they have to adopt wvo-timescale step sizes rule to ensure convergence, which requires actor's 's step sizes to diminish in a faster scale than 48
37
38
39
40
41
42
43
44
45
48
47
48 the critic's step sizes. Due the separation of the step sizes, the critic optimization sub-probie prooloob
50 is solved exactly when the number of iterations tends to co. Such a restriction on the step size will slow down the convergence speed of the algorithm. As a consequence, they only obtain sub-optimal
51
sample complexity of O(s ). In practice, most algorithms are implemented with single-timescale step size rule, where the step sizes for actor and critic updates are of the same order. Though there
52
53
54 are some thcoretical uchievements for migle-timesccle update in other areas such as TDC [31] and bi-level optimization [4]. similar theoretical understanding under AC setting is largely unexplored.
55
Indeed, even when reducing to single-agent setting, the convergence property of simgle-timescale AC algorithm is not well established. The works [9, 10] establish the finite-time convergence result under a special single-timescale implementation, where they attain the sample complexity of O(=-²). However, their analysis is based on an algorithm where the critic optimization step is formulated as a least-square temporal difference (LSTD) at each iteration, where they need sample the transition tuples for O(s-1) times to form the data matrix in the LSTDD problem. Then, they solve the LSTD) problem in closed-form fashion, which requires to invert a matrix of large size. Later, [4] obtains the same sample complexity using TD(0) update for critic variables under i.i.d. sampling. Nonetheless, their analysis highly relies on the assumption that the Jacobian of the stationary distribution is Lipschitz continuous, which is not justified in their work.
57
58
59
60
61
62
63
64
65
The above observations motivate us to ask the following question: 66
67
Can we establish finite-time convergence result for decentralized AC algorithm with single-limescale step sizes rule?!
68
89
Main contributions. By answering this question positively, we have the following contributions:
70
We design fully decentralized AC algorithm, which employs single-timescale step sizes rule and adopts Markovian sampling scheme. The proposed algorithm allows communication between agents for every Ke iterations with K, being any integer lies in [1, O(s- }) rather than communicating each iteration as adopted by previous single-loop decentralized AC algorithms [38, 42].
71
72
73
74
75
Using linear approximation for value and reward estimation, we establish the finite-time convergence result for such an algorithm under the standard assumptions. In particular, we show that the algorithm has the sample complexity of ((=-)), which matches the optimal complexity up to a logarithmic term. In addition, we show that the logarithmic term can be removed under the i.i.d. sampling scheme. Note that these convergence results are valid for all the above mentioned choices for Ke.
76
77
78
79
80
81
To preserve the privacy of local actions, we propose variant of our algorithm which utilizes noisy local rewards for estimating global rewards. We show that such an algorithm will maintain the optimal sample complexity at the expense of communicating at each iteration.
82
83
84
The underlying principle for obtaining the above convergence results is that we reveal the hidden smoothness of the optimal critic variable, so that can derive an approximate descent the averaged critic's optimal gap at each iteration. Consequently, we can resort to the classic convergence analysis for alternating optimization algorithms to establish the approximate ascent property of the overall optimization process, which leads to the final sample complexity results.
85
86
87
88
As convention [9], when we use single-timescale it means we utilize single-loop algorithmic framework with single- timeslleetteeaeeee step sizes rule.

---

89
Another technical highlight is the Lyapunov function we construct for measuring the progress of our algorithm. Such a construction is motivated by [4], which analyzes bi-level optimization algorithm. However, our Lyapunov function is different from theirs as it involves the additional optimal gup of averaged critic and reward estimator, which necessary for dealing with the decentralized setting.
90
91
92
98
We finish this section by remarking that our convergence results are even new for single agent AC algorithms under the setting of single-timescale step sizes rule.
94
95 2
Preliminary
98
In this section, we introduce the problem formulation and the policy gradient theorem, which serves as the preliminary for the analyzed decentralzed AC algorithm.
97
98
Suppose there are multiple agents aiming to independently optimize a common global objective, and each agent can communicate with its neighbors through network. To model the topology, we define the graph as G = EW, where N is the set of nodes with [W] N and & is the set of edges with
99
100
[ E. In the graph, each node represents agent, and each edge represents communication
102
link. The interaction between agents follows the networked multi-agent MDP.
103
2.1
Markoy decision process
A networked multi-agent MDP is defined by a tuple (G, S, {A'JiEN: P, Pr (r")ee 7). G denotes the communication topology (the graph), S is the finite state space observed by all agents, A' represents the finite action space of agent i. Let A == A x .... x A denote the joint action space and s( (s, a): : S x A S [0, 1] denote the transition probability from any state s S to any state E S for any joint action a E A. r² S x A + IR is the local reward function that determines the reward received by agent i given transition (s, a); 7 E [0, 1] is the discount factor.
105
106
108
109
[a For simplicity, we will use a := [a!,... N to denote the joint action, and == [0],... [.',...... .N E Rdox to denote joint parameters of actors, with 0² E Redo Note that different actors may have different number of parameters, which is assumed to be the same for our paper without loss of generality. The MDP goes as follows: For given state 8, each agent make its decision a' based on its policy a² ~ To: (-|s). The state transits to the next state s' based on the joint action of all the
110
111
112
113
agents: ~ P--|s, a). Then, each agent will receive its own reward r' (s, a). For the notation brevity, 115
we assume that the reward function mapping is deterministic and does not depend on the next state without loss of generality. The stationary distribution induced by the policy To and the transition
117
kemel is denoted by Mmo (s).
119 Our objective to find a set of policies that maximize the accumulated discounted mean reward received by agents
8" = arg max J(0) = F(Sk, as)
(1)
0
121
Here, Is represents the time step. F(sk, ax) N Ei=1" (sk, ag) the mean reward among agents at time step k. The randomness of the expectation comes from the initial state distribution Mo(s), the transition kernel P, and the stochastic policy we (-1s).
122
123
124
2.2
Policy gradient Theorem
125
Under the discounted reward setting, the global state-value function, action-value function, and advantage function for policy set 0, state 8, and action a, are defined
126
V= (8) = E
Assk. ax) ax)(s0 =
(2)
k=
(s) == E
> #F )2) = 8, 00
k
Ang (3, a) = Qro (5,
Vio (s).
3

---

127
To maximize the objective function defined in (I), the policy gradient [28] can be computed as follow
VoJ(O)
= E Exnndin aamag
A a) (s,
128 where dz (8) = (1 7) Σ k policy no, and 177 a)
(s = =) the discounted state visitation distribution under log (s, a) is the score function.
130
Following the derivation of [42], the policy gradient for each agent under discounted reward setting
can be expressed as
J(0) E Eywdag araay
1-7 ATo (8, a) (s,
(3)
3 Decentralized single-timescale actor-critic
Algorithm 1: Decentralized sngle-timesen AC (reward estimator version)
1: Initialize: Actor parameter Og. critic parameter wo, reward estimator parameter Ao initial state so.
2: for k = 0,.... KK- do
3: Option 1: i.i.d. sampling:
4: 8k ~ (4,,()), ag ~ To, (+18a), 8x+1 P(-(sa ax).
5: Option 2: Markovian sampling:
6: ax To, (+]se), 8841 ~ P(.(sE..))
7:
Periodical consensus: Compute Wi and Ait by (4) and (7).
9:
10:
for i = 0, .N in parallel do
11:
Reward estimator update: Update 1 by (8).
Update wh++1 by (5). 13: Actor update: Update fi by (6).
12:
Critic update:
14:
end for
15:
end for
135 We introduce the decentralized single-timescale AC algorithm; see Algorithm In the remaining parts of this section, we will explain the updates in the algorithm in details.
135
In fully-decentralized MARL, each agent can only observe its local reward and action, while trying to maximize the global reward (mean reward) defined in (I), The decentralized AC algorithm solves the problem by performing online updates in an alternative fashion. Specifically, we have N pairs of actor and critic. In order to maximize J(0), each critic tries to estimate the global state-value function V= (s) defined in (2), and each actor then updates its policy parameter based on approximated policy gradient. We now provide more details about the algorithm.
138
137
139
140
We will use w² E Rd to denote the in critic's parameter and @ := N Li=1 w to represent the averaged parameter of critic. The ith critic approximates the global value function as VE, (s) (s).
Critics' update.
142
143
144
As we will see, the critic's approximation error can be categorized into two parts, namely, the consensus error >> -= lw @|I, which measures how close the critics' parameters are; and the approximation error (+)ll. which measures the approximation quality of averaged critic.
145
146
147
In order for critics to reach consensus, we perform the following update for all critics
E wiww
if l mod K. otherwise.
at
(4)
148 where W e gaxn is a weight matrix for communication among agents, whose property will be 149 specified in Assumption 5; K. denotes the consensus frequency.
To reduce the approximation error, will perform the local TD(0) update [29] as
w a+1 = Il [ + (E, w())),
(5)
R

---

151
is the update direction, (E, := (s, a) + (s') (s) is the local temporal difference error (TD-error). 3q is the step size for critic at iteration II, projects the parameter into a ball of radius Ra containing the optimal solution, which will be explained when discussing Assumption and 2.
where ;= (s, s') represents transition tuple, gi((,,w) :=
153
155
Actors' update. We will use stochastic gradient ascent to update the policy's parameter, and the stochastic gradient is calculated based on policy gradient theorem in (3). The advantage function An (s, a) can be estimated by
156
157
(g. F(. +V()) V(s).
with a sampled from m-(-))). However, to preserve the privacy of each agents, the local reward cannot be shared to other agents under the fully decentralized setting. Thus, the averaged reward F(sk. ag) is not directly attainable. Consequently, we need strategy approximate the averaged reward. In this paper, we will adopt the strategy proposed in [42]. In particular, each agent will have local reward estimator with parameter X Rd which estimates the global averaged reward as
159
161
163
(skk, ag) z fxe (8k, az).
184
Thus, the update of the in actor is given by
0 0% + agd(En, W Mesr) (sks aft;
(6)
where 8(g, w, = (8, a) + (s') (s) is the approximated advantage function. Oz: is the step size for actor's update at iteration
166
Reward estimators' update. Similar to critic, each reward estimator's approximation error can be decomposed into consensus error and the approximation error.
168
169
For each local reward estimator, we perform the consensus step to minimize the consensus error as
Σ X
WoX
if mod K, = 0 otherwise.
j=
170
To reduce the approximation error, we perform a local update of stochastic gradient descent.
X
[ + magt (ER, X()), Rx
where gf (E, A) := (r) fx as
a) is the update direction. Tk is the step size for reward estimator at iteration k. Note the calculation of (E, A) does not require the knowledge of s'; we use in (8) just for notation brevity. Similar to critic's 's update, IIn, projects the parameter into ball of radius of RA containing the optimal solution.
172
174
175
In our Algorithm 1, we will use the same order for as, 8k, and me and hence, our algorithm is in single-timescale.
176
177
Linear approximation for analysis. In our analysis, we will use linear approximation for both critic and reward estimator variables, i.c. (s) == o(s) (x(s,) a) = spss, a)" X, where d(s) S + dd and (s,, a) S A + As are two feature mappings, whose property will be specified in the discussion of Assumption 1.
178
179
180
181
Agorithm for preserving the local action. Note that Algorithm 1, the reward estimators need the knowledge of joint actions in order to estimate the global rewards. To preserve the privacy of local actions, we further propose a variant of Algorithm 1, which estimates the global rewards by communicating noisy local rewards; see [6] for the original idea. However, maintain the optimal sample complexity, such an approach requires (log(= 2)) communication rounds for each iteration. We postpone the detailed design and analysis of such an algorithm scheme into Appendix B.
183
185
187
Remarks on sampling scheme. The unbiased update for critic and actor variables requires sampling from Mr, and d respectively. However, in practical implementations, states are usually collected from an online trajectory (Markovian sampling), whose distribution is generally different for ua and d Such distribution mismatch will inevitably cause biases during the update of critic and
189
191 actor variables. One has to bound the corresponding error terms when analyzing the algorithm. In this work, we will provide the analysis for both sampling schemes.
5

---

4
Main Results
193
In this section, we first introduce the technical assumptions used for analysis, which are standard in the literature. Then, we present the convergence results for both actor and critic variables under i..d. sampling and Markovian sampling.
195
197
4.1
Assumptions
198
Assumption (bounded rewards and feature vectors). All the local rewards are uniformly bounded,
L.., there exists a positive constants mmax such that (s'(s,)) a) S Tmaxs for all feasible (s, a) and
i E [N]. The norm of feature vectors are bounded such that for all 8 E S, a A. [((()) S
200
1, [1((s, a)ll S
Assumption is standard and commonly adopted; see, e.g., [3, 35, 38, 24, 21]. This assumption can be achieved via normalizing the feature vectors.
203
204
Assumption 2 (negative definiteness of As, and A8.z). There exists two positive constants XØx Xup such that for all policy the following two matrices are negative definite
205
Aa. = (o=ppatslo(s)((0((s)))
)(s)))
Ap.p == po (o) p )))
206
with Amaxx (A0 S Ao. Amax (A0.p) I Xp. where Amax() represents the largest eigenvalue.
207
Assumption can be achieved when the matrices & == .... (() and 4 := (1()1, a.). (((sss): [AA)] have full row rank, which ensures that the optimal critic and reward estimator are unique; see also [24, 34J. Together with Assumption I, we can show that the norm of w (0) and X* (0) are bounded by some positive constant, which justifies the projection steps.
208
210
Assumption 3 (Lipsehitz properties of policy). There exists constants Ca., Teg, La such that for all 0, 0' E S and E A. we have (1)- [aa(as)) (als)) & T, 0' (2). (1ve(s, the (s, a)l < Ly |10 01|| (3). [100(s, a)ll S Co.
211
213
214
Assumption 3 is common for analyzing policy-based algorithms: see, e.g., [33, 32, 11]. The assump-
215
tion ensures the smoothness of objective function (0). It holds for a large range of policy classes such as tabular softmax policy [1], Gaussian policy [7], and Boltzman policy [13].
216
Assumption 4 (irreducible and aperiodic Markov chain). The Markov chain under Ty and transition
218
kernel (118s, a) is irreducible and aperiodic for any 0.
219
a standard assumption, which holds for any uniformly ergodic Markov chains and any time- time-homogeneous Markov chains with sinite-state space. It ensures that there exists constants k and p E (0, 1) such that
Assumption
221
sup (P -sso s, To), pe) S kpp Yk. =ES Assumption 5 (doubly stochastic weight matrix). The communication matrix W is doubly stochastic, i.e. each column/row sum up 1. Moreover, the second largest singular value is smaller than 1.
222
Assumption 5 is common assumption in decentralized optimization and multi-agent reinforcement learning: see, e.g., [27, 5, 6]. It ensures the convergence of consensus error for critic and reward estimator variables.
225
4.2
Sample complexity under i.i.d. sampling
228
Theorem (sample complexity under i.i.d. sampling). Suppose Assumptions 1-5 hold. Consider the update of Algorithm I under i.i.d. sampling. Let ag VK
& for some positive constant &,
Co ag, and Cie ag, Ke S O(a,))) where K denotes the total number of iterations. 2h
2h
Then, we have
K
N
E [| - w- (0.)11
5
K
VK
1=1
K
N
1
ΣΣ
E
5o
1
+ O(app + Esp),
(9)
K
K

---

where Co, C10 are positive constants defined in the proof.
233
The proof of Theorem can found in Appendix E.1. It establishes the iteration complexity of ()) or equivalently, sample complexity of O for Algorithm 1. Note that actors, critics, and reward estimators use the step sizes of the same order. The sample complexity matches the optimal rate of SGD for general non-convex optimization problem. To explain the errors in (9), let us define the approximation error as the following:
234
236
Berwo [V (s) (o) (F(s,a) - fx (x (o)
Eapp
I1xX 0,a
238
The eтror Eppp captures the approximation power of critic and reward estimator. Similar terms also appear in the literature (see e.g.. [35, 1, 21]). Such an approximation error becomes zero in tabular case. The error Esp inevitably caused by the mismatch between discounted state visitation distribution dr,, and stationary distribution May' see, e.g., [38, 24). It is defined as
240
Exp = ( + [1 7)
242
When is close to I, the error becomes small. This is because dz, approaches to Mm when goes to 243 In the literature, some works assume that sampling from dz, is permitted, thus eliminate this error; see, c.g., [4].
244
245
4.3
Sample complexity under markovian sampling
Theorem (sample complexity under Markovian sampling). Suppose Assumptions 1-5 hold. Con- & for some positive constant sider the update of Algorithm under Markovian sampling. Let oi *KK"
247
Co agg, and TiK = Ci ark Ke S O(a 2), where K the total number of iterations. Then, 2A
248 ā. Bg
2hg
we have
K
log K K
E
|l (0+)]] < o
K
1=1
K
log² K VK
B so
O + Exp),
(10)
k=li=l
where Cg, Cio are positive constants defined in proof.
251 We put the proof of Theorem in Appendix E.2. In Markovian sampling, the updates are biased for critics. actors, and reward estimators. The error will decrease as the Markov chain mixes, and the logarithmic term is due to the cost for mixing.
253
Theorem establishes the iteration complexity of Olog? K/vK), or equivalently, sample complex- ity of (s-22) for Algorithm 1. matches the state-of-the-art sample complexity of decentralized AC algorithms, which implemented in double-loop fashion [11, 6].
254
255
4.4
Proof sketch
258
We present the main elements for the proof Theorem 2, which helps understanding the difference between classical two- timescale/double-lnne analysis and single-timescal analysis. The proof of Theorem follows the same framework with simpler sampling scheme.
280
Under Markovian sampling, it is possible to show the following inequality, which characterizes the ascent of the objective.
262
B[J(Ok+ ((x)) 2
Okk E V (0+)1 + ak (fr, 2
2 SC (0+) wf 4C3o 11+
O(log((K)aa) O((Eapp Exp)
(II)
7

---

To analyze the enrors of critic ||w* - 1 and reward estimator ((x) - X the two- timescale analysis requires (aag) < (5x)) (nx)) in order for these two errors to converge.
263
264
285
The double-loop approach runs lower-level update for (log((
times with batch size (
298
to drive these errors below and hence, they cannot allow inner loop size and bath size to be (
257
simultaneously. To obtaín the convergence result for single-timescale update, the idea is further
upper bound these two lower-level errors by the quantity (lag (Er............ Wk++ 141 )13)) (through
series of derivations), and then eliminate these errors by the ascent term 2e 2-lll (Ex, (Ex.WE+1 111?.
270
We mainly focus on the analysis of critic's s error through the proof sketch. The analysis for reward estimator's s error follows similar procedure. We start by decomposing the error of critic as
271
1 w (0x)ll (11a++1 Wh+1ll || w- (0x))
(12)
272
The first term represents the consensus error, which can be bounded by the next lemma.
273 Lemma Suppose Assumptions and 5 hold. Consider the sequence (wf) generated by Algorithm 1, then the following holds
IQwk+1ll S Su²a ||olll || + E4E'
Bi BIVNCs.
(=0
275 where wo = [w'.... W Q := second largest singular value of W.
111 k' ==
[ * Ke. The constant v E (0, 1) is the
276
Based Lemma and follow the step size rule of Theorem 2, it is possible show [|Qwk+11 112 278
E || W - O(K3B)) Let Ke 0(338) have [|w || ((2)), which
N
maintains the optimal rate.
To analyze the second term in (12), first construct the following Lyapunov function
V = (6xk + || w" (0=) + a (0x)))
(13)
Then, it remains to derive approximate descent property of the term || (0x))l in (13). Towards that end, our key step lies in establishing the smoothness of the optimal critic variables shown in the next lemma.
282
284
Lemma 2 (smoothness of optimal critic). Suppose Assumptions 1-3 hold, under the update of Algorithm there exists a positive constant Lp,1 such that for all 0, 0', holds that
285
VV' (0) Vw" (0')l S 1 l'll.
w² where (a) denotes the Jacobian of (0) with respect to O.
This smoothness property is essential for achieving our Ō(I/VK) convergence rate.
288 To the best of our knowledge, the smoothness of w (+) has not been justified in the literature. Equipped with Lemma 2, we are able to establish the following lemma.
290
Lemma 3 (Error of critic). Under Assumptions 1-5, consider the update of Algorithm 1. Then, it holds that
- E| B|lla++1 w0 (00+001))11 < (1 - (0x)l1
aj
Elgh (Ex. w N 1111)1112 O(a.).
(14)
E -00x)1 < (1 Bx) |ا 3(()))l
+ Ck, Br 7x + -Cx Zx BE-
(15)
282 Here, Zx == minf2 E Imp=-= S min(ar, Be, nh }) Co. Xo are constants specified in appendix. and Ck, and Ck, are of order (I))((K)) and Olog" (log)(KK)) respectively.
8

---

Average Reward
Average Reward
S000--al 53AD-R0ISy SUNG
SEAC-AI
SDACenolty
SDAC
DLUMC
1c4
103
Sample Complexity
Communication Complexity
Figure I: Averaged reward versus sample complexity and communication complexity. The vertical axis is the averaged reward over all the agents.
Plug (15) into (14), we can establish the approximate descent property of ||w w² (0x)]l in (13);
294
E| @ - w" (0++1)11²] = (1 + C1an) - 22o3k))laa - w (0x)1l
N
ak
(EK wh+1 / 11=1
O + CK2Ok- Zx 8x)
(16)
Finally, plugging (1)), (14), and (16) into (13) gives the ascent of the Lyapunov function, which leads to our convergence result through steps of standard arguments.
5 Numerical results
298
In this section, our objective is to illustrate the empirical sample complexity and communication complexity of the proposed algorithms. We also implement the algorithm in [6] to serve baseline, which employs double-loop algorithmic framework. Our simulation is based on the grounded communication environment proposed in [19]; see Appendix A for detailed set up. Through the discussion, we refer the algorithm in [6] as "DLDAC", the Algorithm 1 as "SDAC-re", , the Algorithm "SDDCCnoisy" (see Appendix B). We also provide the result which assumes full reward is available to serve as baseline, which we refer as "SDAC-full". for "SDAC-re", "SDAC-noisy", and "SDAC-full". We choose T. 5 (loop size), TV (critic consensus number every iteration). T' = 5 (reward consensus number every iteration) for "DLDAC".
300
302
304
set K, = for "SDAC-noisy": K, =
306
The sample complexity and communication complexity are shown in Figure 1. The results averaged over Monte Carlo runs. As we can see, the proposed two algorithms achieve significantly higher reward than "DLDAC" in terms of both sample complexity and communication complexity. Moreover, their performances approach the baseline "SDAC-full" where the global reward is assumed to be available, indicating that the reward approximation is nearly accurate. Due to space limit, will put additional experiments on the comparison with existing decentralized AC algorithms and the Appendix A.
308
310
312
ablation study of
Conclusion and future direction
315
In this paper, we studied the convergence of fully decentralized AC algorithm under practical single- timescale update for the first time. We designed such an algorithm which maintains the optimal sample complexity of (c--2) under less communications. We also proposed a variant to preserve the privacy of local actions by communicating noisy rewards. Extensive simulation results demonstrate the superiority of our algorithms' empirical performance over existing decentralized AC algorithms.
316
317
319
One limitation of our work is that we only study the convergence stationary point. Thus, we leave the research on the avoidance of saddle points and convergence to global optimum as promising future directions.
321

---

References
323
324 [1] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference Learning Theory (COLT), pages 66, 2020.
326
[2] J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319-350, 2001.
328
329
[3] J. Bhandari, D. Russo, and R. Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on Learning Theory (COLT), pages 1691-1692, 2018.
330
332
[4] Chen, Y. Sun, and W. Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
333
334
335
[5] Z. Chen, Y. Zhou, and R. Chen. Multi-agent off-policy td learning: Finite-time analysis with near- optimal sample complexity and communication complexity. arXiv preprint 17X14 2021.
338
337
[6] Z. Chen, Y. Zhou, R.-R. Chen, and S. Zou. Sample and communication--ffiicnn decentralized actor-critic algorithms with finite-time analysis. In International Conference on Machine Learning, pages 3794-3834 PMLR, 2022.
339
341
[7] K. Doya. Reinforcement learning in continuous time and space. Neural Computation, 12(1):219- 245, 2000.
343
[8]
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
344
345
347
[9] Z. Fu, Z. Yang, and Z. Wang. Single-timescale actor-critic provably finds globally optimal policy. In International Conference on Learning Representations, 2021.
349
[10] H. Guo, Z. Fu, Z. Yang, and Z. Wang. Decentralized single-timescale actor-critic on zero- sum two-player stochastic games. In International Conference on Machine Learning, pages 3899-3909, PMLR, 2021.
350
351
352
[11] Hairi, J. Liu, and S. Lu. Finite-time convergence and sample complexity of multi-agent actor- critic reinforcement learning with average reward. In International Conference on Learning Representations, 2022.
354
[12] S. M. Kakade. A natural policy gradient. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 1531-1538, 2002.
356
357
[13] V. R. Konda and V. S. Borkar. Actor-critic learning algorithms for Markov decision
358
processes. SIAM Journal on Control and Optimization, 38(1):94-123) 1999.
359
[14] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning. arXiv preprint "XXiv:55990097/1 2015.
380
361
[15] Y. Lin, K. Zhang, Z. Yang, Z. Wang, T. Başar, R. Sandhu, and J. Liu, A communication--ffiiieen multi-agent actor-critic algorithm for distributed reinforcement learning. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 5562-5567, 2019.
383
[16] M. L. Littman. Markov games framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
365
[17] Y. Liu, K. Zhang, T. Basar, and W. Yin. An improved analysis of (variance-reduced)) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems, 13:7624-76366 2020.
367
10

---

389
[18] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critie for mixed cooperative-comppttiiee environments. Advances in neural information processing systems, 30, 2017.
370
371
372
[19] I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
373
374
[20] S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning, pages 2681 18900. PMLR, 2017.
376
377
[21] S. Qiu, Z. Yang, J. Ye, and Z. Wang. On the finite-time convergence of actor-critic algorithm. In Optimization Foundations for Reinforcement Learning Workshop at Advances in Neural Information Processing Systems (NeurIPS), 2019.
379
380
[22] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson. Qmix: Mono- tonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304. PMLR. 2018.
381
382
383
[23] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.006377 2017.
384
385
[24] H. Shen, K. Zhang, M. Hong, and T. Chen. Asynchronous advantage actor critic: Non- asymptotic analysis and linear speedup. ArXiv:2012. 2020.
[25] D. Silver, J. Schritwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676) 2017.
388
[26] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 1887-5896. PMLR, 2019.
391
[27] J. Sun, G. Wang, G. B. Giannakis, Q Yang, and 7. Yang. Finite-sample analysis of decentral- ized
394
learning with linear function approximation. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pages 4485-4495, 2020.
395
396
[28] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 1057-1063, 2000.
397
388
399
[29] J. N. Tsitsiklis and B. Van Roy. Analysis of cemporal-diffferentt learning with function approximation. In Advances in neural information processing systems (NIPS), pages 1075- 1081, 1997.
401
402
[30] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-33) 2019.
404
405
[31] Y. Wang, S. Zou, and Y. Zhou. Non-asymptotic analysis for two time-scale TDC with general smooth function approximation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9747-9758. Curran Associates, Inc., 2021.
407
408
409
[32] Y. F. Wu, W. Zhang, Xu, and Q. Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33:17617-17628, 2020.
410
[33] P. Xu, F. Gao, and Q. Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Proc. International Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
412
[34] T. Xu and Y. Liang. Sample complexity bounds for two timescale value-based reinforcement learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 811-819. PMLR, 2021.
415
11

---

417
[35] Xu, Z. Wang. and Y. Liang. Improving sample complexity bounds for (natural) actor- critic
418
algorithms. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
419
420
[36] T. Xu, Z. Yang, Z. Wang, and Y. Liang. Doubly robust off-policy actor-critic: Convergence and optimality, ArXiv:2102.11800 2021.
421
422
[37] C. Yu, X. Wang. X. Xu, M. Zhang, H. Ge, J. Ren, L. Sun, B. Chen, and G. Tan. Distributed multiagent coordinated learning for autonomous driving in highways based on dynamic co- ordination graphs. IEEE Transactions on Intelligent Transportation Systems, 21(2)735-774, 2019.
424
[38] S. Zeng. T. Chen, A. Garcia, and M. Hong. Learning to coordinate in multi-agent systems: A coordinated actor-critic algorithm and finite-time guarantees. ar preprint arXiv:21/0.05597, 2021.
427
[39] H. Zhang, W. Chen, Z. Huang, M. Li, Y. Yang. W. Zhang, and J. Wang. Bi-level actor-critic for multi-agent coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7325-7332, 2020.
430
432
[40] K. Zhang, A. Koppel, H. Zhu, and T. Başar. Global convergence of policy gradient methods to (almost) locally optimal policies. arXiv preprint arXiv:1906.0833 2019.
433
434
[41] K. Zhang, Z. Yang, and T. Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
435
438
437
[42] K. Zhang, Z. Yang. H. Liu, T. Zhang, and T. Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, pages 5872-5881. PMLR, 2018
438
Checklist
440
1. For all authors...
442
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
443
444
(b) Did you describe the limitations of your work? [Yes] The limitation is written in an equivalent form as future works in the conclusion section; see Section 6.
445
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We conduct research about the design and analysis of the fundamental actor-critic algorithm, which should not bring any negative societal impact.
447
449
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
450
451
2. If you are including theoretical results...
452
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
453
(b) Did you include complete proofs of all theoretical results? [Yes]
454
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
496
457
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
459
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
460
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
462
463
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
12

---

485
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
467
488
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
470
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
472
5.
If you used crowdsourcing or conducted research with human subjects...
473
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
478
477
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
478
13


Reviews:
[{"review_id": "jCdWzfulYm9", "reviewer": "Reviewer_fGDk", "summary": "This paper develops the finite-time analysis of a single-timescale multiagent actor-critic algorithms. The sample complexity is $O(\\epsilon^{-2}$, which matches with the sota. Many existing studies use approaches e.g., double loop, minibatch to simplify the analysis, which introduces additional parameters in practice, and thus are not preferred. This paper shows that a single time scale update can also achieve the same order of sample complexity for the problem of decentralized MARL.\n", "questions": "There is one paper in the literature also shows that a single-time scale update can also achieves the same sample complexity as those approaches using minibatch or double loop. Their algorithm does not need a double-loop structure not a mini-batch approach, and only uses one single sample at each time step. The analysis is novel in that they bound the tracking error, (or equivalently, the critic error in this paper) using the gradient norm of the objective so that they can obtain a tighter complexity bound. How does the approach in this paper compare to that one?\nNon-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation, NeurIPS 2021. \n\n", "limitations": "The algorithm in this paper requires the action to be globally observable, which may not be possible in practice. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The analysis in this paper is of more practical interest as it focuses on the single-timescale update rule, which does not involve additional tuning parameters e.g., batch size, inner loop iterations.\n\n2. The complexity of the single-time scale method matches with those double-loop, mini-batch methods in the literature.\n\n\nWeaknesses:\n\n1. This is not essentially a weaknesses, more like a suggestion. It shall be highlighted in the paper why those double-loop and minibatch approaches with inner loop iterations =1, minibatch size =1, fail to obtain a complexity of $O(\\epsilon^{-2}$; and what term needs a tighter bounds than those approaches?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VWv9b5N2EQ4", "reviewer": "Reviewer_hKdD", "summary": "The submission solves corporated MARL via decentralized AC. Specifically, the authors consider single timescale update of AC (i.e., the actor, critic, and reward estimation take stepsizes of the same order). In theory, the authors show that under linear function approximation, AC converges to a stationary point under i.i.d. and Markovian sample schemes, respectively. \n", "questions": "Questions:\n - A discussion on the communication frequency $K_c$ with the final rate of convergence would be interesting. The choice of $K_c$ seems only appear in the introduction (line 71) without much further explanation in the paper. \n - I checked other single timescale methods, especially ALSET [3] and STABLE (Chen, Tianyi, et al.). Compared with the two-timescale algorithm TTSA (Hong, Mingyi, et al.), both ALSET and STABLE seem to put much effort into a careful design of the update formulae, especially on the upper level (eqs (8)(9) in STABLE and eqs (9)&(10) in ALSET). And they claim that this is why they can achieve convergence under single timescale updates (sec 2.2 in (Chen, Tianyi, et al.)). In comparison, Algorithm 1 in the submission seems to be simple yet efficient, since it only consists of a TD(0) in the critic update and an advantage actor in the actor update. It will be great if the authors can explain more on such a comparison, which will definitely bring more intuition to the proof idea. \n - Can the authors also compare with [18] in terms of the proof technique used for single agent AC?\n\nMinor issues:\n - The definition of the function $\\omega^*$ seems first appear in the appendix (line 613) but is used before in the paper (line 145). \n\n - Line 206: $\\varphi(s_{|\\mathcal S|}, a_{|\\mathcal A|})$\n\n - Algorithm 1 line 4: $s_k \\sim \\mu_{\\theta_k}(\\cdot)$\n\n - Eq (5): should be $\\omega_{k+1}^i$ instead of $w_{k+1}^i$\n\n - The supp material seems to be a little bit different to the paper? E.g., reference [2] is different. \n\n\nChen, Tianyi, et al. \"A single-timescale method for stochastic bilevel optimization.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nHong, Mingyi, et al. \"A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic.\" arXiv preprint arXiv:2007.05170 (2020).\n", "limitations": "Yes. The authors can also have a more detailed comparison with the works that prove convergence to a saddle point / global optimality. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - Single timescale AC is widely adopted in practice but lack of theoretical understanding, even in the scenario of single agent with linear function approximation. Most theoretical works aiming to show convergence of AC end up considering two-timescale or bilevel updates, which ensure the approximate convergence of critic update before the update of actor. The submission not only consider the single-timescale AC, but also establish the convergence of such algorithm in MARL setting, which closes the gap between practice and theory for the first time, to the best of my knowledge. \n - As pointed out by the authors, previous works [8] use LSTD in critic update to study single timescale AC. The naive sample complexity of [8] is $O(\\varepsilon^{-4})$. Further, they propose to use off-policy to improve the sample complexity to $O(\\varepsilon^{-2})$, which matches the sample complexity in the submission. In comparison, the submission uses TD(0) in the critic update, which is more similar to the practical use. \n\nWeaknesses:\n - The submission only shows convergence to a saddle point, instead of an optimal point. Previous work [5] seems to have a stronger result showing optimality, though adopting a bilevel scheme. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GMqjCmayzy4", "reviewer": "Reviewer_T2LJ", "summary": "This paper studies a single timescale decentralized actor-critic algorithm. The setting is as follows. Each agent attain their respective action, and reward. There is also a global state which is observed by all the agents. Each of the agents attain a low low dimensional approximation of the rewards function, the value function, and the policy. At each time step the agents update these parameters using a single sample from the environment, either iid or Markovian. The authors show the sample complexity of their algorithm for both iid and Markovian sampling, and show that it attains the lower bound for such sample complexity. ", "questions": "- You did not define what is \\lambda_max in line 205. \n- Can you express the communication cost of the algorithm and compare it with the related work along with the sample complexity?\n- Why not characterize the convergence for natural actor critic as well? I believe establishing the result should be straightforward given the current results. Note that natural actor-critic can guarantee global convergence.", "limitations": "The main suggestion for improvement has already been suggested. I suggest to include the convergence of the natural actor critic as well as the communication cost of the algorithm.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- My main criticism of this work is that I do not see much advantages of the results in this paper over the one in [5]. The authors argue that they study a single timescale algorithm compared to [5] which studies a two timescale algorithm. However, I have two comments about that. First of all, I believe the result in [5] is not two timescale. In particular, the step sizes for both actor and critic are constant. The only issue with [5] might be the need to have an inner loop and an outer loop. But I do not see a significant problem with this structure. The reason is that the sampling is not affected during the move from inner loop and outer loop. In [5] the sampling is through a single trajectory throughout the performance of the algorithm. Can the authors explain why their results are better than [5], or why their algorithm has an advantage over [5]? I am willing to increase my score if this is addressed.\n- The main strength of the paper is a convergence bound which attains the lower bound up to logarithmic factors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject