Multi-Agent Off-Policy TDC with Near-Optimal Sample and Communication Complexities
Authors: Ziyi Chen
Yi Zhou
Rong-Rong Chen
Venue: Accepted by TMLR ()

Abstract: The finite-time convergence of off-policy temporal difference (TD) learning has been comprehensively studied recently. However, such a type of convergence has not been established for off-policy TD learning in the multi-agent setting, which covers broader reinforcement learning applications and is fundamentally more challenging. This work develops a decentralized TD with correction (TDC) algorithm for multi-agent off-policy TD learning under Markovian sampling. In particular, our algorithm avoids sharing the actions, policies and rewards of the agents, and adopts mini-batch sampling to reduce the sampling variance and communication frequency. Under Markovian sampling and linear function approximation, we proved that the finite-time sample complexity of our algorithm for achieving an $\epsilon$-accurate solution is in the order of $\mathcal{O}\big(\frac{M\ln\epsilon^{-1}}{\epsilon(1-\sigma_2)^2}\big)$, where $M$ denotes the total number of agents and $\sigma_2$ is a network parameter. This matches the sample complexity of the centralized TDC. Moreover, our algorithm achieves the optimal communication complexity $\mathcal{O}\big(\frac{\sqrt{M}\ln\epsilon^{-1}}{1-\sigma_2}\big)$ for synchronizing the value function parameters, which is order-wise lower than the communication complexity of the existing decentralized TD(0). Numerical simulations corroborate our theoretical findings. 

Published in Transactions on Machine Learning Research (06/2022)
Multi-Agen Off-Policy TDC with Near-Optimal Sample and Communication Complexities
Ziyi Chen
u12766772@uha.eel
Deportment of Electrical and Computer Engineering
University of Utah
Yi Zhou
yi.chon@@utak edu
Department of Electricol and Computer Engineering University of Utah
Rong-Rong Chen
rehen&ecce utah. edu
Department of Electricol and Computer Engineering University of Utah
Reviewed on OpenReview: /ttps://openreee opwrreiiee.........m net/ forum? id- tnPyQpyk7) Yk7D
Abstract
The finite-time convergence of off- off-policy temporal difference (TD) learning has been compre- hensively studied recently. However, such type of convergence has not been established for off-policy TD learning in the multi-agent setting, which covers broader reinforcement learning applications and is fundamentally more challenging. This work develops a decentralized TD with correction (TDC) algorithm for multi-agent off-policy TD learning under Markovian sampling. In particular, our algorithm avoids sharing the actions, policies and rewards of the agents, and adopts mini-batch sampling to reduce the sampling variance and communication frequency. Under Markovian sampling and linear function approximation, we proved that the finite-time sample complexity of our algorithm for achieving e-accurate solution is in the order of O M where M denotes the total number of agents and d2 network parameter. This matches the sample complexity of the centralized TDC. Morcover, our algorithm achieves the optimal communication complexity O the value function parameters, which order-wise lower than the communication complexity of the existing decentralized TD(0). Numerical simulations corroborate our theoretical findings.
(> for synchronizing
Introduction
Multi-agent reinforcement learning (MARL) is an emerging technique that has broad applications in control Yanmaz et al. (2017): Chalaki & Malikopoulos (2020), wireless sensor networks Krishnamurthy et al. (2088); Yuan et al. (2020), robotics Yan et al. (2013), etc. MARL, agents cooperatively interact with an environment. and follow their own policies to collect local rewards. In particular, policy evaluation a fundamental problem in MARL that aims to learn multi-agent value function associated with the policies of the agents. This motivates the development of convergent and communication-effiien multi-agent TD learning algorithms.
For single-agent on-policy evaluation (i.c., samples are collected by target policy), the conventional TD(0) algorithm [Sutton (1988) Sutton & Barto (2018) and Q-learning algorithm Dayan 1992) have been developed with asymptotic convergence guarantee. Recently, their finite-time (i.e., non aon-asympptoicc convergence has been established under Markovian sampling (i.c., the samples are obtained from Markovian decision process and thus are not i..d.) and linear approximation Bhandari et al. (2018): Zou al. (2019). However, these algorithms may diverge in the off-policy setting Baird 1995) where samples collected by different.

---

Published in Transactions on Machine Learning Research (06/2022)
behavior policy. To address this important issue, a family of gradient-based TD (GTD) algorithms were developed for off-policy evaluation with asymptotic convergence guarantee [Sutton et al. 2008 (2011). In particular, the TD with gradient correction (TDC) algorithm has been shown to have superior performance and its finite-time convergence has been established recently under Markovian sampling Xu
2009); Maei
et al.
p0199; Gupta et al. (2019) Kaledin et al. (020)).
For multi-agent on-policy evaluation, various decentralized TD learning algorithms have been developed. For example, the finite-time convergence of decentralized TD(0) was established with ii.d samples Wai et al. (2018):
: Doan et al. (2019) and Markovian samples (Snn et al. (2020), respectively, under linear function approximation, and an improved result is further obtained in Wang et al. (2020) by leveraging gradient tracking. However, these algorithms do not apply to the off-policy setting. In the existing literature, decentralized off-policy TD learning has been studied only in simplified settings, for example, agents obtain independent MDP trajectories Macua et al. (2014) Stanković & Stanković (2016); Cassano et al. (2020) or share their behavior and target policies with cach other Cassano et al. (2020), and the data samples are either i.idd or have a finite sample size. These MARL settings either are impractical or reveal the agents' policies that may be sensitive. Therefore, we want to ask the following question:
Q1: Can we develop a decentralized off-policy TD algorithm for MARL with interdependent agents and avoids sharing local actions, policies and rewards of the agents?
In fact, developing such a desired decentralized off-policy TD learning algorithm requires overcoming two major challenges. First, to perform dccentralized off-policy TD learning, all the agents need obtain a global importance sampling ratio (see Section 3.2). In Cassano et al. (2020), the authors obtained this ratio by sharing all local policies among the agents, which may lead to information Icakage. Thercfore, we aim to develop saler scheme to syuchronize the global importance sampling ratio among the agents without sharing any sensitive local information. Second, the existing decentralized TD-type algorithm achieves a /M
communication complexity (mumber of communication rounds) of O(c-' --( v((1 Inc" for networks
with M agents and parameter 02 (See Assumption 5 in Section 4 for the definition of σ²) Sun et al. (2020)
This induces much communication overhead when the target accuracy e is small. Hence, we want to ask the following theoretical question:
Q2: Con we develop a decentralized off-policy TD learning algorithm that achieves a near-optimal finite-time sample complerity and near-optimal communication complerity under Markovian sampling?
In this work, we provide affirmative answers to these questions by developing decentralized TDC algorithm that avoids sharing the sensitive information of the agents and achieves the near-optimal sample complexity as well as a significantly reduced communication complexity. We summarize our contributions as follows.
1.1 Summary of Contribution
To perform multi-agent off-policy evaluation, we develop decentralized TDC algorithm with linear function approximation. In every iteration, agents perform two-timescale TDC updates locally and exchange model parameters with their neighbors. In particular, our algorithm adopts the following designs to avoid sharing agents' local sensitive information and reduce communication load.
We let the agents perform local averaging on their local importance sampling ratios to obtain approximated global importance sampling ratios.
All the agents use mini-batch of samples to update their model parameters in each iteration. The mini-batch sampling reduces the sampling variance and the communication frequency, leading to an improved communication complexity over that of the existing decentralized TD(O).
After the decentralized TDC iterations, our algorithm performs additional local averaging steps to achieve a global consensus on the model parameters. This turns out to be critical for achieving the near-optimal complexity bounds.
Theoretically, we analyze the finite-time convergence of this decentralized TDC algorithm with Markovian samples and show that it attains fast linear convergence. The overall sample complexity for achieving
2

---

Published in Transactions on Machine Learning Research (06/2022)
( When there is a single agent (M 1), this sample complexity result matches that of centralized TDC Xu et al. (2019) and matches the theoretical lower bound O(e-)) Kaledin et al. (2020) up to a logarithm factor. In addition, the sample complexity is proportional to M, which matches the theoretical lower bound of decentralized strongly convex optimization [Scaman et al. (017)). Morcover, the communication complexity of our algorithm for synchronizing value function parameters is in the order of O (Y which is significantly lower than the communication complexity /M I)) of the decentralized TD(0) Sun et al. (2020) and matches the communcattin O ve[1-oz]
e-accurate solution is in the order of O Af lne
complexity lower bound Scaman et al. (2017).
Technically, our analysis is a nontrivial generalization of the analysis of centralized off-policy TDC to the decentralized case. In particular, our analysis establishes tight bounds of the consensus error caused by synchronizing the global importance sampling ratio, especially under the Markovian sampling where the data samples are correlated, Moreover, we strategically bound the estimation error of the global importance sampling logarithm-ratio. Please refer to the proof sketch at the end of Scction for details.
1.2 Other Related Work
Centralized policy evaluation. (0) with linear function approximation Sutton 1988 is popular for
on-policy evaluation. The asymptotic and non- -on-asymptotic convergence results of TD(0) have been established
in Sutton 1988); ; Dayan [992); Jaakkola et al. 1993); Gordon [195)] Baird (1995 Tsitsiklis & Van Roy
1997 | Tadić (2001); Hu & Syed (2019) and Korda & La 2015 Liu et al. (2015); Bhandari et al. (2018)
Dalal al. (2018b): Lakshminarayanan & Szepesvari (2018): ; Wang et al. (2019); Srikant & Ying (2019): : Xu
ct al.] (0020b) respectively. Sutton et al. (2009) proposed TDC for off-policy evaluation. The finite-sample
convergence of TDC has been established in Dalal et al.
20188 2020 with i.i.d. samples and in Xu et al.
(2019) Gupla et al. (019); Kaledin el al. 2020) with Markovian samples.
Decentralized policy evaluation. Mathkar & Borkar (0166) proposed the decentralized TD(0) algorithm.
The asymptotic and non-asymptotic convergence rate of dccentralized TD have been obtained in Borkar (2099) and (Sun et al. (2020); Wang et al.] (2020) respectively. Exisitng decentralized off-policy evaluation studies considered simplified settings. |acua et al. 2014];; SStnković & Stankovid (2016) obtained asymptotic result for decentralized off- off-policy evaluation where the agents obtained independent MDPs. Cassano et al. (2020) obtained linear convergence rate also with independent MDPs by applying variance reduction and extended to the case where the individual behavior policies and the joint target policy are shared among the agents.
Decentralized policy control. Decentralized policy control is also an important MARL problem where the goal is to learn the optimal policy for each agent. Many algorithms have been proposed for decentralizod policy control, including policy gradient Chen al. (2021) and actor-critic Qu et al. 2020): Zhang al. 20211.
2 Policy Evaluation in Multi-Agent RL
In this section, we introduce multi-agent reinforcement learning (MARL) and define the policy evaluation problem. Consider a fully decentralized multi-agent network that consists of M agents. The network topology is specified by undirected graph g = (M,E), where M 21,2. ... .M} denotes the set of agents and & denotes the set of communication links. In other words, each agent m only communicates with its neighborhood Nm, := {m' E M:: (m, m') E E}} In MARL, the agents interact with a dynamic environment through a multi-agent Markov decision process (MMDP) specified as {S, (A((m) 1 (R(m) MM To elaborate, S denotes a global state space that is shared by all the agents, A(m) corresponds to the action space of agent m, P is the state transition kernel and Rim) denotes the reward function of agent m. All the state and action spaces have finite cardinality. 7 E (0, 1] is a discount factor.
At any time t, assume that all the agents are in the global state st E S. Then, each agent m takes certain action ((m) E (m)) following its own stationary policy N T(m),, i.c., (m) ~ 7 (-Iss)) After all the actions are taken, the global state transfers to new state 81+1 according the transition kernel P, i.e. 8841 P(. at)
3

---

Published in Transactions on Machine Learning Research (06/2022)
where de {a, (m) J mm=1- At the same time, each agent m receives a local reward R2(m) = R (st, dty Se+1) from the environment for this action-state transition. Throughout the MMDP, each agent has access to only the global state { and its own actions {a+ (a]")) l and rewards {R(""}, The goal of policy evaluation in MARL is to evaluate the following value function associated with all the local policies {i (m()) } m== any global state
M for
M R R(mi)) m=
(*(()) =
8,
(1)
M
t=0
A popular algorithm for policy evaluation in MARL is the decentralized TD(0) Sun et al. 20200. Specifically, consider a popular linear function approximation contains the model parameters and (bs) is a feature vector that corresponds to the state s. In decentralized TD(0), cach agent m collects a single Markovian sample {s, (m)) 88411, R, R(()) at time & (a. ~ (-Iss)
the value function (s(s) == ((s), where gd
(m))
8++1 ~ P(-(st,at) at); R(m) = (sm) at, 84+1)) and updates own model parameters a(Th) with learning rate a >> 0 follows.
Σ my E...
Um.m a(Ai0) + b((m)).
(2)
where U corresponds to doubly stochastic communication matrix, each agent only communicates with neighborhood Nm: and A, = (()))(r((++++)) - d(s.)), 16(m) R ((s)). The above update rule applies the local TD error to update the parameters and synchronize the parameters among neighboring agents through the network. Tt can be inferred that mmm²)) obtained by this decentralized TD(0) algorithm is 6-close to the optimal solution with both sample complexity (the number of required samples) and communication complexity (the number of communication rounds) being O ((e-l+v/ll VM/e(11 - o2) o2)_-))n((e"))) In(e=)).
3
Two- Timescale Decentralized TDC for Off-Policy Evaluation
3.1 Centralized TDC
In this subsection, we review the centralized TD with gradient correction (TDC) algorithm Sutton al. (2009). In RL, the agent may not have enough samples that are collected following the target policy m. Instead, it may have some data samples that are collected under different behavior policy тre Therefore, in this oof-po setting, the agent would like to utilize the historical data to help evaluate the value function va associated with the target policy π.
In Sutton et al. (2009), family of gradient-based TD (GTD) learning algorithms have been proposed for off- off-policy evaluation. In particular, the TDC algorithm has been shown have superior performance. To explain, consider the linear approximation Vo(s) 0 ((s) and suppose the state space includes states $1...., Sn, define total value function Vo (o(ss), .... Vo(s.)) The goal of TDC is to minimize the following mean square projected Bellman error (MSPBE).
MSPBE(() := E,,, IIV - I17 1/
where 14 is the stationary distribution under the behavior policy Th, T is the Bellman operator and I(V)= := arg miny, Vi 2 a projection operator of any state value function S - R onto the space of linear value functions (Vo : Vo(s) = 0 (()).. Given the i-th sample (8), 0i, 8141, R&) obtained by the behavior policy, we define the following terms
P: = x(a
b, b p(Ro(ss). B; = 77:00000
A = -
Tn(a; ):)
C, C==-d(:)d(s))
(3)
Sun al. 2020) does not report sample complexity and communication complexity, so we calculated them based on their finite-time error bound proposition
4

---

Published in Transactions on Machine Learning Research (06/2022)
where Pi is referred to the importance sampling ratio. Then, with learning rates B > 0 and initialization parameters 0o, wo, the-two timescale off-policy TDC algorithm takes the following recursive updates for iterations f = 2,,,..
{
0c+1 0.+ a(A,0, b + B,w+), W++1 wy + ((A,0, b, Cew.)-
(TDC):
(4)
Xu et al. 2019 Xu & Liang 2020)
study slight variations of the above TDC algorithm by using projection and minibatch technique respectively, and obtain that both variants obtain an c-approximation of the optimal model parameter 0" -b- (A = Ex,[A,], b == Er, Bx]) with sample complexity O( ((e)))
3.2 Decentralized Mini-batch TDC
In this subsection, we propose decentralized TDC algorithm for off-policy evaluation in MARL. In the mulli-ageent setting, without loss of generality, we assume that each agent m has a target policy N (1m) and its samples are collected by a different behavior policy Tb"", In particular, agent m is on-policy, then we have %b (m) = (m)) In this mulli-agent off-policy setting, the agents aim utilize the data collected by the behavior policies T6 { π = (m(m)) J
(mm) p to help evaluate the value function Va associated with the target policies
However, directly generalizing the centralized TDC algorithm to the decentralized setting will encounter several challenges. First, the contralized TDC in oq
(4) consumes one sample per-iteration and achieves the sample complexity Olg-l Xu al. 2019 Therefore, the corresponding decentralized TDC would perform one local communication per-iteration and is expected to have communication complexity in the order of O log )) which induces large communication overhead. Second, in the multi-agent off-policy setting, every agent m has a local importance sampling ratio p; ((m) := x (m) (a(") [s:)/ mm/ (mm) (m) (s:). [s).
(a,
However, correctly perform off-policy updates, every agent needs to know all the other agents' local
importance sampling ratios in order to obtain the global importance sampling ratio p := I ("))
To address these challenges that are not seen in decentralized TD(0) and centralized TDC, we next propose decentralized TDC algorithm that takes mini-batch stochastic updates.
To elaborate, note that Pi be rewritten as
P: exp M EM In p, (m))
Therefore, all the agents just need obtain the average 1 Em= In pÚ (m) which can be computed via local
communication of the logarithm-ratios {In Pi (m) MM Im=1 for L rounds. Specifically, every agent initializes (m) Pi.o = In p3 and for iterations = 0,.., T 1 do
Pi,t1
Σ
Um Amm)
(5)
m'eNm
(Output): : p; = exp(M PiL
(6)
In Corollary (see the appendix), prove that all of these local estimates (p(m)) 1 converge exponentially fast to the desired quantity As as L increases. Then, every agent m performs the following two-timescale TDC updates
Jm=1
(1+1)N-1
(fm))
U U
Olmm)
(A (m)) g()
+ 7(m)) + B(()) w (m))
(7)
6+1
N
m E.N....
I=N
(++1)N-1
(m) W
Un (m')
3
Σ (A (m) p(m) + mmm) + C.u
(8)
2+11
N
DY E.
EN
5

---

end end Published in Transactions on Machine Learning Research (06/2022)
where A!m)) B( B/()) are defined by replacing the global variables Pi and R2 involved in As, Bi, ((())) (see eq. 33) with local variables p and Iz(m) respectively. To summarize, every TDC iteration of Algorithm consumes N Markovian samples, and requires two vector commuiccttin rounds for synchronizing the parameter vectors m!m)) (m)) and L scalar communication rounds for estimating the global importance .t
sampling ratio ((pie) : I...... (t++ 1)N 1.m E M) are shared in the l-th communication round). We summarize these update rules in Algorithm Moreover, after the decentralized TDC updates, the agents perform additional T local averaging steps to reach global consensus the model parameters.
Algorithm Decentralized mini-batch TDC.
Input: Batch size N, iterations T,T', learning rates R, Initialize: ppm) wo for all agents m E M.
for ileration l 0, ,.,,,, - do Each agent collects N Markovian samples and computes their local importance sampling ratios p. for communication round (= 0.1 do for agent m E M in parallel do
((m Communicate Piçe via eq.
5 for the N samples i = LN,..., (1 1)N -. 1.
for agent m E M in parallel do
(m) Agent estimates global importance sampling ratios PÚ via eq.
6 for tt,...., (t+ 1)N - 11
and then performs the updates in eqs. and ( end end
for iteration t t=
T - do
for agent m E M in parallel do m(m) Em'END m 0
end end
Output: (0/4// }
4 Finite-Time Analysis of Decentralized TDC
In this section, we analyze the finite-time convergence of Algorithm I Denote pr, as the stationary distribution of the Markov chain { induced by the collection of agents' behavioral policies Tg. Throughout the analysis, we define the following notations.
A = Em [A B = EN Bi], C: Ex, Cil, b, == 1 Em (mm) b: =E [b,],
0. = M M
m= ml))) 8" = A A-1 'b, wi = = (AB: + b),
where Fy denotes the expectation when 8, Pru ((m)) ~ ni (m) (m))(s) and 8141 P(-|st, at), A,, B,, C are defined in eq. (3) with exact global importance sampling ratio Pi, 0r the optimal model parameter, and wt is the optimal auxiliary parameter corresponding to f, It is well-known that the optimal model parameter is 0" = - 1b Xu et al. (2220b) Xu & Liang (2020). We make the following standard assumptions.
Assumption 1. There crist constants and 8 (0, 1) such that for all t 0,
supdrv- (Prr (st 80 = 8) .MT) S v8¹ sES
(9)
where dry denoles the variatarra distance.
Assumption 2. The matrices A and C are invertible.
Assumption 3. The feature vectors satisfy [|o(s)] S 1, Vs.

---

Published in Transactions on Machine Learning Research (06/2022)
Assumption 4. There erist Rmax, Puax 0 such that for all m M: m R(m) (s,as, < Rauax and g,0mm) plm)) (s, ()) < Piax where plm)) (8, alm) ratio..
x-(m) [a) denotes the global importance sampling x/ (a(m))))
Assumption 5. of each column sum up to 1.) and its second largest singuler value satisfies 02 [0,1).
The communication matrix U is doubly stochastic, (f.e., the entries of each row and those
Assumption has been widely adopted in the existing literature Bhandari et al. 2018 Xu el al. (2019);
Xu & Liang (2020); Shaocong et al. (2020: 2021). Tt holds for all homogeneous Markov chains with finite
state-space and all uniformly ergodic Markov chains. Assumptions 2
4 Pare widely adopted in the analysis (2019; Xu & Liang (2200 In particular, Assumption 2 implies that
of TD learning algorithms Xu et al.
A == Amax CACC-AA > 0, >==== Amax CC) > where Amax (C) denotes the largest eigenvalue of matrix C.
Assumption
g can always hold by normalizing the feature vectors. Assumption 4| holds for any uniformly
lower bounded behavior policy, i.e., inf (n..) "b ((m")(s.. (8,,a)= >> 0, which ensures that every state-action pair (8, ()
is visited infinitely often. Assumption 5 is standard in decentralized optimization [Singh et al. (2020); Saha
ct al. (2020) and TD learning [Sun al. (0200:: Wang et al. (2020). dg an important mcasure that reflects communication topology. For example, densely connected network lends to have smaller 02 than sparse network.
We obtain the following finite-time error bound as well as the sample complexity (the number of required samples) and communication complexity (the number of communication rounds) for Algorithm with Markovian samples.
Theorem 1. Let Assumptions hold. Run Algorithm 7 for T iterations with learning rates a (iB)) O 1),,,, 3 S O(),, batch size 2 (O { and L O 1I M+M In Pems
(see eq. (35]
In
38) for the full expressions ). Then, we have the following convergence rate (see eq.
288 for its Full
expression)
-110--001191 - 0-119] S (1-22))
61) (100 0-112 + 1 will²)
0
B
Bo
1/127
(10)
Na
M
Furthermore, after I" iterations of local averaging, the local models of all agents m = I,..., M has the following consensus error (see eq.
33 for its full expression)
B B[ (m)
- ত
< <0² पtt²" 0(1 (1 02) [|]
M
M MBao
2/2
(11)
1 1-02 03
Consequentlyy, by choosing = 01 M,,, T =0=0001-- - In (1-0a)) (See the end of Appendiz B for the full expressions of these hyperperumeters) we obtain that E(110)T)) 112) S for all m. The overall communication complerities for synchromizing mlm)) and imporance sampling ratio p respectively T T" O ((MMa and TL L/M Ine- + M ). The total sample complexity is NT 0 Inc d(1 (010-o2)))
0(1,, T 0 (MM N = 0 ((1)0)))
I. = O VM In
(1-a2)²
(1-02=
(1 (1=0))=) In
The above theorem shows that our decentralized TDC achieves the sample complexity O Mine ((11-))))) which, in the contralized setting (M 1, d2 = 0), matches (nc--Ine-)) of centralized TDC for Markovian samples Xu et al. (2019); Xu & Liang (2020) and matches the theoretical lower bound O(E=1) given in Kaledin et al. (2020) up to logarithm factor. In addition, the sample complexity is proportional to M,, which matches the theoretical lower bound of decentralized strongly convex optimization in /caaaa et al. (017)) Importantly, the communication complexity O / /Mine for syuchronizing ofT*) is substantially lower than the communication
1-
complexity ((e-++ Ve(1-or 12) of receeed (D()) Sun et al. (2020) 2 Intuitively, this is because our algorithm adopts mini- batch sampling that significantly reduces the communication frequency, since
VTT
2Since on-policy evaluation does not involve importance sampling ratio p. we only compare the communication complexity synchronizing m!()) which involved in both on-poliry and off- off-policy evaluation.
7

---

Published in Transactions on Machine Learning Research (06/2022)
communication occurs after collecting mini-batch of samples to compute the mini-batch updates. Moreover, the communication complexity has a logarithm dependence ln on the target accuracy, and this matches the theoretical lower bound of decentralized strongly convex optimization in Scaman et al. (2017). Note that both communication complexity and sample complexity decrease with smaller 02 > 0. Hence, the communication matrix U can be selected with minimuu possible 02, under network topology constraint in practice. In addition, our stepsizes a = 0 (M7) and 3 = (1) do not scale with c. Although a can be small when 02 1 and M is large, it is much larger than min [O(c), (((c) -2))] in decentralized TD (0) Sun et al. (2200) with small
Taking a deeper look, Theorem 1 shows that the average model Br converges to small neighborhood of the optimal solution 6" at a fast. linear convergence rate (10) that matches the convergence rate of
centralized TDC Xu et al. (019)) ; Xu & Liang (2020). In particular, the convergence crror is in the order of
|0 2/2""
M which can be driven arbitrarily close to zero by choosing a sufficiently large mini-batch size N and communication rounds I. (with T fixed), and choosing constant-level learning rates a, f. Moreover, the T" steps of extra local model averaging further help all the agents achieve a small consensus error at a linear convergence rate [1]. Eqs. (10) and (1I) together ensure the fast convergence of all the local model parameters. We want to point out that the T" local averaging steps are critical for establishing fast convergence of local model parameters. Specifically, without the T local averaging steps, the consensus error B[ - 11] would be in the order of at least O(1), which constant-level and hence cannot guaranlee the local model parameters converge arbitrarily close to the true solution.
8 O(Na No +
Proof sketch of Thcorem 1 The proof of the theorem is nontrivial generalization of the analysis of centralized off-policy TDC to the decentralized case. Below, we sketch the technical proof and elaborate on the technical novelties.
We first consider an ideal case where the agents can access the exact global importance sampling ratio Pr at iteration L. In this ideal case, every agent m can replace the estimated global importance sampling ratio ((") involved in ((m) B(")) 8(m) in the update rules 7 and (8) by the exact value Pr
Step 1.
(so A(m)) B((m), D.m) bocome As, B,, ((m²) respectively). Then, with the notations defined in Table in
Appendix A the averaged update rules 14] and [15] become
0.11 +00++a(A1) + by + B,W.),
(12)
wo+1 = + 3(, + br + C.w.),
(13)
This can be seen one step of centralized TDC. Hence, we can bound its optimization error terms [[][W++1
-will [|]] and E [10e+1 - 13].
(by bounding w first) and [[1 0+ [2]. This is done by bounding the gap between the centralized updates 12 and 13 (with exact Pt) and the decentralized updates (14g and (15) (with inexact pt). The
Step 2. We return to Algorithm D lad bound its optimization error terms [[[
.
E [[[W4+1
key is to establish Corollary 2, which strategically controls the gap between the inexact global importance sampling ratio p!!)) and the exact value Pr. Such challenge in bounding the importance sampling ratio gap is not seen in the analysis of decentralized TD(0) and centralized TDC. (mm) in eq. 5 exponentially To elaborate, note that the locally- importance sampling ratios Pt. converges to the value In pt. However, the initial gap In p(m)) - In pel can be numerically large since pm)) may be numerically small positive number. To avoid such divergence issue, our proof discusses two complementary cases. Case 1: the quantity Punin minmE.
(m) p( E ///2 .Pmax]- In this case, the proof is straightforward as the initial gap is bounded. Case 2: the quantity Panin E (0, 02 1//2/. In this case, we show _(m) that the locally-averaged logarithm-ratio PiLL is below a large negative number (1" Puiin < (See eq- (m) 65 which implies that both the global importance sampling ratio Pr and its estimation Py are close 3a min (o())
(1(1 -o2)]] is obtained by letting the convergence rate in proposition 2 from Sun et al. 2020 to be
smaller than
8

---

Published in Transactions on Machine Learning Research (06/2022)
to zero. In both cases, (Pt (m)) JM (
converge exponentially fast Pr as L increases. This prove eq- 10 To the best of our knowledge, this technique for bounding the estimation error of the global importance sampling ratio has not been developed in the existing literature.
Step 3. Finally, prove the consensus error 11 Although the consensus error exponentially decays during the T" extra local average steps in Algorithm 1 it is non-trivial to bound the initial consensus error || of the T" local average iterations (see eq. (34) which is caused by the I" decentralized TDC steps. To bound this error, note that each decentralized TDC step consists of both local averaging and TDC update, which makes the consensus error || diminishes geometrically fast with a noise term Σm=1 ||/m]] (see eq. (30p). Such a noise term is induced by the TDC update and hence its bound depends on both the consensus error and the model estimation error in eq. (10). We need to apply these correlated bounds iteratively for iterations to bound the initial consensus error || |F.
5 Experiments
5.1 Simulated Multi-Agent Networks
We simulate multi-agent MDP with 10 decentralized agents. The shared state space contains 10 states and each agent can take 2 actions. All behavior policies uniform policies (i.e., each agent takes all actions with equal probability), and the target policies are obtained by first perturbing the corresponding behavior policies with Gaussian noises sampled from (0, 0.05) and then performing a proper normalization. The entries of the transition kernel and the reward functions are independently generated from the uniform distribution on [o, 1] (with proper normalization for the transition kernel). We generate all state features with dimension 5 independently from the standard Gaussian distribution and normalize them to have unit norm. The discount. factor is y 0.95.
We consider two types of network topologies: fully connected network with communication matrix U having diagonal entries 0.8 and olf-diagonal entries 1/45, and ring network with communication matrix U having diagonal entries 0.8 and entries 0.1 for adjacent agents. We implement and compare two algorithms in these networks: the docentralized TD(0) with batch size N = 1 (used in Sun et al. (2020) ) and our decentralized TDC with batch sizes N = 1, 10, 20, 50, 100. Note that the original decentralized TD(0) is simply eq. (7) with setting w/(T) = 0 and p. = 1 in the definition of A!TT), B((m) and T,!T), which only works for on-policy evaluation. To adapt decentralized TD()) to off-policy evaluation, we simply use (()) computed by eqs.
and 6 with 3. 3.
Effect of Batch size: We test these algorithms with varying batch size N and compare their sample and communication complexities. We set learning rate = 0.2 for the docentralized TD(0) and 0.2 * N, B = 0.002 * N for our decentralized TDC with varying batch sizes N = 1, 10, 20, 50, 100. Both algorithıns use communication rounds for synchronizing P p(m)) All algorithms are repeated 100 times using fixed set of 100 MDP trajectorics, each of which has 20k Markovian samples.
We first implement these algorithms in the fully connected network. Figure 1 plots the relative convergence error 100, 11/1010'' v.s. the number of samples (t.V) and the number of communication rounds (t) For each curve, its upper and lower envelopes denote the 95% and 5% percentiles of the 100 convergence errors, respectively. It can be seen that our decentralized TDC with different batch sizes achieve comparable sample complexity to that of the decentralized (D)),, demonstrating the sample-officiency of our algorithms. On the other hand, our decentralized TDC requires much less communication complexities than the decentralized TD(0) with N > 10 and the required communication becomes lighter as batch size increases. All these results match our theoretical analysis well.
We further implement these algorithms in the ring network. The comparison results are exactly the same as those in Figure 1 since the update rule of 0g does not rely on the network topology under exact global importance sampling.
Effect of Communication Rounds: We test our decentralized TDC using varying communication rounds
I. = 1,3, 5, 7. We use fixed batch size N 100 and set learning rates 5, B = 0.05, and repeat each
9

---

Published in Transactions on Machine Learning Research (06/2022)
1.00
1.0
Decentralized TDC(N=1)
Decentralized TDC(N=1)
Decentralized TDC(N=10) 10)
Decentralized TDC(N=10)
0.75
0.8
Decentralized TDC(N=20)
Decentralized TDC(N=20)
Decentralized TDC(N=50)
0.6
Decentralized TDC(N=50)
0.50
Decentralized TDC(N=100)
Decentralized TDC(N=100)
Decentralized TD(0)N==1)
0.4
(ecentralizedD(((NNN-]
1|0:
0.25
0.2
0.00
5000
10000 15000
2000C
0
50
100
150
200
number of samples
number of communication rounds
Figure 1: Comparison between decentralized TDC with varying batch sizes and decentralized TD(0).
algorithm 100 times using the set of 100 MDP trajectories. We also implement the decentralized TDC with exact global importance sampling ratios as a baseline. Figure 2 plots the relative convergence error v.s. the number of communication rounds (t) in the fully-connected network (Left) and ring network (Right). It can seen that in both networks, the asymptotic convergence error of the decentralized TDC with inexact, p decreases as the number of communication rounds for synchronizing the global importance sampling ratio increases. In particular, with L 1, decentralized TDC diverges asymptotically due to inaccurate estimation of the global importance sampling ratio. As L increases to more than 5, the convergence error is small that under exact global importance sampling.
Fully connected network
Ring network
1.00
1.00
Decentralized TDC(exact
Decentralized TDC(exact p)
Decentralized TDC(L=1)
Decentralized TDC(L=1)
0.75
Decentralized TDC(L=3)
0.75
Decentralized TDC(L=3)
Decentralized TDC(L=5)
Decentralized TDC(L=5)
Decentralized TDC(L=77)
Decentralized TDC(L=7)
0.50
0.50
0.25
0.25
0
50
100
150
200
0
200 number of communication rounds
50
100
150
number of communication rounds
Figure 2: Effect of communication rounds L on asymptotic convergence error.
We further plot the maximum relative consensus error among all agents max. [10(!)) 0 0,11/||0"|| v.s. the number communication rounds (t) in the fully-connected network (Left) and ring network (Right) in Figure 3 where the tails both figures correspond to the extra T" 20 local model averaging steps. In both networks, one can see that the consensus error decreases as L increases, and the extra local model averaging steps are necessary to achieve consensus. Moreover, it be that the consensus errors achieved in the fully connected network slightly smaller than those achieved in the ring network, as denser connections facilitate achieving the global consensus.
5.2 Two-Agent Cliff Navigation Problem
In this subsoction, we test our algorithms in solving a two-agent Cliff Navigation problem Qiu et al. (2021) in grid-world environment. This problem is adapted from its single-agent version (see Example 6.6 of Sutton
10

---

Published in Transactions on Machine Learning Research (06/2022)
Fully connected network
Ring network
Vivllle * Ill
Decentralized TDC(exact p)
Decentralized TDC(exact p)
|axi -0:ll/ile
Decentralized TDC(L=1)
Decentralized TDC(L=1)
0.2
Decentralized TDC(L=3)
0.2
Decentralized TDCIL=3)
Decentralized TDC(L=5)
Decentralized TDCIL=5)
Decentralized TDC(L=7)
Decentralized TDC(L=7)
0.1
Local averaging starts
0.1
Local averaging starts
TPT rer T
0.0
0.0
0
50
100
150
200
0
50
100
150
200
number communication rounds
number of communication rounds
Figure 3: Effect of communication rounds consensus
S
x
X
D
Figure 4: Two-agent cliff navigation. ("S", "X", "D" donoto starting point, cliff and destination respectively.
The optimal path shown in red.)
Decentralized TDC(N=1)
Decentralized TDC(N=1)
Decentralized TDC(N=10)
Decentralized TDC(N=10)
0.04
Decentralized TDC(N=20)
0.04
Decentralized TDC(N=20)
Decentralized TDC(N=50)
Decentralized TDC(N=50)
Decentralized TDC(N=100)
Decentralized TDC(N=100)
0.02
Decentralized TD(0)(N=1)
0.02
Decentralized TD(O)N=1)
0.00
0.00
5000
10000 15000
20000
50
100
150
200
number of samples
number of communication rounds
Figure 5: Results on two-agent cliff navigation problem.
& Barlo 2018)). ). As illustrated in Figure 4 two agents start from the slarting point "S" on a 3 x grid and aim to reach the destination "D)". Here, global state is defined as the joint location the two agents, and there are total (3 4)² 144 global states. In most states, an agent can choose move up, down, left right by one step and receives -1 reward. However, once an agent falls into the cliff "X", it will return the starting point "S" and receive - 100 reward. When agent reaches "D", it will always stay at "D", and
11

---

Published in Transactions on Machine Learning Research (06/2022)
receives O reward if the other agent also reaches/stays at "D", receives -0.5 reward otherwise. If an agent is not at "X" or "D" and selects direction that points outside the grid, then it stays in the previous location and receives -1 reward.
We apply the aforementioned algorithms with different batchsizes N to solve this problem. The hyperparam- eters and the ways to generate behavior policy and target policy are the same as the previous simulation experiment, except that the communication matrix U has diagonal entries 0.7 and off-diagonals 0.3. All algorithms repeated 100 times using fixed set of 100 MDP trajectories, each of which has 20k Markovian samples. Figure plots the relative convergence error ||0 0* [1/][0] v.s. the number of samples (tN) and the number of communication rounds (t) We can see that compared with the decentralized TD(0), our decentralized TDC achieve comparable sample complexities with different batch sizes and much lower communication complexities with N 2 10 Moreover, the required communication becomes lighter as batch size increases. These properties are similar those shown the simulation and thus have generality.
X
X
X
D1
D2
D3
X
X
X
X
X
X
X
X
X
X
X
X
X
S1 S2 S3 X
X
Figure 6: The map for path finding problem. ("S1", "S2", "S3" denote the starting points of the 3 agents. "D1", "D2", "D3" denote their destinations. "X" denotes an obstacle.)
value function target state
value function of target state
(enttralized-TDCNNN]]
Decentralized TDC(N=1)
Decentralized TDC(N=10)
Decentralized TDC(N=10)
40
Decentralized TDC(N=20)
40
Decentralized TDC(N=20)
Decentralized TDC(N=50)
Decentralized TDC(N=50)
Decentralized TDC(N=100)
Decentralized TDC(N=100)
20
20
Decentralized TD(0)(N=1)
Decentralized TD(O)N=1)
0
2000 4000 6000 8000 10000 number of samples
20
40
60
80
100
number of communication rounds
Figure 7: Results on path finding problem.
12

---

Published in Transactions on Machine Learning Research (06/2022)
5.3 Application to Path Finding Problem
In this subsection, we test our algorithms in solving multi-agent path finding problem Ma el al. (0021 which has broad real-world applications including aircraft-towing vehicles Morris et al. (016), ware- house
and office robots Wurman ct al. (2008); Veloso et al. (2015), and vidco games (ilver (2005); ; Ma et al. (2017). As illustrated in Figure 6 three agents start fron the points "S1", "S2", "S3" on a 10 x 10 grid and aim to reach the destination "D1", "D2", "D3", respectively. In most cases, an agent can choose to move up, down, left or right by one step or stay and receives -0.075 reward. However, when an agent reaches its destination, it always stay and receives 3 reward if all the agents reach their destinations, or receives 0 reward otherwise. If an agent has not reached its destinatiion and collides with an obstacle "X" or another agent, it receives -0.5 reward.
We apply the aforementioned algorithms with different batchsizes N to solve this problem. The implementation details are mostly the same as those of the previous two-agent cliff navigation problem, with the following differences. (1) The behavior policy selects from all available actions uniformly at random, and the target policy selects from available actions that avoid obstacles uniformly
random; (2) The communication matrix U has diagonal entries 0.6 and off-diagonals 0.2; (3) We use 0.08 * N obtained by tuning while (4) The feature vector of any state (all the agents' locations) B = 0.002 * N is the same as aforementioned. is (ss) (s), (s),,, (s)) where (5(m) E {0, 135 defined as follows: <(m)(s) = 1, d ( = or
o (mm) (s) 1 if and only if the weth agent reaches its goal, is 1 step away from its goal, or is horizontal step and vertical step away from its goal, respectively. of(T) (s) = 1 if and only if the m-th agent has not reached its goal and there is at least one obstacle other agents in its 8 surrounding grids. (s) = the m-th agent has not reached its goal and collides with an obstacle "X" or another agent.
if and only if
Figure plots the value function al the target state where all the agents reach the goal (i.e., the sum of entries (0.) + (0.)s + (0))n) v.s. the number samples (tN) and the number of communication rounds (t). We can see that all the algorithms converge to the true value 3/(1- -)) = 60. Compared with the decentralized TD(O), our decentralized TDC achieve comparable sample complexities with different batch sizes and much lower communication complexities with N 10. Moreover, the required communication becomes lighter batch size increases, These properties are similar to those shown in the simulation and thus have generality.
6 Conclusion
In this paper, we develop a sample-efficient and commmication--cfiiinn decentralized TDC algorithm for multi-agent. off-policy evaluation. Our algorithm synchronizes the local importance sampling ratios among the agents and adopts mini-batch stochastic updates to save communication. In particular, it avoids sharing agents' sensitive local information. We prove that the proposed decentralized TDC algorithms achieve a near-optimal sample complexity as well as an optimal communication complexity that improves over the existing decentralized TD()). In the future, we expect that our algorithm can serve as a fundamental component in the design of advanced policy optimization algorithms for MARL.
Acknowledgments
The work of Z. Chen and Y. Zhou was supported in part by U.S. National Science Foundation under the Grants CCF-2106216 and DMS-2134223
13

---

Published in Transactions on Machine Learning Research (06/2022)
References
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Proc. Interna- tional Conference on Machine Learning (ICML), pp. 30 37, 1995.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Proc. Conference on Learning Theory (COLT), volume 75, pp. 1691-1692 2018.
Vivek S Borkar. Stochastic opprocimation: dynamical systems viewpoint, volume 48. 2009.
Lueas Cassano, Kun Yuan, and Ali Н Sayed. Multi-agent fully decentralized value function learning with linear convergence rates. IEEE Transactions Automatic Control, 2020.
Behdad Chalaki and Andreas A Malikopoulos. hysteretic q-learning coordination framework for emerging mobility systems in smart cities. brXic:2011.03137, 2020.
Tianyi Chen, Kaiqing Zhang, Georgios B Giannakis, and Tamer Basar. Communication-eficiennt policy gradient methods for distributed reinforcement learning. IEEE Transactions on Control of Nelwork Systems, 2021.
Gal Dalal, Balazs Szorenyi, Gugan Thoppe, and Shie Mannor. Finite sample analysis of two-timescale stochastic approximation with applications to reinforcement learning. In Proc. Conference on Learning Theory (COLT), 2018a.
Gal Dalal, Balázs Szörényi, Gugan Thoppe, and Shie Mannor. Finite sample analyses for td (0) with function approximation. In Proc. Association for the Advancement of Artificial Intelligence (AAAI), volume 32, 2018b.
Gal Dalal, Balazs Szorenyi, and Gugan Thoppe. A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proc. Association for the Advancement of Artificial Intelligence (AAAT), volume 34, pp.
2020.
Peter Dayan. The convergence of td (A) for general X. Machine learning, 8(3-4)3341333 362, 1992.
Thinh Doan, Siva Maguluri, and Justin Romberg. Finite-time analysis of distributed TD(0) with linear function approximation on multi-agent reinforcement learning. In Proc. International Conference on Machine Learning (ICML), volume 97, pp. 1626 1635, 09 15 Jun 2019.
GeofTrey J Gordon. Stable function approximation in dynamic programming. In Machine Learning Proceedings 1995, pp. 261-268. 1995.
Harsh Gupta, R. Srikant, and Lei Ying. Finite-time performance bounds and adaptive learning rate selection for two time-scale reinforcement learning. In Proc. Advances Neural Information Processing Systems (NeurIPS), volume 32, pp. 4704-4713, 2019.
Bin Hu and Usman Ahmed Syed. Characterizing the exact behaviors temporal difference learning algorithms using markov jump lincar system theory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 8479 $490, 2019.
Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic pro- gramming algs In Proc. Advances in Neural Information Processing Systems (NTPS), volume 6, pp- 703 710, 1993.
Maxim Kaledin, Eric Moulines, Alexey Naumov, Vladislav Tadic, and Hoi-To Wai. Finite time analysis of lincar woo-timsscale stochastic approximation with markovian noise. In Proc. Conference on Learning Theory (COLT), pp. 2144 2203, 2020.
Nathaniel Korda and Prashanth La. On td (0) with function approximation: Concentration bounds and a centered variant with exponential convergence. In Proc. International Conference on Machine Learning (ICML,), pp. 026-634, 2015.
14

---

Published in Transactions on Machine Learning Research (06/2022)
Vikram Krishnamurthy, Michael Maskery, and George Yin. Decentralized adaptive filtering algorithms for sensor activation in an unattended ground sensor network. IEEE Transaclions Signal Processing, 56 (12):6086 6101, 2008.
Chandrashekar Takshminarayanan and Csaba. Szepesvari. Linear stochastic approximation: How far does constant step-size and iterate averaging go? In Proc. International Conference Artificial Intelligence and Stalistics (AISTATS), pp. 1347 1355, 2018.
Bo Liu, Liu, Mohammad Ghavamzadch, Sridhar Mahadevan, and Marck Petrik. Finito-sample analysis of proximal gradient td algorithms. In Proc. Conference Uncerlainly in Arlificial Intelligence (UAT), pp- 504-513, 2015.
Hang Ma, Jingxing Yang, Liron Cohen, TK Kumar, and Sven Koenig. Feasibility study: Moving non- homogencous teams congested vidco game environments. In Proc. the AAAT Conference on Artificial Intelligence and Interactive Digilal Enterlainmend, volume 13, pp. 270 272, 2017.
Ziyuan Ma, Yudong Luo, and Hang Ma. Distributed heuristic multi-agent path finding with commuicationn. In Proc. IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2021.
Sergio Valcarcel Macua, Jianshu Chen, Santiago Zazo, and Ali H Sayed. Distributed policy evaluation under multiple behavior strategies. TEEE Transactions on Automatic Control,
2014.
Hamid Reza Maci. Grodient compora--diferennc learning algorithms. PhD thesis, University of Alberta, 2011.
Adwaitvedant Mathkar and Vivek S Borkar. Distributed reinforcement learning via gossip. IEEE Transactions on Automatic Control, (23):1466-11700 2016.
Robert Morris, Corina S Pasareanu, Kasper Luckow, Waqar Malik, Hang Ma, TK Satish Kumar, and Sven Koenig. Planning, scheduling and monitoring for airport surface operations. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. In Proc. A Advances in Newral Information Processing Systems (NeurIPS), 2021.
Guannan Qu and Na Li. Harnessing smoothness accelerate distributed optimization. IEEE Transactions on Control of Network Systems, 5(3:12245 1260, 2017.
Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control (L4DC), pp. 256-266, 2020.
Rajarshi Saha, Stefano Rini, Milind Rao, and Andrea Goldsmith. Decentralized optimization over noisy, rate-constrained networks: How to agrec by talking about how we disagrec. ArXiv:2010. 11292, 2020.
Kevin Scaman, Francis Bach, Sébastien Bubeck, Yin Tat Lec, and Laurent Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In Proc. International Conference on Machine Learning (ICML), volume 70, pp. 3027-3036 2017.
Ma Shaocong, Zhou Yi, and Zou Shaofeng. Variance-reduced off-policy tdc learning: Non-asymptotic convergence analysis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ma Shaocong, Chen Ziyi, Zhou Yi, and Zou Shaofeng. Greedy-[ga} with variance reduction: Finite-tine time analysis and improved complexity. In Proc. International Conference on Learning Representations (ICLR), 2021.
David Silver. Cooperative pathfinding, In Proc. the AAAI conference artificial intelligence and interactive digital entertainment, volume 1, pp. 117-122, 2005.
Navjot Singh, Deepesh Data, Jemin George, and Suhas Diggavi. Squarm-sgd: Commmicatiin-cfffiicntt momentum sgd for decentralized optimization. Xio:22055.0044 2020.
15

---

Published in Transactions on Machine Learning Research (06/2022)
Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and td learning. In Proc. Conference on Learning Theory (COL.)), pp. 2803-2888 2019.
Miloš S Stanković and Srdjan S Stankovié. Multi-agent demporal-difference learning with linear function approximation: Weak convergence under time-varying network topologies. In Proc. American Control Conference (ACC), pp. 167-172, 2016.
Jun Sun, Gang Wang, Georgios B Giannakis, Qinmin Yang, and Zaiyue Yang. Fimite-sample analysis of decentralized cemporal-diffrenne learning with linear function approximation. In Proc. International Conference on Arlificial Intelligence and Stalistics (AISTATS), pp. 4485 (485-4495, 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Richard S Sutton, Csaba Szepesvári, and Hamid Reza Maei. A convergent o(n) algorithm for off-policy temporal-difference learning with linear function approximation. In Proc. Advances in Neural Information Processing Systems (NIPS), volume 21, pp. 1609-1616, 2008.
Richard S Sutton, Hamid Reza Maci, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-differecee learning with linear function approximation. In Proc. International Conference on Machine Learning (ICML), pp. 993-90000 2009.
Vladislav Tadié. On the convergence of temporal-differenee learning with linear function approximation. Machine learning, 42(3))241-261) 2001.
John N Tsitsiklis and Benjamin Van Roy. An analysis of remporal-differnnce learning with function approxi- mation. IEEE transactions on automatic control, 42(5):674-620) 1997.
Manuela Veloso, Joydeep Biswas, Brian Coltin, and Stephanie Rosenthal. Cobots: Robust symbiotic au- tonomous mobile service robots. In "wenty-Fourde International Joint Conference on Artificial Intelligence, 2015.
Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, and Mingyi Hong. Multi-agent reinforcement learning via double averaging primal-dual optimization. In Proc. Advances in Neurul Information Processing Systems (NeurIPS), pp. 9672 9683, 2018
Gang Wang, Bingcong Li, and Georgios B Giannakis. A multistep Iyapunov approach for finite-time analysis of biased stochastic approximation. ArXiw:1909.00 04290, 2019.
Gang Wang, Songtao Lu, Georgios Giannakis, Gerald Tesauro, and Jian Sun. Decentralized td tracking with lincar function approximation and its finite-time analysis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Peter R Wurman, Raffaell D' D'Andrea, and Mick Mountz. Coordinating hundreds of cooperative, autonomous vehicles in warehouses. Al magazine, 29( 29)1) 11:99999 9, 2008.
Tengyu Xu and Yingbin Liang. Sample complexity bounds for two timescale value-based reinforcement learning algorithms. ArXiv:2011.05059, 2020.
Tengyu Xu, Shaofeng Zou, and Yingbin Liang. Two time-scale off-policy td learning: Non-asymptotic analysis over markovian samples. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 10634-10644, 2019.
Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. In Proc. Advances in Neural Information Processing Systems (NewrIPS), volume 33, 2020a.
Tengyu Xu, Zhe Wang, Yi Zhou, and Yingbin Liang, Reanalysis of variance reduced temporal difference learning. In Proc. International Conference on Learning Representations (ICI.), 2020b.
16

---

Published in Transactions on Machine Learning Research (06/2022)
Zhi Yan, Nicolas Jouandean, and Arab Ali Cherif. A survey and analysis of multi-robot coordination. International Journal of Aduanced Robolic Systems, 10(12):399, 2013.
Evgen Yanmaz, Markus Quaritsch, Saeed Yahyanejad, Bernhard Rinner, Hermann Hellwagner, and Christian Bettstetter. Communication and coordination for drone networks. In Proc. International Conference on Ad Hoc Networks, pp. 79-91, 2017.
Mingqi Yuan, Qi Cao, Man-on Pun, and Yi Chen. Towards user scheduling for 6g: A fairness-oriented scheduler using multi-agent reinforcement learning, ArXiv:2012. 15081, 2020.
Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Marl with general utilities via decentralized shadow reward actor-critie. ArXiv:2106. 00548, 2021.
Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Fimite-sample analysis for sarsa with linear function approximation. In Proc. Advances in Newral Information Processing Systems, pp. 8665 8675, 2019.
A Notations and Filtration
A.1 Notations rewrite update rules in Algorithm 1
We introduce the notations in Table that will be used Chroughout our proof
Table 1: List of notations
Notations
Explanation
Ps. A,, B, C. (defined in eq. (3),
These quantities of the i-th sample use exact global importance sampling ratio Ps.
b m) = p( ((s:)
A
Pi (m)) (ss:) (70
((s)))
Replace Pi and R2 in Ai, B; (see eq.
B := p,mm ((s )o(s)))
with ((()) and s!!"") respectively.
(m = p, m) (m)) ((8))
Replace Pi in b(m) with p (defined by eqs. 5 and (6)).
0, = 37 Em= m))
Agents' average model parameter.
Wr and b, are defined similarly.
Minibatch average of global quantity.
A. N E(11NN- A;
B, and C₂ are defined similarly.
A(m) N E(HNNN- - A
Average local quantity over the minibatch at the f-th iteration.
x(m) B m)) ((m) and b, are defined similarly.
6
((+1) -1 b,
M
(m)
Average over both agents and minibatch.
NN
A : Exg A.J, B : E, B
Expected quantities.
C = E,,, [Ci), b: = Ex, 5
0+ - - b
The optimal model parameter The optimal auxiliary parameter corresponding to O,.
wi (AO,
||||F Eg,g A3,) 1/2
Frobenius norm.
Then, by averaging the update rules average 0,, W.
and over m, we obtain the following update rules of the model
M
x()
Oct1
=0, +
(Alm) ofm))
+ -b + B((m) we (m))
(14)
M
3
M
A (m) of
+ br + Cpw(m)) (m)).
xmm
We+1
W +
(15)
M
m=1
17

---

Published in Transactions on Machine Learning Research (06/2022)
A.2 Filtration
Define the filtration Fi = =0((0 ar a0(60, u {sWN)). Then,
((m) bt F
Be, Cr b, A B(
",,, wn Tr w E Fi/Fi-1.
B Proof of Theorem
Step Bounding optimization error for ideal case. We first consider an ideal case where the agents can access the exact global importance sampling ratio Pt at iteration In this ideal case, every agent m can replace the estimated global importance sampling ratio P; (!"") involved in (m) B(m), B.m)) in the update rules
and by the exact value Pi- Then, with the notations defined Appendix A the averaged update rules
and respectively become eqs.
[12 and (13) as repeated below
0,+1 B + a b, + B,W.). x++1 - + 3 + by + C...)
The aim of Step 1 is to bound the following optimization errors of W411 Bet1 obtained by the centralized update rules
13 and
12 respectively.
E
[1111
w||1 Fi
wo 112 + 23 (w. - - " B[A,0, + tb Cr Fil + B|IA,, b + C. [|....)
(16)
(IT)
E |1 [3.1
110 1²21 2a (0, -0) T + b. B. +a A + b, + B,u ||/
(17)
(ITr)
(IV)
The above four terms (T)-(IV) are respectively bounded below.
(1) =0 (T, (w, T [ + [,
@ 2(( wi) [E c| (w) wt) +
- - TC C(w. w() + 2(w, =w²
B[(Ā. C, b, [b(Fi
(0)
< 2|1E,C, - C| ||||||| - wt ||² 2>2 | - w + de|| w
3
E - C, [,
A
E (l C,C-1A) F:] 0, - 1²1/2 + -
3
El[lA - C, ||
X2
A₂
(H8)
Avpmax -) - )2)|IT, wi wi
96p2 (v +1)
1 }
) 2 (1 0-1 + 110 1| + 12
(18)
N12 8)
where (i) uses the notation that w C-- + b), (ii) uses de - -max (C), the inequality that
|Azl| ||A|| |||a|| for any matrix A and vector x ; and the incquality that 2a, a2 S o ||a 1| +o [az]] 12 for
any a1, 02 E Rd and o 0, and applies Jensen's inequality to convex functions || and || ||2 and uses eq.
(iv) uses eqs.
47 and 48
(IT) = E[| A.O. + by C.w, [1]]
18

---

Published in Transactions on Machine Learning Research (06/2022)
E - -) -A,
0*)
0, A)0' + +bg + + w;) w;)
(1)
S 4B[II, - č, F:] B. - 10 + a|| - wi 1|
+ 4 [14, - A Fr] 1|
[1 - +
(100 128p a maxx (v + 1)
(1 ) (10. 11 + |10 | + R + 4| -w(l 11
(19)
N
where (i) uses the notation that w = -C- (AB, + b). (ii) uses || 1Ek= 1xl1 S 4 Σin
| for any
01, O2, a3, 04 E Rdd and eq. 141 ; and |xx| = || ||||||>]|| for any matrix A and vector x (iii) uses eqs. 47
and 48
(111) = (0. 0*) T E[A, +b, B, [i]
@ 2(0 87) E[ - B-C-AA- A) 00) + A - 0¹)
+ +Ā. B.C A - A-- + B, - w/) + b - bb (B. - B)C
(0)
2E I|Ā B.C- A - A A Al || 0 0-11 - 2211 B. 111 + 111100 - 0011 + B. w/)l|
+ B[IIA: -
4
A
1|/11100 + +16 6 b1le + +l[(B, - 1000=o11172
A1
(20)
64 64p (v +1) (1
pmax]
)²
2 - ->i1 >1)1100 110,1 -0'12 + A1
4pmnxx
The - wwil?
N(1 - 8)
d22
32p² maxe (v + 1) (4(1+
Pmaxx
"110711== + RR +
Piaxx Rmax
+ NAi(1 8)
+
(20)
A2
A2
where (i) uses the notations that w, = -AB b) and that 6 A00, and the relation that C- B = AT
(ii) uses the notation that X ->maxx (ATC1A) and the inequality that 20, a2 I <0-1[aa]] 1|2 + + [||a2|] 1| for
any a1, a2 E Rd and > O, and applies Jensen's inequality to the convex functions || - and || -|", (iii) uses
eqs.
40
42
45
(46) and 49h.
(IV)
= E[]], + T. + .. [|]
@ E[] - -B,C-' A) 0²) b, - b b. (T, w()
+ +A,- - -B.C-IA - ATC-1A)6 -B. - B)C 61| 1 [F]]
(00
< 10B ||| A11. + || 1 F. 11 1- + 5B A. BBC- A-A--- |F 1=
5E [1 - -bl b1] + 5E [|| B, || | - w 1| + - 11]
(00)
< 40p max (1 + Pmmx
"110, 0-1 1|2 + 55p |,, - w
A22
p
(v +1) 1)
(1+ Pmmxx
2 ((100] 1 + RR
(21)
N(1
8) ())
d2
where (i) uses the notations that wi -A- + b) and that b= -A0, and the relation that C- B = AT,
(ii) uses ||Ax|| < [A] for any matrix A and vector x and Σ K
axll²
< K
<K k=1 ||ax|2 2 for any
K
an E Rd and eqs. (40) (45), (4G) and (49), (iii) uses eqs. (39), (40). (42) and (43) and (1 + P/nxe/ X2) S
(1 + Pmax/Ag) /22)2, Substituting the above terms 18 21 into eqs. 116 and
(17 gives the following upper
bounds of E [ will [ and [| 8- 12 Fi].
E [1 - w]
< [ + 23 41/mmax - 12)) + 433
w1 wi] 2
N(1 -8)
19

---

Published in Transactions on Machine Learning Research (06/2022)
64pmax (v 1)+1((1 (1+ ) 2 33
2(35 + +232 ( (18: -0-12 + +100112 + Rmax)
N11 5)
X2
A2
(6) 1 2
812)
|wue -wwllll will?
320 max (V 1+11// (1 (1
1/ ) (11., 0- + -100111 + R
1
(22)
N12(1 - 8)
where (i) uses N 2
Bugmux and 3 S min (5/ )t) x=(1-5)
E -110411-0 FF]
+ (1
1280x/mmas (v + 11+1 (1+
(1+
Pmax
2ad1 + 40a P (1 + Pmnxx
110 - 011
N(11 6)
A2
A2
+ apmax
8
+
5aa) 10)1\00= -
1|000--0 wFll
64cpmmax (v +1) 1)
(4(1
(11 + Pmaxx
010010 1| + R
Pmax Rmaxxmaxx
11
Nd1(11 -8) 8)
A2
A2
160a² P (2 12+1) +1 (1+ Puaxx)
(110-11² * [1+
(11- - 5)
A2
(1)
S 1 ad1
2 ) 110
1 -
-0'1- 0²12+
13a
224a ( (1) (1+
Punax NA - -6) 22
2 "(41001112 + +2R max + +1),
A1
(23)
where (1) USOS N2
128pm (v+1)
(1
(1+ Penux ) S min
20pmae
A
(1
Panax
-2 1 and
A..
Pmax Ramax
da
Prnax S (1+
+
Ax
+ R2
max-
Step 2: Bounding optimization error of Algorithm wt
With the above upper bounds
12
and
we derive the upper bounds of -[[ Fr), E
- [[+= and B[[ - 0- 1| F as follows.
E [e - w F.F
(
(1+ 6/(BA2) -
B(1lū, - -w, wFFFi + (1+ B\₂ 3 33)E(( - -well
(20 6 2,32 [1
B222)
| will + 320,3 max (v + 1) (1+ )
2 (10, - 1r11 + 11 + R
6 -882
2
N12 -6)
+
6 E
B M
Σ
[A - Am)o(m²)
+ b/
2(m)
- bmmm) |||
322
M
m=1
(18)
(1
B12
|
wi 1| + p2003pm max (u 11
(1 + xo
1 ) (110. 1-12+ + 110-112 + R
3
N)2(1 - 5)
+
6,3
Σ (I((A(()) - A.)e!!! mm)
M
+ +b, - -b(m (|) 11²
2(m)
[r]
A2
M
m=
(10)
|| w1ll²= 427 Paax (v (u+1) (1+ 1 _ (1τ² - 0-11² + 110-112 + R²max)
N12(1 - 8)
N22
+ MX2
12,00
L.4
166 max (1106m)] MEM
|| + ||m! (m)) || ||+ Rmax) )2 + 3(p2maxx + +3) + R ,
(24)
where (i) uses the inequality that |a + |2l < a)laal + + o )|2e2 for any a1, 02 E Rd and
a >> 0, (ii) uses eqs. (15). 13 and (22). (iii) applics Jensen's 's incquality to the convex function || and
uses B S , (iv) uses the condition that 8 < X2 which implies that 1 + 6/(822)-3 S 2, the inequality that
||a + 22 < 2||ai|| 2 + 1| for any 0s, a2 Rdd and eqs.
52
54 and 68
E 1141 - 11
[+
2/3( 11 13 [[[ - [
[[]||// + [1 + 2(3/(3X))
1)] [1 - - 11]11
20

---

Published in Transactions on Machine Learning Research (06/2022)
(8)
6/(3)2)
1 (1
(22)
3 | -w,12 4273pmaxt (v 11 1) (1+
X ) (11,, (116--0 -0112+ + -10011 ? + R
2(32
N ( -6)
12900 4/4
166 max (1le) || + || + Rmuxx) 7[1 + 3(2Pmax + +3)]2t + R ) MEM
M2
6
+
E B|IC0-1
A(B - - 12
B22
(88)
Bd2)
1 wwill 5343 max (v ( (1 Xx2 ) (10, -01 + +110-12 + R
1
6
N12(1 -8) 5)
15,302 L/4
(17
max (116)m)) mEM
( || 11100 ||00
|| + Rmax) + 3(2pmax + +3)
24a² Pmax 8\8
Pmax
E
Σ m
M
+ Blmi) (m)) + b,
=(m)
w 11/ Fi.
(25)
M
(V)
where (1) uses the incquality that ||a + 22ll S (1 + + (1+11+++0 0 || o for any 01, a2 E IRd and a > 0,
(ii) uses eq. [24] as well as the notation that w = CA- + b), (iii) uses B S 1
bg (this implies that
2 (3/(3))) -1 I ) and eqs. (14). 039
and 43 The above term (V) can be upper bounded as follows.
(V)= = B
1 M
Σ
A
m)
mm)))
+ b,
(m))
+ B. (m))
1|9 [i]
M
(
S 2E
M
((A(m))
Am)o!m
+ (b. - -Bkml) + (B(m) B.jus (m)) 12 Fi
(m)
M
m==1
1 M M
(A. + bmm) + Benu (mm)) >| 11
+
2E
m==1
(10
2
Σ
M
B ((Am)) Am) + (b.
(m)
-m.m) + (B(m) B.)w,, || 2E( b. + 1.
M
(R13)
M
E [[|[A((m)
((m)
100 m + [m(m)) -Bel (1
((m)
M
+ 2 40pmaxx
(1 (1+
Pmax
2 "18.-00 -
18-12- + 5pmax W2 -
-will will 160p max (v +1)
(1
Pinxx 2 (110-11- + R
A2
N(1- 5)
A2
(1v) 60% L/1
20 max mEM
(118(
+ 1111e( || + Rmax )211 + 351++(2puum 3(2pmaxx + +3)12t + Rmmax)
M
+ 2 4000maxx
(1+
Pmnax
2 110 10-1-
8-11
+ 5pmnax [1 will 11
160p (v +1+1) (1
Pinax
2
(110~112 + 1|)0s|+RRnnss) Rmaax)
A2
N ()
A2
where (i) uses the incquality that ||a: + au212
1|
< 2
2||ag + 2| for any 01, a1 E Rd,
(ii) applics Jensen's 's (21 and the inequality that ||a +ag < 3 -== || 1|
inequality to the convex function 11- (iii uses eq.
for any a1, a2, as E Rd, (iv) uses cqs.
52
53
(54 and 188. Substituting the above incquality into eq.
25 yields that
E W++1
w
1
120a² phaxx
[ - wo'l 1|
pp Pmax (v (v+1) (1+
2 +
1920 Pinxx
(1 +
Punxx 27
)
10 - 11
6
Ndx(1 - 8)
x3
BX3
A
3840
(v +
1)
3
1+
1
2
+
a² Prnxxx
1+ Pimax
2 2
(110 122 + R2aax)
N(1
5)
Xx2
21

---

Published in Transactions on Machine Learning Research (06/2022)
25502
L/4
(3 (3+ +
120² Piax
(110)")) || + 11w0 (106m)11 || + ) + Bm2pmax + 3))²²
max MEM
(26)
Md2
312
E [1 411 - [e.]
(
2 (1+ 6/(a21) - -3
B100+1- - -0 + (1+
adi -3 E[0c++1-10 - oz+1 Fil
6
(0) 6 2ad1
ad1
10: - -0-1== 13apmax
m7,- -w;l- will
6 1
2
A1
224apmuxx
(v + 1)
(4|10~12 + 21 + +1)(1 (1+ 1+ Pimxx )9]
+
NAI
5)
A2
6
E
a
M
[(A(m)] - Al) (m)
+ b
x(m
[6m) + (B(m)) - -B.)u, [|]
a²1
M
m== 1
(618)
(1
ar1
|10, 8¹11²
18apmax
W w 1|
300 (v +1)
(4110-11 + -2R2 + +1)(1 (1
Pinax 2
)
3
A1
N11 -0)
A2
+ MA
6a
M
[[[((A((T)
+
b
x(m)
Timl + (B(m)) Bo)u,, (10)| (m) [|]{]]]
m=1
(1v)
1 ad1
|10 - 0-1
18ap
T2 will + N11 -
300cxp (u + 1)
(4|10-112 + 2R max
1) (1
Pmax 2
3
d2
+
1800
L/4
20 max (1166²)) + (1/m6m) (m)] MEM
+ Rimax )2(1 + (22 + +3))20 + R
(27)
MA1
where (i) uses the inequality that ||a + azll2 s (1 11) ola=")lle for any 01, ag E I and o > 0,
(ii) uses eqs. [1], 12 and (23). (iii) uses 1/A1 and applics Jensen's 's incquality to the convex function
- 11, and (iv) uses the inequality that ||a + a2 + azll² < 3 E ||ax|| 2 for any a1 a2 0g E Rd and then uses
eqs. 52 54].
Taking expectation on both sides of eqs. 26 and 27 and summing up the two inequalities yields that
E( 0 1+1 - 11) + E([[++++ wi+1 11²)
322
120c² PLaxx
180xpimax
E( WT will )
6
BA3
adı
5343
pm
max (u 1)
11
2 )
1
1920c² Maxx
(1
Pmax
|[(10c-0011 - 111)
N12(1 - 5)
A2
A2
3840p 2mx0p
(v + 1)
3
11 112
2
a² p
Pmax
1+ Pmax
2
+
a
[100|122 + Re + 1)
N(1 -5)
5)
A2
BA3
Av2
2550 L/4
(3+
12a² 12a?pa
p
ad2
max (116!m))
|| + 1|w0
+ Rmmax) )211 + 3(2pmax + +3))2
M
B23
d
MEM
(1)
ad1
6000 (v+ (1+ Pinux) ) 2 (11091 + Ri + 1)
[E(| - 11) + E(IlwT w [2:)] +
6
N1 - 8) A
574302
L/4
max (110) | + |!l0 |m0
(m)) || + Rmax) )211 + (22 + +3)121
MA₂
MEM
where (i) uses the conditions that N >
64088p2 (u+1)
(1
x
(1-o)(
a I min 23040p 53p
BA
Sd Adg
lterating the inequality above yields the following
432p PaRA
2A1
2hg
Apanas
convergence rate of the optimization error E [|
- w wit witt| and [[ - -1
B[[ 0-12 |WT w0' w01133
22

---

Published in Transactions on Machine Learning Research (06/2022)
a11 T (110-
+1 0²11²= + (00- - will?)
6
T-1
ad1
7-100 max +
Pmax 2 (110712 1 + +R Rmax + 1)
11
6
Nd2 (1 8)
)2
1=
574302
1./4
max (1(0)
+ 1110 [m]]
+ Rmax) | + 5(2pmax + +3))211
M
meM
aλ1 T
8 - 0-1 1| + T w513) + 36000,3 max (v -1+1 (1+
Pinax )² "(10911 +R Rmax + 1)
(1 -
A
57430z 2T
(110/=>]]
| + Rmmax) )a
max
Md
MEM
= -(1- 6 ( 0-11 + 120y - w)ll3) + 0 3 Boz 2/2
)
T
+
(28)
Na
M
where (1) uses the conditions that a S + and 5(2pmax+3) which respectively imply that a1
and
that + 3(2pmax + 3) S /2. This proves eq.
10
Step 3: Proof of consensus error 11
Note that the local model averaging iterations can be rewritten
into the matrix-vector form O++1 UO where T TSIST T and O: e :...:; MT Hence, it can be derived from Lemma C.3 that
A | Or = 110 A0TIF S of || |F.
(29)
Hence, we only need to obtain upper bound of the initial consensus error EJ B||001| Subtracting eq. 14 from the local update rule yields that for any 0 0 T1,
(m) 0+1
Oc+1=
Σ
Um - 01) +
(mm)
-0,)
M -
a (Amm) og:))
+ +b,
x(m)
+ B( we (mm)
M
mi E.
O
Σ
M
(Amm))
+ b + B(m0') wi B (md)).
x(m')
M
m = 1,me
of
This can be rewritten into the following matrix-vector form,
A0 UAO, + [his h2i.............. har]]
where hm - M-1 (AA(m)) m/ + b, M M
(m)
+ T(Th (m)) - M mm'=10 mmm'= #m (A(m'))(m') + be + B(mn') we (m))).
Em
=(m²)
The item 2 of Lemma C.3 implies that for any 0 S t s T 1,
M
M
|0 S oz A2llA +
Σ
|mmll 112 s &211A011|F + Σ
|| hmm ||-
(30)
m=1
Then, using eqs.
55
57
yields that
Σ h
M
1 (2o) (2n)((ppnr)) pmpx +2)
Σ (10!!)) + Remax 1|:00 (1l))
M
m=
M
4erlpmxx + 1) Σ
(10!!)) - 0ell + ||l! well + 110, - 001 + || - wi'l + ||0*|| ||C-- + b)
(8)
M
40(pmax + +1) Σ
(m - -l - -1[w (m)
(m)
-wel)
m=1
23

---

Published in Transactions on Machine Learning Research (06/2022)
M (1+
2pmax A2
110, - 001 + M lT, will
M Pmax (21101) || + Rmax))
A2
where (1) uses the notations that w²= = -0------00 (AB2 + b). (ii) uses cqs. that
39
42
and
43
Hence, we obtain
E(
112)) S
(1)
1
-
(1 + (1+ 02 - )
M ll
)
2
m=1
(40
1 + 2
48a 48a² + +3))
B( B(1A0:11)) +
(Pmax + 1 2M
(11)(m) - -Bel2 + +1(w)()) - w21
2
1-02
m=1
2pmax (110, 1- + 100 w;l3))
4M² Pma
22 1 +
(11011 || + Rmax)2
(31)
A2
X
where (i) uses eq.
30 and the fact that (u + u)² S (1 + oju² + (1 + o ))²² for any u, 2 0, (ii)
uses
1= 9:)
S n
of for any qi E R and n E N Similarly, we obtain from the update rule of ( 1T paaddd and eq. 15 that
(2)
We- [ w
wi
B(IAW 1 (2))
1 1+
E B(IAW.1))) + 48a o3))
(Pmax + 1) 2M
Σ
M
(110( - 0,ll + 11 (m) w2l13)
2
1-03 02
m=1
+ .22
2mmax
2 (100 1-2 2/ 1|us= 10) +
4M
Pinax (110)) || + Rumax) X/
(32)
A2
Summing up eqs.
31 and
32
yields that
0(| + |AW 11
(0) 1+ B A10 + 117 + 96a + +
(pmax + =1 2M (0 [100,1] + ||
2
1-03
+ 22 (1 (1+
2pmax 2 ((1-
ad1
1 ) (100 - -0-11= + +11100 - -w:lll m1ll3)
A2
6
36000,pp² maxe (v + 1)
Pmax
(110"132 + n² 12 +1) +
574,80,/
4/14t
max (110gm)) || + |1m0@m) || + Rmmax 12)
1+
22 8)
da
MA2
meM
2M Pm
(110'1 + Rmmax) )
(0)
2 + of
E(IA0, + (AW
2
(Pmax + +1)
2Pmax
)((18, - -0-12++ +1100 -w6l1))
3
1 1- oî
A
282, a22
+
8M2 2
(110-12 + Raax + 1) 574302
1./42'
max
(110m=))]
+ |110 || Rmax) )3
(33)
X2
mEM
where (i) uses eq-
28
and (ii) uses (110'1 Rmax)* )2 < 1/ + 27
1-0z
and N >
50v 50 (M +1)
12802. (v+1) (1 Pmazx A1(1-6)
An
Iterating eq. 331
yields the following convergence rate of the initial consensus error E(IAOr
E [[ S
11 + (1)
2
(I 1 + (1
2 +
(Pmax + +1)
1+
2pmmxx
3
1- oz
A2
3
(100 0" 11/11T00 will
+ 2828
8.M² 3
(110^12
12 + +1))
1 02
X2
574,302
L/12T
max (1166m))
+|1110m || |++ Runax }
||
MA₂
24

---

Published in Transactions on Machine Learning Research (06/2022)
1 | 10 + |aWoll 0 (1
M
(
(a+M²) a
M MBa2o22
2/
(1-02)2
02
(9)
Ma
MBuo2/42
0
(34)
02
1-02
where (i) uses
8
(B). Substituting the above inequality into eq. 29 proves eq.
11
2p
Hyperparameter choice and complexities. To summarize, the following conditions of the hyperparame- ters are used in the proof of Theorem including those required by Corollary 2 and lemma
C.4
S &min
B
A1
1+
Pmax
BA2
BA
BA
3A A2
2pmax +
40p
imxx
A2
A1
4pmax
23040p max
53
432p max
1 02
3A2
B71
= mii(((())) OM
(MM 02))}
(35)
50 (Pmax
1)
2A1
2x2
2
S min
5m2pmax +3) = (1))
(36)
8yPmax
6408,ppmss (v+1)
1
2 128p 118ppax (V (v/ + (1+ Pimxx ( (37)
N 2 max
1+
A2 8) ad112(11 - 0)
A
A1 -6) 6) A2
2 12 M + (8M- + 10) In Pmax 0
M
(38)
lnoz
1
Under the above conditions, we choose the following hyperparameter values.
O0 10M-112(11111 -0))), 3 = 0(1)
T Ine = ( VM VMI
Inn
ad1
1- 02
N B
= =0 VM
c(1 - 02)
8
12 M (8M + 10) In Pmax
0 (1-02)² + "I- - s0 (
VM
M
M In c- 02)
L 1n 2
In
Me/
In
T'
1
(c10(1 + (1 (1-02)2
M²3a
+
M MBa02 1/422T
ln
11
1- 02
1
In (c:0
1+
335
+ 1-
aM22
02))
0
1
In
Eel1 - o2)
M
Ino
1 02
1
02
Substituting these hyperparameters into eqs.
10 and
implies R[[0r - 0+1(3 B B(100T)) - 0+ S O(e),
(m) so E((0(T)) - 0 (13) < 2E(10(/T)
0r 1r
2
1- s O(e). Therefore, the overall communication
complexity for synchronizing !!!² is T + T" 0 /M In
), and the total sample complexity NT 1-02
o( M Inn-1 (1-6a))
C Supporting Lemmas
In this section, we prove some supporting lemmas that are used throughout the analysis of Algorithm D
Lemma C.1. Regarding the terms defined in Appendia A their norms have the following upper bounds.
||4 ||. (mm)
[|, || (l ||A|| & 2pmaxx
(39)
|| [|] [B
||. B ll. BB S Pmaxx
(40)
||G]]|; [[Cll ||]] $ 1,
(41)
25

---

Published in Transactions on Machine Learning Research (06/2022)
||0 (m)) 11, 110, 10
II, 1|. [illl |b||| & Pmmx Riaax
(42)
||c - 11 =
(43)
Proof. Consider any two vectors u, v E RRd we have that || || = Vtr(ew we we:) = ||u|||/||| Therefore, by we obtain that
Assumption 3
|Alee (pile(())lllo((()))))) x(s;))l S Pmax (olnxxx)ll@(s+11111 +|1((=))))+++ S 2pmax
|B: E=pall((s S Pinax:
||C|||e (ille>sllo(ss)) <11 1.
|b6"">| sp. [|((s S Pmaxx
The proof for |14 rr |ا || etc. similar. On the other hand, by Jensen's inequality, we obtain that
(++1)--1
||||| = [[,ll S 1..ll S 2pmax:
| S
Σ
||Aill S 2pmax
==
The proof for the other remaining matrices in eqs. Finally, we prove eq.
39
42
is similar by using the Jensen's 's inequality.
(33 Note that -C Em, (o(s, (s.)) > 0 with Amin (min(-))= = ->max(()) = >2. Hence,
||C-||| Amax(
>max(-C-1)
(-C) Aoo
Lemma C.2. Suppose the MDP trujectory {st, a:000 is generated following a behavioral policy To where
(a(m)) ar A {a, Jm- For any deterministic mappings Y S x Ai x... Am S + 12xa such that || (s, a, 8'Ill S
Cy, Vs, E S,a!! E Am where {a!m)) Ims we have
(++1)-- -1
E
Σ
Y(s ai S1+1) Fo
Fil
-Y
- N(1 8)'
2vC, y
N
EEN
(++1)--1
8C3(V +1) 5)
Σ
-
N
IN
where Y EY (si, 04, 81+1).
Note: A simplified version of the above lemma has been proposed and proved in Xu ct al. 2020a),, where and 8;+1 are omitted in the above inequality. We add a, and 8:41 so that this lemma can be better applied to the quantities Ai, Br, Cj and ((m) which rely on 8; as well as os and 81+1- The proof logic very similar to that of Xu et al. (2020a) and thus omitted here.
Corollary 1. Regarding the terms defined in Appendir A
they have the following upper bounds.
2VPmmxx
EE [[[[ C1IF Fi] S
(44)
N(1-0)
8p (v + 1)
[ B B s
(45)
N(1 8)
8p R22 (u + 1)
E [ T. b1 5
(46)
N(1 - 8)
p2 max (v +1
1
E
I1 C.c--All?
1+
(47)
N(1 -8)
8)
E 1. 8,C- 100--11111 [1
8p2 12
max max (v + 1)
1 +
1 2
(48)
N(1 6)
xx2
E
[ A, -. A- All
A
Fc
32p
(v + 1)
1 +
Pmax
(49)
N(1 6)
A2
26

---

Published in Transactions on Machine Learning Research (06/2022)
Proof. Let =
in Lemma
C.2 Then it can be checked that Y(s ae St-1)
Br, Cy = Pmaxs Pmaxs N D(++)/-1 Y (s,, on 81+1) = B,, and
Ex, Y (8;, as 8141) B.
Applying Lemma C.2 these equations proves eq.
45 The eqs.
44 and 46)
can be proved in similar
way.
Let ((s,as') = p(s, (()) - [[8)]T + a, (s,,
C-1A. Then, it can be checked that = A, B,C-A,, tV - ((,,, as 81+1) = A. B,C Moreover,
a S Pmax (r + 1) + P/ |C
||||||| S 2pmax + Pmax (22 (22') (2pmaxx = 2p (1 + Pmax/ (22) = Cy: Y = Ex, (8is dis 81+1) A BC- A = C-
Applying Lemma C.2 to these equations proves eq.
49
The equations
47 and 48
can be proved in
similar way.
Lemma C.3. The doubly stochastic matriz U and the difference matriz A
M 11 have the following
properties:
1. AU = -U= 11T
For any x E R and n E N+, |U" Ar| S 02 Az| (02 is the second largest singular value of U). Hence, for any H E RMM ||U S o AH
Proof. The first item can be proved by the following two equalities.
AU = (T- 1 -=
U = 11 M Unn =U -U-T==U-
1
M
M
UA = -1)
=
1
M 11
M
The proof of the item 2 follows from the claim page of Qu Li
2017 that
U 15 S o x 17/,
(50)
where we replace their W E _xx and vector w 12" into our U E MMM and vector H E RM respectively, H = 1 x and the largest singular value of - M 111, We first prove that is also the second largest singular value of U, i.e., o= 02.
Consider the singular value decomposition of the doubly stochastic matrix U TT DQ, where matrices
Q, Q are unitary and matrix D diag(1, 02, 03. . TM) is diagonal with >> 02 2 03 2 aM 2 0. Note that 1 U1 Q Dर1 which implies that Q1 DQ1. Similarly, U @QDQQ Q1 DQ1. Combining the above two results, we conclude that
Q1 DQQ DDU1 D²UL.
that is, (I- O. Since I - D is a diagonal matrix where the first diagonal entry is zero but the rest diagonal entries are strictly positive, it must hold that all the entries of Q1 are zero except for its first entry, i.e., Q1 = αε₁ where E R is the first entry of Q1 and e₁ = (1, 0...0. 0) is a basis vector. Hence, we conclude that Q1 = DQ1 = aDe = ae1- Taking the norm of both sides yields that | = |a|l|e]]],, i..., |a|| = || = |1|| = VM. Therefore, Q - _11 = QT ar (Q1) Q where
1/ (Q1) = D 01c = D D-eieT = diag(0, 02, .g....,,MM. which proves that 02 is the largest singular value of U '11 Lc.., =d2-
Then, substituting a =02 Ux 1r (U
x
UAr (the last step follows from the item 1 of this yields that
Lemma) and 17 (1 11 )r Ar into eq.
50
| 02 Az|
(51)
27

---

Published in Transactions on Machine Learning Research (06/2022)
A simple induction based on the above equality proves that |0"AA1| $ <ollAr|1 for any N Therefore, for any matrix H = [h.....,huu RMXM we can prove that
I" AHE
Σ |>
Σ (o 1|) = o
Based on Lemma
we obtain the following inexactness importance sampling ratio estimation p!T)) Pe- Corollary 2. Under Assumption and choosing L Σ O
C.3
Pmmx ), the estimation error of the inexact In
global importance sampling ratio P satisfies ΣM (p: (p(m) hold.
-pi)
1/4 Therefore, the following inequalities S0z
M
(m)
A.
A (m)
Aell; 11 I 40 L/4
(52)
m=1
B
(m) B
F
[ (m)
Bellz- 2 S 02
L/4
(53)
m=1
m=1
(m) bg
mm)
(m) ||?
b((m) 1|² S 02
L/4 R² Rmaax.
(54)
As result, the following upper bounds hold.
(m))
< 2mmax + 2
(55)
B((m)
S Piax +1 1
(56)
(m)
16 mm)
11.
66,
|| I Rmaxx (Pmax + 1)
(57)
Proof. Eq. 15 can be rewritten into the following matrix form.
p -
PP U. Pi.0 ... :P.. P...
Hence, the item 1 of Lemma C.3 yields that
A A : : (M) = U
..........
[MM
Then the item 2 of Lemma
C.3 yields that
PELL
(M) < 02
2L
A
P1.0 :..: 11
(1)
(58)
Denote Pmin minmeM M P; (m) can be proved by iterating eq.
Then Assumption implies that plm) In (m)) [ln Pminn Pmax]. Then it that p/m) [In Pmiin In Pmax]. Hence,
1
In P:
1
Σ m=1
In (m) [In Pminn In Pmax]
(59)
M
M
Then eqs.
58 and 59 imply that
M (m)
M
PiL
1
In ps)
<o²
2L
Σ
Pi.
(m)
In pi)) Mo?" In2 (pmax/ (0mmax/mmii)) Pmiu).
(60)
M
M
m=1
28

---

Published in Transactions on Machine Learning Research (06/2022)
Hence,
PiL
In ps
SVMo2 in
Pmax
(6)
In
Pmax
(61)
M
Pmin
2M
Pmin
where (i) uses the conditions that (BM Pmnce and 02 E [0, 1). Hence, eqs.
In(o,"
')
59 and 60
imply that
PiLL S In Pmax
In (Pmax/ Pmii)
(62)
2M
Therefore, we obtain that
Σ
P
(m)
pb)
(e MAR plp2
(im)
(69)
max (eM ,e (MP - In pi)
(60)
M
2L
Panax v Pmaxx/ Prain In? (Pmax/ Punin)
(iv)
M MM255 /P
(63)
where (i) uses eq. 66 (ii) uses the Lagrange's Mean Value Theorem, (iii) uses eqs.
59
60
and
(iv)
uses the inoquality that In < x for x Puaax/Puaa Pmin 2 1.
Since at least one of (0 Meem equals In Pminn we have
M (m) Pso
In P.
Σ
In Pini + (M 1) In Piaxx
(64)
m=1
Then, eqs.
61 and 64 imply that
(m) PiL
S
1
In Pmin + (1-
1
Pmax
(65)
2M
2M
Hence, we conclude that
Σ
M
(p (m)
pi)
= Σ
M
cnnps)2 2 < Σ max (e
2Mppin
mmm)
(6) 2mm) < M
PmminPma 2M- -1
(66)
where (i) uses eq.
(ii) uses cqs. 64 and 65
When Pmin 022 eq. 63 implics that Σ Em=1 (p. m))
1/2
pa)2 3 M42.5 02
0.755, When Puin
0
L/2
1,
eq.
(66] implics that mm= (p/m))
p.)2 x M MPMM-1 11 L/2
1o Both imply ΣM (p!m) p.) - 02
L/1
since
12th (+M+++10)) In Pmax
L >
In(a.")
Then, eq.
52 can be proved as follows.
M
A (m))
A:ll= < ([(s)))70((H4)) (bx:)) 112 Σ (p(m))
(m)
p:)² < (1+7)20
L/4
<
40
L/4
(67)
The above inequality implies that MM |
A = Σ M
(++1)N
(A (m) A 1 < 40 L/4
==N
where < applies Jensen's inequality to the convex function || 112-
Eqs.
53
and 54 can be proved similarly.
Eq. implies that [4(mm A. p: Amm)
À. 2. Hence, eq.
555 can be proved using triangle
inequality and eq. 139 Eqs. (56
and 57 be proved similarly.
29

---

Published in Transactions on Machine Learning Research (06/2022)
The proof of Corollary introduces new technique, which includes discussion of two cases: Puin (m)
minmeM P3 (m) lies in |or4/2 Pmmx) and (0, o o4/3]. This necessary the local average applied In Pj
which may be large negative number that cannot ensure a small consensus error for a fixed number of local average steps L.
Lemma C.4. Under the update rules of Algorithm and choosing 12 M-(8M+100) Praxx a the parameters have the following upper bound.
In(o,
PPaas +2
max MEM
|0|m)
max
| (m) S
max (116(m) mEM
+ 11 + Rmma \1 + 8(2pmax + 3)}²
(68)
mEM
Proof. Since L> 1n (+88M+ 10) In eqs.
55
57
hold. Hence, these equations and the update rule
In(o5')
imply that |16(7|11 S E Um 110( '11 + ad Pmax + (2 (1 || Rmax + ||1, ||) Taking maximum with
respect to m yields that
max
||!|ح|||
S
max mEM
Σ
Unm max [10(m")) + a(mmax + 1) mEM
max |10(m)) + Rmax + max (m))
meM [|wu |).
mEM
M²EM
me' E.
<alpmax 1) max
||0|-T]1 Rmaxx
+ max mEM
1m11 (1) + max 1
(69)
Similarly, it can be obtained that
[m]] MEM 241
max
||
|| S (8
1) max (0(Tm)
(1+ + 3) max |!w (mm) + 3 Rmmax (Puax + 1).
(70)
mE.M
mE.
Adding up eqs.
69 and yields that
m2) 10
(m)
++1
[wt+il
mEM
< 2(a PPmax + 1)
max mEM
110!m01
(apmax + a++++ 3 1) max mEM
|! (m) ||w, Rmax (Pmax + 1)
(0)
< 6(2pmax 3) max mEM
110( m
+ (1.53 +
max
|lu6 (mr)
0.53R Rax (2pmax + 3)
mEM
$ [L 8(2pmax 3){(
max
[10(m)
+ max mEM
[mm(m (]) + 0.5,3 (2pmax + 3),
MEM
where (i) uses the condition that a S +2 and (ii) uses Pmax 2 By iterating the incquality above and
using maxmeM ||6|T\]| + max 11w0 (m)) S 2 (( + ||m(]) ()))))) we prove eq. 68)
30


Reviews:
[{"review_id": "28vYuznseq", "reviewer": "Reviewer_nejn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the present paper gives theoretical analysis for off-policy TD learning, which is an essential contribution. \nWeaknesses: The gap between off-policy and on-policy in theoretical analysis is not highlight. The idea in the analysis to address the challenges of off-policy is not clearly presented. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper develops a sample-efficient and communication-efficient decentralized TDC algorithm for multi-agent off-policy evaluation. The communication complexity and sample complexity are attained via theoretical analysis. Experimental results verify the theoretical results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Comments:\n1.\tOne of the main contributions of this paper is to establish the convergence of the off-policy TD algorithm. So, the authors are expected to provide more details to highlight the challenges in the off-policy setting compared with the on-policy counterpart. Moreover, if the intuition of how the challenges are overcome in the analysis is given, the contribution will be clearer to readers.\n2.\t From the algorithmic perspective, it seems that there is nothing fundamentally new; thus, maybe the authors should focus more on exhibiting the theoretical analysis, at least, explaining the ideas behind the proof that did not appear in the existing literature.\n3.\tAs for assumption 5, as long as V is doubly stochastic, it can be derived that the second largest singular value is in range [0, 1). \n4.\tThe communication complexity does not take the communication topology into account. The total communications differ with different topologies even if number T+T’ is fixed. Densely connected network results more communications with the same number of iterations. The choice of communication matrix V also partially determines the convergence rate. The authors did not specify how V is chosen or optimized.\n5.\tFor the figures where the x-axis represents sample complexity (resp. communication complexity), the reviewer suggests using number of samples (resp. number of communication rounds).\n6.\tThe experiment settings are simple. The results are more convincing if more practical models are used. \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ESqRgMd3CF", "reviewer": "Reviewer_M45B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths\n- Algorithmic contributions: While Alg1 (decentralized mini-batch TDC) is built around a simple adaptation of TDC to the distributed scenario, the algorithm contains a few crucial ingredients that are non-trivial (i.e., they were not used in previous works) and have clear impact on the guarantees of the algorithm. Notably, the use of local mini-batches allows reducing the communication frequency and eventually the overall communication complexity. Furthermore, the agents perform a local estimation of the importance sampling ratios that allow computing a very accurate estimate of the global importance sampling ratio, which guarantees that accuracy of the updates. Finally, the algorithm includes a final communication step to ensure a desirable level of consensus is reached among all agents.\n- Theoretical contributions: The authors prove the effect of each algorithmic choice in deriving the final sample and communication complexity bounds. The results are interesting and clearly show the advantage wrt distributed TD.\n\n** Weaknesses\n- While the paper is overall well written, it would benefit from more details and clearer explanations in the review of existing results, the algorithmic structure and theoretical results (see below). This would make the paper more accessible and contributions stronger.\n- Experiments are designed more as a validation of the theory than a proper empirical assessment of the quality of the algorithm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel algorithm for distributed off-policy multiagent policy evaluation adapting the TDC (TD with gradient correction) algorithm. Using local mini-batch and distributed estimation of the global importance sampling ratio the authors show their algorithm achieves a sample complexity which matches the centralized case with a communication complexity that greatly improves over decentralized TD. Furthermore, their algorithm avoids sharing policy information, which may be considered as sensitive information in certain applications. The theoretical findings are supported by preliminary numerical simulations in small environments.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "** Required changes/clarifications\n- In order to have an easier way to contrast the sample and communication complexity of Alg.1 with previous literature, please include in Sect.2 the complexity of decentralized TD(0) and in Sect.2 the complexity of centralized TDC.\n- From my understanding, in order to compute local estimates of the global importance sampling ratios, Eq.5 requires communicating the local estimates $\\tilde \\rho$ across agents according to the topology of the communication graph. Since Eq.5 is reiterated L times and this is executed at each of the T iterations of the first phase of the algorithm, I would expect the total communication complexity to amount to T*L. But this does not seem to be the case. Please clarify this point.\n- At each iteration t, \"each agent collects N Markovian samples\". I guess this means that we simulate pi for N steps, but agents only store the actions they play. Is that correct?\n- At the beginning of Sect.4, the matrices A_t, B_t, C_t are never really properly defined.\n- The big-O notation used in the statement of Thm.1 is a bit inaccurate, since it is not clear what is the variables considered to the define it.\n- $\\bar\\theta_t$ is never defined\n- I would expect Eq.10 to reduce to the centralized TDC case for M=1.\n- In Fig.1 it looks like TDC is less accurate/slower than TD in terms of sample complexity. I guess this is the price to pay for the minibatches, which on the one hand allows less frequent communication, but on the other hand makes updates lag behind wrt to TD. Is that the case?\n- Distributed TD(0) as introduced in Eq.2 is on-policy, but in the experiments you run an off-policy version. Clarify what the algorithm is.\n- In Fig.2 TDC(L=1) tends to diverge as the importance sampling ratios are not accurate enough. This doesn't seem to affect the consensus as shown in Fig.3. Is this expected?\n- The notation is A.1 is very difficult to parse. I suggest to replace it by a table with an explanation of the meaning of each term and their role in the proof.\n- While the authors provide a sketch of the proof in the main paper, the appendix is difficult to browse. The authors did a good job in detailing and explaining all steps accurately, but the bigger plan or what is actually \"happening\" in the proof is hardly explained. I suggest to use the proof sketch to structure App.A and guide the reader through the different steps of the proof.\n\n** Suggestions for improvement\n- \"Markovian samples\" is typically used in theoretical papers (in particular in the analysis of TD-like algorithm), but it may be less obvious to the broader community. Clarify the meaning and contrast it with the iid case.\n- The undirected graph G=(M,E) is introduced but never really used. You may want to introduce the notation N_m already when introducing the graph.\n- The use of $V_{m,,m'}$ for the communication matrix clashes with the notation of the value function. \n- Define formally a \"doubly stochastic matrix\"\n- When introducing decentralized TD(0) (Eq.2) specify how the samples are obtained (i.e., on-policy). Also, it is never really explained what agents communicate exactly. Make it more explicit.\n- The stationary distribution $mu_b$ is never properly defined.\n- In MSPBE you define $\\Pi$ as a projection operator, please define.\n- In Sect 3.1 the use of index i (in Eq.3) and t (in Eq.4) is confusing. Please clarify.\n- In Sect.5, you write $theta^* = A^{-1}b$ is the optimal model parameter. It would be good to introduce this notion much earlier, for instance when introducing TDC.\n\n", "broader_impact_concerns": "The paper's contribution is more on the theoretical side for a generic multiagent policy evaluation problem. I do not have any concern to flag.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XwzQ19S7mo", "reviewer": "Reviewer_TD45", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is the algorithmic contribution, with sample complexity analysis and being fully decentralized with less restrictive assumptions than previous works, makes for a potential strong contribution in the area. However, I find several issues in the presentation of the work regarding some important limitations that are not fully discussed but are only reflected in the proofs, as well as possible theoretical issues  and other more minor issues including clarity in algorithm description, and notation. I go into more detail below.\n\n## Clarity in algorithm description an communication complexity\n\nFrom the pseudo-code and algorithm description it is not clear how exactly the minibatches are utilized. I would assume that for a mini-batch of size $N$ and $L$ iterations of approximating $\\hat{\\rho}_i^{(m)}$ that there should be $N$ estimates of $\\hat{\\rho}_i^{(m)}$ (one for each sample in the mini-batch). Is this correct? If so, how is it possible for the M agents to produce one mini-batch update in parallel (as claimed in second for loop of the pseudo-code)? From my understanding, L rounds of equations (5-6) cannot be implemented in parallel, iteration $\\ell \\in \\{0,\\cdots,L-2\\}$ needs to be completed (requiring information from all agents within the neighbourhood of agent $m$) before iteration $\\ell +1$. Additionally, it is unclear how these approximation steps are reflected in communication complexity, shouldn't complexity increase as $L$ increases? These global importance sampling approximation steps don't seem to be reflected in the communication complexity. Additionally, it is unclear what is exactly meant by communication complexity it would be useful to formally define it in the paper.\n\n## Limitations of algorithm and Theorem 1\n\nThe presentation of theorem 1 in the main body suggests a fast linear convergence result, however it does not clearly state stepsize or mini-batch conditions which is common in other works. See for example the closely related work of Sun et al. (2020) (Theorem 1) where the stepsize condition is clearly stated without using big O notation. I believe the big O notation hides significant restrictions that are not discussed in the paper. From looking at the proof it seems that the linear convergence comes at a significant cost with a __required__ mini-batch size greater than 1 and possibly small stepsizes. More specifically the minibatch condition (which is required for each step of phase 1) is\n\n$N \\geq \\max  \\left( \\frac{8 \\nu \\rho_\\max}{\\lambda_2(1-\\delta)}, \\frac{128 \\rho_\\max^2 (\\nu+1)}{\\lambda_1(1-\\delta)}\\left(1+\\frac{\\rho_\\max}{\\lambda_2}\\right)^2, \\frac{6408 \\beta \\rho_\\max^2(\\nu+1)}{\\alpha \\lambda_1 \\lambda_2(1-\\delta)}\\left(1+\\frac{1}{\\lambda_2}^2\\right)\\right).$\n\nThis seems that $N$ cannot be taken to be a value of $1$ and may in fact be very large. This seems like an important limitation of the algorithm that needs to be discussed in the main body. For example, the proposed method may not work in cases where one cannot query the environment twice at the same state. From my understanding, a large benefit of off-policy methods is to learn with minimal control over the data collection procedure. For example, in some applications I would imagine that it would be difficult or impossible to sample the behavioural policy more than once at a given state. Additionally, the potentially large mini-batch requirement suggests that the mini-batches are offering more than just variance reduction if convergence is not possible for $N=1$, indeed the experiments only show for the case $N \\geq 10$. Is convergence observed in practice for $N=1$?\n\nSimilarly to the minibatch condition, the stepsizes condition is not clearly stated and may require significantly small stepsizes which should be better stated in the theorem and discussed in the main body.\n\n## Possible issue with Lemma C.3\n\nThe proof of the second item in Lemma C.3 is claimed to be given in the work of Qu & Li (2017), however, from my understanding the result in Qu & Li (2017) uses the __largest__ singular value while Lemma C.3 uses $\\sigma_2$ which is the __second largest__ singular value. Therefore, it is unclear how the result from Qu & Li applies. I would recommend including the full arguments in the paper. \n\n## Notation\n\nThe current notation and lack of definitions makes the paper difficult to follow, especially the proofs. Below I outline some possible issues with notation.\n\n1. The notation used to describe the average updates (eq 12-13) should be defined if used in the main body. Furthermore, the average notation definitions are difficult to and are counter intuitive. For example, bar notation such as $\\bar{b}_t$ is used to sometimes denote average over the $M$ agents but sometimes it is used to denote the average over the mini-batch (such as in $\\bar{A}_t^{(m)}$). It would be much easier to follow with a more consistent stystem of notation.\n2. $\\rho^{(m)}(s,a)$ used in assumption 4 does not seem to be defined. \n3. I assume $||A||_F$ denotes the frobenious norm but it is never formally introduced as such. It may also be helpful to state the fact that $||A x||_2 \\leq ||A||_F ||x||_2$ which is often used in the proofs but never stated.\n4. I assume $\\lambda_\\max(A)$ denotes the largest eigen value of $A$ but this notation is never introduced.\n5. The following values used at the end of theorem 1 seem to never be defined: $c_{10},c_{11}, c_{8},c_{var},c_{sd}$. Is is mentioned that some of these are defined in appendix A but I cannot seem to find them.\n6. What is the significance of $w_t^\\ast$? It would be helpful to understand what this target parameter means. At the moment, the \"optimization error\" $\\mathbb{E}[||w_{t+1}-w_t^\\ast||^2]$ does not carry any meaning without further discussion.\n\n## Other questions and comments\n\n1. Do the specific values within the matrix $V$ carry any meaning? In Sun et al. I believe it is mentioned that the values of $V$ can be used as a design choice to optimize the value $\\sigma_2$, is it the similar case here? For, example is there a reason for specific choices for the diagonal of $V$ used in the experiments? Why is the diagonal non-zero and why do the values differ across application? Is the matrix $V$ an important tuning parameter (the weights for the neighbors of each agent)?\n2. It is mentioned that the analysis \"establishes tight bounds of the consensus error cause by synchronizing the global importance sampling ratio, [...] where samples are correlated,\" I fail to see how samples are correlated when estimating the global sampling ratio, aren't the mini-bath samples i.i.d.? It is possible I am misunderstanding this sentence.\n3. I think there is a typo in equation (3), shouldn't there only be subcript $i$ for $\\rho$ and not a $t$ subscript?\n\n# References\n\nSun, J., Wang, G., Giannakis, G. B., Yang, Q., & Yang, Z. (2020, June). Finite-time analysis of decentralized temporal-difference learning with linear function approximation. In International Conference on Artificial Intelligence and Statistics (pp. 4485-4495). PMLR.\n\nQu, Guannan, and Na Li. \"Harnessing smoothness to accelerate distributed optimization.\" IEEE Transactions on Control of Network Systems 5.3 (2017): 1245-1260.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new decentralized off-policy method for the multi-agent policy evaluation problem under linear function approximation. \nThe novel method is derived from the two-time scale TDC method from single-agent problems with two phases. The first phase includes decentralized TDC with a novel method to efficiently compute an approximation of the global importance sampling ratio while preserving communication restrictions across agents. Futhermore, the first phase requires mini-batch sampling, which can help reduce variance. The first phase is then followed by a second local averaging phase to guarantee consensus of the agents' parameters and prove convergence. The sample complexity of the proposed method is given and contrasted with related works. The method seems to be the first fully decentralized TDC method without making assumptions such as independent MDPs or policy sharing across agents.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I propose the following changes that I think would improve the work.\n\n## Critical changes\n* Introduce all notations used within the main body, without exception, and to include missing definitions in the appendix. Please see my comment above.\n* Discussion about the __requirement__ of a mini-bath size greater than 1 and why this is needed in practice or theory. If it is a conjectured to be a limitation of the analysis it would be helpful to include experiments with a batchsize of $N=1$.\n* Address the aforementioned issue regarding Lemma C.3 above.\n\n## Changes to strengthen the paper\n* Resolving all issues regarding notation. I would also suggest to have an index of notation at the start of the appendix for __all__ the symbols. I think this would greatly improve the readability of the paper.\n* Address clarity issues mentioned above. Mostly regarding the mini-batches and how it is used in conjunction with the global importance estimating procedure.\n* Include the discussion of the choice of values in the matrix of V and how this may affect performance (see more details above).", "broader_impact_concerns": "I do not have any concerns regarding broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Much of the reviewers' concerns revolved around clarity in the presentation: including notation, claims in prior work, innovations needed to generalize prior work to the off-policy case, etc.  The authors did a thorough, and commendable, job of addressing these concerns and making considerable improvements to the manuscript!
Decision: Accept (poster)