Title: Finite-Time Error Bounds for Distributed Linear Stochastic Approximation
Authors: Yixuan Lin
Ji Liu
Vijay Gupta
Venue: NeurIPS 2021

Abstract: This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-type distributed stochastic approximation algorithm and provides its finite-time bounds for the performance by leveraging the analysis for the consensus-type algorithm with stochastic matrices.


Review discussion:
Originality: This paper studies distributed Markovian stochastic approximation algorithms with stochastic interconnection matrices, which has not yet been covered in the related literature. The results are a bit different than what is known for related algorithms having doubly stochastic interconnection matrices.
Further, a push-type algorithm is proposed for decentralized implementation.
Quality. The results are technically sound and theoretically supported. The algorithm proposed is also new.
Clarity. The paper is well written and easy to follow. The results are nicely organized and the contributions relative to existing works are clear too.
Significance. The results are important and complement the existing distributed Markovian stochastic approximation results which have only focused on doubly stochastic interconnections among agents. The tools developed for analysis may also be useful for further use.
No
Thanks for the very positive comments.
We did not provide numerical simulations because the finite-time analysis is a purely theoretical work and many published NIPS, ICML and COLT papers on finite-time analysis
do not provide simulations; see e.g. [3, 13, 19, 21].
Notwithstanding this, comparing with existing algorithms is a good suggestion and we are happy to do this, as well as to validate our theorems, in the final version.
As mentioned, the main contribution is a new analysis on the distributed linear stochastic approximation algorithm given in (1), encompassing constant stepsize and time varying step size, Markovian samples. To achieve this, the study introduces a technique known as the absolute probability sequence to track the evolution of error in light of the time varying directed graph which results in a row-stochastic interconnection matrix. This technique can be interesting as it leads to a simple recursion of the iterates in (4), simplifying the subsequent analysis.
However, there are a number of drawbacks which outweigh the benefits of the current paper, as discussed below:
- Though the paper title reads "distributed linear stochastic approximation", the algorithm in (1) only considers a restricted setting with an identical driving matrix $A(x)$ across agents (and different $b^i(x)$ across agents). This setting may be suitable for the distributed policy evaluation problem where a number of prior works have appeared, but it does not include many interesting cases for distributed linear stochastic approximation including classical ones such as linear regression. In this regard, the paper may better position itself as a paper for distributed policy evaluation insteado of the more general linear stochastic approximation.
- The main theorems (Theorem 2 & 3) require Assumption 6 which is not well justified. Particularly, this assumption requires the absolute probability seqeunce to admit a limit vector. The Remark 3 which follows the assumption fails to provide a convincing case for this assumption to hold. Therein, two examples are given: the case $W_t$ is doubly stochastic is rather trivial as it defies one of the main relaxation considered in this paper; the case when the number of neighbors of agents are unchanged over time (also, is it the number of out-neighbors? or in-neighbors?) appears to be too artificial. The paper shall include a more concrete construction on how the assumption holds.
- In addition, it is not clear what are the benefits of employing the absolute probability sequence technique in analyzing the convergence of distributed algorithms, e.g., when compared to the classical approach in [52] for the push-SA algorithm.
As for the convergence to a weighted average of the equilibrium point when a row-stochastic mixing matrix is employed, some relevant work can be found in the static graph setting, e.g.,
-- J. Chen, A. Sayed, "Distributed Pareto Optimization via Diffusion Strategies", IEEE Transactions on Signal Processing, 2013.
- The presentation of the main theorems lacks useful information that a finite-time error analysis should encompass.
For instance, it is mentioned that "$C_2$ depends on $L, \gamma_{min}, \gamma_{max}, A_{max}, b_{max}$ ..." which does not describe how small/large the bound is - in fact, without a more detailed description, the bound in (7) is not useful since $C_2$ can be large; the result for push-SA is summarized in Theorem 5 *only* for the case of time varying step size, and it is mentioned that "our current analysis approach cannot be directly applied for this case" which lacks further expatiation; in theorem 3, how does that the convergence rate of $\eta_t$ affects the finite time error bound in the constant step size case?
**Post Rebuttal**
The reviewer appreciates the detailed response from the authors. Unfortunately, the response has not addressed my previous concerns and therefore my previous evaluation stands.
In particular, the promised "simulation based" study will not be very convincing since cases which violate Assumption 6 may still include a lot of cases.
Instead, it would be better to construct an **analytical** example to illustrate the necessity of Assumption 6.
In summary, my main concern is that Assumption 6 and its corresponding analysis due to the "absolute probability sequence" is one of the main innovations claimed in the paper, yet its applications are so far only limited to the previously known cases.
Moreover, while it is acceptable that $C_2$ can be a large constant, it is also important to point out **how** does $C_2$ depends on the system parameter $L, \gamma_{min}, \gamma_{max}, A_{max}, b_{max}$. The current presentation has so far eluded from discussing about that.
No
Our paper has two main contributions.
In addition to the finite-time analysis of a novel consensus-based distributed linear stochastic approximation algorithm, we also proposed a push-sum based distributed stochastic approximation algorithm and established its finite-time performance bound. Both algorithm analyses appear to the properties of an absolute probability sequence, in a different manner.
(1) Our distributed setting is motivated by the work of Kushner and Yin [33], which considers an identical $A(x)$ across the agents and is titled ''Asymptotic properties of distributed and communicating stochastic approximation algorithms''. Following [33] we used ''distributed linear stochastic approximation'' in the title.
As mentioned in 'Introduction' and 'Concluding Remarks', our algorithms can be directly applied to distributed TD learning (policy evaluation) because both TD(0) and TD($\lambda$) are special cases of linear stochastic approximation. Whether/how our algorithms can be applied to linear regression is a very interesting future direction.
(2) As we mentioned in Remark 3, Assumption 6 is a sufficient condition to guarantee our distributed algorithm (1) to converge when the interaction matrix is row stochastic and time-varying. We conjecture that the condition is also necessary. To see this, look at the (time-varying) weighted average dynamics (4), if Assumption 6 does not hold (i.e., the $\pi$ sequence does not have a limit), then the linear stochastic approximation (4) does not have a limit. A formal proof of this necessity has so far eluded us. As we mentioned in Remark 1, Assumption 6 is an analogy for condition (C3.4') in [33] which was needed to analyze the convergence of a different form of distributed stochastic approximation; this can be regarded as another justification for the need of Assumption 6.
Simple simulations show that without Assumption 6, both our algorithm (1) and the algorithm in [33] may not converge. We will add the above discussion and simulations to avoid possible confusion in the final version.
One purpose of Remark 3 was to show that our analysis methodology and results subsume the existing finite-time analyses for single-timescale distributed linear stochastic approximation (e.g., with doubly stochastic matrices or a fixed row stochastic matrices) as special cases; in these cases, Assumption 6 will be automatically satisfied and thus not needed.
Meanwhile, we would like to argue that the case when the number of neighbors of agents is unchanged over time (the number of in-neighbors) is not artificial. It is a biological behavior observed in fish and has been adopted in bio-inspired distributed algorithm design [''Consensus Over Numerosity-Constrained Random Networks'' by Abaid and Porfiri].
Other possible scenarios which satisfy Assumption 6 include the interaction matrix arbitrarily changes over time for an initial period and finally becomes a fixed matrix; the existing analyses cannot be directly applied to such scenarios. We will clarify these in the final version.
It is also worth emphasizing that our push-SA algorithm does not require Assumption 6 or similar.
(3) First, for the consensus-based algorithm (1), we are not aware of any existing techniques which can be applied to analyze time-varying row stochastic matrices cases. Our technique employing the absolute probability sequence is the first one, to our knowledge, that achieves this.
Second, we would like to emphasize that the classical approach in [52] cannot be applied to the push-SA algorithm (9); see detailed reasons in the 'Technical Innovation and Contributions' on page 3 and 'Proof Sketch and Technical Challenge' on page 9. One critical point is in the push-SA we need to prove its boundedness, whereas in the push-subgradient [52] its subgradients are assumed to be bounded.
It is worth mentioning that our novel approach for push-SA can be used to prove a better convergence rate for the push-subgradient algorithm in [52], which is under preparation and to be submitted somewhere else.
Besides, thanks for letting us be aware of this paper from distributed optimization. After carefully checking the paper, we found that it has an interesting idea to mix information among the agents (which leads to a row stochastic matrix);
however, it does not consider time-varying graph/matrix cases, and thus cannot be (directly) applied to our finite-time analyses.
Nevertheless, we are happy to include the paper in our final version and consider its idea as a possible future direction.
(4) In the fixed step-size case, the algorithm (1) will not converge with probability one or in the mean square;
instead, it will weakly converge. This can be validated via simple simulations and seen from Theorem 3.1 in [33] (though it is for a different form of distributed stochastic approximation).
That is to say, an asymptotic constant error in mean square always exists in the fixed step-size case, and $C_2$ serves as a way to quantify this error. Our $C_2$ shows that how this error depends on various parameters; theoretically, it could be large.
When it is large, our result implies that the algorithm with the given fixed step-size does not perform well, and then we can adjust the step-size (and possibly other parameters) to reduce the constant error, though cannot be arbitrarily small.
Thus, we respectfully disagree that (7) is not useful since $C_2$ can be large. It is worth mentioning that all existing finite-time analyses for the fixed step-size case have such a constant error which can be large; see e.g. [3, 13, 39]. In the next paragraph, we will explain how $C_2$ can be slightly improved.
Our current bound (7) is independent of $\eta_t$ because we used 2  to bound the term $\eta_t$ =  $||\pi_t-\pi_\infty||$ (because each $\pi$ is a stochastic vector) in our analysis. By doing this, the overall bound can be cleanly written as a sum of an exponentially decaying term and a constant term $C_2$, though slightly sacrificing the tightness of $C_2$. If we keep $\eta_t$ in our analysis, $C_2$ will split into two parts: a new constant, which is slightly smaller than $C_2$, and an asymptotically decaying term $\sum_{k=0}^{t-T_1} \eta_{t+1-k} (1-\frac{0.9 \alpha}{\gamma_{\max}})^k$. The latter term shows how the convergence rate of $\eta_t$ affects the finite time bound.
We will add this in the final version, and we are happy to replace (7) with this new expression if the reviewer feels it is a better-bound expression.
An important step in analyzing stochastic approximation processes is to bound the overall trajectory; see [''Stochastic approximation: A dynamical systems viewpoint'' by Borkar].
For the consensus-based algorithm (1), the boundedness of $||\Theta_t||$ can be proved in the same way for both fixed and time-varying step-sizes.
However, it is not the case for the push-SA algorithm (9). To establish the boundedness, we introduced an absolute probability sequence for the push-SA and utilized its properties (see Lemma 14), which has never been reported in the literature. Unfortunately, how to extend this approach to the fixed step-size case has so far eluded us. We will add this point in the final version, and would like to emphasize that our paper is, to our knowledge, the first one to analyze push-sum based stochastic approximation.
Strength:
This paper relaxes the commonly adopted assumption that the interconnection matrix is doubly stochastic to a more general one. The assumptions and results obtained in the work are well explained relative to existing works.
Weakness:
The motivating context concerns the application of SA approximation in reinforcement learning (RL).  It is suggested to briefly discuss how SA is applied in RL, for example, TD learning.
Some typos to be fixed:
Line 36, “arguably” —> “inarguably”;
Line 37, “differing” —> “different”
-------- after rebuttal ---------
Thanks for the reply! After reading other reviews and the committee discussion, I decide to change the score to weakly reject, which is in agreement with other reviewers, who raise detailed and constructive concerns.
No
We thank Reviewer kdHV for the positive feedback and for pointing out the typos. We will correct the typos in the final version.
We claimed in 'Introduction' and 'Concluding Remarks' that our algorithms and analyses can be directly applied to TD learning because both TD(0) and TD($\lambda$) are special cases of linear SA. Due to space limitations, we did not provide details of tedious applications to TD learning. We are happy to include a brief discussion on this in the final version.
Overall I think the results in this paper is solid and should be some interesting to the community. I have the following comments about the paper.
(1) Only convergence rate is characterized in the main theorems. It would be better to see the sample complexity result (total number of samples that required to achieve $\epsilon$-accuracy level) so that the readers can compare the contributions in this paper with the state-of-art.
(2) Although it is a theoretical paper, empirical verification is also suggested even with toy example.
(3) As claimed by the author, the dependence on some conditional numbers is not clear.
However those conditional numbers could affects the convergence rate significantly. A detailed discussion about those numbers would makes the paper more convincing.
-------- after rebuttal ---------
In the discussion, since the area chair and reviewer AANE raised some concerns about assumptions and the author did not address these concerns convincingly, I change my score to weakly reject.
No
We thank Reviewer KMZp for the positive feedback and suggestions.
(1) In our understanding, both ''finite-time'' and ''sample complexity'' analyses aim to study how many iterations it takes to get within a certain distance of the optimum (or equilibrium). It appears to us that some people use ''finite-time'' to stress the distinction between finite-time results and asymptotic results (e.g. [3, 13]), while some people use ''sample complexity'' because it refers to how many samples are needed to obtain a given level of accuracy (e.g. [''On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximation'' by Kumar, Koppel, Ribeiro]), possibly due to different communities' convention.
Therefore, the boundary between finite-time analysis and sample complexity analysis is blurry to us.
For example, from Theorem 5, the convergence rate (i.e., finite-time error bound) of our push-SA algorithm is at the order of $O((\log^2 t)/t)$;
then the sample complexity can be numerically computed by solving $t$ for the equality $(\log^2 t)/t = \varepsilon$, where $\varepsilon$ is the accuracy level.
For our consensus-based algorithm (1), its sample complexity can be derived in the same way from Theorem 3. In a word, sample complexity can be uniquely determined by the finite-time analysis.
Of course, there may be finer distinctions that we are not aware of. If it is the case, we are happy to discuss this in the final version.
(2) We did not provide numerical simulations because the finite-time analysis is a purely theoretical work and many published NIPS, ICML and COLT papers on finite-time analysis do not provide simulations; see e.g. [3, 13, 19, 21].
Notwithstanding this, comparing with existing algorithms is a good suggestion and we are happy to do this, as well as to validate our theorems, in the final version.
(3) We have discussed in Remark 4 and Remark 5 (in the appendix) about how the parameters $L, \beta, \alpha, \pi_{\min}, \gamma_{\max}$ and $ \delta_{\max}$ affect the convergence rate and error bound.
We agree that the influence of the remaining parameters is also important, which, however, has so far eluded us. We will add some simulations to illustrate the influence in the final version/appendix. We would like to mention that such a complete characterization of all parameters' influence on the bounds is also missing in the existing finite-time analysis papers; see e.g. [3, 13, 39].
We thank all reviewers and the area chair for their efforts to review and handle our submission.
After carefully checking all the post rebuttal review updates, we find that the only main remaining concern, which was raised by Reviewer AANE, is the necessity of Assumption 6, and feel that our earlier response to this issue was possibly overlooked or misunderstood. We will provide a more detailed response to elaborate on the necessity of Assumption 6 and its generality later today.
Meanwhile, it will be greatly appreciated if any reviewer or area chair feels any other issue was not well responded, and we will have a chance to respond again.
We thank all reviewers and the area chair for their efforts to review and handle our submission.
We have carefully checked all the post-rebuttal review updates. The main remaining concern, which was raised by Reviewer AANE (and also by the area chair, it seems), is the necessity of Assumption 6. Reviewer AANE summarizes this concern as follows:
["In summary, my main concern is that Assumption 6 and its corresponding analysis due to the "absolute probability sequence" is one of the main innovations claimed in the paper, yet its applications are so far only limited to the previously known case."]
We respectfully disagree with this statement, as it seems to overlook or misunderstand some related parts of our paper and our earlier response. In what follows, we contend that Assumption 6 has more general applications than the previously known case and that it is in fact necessary, and we reiterate the overall contributions of our paper.
1. Applications
First, as mentioned in our earlier response to Reviewer AANE, there are at least two cases which satisfy Assumption 6, yet cannot be directly handled by the existing analysis tool, which was developed only for doubly stochastic matrices. Case 1 is when the number of in-neighbors of agents is unchanged over time. This case has an interesting behavioral interpretation in fish biology, and has been adopted in bio-inspired distributed algorithm design ["Consensus Over Numerosity-Constrained Random Networks" by Abaid and Porfiri].
Case 2 is when the interaction matrix changes arbitrarily over time during an initial period, after which it finally becomes fixed. As we describe below, Case 2 occurs naturally in certain multi-agent systems.
In "Technical Innovation and Contributions" part in the introduction, we have explained why the existing analysis for doubly stochastic matrices is relatively easy and already well-understood within the vast distributed optimization literature.
Case 1 is mathematically equivalent to the situation when all stochastic matrices share the same left dominant eigenvector, which subsumes doubly stochastic matrices as a special case;
thus it could be analyzed by carefully choosing a fixed norm. There may be different choices: one choice is to apply our time-varying quadratic Lyapunov comparison function $\sum_{i=1}^N \pi_{t}^i \mathbf{E}[||\theta_{t}^i - \theta^*||_2^2]$ to the time-invariant case (i.e., $\pi_t^i$ does not change over time), which leads to the weighted Frobenius norm defined in the appendix.
The extension to Case 1 just described is straightforward, but Case 2 is not. As we proved in Theorems 2 and 3, when the interaction matrix arbitrarily changes over time for an initial period, say of length $T$, and finally becomes a fixed matrix or enters Case 1, all agents' trajectories determined by (1) will converge in mean square.
Also, recall that the corresponding finite-time error bounds in this case were derived using the "absolute probability sequence" technique. Note that the existing techniques can only be applied to analyze (1) after time $T$;
when $T$ is very large, such an analysis is undesirable, since the focus and challenge here are for "finite" time.
It is important to note that Case 2 provides a realistic model for certain systems. Consider scenarios in which some agents do not function stably and thus they communicate with their neighbors sporadically for a certain period, leading to a time-varying stochastic matrix. Such scenarios occur naturally when there is unstable communication due to environmental changes or movement of agents (e.g., robots or UAVs may need to move into a new formation while continuing computation). After this unstable period, which could be long, the whole system then enters a stable operation status. This satisfies Case 2 and our finite-time analysis can be applied to the whole process, no matter how long the unstable period could be, as long as it is finite.
In addition to this example, Case 2 and our analysis can be applied to certain scenarios in the presence of malicious agents. Suppose the system is aware that a small subset of agents have potentially been attacked and are thus behaving maliciously. To protect the system, the consensus interaction among the agents can switch to resilient consensus algorithms such as [46,47] in order to attenuate the effect of malicious agents. In this situation, the resulting dynamics of the non-malicious agents are in general characterized by a time-varying stochastic matrix.
After identifying and/or fixing the malicious agents, which could be a very slow process, the system can switch back to normal operation status. This example again satisfies Case 2, and our analysis can be applied to the whole procedure. As we mentioned in Remark 3, if some malicious agents always exist, the non-malicious agents in general will not converge, and thus a finite-time analysis is probably meaningless. The non-convergence issue will be further explained in the next "Necessity" section.
Whether Assumption 6 can represent more realistic/analytic examples is a very interesting future direction.
Though consensus has been extensively studied and the "absolute probability sequence" was proposed decades ago, this question has never been explored. The development of more advanced analysis tools is an interesting topic as well.
2. Necessity
We now elaborate on why Assumption 6 is not restrictive from a theoretical point of view.
As mentioned in Remark 3, distributed SA with time-varying stochastic matrices does not converge, in general, if Assumption 6 does not hold; in response to Reviewer AANE, it is worth emphasizing that the promised "simulation-based" study is only for illustrating this fact (consistent with our Remark 3). As mentioned in our earlier response to Reviewer AANE, Assumption 6 is sufficient to guarantee the convergence of the distributed SA algorithm (1) when the interaction matrix is row stochastic and time-varying. Let us denote the necessary and sufficient condition for convergence of consensus-based distributed SA as Condition A, which is currently unknown. It is possible that there is a large gap between Assumption 6 and Condition A.
But Assumption 6 is (to our knowledge) the most general sufficient condition that has been proposed so far; one indirect justification of this claim is, as mentioned in our earlier response to Reviewer AANE, Assumption 6 is an analogue of condition (C3.4') in the seminal work by Kushner and Yin [33], which is itself a sufficient condition guaranteeing the asymptotic convergence of a different form of distributed SA.
While [33] only provided asymptotic analysis, we provided both asymptotic and finite-time analyses using a novel tool. Assumption 6 subsumes the existing analysis for doubly stochastic matrices as a special case, and can be used for more general, nontrivial cases (see the examples provided in the discussion of Case 2 above).
Existing analysis tools cannot be applied to Case 2. From a theoretical point of view, our paper reduces the gap between the doubly stochastic matrices assumption and Condition A to the smaller gap between Assumption 6 and Condition A, for finite-time analysis of consensus-based distributed SA.
In addition, as mentioned in our earlier response to Reviewer AANE, the other equally important main contribution of our paper, push-SA, does not need Assumption 6, though its analysis still relies on the "absolute probability sequence" technique.
3. Contributions
Next, we present a high-level view of our paper, which may help the area chair and reviewers to better understand our overall contributions.
There are three major information fusion schemes in the vast distributed algorithms literature: "consensus" (time-varying stochastic matrices), "averaging" (time-varying doubly stochastic matrices which include gossiping), and "push-sum" (time-varying column stochastic matrices). The consensus-based scheme can guarantee an agreement among the agents, but the agreement point in general cannot be specified, especially when the interaction is time-varying. The averaging scheme can specify the agreement point to be the average among all agents using doubly stochastic matrices, but these only work for undirected graphs (i. e. , bi-directional communication is required between any pair of neighbors); typical examples are the Metropolis algorithm [44] and gossiping [62].
The push-sum scheme is able to not only achieve agreement on the average, but it also works for directed graphs, allowing uni-directional communication. The push-sum scheme can also be straightforwardly modified to achieve any given convex combination agreement among all agents. The three schemes are widely used, depending on task specifications. Push-sum appears to be the most powerful, but the other two also have advantages; e. g. , consensus can be modified to be more resilient against malicious agents, and averaging is easier in algorithm design (especially gossiping) and analysis (due to nicer properties of doubly stochastic matrices).
There is a very recently proposed scheme called push-pull, but it is not yet that popular, so we focus our attention on the three major schemes.
With the above background in mind, there are three major information fusion schemes that can be used to design distributed SA (as well as RL). The existing literature has only analyzed the averaging scheme (doubly stochastic matrices), which to us appears to be the easiest among the three. Finite-time analyses of the other two schemes are untouched in the literature. Our paper is the first to consider both.
As explained in the preceding "Necessity" section, our result and analysis for the consensus scheme (based on Assumption 6) are the most general so far and generalize the existing tools in a nontrivial manner.
This leads to very interesting, open research problems -- like necessary and sufficient condition for distributed SA convergence -- as well as how to design resilient consensus fusion, which can guarantee convergence of distributed SA.
Regarding the push-sum scheme, the finite-time analysis of push-SA relies on our development of the "absolute probability sequence" technique. This is the first time the technique is used in analyzing push-sum-based distributed algorithms, and the existing finite-time analysis tools for distributed (multi-agent) SA and RL cannot be applied here. The technical challenges and contributions are explained in "Technical Innovation and Contributions" on page 3 of the paper and "Proof Sketch and Technical Challenge" on page 9. As mentioned in our earlier response to Reviewer AANE, our analysis is nontrivial: in particular, boundedness of the trajectory generated by the push-SA scheme is proved using our development of the "absolute probability sequence" technique, which cannot be achieved -- or is at best very hard to achieve -- using the existing tools.
In summary, our paper has two equally important contributions, each of which is non-trivially novel and the most general yet achieved within the existing multi-agent finite-time analysis literature.
4. $C_2$ Dependency
Finally, we also respond to the (minor) issue of $C_2$ dependency raised by Reviewer AANE.
Based on the expression for $C_2$ given in Appendix A.1, it is easy to verify the dependency of $C_2$ on all the system parameters, except for $A_{\max}$.
Specifically, $C_2$ is monotonically increasing for $\gamma_{\max}, b_{\max}, \delta_{\max}, L$ and $ N$, and monotonically decreasing for $\gamma_{\min}, \pi_{\min}$ and $\beta$. The relationship between $C_2$ and $A_{\max}$ is less straightforward, and will depend on other parameters, but this relationship can be easily estimated by carrying out simulations using a range of different values for the remaining parameters.
We thank Reviewer AANE for raising this issue, which turns out to be easy to address and helps us improve our paper. We will definitely add the above discussion and simulations in the final version/appendix.

Meta-review:
role: PC
data: comment: The paper has received some discussion among the reviewers, with a weak consensus centering around rejection. A key concern is the presence of Assumption 6 in the paper. This issue has received substantial discussion with a lengthy reply from the authors. Nevertheless, I believe the authors have not successfully rebutted the concerns raised. 

To summarize the main contribution of the paper: existing analysis of distributed stochastic approximation are based on doubly stochastic matrices. The authors propose an analysis which can handle more cases, based on so-called absolute probability sequences, which is where Assumption 6 comes into play. However, the main concern is that the applicability of these new results is quite narrow. 

The authors list two cases where their results are applicable: (i) when all the matrices appearing in the algorithm share the same left-eigenvector (ii) when the matrices in question change until time T, at which point they remain constant. However, generalizing existing results from doubly stochastic matrices to cover case (i) is quite easy. The authors implicitly point out the key element of this in their response, which is that in this case we can write down a quadratic Lyapunov function using the quantities pi as coefficients -- there is no need for an absolute probability sequence. 

This leaves only case (ii). But if this is the main case where the results of this paper apply, the applicability already is quite narrow. Additionally, while in this case the authors remove scaling with the time T, they add a scaling with pi_{min} because the errors are scaled by pi_{min} in the bounds. There are bounds in the literature for this quantity which are exponentially small in the number of states; for example, see the proof of Theorem 4 in Touri and Nedic,  ''On the existence of a quadratic comparison function...", Proc. CDC 2011. So even in this case, it is not at all clear the results are better than the naive approach of simply applying existing bounds after time T; rather, there are tradeoffs.  Thus I believe this paper does not make the cut for NeurIPS. 

My suggestion to the authors for future submissions is to center the paper around their second main result, Theorem 5, provided it has good applications in distributed ML (this would need to be convincingly argued). 

Decision: Reject