Zero-Order One-Point Gradient Estimate in Consensus-Based Distributed Stochastic Optimization
Authors: Elissa Mhanna
Mohamad Assaad
Venue: Accepted by TMLR ()

Abstract: In this work, we consider a distributed multi-agent stochastic optimization problem, where each agent holds a local objective function that is smooth and strongly convex and that is subject to a stochastic process. The goal is for all agents to collaborate to find a common solution that optimizes the sum of these local functions. With the practical assumption that agents can only obtain noisy numerical function queries at precisely one point at a time, we consider an extention of a standard consensus-based distributed stochastic gradient (DSG) method to the bandit setting where we do not have access to the gradient, and we introduce a zero-order (ZO) one-point estimate (1P-DSG). We analyze the convergence of this techniques using stochastic approximation tools, and we prove that it \textit{converges almost surely to the optimum} despite the biasedness of our gradient estimate. We then study the convergence rate of our method. With constant step sizes, our method competes with its first-order (FO) counterparts by achieving a linear rate $O(\varrho^k)$ as a function of number of iterations $k$. To the best of our knowledge, this is the first work that proves this rate in the noisy estimation setting or with one-point estimators. With vanishing step sizes, we establish a rate of $O(\frac{1}{\sqrt{k}})$ after a sufficient number of iterations $k > K_0$. This rate matches the lower bound of centralized techniques utilizing one-point estimators. We then provide a regret bound of $O(\sqrt{k})$ with vanishing step sizes. We further illustrate the usefulness of the proposed technique using numerical experiments.

Published in Transactions on Machine Learning Research (11/2024)
Zero-Order One-Point Gradient Estimate in Consensus- Based Distributed Stochastic Optimization
Elissa Mhanna
elissa Salesm.boonnnaeeeooeeee
Mohamad Assaad
Laboratoire des Signaur & Systèmes CenéroleSupélec
rue Joliot Curie
91190 Gif-sur-Yuen France
Reviewed on OpenReview: https: /ttps://openreee opwreeiee........ net/ per/forum? id- tksub 7XoYB
Abstract
In this work, we consider a distributed multi-agent stochastic optimization problem, where each agent holds a local objective function that is smooth and strongly convex and that subject to stochastic process. The goal is for all agents to collaborate to find common solution that optimizes the sum of these local functions. With the practical assumption that agents only obtain noisy numerical function qucries at preciscly one point at lime, consider an extention of standard consensus-based distributed stochastic gradient (DSG) method to the bandit setting where we do not have access to the gradient, and we introduce a zero-order (ZO) one-point estimate (1P-DSG). We analyze the convergence of this techniques using stochastic approximation tools, and we prove that it converges almost surely to the oplimum despite the biasedness our gradient estimate. We then study the convergence rate of our method. With constant step sizes, our method competes with its first-order (FO) counterparts by achieving lincar rate O(e")
function of number of iterations k. To the best of our knowledge, this is the first. work that proves this rate in the noisy estimation setting or with one-point estimators. With vanishing step sizes, we establish a rate of O after a sufficient number of iterations k > Ko- This rate matches the lower bound of centralized techniques utilizing one-point estimators. We then provide regret bound of O(vk) with vanishing step sizes. We further illustrate the usefulness of the proposed technique using numerical experiments.
1 Introduction
Gradient-free optimization is an old topic in the research community; however, there has been an increased interest recently, especially machine learning applications, where optimization problems are typically solved with gradient descent algorithms. Successful applications of gradient- methods in machine learning include competing with an adversary in bandit problems (Flaxman
al. 20004) Agarwal al. 2010). generating adversarial attacks for deep learning models (Chen al.) 2019 Liu et al. 2019) and reinforcement learning (Vemula et al. 2019). Gradient-free optimization aims to solve optimization problems with only functional ZO information rather than FO gradient information. These techniques are essential in settings where explicit gradient computation may be impractical, expensive, impossible. Instances of such settings include high data dimensionality, time or resource straining function differentiation, or the cost. function not having a closed-form. ZO information basatiio--basddd methods inchude direct search methods (Golovin et al. 2019 1-point methods (Flxxman et al.) 2004 Bach & Perchet 2016 Verula al. 2019 Li Assaad [021] where function f). Ed + is evaluated at single point with some randomization estimate the gradient such
(1, S) d +2,
(1)

---

Published in Transactions on Machine Learning Research (11/2024)
with the optimization variable, > 0 small value, and random vector following symmetrical
distribution. 70 also includes 2- more point methods (Duchi et al. 2015 Nesterov & Spokoiny 2017
Gorbunov et al. 2018; Bach & Perchet
2016
Hajinezhad et al. 2019
Kumar Sahu et al. 2018
Agarwal
cct al, 2010: Chen ct al. 2019 Liu et al. 2019
Vemula et al. 2019 where functional difference at various
points employed for estimation, generally
having the respective structures
f( + 72,
f(x 72,
( (x, d
(2)
27
f(x Yeg, 5) - f(x - yeg, 5) ej
and g.²4 (I, () =
(3)
2;
j=1
where (eg) is the canonical basis, and other methods such as sign information of gradient estimates LLu et al. 2019
Another area. of great interest. distributed multi-agent optimization, where agents try to cooperatively solve problem with information exchange only limited to immediate neighbors in the network. Distributed computing and data storing are particularly essential in fields such vehicular communcattins and coordi- nation, data processing and distributed control in sensor networks (Shi & Eryilmaz, 2020), big-data analytics [Daneshmand et al., 2015), and federated learning (McMahan et al. 2017). More specifically, one direction of research integrates (sub)gradient-baser methods with a consensus/ jousensus/averaging averaging strategy; the local agent incorporates one or multiple consensus steps alongside evaluating the local gradient during optimization. Hence, these algorithms tackle fundamental challenge: overcoming differences between agents' local data distributions.
1.1 Problem Description
Consider a set of agents A = (1, connected by communication network. Each agent is associated with a local objective function Si( K. - R, where C Rdd is a convex feasible set. The global goal of the agents is to collaboratively locate the decision variable x E that solves the stochastic optimization problem:
min F(x) zER
B(x)
(4)
where
Fi(x)= = EsJ((,
with S E denoting an i.i.d. ergodic stochastic process describing uncertainties in the communication system.
We assume that at cach time step, agent can only query the function values of ft at exactly one point, and can only communicate with its neighbors. Further, we assume that the function queries are noisy ft f++ with G some additive noise. Agent must then employ this query to estimate the gradient of the form g. St).
1.2 Function Classes and Gradient Estimate Assumptions
Consider the following five classes of functions:
The convex class Caez containing all functions Rdd - that are convex.
The strongly convex class C.e containing all functions f Red + IR that are continuously differentiable and admit a constant Xy such that
(V/(=) Vl
- -) Ay || vll Var Rdd
2

---

Published in Transactions on Machine Learning Research (11/2024)
The Lipschitz continuous class Cúpp containing all functions R IR that admit a constant Lj such that
If(x) ( - Ly||æ - yll, Væ, y E RR
The smooth class Cemo containing all functions f I2 + IR that are continuously differentiable and admit constant Gj such that
2 Gylye vll, Vx, E Rd.
The gradient dominated class Cyd containing all functions f R + R that are differentiable, have a global minimizer x", and admit a constant such that
20y(f(z) f(x))) & [V Vx E Rd
This gradient domination property can be viewed as a nonconvex analogy of strong convexity.
In addition, consider the following assumptions on the gradient estimate:
A gradient estimate g said be unbiased w.r.t. the true gradient V if for all E Rd and independent S S, it satisfies the following equality
Es Eslg(z,)))az = Vf(2).
Otherwise, is said to be biased and satisfies
Es|g(z, V/())
with b(x) some bias term
gradient estimate g is said to have bounded variance when for all x E Rdd and independent. $ E S
Esllgg(r, - [f())) s for some o > 0.
Otherwise, when this bound is unknown or does not exist, it is said to have unbounded variance.
In general, FO stochastic gradient estimates unbiased and have bounded variance. 70 estimates, the other hand, are biased. While multi-point Zo estimates have bounded even vanishing variance, one-point estimates have unbounded variance Liu et al. (2020).
1.3 Related Work
FO Consensus-Based Distributed Methods: The optimal convergence rate for solving problem 4 suming the objective function F is strongly convex with Lipschitz continuous gradients, has been established as O(+) under a diminishing step size with full gradient information Pu & Nedić (2018); Nemirovski et al. (2009). However, when employing a constant step size a > 0 that is sufficiently small, the iterates produced by a stochastic gradient method converge exponentially fast (in expectation) the optimal solution (Pu Nodić 2018); this known as the lincar rate O(e*). The literature dedicated solving problem
an
of
vast. In what follows, we highlight some of the contributions.
Towfie et al. 2016 Tu & Sayed (012)) study distributed stochastic gradient methods where they compare (ATC) and combine-then-adapt (CTA) strategics, and prove that the ATC strategy
the adapt-then-combet oulperforms CTA one in terms of convergence rate, whether with vanishing or with constant step sizes and that it is more robust against data distribution drifts and network topology. Jakovetic et al. (2018) consider the CTA strategy with noisy FO gradients over random networks and establish an O convergence rate for strongly convex and smooth objectives and vanishing step size. Matei & Baras (2011): Yuan al. (2016) also consider random networks and solve problem 4 using a noise-free (sub) suub)gradient instead and achieve

---

Published in Transactions on Machine Learning Research (11/2024)
FUNCTION CLASS
STEP SIZE
REGRET BOUND
CONVERGENCE RATE
ESTIMATE
OP
Centralized
Cevz ACis
f.
O(ki)
0
Flaxman et al.
2004
One- point
Centralized
Csc CCip Csmo
V.
O(vk)
0
Bach Perchet
2016
Distributed Distributed
Cac SoenCann Cymo V.
O(vk)
O DS-
Cac Cxmo
[.
O(g))
IP-DSG
Centralized
Cevz 0 Cisp
V.
O(vk)
Vi. log h
Agarwal et
2010
70
Two-
Centralized
Cac
Cisp
V.
O k)
Agarwal et al.
2010
point
Distributed
Cisp
Cema
log k)
Tang
2021
VT
Distributed
Camo
ncd
0
Tang
2021
Distributed Distributed Distributed
Camo Camo Ngd
f.
O
Tang et al.
2021
2d- point
f.
O
Tang et
2021
Car Cxmo
0
Kumar Sahu et al.
2018
vF
Distributed
Cec Camo
O Matei & Baras
2011
Unbiased /BV
Yuan et al. 2016 i Pu & Nedic
20018
FO
Distributed
Csc Csmo
O Jakovetic
2018
Pu & Nedić
2018
Table 1:
Convergence rates for various algorithms related our work, classified according to the nature of the gradient estimate, whether the optimization problem (OP) is centralized or distributed, the assumptions
the objective function, whether the step size is fixed (f.) varying (v.), and the achieved regret bound and convergence rate
linear rate to a neighborhood of the optimum with constant step sizes. Nedić & Olshevsky 2006 consider time-varying and directed networks and present subgradient-push method based on noisy FO gradients that achieves an O ((1nk) rate under the same assumptions on the objective function and vanishing step size. Both the works Shi et al. (2015) and an & Li (2018) consider a static version of the objective function and propose methods that employ history information of the gradient. They both obtain rate of ( for general convex and smooth objectives with constant step sizes. Under the further strong convexity assumption, the static nature of the objective allows them to establish a linear convergence rate to the exact solution instead of neighborhood of it. Qu & Li (2018) inspire the vast. literature on gradient tracking extended to the stochastic setting (Pu Nedić 2018 Pu 2020 Xin et al., 2019) that utilizes local auxiliary variables to track the average of all agents' gradients, the lincar rate, however, established to neighborhood of the optimum.
ZO Centralized Methods: ZO methods are known to have worse convergence rates than their FO coun- terparts under the same conditions. For example, under convex centralized setting, Flaxman et al. (2004)
prove regret bound of O(k²) (or equivalently a rate of O Ve )) with a one-point estimator for Lipschitz
continuous functions. For strongly convex and smooth objective functions. Hazan & Levy 20144) and Ito (2020) improve upon this result by proving regret of O(v log k)) Bach & Perchet 2016 that of O k).
In the work of Agarwal et al. (2010), when the number of points is two, they prove regret bounds of Ö with high probability and of (llog())) in expectation for strongly convex loss functions. When the number is d + point, they prove regret bounds of O(Vk) and of (log(k)) with strong convexity. The reason why the performance improves with the addition of number of points in the estimate, is that their variance can be bounded, unlike one-point estimates whose variance cannot be bounded (Liu et al.) 2020). However, when the function queries are subjected to noise, multi-point estimates start behaving like one-point ones. In noisy function queries (centralized) scenario, it has been proven that gradient-free methods cannot achieve better
4

---

Published in Transactions on Machine Learning Research (11/2024)
convergence rate than 2 ) which is the lower bound derived by Duchi et al. (2015); Jamicson et al. (0012) Shamir (2013) for strongly convex and smooth objective functions. In the work of Bubeck et all (2021), a kernelized loss estimator is proposed where a generalization of Bernoulli convolutions is adopted, and an annealing schedule for exponential weights is used to control the estimator's variance in a focus region for dimensions higher than 1. Their method achieves regret bound of O(vi).
zo Tansensuus-Bsses Distributed Methods: In distributed settings, Tang el al. 2021] develop two algorithms for a noise-free nonconvex multi-agent optimization problem aiming at consensus. One of them is gradient-tracking based on a 2d-point estimator of the gradient with vanishing variance that achioves a rale of O(2) with smoothhess assumptions and a linear rale for an extra v-gradient dominated objective assumption and for fixed step sizes. The other is based on a 2-point estimator following an ATC strategy log under Lipschitz continuity and smoothness Vk
instead of gradient tracking and achieves a rate of 0 O
conditions and O(E) under an extra gradient dominated function structure. In the nonconvex setting, a gradient-tracking method is also proposed, but with a one-point estimator (Mhanna & Assaad, 2023) where a convergence rate of O Vे ) is established with Lipschitz continuous and smooth functions. Kumar Sahu el al. (2018) propose a standard CTA method where they consider a 2d-point estimate with noisy function queries over random networks. Under smoothness and strong convexity, they establish an O( convergence rate with vanishing stop sizcs. Waan ct al. (2020 2022) propose a projection-free method with onc-point gradient estimate where a linear optimization step is performed instead of projection. They prove a regret bound of O(k)) } for convex losses and an improved regret of O (log k)}) ) for strongly convex ones.
(
We highlight some of the mentioned convergence rates from the literature in Table 1
1.4 Contributions
While consensus- based distributed methods have been extended to the ZO case (Tang et al.] 2021 Ku-
mar Sahu et al. 2018), their approach relies on a multi-point gradient estimator and in the case of Mhanna
& Assaad (2023 Wan et al. 22020 2022). the rates established for one-point estimates are slow. The multi-
point estimation technique assumes the ability to observe multiple instances of the objective function under
identical system conditions, i.e., many function queries are done for the same realization of S in (2) and (3) However, this assumption needs to be revised in applications such as mobile edge computing (Mao et al.,
2017 Chen et al.) 2021 Zhou et al.) (022) where computational tasks from mobile users are offloaded to servers within the cellular network. Thus, queries requested from the servers by the users are subject to the wireless environment and are corrupted by noise not necessarily additive. Other applications include scnsor selection for an accurate parameter estimation (Liu et al.) 2018) where the obscrvation of cach sensor is continuousl changing. Thus, in such scenarios, one-point estimates offer vital alternative to solving online optimization iaaiiiiaattoo/laennngggg /learning problems. Yet, one-point estimators are not generally used because of their slow convergence rate. The main reason is due to their unbounded variance. To avoid this unbounded variance, in this work, we don't use the estimate given in ( we extend the one point approach in Li & Assaad (202T)'s 's work where the action of the agent is a scalar and different agents have different variables, to our consensual problem with vector variables. The difference is that in our gradient estimate, we don't 't divide by 7. This brings additional challenges in proving that our algorithm converges and consensus can be achieved by all agents. And even with bounded variance, there's still difficulty achieving good (linear) convergence rates with two-point estimates due to the constant upper bound of the variance ( (Tang et al. 2021). Here, despite this constant bound, we were able to go beyond two-point estimates to achieve a linear rate. Moreover, while it requires 2d points with the gradient tracking method to achieve a linear rate in TTagg et al. (021))ss 's work, which is twice the dimension of the gradient itself, here we only need one scalar point or query. This is much more computationally efficient. We further replace the gradient tracking method by standard ATC strategy which is more communication efficient as it requires the sharing of only one vector instead of two.
We summarize our contribution in the following points,
We consider smooth and strongly convex local objectives, and we consider the distributed stochastic gradient method in the case where we do not have access to the gradient in the noisy setting. Under
5

---

Published in Transactions on Machine Learning Research (11/2024)
the realistic assumption that the agent only has access to single noisy function value at cach time without necessarily knowing the form of this function, we propose one-point estimator in a stochastic framework.
Naturally, one-point estimators are biased with respect to the true gradient and suffer from high variance (Liu et al., 2020); Despite this, in this work, we analyze and indeed prove the algorithm's convergence almost surely with a biased estimate. This convergence is stronger than expected convergence analysis usually established for Z0 optimization. We also consider that a stochastic process influences the objective function from one iteration to the other, which provides a practical modcling for real-world scenarios that involve various sources of stochasticity, not necessarily additive noise.
We then study the convergence rate and we demonstrate that with fixed step sizes, the algorithm achieves a linear convergence rate Oge²) to neighborhood of the optimal solution, marking the first instance where this rate is attained in Zo optimization with one-point/ twoo-oont estimates and in a noisy query sctting, to the best of our knowledge. This linear rate competes with FO methods and even centralized algorithms terms of convergence speed (Pu Nedić 2018
When the step-sizes are vanishing, we prove that a rate of O (²) attainable to converge to an exact solution after sulficient number of iterations k > Kn- This rate satisfies the lower bounds achieved by its centralized counterparts in the same derivative-free setting (Duchi et al. 2015 Jamieson et al. 2012 Shamir 2013
We then show that a regret bound of O(Vk) is achieved for this algorithm.
Finally, we support our theoretical claims by providing numerical evidence and comparing the algo- rithm's performance to its FO and centralized counterparts.
The rest of this paper is divided as follow. In subsection 1.5 present the mathematical notation followed in this paper. In subsection [.6] we present the main assumptions of our optimization problem. We then describe our gradient estimate followed by the proposed algorithm in subsection 2.1 We then prove the almost sure convergence of our algorithm in subsection 3.1 and study its rate in subsection 3.2 with varying step sizes. In subsection 3.3 we find its regret. bound. And in subsection 3.4 we consider the case of fixed step sizes, study the convergence of our algorithm and its rate. Finally, in section
4 we provide numerical
evidence and conclude the paper in section 5
1.5 Notation
In all that follows, vectors are columu-shaped unless defined otherwise and denotes the vector of all entries equal to 1. For two vectors a, of the same dimension, (a, b) the inner product. For two matrices A, B Raxd, we define
(A, B) = Σ (A B,)
where A, (respectively, B.) represents the i-th row of A (respectively, B). This matrix product is the Hilbert- Schmidt inner product which written as (A, B) = tr(ABT). ||-I1 denotes the 2-norm for vectors and the Frobenius norm for matrices.
We next let [x(+)) denote the Euclidean projection of vector on the set K. We know that this projection closed convex set K. is nonexpansive (Kinderlehrer & Stampacchial (2000) Corollary 2.4), ie..
[[]((>) - [x(x'))l | 1--- 2'11, Vxr, ERd
We assume that each agent maintains a local copy D4 E of the decision variable and each agent's local function is subject to the stochastic variable S; E mm. At iteration k, the respective values denoted Lik and S... Bold notations denote the concatenated version of the variables, i.e.
x= [mi,
F2, [2,
.., xn)² is of dimension and S = [Si, ..,.... ... S.)T of dimension n X m.

---

Published in Transactions on Machine Learning Research (11/2024)
We then define the mean of the decision variable as == 1 whose dimension is d.
We define the gradient of FÚ at the local variable (x;) E Rd and its Hessian matrix v2F;(z;) E Rdxd and we let
VF(x) = (z1), (22 VF E Radd
and
g = g(x, S) : |6] (I1, S1), (2 S2). In (Zn, Sn)]² Ruxd
We define its mean y = 1 E R and we denote each agent's gradient estimate at time k by Di,k = g: (z S.,).
1.6
Basic Assumptions
In this subsection, we introduce the fundamental assumptions that ensure the performance of the 1P-DSG algorithm.
Assumption 1.1. (on the graph) The topology of the network is represented by the graph g = (N,E) where the edges in & N N represent communication links. A gruph G undirected, ie., (1,J) E is (i.1) E, and connected (there erists a path of links between any two agents).
W [wsy] E Rax* denotes the agents' coupling matrix, where agents and j are connected iff wi w 0 (way W/j = 0 otherwise). W is monmegaline matrix and doubly slochaslic, i.e., W1= =1=11 and 1 = 1 All dingonal elements wai are strictly positive.
A Assumption 1.2. (on the objective function) We assume the eristence and the continuity of both (z)
and (x). Let x² E IC denote the solution of the problem 4 such that F(x*) mingek F(x). We next assume that F(x) is X-strongly convex where
A 1lv - Vr,yeK
F ( z)
We further assume the boundedness of the local Hessian where there ezists constant e₁ R+ such that
[V C1, YI
where here il suffices to the spectral norm (keeping in mind for matris A, ||ll2 S [||[F)) Assumption 1.3. (om the additive noise) Çi,k is zero-mean uncorrelated noise with bounded variance, where E(Gr,)) = and E(S?,) = C2 < co, vi E N.
Lemma 1.4. Qu & Li. 2018) Let pw be the spectral norm of W 111 When Assumption 1.1 is satisfied, we have the following mequalily
1 1
IW 15 S pwllw 1w|l, Vo E Baxd and w
and Pw < 1.
== Lemma 1.5. (uu & Nedić 2018) Define h(x) = 11" E Rixd, Due to the boundedness of the second
derivative in Assumption 1.2 there erists a scalar > 0 such that the objective function is L-smooth, and
VFF(z) x(x)ll S Vn |xx 17/l
Proof: See Appendix
2 Distributed Stochastic Gradient Methods
We propose to employ a zero-order one-point estimate of the gradient subject to the stochastic process S and additive noise Ç while stochastic perturbation and step size introduced, and we assume that

---

can then be written in the following compact matrix form for Published in Transactions on Machine Learning Research (11/2024)
each agent perform this estimation at each iteration. To elaborate, let Ii,k denote the aforementioned gradient estimate for agent at time k, then we define it as
Dt.k = fI,I + 7a Pik. St.k)
(6)
= (fe + 7 Lik St,k) + (c,k);
where Yh is vanishing step size and Pi,k E Rd is perturbation randomly and independently generated by each agent i. Di.k is in fact biased estimation of the gradient (ar,)) and the algorithm can converge under the condition that all parameters are properly chosen. For clarification on the form of this bias and more on the properties of this estimate, refer to Appendix
2.1 The 1P-DSG Algorithm
We consider a zero-order distributed stochastic gradient algorithm aiming for consensus with one-point estimate. We denote it as 1P-DSG employing the gradient estimate Ji,k in ( Every agent initializes its variables with an arbitrary valued vector Fı,0 E K and computes 94.0 at that variable. Then, at each time k N, agent i updates variables independently according the following steps:
24,K+1 wiy (2 (2,,,
Ti (x(
perform the action: 24,4+1 + 7k+1 01,4+1
where an is a step size. Algorithm clarity of analysis:
2++1 W(x: -ggkk)
X [an 11,, #2,41: Fn,,tt,,.,,,,,tt1 In
(8)
perform the action: Xk+1 + 7k+1 4 k+1
where $ Radd is defined as 4 101, P2.k...., Pn.
As evident from the update of the variables, the exchange between agents is limited to neighboring nodes, and it encompasses the value X agga or the local gradient descent step.
We remark the effect of the gradient estimate variance on the convergence by carefully examining the steps in (8). Naturally, when the estimales have a large variance, the estimated gradients can vary widely from one sample to another. This means that the norm X++1, which is directly affected by this variance, may also grow considerably. Thus, may then take longer to converge to the optimal solution because it cannot reliably discern the direction of the steepest descent. In the worst case, the huge variance causes instability the optimizer may oscillate around the optimum making converging to a satisfactory solution difficult. In this work, use the fact that the local functions and the noise variance are bounded to prove that the variance of gradient estimate presented in (6) is indeed bounded. This boundedness, alongside the properties of the matrix W in Assumption 1.1 allows us to find then an upper bound on the variation of x with respect to its mean and the variation of this mean with respect to the optimizer at every iteration and analyze the convergence both.
even diverge if the variance is too high,
We then consider the following assumptions for the subsequent convergence analysis. We must note that the first assumption is only taken into account when study the algorithm's behavior with varying step sizes, herwwsee it is dropped.
Assumption 2.1. (on the step-sizes) Both an and 7 vanish to 0 k co, and satisfy the the following sums
80
20
0 = 00, Σ
2 < 00, and Σ a 2 < 00.
Am
8

---

Published in Transactions on Machine Learning Research (11/2024)
Assumption 2.2. (on the random perturbation) Let Di,k ( Pi.k..... pi,k)
Each agent i chooses its Di,k vector independently from other agents j # i. In addition, the elements of Li.k are assumed i.i.d with E 0 for d1 + d2 and there exists Cg > 0 such that B(o = cz, vdj, Vi, almost surely. We further ossume that there erists constant с₁ > where || S c4, Vi, almost surely. Example 2.3. One example is to take ag =ook + 1)- and * =o0( + 1)="a with the constants oo, "o, U1, 1/2 E IR+ As Ek=1 Okkik diverges for e1 U12 1,, Ek=1 og converges for conwerges for U1 + 202 > 1, we can find pairs of Us and V2 so that Assumption 2 is satisfied.
0.5, and Σk=1 04
we can choose the probability distribution of 0 to be the symmetrical Bernoulli distribution where Qik {- Va Fab with P(O - =( -t 0.5, Vdj, Vi.
To achieve the conditions in Assumption 2.2 9
Assumption 2.4. (on the local functions) K. is compact conver set and all local functions x S) are bounded on the c4 nooghhighboroood of K., i.e.,
S)( < co, Vo Neroo (IC), e R
where Nearr (KC) = {xc Rd infaek | all c470 is the c470 beighboogooo of K.
3
The 1P-DSG Algorithm
In this section, analyze Algorithm 1P-DSG presented in and
S
3.1 Convergence Results
The goal of this part is to analyze the asymptotic behavior of Algorithm 18 We start the analysis by defining Ha the history sequence (2o, yo. So. ... 1k-1, yk 2k and denoting by [[[[Ha)] as the conditional expectation given Hy.
We define ÿR to be the expected value of ãh with respect to all the stochastic terms S, &, given Hys i.c.,
go = Es 4.c < [H+].
In what follows, we use Oz- = [E H4.] for shorthand notation.
We define the error to be the difference between the value of single realization of Tr and its conditional expectation Tk. i.c.,
C 5k gs.
where can be seen as stochastic noise. The following lemma describing the vanishing of the stochastic noise is essential for our main result. Lemma 3.1. If all Assumptions
1.2
2.2
and 2.4 hold, then for any constant > O, we have
K' aseA 2 v) = 0, An > 0.
P( lim
sup
K
K'ZK
k= K
Proof: See Appendix
For any integer & 2 O, we define the divergence, or the orror between the average action taken by the agents Es and the optimal solution x" within K. as
ds = [ - 1²
(9)
The following theorem describes the main convergence result.
2.2 and 2.4 hold, then as k + 0o, ds + 0, Is + x², and Fi,k Tk, for all N, almost surely by applying the Algorithm.
Theorem 3.2. If all Assumptions I.1 I.3
Proof: See Appendia 7

---

Published in Transactions on Machine Learning Research (11/2024)
3.2 Convergence Rate with Vanishing Step Sizes
This part deals with how fast the expected divergence vanishes to find the proposed algorithm's expected convergence rate. To do so, we define the expected divergence as
Dr
T x ==
The goal is to bound this divergence from above by sequences whose convergence rate is known. The analysis is highly associated with the parameters ag and " that play significant role in determining this upper bound. Hence, in what follows, the analysis starts with general form of og and *k. then particular case is considered.
3.2.1 General Form of 0g and %
We first study the rate of convergence of the consensus error by introducing the following lemma.
Lemma 3.3. Let Assumptions
and 2.4 hold. Define
K1 arg min k. 1t
Then, for k 2 K1, there crist 0 < 01, 02 < oc, such that
| 1Txl12 < 8. and |1 - 15x 2 S Bzag
(10)
Proof: Refer to Appendiæ D.S
Our main result regarding the convergence rate is summarized in the following theorem.
Theorem 3.4. Let Assumptions
1.1 1.3
2.1
2.2
and 2.4 hold. We then define the constants A Aca
2
B 4 cee.
cxt E
E=
An
n
K2 arg min k and Ko = max(K1, K2}.
Asgrs1
We finally define the following parameters:
7h+1
max ai
Kg
01 k2Ka
max Fk:
03
max
k2 Ko %
k2
Tk
(11)
Tk
04
Tk:
05
06
T6.
k2Ka
Ko
k2Ka
If RR < A for any lo 2 Ku, then
D3 S S1 Vk 2 Ko.
(12)
with
DKo
Bo + Eos
S 2 max
(13)
kko
A-a 01
If Th for any 2 Ko, then
D₁ $2
Wk Ko.
(14)
7ik
with
DooTKo
Bos + Cos +
c 2 max
(15)
ako
A -
Proof: See Appendia
E.1
10

---

Published in Transactions on Machine Learning Research (11/2024)
3.2.2 A Special Case of ag and %
We now consider the special case mentioned Example 2.3
0rk: aoo 1)- and No 7o K 1)=
(16)
where 0.5 < U1 <1,0< 02 02 I 1 U1 and 11 + 20² > 1.
Theorem 3.5. Let ark and "/h have the forms given in 16) and consider the same assumptions of Theorem
3.4
If 0070 2 (2x2222,21) - A then we can say that there erists T < co, where
D& S + 12m Wk 2 Ko.
Proof: See Appendir E.4
The parameters clearly affect the upper bound of the convergence rate or rate of expected divergence decay As is evident that
in Theorem 3.5
max (202, v1 - U2 S 0.5,
the best choice when equality holds for en 0.75 and U/ = 0.25. With the sulficient condition on the parameters in Theorem 3.5 we can finally state that our algorithm converges with a of 0 xx)) after a sufficient number iterations & > Ky when the step sizes are vanishing.
3.3 Regret Bound
To further examine the performance of our algorithm, we present the following theorem on the achieved regret bound.
Theorem 3.6. Let the assumptions of Theorem 3.4 hold. When og and nh have the forms of 16 with 11 0.75 and 1/2 = 0.25, the regret bound given by
E
R( - Fi(x') <O(VK).
Proof: See Appendiz
3.4 Convergence Rate with Constant Step Sizes
In this subsection, we the slep sizes to ag = a > and 7/2 = > 0, vk 2 0, and we assume them to be two arbitrarily small values. We also define the following terms, A
B tcgL² C
2
esA
and R. ||x0 12 We let M denote the upper bound on 1| We then let G, 2n
and
(1-p2)"
G2 nM
1+ 1-
I+ I-pt
We finally define e = 1 Aayy and 02
1-02 2
Theorem 3.7. Assume any 1 and a < y. Let Assumptions 1.3
2.2
and 2.4 hold, then
| 15 2|2 s eh 1+1 a²G, and | 15x11
02
R + a²Gz.
(17)
Meaning, [|x+11 17 converges with the linear rate of ((e)) for an arbitrary small almost surely. Further,
When 01 S 02.
2R Bay + BG
DE+T Se k+1 Do + d
k+l
2Ax7 + 12. - +
+
G2 nd +
+, 2
(18)
Then, for arbilrary small step sizes, Dr. converges with the linear rate of 0 (
11

---

Published in Transactions on Machine Learning Research (11/2024)
When 21 > 22.
D4+1 Sei k+1 (Do + 2RBar + 2R
BG
a G2 C
+ a A
? AnA + 72
(19)
- 2A07 - -
Then, for arbitrary small step sizes, Dj converges with the linear rate of 0 (g)).
Proof: See Appendix G
Taking arbitrarily small values of a, 7 satisfying an < A and < Y, and setting e max max (2), the convergence rate becomes O(e*), achieving the same rate as with FO information.
4 Numerical Results
In this section, we provide numerical examples to illustrate the performance of the algorithm 1P-DSG. We compare it with FO distributed methods aiming to achieve consensus, namely DSGT (Pu & Nedić 2018) and EXTRA (Shi et al. 2015), ZO distributed algorithm denoted 2P-DSG based on the two-point estimate
in (2) (Tan et al. 2021), J and a Zo centralized algorithm based on gradient descent (e.g. Flaxman et al.
(2004) and Bach & Perchet (2016) using another one-point estimate which is presented in (1). We denote the Z0 centralized algorithm by IP-GD. We also compare with a centralized version of our algorithm where we use the estimate in (). For DSGT and EXTRA, we calculate the exact gradient and add white noise to
it to form an unbiased FO estimator and for all the 70 algorithms, we consider that the function queries are noisy, The network topology is connected Erdös- Erdös-Rényi random graph with a probability of 0.05.
We consider a logistic classification problem to classify m images of the two digits, labeled as Vij +1 or -1 from the MNIST data set (LeCun & Cortes] 2005). Each image, Xij, is a 785-dimensional vector and is compressed using a lossy autoencoder to become 10-dimensional denoted as X!ji ie., = 10. The total images are split equally among the agents such that cach agent has m; a ones for privacy constraints. However, the goal is still to make use of all images and to solve collaboratively
m images and no access to other
min- BEK
BaNA((Lo))) In(1 + exp(- XT + c
j=1
while reaching consensus on the decision variable KC with K = [-10, 10ja. We note here that u models some perturbation on the local querying of every example to add to the randomization of the communication process.
We consider classifying the digits 1 and 2 with 12700 images. There and thus each has a local batch of m, 127 images. We take o, 0.01 and let a₃ 0.05 + 10-0. and ? = 0.8(k: + 1)-0.23 for DP-DSG with vanishing step sizes, and a = 0.05 and ? = 0.6 with constant step sizes. We choose t E (- "a 3a2 with equal probability. Also, every function query is subject to white noise generated by the standard normal distribution. For the DSGT algorithm, we set the step size to aş = 0.015(k + 1)-1 when it is vanishing and = 0.015 when constant, and we do not consider the perturbation on the objective function nor the noise on the objective function, only the noise on the exact gradient. Similarly for EXTRA and we set its step size to a = 0.01. For 2P-DSG, we consider og = 0.01 + 1) and 7 = 0.01 + 1)-0.25 For the centralized 1P-GD algorithm, we set a = 0.005 and 7 0.5 (a 0.03 and 7 = 0.6 with estimator 6 We let e 0.1, and the initialization be the same for all algorithms, with 6.o uniformly chosen from 1-0.5, 0.5]d, Vi AV, per instance. We finally average the simulations over 30 instances.
100 agents in the network
The expected evolution of the loss objective function is presented in Figure I and the graphs are zoomed in on in Figure 2 Experimental results seem to validate our thcoretical results: Our proposed algorithm converges linearly fast with constant step sizes, however the final gap is due to converging to an O(a)- neighborhood of the optimal solution. 1P-DSG with vanishing step sizes converges with an 0 VF while
DSGT with vanishing step size converges at a rate of O(E)- Using constant vanishing step size does not.
12

---

Published in Transactions on Machine Learning Research (11/2024)
seem to affect the convergence rate of the loss function of DSGT. EXTRA consistently performs similarly to DSGT. The most interesting point is that IP-DSG, with vanishing and constant step sizes, outperforms the centralized Zo counterpart 1P-GD highlighting an advantage of our gradient estimate, In addition, there seems to be an evident advantage to using our one-point estimate to two-point one when the queries are noisy, as our algorithm outperforms 2P-DSG. We also note that the estimate we use seem more stable than the other 70 counterparts as shown in Figure 3 where we plot the average loss error bar of these algorithms. A possible explanation is that not dividing by y in the estimate provides some stability against noisy queries (this is slightly evident in 1P-GD with the different estimators and in 1P-DSG 2P-DSG).
BSST concran stap
1250 constar stop
20
EXTRA with sertalunt step skzz
20
EXTINA with consvarr. step sias
IF-00 OD with constant
GD with constant step slass
with
(6)
Iaih La22 txokion
25-
with vanssting step dass
Thei uas Exoatton
25
with vanis kig stop sioes 2P-250 with constaitt scap sizes
1P DOG with constaitt step sizes
2F-D50
2P-050
30-
10
5-
leration
Figure 1: Expected loss function evolution of the proposed algorithm DSGT, EXTRA, and 1P- GD considering vanishing vs. constant step sizes.
Figure 2: Expected loss function evolution of the proposed algorithm DSGT, EXTRA, and 1P- GD considering vanishing vs. constant step sizes.
10
IP-00
1P-GG with intimatoo (6)
0.-
Expacted Accuracy
0.5
Rug
DSST wihh vonisting step sibe
constart consuan: scep
6-
1P-G0 with constant step sizes
3P GO with entmmator (6)
3P-D5G with
1P-D50 with constant
-10
2P-066 with notee
600
2000
Eoration
Figure 3: The average loss error bar evolution of the proposed algorithm DSGT.
Figure 4: Expected test accuracy evolution of the proposed algorithm DSGT, EXTRA, and 1P- GD considering vanishing vs. constant step sizes.
In Figure we mcasure at every itcration the classification accuracy against an independent test set of 2167 images using the updated mean vector 0₂ = + Σx= 0i, of the local decision variables. The interest of the constant step sizes appears in the convergence rate of this accuracy, where our algorithm is able compete with DSGT with full FO information, and to outperform DSGT with vanishing step size. This is important result as it shows that the classification goal with ZO is well met despite the limiting upper bounds of convergence rale and that nei)-hhoihboroooo of the optimal solution achieved linearly fast can be sufficient to achieve the best possible accuracy.
The reason for this better accuracy attainment is generally because the step sizes affect the bound on the generalization error. For example, Hardt et al. (2016) prove theoretically that the bound on the generalization error for strongly convex objectives is smaller when the step sizes are constant (theorem 3.9) than when they
13

---

Published in Transactions on Machine Learning Research (11/2024)
are vanishing (theorem 3.10), wherein the latter, there is extra element containing the supremum of the function. Naturally, the step sizes seem to play a role in affecting the bound on the evolution of iterates, which in turn affects the uniform stability of the SGD method (stable means that the loss function is not affected much one datapoint different) and the gencralization error of an SGD-trained model is upper bounded by the uniform stability bound.
This result seems to be confirmed by the centralized 1P-GD vs IP-DSG with vanishing step sizes. Despite the latter outperforming the first in convergence speed (of the objective function), the first with constant step sizes seems to generalize better.
In Figures 6 and the curves are those of the evolution of the expected consensus error, or EE| Σ 10 0xll which the expected error between the local decision variables and their average. For all algorithms, the error again validates the theoretical bounds and decreases quite fast. Generally, evident in Figure 7 for all algorithms (expect 2P-DSG where a noise term is always multiplied by ) vanishing step sizes allow the consensus error to completely vanish while constant step sizes leave an O O(a") )-gap.
We add other numerical examples for different image labels in Appendix H
5 Conclusion
In this work, exlended the distributed stochastic gradient algorithm to present a practical solution to a relevant problem with realistic assumptions. A novel ZO algorithm was studied and proved to converge with a biased and high variance one-point gradient estimate and a stochastic perturbation on the objective function. In the context of noisy ZO optimization, we have successfully established a linear convergence rate with vanishing step sizes. These rates align with the optimal expectations examined in the existing literature. We also prove a regret bound that of O(VR) with vanishing step sizes. numerical application confirmed the success and efficiency of the algorithm.
of Oge²) using fixed step sizes and 0 V
6 Acknowledgment
We are grateful for the comments made by the editor and the reviewers that have substantially improved the quality of this work.
DSC1 with vanishing size
12001 with vanishing strp size
smap
1000
with comolant stsp sizz
3000
KA canorani step
-M-
18 DSG with vanisking step sioes
- 2P 250 with vanisning stop sizes
Expreted Corsentiu E-t0
1F DSG with constaint stop sizes
Expected Conceruus Conoeruu Enor
1P DSG with constant stop sibes
DSG
with
500
600
400
400
0-
400
800
1000
0.0
2.5
7.5
10.0
15.0
17.5
Reration
Beration
Figure 5: Expected consensus error evolution of the proposed algorithm vs. DSGT and EXTRA considering vanishing vs. constant step sizes.
Figure 6: Expected consensus error evolution of the proposed algorithm vs. DSGT and EXTRA considering vanishing vs. constant step sizes.
14

---

Published in Transactions on Machine Learning Research (11/2024)
3.5
Expacted Converun Eneor
DSS with varishing stop sac constant
conssont step
1P-050 wih verishing step sies
15.056 with constant stop sees
2P-056
0.0
1200
340
200
1000
Figure 7: Expected consensus error evolution of the proposed algorithm vs. DSGT and EXTRA considering vanishing vs, constant step sizes.
References
Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi- point bandit feedback. In COLT, 2010.
Francis Bach and Vianney Perchet. Highly-smmothl zero-th order online optimization vianney perchet, 2016.
URL https://arxiv http://aaaxvv.../raaaeee arggabs//600 00065
Sébastien Bubeck, Ronen Eldan, and Yin Tat Lee. Kernel-based methods for bandit convex optimization. J. ACM, 68(4), jun 2021. ISSN 0004-5411, doi: 10.1145/3458721. /ttpp::::///oii.orr www.oo..g//ooooo.ooooooooooooo.o 11155/001114553332222
Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. Zo-adamm: Zeroth- order adaptive momentum method for black-box optimization. In NewrIPS, 2019.
Ying Chen, Zhiyong Liu, Yongchao Zhang, Yuan Wu, Xin Chen, and Lian Zhao. Deep reinforcement learning- based dynamic resource management for mobile edge computing in industrial internet of things. IEEE Transactions on Industrial Informatics, 17(7):4925-4990 4934, 2021. doi: 10. 10.1109/11.. 10.1109/TTII.2 TT1.202..20000288866
Amir Daneshmand, Francisco Facchinei, Vyacheslav Kungurtsev, and Gesualdo Scutari. Hybrid ran- dom fomm/determiistic parallel algorithms for convex and nonconvex big data optimization. IEEE Transactions on Signal Processing, 63(15):3914 3929, 2015. doi: 10.1099/TSP.2015.225500
Joseph L. Doob. Stochastic processes. 1953.
John C. Duchi, Michael Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5)22788 2806, 2015.
doi: (0.1109/TIT: 22115201..22. 201.2205..4409956.
Abraham Flaxman, Adam Tauman Kalai, and H. Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without gradient. CoRR, cs.LG/0408000 2004. URL http://arxiv. org/abs org/abs/cs.u wtc.cs..G//0000000
Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi (Richard) Zhang, Gradientless descent: High-dimensional zeroth-order optimization. CoRR, abs/ 1bs1191.066317, 2019. URL wtw.:/rg////bbaa/eeeeeaee
http://arxiv.org/aaa abaaaaas//99 http://www....g//a////1111111111
Eduard Gorbunov, Pavel Dvurechensky, and Alexander Gasnikov. An accelerated method for derivativo-free smooth stochastic convex optimization, 2018. URT https://arxiv, org. absa
wwt.://rr.bb....../... 09022
15

---

Published in Transactions on Machine Learning Research (11/2024)
Davood Hajinezhad, Mingyi Hong, and Alfredo Garcia. Zone: Zeroth-order nonconvex multiagent op- timization over networks. IEEE Transactions on Aulomalic Control, (4(10):399544010 2019. doi: 10.1109/TAA:220192296777
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016. URL httts::///axiiiii wwt.:/r/gbbbb.....//// htttp:://aaxv..or///////1/. 01240
Elad Hazan and Kfir Levy. Bandit convex optimization: Towards tight bounds. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems, volume 27. Curran Associates, Inc., 2014. URL. htttp::/prrceeddii wwaeinng.nnneurgee neur cc/
paper wtt.apppwwe/.l/..coeesaaaeeeeeaa
Shinji Ito. An optinal algorithm for bandit convex optimization with strongly-convex and smooth loss. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenly Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Mochine Learning Research, pp.
PMLR, 26-28 Aug 2020. URL https://procceddiggs mlr. press/v108/1to2000 htttomm0m.mmmmll1
Dusan Jakovelic, Dragana Bajovic, Anit Kumar Sahu, and Sourmmya Kar. Convergence rates for distributed stochastic optimization over random networks. In 2018 IEEE Conference on Decision and Control (CDC), pp. 4238-4245, 2018. doi:
Kevin G. Jamieson, Robert D. Nowak, and Benjamin Recht. Query complexity of derivative-free optimiza- tion. In NIPS, 2012.
David Kinderlehrer and Guido Stampacchia. An introduction to variational inequalities and their application. 31, 01 2000. doi:
Anit Kumar Sahu, Dusan Jakovetíc, Dragana Bajovic, and Soummya Kar. Distributed zeroth order opti- mization over random networks: A kiefer-wolfowitz stochastic approximation approach. In 2018 IEEE Conference Decision and Control (CDC), pp. 1551-998, 2018. doi: 10. 10.1109/ 10.109/CDC.201 2CC.2018.866900
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.
Wenjie Li and Mohamad Assaad. Distributed stochastic optimization in networks with low informational exchange. IEEE Transactions on Information Theory, 67(5):2989-3000 2021. doi: 10.1109/TTI 2t0//IIT.2222 3064925.
Sijia Liu, Jie Chen, Pin-Ye Yu Chen, and Alfred Hero. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In Amos Storkey and Fernando Perez- Cruz (eds.), Proceedings of the Twenty- First International Conference Artificial Intelligence and Statistics, vol- une 84 of Proceedings of Machine Learning Research, pp. 288 297. PMLR, 09 11 Apr 2018. URL attps://proceediin mlr. www..press/veee vrres//844/iul8a. html
First--First
Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In ICLR, 2019.
Sijia Liu, Pin-Y Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney, A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEFE Signal Processing Magazine, 37(5):43 54, 2020. doi: 10.1 10.1109/ MSP.2020030033355
Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled Letaief. A survey on mobile edge computing: The communication perspective. IEEE Communications Surveys Tutorials, 19(4):2322-25) 2358, 2017. doi: 10.1109/ CO000/COMST2200 2017.274522.
Ion Matci and John S. Baras. Performance evaluation of the consensus-basee distributed subgradient method under random communication topologies. IEEE Journal of Selected Topics in Signal Processing, 5(4):754- 771, 2011. doi: 10.1109/JSTSPP200 2...SS....1..200000
16

---

Published in Transactions on Machine Learning Research (11/2024)
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communiiationn-Ffiiiinne Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 20-22 Apr 2017. URL mlr. wttppp//////mmaaaaaaaaeaeeeaaeae
Elissa Mhanna and Mohamad Assaad. Single point-based distributed veroth-order optimization with a non- convex stochastic objective function. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 24701 24719. PMLR, 23-29 Jul 2023. URL htttps://proceedingg .ml.iiinnsmmrrr....mm. press/ press/v202
v202 mwanna23a......mmmma htmml
Angelia Nedić and Alex Olshevsky. Stochastic gradient-push for strongly convex functions on time-varying directed graphs. JEEE Transactions on Automatic Control, 61 (12))33933 -9447, 2016. doi: 10.1109/TAC.
2016.2529285.
A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1009, 2009. doi: 10.1137/070704777. URL
https://doi.ors 11170000717777740400
Yurii Nesterov and Vladimir G. Spokoiny. Random gradient-free minimization of convex functions. Foun- dations of Computational Mathematics, 17:527-555 0667-566,47 2017.
Shi Pu. A robust gradient. tracking method for distributed optimization over directed networks. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 2335-2341, 2020. doi: 10.1109/CDC423400. 2020.9303917.
Shi Pu and Angelia Nodić. Distributed stochastic gradient tracking methods, 2018. URL https://arxiv.
org/ hog//bb/18855.0 11454
Guannan Qu and Na Li. Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on Control of Network Systems, 5(3):1245-122 1260, 2018. doi: 111099TCCS.201..22000
Ohad Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In Shai Shalev-Shwartz and Tngo Steinwart (eds.), Proceedings of the 26th Annual Conference on Learning Theory, volume 30 of Proceedings of Machine Learning Research, pp. 3 24, Princeton, NJ,, USA, 12 14 Jun 2013.
PMLR. URL https:://proceeddnng mlr press/ v30 w030/33//////a/ Sttmm//aaaaiii..a html
Wei Shi, Qing Ling, Gang Wu, and Wotao Yin. Extra: An exact first-order algorithm for decentralized consensus optimization. SIAM Journal on Oplimizalion, 35(2):944-966) 2015. doi: 10.1137/ 40916661996..
URL attpp:://doi.orr/10... 1117 1137/14096668X
Zai Shi and Atilla Fryilmaz. A zeroth-order admm algorithm for stochastic optimization over distributed processing networks. IEEE INFOCOM 2020 IEEE Conference on Computer Communications,, pp. 726 735, 2020. doi: N0.109//NFOCOMM1004338 202209115000000019555550000
Yujie Tang, Junshan Zhang, and Na Li. Distributed zero-order algorithms for nonconvex multiagent opti- mization. IEEE Transactions on Control of Network Systems, 8(1):269-28)) 2021, doi: 10.1109/TCNS. 2020.3024321.
Zaid J. Towfic, Jianshu Chen, and Ali H. Sayed. Excess-risk of distributed stochastic learners. IEEE Transactions on Information Theory, 62(10))5553-5777 5778,, 2016. doi: 10.1109/TIT.2016, NTTT.2006.2597669
Sheng-Yuan Tu and Ali H. Sayed. Diffusion strategies outperform consensus strategies for distributed es- timation over adaptive networks. IEEE Transactions Signal Processing, 60(12):6217 6234, 2012. doi: 10.1009/7SP.201222217333
Anirudh Vemula, Wen Sun, and J. Andrew Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. ArXiu, hbs/1990..11000 2019.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Yuanyu Wan, Wei-Wei Tu, and Lijun Zhang. Projection-free distributed online convex optimization with o(VT) communication complexity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, htmmlllbbmmmmm
pp. 9818-9828. PMLR, 13-18 Jul 2020. URL httpp:://proceeing....... mlr wwwpenngs..rr...rrr....
Yuanyu Wan, Guanghui Wang, Wei-Wei Tu, and Lijun Zhang. Projection-fres distributed online learning with sublinear communication complexity. Journal of Machine Learning Research, 23(172):1-53, 2022.
URL htt.:ppp///.pp.r.o//pppppssss//s wttp..rg/ppppp/////2000----
1239. html
Ran Xin, Anit Kumar Sahu, Usman A. Khan, and Soummya Kar. Distributed stochastic optimization with gradient tracking over strongly-connected networks. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 8353-8358, 2019. doi:
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 26(3):1835-1825 1854, 2016. doi: 10.1137 11117111094.70. URL https://doi.orge:
130943170
Fanqin Zhou, Lei Feng, Michel Kadoch, Peng Yu, Wenjing Li, and Zhili Wang. Multiagent rl aided task offloading and resource management in wi-fi 6 and 5g coexisting industrial wireless environment. 1EEE Transactions on Industrial Informatics, 18(5):2923 2933, 2022. doi: 10.1109 10.1109/TTI.200 211.
A L-Smoothness Property
-11 Σ (V)(()) VF(()))
VF h(x)]l
(F(()) VF
- x ||
Σ
1=1
(a)
I 2
1=1
L
x
1z
where (a) is by applying the Cauchy-Schwarz inequality,
Σ a; 1| < (Ei a?) .E 12)1
nd (E EE=13 a and (b) is by definition of the Frobenius norm, xx 12| Σ || =/12.
B Estimated Gradient
In this section, derive the bias of the gradient estimate with respect to the real gradient of the local objective function. Let
G..
Thus, by Assumption 1.3 and the definition in 14
Ji. = (.s.o.c4.....
+ n 2i Si.e) + Ci,)) Ho]
= Di f [74c]
= E. P F. F((Di, + + 0.
18

---

Published in Transactions on Machine Learning Research (11/2024)
By Taylor's theorem and the mean-valued theorem, there exists Hi located between 24,8 and 24,6 + Yh Pi where
Th (B (
Fi + p.,)) (. + 7A (Pi, VF((a))) +
substituting in the previous definition,
"k E+ F((L,))
A (.(. + W
ca + by. Thus, the estimation bias has the form
Di.k
d
C3
7/ Es
D2.s
V (..,,)) P.. [Hx].
2c3
Let Assumptions 1.2 and 2.2 hold. Then, we can bound the bias as
"k Exp
|b, -
203
(20)
$ 7k c1
203
We can see |lb, - 0 as le > da since " is vanishing. We remark that
gx Elgx [44]
B[gr, [44]
(21)
Ya V F. (T..)) + dg.k]
cs7 (x<x)) bel
is also a biased estimalor of ((xx) with
|
-1
[=1
||b
(22)
1 72
T'k cic1
71
203
ci
7/k
203
Lemma B.1. Let all Assumptions that 3111@n1191Ha. < M.
1.3 2.2 and 2.4 hold, then there erists bounded constant M > 0, such
Proof. Vi E N, we have
[H
(file, + Ti Si,k) + (1, Bil
-Bll 11] nu P St,e) + C. (1HH
(a)
(F P.. S..) Cr.e) 2
(6)
GEF, (œi, + P. [4, c202
S
00,
19

---

Published in Transactions on Machine Learning Research (11/2024)
where (a) is due to Assumption
2.2 (b) Assumption
L.3 and (c) Assumption 2.4
Then, Ell gk [Ha] E
Σ== 1 12
Ex=
Ell [ <
E
and
|1 17+c]
=E
9 11
H4
11
Pi.A
712
I2
119
11
71
C Stochastic Noise
K To prove Lemina 3.1 we begin by demonstrating that the sequence {Ek k 0uEEkK'KK is martingale. To
do so, we have to prove that for all K' 2 K, Xx E k orez satisfies the following two conditions:
K'S
(i) E[XK+1||KK] =
(it) X [2] < 00
We know that
Elex] Blgx
[[gh|HHa]] Ext
B gxx
E[gx
Hol
His
by the law of total expectation. Hence,
K'
K'
K'
G[X = E aax411
+
Σ
aLEk
0 +
Σ k=
aget = XK..
(23)
k=
l
In addition, ek and ey uncorrelated for any ie + We' since (assuming >> K) E c cw| = B
eN [P
E Ble [Ha] 0. Thus,
K' E(Il
K'
Σ
agez 11
E
Σ
aga (ek, ex))
K'
E
k=
Bia Ig+
E[gx (Ha]l
K
E(1ãa 12) - Eze, [[lE[[n//HH]])
E.2x( 113)
(24)
k= K
(6)
(c)
M
< 00,
k= K
20

---

Published in Transactions on Machine Learning Research (11/2024)
where (a) is due to the uncorrelatedness Ei(ee,, ex)) 0, (6) is by Lemma B.1 and (c) is by Assumption 2.1 Therefore, both (i) and (ii) are satisfied and we can say that {E axCc a martingale. This permits us to use Doob's martingale inequality Doob [953]]
For any constant 0,
K'
K'
P( sup K'2K k=k
||
Σ ckeall 2 v)
E(1
Σ
12
k=K
(25)
(a)
M
V2
ak
k=k
where (a) following the exact same steps as 24
Since M bounded constant and limg ooo Σk= K o
by Assumption 2.1 we gel
M lim 77 2k= o for any bounded constant Hence, the probability that 1 CReEll also vanishes K + oc, which concludes the proof.
D Proof of Convergence
We start by stating the following lemma that will be useful for the proof of convergence.
Lemma D.1. If all Assumptions 1.1
1.3 2.1 2.2
and 2.4 hold, then limg-+oo ||x 10 = O. In fact, we
have
20
XR 1xp
1 00,
Σ
2,+11
1xx11 oc, and aa X 1xxll 20.
almost surely.
Proof: See Appendia D.2
D.1 Proof of Theorem 3.2
By using the compact form of the algorithm in {8] we know that
2k+1
1 aggs)
(a) T (xx - agg;) = 23. - -aggk.
(26)
n
where (a) again due to the doubly stochastic property of W.
The divergence time + 1 can then be written as
d++1 Tk+1 x= 1|2
(0i 11
11
|0 - x111
|2,,+1 - 12
21

---

Published in Transactions on Machine Learning Research (11/2024)
21,4+11 Ft Bkk x=1
|22,1+11 - 1x +2 --
22, Tx. 2g xx)
[ x "1
im1
km
|b++1 - 1xxl1² + +((2x+1
Tke Th - x²) + || - x 112
1
Ak++1 15xl1² + 2 04 Ōk. Tx x²) + dx
1 dx 20r (Tx -=" 0 Elg H4) + E [HHx)> + -
-1| ||0 1
n
(27)
-d- 20 (6x x², Elgx [+++) 20 (Tx - x=,,x) (x) + || - 155x/'
( dx 2037R0x x" h h(xx)) + bx) 22x (Tx - xx",,""" ,e,) + =||z++1 1
1
n
dx 2033R00 x", vF((x)) + 203 (Tx - -x", (F(Tx) 11(xx))
1
20 x² ba) - 2ax - x², cc))
| 1.
71
( S dx 20sTRa x", vF((E))) + 20a 2
|| - x² ||<x - 1xx
vn
+ 2037Raul13 x bx|| 2 x², eck
|z++1 1Fxl1
where (a) is by the projection inequality (5) noting that x" E K (so projecting it onto IC gives us the same
point), (b) is by 26 (c) is due to (1 and (d) is due to Lemma 1.5
By recursion of inequality 27
we have
20IL K vn
dk+1 S do - 20s
naa (3x - x", VF(ik) + bu) +
Σ
naa |0 x= x=1||xx
1xx
k=0
k=
(28)
K
+ 203
naa Eb x² |||bb||||
2
Σ
a (ix x² ex)
+
|k41
1xx112
n
k=0
k=
k=0
By Lemma 3.1
we have limx-
E
k= aLEA
K oo almost surely. Since | - x+1l 00 by the compactness
of KC in Assumption 2.4
hence
lim
ag x², x=,ex>>ll= 80.
(29)
00
km
From 42 in Lemma D.1 we have
K
lim
||6+1 12xl12 oc.
(30)
+00
k=
As stated in Lemma D.1 we have E n 7h | - -1xx/1 < oo, adding to || xI1l < oo by Assumption
2.4 then
K
lim
naa - x xx|||| 15xll <c.
(31)
K +00
k=
By 222, we know that |lbel| <
20g Th and
|| x1 < oc by Assumption 2.4
K
lim
2 az
li x²1l < 00,
(32)
K +000
k==
22

---

Published in Transactions on Machine Learning Research (11/2024)
by Assumption 2.1
From the above inequalities 28 KK defined as
32
we see that there exists 0 < D' < 00 such that dx+1 S D' + +2,, with
2K
Taa - N VF((a)). 203
(33)
By the strong convexity, we have
A 1 - x² S 0,
(Tx - x", VF(Ta))
(34)
as F(0x) F(x*) by the definition of being the optimum in and 3y (by the property of convex set).
Thus, ²K < 0, confirming dx+1 < cc.
Let's assume that Yea > 0, EKa, such that | - x²12 > en for ko 2 Kh, meaning
K T - x²1|3 < -Eh lim
K
lim
7hap < -00,
(35)
k=
since Σx 0%7 diverges by Assumption 2.1 However, this implics that 2 -00 and consequence,
dx+1 < D'
-oo which is contradiction as dx+1 2 0. We conclude that limg da = 0 and lim T2 = x,, almost surely.
D.2 Proof of Lemma D.1
The goal is to bound X
1 by | - 1xxl and other vanishing terms.
| 15 2
X6+1
15x + 15x 1 ||
X
1 + 2(xx+1 17,, 17, 17x+1) + | - 12
Xk+1 1=x|12 || 15
X6+1
1Txl1
| Fxl
|| 12
Z 11
Wx2 - a W gt: 1
Wxx 11 - 20 ( (Wx
1.k. Wgn) + a gx|
1- p2
2p
[Xx
1 + an
[[xx Xkk 15xl1² 2
| gv | + a |g
2p okk
1 P2
(4)
1
P
2p a
|X
15
+ 0kk
15 12+, +
[[
g + az W
20
p
+
22
+ 2
||x
17
+ 0%
[gghll g
2
P
+
2
1
+
2
[x
17
2
0
W gi
17h + 17x
2
1
P
(e)
+
Pe
+
p
n2(1 + 22)
2
|xx
1x
2
0
2
Wig
gx
19xl² + a
1 p
1h |
23

---

Published in Transactions on Machine Learning Research (11/2024)
+
Pu
Piu + (P))
n1(1 + p
2
Xx
17
0k
1 P
Bk
10 + Otk 1 P
(36)
(
+ 2
7 - pw)
2
|x 1
0k
2
Pio
M.
where (a) is by 377, (b) is the projection inequality 5 noting that In E K since K. is convex set (so
projecting it onto K gives us the same point), (c) is by
2c x 1 (0, = -2(ea, b) - e2|la11 + 1b1 (d) is
by Lemma 1.4 (e) is by 388 and (ʃ) is by 39 and
40
2(x 12 15 11x+1) =2
[iit+1 - Iz - ox+1)
Σ (0i,K+1 3x), Tx Bk+1)
(37)
2mn(Tx+1 Tx), 3s. Tk+1)
2n(Ix 2k+1, Ik xx+1)
2n |2 Ik+1ll
= 2| - 15x+1
(W gk 15,, 19h)
Willii Āk. Tk)
i=1 j=1
Wij9jA nāgk kk)
2m j=
(38)
Wky }gi,k Tig gk)
gj.k nik- 9h
1=1
=0.
From Lemma B.1 we know that 11 M oo almost surely,
Ba
1gx 1|
Σ
11
Σ j=1
gin 2
1 n
Σ
119r.
2(gi.
Σ
gi.k) + ||
(39)
n
i=1
1=1
|gga| 2 -2rllgxoll 112 + n r0ll@x
gg
2 -1|
-n|l@xll 2
Then,
1g + nl|ga 1|2 = pullg* + n(1 - p 1l
P + n((1 - PM
(40)
= n.M
24

---

Published in Transactions on Machine Learning Research (11/2024)
1. Proving lim--+0 IR=0 ||os
18xl1² oo, limo-+00 Σk=0 |z++1 1Fx11? oo, and limg- I
| =0
Reconsider 36
1+
1
Pw
2 n1(1 + P
X4+1 - 10s+1 23
Xx -
10x 12 + 0%
M
2
- P
1 +
Pw
n1 + pw)
|x 18x 1
Xk-1
18x 11 + a
M
2
1 - p
(41)
1 +
p
mm(1 + P
||x] 15, 12 2
2 xo 15
+ 000
M.
1 P
Adding all inequalities in
41 we obtain
|xx+1 12 2
Par
x 1x/l1²
+
1 Pm
X 1rol1
n21 + Pm
M
ai
1
p
t=0
Let 1e + bo, then the second and third terms are bounded due to Assumption
2.1 There are then
2 cases: Eτ xx] - L either diverges or converges. Assume the validity of the hypothesis H2)
Σ, || 1x/l2 diverges, i.e., E ||x] 1i; 1|2 + cc. This leads
| 15x+ -x0
1 1-p2 However, ||x+11 17 2|2 should be positive. Thus, hypothesis H2 cannot be truc Σ1 X 15 converges. Hence, lim ||xa - 1Tx 122 0 almost surely.
as
and
Thus, reconsider (36
1
P
n(1 + Pmn)
1b+11
1²
[ 1xx 1|
+ +ak
0k
M
2
1 P
1+++1
1x
1 +
P
|
1
n(1 + P
M
2
(42)
1 Pd
k=0
k=
<00.
2. Proving Σ maalIx LFxl1 < 00
By induction from 36), we have
| 15 1 <
1 + p
2
k+1
Ixo
1Tol
+
2n.M
2
Σ
4:
+
2
12
jt1
Ok
2
(43)
j=0
Since Va + b Va+ Vb.
+
P
L+1
2nM
1
22
it
|x 15x-11 |||
Xx
15 +
ag-j
(44)
2
Piu 1=0
Then, substituting into the sum E
1x
Σ
8
7Rock
+ 2
|xx0 110ll
2n.
k-
1 1+ 2
1+1
0k-
2
1
p
2
A=1
1=0
53000|1200
15ol
/2
1 + p
/ + Pt
+
V
2nM
p k=1
70
Σ
k-1
1 P
it
0k-1-3
0=0
25

---

Published in Transactions on Machine Learning Research (11/2024)
where the inequality is due to the fact that 7 and ag both decreasing step-sizes and we have geometric sum of ratio
1+22
1. We then study the sums in the second term,
k-1
1 +
Pw
k-1
P
2
a--1-j
7
2
1-j
k=1
3=0
k=1
1=0
k
P
h=j+1
2
2
j-1
1=1
1=1
2
+
Pu
k-
j-1
7k
2
j=1
k=
2
+
Pm
900
0
2
j=1
k=j
1 + P
8
=70 V2
1 +
P
f=1
<00,
as Σ az converges by Assumption 2.1
Finally, Ekzo Yk |Xx Xx
17g| < Co.
D.3 Convergence Rate of the Consensus Error || 15xl1 11 and of ||x++1 1Txl2
As Σx ||x 1 < oc, let us assume that ||x 17x 122 vanishes with the same rate og Then, there must be a scalar 01 > 0 such that ||x* 1rxll² < Sla.. To test if such 01 exists, we employ 36 to check
whether ||x++1 18 1| < saak1 holds,
1 + P
n11 +
P2)
|x+1 15
12
X5 17
O
1
Pu
1 P 0.00 0 +
n11 + )
OKk
(45)
P 0 +
n11 + P
1 1 P 0/k
Then, testing
1 1
2
n11 p M
0 +
10 S 19
2
n (1 + 2MM
a
+ 2
1 P2
0
2
(46)
m21++02 )M
P
< 0..
1+ 2
4
Thus, 0< 0< < 0o whenever 2f
1+p2 2
2v
Let consider ag having the form in Example 2.3 then taking values between and 1, and define
A+1
is increasing function of k
ag
k+
K₁ arg min k.
26

---

Published in Transactions on Machine Learning Research (11/2024)
To test whether K1 grows very large, we find the intersection
T
1+ 2
+ )
k k+1 201
1
Pmu
++
( Pee
1: +
2
h 1
(k 2)
+
Pm
201
(47)
2
1+p
20g
2
k:
1+p2
201
Define the function h(,; u1) Jor
for 0 < <1 and 0.5 1.
26T
@h(a.o))
exp(
In 20g
0 for a fixed 0 0<<<' <1 1.
201(1-22 ToI
Buy +1 2v1
0b(a.v1)
for a fixed 0.5 01
0x
20r (1-x 201
Taking extreme case of x (0.95, 0.95) s 36 iterations. It decreases even more drastically for realistic choices of and v1. Thus, it is reasonable to study the rate for k 2 K1.
0.99, obtain h(0.99, 0.99) 196 iterations. For 21 0.95,
We conclude that for 2 K1, there exists 01 < 00, such that
- 1 < 89a2.
(48)
Thus, from 36 for k 2 K1, we also have
1
+
P
n(1 + P M
|z++1
1xx 1
| 1r
+ ak
2
1 pu
1
Pm
07 +
n11 + p
ak
(49)
1 P2
E
Convergence Rate
Our primary result, stated in the following Lemma, is based finding a relation between two successive iterations of the expected divergence.
Lemma E.I. Let A = Adg
AcgL200
An C
cec.
Then, for k > K1, caa and d2 a
2
D&+1 S (1 Aa + Bo Coun + Eok.
(50)
Proof: See Appendix E.
Next, we let
K₂ arg min Aaara1
and Ko = max(Ki, K2). For the ensuing part, the purpose is to locate a vanishing upper bound of Dis making use of the inequality (50). The idea is to propose a decreasing sequence Ux41 S Ux and suppose that Dj < Un, Yk 2 Ko, and then verify that De+1 S U8+1 by induction. The choice of Un the most difficult component as one has to keep in mind the general forms of ag and % (50) and what kind of decisions take regarding these forms. An essential property of Up is presented in the subsequent lemma.
27

---

Published in Transactions on Machine Learning Research (11/2024)
Lemma E.2. If decreasing sequence Ukt1 Uk for Ko erists such that D S Up+1 can be deduced from Da S Us and 500
then
B
C
E 0r
Uk 2 A
0
A Tk
(51)
7
Proof: See Appendir E.9
An important remark is that the lower bound of U₁ in [51 is vanishing as az 16: and ae are all vanishing. This lower bound provides an insight on the convergence rate of D3 as cannot better than that of a 7k.
ax "
The previous Lemma allows us to move forward in confirming the existence of the constants and $2 that permit D3 S <in and Dp as in Theorem 3.4 respectively.
E.1 Proof of Theorem 3.4
Proof of 12
By definition of Sı, DKo S11Tk. The next step is make sure that D4+1 S Uk+1 can be obtained from DE UR, Vk 2 Ko. Take UR si let De Un hold, and substitute in (50g
D4+1 (1 Aawn)ssnt Baies + Car Eaq.
We solve Dk+1 Uk+1 for ç R
Aaxtn)sin Baon + Caun + Faj < U2+1 = 117241
1-(2)21)
Then, by considering K%
0 given in 111.
06%
Bat TK
2 + Eau 3 + S (1 kk),
and assuming A - Kx > O, we find a constant si such that
Bat EO +C
A- Kg
keeping in mind that Bo Fa 3 + C is positive by definition. Examine the parameters d 02 and dg as they are introduced in then
Boz + Eos +C
A A-61 01
We conclude that D2 S sins where i satisfies the definition
13
Proof of [4 7Ka
DK. I 52 axo
by definition of $2. Wk Ko, let De S 52 on
then
3x
D&+1 & (1 Aon On
Batra Cau + Eot.
7
Solving De+1
22
atil for 22 Rt,
78+1
(1 Aox 7k 52 as
+ Ba + Cours + Eag s 52 ax+1
7
7++
ak ott1
Take 72 76+1
> 0 as given in 11
then
ob
Back + Ca
F. < (A Tx)
28

---

Published in Transactions on Machine Learning Research (11/2024)
If ok-1 76 76t1
Aag, then & such that
$2 Ba + Caa -1 1
7k
+ E
Tk)
Examine 01, 0s, and that defined in
we say
Bos + Caa + E
C S
(A- 04)
We conclude that D2 S 52 an with 52 satisfying
15
E.2
Proof of Lemma E.1
Starting with the same steps as in 27
Da+1
=B
[|| +1 x²1 1²
<
Zk+1 - 1xxl1² + 2( 2(-ax 26 - x²) + dsl
=D:
Bl|lz411 15 || 20 El [Tx x² .ghbl
11
(a)
D2
B[[| 1² 26 x², h(x=) + bb))
DD
B[[|
[x+1
1 2c x", V.((F)))) 2cgar Ex - -x", VF(())) h(xx)))
BB - -x",, bb))
(52)
where (a) is due to both E[ex [H(}] and 21
EE [x x², gg>)
[2 x², E Gg [H4] B[gg HH&))
[T _x,eex)) ex)) + ( x", B[gx 17<+}}
EHH [T x=", .x)) He&]] - B( - x², El He&)
+ [x - xx, Ei [Hu]))]
From Lemma B.1 we have 01 < M with M a bounded constant.
By the strong convexity in Assumption 1.2
we have
RR x², VF(Gx))) s 2030 71 B[F(x*) F(Tx)) 7/ B(| - x= x²11
S - 76 - x= 1=]
(53)
-c33au
where we used the fact that F(x*) F(xx) SO.
Next, from Lemma 1.5 have
2030mm - x" VF((x) (xxx))) < 20 -
Th x x'11[xx 15x
(0) Ac3
122
| x:" $12 4c300774 An
xx
17 12
4
where (a) due to 2 (a, b) 2(Vea, 1 < e||a|| + 11111 From
48 we have for > K1,
Xx
1 < 0.
29

---

Published in Transactions on Machine Learning Research (11/2024)
Hence,
Bl x., VF(()) h(xx))1 Aczasnt Dr
40gL2 ax
(54)
4
An
From 22
Acza4 7k D2 + 4cz0rk 7k
2030u x²,, xb)bb))
18 [1bb
(55)
Aegan7k 7k
cifaam
4 D3 +
c3
From 49 for h 2 K1, we have
32 S
Bil 2++1
12
ak.
(56)
Finally, by combining 62
53
and 56
we get 50
E.3 Proof of Lemma
Since 1 - Ya > when Ko, we may substitute D2 S Us in 50
D4+1 Aax7)uu + Ba + Cam 3 + Eag
Testing Dk+1 $ Ukt1 in the previous incquality, get
(1 Au + Bait + CaN Eat S Uk+1 S Un
B
C Ook < Up.
a + 1h
(57)
A
A
A 7
E.4 Proof of Theorem 3.5
Thcorem 3.4 indicates that the convergence rate is a function of en and U2, as 7 (A: + 1)-222 and & (k +1 (ui (w1- Nonetheless, we must still verify the validity of the assumptions presented in the theorem, meaning:
Are 0i < and σ4 A fulfilled?
Are si and 52 bounded?
We must remark that in what follows, the analysis is done for k 2 Ky-
Let ag and 7t have the forms given in [6
Verifying that σ1 A and 04 <
The idea is to find bound on and no to guarantee 0s < A and 04 < A. We start by bounding σn and 04 from above, i.e.,
(
1- (1 ) -20
202
01
max k2 Ko
k2Ko
0000 11-11-12
and
02417
1- (1 } -v(11)))
0
max
max R2
R2 Kg
Olk 7kk
0070 (k + -21-2
30

---

Published in Transactions on Machine Learning Research (11/2024)
To do so, define function ((z) == (1 have ((x)) I x1 (1 + x)--) = r(x). To further bound g(x), We study the derivative of r(x) it is simpler to do s0,
=11 x)
with a, a, (0,1). Since x==x== we
((x)= [ ( 1)(1 +2)+--1-111-- -1---1=-(
Hence the sign of (x) is that of s(x). We again calculate the derivative of s(x) find its sign,
s s'(x) = b(b + 1)x(11 + x--
6- 1 0
since > 0 and > 0. Then, s(z) is decreasing function of over (0, We remark that so s(x) = O, mcaning s(x) < 0 and (x)) < O, Vr E (0, 1]. Finally,
-1 x) b x
r(x) lim r(x)
z z0
and ((x) r(x) b, noting that o ((x) b for a We conclude that 0r
.222. and ao7o
04 < -1-22, For < A and aa < A to be valid, we must have
0070
0o7o 2 max (202, in U2//A.
(58)
Verifying that Si and S2 are bounded
The goal is to verify that the constant term in the convergence rate is bounded. Thus, we must check that the lower bounds given in [1 and 15 are indeed finite. We start by analyzing 02 and
02 -0% 2.-2 max (1 + --- 2(01 02)
a0%00(1 Ko) 2(01 v2)) -
0 U2 S U1.
12
and
F5 on To max (1 -)) =1) Ko) < 2/2 U1. Ko
We end with the analysis of Jy and 0e, i.c.,
03 00% 33 max (1 + (1 B0s)
aano 31 Ko) Ko)-(11 (v1))
if 2 3U2
k2Kp
if U1 < 3
and
au 70 + Ko)
3uz if v1 S 332
06 = on
1 1o max (1 + -302
k2Ko
00, if 11 > 302-
There are clearly 3 cases:
V1 > 302
Thus, is bounded.
Since 02 and S (by definition) are also bounded provided that oo > 202 in 58
However, S2 - 0o since as - so resulting a loose upper bound in 14
To that end, we can write D & T,(1 + k) 2ng with Y, a bounded constant.
Ui < 302
Similarly, dg is bounded while 03 - co. Then, 3 Tg < oo, where Dy < Y2(1 + k) (01 provided that an 2 15 A
V1 3t2
Both d3 and 0e are bounded allowing both previous inequalities corresponding to D3 to be valid.
31

---

Published in Transactions on Machine Learning Research (11/2024)
By this analysis, conclude the proof of Theorem 3.5
We present Figure S for easier reading of the conditions on the step sizes' exponents where we plot V/ vs. Vt-
01 + 12
11
171
Figure 8: Plot of Up vs. where the yellow shaded area is the fcasibility re- gion determined by Assumption 2.1
F Regret Analysis
Consider the following modified definition of expected divergence that denote by D'-
D'd = E
1 x x²1
We then develop this entity,
D/+1
=E
-
[0,,+11 - 11
72
(a)
E
x²112]]
72
wiy(ay, 04 -xn)
x²) |
72
- wiy |,g. oxgiA - x² x²1
Σ 1a C.Jji - x²1]]]
1
=D/- - 2ak
[2 x 9"", + ax
-
Elllgh
j=1
1=1
=D- - 203047
EllTzA x3, (1)()))) + bلnرن
+ oz
EE E111g,.111
j=1
1
D/-
20
- -F((") + C3O
Bll x²11 111 af.
n
S.
D/ 20304
Σ
-E((() c c3a07
+ az,,
71
403
32

---

Published in Transactions on Machine Learning Research (11/2024)
where (a) is by applying the projection inequality 5 (6) is by the convexity of the norm square function, (c) is by the doubly stochastic nature of the matrix W, (d) is by the convexity of the objective function and Lemma B.1 and (e) is by 20
Then,
([ - F((2)) S c30
D2 D2- -Di+1
+ D2 +
c 2
4r3
7
C3
OA M.
(59)
We know that D/g can be written as
D2
=E
1iik - 2²1
n
18
1a Ik + 2b. xx"1] xx1
72
(60)
18
|1:0,1 xx + 2(z
Tx.. Tb. - xx) + 1| x 113)
X 1 2 - 2 - Ik. 3x x²) + 1 - x x²113]]
n
E
18x
1xx 11= 11 - x² $12
Hence, to find the regret bound, we write
E
F((EAK))) F((x*) -F((=)
Ki
K
D/ D/+1
+ D' +
cJ 2
OAk M cssk
Cya
4gg
k= Ka
K
D'
C30k-17k-1
C3000 7Ko
D'Ko
D
+ Σ
K
D'
4eg
7k
+
C37N
OK
M
k= K1+1
k=
(
1
+1
Σ
D/
+ -1)D'rs
D'+11
Mao
1
70000
C300K,
C3OK+1TK*
4
Vk +
0370
vk
k=Ko+1
k=Ka
PPad
1
(K +2)
/Pa0d
+1
+T
+ r
C3
(k: + +)
V
C30070
(K +2)
/K + 2
k= ko+1
Mao)
K
+ D'k
Σ
+
+
+
CzacK. Yoo
403
Co
+I
k=
where (a) is following up from 59 (b) is by substituting a% = aolk + 1)- and 7 = 7o(k + 1)-2, and (c)
is by 60 Lemma 3.3 and Theorem 3.5
To find an upper bound, we interpret the sums over Ko+1 S k < K as Riemann sums in which the functions - evaluated at the right endpoint of the interval [ 1, for i = Ko Ko 2.... K.
(=+1)* and
Since the functions
(w+1) and V are monotonically decreasing, the sums are in fact lower Riemann sums and therefore bounded from above by the integrals f (u+1) du and fkk Vu du, respectively.
Σ
1
/
K 1 du 2
(k + 1)
Ko (u + 1)
VKn
VK +1
k1
33

---

Published in Transactions on Machine Learning Research (11/2024)
k
Σ
du 2 v +1)
+1
Ko Vu +
k= k=
Finally;
K
Biag
1
E
F((za)) (actx)-P((())) P((00)
+1)
+ M[[V+IIVKo+H
C30070
VKo+ +1
VK+1
k= Ko
1
[Paa
1
+ +TVK+2)
0300000
n
VK+ +2
+
1
D'o +
199970
+
2Moo 0370
)
(VK+I Vk)
C3aK, YKo
2c3
G Convergence Rate with Constant Step Sizes
We start by going over previous derivations,
Js.k Es (R....l + y Sc,,) + [144
Eo(Di, Fi (37,e +000,,,+-900 +1H,)))HH
22 F(((i,)) Eo (i.kk] + yEo P T. H4 (F - Ro
Bo P t (
F + b.,b).
Thus, 61.6
2cg
1, ( k 2 ( [H)]
Let Assumptions
1.2 and 2.2 hold. Then, we can bound the bias as
7 Exp [t.ll
|bb -
Hx]
203
caa
S 7 203
We remark that
Eigu [Hp]
Elge, [PA]
n
(61)
1
c3>l VV + b
n
c/ + bn]
is also a biased estimator of (xx)) with
||b1|| = -11-
Σ belell
12 i=1
||br,a|| |I
(62)
cee
20s
1=1
? 203
34

---

Published in Transactions on Machine Learning Research (11/2024)
Lemma G.1. Let all Assumptions 1.3. 2.2 and 2.4 hold, then there erists bounded constant M > 0, such that E El I?]] < M.
Proof. Vi E N, we have
Blllg, = (f(., G4
([ S.,) Si, 17
[B[(J + +7 St,e) G. 2.)) [
GBB + t S Hm) + c¹A2
xc,
where (a) due to Assumption 2.2 (b) Assumption
1.3 and (c) Assumption 2.4
The stochastic noise is still defined eg 9k 9L and retains its property
Elea]
0. 10, | and Z6+1 1 converge linearly
Blox E[ Ha]] = Ex 9r Elin [Hx] His
1. Proving ||X
X++1
18
=
| 15x + 12x 1 1|
xk+1 1zx 112 2(xx+1 15 1Tx 15441) + ||r 1=
Xh+1 15x 1|
1.xx 15x+111
Xk+1
15x 1|2
|,0,,+111 Fx 1|
(6)
||,,6+11 - Tx
Mx+1 1 112
W aW g: 12xl²
| Xx 1 212 2a X2:
17,, W Wgs)
a² [| 1
22
2p a
W
X
17 2 + 01|
I Xx:
15x + 1 p
|[ [2] + a² Wg |2
2p
(d)
2 Pie
2p a
Pio
2
[
152 11 + a
17
+
| g:
1 + a
W gx |
20
1- p
1
1+
2
1
+
P
| 1r 1
+
2
| gi: g
2
1
P
1
+
1
+
2
15
2
| gi:
15k + 1 11
1
P
(e)
1 +
P
X 15
2
1
2
[ gg
1 2 + a nT + p
1 1|
2
P
1 P2
1
Pm
X
17
2
+ 2
Piu
P
gg 1g 12 + a² 11(1 p
|1gh
p
(/)
<
+ 2
| 17
2
+ a²
n(1
2
M.
2
1
P
(63)
35

---

Published in Transactions on Machine Learning Research (11/2024)
where (a) is by (37,, (b) is the projection inequality 5 noting that By E K. since K. is convex set (so
projecting it onto K gives us the same point), (c) is by 2x x (a, b) 2 bb - e ||al| + / |b|
(d) is by Lemma 1.4 (e) is by (38 and (f) is by (39 and (40).
By induction, we have
| 10
11
-
+
pe
k+1
|x0
1
+ a² 2n.
k
1 +
j+1
2
1 - Pte j=0
2
(64)
1 2
+
P
k+1
||x0
15
a 2n. (1 + Pie
(1 12
where the last inequality is due to the geometrie sum with 1+ <1
We conclude that |X 1 converges lincarly to an a neighborhood of 0, almost surely.
Substituting in 63
1 +
P
+a a.2 m2(211111 + Pw)
| 15x
Xs: 17xl
M
2
1 pmu
1 + p
1 P
||x0 10o11 + a² 21 + Pio)
+ 2 n(1 + +
M
(65)
2
2
Pdo
1 +
Pie
k+1
||x
17
n
1 Pm
+
+ P
2
Puu
Finally, [|8441 17 converges linearly to an a² neighborhood of 0, almost surely, as well.
2. Proving Dr- =B[l x [|3] converges linearly
Dx+1
18
[ +1 x²11 211
7 1ā 2 + aag. Fg - x²) + dxl
=Dx
El [zx+1 1z || 20<B((xx - 99h]
(n)
Dr
Bl Zz441 17g [1 20300111 x", l((xx) + b6ll
Dp Bl| |zx+1 12x 20307 x", vF(xx)) + Blga x", VF(Tx) h(xx))]]
2( xz, bb>]
(66)
where (a) is due to both E[ex|HH]] 0 and (1
Bl x², ,'h']]
[5 xx," Elga [HN] + Ejga [Ht])]]
(Z x=" _,,c=>)] + ( - x=, E 17
EHg [T[ - _x,xx) ex) [7 + E[ - ==, &h 176111
+ B[( - ,,, LLgh Hx])-
From Lemma G.1
we have B 12] < M with M bounded constant.
By the strong convexity in Assumption 1.2 we have
-2c3ay - x", VF(xx))) < ((z") - F(xx)] Ac3a7 - -x*13
< -xc3ary x²1
(67)
= AegarrDes
36

---

Published in Transactions on Machine Learning Research (11/2024)
where used the fact that F(x*) - F(āx) s O.
Next, from Lemma 1.5 we have
L
20300/ - x,, VF(a)) m(xx)) & 2030r
-
|0g - xx x=11|x 1xxl
vn
(g) Aegar
L²
|T x=1 1|2 + 4egary
||x 15x 1|
An
where (a) is due to 2 x (2 b) = 2(vEa, ) S E||a| 12 + 116112
211 In (64) and (651, we let R 1|x00 1ooll and G (1-p
G2 = n.
[+tp
1-
+
P-
1+
+ p |xx 16 S
R + a² and ||a+1 - 1 S
(1 + P
k+1
R + a2202
(68)
2
2
Hence,
2c - x², (F(zz)) h(xx)))) tegar
L
1 p
D2 + 40za7
R a
(69)
An
From
62
2c - xx, b",b,>l s Aegar
4cg00
D3 +
Bllbba11
4
A
(70)
Acgar
DE + ar cf
4
C3
Finally, by combining
66),
67
688
69
and
77
and setting now A Aea
B
4.a
and
2
An
C= 62.6 ca
we get
DE+1 5(1 Da Bay( (
1+ pe
R+ 2G]
R R+
1 k+1 R+ + Coa
2
(71)
1 ++PPps Punj a + a² oz2 Gz pi)]
=(1 - A R Bay + n 2
G2 ag²
n
Let = Aay and 02 = ( (+t22)) ). Then, assuming a9 < A and taking the telescoping sum
k BGi + 0² ---co + ay Σ o
Det1 Sot ++1 Do + R Bay+ + 02
Σ
cie A + BBBBBB
1=0
1=0
1 - 0 k+1
= Do + 12 Bary
ei k- + (a²
a G2 2)
1 0
1=0
(72)
- Do + R Bay
02 n
Σ
k- (a²
BG1
7 C 7 nd A
G2
(1- ditl)
+
sol+i Do + R Bay +
@2
k
+a²
BG
Gz + nd A
+ 2 C
A
where in the last equality, we further imposed the step sizes to satisfy a < In what follows, we discuss the summation in the second term of the incquality to avoid setting loose bounds. We know that this summation can be written as follows,
k
k
Σ
Σ
05
(73)
1=0
1=0
37

---

Published in Transactions on Machine Learning Research (11/2024)
Thus, without imposing further assumptions on the step sizes, consider the following function the two cases:
When d1 02, use the left hand side of the previous equality
D4+1 Sen ++1 Do + R Bay +
02
0
+
a
BG1
a
G2
+
A
nd
A
1=0
Sen k+1 Do + R Bayy +
22
02
+
BG
G2
C
(74)
2
A
nA
A
2R Bay + 02
BCI
a G2 2C
= k+1 Do + 02
k+1
2A07 + P - + a
A
? nn
A
Then, for arbitrary small step sizes satisfying ay < 1 and < 7, Dy converges with the linear rate of (g0)).
When @1 > 02, we use the right hand side
D&+1 Sei k+ Do + R Bay +
L2
0
02
+
a
BG
G2
+
C
A
nd
A
t=
02 Se k+1 Do + R Bay +
01
BG
a G2
C
(75)
1
m
nA
A
2RBay +
2R
BG1
a G2
2
=1 k+1 Do + 1 - 2A07
n
+ a
PM
A
nA
A
Then, for arbitrary small step sizes satisfying ay < and < 7, D₁ converges with the linear rate of O(e)).
H Additional Numerical Examples
depict the classification of images with the labels 2 and 3 and Figures 12 14 depict those with the labels 3 and 4.
Figures g 11
DSCT venistngg
step with constant stap vina
1F-G3 with constett step vee
1F-00 with estimator
Irahn Loza Evolubin
vambding stop
with comtaatt stop
0.8
25 DSG with noisa
DSGT with vonishing step size
Explccted
DSST with constart stco site
step
0.6
GO with estimator (6)
JP-DSG with vanisiigg stop sizes
3P-D5G with
with notee
Eoration
Figure 9: Expected loss function evolution of the proposed algorithm vs. DSGT, EXTRA, and 1P- GD considering vanishing vs. constant step sizes classifying images with labels 2 and 3.
Figure 10: Expected test accuracy evolution of the proposed algorithm vs. DSGT, EXTRA, and 1P-GD considering vanishing vs. constant step sizes classifying images with labels and 3.
38

---

Published in Transactions on Machine Learning Research (11/2024)
DSCT with venisting step size
--- DSGT min vanisting step size
constart step
20
1F-04D with caniching step rioes
1P-23 constant step
Fipacted Cansersus Frtor
1F-DGG with corstant siep sizes
3P-G3 with estimator (6)
500
2F-D0G with
Thain Loos Exclutoon
3P-050 with
3P-900 with naive
400
200
60
300
400 800 800 2000
Beration
Figure 11: Expected consensus error evolution of the proposed algorithm vs. DSGT and EXTRA considering vanishing vs, constant step sizes clas- sifying images with labels 2 and 3.
Figure 12: Expected loss function evolution of
the proposed algorithm vs. DSGT, EXTRA, and 1P-GD considering vanishing vs. constant step sizes classifying images with labels and 4.
3000
DSGT with venishigg step size
constart
3000
step 3P-D0G with caniriing stop sinm
Expected Corseneud Ereer
3P-30D with cenciant step siow
2500
2P-25G with noise
2000
vanisting step size
1500
DSCT with constait step size
0.7
EXTRA with constant stop aze
1F-00 with constant
estimator
1F-00G with venshtigg step tiess
300
IF DSG with consrant step sizes
3F-DGC with noiee
200
400
608
800
1000
20
60
80
100
keration
Rorabon
Figure 13: Expected Lest accuracy evolution of the proposed algorithm vs. DSGT, EXTRA, and 1P-GD considering vanishing vs. constant step sizes classifying images with labels and
Figure 14: Expected consensus error evolution of the proposed algorithm vs. DSGT and EXTRA considering vanishing vs. constant step sizes clas- sifying images with labels 3 and 4.
39


Reviews:
[{"review_id": "TsfQkFXFsr", "reviewer": "Reviewer_WN2N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The problem addressed in the paper is clearly defined and strongly motivated.\n\n* The solution method is explained in a straightforward and intuitive manner, making it relatively easy to follow.\n\n* The paper offers convergence guarantees for the one-point ZO DSG method, which aligns with those of the centralized version, presenting a seemingly novel and interesting result.\n\n**Weaknesses**\n\n* In the comparison of SOTA methods, it is shown that for SOTA centralized one-point ZO analysis, $\\mathcal{C}_{lip}$ is used. However, in the proposed distributed ZO analysis, this assumption is not needed. It is unclear whether this need for weaker assumptions comes at the cost of a different set of assumptions or if the provided analysis can improve the centralized analysis by removing existing assumptions. It would be insightful to elaborate on this disparity between the assumptions needed for the existing SOTA centralized one-point ZO analysis and those provided in this work for its decentralized counterpart.\n  \n* Given that the proposed algorithm involves significant randomness at several levels, it might be useful to add error bars to show the variance of the proposed method compared to that of other methods.\n  \n* For completeness, it seems necessary to add a centralized version of 1P-DSG with the proposed ZO gradient estimate as a baseline. This would provide a fairer and more informative comparison to demonstrate the validity of the proposed one-point gradient estimate compared to the performance of the SOTA one-point gradient estimate in (1).\n  \n* It is unclear in the main text how the convergence of the proposed method can be ensured without the normalizing constant $1/\\gamma$ used in the existing one-point ZO gradient estimate. While the details can be deferred to the proof, it would be insightful to describe the technical innovation, compared to existing analysis, that allows for establishing convergence without $1/\\gamma$.\n\n* The authors should justify (e.g. provide citations) the statement \"the good balance between exploitation and exploration\" of the constant stepsize version of a gradient method (pages 12 (end) and 13 (beginning)).\n\n**Minor Weaknesses**\n\n* In Table 1, line 2, is $\\mathcal{C}_{sm}$ a typo?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a novel zero-order (ZO) one-point gradient estimate for the distributed stochastic gradient (DSG) method, along with the corresponding 1P-DSG algorithm, which provably converges under certain assumptions. The paper provides convergence rates and regret bounds for the 1P-DSG algorithm under both constant and diminishing step-size choices. With the novel one-point ZO gradient estimation, the authors match the convergence guarantees of the centralized counterpart of ZO methods. They validate the applicability of the proposed method through numerical experiments, demonstrating that it can perform comparably to its centralized and first-order (FO) counterparts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please provide some clarification/justification for the first point and last two points in Weaknesses.\n\n* Please add the error bars and additional baseline mentioned in the second and third points of Weaknesses\n\n* For minor changes needed, refer to Minor Weaknesses.", "broader_impact_concerns": "N/A, mostly theoretical work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hhgdaaV9rT", "reviewer": "Reviewer_vy8y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe theoretical analysis of the proposed method is complete. \n\nweaknesses:\n\n1.Please provide some application background of your considered problem settings and the proposed algorithm. The motivations of this work seem to be weak.\n\n2.What is the main contribution of this work. Is it the gradient estimation? If it is, the reviewer suggests the authors highlight the design of the gradient estimation. Please explain the idea, and the sketch the proof involved, and introduce the novelty that is unique in your work. Otherwise, the present paper is just a simple combination of zero-th order optimization and consensus algorithm. The assumptions are strong, and the algorithm framework are commonly seen. The results seem not to be new.\n\n3. The experiments are performed with a toy low-dimensional classification problem. Please compare you algorithm with others in more practical scenarios. In this convergence figures, the linearly-converging algorithms seem to converge at a slower rate than the sublinearly-converging algorithms. This may happen in principle. But meanwhile this implies: the problem is too simple, or the hyperparameters are suboptimal. In either case, it is not good to verify the constant learning rate leads to linear convergence and diminishing learning rate leads to sublinear convergence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses a distributed multi-agent stochastic optimization problem. Each agent holds a local objective function that is smooth, strongly convex, and subject to a stochastic process. The goal is for agents to collaboratively optimize the sum of their local functions, despite only having access to noisy function queries at one point at a time, and without access to the gradient. The authors propose a zero-order (ZO) one-point distributed stochastic gradient (1P-DSG) estimator, extending the standard consensus-based distributed stochastic gradient method to a bandit setting where gradients are unavailable. They prove the algorithm converges to the optimum despite the gradient estimate's bias and analyze its convergence rate. With constant step sizes, the method achieves a linear rate, and with vanishing step sizes, it reaches a rate that matches centralized techniques using one-point estimators. A regret bound is also provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major revision:\n\n1. Please highlight the motivations and contributions of the work. \n\n2. Please verify the proposed approach with more practical experiments.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RfD1lPWLu6", "reviewer": "Reviewer_aMVf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths\n1) The authors have made some efforts to address concerns in the first round review, such as removing the restrictive assumption and replacing the gradient tracking technique.\n2) A regret bound of $O(\\sqrt{k})$ and a convergence rate of $O(1/\\sqrt{k})$ are achieved for smooth and strongly convex functions.\n3) A linear rate to $O(\\alpha)$-neighborhood of the optimal solution is achieved for the setting with only one-point estimators.\n\n#Weakness\n1) The $O(\\sqrt{k})$ regret bound and the $O(1/\\sqrt{k})$ convergence rate have already been established in the centralized setting. It is not surprising that such results can be extended to the decentralized setting.\n2) In the proposed algorithm, i.e., Eq. (7) and Eq. (8), although $ x_{i,k+1}$ is in the feasible set $\\mathcal{K}$, the performed point $x_{i,k+1}+\\gamma_{k+1}\\phi_{i,k+1}$ may not be in the set, which could affect the availability of the one-point feedback.\n3) The problem studied in this paper is very close to distributed online bandit optimization (DOBO). Moreover, there do exist some previous studies on DOBO [1, 2] that have established $O(k^{3/4})$ and $O(k^{2/3})$ regret bound for convex and strongly convex functions, respectively. However, the authors did not provide any discussions about these results.\n4) In addition, the authors missed an existing study on distributed stochastic optimization with single-point estimation, though it focused on a different type of function. \n\n[1] Projection-free Distributed Online Convex Optimization with $O(\\sqrt{T})$ Communication Complexity. In ICML, 2020.\n[2] Projection-free Distributed Online Learning with Sublinear Communication Complexity. In JMLR, 2022.\n[3] Single Point-Based Distributed Zeroth-Order Optimization with a Non-Convex Stochastic Objective Function. In ICML 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider a distributed stochastic optimization problem under the bandit setting, and propose a method (1P-DSD) with zero-order one-point estimators. They analyze the convergence rate of their method under constant and vanishing step sizes, and provide numerical experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) In numerical experiments, it would be better if the authors could compare the proposed method with other zero-order distributed methods such as the 2-point estimate method in Tang et al. (2021)?\n2) From Table 1, the Lipschitz condition is required for existing centralized stochastic optimization to achieve the $O(\\sqrt{k})$ regret bound with only one-point feedback. By contrast, it seems that this condition is not required by the proposed method. It would be better if the authors could provide more insights on this improvement.", "broader_impact_concerns": "NULL", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a significant improvement of the previous TMLR submission with the title "Zero-Order One-Point Estimate with Distributed Stochastic Gradient Techniques". The paper addresses all the concerns on the previous submission, such as the removal of restrictive assumptions $\nabla \mathcal{F}(w^*)=0$, the removal of the unrequired gradient track technique and the improvement of the presentation. All the reviewers are satisfied with the current version. The theoretical and empirical analysis is convincing. Therefore, I would like to recommend the acceptance of the paper for publication.
Decision: Accept (poster)