SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenzhi Gao*
Stanford University gwz@stanford.edu
Qi Deng†
Shanghai University of Finance and Economics qideng@sufe.edu.cn
This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of  $\mathcal{O}(\tau_{\mathrm{max}} / \sqrt{K})$  which depends on the maximum information delay  $\tau_{\mathrm{max}}$ . In this work, we show that the delayed stochastic subgradient method (DSGD) obtains a tighter convergence rate which depends on the expected delay  $\bar{\tau}$ . Furthermore, for an important class of composition weakly convex problems, we develop a new delayed stochastic prox-linear (DSPL) method in which the delays only affect the high-order term in the complexity rate and hence, are negligible after a certain number of DSPL iterations. In addition, we demonstrate the robustness of our proposed algorithms against arbitrary delays. By incorporating a simple safeguarding step in both methods, we achieve convergence rates that depend solely on the number of workers, eliminating the effect of the delay. Our numerical experiments further confirm the empirical superiority of our proposed methods.
In this paper, we consider the following stochastic optimization problem
$$
\min  _ {x \in \mathbb {R} ^ {n}} \psi (x) := \mathbb {E} _ {\xi \sim \Xi} [ f (x, \xi) ] + \omega (x), \tag {1}
$$
where  $f(x, \xi)$  is a nonconvex continuous function over  $x$  and  $\xi$  is a random variable sampled from some distribution  $\Xi$ ;  $\omega(x)$  is lower-semicontinuous and proximal-friendly. We assume that both  $f(x, \xi)$  and  $\omega(x)$  belong to a general class of nonsmooth nonconvex functions that exhibit weak convexity. Here, we say that a function  $g(x)$  is  $\kappa$ -weakly convex if  $g(x) + \frac{\kappa}{2} \|x\|^2$  is convex for some  $\kappa \geq 0$ . Weakly convex optimization has attracted growing interest in machine learning in recent years, and we are particularly interested in a general type of weakly convex problems with the following composition structure [12]
$$
f (x, \xi) = h (c (x, \xi)), \tag {2}
$$
where  $h$  is a convex Lipschitz function and  $c(x, \xi)$  is smooth. Optimization in the above composition form is pervasive in applications arising from machine learning and data science, including robust phase retrieval [14], blind deconvolution [8], robust PCA and matrix completion [5], among others.
Stochastic (sub)gradient method and its proximal variants [39, 10, 30, 28, 31] (all referred as SGD in our paper) are arguably the most popular approaches for solving problem (1). Typically, SGD iteratively solves  $x^{k + 1} = \arg \min_{x}\left\{\langle f^{\prime}(x^{k},\xi^{k}),x - x^{k}\rangle +\omega (x) + \frac{\gamma_{k}}{2}\| x - x^{k}\|^{2}\right\}$ , where  $\xi^k$  is
*Work done while at SHUFE.
†The corresponding author.
a random sample and  $f'(x^k, \xi^k)$  denotes a subgradient of  $f(x^k, \xi^k)$ . However, despite its wide popularity in machine learning, the sequential and synchronous nature of SGD is not suitable for modern applications that require parallel processing in multi-cores or over multiple machines.
To further improve SGD in parallel and distributed environments, recent work considers a more practical asynchronous setting where the parameter updates allow outdated gradient information. See [1, 32, 38, 29, 21]. In the asynchronous setting, it is crucial to know how the stale updates based on delayed information affects convergence. For smooth convex optimization, Agarwal and Duchi [1] show that delayed stochastic gradient method (DSGD) obtains a rate of  $\mathcal{O}(\sigma / \sqrt{K} + \tau_{\sigma^2} / (\sigma^2 K))$ , where  $\tau_{\sigma^2}$  bounds the second moment of random delays. DSGD with adaptive stepsize has also been studied by [25, 32, 38] to achieve better empirical performance and the latest work [2, 33] further improves the rate to  $\mathcal{O}(\sigma / \sqrt{K} + \tau_{\max} / K)$ . For general smooth (nonconvex) problems, the work [33] shows that DSGD converges to stationarity at a rate of  $\mathcal{O}(\sigma / \sqrt{K} + \tau_{\max} / K)$ . In a follow-up study [7], the authors propose a more robust DSGD whose rate depends on the average delay  $\tau_{\mathrm{avg}}$  rather than the maximum delay. Recently, Koloskova et al. [18] develop a sharp analysis for asynchronous SGD and then a simple delay-adaptive stepsize to achieve the best rate  $\mathcal{O}(\sigma / \sqrt{K} + \tau_{\mathrm{avg}} / K)$ . Based on novel delay-adaptive step sizes and virtual iterate-based analysis, a concurrent work [26] has established new convergence rates that depend on the number of workers rather than the delay of the gradient.
Despite much progress in distributed smooth optimization, it remains unclear how to develop efficient asynchronous algorithms for nonsmooth and even nonconvex problems. For general nonsmooth convex problems, the pioneering work [27] has shown that the asynchronous incremental subgradient method obtains an  $\mathcal{O}(\sqrt{\tau_{\mathrm{max}} / K})$  convergence rate. The aforementioned work [26] shows that DSGD achieves a delay-independent rate of  $\mathcal{O}(\sqrt{m / K})$ , where  $m$  is the number of workers. However, it is still unknown whether their key technique by telescoping the virtual iterates can be extended to composite nonconvex optimization.
For distributed weakly convex optimization, Xu et al. [34] shows that Delayed Stochastic Extrapolated Subgradient (DSEGD), an inertial version of DSGD, exhibits a convergence rate of  $\mathcal{O}((1 + \tau_{\max}) / \sqrt{K} + \tau_{\max}^2 / K)$ , which has an undesired dependence on the maximum delay  $\tau_{\max}$ . This issue is further exacerbated in real heterogeneous environments where the maximum delay is dictated by the slowest, or the "straggler" nodes. It remains unclear
1) whether such delay dependence in  $DSGD$  is still improvable or not for weakly convex problems?
Nevertheless, even if the rate is improvable in terms of  $\tau$ , there yet remains a fundamental challenge, as the delay significantly influences the leading term  $\mathcal{O}(1 / \sqrt{K})$  of the convergence rate. This contrasts with smooth distributed optimization, where the delay only affects a higher order term  $\mathcal{O}(1 / K)$  and hence is negligible in the long run. Such a performance gap highlights a substantial limitation of the prior study when nonsmoothness is present. Hence, it is natural to ask: For distributed weakly convex optimization,
2) can we design algorithms with a negligible penalty caused by delays?
3) Moreover, can we make convergence rates delay independent?
The goal of this paper is to address the above three questions.
Contributions In this paper, we answer the above questions positively (Table 1). Our contributions are as follows.
1) For distributed weakly convex optimization, we provide a sharp convergence analysis of DSGD under statistical assumptions on the delays and obtain a rate of  $\mathcal{O}(\bar{\tau} /\sqrt{K} +\tau_{\sigma^2} / K)$ , where  $\bar{\tau}$ ,  $\tau_{\sigma^2}$  are respectively first and second moments of stochastic delays. Our result significantly improves upon the previous  $\mathcal{O}(\tau_{\mathrm{max}} / \sqrt{K} +\tau_{\mathrm{max}}^2 /K)$  rate for DSGD [34].
2) For weakly convex problems with composition structure (2), we propose a new delayed stochastic prox-linear method (DSPL) which can exploit the structure (2) more effectively. Unlike SGD, the stochastic prox-linear algorithm ([9, 11]) partially linearizes the inner function  $c(\cdot, \xi)$  while retaining the outer function  $h(\cdot)$ . Then it iteratively solves:  $x^{k+1} = \arg \min_{x} \{ h(c(x^{k}, \xi) +$
$\langle \nabla c(x^k,\xi),x - x^k\rangle) + \omega (x) + \frac{\gamma_k}{2}\| x - x^k\| ^2\}$ . To the best of our knowledge, this is the first study of SPL in the asynchronous distributed setting. We show that the new DSPL method achieves a rate of  $\mathcal{O}(1 / \sqrt{K} +\tau_{\sigma^2} / K)$ , which is consistently better than that of DSGD in terms of the dependence on delay. Interestingly, our result implies that the delay is negligible when  $K$  is sufficiently large.
3) We propose a simple yet effective safeguarding step that skips the iteration when the delay is significantly large. This enhancement ensures that the rate depends only on the number of workers rather than on the delay explicitly. Specifically, in an environment of  $m$  workers, we obtain an  $\mathcal{O}(\sqrt{m} / \sqrt{K} + m / K)$  rate for DSGD and  $\mathcal{O}(1 / \sqrt{K} + m^2 / K)$  for DSPL, making both methods robust to arbitrary delays. As per our knowledge, these are the first delay-independent rates for distributed nonsmooth nonconvex optimization. Prior to our work, delay-independent rates were only known for smooth or convex optimization [26], which were derived through a distinctly different delay-adaptive stepsize strategy.
Table 1: Rates of delayed algorithms for nonconvex optimization.  $m$ : the agent number;  ${\tau }_{\max }$  : the maximum delay;  $\bar{\tau } = \mathbb{E}\left\lbrack  {\tau }_{k}\right\rbrack  ;{\tau }_{{\sigma }^{2}} = \mathbb{E}\left\lbrack  {\tau }_{k}^{2}\right\rbrack  ;{\tau }_{\text{avg }}$  average over arbitrary delays.
Delay	Work	Setting	Problems / algorithms	Convergence rate
Non-robust	[34]	weakly convex	f + ω / DSGD	O(τmax/√K + τ2/K)
Ours	f + ω / DSGD	O(τ/√K + τσ2/K)
h o c + ω / DSPL	O(1/√K + τσ2/K)
Robust	[26, 18, 7]	smooth nonconvex	f / DSGD	O(1/√K + m/K) or O(1/√K + τavg/K)
Ours	weakly convex	f + ω / DSGD	O(√m/√K + m/K)
h o c + ω / DSPL	O(1/√K + m2/K)
Structure of the paper Section 2 introduces the notations and problem setup. Section 3 analyzes the delayed stochastic (proximal) subgradient method (DSGD). Section 4 develops the delayed stochastic prox-linear method (DSPL). Section 5 proposes a simple strategy to make the asynchronous algorithms robust to arbitrary delays. Section 6 conducts experiments to verify our theoretical results. We draw conclusions in Section 7 and leave all the proofs and further discussions in the appendix.
Notations We use  $\| \cdot \|$  and  $\langle \cdot ,\cdot \rangle$  to denote the Euclidean norm and inner product. The subdifferential of a function  $f$  is defined by  $\partial f(x)\coloneqq \{v:f(y)\geq f(x) + \langle v,y - x\rangle +o(\| x - y\|),y\to x\}$  and  $f^{\prime}(x)\in \partial f(x)$  denotes a subgradient. If  $0\in \partial f(x)$ , then  $x$  is called a stationary point of  $f$ . The domain of  $f$  is defined by  $\operatorname {dom}f\coloneqq \{x:f(x) < \infty \}$ . At iteration  $k$ , we use  $\mathbb{E}_k[\cdot ]$  to denote the expectation conditioned on past iterations  $\{x^{1},\ldots ,x^{k - 1}\}$ .  $\mathbb{I}\{\cdot \}$  denotes the 0-1 indicator function of an event. Given a delay sequence  $\{\tau_k\}$ , we write  $\Delta_1\coloneqq \frac{1}{K}\sum_{k = 1}^{K}\tau_k$  and  $\Delta_2\coloneqq \frac{1}{K}\sum_{k = 1}^{K}\tau_k^2$ .
Our analysis will adopt the Moreau envelope as the potential function, a technique initially identified in the work [8]. Let  $f$  be a  $\kappa$ -weakly convex function. Given  $\rho > \kappa$ , the Moreau envelope and the associated proximal mapping of  $f$  are respectively defined by
$$
f _ {1 / \rho} (x) := \min  _ {y} \left\{f (y) + \frac {\rho}{2} \| x - y \| ^ {2} \right\} \quad \text {a n d} \quad \operatorname {p r o x} _ {f / \rho} (x) := \underset {y} {\arg   \min } \left\{f (y) + \frac {\rho}{2} \| x - y \| ^ {2} \right\}.
$$
By the optimality condition and convexity of  $f(y) + \frac{\rho}{2} \| x - y\| ^2$ ,  $0\in \partial f(\mathrm{prox}_{f / \rho}(x)) + \rho (\mathrm{prox}_{f / \rho}(x) - x)$ . The Moreau envelope can be nicely interpreted as a smooth approximation of the original function. Specifically, it can be shown that  $f_{1 / \rho}(x)$  is differentiable and its gradient is  $\nabla f_{1 / \rho}(x) = \rho (x - \mathrm{prox}_{f / \rho}(x))$  (See [8]). Combining the above two relations, we obtain  $\nabla f_{1 / \rho}(x)\in \partial f(\mathrm{prox}_{f / \rho}(x))$ . Therefore, the Moreau envelope can be used as a measure of approximate stationarity: if  $\| \nabla f_{1 / \rho}(x)\| \leq \varepsilon$ , then  $x$  is in the proximity (i.e.  $\| x - \mathrm{prox}_{f / \rho}(x)\| \leq \rho^{-1}\varepsilon$ ) of a near stationary point  $\mathrm{prox}_{f / \rho}(x)$  (i.e.  $\mathrm{dist}(\partial f(\mathrm{prox}_{f / \rho}(x)),0)\leq \varepsilon)$ ).
2.1 Assumptions
Throughout the paper, we make the following assumptions.
A1: (i.i.d. sample) We draw i.i.d. samples  $\{\xi^k\}$  from  $\Xi$ .
A2: (Lipschitz continuity)  $\omega(x)$  is  $L_{\omega}$ -Lipschitz continuous over its domain.
A3: (Weak convexity)  $\omega$  is  $\kappa$ -weakly convex.
A4: (Bounded moments) The distribution of the independent stochastic delays  $\{\tau_k\}$  has bounded first and second moments. i.e.,  $\mathbb{E}[\tau_k] \leq \bar{\tau} < \infty, \mathbb{E}[\tau_k^2] \leq \tau_{\sigma^2} < \infty, \forall k$ .
Remark 1. Assumptions A1 to A3 are typical in stochastic weakly convex optimization [8], while A4 is common in distributed optimization [3, 32]. Moreover, throughout our analysis, we regard  $\{\tau_k\}$  as a pre-defined random sequence independent of data sampling and the algorithm [32, 34].
3 Delayed proximal subgradient method
In this section, we analyze the convergence rate of the delayed stochastic proximal subgradient method (DSGD) for weakly convex optimization. DSGD is the workhorse of most applications and is frequently analyzed in the literature on centralized distributed optimization.
Algorithm 1: Delayed stochastic proximal subgradient method
Input:  $x^{1}$
for  $k = 1,2,\ldots$  do
Let  $g^{k - \tau_k}\in \partial f(x^{k - \tau_k},\xi^{k - \tau_k})$  be computed by a worker with delay  $\tau_{k}$
In the master node update
$$
x ^ {k + 1} = \underset {x \in \mathbb {R} ^ {n}} {\arg \min } \left\{\langle g ^ {k - \tau_ {k}}, x - x ^ {k} \rangle + \omega (x) + \frac {\gamma_ {k}}{2} \| x - x ^ {k} \| ^ {2} \right\}. \tag {3}
$$
We restrict  $f$  to have a bounded subgradient and make the following extra assumption.
B1:  $f(x, \xi)$  is  $\lambda$ -weakly convex,  $\mathbb{E}_{\xi}[\| g \|^{2}] \leq L_{f}^{2}, g \in \partial f(x, \xi)$  for all  $x \in \operatorname{dom} \omega \subseteq \operatorname{int}(\operatorname{dom} f)$  and all  $\xi \sim \Xi$ .
Now we are ready to present the convergence analysis of DSGD. Our analysis relies on Moreau envelope, denoted as  $\psi_{1 / \rho}(x^k)$ , which serves as the potential function. After assessing the descent of the potential function with delays separated as an error term, we derive the convergence result by bounding the stochastic delays. The following lemma provides the descent property.
Lemma 1. Let  $\hat{x}^k \coloneqq \mathrm{prox}_{\psi / \rho}(x^k)$ . Suppose A1, A2, A3 and B1 hold. If  $\rho > 2\lambda + \kappa, \gamma_k \geq \rho$ , then
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} \leq \psi_ {1 / \rho} (x ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} (x ^ {k + 1}) ] - \frac {\rho (\gamma_ {k} - \rho)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {\rho \lambda}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] \\ \end{array}
$$
Remark 2. Lemma 1 shows that aside from the noise and delay-related terms, unless we are close to approximate stationarity characterized by  $\| \hat{x}^k - x^k \|^2$ , there is always sufficient decrease in the potential function. Intuitively, if we take sufficiently large regularization  $\gamma$  and bound the delays using A4, convergence is almost immediate. Now we show convergence of DSGD in Theorem 1.
Theorem 1. Under the same conditions as Lemma 1 as well as A4, taking  $\gamma_{k} \equiv \gamma = \rho + 4\lambda + \kappa + \sqrt{K} / \alpha$  for some  $\alpha > 0$ , letting  $k^{*}$  be chosen between 1 and  $K$  uniformly at random, then
$$
\mathbb {E} \big [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} \big ] \leq \frac {2 \rho}{\rho - 2 \lambda - \kappa} \Big [ \frac {(\rho + 2 \lambda) D}{K} + \frac {D}{\sqrt {K} \alpha} + \frac {4 \rho L _ {f} (L _ {f} + L _ {\omega}) \alpha}{\sqrt {K}} (\Delta_ {1} + 1) + \frac {8 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K} \Delta_ {2} \Big ],
$$
where  $D = \psi_{1 / \rho}(x^1) - \inf_x\psi (x)$  and recall our notation  $\Delta_1 = \frac{1}{K}\sum_{k = 1}^{K}\tau_k,\Delta_2 = \frac{1}{K}\sum_{k = 1}^{K}\tau_k^2.$
Remark 3. Note that  $\alpha$  controls the trade-off between noise, delay and optimization. In practice we can set  $\alpha$  as a hyper-parameter and tune it to improve performance.
The bound for DSGD states that
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} ] = \mathcal {O} \big (\frac {1}{\sqrt {K}} + \frac {\Delta_ {1}}{\sqrt {K}} + \frac {\lambda \Delta_ {2}}{K} \big) \mathrm {o r} \mathbb {E} [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} ] = \mathcal {O} \big (\frac {1}{\sqrt {K}} + \frac {\bar {\tau}}{\sqrt {K}} + \frac {\lambda \tau_ {\sigma^ {2}}}{K} \big),
$$
if we take  $\mathbb{E}[\Delta_1] = \bar{\tau}$  and  $\mathbb{E}[\Delta_2] = \tau_{\sigma^2}$ . Here  $\Delta_1$  is the average delay and is considered "robust" in the distributed setting. However, there exists a second term  $\lambda \Delta_2$  arising from  $\lambda$ -weak convexity. As we will show later, this term, in the worst case, can not be trivially bounded. Therefore, we now resort to our statistical assumption A4 and use the first and second moments to give a bound.
4 Delayed stochastic prox-linear method
In this section, we present a delayed stochastic prox-linear method (DSPL) for the composition optimization problem with  $f(x,\xi) = h(c(x,\xi))$ , where  $h(\cdot):\mathbb{R}\to \mathbb{R}$  is a convex nonsmooth function and  $c(\cdot ,\xi):\mathbb{R}^n\to \mathbb{R}$  is a smooth, potentially nonconvex function. For brevity, we denote  $f_{z}(x,\xi) = h(c(z,\xi) + \langle \nabla c(z,\xi),x - z\rangle)$ , a partial linearization of the objective, and we use these two notations interchangeably to describe DSPL. We will show that this linearization scheme improves the accuracy of approximating the objective, thus giving improved convergence rates in the presence of delay. It should be noted that while we assume  $c(\cdot)$  to be a smooth function, our analysis can seamlessly extend to the case where  $c$  is a smooth mapping with a bounded Jacobi matrix.
Algorithm 2: Delayed stochastic prox-linear method
Let  $c(x^{k - \tau_k},\xi^{k - \tau_k})$  and  $\nabla c(x^{k - \tau_k},\xi^{k - \tau_k})$  be computed by a worker with delay  $\tau_{k}$
$$
x ^ {k + 1} = \underset {x \in \mathbb {R} ^ {n}} {\arg \min } \left\{f _ {x ^ {k - \tau_ {k}}} \left(x, \xi^ {k - \tau_ {k}}\right) + \omega (x) + \frac {\gamma_ {k}}{2} \| x - x ^ {k} \| ^ {2} \right\} \tag {4}
$$
We describe DSPL in Algorithm 2. DSPL can be deployed in a multi-agent network where all the workers are connected to a master server. We assume the workers access random samples  $\xi$  and compute the function value  $c(x, \xi)$  and gradient  $\nabla c(x, \xi)$ . At the  $k$ -th iteration, the master node receives a delayed value  $c(x^{k - \tau_k}, \xi^{k - \tau_k})$  and gradient  $\nabla c(x^{k - \tau_k}, \xi^{k - \tau_k})$  of an earlier point  $x^{k - \tau_k}$ . Next, it performs a proximal update (4) to obtain the next iterate  $x^{k + 1}$ . Figure 1 illustrates the operational dynamics of DSPL and contrast it with DSGD. Throughout the rest of this paper, we assume that the proximal update is easy to compute. To derive the convergence results of DSPL, we now formally state the assumptions on  $h$  and  $c$ .
C1:  $h(x)$  is convex and  $L_{h}$ -Lipschitz continuous over its domain;  $c(x, \xi)$  has  $C(\xi)$ -Lipschitz continuous gradient and is  $L_{c}(\xi)$ -Lipschitz continuous over  $\operatorname{dom} \omega$ . Moreover, we assume that  $\mathbb{E}_{\xi}[C(\xi)^2] \leq C^2$ ,  $\mathbb{E}_{\xi}[L_{c}(\xi)^{2}] \leq L_{c}^{2}$ .
Remark 4. The assumption of Lipschitz continuity for  $c$  is quite prevalent in the literature of weakly convex optimization [8, 12, 13]. However, it may not hold if  $\operatorname{dom} \omega$  is unbounded and  $c$  is nonlinear, such as in the case of quadratic functions. To circumvent this problem, we introduce the relative Lipschitzian property and extend the method to the Bregman proximal iteration. This involves replacing the Euclidean regularization term  $\frac{1}{2} \| x - x_k \|^2$  in (4) with the divergence  $V_d(x, x^k)$ , which is generated by a strongly convex function  $d(\cdot)$ . The Bregman DSPL encompasses Algorithm 2 as a special case; the details are provided in the appendix for a thorough examination.
Under the above assumptions,  $f_{z}(x,\xi)$  satisfies the following desirable properties.
Proposition 1 (Properties of  $f_{z}(x,\xi)$ ).
$\pmb{P1}$  ..  $f_{z}(x,\xi)$  is convex for all  $x,z\in \mathrm{dom}\omega$  , and every  $\xi \sim \Xi$
P2:  $|f_z(x,\xi) - f(x,\xi)| \leq \frac{L_h C(\xi)}{2} \| x - z \|^2$  for all  $x, z \in \mathrm{dom} \omega$  and  $\xi \sim \Xi$ .
P3:  $f_{z}(x,\xi) - f_{z}(y,\xi)\leq L_{h}L_{c}(\xi)\| y - x\|$  for all  $x,y,z\in \mathrm{dom}\omega$  and all  $\xi \sim \Xi$
From Proposition 1, we see  $f(x,\xi)$  is  $L_{h}C(\xi)$ -weakly convex since the error between  $f(x,\xi)$  and a convex function is bounded by a quadratic function. Furthermore, we know that  $f(x)$  is  $L_{h}L_{c}$  Lipschitz-continuous and  $L_{h}C$ -weakly convex after taking expectation. For a unified analysis, we take  $L_{f} = L_{h}L_{c}, \lambda = L_{h}C$  and use these constants to present the results. The following lemma characterizes the descent property for our potential function in DSPL.
Lemma 2. Suppose A1, A2, A3 and C1 hold, if  $\rho > 2\lambda + \kappa$ ,  $\gamma_k \geq \rho$ , then
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} \leq \psi_ {1 / \rho} (x ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} (x ^ {k + 1}) ] - \frac {\rho (\gamma_ {k} - \rho)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} + \frac {3 \rho \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ]. \\ \end{array}
$$
Similar to the analysis of DSGD, we bound the delays to give the convergence result.
Theorem 2. Under the same conditions as Lemma 2 as well as A4, taking  $\gamma_{k} \equiv \gamma = \rho + 6\lambda + \kappa + \sqrt{K} / \alpha$  for some  $\alpha > 0$ , letting  $k^{*}$  be uniformly chosen between 1 and  $K$ , then
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} ] \leq \frac {2 \rho}{\rho - 2 \lambda - \kappa} \Big [ \frac {(\rho + 4 \lambda) D}{K} + \frac {D}{\sqrt {K} \alpha} + \frac {2 \rho L _ {f} ^ {2} \alpha}{\sqrt {K}} + \frac {6 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K} \Delta_ {2} \Big ],
$$
where  $D = \psi_{1 / \rho}(x^1) - \inf_x\psi (x)$
If we take  $\mathbb{E}[\Delta_2] = \tau_{\sigma^2}$ , then Theorem 2 implies that
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} ] = \mathcal {O} \left(\frac {1}{\sqrt {K}} + \frac {\lambda \Delta_ {2}}{K}\right) \text {o r} \mathbb {E} [ \| \nabla \psi_ {1 / \rho} (x ^ {k ^ {*}}) \| ^ {2} ] = \mathcal {O} \left(\frac {1}{\sqrt {K}} + \frac {\lambda \tau_ {\sigma^ {2}}}{K}\right).
$$
Compared to DSGD, the delays for DSPL appear only in a higher-order term with respect to  $K$ . Different from DSGD where  $\lambda$  only characterizes weak convexity,  $\lambda$  for DSPL also represents the quadratic upper-bound on  $f(x,\xi)$  in P2, which is not 0 even if the problem is convex.
Figure 1: DSGD and DSPL in a master-worker architecture.