Title: Delayed Algorithms for Distributed Stochastic Weakly Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stochastic optimization

Distributed optimization

Prox-linear method

Stochastic gradient method
Abstract: This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node.  Recently, Xu~et~al.~2022  showed that an inertial stochastic subgradient method   converges at a rate of $\mathcal{O}(\tau_{\text{max}}/\sqrt{K})$ which depends on the maximum information delay $\tau_{\text{max}}$. 
In this work, we show that the delayed stochastic subgradient method ($\texttt{DSGD}$) obtains a tighter convergence rate which depends on the expected delay $\bar{\tau}$. Furthermore, for an important class of composition weakly convex problems, we develop a new delayed stochastic prox-linear ($\texttt{DSPL}$) method in which the delays only affect the high-order term in the rate and hence,  are negligible after a certain number of $\texttt{DSPL}$  iterations.  In addition, we demonstrate the robustness of our proposed algorithms against arbitrary delays.  By incorporating a simple safeguarding step in both methods, we achieve convergence rates that depend solely on the number of workers, eliminating the effect of delays. Our numerical experiments further confirm the empirical superiority of our proposed methods.

Delayed Algorithms for Distributed Stochastic Weakly Convex Optimization

Abstract

This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of O ( τ max / √ K ) which depends on the maximum information delay τ max. In this work, we show that the delayed stochastic subgradient method ( DSGD ) obtains a tighter convergence rate which depends on the expected delay ¯ τ . Furthermore, for an important class of composition weakly convex problems, we develop a new delayed stochastic prox-linear ( DSPL ) method in which the delays only affect the high-order term in the complexity rate and hence, are negligible after a certain number of DSPL iterations. In addition, we demonstrate the robustness of our proposed algorithms against arbitrary delays. By incorporating a simple safeguarding step in both methods, we achieve convergence rates that depend solely on the number of workers, eliminating the effect of the delay. Our numerical experiments further confirm the empirical superiority of our proposed methods.

1 Introduction

In this paper, we consider the following stochastic optimization problem

where f ( x, ξ ) is a nonconvex continuous function over x and ξ is a random variable sampled from some distribution Ξ ; ω ( x ) is lower-semicontinuous and proximal-friendly. We assume that both f ( x, ξ ) and ω ( x ) belong to a general class of nonsmooth nonconvex functions that exhibit weak convexity. Here, we say that a function g ( x ) is κ -weakly convex if g ( x ) + κ 2 ∥ x ∥ 2 is convex for some κ ≥ 0 . Weakly convex optimization has attracted growing interest in machine learning in recent years, and we are particularly interested in a general type of weakly convex problems with the following composition structure [12]

where h is a convex Lipschitz function and c ( x, ξ ) is smooth. Optimization in the above composition form is pervasive in applications arising from machine learning and data science, including robust phase retrieval [14], blind deconvolution [8], robust PCA and matrix completion [5], among others.

Stochastic (sub)gradient method and its proximal variants [39, 10, 30, 28, 31] (all referred as SGD in our paper) are arguably the most popular approaches for solving problem (1). Typically, SGD iteratively solves x k +1 = arg min x {⟨ f ′ ( x k , ξ k ) , x -x k ⟩ + ω ( x ) + γ k 2 ∥ x -x k ∥ 2 } , where ξ k is

a random sample and f ′ ( x k , ξ k ) denotes a subgradient of f ( x k , ξ k ) . However, despite its wide popularity in machine learning, the sequential and synchronous nature of SGD is not suitable for modern applications that require parallel processing in multi-cores or over multiple machines.

To further improve SGD in parallel and distributed environments, recent work considers a more practical asynchronous setting where the parameter updates allow outdated gradient information. See [1, 32, 38, 29, 21]. In the asynchronous setting, it is crucial to know how the stale updates based on delayed information affects convergence. For smooth convex optimization, Agarwal and Duchi [1] show that delayed stochastic gradient method ( DSGD ) obtains a rate of O ( σ/ √ K + τ σ 2 / ( σ 2 K )) , where τ σ 2 bounds of the second moment of random delays. DSGD with adaptive stepsize has also been studied by [25, 32, 38] to achieve better empirical performance and the latest work [2, 33] further improves the rate to O ( σ/ √ K + τ max /K ) . For general smooth (nonconvex) problems, the work [33] shows that DSGD converges to stationarity at a rate of O ( σ/ √ K + τ max /K ) . In a followup study [7], the authors propose a more robust DSGD whose rate depends on the average delay τ avg , rather than the maximum delay. Recently, Koloskova et al. [18] develop a sharp analysis for asynchronous SGD and then a simple delay-adaptive stepsize to achieve the best rate O ( σ/ √ K + τ avg /K ) . Based on novel delay-adaptive stepsizes and virtual iterate-based analysis, a concurrent work [26] has established new convergence rates that depend on the number of workers rather than the delay of the gradient.

Despite much progress in distributed smooth optimization, it remains unclear how to develop efficient asynchronous algorithms for nonsmooth and even nonconvex problems. For general nonsmooth convex problems, the pioneering work [27] has shown that the asynchronous incremental subgradient method obtains an O ( √ τ max /K ) convergence rate. The aforementioned work [26] shows that DSGD achieves a delay-independent rate of O ( √ m/K ) , where m is the number of workers. However, it is still unknown whether their key technique by telescoping the virtual iterates can be extended to composite nonconvex optimization.

For distributed weakly convex optimization, Xu et al. [34] shows that Delayed Stochastic Extrapolated Subgradient ( DSEGD ), an inertial version of DSGD , exhibits a convergence rate of O ((1 + τ max ) / √ K + τ 2 max /K ) , which has an undesired dependence on the maximum delay τ max. This issue is further exacerbated in real heterogeneous environments where the maximum delay is dictated by the slowest, or the 'straggler' nodes. It remains unclear

1) whether such delay dependence in DSGD is still improvable or not for weakly convex problems?

Nevertheless, even if the rate is improvable in terms of τ , there yet remains a fundamental challenge, as the delay significantly influences the leading term O (1 / √ K ) of the convergence rate. This contrasts with smooth distributed optimization, where the delay only affects a higher order term O (1 /K ) and hence is negligible in the long run. Such a performance gap highlights a substantial limitation of the prior study when nonsmoothness is present. Hence, it is natural to ask: For distributed weakly convex optimization,

2) can we design algorithms with a negligible penalty caused by delays?

Moreover, can we make convergence rates delay independent?

The goal of this paper is to address the above three questions.

Contributions In this paper, we answer the above questions positively (Table 1). Our contributions are as follows.

For distributed weakly convex optimization, we provide a sharp convergence analysis of DSGD under statistical assumptions on the delays and obtain a rate of O (¯ τ/ √ K + τ σ 2 /K ) , where ¯ τ, τ σ 2 are respectively first and second moments of stochastic delays. Our result significantly improves upon the previous O ( τ max / √ K + τ 2 max /K ) rate for DSGD [34].

For weakly convex problems with composition structure (2), we propose a new delayed stochastic prox-linear method ( DSPL ) which can exploit the structure (2) more effectively. Unlike SGD, the stochastic prox-linear algorithm ([9, 11]) partially linearizes the inner function c ( · , ξ ) while retaining the outer function h ( · ) . Then it iteratively solves: x k +1 = argmin x { h ( c ( x k , ξ ) +

⟨∇ c ( x k , ξ ) , x -x k ⟩ ) + ω ( x )+ γ k 2 ∥ x -x k ∥ 2 } . To the best of our knowledge, this is the first study of SPL in the asynchronous distributed setting. We show that the new DSPL method achieves a rate of O (1 / √ K + τ σ 2 /K ) , which is consistently better than that of DSGD in terms of the dependence on delay. Interestingly, our result implies that the delay is negligible when K is sufficiently large.

We propose a simple yet effective safeguarding step that skips the iteration when the delay is significantly large. This enhancement ensures that the rate depends only on the number of workers rather than on the delay explicitly. Specifically, in an environment of m workers, we obtain an O ( √ m/ √ K + m/K ) rate for DSGD and O (1 / √ K + m 2 /K ) for DSPL , making both methods robust to arbitrary delays. As per our knowledge, these are the first delay-independent rates for distributed nonsmooth nonconvex optimization. Prior to our work, delay-independent rates were only known for smooth or convex optimization [26], which were derived through a distinctly different delay-adaptive stepsize strategy.

Structure of the paper Section 2 introduces the notations and problem setup. Section 3 analyzes the delayed stochastic (proximal) subgradient method ( DSGD ). Section 4 develops the delayed stochastic prox-linear method ( DSPL ). Section 5 proposes a simple strategy to make the asynchronous algorithms robust to arbitrary delays. Section 6 conducts experiments to verify our theoretical results. We draw conclusions in Section 7 and leave all the proofs and further discussions in the appendix.

2 Preliminaries

Notations We use ∥ · ∥ and ⟨· , ·⟩ to denote the Euclidean norm and inner product. The subdifferential of a function f is defined by ∂f ( x ) := { v : f ( y ) ≥ f ( x ) + ⟨ v, y -x ⟩ + o ( ∥ x -y ∥ ) , y → x } and f ′ ( x ) ∈ ∂f ( x ) denotes a subgradient. If 0 ∈ ∂f ( x ) , then x is called a stationary point of f . The domain of f is defined by dom f := { x : f ( x ) < ∞} . At iteration k , we use E k [ · ] to denote the expectation conditioned on past iterations { x 1 , . . . , x k -1 } . I {·} denotes the 0-1 indicator function of an event. Given a delay sequence { τ k } , we write ∆ 1 := 1 K ∑ K k =1 τ k and ∆ 2 := 1 K ∑ K k =1 τ 2 k .

Our analysis will adopt the Moreau envelope as the potential function, a technique initially identified in the work [8]. Let f be a κ -weakly convex function. Given ρ > κ , the Moreau envelope and the associated proximal mapping of f are respectively defined by

By the optimality condition and convexity of f ( y ) + ρ 2 ∥ x -y ∥ 2 , 0 ∈ ∂f ( prox f/ρ ( x )) + ρ ( prox f/ρ ( x ) -x ) . The Moreau envelope can be nicely interpreted as a smooth approximation of the original function. Specifically, it can be shown that f 1 /ρ ( x ) is differentiable and its gradient is ∇ f 1 /ρ ( x ) = ρ ( x -prox f/ρ ( x )) (See [8]). Combining the above two relations, we obtain ∇ f 1 /ρ ( x ) ∈ ∂f ( prox f/ρ ( x )) . Therefore, the Moreau envelope can be used as a measure of approximate stationarity: if ∥∇ f 1 /ρ ( x ) ∥ ≤ ε , then x is in the proximity (i.e. ∥ x -prox f/ρ ( x ) ∥ ≤ ρ -1 ε ) of a near stationary point prox f/ρ ( x ) (i.e. dist( ∂f ( prox f/ρ ( x )) , 0) ≤ ε ).

2.1 Assumptions

Throughout the paper, we make the following assumptions.

A1: (i.i.d. sample) We draw i.i.d. samples { ξ k } from Ξ .

A2: (Lipschitz continuity) ω ( x ) is L ω -Lipschitz continuous over its domain.

A3: (Weak convexity) ω is κ -weakly convex.

A4: (Bounded moments) The distribution of the independent stochastic delays { τ k } has bounded first and second moments. i.e., E [ τ k ] ≤ ¯ τ < ∞ , E [ τ 2 k ] ≤ τ σ 2 < ∞ , ∀ k .

Remark 1 . Assumptions A1 to A3 are typical in stochastic weakly convex optimization [8], while A4 is common in distributed optimization [3, 32]. Moreover, throughout our analysis, we regard { τ k } as a pre-defined random sequence independent of data sampling and the algorithm [32, 34].

3 Delayed proximal subgradient method

In this section, we analyze the convergence rate of the delayed stochastic proximal subgradient method ( DSGD ) for weakly convex optimization. DSGD is the workhorse of most applications and is frequently analyzed in the literature on centralized distributed optimization.

Algorithm 1: Delayed stochastic proximal subgradient method

Input:

for k =

x

1

;

1, 2,...

do

Let g k -τ k ∈ ∂f ( x k -τ k , ξ k -τ k ) be computed by a worker with delay τ k ;

In the master node update

end

We restrict f to have a bounded subgradient and make the following extra assumption.

B1: f ( x, ξ ) is λ -weakly convex, E ξ [ ∥ g ∥ 2 ] ≤ L 2 f , g ∈ ∂f ( x, ξ ) for all x ∈ dom ω ⊆ int (dom f ) and all ξ ∼ Ξ .

Now we are ready to present the convergence analysis of DSGD . Our analysis relies on Moreau envelope, denoted as ψ 1 /ρ ( x k ) , which serves as the potential function. After assessing the descent of the potential function with delays separated as an error term, we derive the convergence result by bounding the stochastic delays. The following lemma provides the descent property.

Lemma 1. Let ˆ x k := prox ψ/ρ ( x k ) . Suppose A1 , A2 , A3 and B1 hold. If ρ > 2 λ + κ, γ k ≥ ρ , then

Remark 2 . Lemma 1 shows that aside from the noise and delay-related terms, unless we are close to approximate stationarity characterized by ∥ ˆ x k -x k ∥ 2 , there is always sufficient decrease in the potential function. Intuitively, if we take sufficiently large regularization γ and bound the delays using A4 , convergence is almost immediate. Now we show convergence of DSGD in Theorem 1 .

Theorem 1. Under the same conditions as Lemma 1 as well as A4 , taking γ k ≡ γ = ρ +4 λ + κ + √ K/α for some α > 0 , letting k ∗ be chosen between 1 and K uniformly at random, then

Remark 3 . Note that α controls the trade-off between noise, delay and optimization. In practice we can set α as a hyper-parameter and tune it to improve performance.

The bound for DSGD states that

if we take E [∆ 1 ] = ¯ τ and E [∆ 2 ] = τ σ 2 . Here ∆ 1 is the average delay and is considered 'robust' in the distributed setting. However, there exists a second term λ ∆ 2 arising from λ -weak convexity. As we will show later, this term, in the worst case, can not be trivially bounded. Therefore, we now resort to our statistical assumption A4 and use the first and second moments to give a bound.

4 Delayed stochastic prox-linear method

In this section, we present a delayed stochastic prox-linear method ( DSPL ) for the composition optimization problem with f ( x, ξ ) = h ( c ( x, ξ )) , where h ( · ) : R → R is a convex nonsmooth function and c ( · , ξ ) : R n → R is a smooth, potentially nonconvex function. For brevity, we denote f z ( x, ξ ) = h ( c ( z, ξ ) + ⟨∇ c ( z, ξ ) , x -z ⟩ ) , a partial linearization of the objective, and we use these two notations interchangeably to describe DSPL . We will show that this linearization scheme improves the accuracy of approximating the objective, thus giving improved convergence rates in the presence of delay. It should be noted that while we assume c ( · ) to be a smooth function, our analysis can seamlessly extend to the case where c is a smooth mapping with a bounded Jacobi matrix.

Algorithm 2: Delayed stochastic prox-linear method

We describe DSPL in Algorithm 2 . DSPL can be deployed in a multi-agent network where all the workers are connected to a master server. We assume the workers access random samples ξ and compute the function value c ( x, ξ ) and gradient ∇ c ( x, ξ ) . At the k -th iteration, the master node receives a delayed value c ( x k -τ k , ξ k -τ k ) and gradient ∇ c ( x k -τ k , ξ k -τ k ) of an earlier point x k -τ k . Next, it performs a proximal update (4) to obtain the next iterate x k +1 . Figure 1 illustrates the operational dynamics of DSPL and contrast it with DSGD . Throughout the rest of this paper, we assume that the proximal update is easy to compute. To derive the convergence results of DSPL , we now formally state the assumptions on h and c .

C1: h ( x ) is convex and L h -Lipschitz continuous over its domain; c ( x, ξ ) has C ( ξ ) -Lipschitz continuous gradient and is L c ( ξ ) -Lipschitz continuous over dom ω . Moreover, we assume that E ξ [ C ( ξ ) 2 ] ≤ C 2 , E ξ [ L c ( ξ ) 2 ] ≤ L 2 c .

Remark 4 . The assumption of Lipschitz continuity for c is quite prevalent in the literature of weakly convex optimization [8, 12, 13]. However, it may not hold if dom ω is unbounded and c is nonlinear, such as in the case of quadratic functions. To circumvent this problem, we introduce the relative Lipschitzian property and extend the method to the Bregman proximal iteration. This involves replacing the Euclidean regularization term 1 2 ∥ x -x k ∥ 2 in (4) with the divergence V d ( x, x k ) , which is generated by a strongly convex function d ( · ) . The Bregman DSPL encompasses Algorithm 2 as a special case; the details are provided in the appendix for a thorough examination.

Under the above assumptions, f z ( x, ξ ) satisfies the following desirable properties.

Proposition 1 (Properties of f z ( x, ξ ) ) .

P1: f z ( x, ξ ) is convex for all x, z ∈ dom ω , and every ξ ∼ Ξ .

P2: | f z ( x, ξ ) -f ( x, ξ ) | ≤ L h C ( ξ ) 2 ∥ x -z ∥ 2 for all x, z ∈ dom ω and ξ ∼ Ξ .

P3: f z ( x, ξ ) -f z ( y, ξ ) ≤ L h L c ( ξ ) ∥ y -x ∥ for all x, y, z ∈ dom ω and all ξ ∼ Ξ

From Proposition 1 , we see f ( x, ξ ) is L h C ( ξ ) -weakly convex since the error between f ( x, ξ ) and a convex function is bounded by a quadratic function. Furthermore, we know that f ( x ) is L h L c Lipschitz-continuous and L h C -weakly convex after taking expectation. For a unified analysis, we take L f = L h L c , λ = L h C and use these constants to present the results. The following lemma characterizes the descent property for our potential function in DSPL .

Lemma 2. Suppose A1 , A2 , A3 and C1 hold, if ρ > 2 λ + κ, γ k ≥ ρ , then

Similar to the analysis of DSGD , we bound the delays to give the convergence result.

Theorem 2. Under the same conditions as Lemma 2 as well as A4 , taking γ k ≡ γ = ρ +6 λ + κ + √ K/α for some α > 0 , letting k ∗ be uniformly chosen between 1 and K , then

where D = ψ 1 /ρ ( x 1 ) -inf x ψ ( x ) .

If we take E [∆ 2 ] = τ σ 2 , then Theorem 2 implies that

Compared to DSGD , the delays for DSPL appear only in a higher-order term with respect to K . Different from DSGD where λ only characterizes weak convexity, λ for DSPL also represents the quadratic upper-bound on f ( x, ξ ) in P2 , which is not 0 even if the problem is convex.

Master

x

,

y

Worker

x

,

f

(

x

,

ξ

)

Worker

x

,

f

(

x

,

ξ

)

Worker

x

,

f

(

x

,

ξ

)

f

′

x

f

′

x

f

′

x

Master

,

x

,

y h

Worker

x

,

c

(

x

,

ξ

)

Worker

x

,

c

(

x

,

ξ

)

Worker

x

,

c

(

x

,

ξ

)

c

, ∇

c

x

c

, ∇

c

x

c

, ∇

c

x

f

(

x

,

ξ

) =

h

(

c

(

x

,

ξ

))

DSGD

DSPL

DSPL vs. DSGD We give some further insights on the behavior of DSPL and DSGD for the composition function (2). Intuitively, as the algorithm converges, we have lim k → 0 ∥ x k -x k +1 ∥ = 0 a.s. When x k ≈ x k -τ k , DSPL enjoys an increasingly stable estimation of the proximal mapping (4), as the influence of delay is diminishing and the error is mainly driven by stochastic sampling. The same conclusion holds on smooth DSGD due to the Lipschitz continuity of ∇ f . On the other hand, when DSGD is applied for a nonsmooth problem, the master node receives an out-of-date subgradient f ′ from the worker and solves (3). Since f ( x, ξ ) is nonsmooth, the subgradient f ′ ( x k -τ k , ξ ) may differ significantly from f ′ ( x k , ξ ) even when the sequence { x k } converges. Hence, DSGD will constantly suffer from delay during all the updates (3).

DSPLwith momentum Wealso remark that the momentum technique from DSGD can be extended to DSPL , which gives us the same O ( τ max / √ K + τ 2 max /K ) convergence rate as in [34]. We refer the interested readers to the Appendix F.

When f is smooth, the analysis of DSPL can be adapted to yield a comparable convergence rate for the proximal stochastic gradient method for minimizing ψ ( · ) .

Theorem 3. Suppose all the assumptions in Lemma 1 and A4 hold, and that f is λ -smooth. Let γ k ≡ γ = ρ +6 λ + κ + √ K/α for some α > 0 , then E [ ∥∇ ψ 1 /ρ ( x k ∗ ) ∥ 2 ] = O ( 1 √ K + λ ∆ 2 K ) .

So far we have spent two sections analyzing the two algorithms so that enough intuition can be established. All these serve for our ultimate goal: making DSGD and DSPL robust to arbitrary delays.

5 Weakly convex optimization robust to arbitrary delays

This section proposes robust variants of DSGD and DSPL , for which the explicit delays are eliminated from the convergence rate. What we will do is reduce the impact of delay to the number of agents in the network . Moving forward, we substitute A4 with D1 .

D1: The distributed environment has m workers.

The previously established results have provided sufficient intuition on how delays impact our algorithms. In view of Theorem 1 and Theorem 2 , we have isolated delay-dependent O (∆ 1 / √ K ) and O (∆ 2 /K ) in the proof. Although ∆ 1 / √ K seems larger, it turns out that ∆ 2 stands in our way. The following lemma shows ∆ 1 = 1 K ∑ K k =1 τ k is bounded by m .

Lemma 3. In a distributed environment of m workers ∆ 1 = 1 K ∑ K k =1 τ k ≤ m -1 ≤ m .

Given Lemma 3 , we know ∆ 1 / √ K = O ( m/ √ K ) and we can replace dependency on ∆ 1 by m . Now we consider ∆ 2 = 1 K ∑ K k =1 τ 2 k . Even if we have a larger denominator K neutralizing the effect of ∆ 2 , the following example shows that in the worst case, ∆ 2 can be of O ( K ) and result in O (1) error.

Example 1 (Why ∆ 2 hurts performance) . Given sequence of delays { τ k } such that τ k = 0 , k ≤ K -1 , τ K = K . Then ∆ 2 = K and ∆ 2 /K = 1 .

The example tells that delays of O ( K ) ruin our convergence. In other words, to recover an overall O (1 / √ K ) convergence rate, we need ∆ 2 /K = O (1 / √ K ) ⇒ ∑ K k =1 τ 2 k = O ( K 3 / 2 ) . The next lemma provides a hint for our algorithm design.

Lemma 4. If a sequence of nonnegative integers { τ k } satisfy ∑ K k =1 τ k ≤ mK , then given T ≥ 0 ,

Lemma 4 tells us two facts about a nonnegative integer sequence of length K with sum bounded by O ( K ) : 1) . if we restrict the elements to be less than O ( T ) , then ∑ K k =1 τ 2 k ≤ O ( KT ) . 2) . there will be Ω( K -mKT -1 ) elements bounded by T . Back to our context, this implies 1) . To reduce ∆ 2 /K to O (1 / √ K ) , we can discard the iterations of delays greater than T = O ( √ K ) . 2) . We skip no more than O ( √ K ) iterations and optimization works with O ( K -√ K ) = O ( K ) iterations left.

Having established the foundational understanding, we outline the main steps in Algorithm 3 . It is based on the aforementioned intuition and incorporates a 'safeguarding' step to discard inaccurate information which could potentially hurt the convergence performance. As the above argument on the accumulated delay is independent of any specific algorithm, the safeguarding strategy can be applied to both DSGD and DSPL . To make our parameter setting more general, in our analysis we consider T = r -1 mK β , r > 0 , β ≥ 0 .

Intuitively, under worst-case scenarios, Algorithm 3 will perform after K iterations in a manner similar to its non-safeguarded counterpart. However, the maximum delay will be capped at r -1 mK β , and the iteration count will be K (1 -rK -β ) .

Theorem 4 (Safeguarded DSGD ) . Under the same conditions as Lemma 1 as well as D1 , taking 1) β > 0 , K > r 1 /β or 2) β = 0 , r < 1 , then letting γ k ≡ γ = √ K α √ η + ρ + κ +4 λ, η = 1 + r K β -r for

some α > 0 and k ∗ be uniformly chosen between iterations where τ k ≤ T = r -1 mK β ,

Theorem 5 (Safeguarded DSPL ) . Under the same conditions as Lemma 2 as well as D1 , taking 1) β > 0 , K > r 1 /β or 2) β = 0 , r < 1 , then letting γ k ≡ γ = √ K α √ η + ρ + κ +6 λ, η = 1 + r K β -r for some α > 0 and k ∗ be uniformly chosen between iterations where τ k ≤ T = r -1 mK β ,

where D = ψ 1 /ρ ( x 1 ) -inf x ψ ( x ) .

Remark 5 . Theorem 4 and 5 show that by employing a safeguarding step, both DSGD and DSPL can achieve delay-independent rates. It is also interesting to see how the choice of β and r affects performance. To recover a convergence rate of O ( K -1 / 2 ) , it is sufficient to have β ≤ 1 / 2 . If we set β > 0 , then we skip rK -β of all the iterations and incur a penalty of up to η 3 / 2 in DSGD and η 1 / 2 in DSPL . However, this loss becomes negligible for large K , as η = 1 + r K β -r → 1 . Alternatively, if we set β = 0 , DSGD achieves O ( √ m/ √ K + m/K ) rate with α = 1 / √ m while DSPL yields a rate of O (1 / √ K + m 2 /K ) with α = 1 . This setting aligns with [7, 18] and achieves optimal rate on m . However, the penalty from η > 1 is non-negligible and adversely affects the overall convergence rate by a constant factor. Therefore, we can strike a balance in practice by choosing β in (0 , 1 / 2] and allow delays of up to O ( K β ) .

6 Experiments

This section performs numerical experiments on the robust phase retrieval problem to demonstrate the efficiency of our methods. Given a measuring matrix A ∈ R m × n and a set of observations b i ≈ |⟨ a i , ˆ x ⟩| 2 , 1 ≤ i ≤ m ( m in this section represents the number of samples), robust phase retrieval aims to recover the true signal ˆ x from

where δ S denotes the set indicator function and the ℓ 1 loss function improves robustness. Our experiment contains three parts. The first part profiles algorithms in an asynchronous environment simulated via MPI; our second experiment runs sequentially with simulated delays from common distributions; our last experiment also runs in the simulated environment and demonstrates the effectiveness of safeguarding step under adversarially chosen delays.

6.1 Experiment setup

Synthetic data . For the synthetic data, we take m = 300 , n = 100 in the experiments of simulated delay and m = 1500 , n = 500 in the asynchronous environment. Data generation follows the setup of [11], where, given some κ ≥ 1 , we compute A = QD,Q ∈ R m × n , q ij ∼ N (0 , 1) and D = diag ( d ) , d ∈ R n , d i ∈ [1 /κ, 1] for all i . Then we generate a true signal ˆ x ∼ N (0 , I ) and obtain the measurements b using formula b i = ⟨ a i , ˆ x ⟩ 2 . Last we randomly choose p fail-fraction of the measurements and add N (0 , 25) to them to simulate data corruption.

Real-life data . The real-life data is generated from zipcode dataset, where we vectorize a 16 × 16 hand-written digit from [16] and use it as the signal. The measuring matrix A comes from a normalized Hadamard matrix H ∈ R 256 × 256 : we generate three diagonal matrices S j = diag ( s j ) , j = 1 , 2 , 3 ; each element of s j ∈ R 256 is taken from {-1 , 1 } randomly and we let A = H [ S 1 , S 2 , S 3 ] T ∈ R 768 × 256 . Finally p fail-fraction of the observations are set 0.

Dataset . In the asynchronous environment, we keep up with [34] setting κ = 1 , p fail = 0 and in the simulated environment, we follow [11] setting κ ∈ { 1 , 10 } and p fail ∈ { 0 . 2 , 0 . 3 } .

Initial point and radius . Synthetic data: we generate x ′ ∼ N (0 , I n ) and start from x 1 = x ′ ∥ x ′ ∥ ; zipcode data: we generate x ′ ∼ N (ˆ x, I n ) and take x 1 = 10 x ′ . M = 1000 ∥ x 1 ∥ .

Stopping criterion . We run algorithms for 400 epochs ( K = 400 m ). In the asynchronous environment, algorithms run until reaching the maximum iteration. In the simulated environment, algorithms stop if f ≤ 1 . 5 f (ˆ x ) . When f contains corrupted measurements, f (ˆ x ) > 0 .

Stepsize . We set γ = √ K/α , where α ∈ { 0 . 1 , 0 . 5 , 1 . 0 } in the asynchronous environment, α ∈ [10 -2 , 10 1 ] for synthetic data and α ∈ [10 1 , 10 2 ] for the zipcode dataset.

Simulated delay . In the simulated environment, we generate τ k from two common distributions from literature, which are geometric G ( p ) and Poisson P ( λ ) [37]. After the delay is generated, it is truncated by twice the mean of the distribution.

Adversarial delay . We let delay happen at the last iteration of each epoch and use the information of x 1 to update. Safeguarding parameter is set to T = 0 . 1 √ K . 表格 1

worker

SGD

SPL

worker

SGD

SPL

Trade-off between computation and communication . In the asynchronous environment, the numerical linear algebra on the worker uses a raw implementation (not importing package) to balance the cost of gradient computation and communication. 1 316 341 1 19.6 34.7 2 157 173 2 9.04 24.3 4 82 90 4 8.8 24 8 49 55 8 9.2 25.2 16 28 33 16 10.9 33.1

worker

SGD

SPL

1

1

1

6.2 Asynchronous environment 2 2.01273885350318 1.97109826589595 4 3.85365853658537 3.78888888888889

8

6.44897959183673

6.2

16

11.2857142857143

10.3333333333333

Our first experiment runs in an asynchronous environment implemented by MPI Python interface and is profiled on an Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz machine with 10 cores and 20 threads. This experiment runs on a single machine to verify our theoretical finding rather than to test the algorithm's real performance on a specific distributed architecture.

0

3

6

9

12

1

2

SGD

SPL

0

100

200

300

400

1

2

4

8

16

SGD

SPL

Running time

Number of workers

0

10

20

30

40

10 -6

10 -4

10 -2

10 0

DSGD 1 workers

DSPL 1 workers

DSGD 2 workers

DSPL 2 workers

DSGD 4 workers

DSPL 4 workers

DSGD 8 workers

DSPL 8 workers

DSGD 16 workers

DSPL 16 workers

Distance to optimal

Number of epochs

0

10

20

30

40

10 -6

10 -4

10 -2

10 0

DSGD 1 workers

DSPL 1 workers

DSGD 2 workers

DSPL 2 workers

DSGD 4 workers

DSPL 4 workers

DSGD 8 workers

DSPL 8 workers

DSGD 16 workers

DSPL 16 workers

Function value gap

Number of epochs

1

The first figure plots the wall-clock time (in seconds) for DSGD and DSPL to complete 400 epochs when the number of workers increases. It is observed that both algorithms exhibit speed-up with more workers and note that DSPL takes more time than DSGD due to the need to pass the function value to the master and slightly more expensive updates. But as the second and the third figure suggest, this extra cost is justified by the superior convergence: in the first several epochs DSPL reaches a high accuracy of 10 -6 in both function value and distance to the optimal solution, while DSGD stagnates at a relatively low-accuracy solution of 10 -2 . These observations suggest that DSPL offers better convergence behavior than the methods only based on subgradient. Finally, our experiments suggest both DSGD and DSPL are not sensitive to the increase in the number of workers when there are relatively few workers.

6.3 Simulated environment

The second part of our experiment compares the performance between DSGD and DSPL and is based on the simulated delay, where the algorithm runs sequentially but the gradient information is computed from the previous iterates.

Figure 3 plots the number of iterations for each algorithm to converge under different datasets and delay parameters. We see that in spite of delays, DSPL admits a wider range of stepsize parameters ensuring convergence than DSGD , and the performance slightly deteriorates as delay increases.

Iteration count

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

6.4 Adversarial delay

The final experiment verifies the efficacy of our safeguarding step when delays are introduced in an adversarial setting. We evaluate the performance of DSGD , DSPL , both with and without the safeguarding step, under the delay patterns we have generated.

Function value

3.5

3.55

3.6

3.65

3.7

3.75

3.8

10 4

0.42

0.44

0.46

0.48

0.5

0.52

DSGD

Safe-DSGD

DSPL

Safe-DSPL

2.35

2.4

2.45

2.5

2.55

2.6

2.65

10 4

0.2

0.3

0.4

0.5

0.6

0.7

0.8

DSGD

Safe-DSGD

DSPL

Safe-DSPL

2.5

3

3.5

10 4

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

DSGD

Safe-DSGD

DSPL

Safe-DSPL

2.5

3

3.5

4

4.5

5

10 3

0.4

0.6

0.8

1

1.2

DSGD

Safe-DSGD

DSPL

Safe-DSPL

Figure 4 clearly illustrates the superior performance of our method incorporating the safeguarding step. As suggested by our theory, DSGD is notably sensitive to delays of O ( K ) . However, upon discarding outdated information, DSGD exhibits much greater robustness. Interestingly, even under our adversarial setup, the performance of DSPL remains acceptable, albeit slightly inferior to its safeguarded version, which aligns well with our theoretical findings.

7 Conclusions

We offer a sharp analysis of delayed stochastic algorithms for weakly convex optimization, discussing the widely utilized DSGD method and introducing the novel DSPL method for problems with a composition structure. Through careful examination of delay factors, we propose a straightforward safeguarding approach to eliminate the effect of delays, and instead derive bounds depending on the number of agents in the distributed environment. This makes our algorithms resilient to arbitrary delays. A promising future direction is the application of these prox-linear methods in more diverse distributed settings, such as decentralized networks.

References

Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In J. ShaweTaylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory , pages 111-132. PMLR, 2020.

Karl Bäckström, Marina Papatriantafilou, and Philippas Tsigas. Mindthestep-asyncpsgd: Adaptive asynchronous parallel stochastic gradient descent. In 2019 IEEE International Conference on Big Data (Big Data) , pages 16-25. IEEE, 2019.

Dimitri Bertsekas and John Tsitsiklis. Parallel and distributed computation: numerical methods . Athena Scientific, 2015.

Vasileios Charisopoulos, Yudong Chen, Damek Davis, Mateo Díaz, Lijun Ding, and Dmitriy Drusvyatskiy. Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Foundations of Computational Mathematics , 21(6):1505-1593, 2021.

Shixiang Chen, Alfredo Garcia, and Shahin Shahrampour. On distributed non-convex optimization: Projected subgradient method for weakly convex problems in networks. IEEE Transactions on Automatic Control , 2021.

Alon Cohen, Amit Daniely, Yoel Drori, Tomer Koren, and Mariano Schain. Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems , 34, 2021.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019.

Damek Davis, Dmitriy Drusvyatskiy, and Kellie J MacPhee. Stochastic model-based minimization under high-order growth. arXiv preprint arXiv:1807.00255 , 2018.

Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems , 25, 2012.

Qi Deng and Wenzhi Gao. Minibatch and momentum model-based methods for stochastic weakly convex optimization. Advances in Neural Information Processing Systems , 34, 2021.

Dmitriy Drusvyatskiy and Courtney Paquette. Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming , pages 1-56, 2018.

John C Duchi and Feng Ruan. Stochastic methods for composite and weakly convex optimization problems. SIAM Journal on Optimization , 28(4):3229-3259, 2018.

John C Duchi and Feng Ruan. Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval. Information and Inference: A Journal of the IMA , 8(3):471529, 2019.

R Fletcher. A model algorithm for composite nondifferentiable optimization problems. In Nondifferential and Variational Techniques in Optimization , pages 67-76. Springer, 1982.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning , pages 3252-3261. PMLR, 2019.

Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. Advances in Neural Information Processing Systems , 35:17202-17215, 2022.

Adrian S Lewis and Stephen J Wright. A proximal method for composite minimization. Mathematical Programming , 158(1):501-546, 2016.

Xiao Li, Zhihui Zhu, Anthony Man-Cho So, and Jason D Lee. Incremental methods for weakly convex optimization. arXiv preprint arXiv:1907.11687 , 2019.

Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. Advances in Neural Information Processing Systems , 28, 2015.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning , pages 3043-3052. PMLR, 2018.

Vien Mai and Mikael Johansson. Convergence of a stochastic gradient method with momentum for non-smooth non-convex optimization. In International Conference on Machine Learning , pages 6630-6639. PMLR, 2020.

Brendan McMahan and Matthew Streeter. Delay-tolerant algorithms for asynchronous distributed online learning. Advances in Neural Information Processing Systems , 27, 2014.

Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Angelia Nedi´ c, Dimitri P Bertsekas, and Vivek S Borkar. Distributed asynchronous incremental subgradient methods. Studies in Computational Mathematics , 8(C):381-407, 2001.

A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization , 19(4):1574-1609, 2009.

Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. volume 24, 2011.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400-407, 1951.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In COLT , volume 2, page 5, 2009.

Suvrit Sra, Adams Wei Yu, Mu Li, and Alex Smola. Adadelay: Delay adaptive distributed stochastic optimization. In Artificial Intelligence and Statistics , pages 957-965. PMLR, 2016.

Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed updates. Journal of Machine Learning Research , 21:1-36, 2020.

Yangyang Xu, Yibo Xu, Yonggui Yan, and Jie Chen. Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems. SIAM Journal on Imaging Sciences , 15(2):550-590, 2022.

Jinshan Zeng and Wotao Yin. On nonconvex decentralized gradient descent. IEEE Transactions on signal processing , 66(11):2834-2848, 2018.

Junyu Zhang and Lin Xiao. Stochastic variance-reduced prox-linear algorithms for nonconvex composite optimization. Mathematical Programming , pages 1-43, 2021.

Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji Liu. Staleness-aware async-sgd for distributed deep learning. arXiv preprint arXiv:1511.05950 , 2015.

Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning , pages 4120-4129. PMLR, 2017.

Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. Advances in neural information processing systems , 23, 2010.

Appendix

Table of Contents

Structure of the appendix In this paper, we strive to provide a comprehensive treatment of delayed stochastic algorithms for stochastic weakly convex optimization in both theoretical and practical aspects.

1) Theoretically , we choose to employ a more general Bregman divergence context in our proof. This broader context covers the primary results for the Euclidean setting discussed in the main section. Specifically, this context will be invoked in our analysis for DSPL method to accommodate real-life applications of prox-linear method and by relaxing the Lipschitz condition C1 .

2) Practically , we provide a detailed discussion on how to implement the DSPL algorithm, both in Euclidean and Bregman context. We specifically address the selection of the Bregman divergence kernel and propose efficient subroutines to solve proximal subproblems. Additionally, in view of the wide usage of momentum techniques in stochastic optimization, we analyze DSEPL , the momentum variant of DSPL . While this is not the central contribution of our paper, we believe it represents an important step for making DSPL applicable to real-world problems.

Main results

Practical extensions

(Optional)

Section A

Other related works

Section B

1. Context setup

2. Auxiliary results

Bregman

Euclidean

∋

Section C

Convergence of DSGD

(Euclidean)

Section D

Convergence of DSPL

Section E

Delay-independent

Analysis for DSGD

and DSPL

Section F

Convergence of

DSEPL (Euclidean)

Section G

Section H

Additional

Experiments

The appendix is organized as follows.

The first five sections address the theoretical aspects of delayed stochastic algorithms.

In Section A, we first conduct an extended review of other related works. In Section B, we introduce the Bregman context with some auxiliary results. In Section C, we perform convergence analysis for the DSGD method, and Section D presents a more general Bregman DSPL that relaxes the Lipschitzness assumption. Section E presents the safeguarded variant of our algorithms that are robust to arbitrary delays.

(Optional) The last three sections are devoted to more practical aspects.

In Section F, we present the convergence analysis of DSEPL , namely, the momentum variant of DSPL using variable extrapolation. Section G discusses how to construct the kernel and solve the Bregman proximal subproblems. Finally, Section H displays additional experiment results.

A More on the related works

First, we review the literature on stochastic weakly-convex optimization. The Prox-Linear (PL) method [19, 12, 15] has received more attention lately. The seminal work [8] conducts a novel complexity analysis using the Moreau envelope as the potential function. They show that SGD (and many other stochastic algorithms) achieves an O ( 1 ε 4 ) complexity in terms of convergence to the proximity of approximate stationary points. [20] analyzes an incremental subgradient method for finite-sum problems. [36] extends the prox-linear algorithm to the finite-sum setting and provides analysis based on variance-reduction. SGD with momentum is studied in [24] and [11] reveals that SPL can be accelerated by minibatching even for nonsmooth objective functions. [11] also employs heavy-ball momentum to further improve the model-based optimization.

There is substantial literature on distributed and asynchronous optimization. We refer to the seminal work [4]. In addition to the aforementioned asynchronous issues, another important research direction concerns reducing the network communication cost by gradient compression. For example, see [17]. Besides the centralized setting, there is also growing interest in decentralized algorithms. Recently, [6] presents a decentralized SGD in the network where nodes exchange parameters and subgradients locally. However, their method requires all the nodes to update synchronously and only

proves asymptotic convergence of SGD . An extensive study of decentralized optimization is beyond our work. Therefore, we refer interested readers to [22, 23, 35] for more recent advances.

B Bregman context and auxiliary results

We initiate the theoretical discussion by introducing the Bregman context, which includes the concepts of Bregman divergence, relative Lipschitzian property, and Bregman Moreau envelope. Following the provision of requisite definitions, we present some auxiliary results that will be used frequently in our analysis.

B.1 Context setup

Bregman divergence and relative Lipschitzness Given a Legendre function d ( · ) [7] that is proper, closed, strictly convex and essentially smooth, it induces the Bregman divergence defined by

and d is called the kernel of V d . Since d is strictly convex, we know V d ( x, y ) ≥ 0 and the equality holds if and only if x = y . The notion of Bregman divergence greatly enhances the coverage of the algorithms and has recently been adopted in the context of weakly convex optimization [8, 3] to tackle a broader class of nonsmooth nonconvex optimization problems. In this paper, we mainly leverage the concept of relative Lipschitzian property to extend the analysis of delayed stochastic algorithms. First, we give its formal definition.

Definition 1 (Relative Lipschitzian property) . [3] A function f satisfies L -relative Lipschitzian property to kernel d if for all x, y ∈ dom d ,

Bregman proximal mapping and Bregman envelope

As in conventional weakly convex optimization, we need an approximate measure of stationarity in the non-Euclidean setting. A naturally choice is the Bregman envelope [3]. Assume that d is 1-strongly convex, then for any ρ > 0 , the Bregman envelope of f with respect to kernel d is

and the Bregman proximal mapping is represented using

We sometimes refer to prox d ψ/ρ ( x ) as ˆ x and proximal distance E [ V d (ˆ x k , x k )] has been proven a proper measure of the convergence of stochastic algorithms [3] under specific choice of d and ρ .

Connection with Euclidean geometry One advantage of using Bregman context is that it subsumes the Euclidean case. If we take d ( x ) = 1 2 ∥ x ∥ 2 , then V d ( x, y ) = 1 2 ∥ x -y ∥ 2 and relative Lipschitzian property becomes standard Lipschitz condition. Particularly we have the following relation between V d (ˆ x, x ) and Moreau envelope.

When the context is clear, we will use ψ = ψ d directly if d ( x ) = 1 2 ∥ x ∥ 2 . In the next section, we present several useful auxiliary results that appear frequently in the proof.

B.2 Auxiliary lemmas

In this section, we present all the auxiliary lemmas that will be used in the proof of our main results. While some of these lemmas are widely recognized in the field, we reproduce them here to ensure that our work remains self-contained. To begin, we introduce the well-known three-point lemma.

Lemma 5 (Three point lemma) . Let f be a closed convex function and define

for some y ∈ dom d . Then we have

The following lemma bounds, for a deterministic function, the proximal step by the regularization term and relative Lipschitzian property.

Lemma 6 (Bounding the proximal step) . Given the divergence V d generated by some 1-strongly convex function d . Let the function f ( x ) satisfy the relative Lipschitzian property (5) . For any γ > 0 such that f + γd is strongly convex, if we define

Proof. By the optimality of x + and relative Lipschizian property, we have

Dividing both sides of (8) by √ V d ( x + , x ) gives (9); (10) uses V d ( x + , x ) ≥ 1 2 ∥ x + -x ∥ 2 .

Lemma 7 (Bounding the proximal step in expectation) . Given the divergence V d generated by some 1-strongly convex function d . Let convex stochastic function f ( x, ξ ) satisfy the relative Lipschitzian property (5) with L = L ( ξ ) for ξ ∼ Ξ . For any γ > 0 , if we define

then and

and then

and

Proof. We can apply Lemma 6 . Then we have, for each ξ ∼ Ξ , that

Taking expectation with respect to ξ , the fact that V d ( x + , x ) ≥ 1 2 ∥ x + -x ∥ 2 completes the proof

C Convergence Analysis of DSGD

In this section, we present the convergence analysis of DSGD . The proof follows a standard inexact potential reduction scheme and is done in the Euclidean setup, where Lemma 5 and Lemma 7 hold for d = 1 2 ∥ · ∥ 2 .

C.1 Proof of Lemma 1

Since ρ > λ + κ and γ k ≥ ρ , by three-point lemma we get the following two relations

Summing up (11) and (12) and taking expectation, we have

where (13) follows from λ -weak convexity of f ( x, ξ ) and B1 (using [1]). Re-arranging the terms, we have

where (14) follows by Cauchy's inequality ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 . Now we re-arrange the terms and divide both sides by γ k -2 λ -κ to derive

Finally, we measure the reduction in potential function ψ 1 /ρ ( x k ) and successively deduce that

where relation (16) plugs (15) in. A simple re-arrangement completes the proof.

C.2 Proof of Theorem 1

By the descent property revealed in Lemma 1 , we multiply both sides of the inequality by γ k -2 λ -κ , telescope over k = 1 , . . . , K and deduce that

where (17) is due to ψ ( x ∗ ) ≤ E k [ ψ 1 /ρ ( x K +1 )] . Now it remains to bound the error from the stochastic delays. Let's first consider applying ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 to get

and we can bound the first term using

where (19) uses the fact ∥ ∑ k i =1 a i ∥ 2 ≤ k ∑ k i =1 ∥ a i ∥ 2 , (20) invokes Lemma 7 since ⟨ g, x ⟩ + ω is ( ∥ g ∥ + L ω ) Lipschitz continuous with E ξ [ ∥ g ∥ 2 ] ≤ L 2 f , and we let x k -j = x 1 if k ≥ j . On the other hand, we bound the first-order terms by

where (21) again invokes Lemma 7 . Plugging the bound back, we deduce that

where (22) is by γ = √ K/α + ρ + 4 λ + κ and we cancel the first summation term in (18) with -∑ K k =1 γ -ρ 2 ∥ x k +1 -x k ∥ 2 since γ ≥ ρ +4 λ . The proof is complete after dividing both sides by ρ ( ρ -2 λ -κ ) and using the fact that 1 2 ∥ ˆ x k -x k ∥ 2 = 1 2 ρ 2 ∥∇ ψ 1 /ρ ( x ) ∥ 2 .

D Convergence analysis of DSPL

In this section, we introduce a more general DSPL associated with Bregman divergence and establish its convergence analysis. After proving the result in our Bregman context, the Euclidean results in Section 4 follows immediately as a special case.

Why use Bregman divergence? Recall that in C1 we assume Lipschitz continuity of both c and ∇ c , which does not hold without assuming a bounded set. The main issue here is to expect a Lipschitz smooth function to also be Lipschitz continuous. Using the Bregman context, we can relax such Lipschitz continuity to relative Lipschitzian property, so that we do not necessarily need the bounded set assumption.

D.1 Preliminaries and analysis of Bregman DSPL

In this section, we present the convergence results of DSPL . As we mentioned in Remark 4 , we replace 1 2 ∥ · ∥ 2 regularization by divergence V d and summarize the update in Algorithm 4 .

Algorithm 4: A delayed stochastic prox-linear method with Bregman proximal updates

Now we overload the assumptions used in Section 4.

A1 ′ : (i.i.d. sample) It is possible to draw i.i.d. samples { ξ k } from Ξ .

A2 ′ : (Relative Lipschitzian property) ω ( x ) satisfies relatively Lipschitzian property (5)

A3 ′ : (Weak convexity) ω ( x ) is κ -weakly convex.

A4 ′ : (Strongly-convex kernel) Kernel function d is 1-strongly convex.

A5 ′ : (Bounded moment) The distribution of the independent stochastic delays { τ k } has bounded first and second moments. i.e., E [ τ k ] ≤ ¯ τ < ∞ , E [ τ 2 k ] ≤ τ σ 2 < ∞ , for all k .

C1 ′ : h ( x ) is convex and L h -Lipschitz continuous; c ( x, ξ ) has C ( ξ ) -Lipschitz continuous gradient, for all ξ ∼ Ξ . Moreover, during the algorithm the stochastic function f z ( x, ξ ) , z ∈ dom d is L f ( ξ ) -relative Lipschitzian to d for all ξ ∼ Ξ . Moreover, we assume E ξ [ C ( ξ ) 2 ] ≤ C 2 , E ξ [ L c ( ξ ) 2 ] ≤ L 2 c .

Remark 6 . It can be seen that we added extra assumptions on the kernel and relaxed the Lipschitz continuity of c in C1 ′ and ω in A2 ′ . Specially if c is L c ( ξ ) -Lipschitz continuous, we have f z ( x, ξ ) is L h L c ( ξ ) -Lipschitz continuous since

With the above assumptions, we can further derive a more general version of Proposition 1 .

Proposition 2 ( Proposition 1 in the Bregman setting) . Let { f z ( · , ξ ) } be the sequence of stochastic functions queried during the DSPL algorithm, then

P1 ′ : (Convexity) f z ( x, ξ ) is convex, for all x ∈ dom d and ξ ∼ Ξ .

P2 ′ : (Two-sided approximation) | f z ( x, ξ ) -f ( x, ξ ) | ≤ L h C ( ξ ) 2 ∥ x -z ∥ 2 , for all x ∈ dom d and all ξ ∼ Ξ .

P3 ′ : (Relative Lipschitzian property) f z ( x, ξ ) -f z ( y, ξ ) ≤ L f ( ξ ) √ 2 V d ( y, x ) for all x, y ∈ dom d and all ξ ∼ Ξ .

As we did in Section 4, we take λ = L h C and use these constants to present the results. Our analysis adopts the conventional potential reduction framework but a more careful treatment of stochastic noise is needed to improve the convergence result. The next lemma bounds the stochastic noise of the algorithm and is key to our analysis of DSPL .

Lemma 8 (Stability of the stochastic iteration) . Assume that the assumptions A1 ′ to A4 ′ hold. If f x ( · , ξ ) is convex and satisfies Proposition 2 , then the proximal iteration

satisfies for any γ k > κ .

Now we can get the lemmas that match our main results in a Bregman context.

Lemma 9 ( Lemma 2 in the Bregman setting) . Suppose A1 ′ , A2 ′ , A3 ′ , A4 ′ and C1 ′ hold, if ρ > 2 λ + κ, γ k ≥ ρ , then

Theorem 6 ( Theorem 2 in the Bregman setting) . Under the same conditions as Lemma 9 , as well as A5 ′ , taking γ k ≡ γ = ρ + 6 λ + κ + √ K/α for some α > 0 , letting k ∗ be uniformly chosen between 1 and K , then

where D = ψ d 1 /ρ ( x 1 ) -inf x ψ ( x ) .

Proof Proposition 2

f z ( x, ξ ) inherits convexity from h . The other properties hold by

and P3 ′ is by C1 ′ .

Proof of Lemma 8

Without loss of generality, we consider the following proximal mapping

where A denotes the proximal mapping from two past iterates z, x and a given sample ξ ∼ Ξ . Then we invoke the three-point lemma to get, for any u ∈ dom d that

Similarly, given some ξ ′ ∼ Ξ , v ∈ dom d , we have

Letting u = A ( z, x, ξ ′ ) in (24) and v = A ( z, x, ξ ) in (25), we sum the two relations up and get

where (26) is by Lipschitzness from A2 ′ , P3 ′ and (27) uses the relation √ a + √ b ≤ √ 2 √ a + b . Then this implies

Last we successively deduce that

where the third equality holds since ξ and ξ ′ are from the same distribution; (29) follows from Jensen's inequality and (32) uses (28). Plugging in z = x k -τ k , x = z k , ξ ′ = ξ k -τ k and γ = γ k into (32) completes the proof.

Proof of Lemma 9

First, we have, by the three-point lemma and optimality of ˆ x k , that

and that

Summing the above two relations and taking expectations, we have

where (33) uses P2 and Lemma 8 . Next we lower-bound the left-hand side using the 1-strong convexity of kernel d

Re-arranging the terms,

where the (36) uses ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 and (37) again follows by V d (ˆ x k , x k +1 ) ≥ 1 2 ∥ x k +1 -ˆ x k ∥ 2 . Now we re-arrange the terms and divide both sides by γ k -2 λ -κ to get

Now we are ready to evaluate the decrease in the potential function.

where (39) plugs in the relation from (38). Another re-arrangement completes the proof.

Proof of Theorem 6

Summing the relation from Lemma 9 from k = 1 , . . . , K and multiplying both sides by γ -2 λ -κ ,

where (41) uses the relation ψ d 1 /ρ ( x K +1 ) ≥ inf x ψ d ( x ) . Now it remains to bound the error of delays and recall that we have

and plugging the bounds back, we have

where in (43) we cancel the first summation from (42) since γ -ρ = √ K/α +6 λ + κ ≥ 6 λ . Finally we divide both sides by ρ ( ρ -2 λ -κ ) to complete the proof.

D.2 Proof of Lemma 2 and Theorem 2

First d ( x ) = 1 2 ∥ x ∥ 2 satisfies A4 ′ . Since A1 , A2 , A3 , A4 and C1 are equivalent to A1 ′ , A2 ′ , A3 ′ , A5 ′ and C1 ′ , noticing that Lemma 2 is a special case of Lemma 9 and Theorem 2 is a special case of Theorem 6 , we complete the proof.

D.3 Proof of Theorem 3

Recall that in the proof of DSPL , we actually used A4 and the properties from Proposition 1 that are deduced from A1 , A2 , A3 and C1 . Indeed, it is straightforward to verify that f z ( x, ξ ) = ⟨∇ f ( z, ξ ) , x -z ⟩ satisfies Proposition 1 given A1 , A2 , A3 and λ -smoothness of f .

E Delay-independent analysis

In this section, we further improve our analysis and show that by adopting a simple safeguarding strategy during the iterations.

E.1 Proof of Lemma 3

We show ∑ K k =1 τ k ≤ mK by noticing the following facts: 1) . at each iteration there are at most m -1 agents accumulating 1 unit of delay; 2) . τ k must come from a machine's previous delay accumulation. 3) . once the gradient is used, delay on a machine starts count from 0. Assuming that at k = 1 there are no previously accumulated delays on each agent, then we have ∑ K k =1 τ k ≤ ( m -1) K ≤ mK . We use a loose bound mK to simplify notation.

E.2 Proof of Lemma 4

We know that T ∑ K k =1 I { τ k ≥ T } ≤ ∑ K k =1 τ k ≤ mK and recall that T = r -1 mK β :

Also we have ∑ K k =1 τ 2 k ≤ T ∑ K k =1 τ k ≤ mKT .

E.3 Proof of Theorem 4

First we rewrite Lemma 1 by associating potential reduction with I k := I { τ k ≤ T } .

where we have x k +1 = x k , thus E k [ ψ 1 /ρ ( x k +1 )] = ψ 1 /ρ ( x k ) if I k = 0 .

From Lemma 4 , we know that ∑ K k =1 I k ≥ K -rK 1 -β = K (1 -rK -β ) = η -1 K . Then similar telescopic sum over the un-skipped iterations will give

where (44) uses the fact that γ k ≡ γ is the same for all the iterations; (45) uses the relation γ = √ K α √ η + ρ + κ +4 λ ≥ max { √ K α √ η , ρ + κ +4 λ } . Then we bound the above terms, respectively, by

where we used the fact that ∑ K k =1 I k ≤ K . Meanwhile,

where (48) uses a tighter bound ∑ K k =1 τ k ≤ ( m -1) K . Finally, we bound the delays using

and that 1 K 2 ∑ K k =1 τ 2 k I k ≤ mKT K 2 = m 2 rK 1 -β . Putting the above bounds back, using the same technique as in (18) to cancel the error 2 λ ∥ x k +1 -x k ∥ 2 with -∑ K k =1 γ -ρ 2 ∥ x k +1 -x k ∥ 2 , γ -ρ ≥ 4 λ , we can re-arrange the terms to complete the proof.

E.4 Proof of Theorem 5

We prove Theorem 5 by showing its Bregman version.

Theorem 7 (Safeguarded DSPL in Bregman setting) . Under the same conditions as Lemma9 as well as D1 , taking 1) β > 0 , K > r 1 /β or 2) β = 0 , r < 1 , then letting γ k ≡ γ = √ K α √ η + ρ + κ +6 λ, η =

1 + r K β -r for some α > 0 and k ∗ be uniformly chosen between iterations where τ k ≤ √ K , then √ √

Proof of Theorem 7

Similar to the proof of Theorem 4 , we write the modified potential reduction by

Then telescoping gives

where (49) reuses (46), (47) and (48). A re-arrangement completes the proof of Theorem 7 . Taking d ( x ) = 1 2 ∥ x ∥ 2 completes the proof of Theorem 5 .

F DSPL with momentum

F.1 Preliminaries of DSEPL

Momentum has been an important ingredient for a stochastic algorithm to be implemented in practice. In this section, we incorporate the extrapolation technique into the delayed prox-linear algo-

rithm. Before the master performs a proximal update, it uses two recent iterates to compute an extrapolated iterate y k . Then a proximal update is done centered around y k with delayed information. We summarize the procedure in Algorithm 5 . Throughout this section we take γ k ≡ γ .

Algorithm 5: Extrapolated DSPL

Input:

,

x

1

, β

;

for k =

x

0

1, 2,

. . .

do

Let c ( x k -τ k , ξ k -τ k ) and ∇ c ( x k -τ k , ξ k -τ k ) be computed by a worker with delay τ k ; In the master node update

end

To analyze DSEPL , we extend the framework from [5] to the delayed case and the analysis is based on a virtual iterate

where the extrapolation parameter, also known as momentum, is fixed at some constant β ∈ [0 , 1) and θ = 1 -β . Using the virtual iterate, DSEPL uses a more complicated potential function.

As extrapolation increases the instability of the iterations, analysis of DSEPL furthers requires boundedness of the delays.

E1: (Bounded delay) Independent delays are bounded by τ < ∞ .

The following lemma presents a similar descent property for the potential function.

Lemma 10. Suppose A1 , A2 , A3 and C1 hold. Given 0 ≤ β < 1 , ρ > 3 λ +2 κβ + κ and γ ≥ ρ ,

where ε k = ( λθ + λ 2 ) ∥ x k +1 -x k -τ k ∥ 2 + λ (1 -θ ) 2 ∥ x k -x k -τ k ∥ 2 characterizes the error of delay.

We next bound the stochastic delay using E1 and eliminate the delays.

Lemma 11. Under the same conditions as Lemma 10 as well as E1 , given γ > max { ρ, 2( ρ + κβ ) θ -1 +2 ρβ 2 θ -3 } ,

Remark 7 . Lemma 11 reveals the effect of extrapolation. After simplification θ = 1 -β appears in the denominator of the error from delay and will enlarge such an error if β → 1 . This is intuitive since when extrapolating using two iterations generated from delayed information, the error is magnified.

By taking less aggressive steps, we bound the term ∑ K k =1 E [ ∥ x k +1 -x k ∥ 2 ] and arrive at the final convergence result for DSEPL .

Theorem 8. Under the same conditions as Lemma 11 , if we further choose γ ≥ 2 θ -1 ( ρ + κβ ) + 2 ρθ -3 β 2 +2 λθ -2 β +3 λθ -1 τ 2 , then

which implies that the delay is negligible if τ = o ( K 1 / 4 ) .

Remark 9 . Although our convergence result is based on the extrapolated sequence, we can leverage Lemma 10 to show that E [ ∥ x k ∗ +1 -x k ∗ ∥ 2 ] , and subsequently E [ ∥ z k ∗ -x k ∗ ∥ 2 ] is O ( 1 K ) . Using smoothness of the Moreau envelope, we finally have E [ ∥∇ ψ 1 /ρ ( x k ∗ ) ∥ 2 ] is O ( 1 √ K ) .

F.2 Convergence analysis of DSEPL

In this section we present the convergence analysis for DSEPL .

F.2.1 Auxiliary results

To show the convergence of DSEPL , we need to define an auxiliary sequence

Given x k , define ¯ x = βx k +(1 -β ) x for x ∈ dom( ω ) and θ = 1 -β . The following identities hold

and will be used frequently in the analysis.

Proof of Lemma 10

First by the ( γ -κ ) -strong convexity of the proximal sub-problem, we have

Also, since f x k -τ k ( · , ξ k -τ k ) + ω ( · ) + κ 2 ∥ · -x k ∥ 2 is convex, we plug ¯ x in and apply convexity to get that

where the second inequality leverages two-sided approximation since ∣ ∣ f x k -τ k ( x, ξ k -τ k ) -f ( x ) ∣ ∣ ≤ λ 2 ∥ x -x k -τ k ∥ 2 . Then a simple re-arrangement gives

Now combining the above inequality with (51), we get that

From now on we let x = ˆ z k in (52) and successively deduce that

where the second inequality (53) applies ∥ ˆ z k -x k ∥ 2 ≤ 2[ ∥ x k +1 -x k ∥ 2 + ∥ ˆ z k -x k +1 ∥ 2 ] and the last inequality applies ∥ ˆ z k -x k -τ k ∥ 2 ≤ 2 ∥ ˆ z k -x k +1 ∥ 2 +2 ∥ x k +1 -x k -τ k ∥ 2 .

By definition of ˆ z k , for ρ > λ + κ , we have ( ρ -λ -κ ) -strong convexity of ψ ( x ) + ρ 2 ∥ · -x ∥ 2 and

Multiplying (55) by θ and adding it to (54), we have

Then we bound the first line of the right hand side in (56) by

where the inequalities invoke two-sided approximation ∣ ∣ f x k -τ k ( x k , ξ k -τ k ) -f ( x k , ξ k -τ k ) ∣ ∣ ≤ λ 2 ∥ x k -x k -τ k ∥ 2 and bound E ξ [ f x k -τ k ( x k +1 , ξ ) ] -f x k -τ k ( x k +1 , ξ k -τ k ) with Lemma 8 . Now plugging (57) back into (56) gives

Isolating the delayed error by ε k = ( λθ + λ 2 ) ∥ x k +1 -x k -τ k ∥ 2 + λ (1 -θ ) 2 ∥ x k -x k -τ k ∥ 2 and we have, by algebraic manipulations of the momentum terms, that

and that

Combining (59), (60) with (58) and taking expectation, we have

After re-arrangement, we arrive at

Dividing both sides of (61) by ( γ -κ ) θ 2 2 , we obtain that

Next, we consider the Moreau envelope and

where (63) uses the relation from (62). Last we re-arrangement the terms in (64) to get

Recalling that ∥∇ ψ 1 /ρ (ˆ z k ) ∥ 2 = ρ 2 ∥ ˆ z k -z k ∥ 2 , we complete the proof. Proof of Lemma 11

Summing up the inequality (65) from k = 1 to K , multiplying both sides by γ + κ > 0 and taking expectation, we have

where the last inequality uses x 0 = x 1 and D = max { ψ 1 /ρ ( z 1 ) -ψ 1 /ρ ( z ∗ ) , ψ ( x 1 ) -ψ ( x ∗ ) } . Now we divide both sides of the inequality by ρ -κθ 2 ρθ to get

Last we bound ∑ K k =1 E [ ε k ] . First, we have the following relations

where the first inequality is by E1 and the second inequality uses the fact that x 0 = x 1 . Similarly

where the first inequality uses ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 and the second inequality re-uses the bound of the first term. Plugging the above bounds back into ∑ K k =1 E [ ε k ] , we successively deduce

that

which implies

and this completes the proof.

Proof of Theorem 8

By Lemma 11 , it remains to bound the quantity ∑ K k =1 E [ ∥ x k +1 -x k ∥ 2 ] . To this end we consider the relation in (66), where we have

Since we choose β, γ such that γθ 2 -2 θ ( ρ + κβ ) -2 ρβ 2 θ -1 -3 λτ 2 -2 λβ > 0 , then

Plugging the bound back, we have

This completes our proof.

G Kernel and subproblems

In this section, we delve into the practical considerations related to Bregman proximal methods. In particular, we discuss how to construct suitable divergence kernels for DSPL so that we can accommodate non-Lipschitzness of the objective function. We also propose efficient subroutines for solving Bregman proximal subproblems, which are applicable to a wide range of loss functions typically encountered in machine learning tasks.

G.1 Choosing the divergence kernel

The selection of the kernel function is a critical aspect of the Bregman proximal method. The choice of kernel has been widely discussed in the literature [3, 6], and we incorporate these findings to construct suitable kernels for the DSPL method. Given the assumptions we have made throughout the paper, it suffices to consider kernels d ( x ) = P p n ( ∥ x ∥ ) := ∑ n k =0 p k ∥ x ∥ k , which are degreen polynomials in ∥ x ∥ . We recall the following lemma and refer the readers to [3] for its proof.

Lemma 12. Given f ( x ) such that f ( x ) -f ( y ) ∥ x -y ∥ ≤ P p n ( ∥ x ∥ ) + P p n ( ∥ y ∥| ) = ∑ n k =0 p k ( ∥ x ∥ k + ∥ y ∥ 2 ) , p ≥ 0 , ∀ x ∈ dom h , then f satisfies 1 -relative Lipschitzian property with kernel d ( x ) = ∑ n k =0 3 k +7 k +2 p k ∥ x ∥ k +2 .

The above lemma implies that once we establish a polynomial upper bound on the Lipschitzness of f z ( x, ξ ) + ω ( x ) , a kernel d is immediately available.

We note that the above kernel covers most of the applications [2] of the prox-linear method, including phase retrieval, blind deconvolution, matrix completion, covariance estimation, robust PCA, and so on. In many machine learning applications, c has Lipschitz continuous gradient, meaning that ∇ c is bounded by first-order growth. For example, in the phase retrieval problem, we have ω = 0 , h ( x ) = | x | , c ( x, ξ ) = ⟨ a, x ⟩ 2 -b . Hence, we get

Now assume that z = x , then for any p 0 ≥ 0 and p 1 = 2 α 2 , α = sup a ∼ Ξ ∥ a ∥ we know that 2 ∥ a ∥ 2 ∥ x ∥ ≤ p 0 + p 1 ∥ x ∥ . Hence, we can take d ( x ) = 7 2 ∥ x ∥ 2 + 10 α 2 3 ∥ x ∥ 3 as our kernel. Recall that in DSPL z = x k -τ k , x = x k , as long as there exist some M,N such that ∥ x j ∥ ≤ M ∥ x k ∥ + N,k ≥ j for all k , our assumptions are satisfied.

Now that we have specified ways to construct kernels for DSPL , we discuss how to solve the Bregman proximal subproblems in the next section. The subproblems' solution determines the practical applicability of our methods, and we show that these subproblems can be efficiently solved for a wide range of problems.

G.2 Bregman proximal subproblem

In this section, we discuss the solution of the proximal subproblems for DSPL when ω = 0 , where the Bregman proximal subproblem can then be expressed as

Consider a piece-wise linear h ( x ) = max { α 1 x + β 1 , α 2 x + β 2 } . This formulation characterizes common nonsmooth loss functions including ℓ 1 loss | x | = max { x, -x } and hinge loss max { 0 , x } . Substituting h in and removing terms irrelevant to x , we arrive at the following DSPL subproblem.

The next proposition provides a solution to the above subproblem.

Proposition 3 (Solving the proximal subproblem) . The proximal subproblem can be solved by evaluating solutions to the following three problems

where the solution to the third problem satisfies x = u + αv , where u = -( a 1 -a 2 )( b 1 -b 2 ) ∥ a 1 -a 2 ∥ 2 , v =

( ∥ a 1 ∥ 2 -∥ a 2 ∥ 2 ) 2 ∥ a 1 -a 2 ∥ 2 ( a 1 -a 2 ) -a 1 + a 2 2 and α is a positive root of the following equation

Proof. Recall that we are solving the following convex optimization problem

and

where d is strongly convex and thus the problem admits a unique minimizer x ∗ . Then we do case analysis.

Case 1 . ⟨ a 1 , x ∗ ⟩ + b 1 > ⟨ a 2 , x ∗ ⟩ + b 2 . In this case solving min x ⟨ a 1 , x ⟩ + d ( x ) produces x ∗ .

The above two cases degenerate into the subproblems from stochastic mirror descent and its solution has been discussed in [6]. The last case is a bit more tricky.

In this case, the proximal subproblem becomes an equality constrained problem.

Letting λ be the multiplier of the equality constraint and appealing to the optimality condition, we have

Summing (70) (71) and dividing both sides by 2, we have

Multiplying both sides of (73) by a 1 -a 2 , we arrive at

Since d ( x ) is a polynomial in ∥ x ∥ , ∇ d ( x ) = ζx, ζ > 0 and using (72), we have

Then substituting λ into (74),

we get

for some ζ > 0 . Without loss of generality, we express x = u + αv for u = -( a 1 -a 2 )( b 1 -b 2 ) ∥ a 1 -a 2 ∥ 2 and v = ( ∥ a 1 ∥ 2 -∥ a 2 ∥ 2 ) 2 ∥ a 1 -a 2 ∥ 2 ( a 1 -a 2 ) -a 1 + a 2 2 and α > 0 . Substituting back gives ζ = ∑ n k =0 p k ∥ x ∥ k and

Therefore, we only need to verify the positive roots of ∑ n k =0 p k α ∥ u + αv ∥ k -1 = 0 , which can be done efficiently if n is not large. This completes the proof.

G.3 Euclidean subproblem with nonsmooth regularizer

In this section, we discuss the solution of the proximal subproblems for DSPL when d ( x ) = 1 2 ∥ x ∥ 2 and ω takes on one of two forms: 1) . Indicator function of ball δ {∥ x ∥≤ R } . 2) . ℓ 1 regularizer ∥ x ∥ 1 . Westill assume h ( x ) = max { α 1 x + β 1 , α 2 x + β 2 } , which covers all the popular applications of SPL from [2, 4]. Following the same argument as in the previous section, each DSPL iteration is reduced to

and after doing case analysis on ⟨ a 1 , x ∗ ⟩ + b 1 and ⟨ a 2 , x ∗ ⟩ + b 2 as in Proposition 3 , it suffices to solve

efficiently.

Case 1. ω ( x ) = δ {∥ x ∥≤ R } . In this case, we need to solve

whose KKT condition is given by

After simplification, we arrive at

Then either λ = 0 or the solution can be obtained using bisection.

Case 2. ω ( x ) = ∥ x ∥ 1 . In this case, we need to solve

Given the Lagrangian multiplier ν associated with the equality constraint, we have that x ( ν ) = S ( y -νa ) , which is given by the soft-thresholding operator. The residual of the equality constraint is given by:

This function is univariate in ν and is semi-smooth. Therefore, semi-smooth Newton method can efficiently find its root. This makes the computation tractable.

H Additional experiments

In this section, we present additional experiments incorporating momentum. Our experiments focus on both robust phase retrieval and blind deconvolution problems.

H.1 Robust phase retrieval

We refer readers to Section 6 for the detailed formulation of phase retrieval. Most of the experiment settings in this section are consistent with Section 6.

H.1.1 Experiment setup

Initial point and radius . For the synthetic data, we generate x ′ ∼ N (0 , I n ) and start from x 0 = x 1 = x ′ ∥ x ′ ∥ and for zipcode data, we generate x ′ ∼ N (ˆ x, I n ) and take x 0 = x 1 = x ′ . M = 1000 ∥ x 0 ∥ .

Stepsize . We tune the stepsize parameter setting γ = √ K/α , where α ∈ { 0 . 1 , 0 . 5 , 1 . 0 } in the asynchronous environment, α ∈ [10 -2 , 10 1 ] for synthetic data in the simulated environment and α ∈ [10 0 , 10 1 ] for the zipcode dataset.

Momentum parameter . In the asynchronous environment, we allow β ∈ { 0 , 0 . 1 , 0 . 3 , 0 . 6 } and in the simulated environment, we test β ∈ { 0 . 6 , 0 . 9 } for the synthetic and zipcode data respectively.

Others . Other settings are consistent with Section 6.

H.1.2 Asynchronous environment

0

10

20

30

40

10 -6

10 -4

10 -2

10 0

DSEGD

= 0

DSEPL

= 0

DSEGD

= 0.1

DSEPL

= 0.1

DSEGD

= 0.3

DSEPL

= 0.3

DSEGD

= 0.6

DSEPL

= 0.6

0

10

20

30

40

10 -6

10 -4

10 -2

10 0

DSEGD 1 workers

DSEPL 1 workers

DSEGD 2 workers

DSEPL 2 workers

DSEGD 4 workers

DSEPL 4 workers

DSEGD 8 workers

DSEPL 8 workers

DSEGD 16 workers

DSEPL 16 workers

The first figure plots the effect of different momentum parameters in the distributed environment. It can be seen that both DSEGD and DSEPL perform better with momentum. Moreover, the performance of DSEPL dominate DSEGD as we observed in the previous experiments.

H.1.3 Simulated Environment

10 1

10 2

10 3

0

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

10 1

10 2

10 3

0

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

10 2

10 3

0

0.5

1

1.5

2

2.5

3

3.5

10 5

DSGD

DSEGD

DSPL

DSEPL

10 2

10 3

0

0.5

1

1.5

2

2.5

3

3.5

10 5

DSGD

DSEGD

DSPL

DSEPL

Figure 7 plots the impact of staleness on the number of iterations each algorithm takes to reach the desired accuracy. It can be seen that with other parameters fixed, DSPL tends to be more robust against delays than the pure subgradient-based methods, which is consistent with the theoretical results. Moreover, we observe that when extrapolation is used, the algorithm converges faster at the cost of less robustness as delay increases.

Our last experiment investigates the robustness of DSPL compared to DSGD and justifies the use of extrapolation in presence of delay. Figure 8 plots the number of iterations for each algorithm to converge with different datasets, extrapolation parameters, and delays. In spite of delays, DSPL and DSEPL still admit a wider range of stepsize parameters ensuring convergence than DSGD . Also, when the stepsize is not large, the use of extrapolation can effectively accelerate the convergence.

H.2 Blind deconvolution problem

In this section, we present the additional experiments on blind deconvolution problem to further illustrate the efficiency of DSPL . The blind deconvolution problem, unlike robust phase retrieval,

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 0

10 1

0

0.5

1

1.5

2

2.5

3

10 5

DSGD

DSPL

DSEGD

DSEPL

10 0

10 1

0

0.5

1

1.5

2

2.5

3

10 5

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 0

10 1

0

0.5

1

1.5

2

2.5

3

10 5

DSGD

DSPL

DSEGD

DSEPL

10 0

10 1

0

0.5

1

1.5

2

2.5

3

10 5

DSGD

DSPL

DSEGD

DSEPL

aims to recover two signals from their convolution and its mathematical formulation is given by

where { b i } are the measurements, { ( u i , v i ) } are the measuring data and x, y are the optimization variables corresponding to the signals. Similar to phase retrieval, we first present the detailed setup and then inspect the performance of DSPL in both real and simulated asynchronous environments.

H.2.1 Experiment setup

Data generation. We use synthetic data for blind deconvolution problems.

Synthetic data . We take m = 300 , n = 100 in the experiments of simulated delay and m = 1500 , n = 150 in the asynchronous environment. The data is generated similar to phase retrieval, where, given some conditioning parameter κ ≥ 1 , we compute U = Q 1 D 2 , V = Q 2 D 2 , Q ∈ R m × n , q ij ∼ N (0 , 1) and D = diag ( d ) , d ∈ R n , d i ∈ [1 /κ, 1] , ∀ i . Then two true signals are generated like ˆ x in phase retrieval and we random corruption is applied.

Dataset . In the asynchronous environment, we set κ = 1 , p fail = 0 and in the simulated environment, we set κ = 1 and p fail ∈ { 0 . 2 , 0 . 3 } .

Initial point and radius . We generate x ′ , y ′ ∼ N (0 , I n ) and start from x 0 = x 1 = x ′ ∥ x ′ ∥ , y 0 = y 1 = y ′ ∥ y ′ ∥ . ∆ = 1000 ∥ ( x 0 , y 0 ) ∥ .

Stepsize . We tune the stepsize parameter setting γ = √ K/α , where α ∈ { 0 . 1 , 0 . 5 , 1 . 0 } in the asynchronous environment, α ∈ [10 -2 , 10 1 ] for synthetic data in the simulated environment.

The rest of the experiment setup are consistent with in phase retrieval.

H.2.2 Asynchronous environment

The experiments for blind deconvolution again justifies the effect of DSPL in its convergence behavior and robustness to the stochastic delays. By retaining the smooth structure of the inner composite function, DSPL gives a more accurate approximation of the original objective function only at cost of transmitting one more scaler (i.e., the inner objective value). Hence DSPL serves as a competitive alternative to DSGD when the problem enjoys composite structure (2) in the distributed setting.

4

8

16

3.6718888624413

7.01060403661639

12.2050820986005

3.67609549531383

6.88793136366286

10.2571707328752

0

3.5

7

10.5

14

1

2

SGD

SPL

0

100

200

300

400

1

2

4

8

16

SGD

SPL

0

10

20

30

40

10 -2

10 -1

10 0

DSEGD

= 0

DSEPL

= 0

DSEGD

= 0.1

DSEPL

= 0.1

DSEGD

= 0.3

DSEPL

= 0.3

DSEGD

= 0.6

DSEPL

= 0.6

0

10

20

30

40

10 -6

10 -4

10 -2

10 0

DSEGD 1 workers

DSEPL 1 workers

DSEGD 2 workers

DSEPL 2 workers

DSEGD 4 workers

DSEPL 4 workers

DSEGD 8 workers

DSEPL 8 workers

DSEGD 16 workers

DSEPL 16 workers

1

10 1

10 2

10 3

0

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

10 2

10 3

0

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

10 1

10 2

10 3

0

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

10 1

10 2

10 3

2

4

6

8

10

12

10 4

DSGD

DSEGD

DSPL

DSEPL

H.2.3 Simulated environment

In the simulated experiments for blind deconvolution, we can see similar robustness of DSPL against delay and stepsize selection, which confirms our previous observations.

H.3 Proximal sub-problems

This subsection shows how the sub-problems from DSPL and DSPL are solved for completeness. The results are a direct adaptation from [1] and interested readers can check [1] for the detailed derivations.

Phase retrieval problem In phase retrieval problem, we denote x to be the point where the stochastic model function is constructed and y to be the center of the proximal term. Then the DSPL proximal subproblem is given by

and it admits closed-form solution when away from the boundary

where δ = γ -1 ( ⟨ a, z ⟩ 2 + 2 ⟨ a, z ⟩⟨ a, x -z ⟩ -b ) and ζ = 2 γ -1 ⟨ a, z ⟩ a and Proj [ -1 , 1] ( · ) denotes projection onto the box [ -1 , 1] .

Blind deconvolution problem In the blind deconvolution problem, we use z = ( z x , z y ) to denote the point of model construction and w = ( w x , w y ) to denote the proximal center. Then the subproblem is given by

First we assume that the constraints are inactive, then the solution is available in closed form

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

10 -2

10 -1

10 0

10 1

0

2

4

6

8

10

12

10 4

DSGD

DSPL

DSEGD

DSEPL

where

H.4 Separate figures of Section 6

0

10

20

30

40

Number of epochs

10 -1

10 0

Distance to optimal

DSGD 1 workers

DSGD 2 workers

DSGD 4 workers

DSGD 8 workers

DSGD 16 workers

0

10

20

30

40

Number of epochs

10 -6

10 -4

10 -2

10 0

Distance to optimal

DSPL 1 workers

DSPL 2 workers

DSPL 4 workers

DSPL 8 workers

DSPL 16 workers

0

10

20

30

40

Number of epochs

10 -1

10 0

Function value gap

DSGD 1 workers

DSGD 2 workers

DSGD 4 workers

DSGD 8 workers

DSGD 16 workers

0

10

20

30

40

Number of epochs

10 -6

10 -4

10 -2

10 0

Function value gap

DSPL 1 workers

DSPL 2 workers

DSPL 4 workers

DSPL 8 workers

DSPL 16 workers

References in the appendix

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019.

Damek Davis and Dmitriy Drusvyatskiy. Graphical convergence of subgradients in nonconvex optimization and learning. Mathematics of Operations Research , 47(1):209-231, 2022.

Damek Davis, Dmitriy Drusvyatskiy, and Kellie J MacPhee. Stochastic model-based minimization under high-order growth. arXiv preprint arXiv:1807.00255 , 2018.

Damek Davis and Benjamin Grimmer. Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems. SIAM Journal on Optimization , 29(3):1908-1930, 2019.

Qi Deng and Wenzhi Gao. Minibatch and momentum model-based methods for stochastic weakly convex optimization. Advances in Neural Information Processing Systems , 34, 2021.

Haihao Lu. "relative continuity" for non-lipschitz nonsmooth convex optimization using stochastic (or deterministic) mirror descent. INFORMS Journal on Optimization , 1(4):288303, 2019.

R Tyrrell Rockafellar. Convex analysis princeton university press. Princeton, NJ , 1970.

Siqi Zhang and Niao He. On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization. arXiv preprint arXiv:1806.04781 , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates delayed stochastic optimization problems with composite objective functions. The authors demonstrate that, under weakly convex conditions, the convergence rate of the delayed stochastic subgradient method can be improved from $\\mathcal{O}(\\tau_{\\max}/\\sqrt{K})$ to $\\mathcal{O}(\\bar{\\tau}/\\sqrt{K})$. Additionally, when the stochastic part $f$ in the objective function follows the structure $f = h\\circ c$, the authors propose a delayed stochastic prox-linear method to address the impact of delayed information. Lastly, by utilizing a hard-thresholding technique, the authors effectively manage the magnitude of delay, mitigating the issue of excessive delay compromising the robustness of the original algorithm.\n\nSTRENGTHS:\n1. The paper is written in a clear and easily understandable manner. The language and organization make it accessible to readers.\n\n2. The majority of the proofs presented in the paper are accurate and sound. This demonstrates the authors' proficiency in the mathematical aspects of their work.\n\n3. The authors provide valuable numerical experiments that effectively showcase the strengths and weaknesses of the DSGD and DSPL methods, specifically in terms of their step size selection. This empirical evidence enhances the practical relevance of the research.\n\nWEAKNESSES:\nWeaknesses:\n\n1. The research direction pursued in the paper appears to be relatively narrow, as the authors focus solely on improving the convergence rate under a specific setting. Moreover, the paper lacks substantial evidence to demonstrate the significance of these improvements. It would benefit from a more comprehensive exploration of the broader implications and potential applications of the proposed methods.\n\n2. The paper lacks in-depth explanations of the theoretical results. For instance, the authors fail to clarify why DSGD improves the maximum delay $\\tau_{\\max}$ to $\\bar{\\tau}$ compared to the reference [1]. Similarly, there is a lack of explanation regarding why DSPL can eliminate the impact of delay in objective functions that do not possess the $h\\circ c$ structure. And if there is a minimax lower bound for the latter? It would be helpful to provide further insights and discussions to enhance the readers' understanding.\n\n3. The assumptions made about the objective function predominantly revolve around the uniformity of the randomness $\\xi$. For example, requiring $f(x,\\xi)$ to be weakly convex for any $\\xi$, compared to the less restrictive assumption of $ E_{\\xi} f(x,\\xi) $\nbeing weakly convex in the reference [1]. This raises concerns about whether the improvements in convergence rate are solely attributed to the introduction of stronger assumptions. Additionally, the assumption $\\sup_{g\\in \\partial f(x,\\xi)}\\|g\\| \\le L_f, \\forall \\xi$ may not hold for the robust phase retrieving problem considered in the authors' experiments, where the measuring matrix is sampled by a Gaussian ensemble. These limitations should be acknowledged and discussed to provide a more comprehensive analysis.\n\n[1] Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems.\n\nQUESTIONS:\nMost of the questions are raised in the Weaknesses section. Another question worth considering is whether an experiment can be conducted to demonstrate that the convergence of DSPL is unaffected by the magnitude of delay.\n\nLIMITATIONS:\nThe main limitations of the paper center around its limited scope, incremental contributions, and insufficient exploration of key theoretical aspects, undermining its broader impact and significance in the field.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies distributed nonsmooth optimization with delayed updates. The authors study two kinds of objective functions, the first class is the sum of a weakly-convex nonsmooth function and a computation-friendly nonsmooth regularizer, while the second class is the sum of a composite function with a regularization, where the inner function of the composite function is smooth. For the first class of functions, the paper improves the results in Reference[1] by replacing the maximum delay with the expected delay and the second momentum of the delays. For the second class, they propose a prox-linear algorithm that can further get rid of the delay in the dominant term of the convergence rate. Convincing numerical experiments also demonstrate the effectiveness of the proposed algorithms.\n\nSTRENGTHS:\n* the delay in the dominant term is removed for the composite functions where the inner function is smooth. This is the first result to remove the dependence on the delay in the dominant term for nonsmooth and nonconvex functions.\n\n* for weakly-convex nonsmooth functions, the maximum delay in the convergence rate is improved to the expected delay. \n\n* the presentation and structure of the paper are quite clear.\n\nWEAKNESSES:\n* for prox-linear methods, the inner function is usually vector-valued. I would like to know whether it is possible to extend Algorithm 2 to the cases when the inner function is a mapping while the convergence rate remains invariant.\n\nQUESTIONS:\n* Is it possible to generalize the inner function of the composite function to a mapping while the convergence rate remains invariant. \n\n* I would like to know about the maximum magnitude of numerical errors in the solutions to the subproblem in Algorithm 2 so as to keep the convergence rate invariant.\n\nLIMITATIONS:\nI am not aware of any potential negative social impact of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper presents DSPL, which is an optimization for a certain structured weakly convex problems. This algorithm can work with arbitrarily delayed updates, and outperforms existing algorithms in both theoretical rates and numerical experiments.

STRENGTHS:
1. The algorithm is robust to arbitrary delays. The robust rate doesn't rely on $\tau$.
2. The safeguard is a simple procedure, which makes it more practical.
3. The authors conduct numerical experiments which showcase the effectiveness of proposed algorithms.

WEAKNESSES:
1. The authors compare the proposed algorithm with some non-convex algorithms. Is this paper's setting inherently simpler than nonconvex optimization?

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper studies delayed stochastic algorithms for weakly convex optimization. For a general weakly convex problem, the authors provide a sharper bound that improves the max delay in the previous bounds to "average" delays. For a weakly convex problem with a composition structure, they propose a variant, DSPL, of the delayed SGD (DSGD) by only linearizing the inner function and provide a convergence analysis. Moreover, they propose variants of DSGD and DSPL that are robust to arbitrary delays by checking if the delay is too outdated. Finally, they provide some empirical results to support their theoretical findings.

STRENGTHS:
1. I'm not an expert in this literature but the work seems to be technically solid and make a significant contribution.
2. The paper is very well written with enough intuition and discussions given after each main results. I enjoyed reading it.
3. The theoretical results are strong and supported with empirical evidence

WEAKNESSES:
The experiment section seems to be rushed. I left a few questions below.

QUESTIONS:
1. Could you add labels to the axes in your figures?
2. The right two figures in Figure 1 are impossible to parse given that many curves are very close to each other. I suggest you split each of them into subplots. For example, you could have a subplot showing DSGD 1 worker v.s. DSPL 1 worker, one showing DSGD with different workers, and one showing DSPL with different workers.
3. The two types of delays in Figure 2 are not defined.
4. In Section 6.4 it is claimed that "upon discarding outdated information, DSGD exhibits much greater robustness". How to see that from the plots?
5. The bound of DSPL also has the $\Delta_2$ term. Why is it robust to your adversarial delay? Does that mean your bound could be improved?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-27

comments: SUMMARY:
The authors study a class of weakly convex optimization over a distributed network under the presence of delayed derivatives. They first analyze an existing algorithm and show that they can improve its complexity bound in terms of the dependence on the properties of the delay. They further provide another algorithms and show even a more relaxed bound dependent on the delay.

STRENGTHS:
The authors can improve the complexity bounds in terms of their dependence on the values of delay.

WEAKNESSES:
Assumptions B1 and C1 seems to be much stronger than the ones in the existing works as they have been made on the stochastic functions.

QUESTIONS:
It seems that one of the reasons to get a better dependency of the delay, is to have direct assumption on the stochastic functions and not the expectation ones. The authors need to clarify this.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-27