SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\rho - \kappa \theta}{2 \rho \theta} \mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \\ \leq \frac {\gamma - \kappa}{K} \left\{\psi_ {1 / \rho} \left(z ^ {1}\right) - \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} \left(z ^ {K + 1}\right) \right] \right\} + \frac {\rho \beta}{\theta^ {2} K} \left\{\psi \left(x ^ {k}\right) - \mathbb {E} _ {k} \left[ \psi \left(x ^ {k + 1}\right) \right] \right\} \\ + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 \theta^ {2} K} \| x ^ {1} - x ^ {0} \| ^ {2} + \frac {2 \rho L _ {f} ^ {2} K}{(\gamma - \kappa) \theta^ {2}} + \frac {\rho}{\theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \varepsilon_ {k} ] \\ - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ \leq \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} + \frac {\rho}{\theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \varepsilon_ {k} ] \\ - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \tag {66} \\ \end{array}
$$
where the last inequality uses  $x^0 = x^1$  and  $D = \max \{\psi_{1 / \rho}(z^1) - \psi_{1 / \rho}(z^*),\psi (x^1) - \psi (x^*)\}$ . Now we divide both sides of the inequality by  $\frac{\rho - \kappa\theta}{2\rho\theta}$  to get
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \leq \frac {2 \rho \theta}{\rho - \kappa \theta} \left[ \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} + \frac {\rho}{\theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {k} [ \varepsilon_ {k} ] \right].
$$
Last we bound  $\sum_{k=1}^{K} \mathbb{E}[\varepsilon_k]$ . First, we have the following relations
$$
\sum_ {k = 1} ^ {K} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} \leq \tau^ {2} \sum_ {k = 1} ^ {K} \| x ^ {k} - x ^ {k - 1} \| ^ {2} \leq \tau^ {2} \sum_ {k = 1} ^ {K} \| x ^ {k + 1} - x ^ {k} \| ^ {2},
$$
where the first inequality is by E1 and the second inequality uses the fact that  $x^0 = x^1$ . Similarly
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} \leq \sum_ {k = 1} ^ {K} 2 [ \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} + \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ \leq 2 (\tau^ {2} + 1) \sum_ {k = 1} ^ {K} \| x ^ {k + 1} - x ^ {k} \| ^ {2}, \\ \end{array}
$$
where the first inequality uses  $\|a + b\|^2 \leq 2\|a\|^2 + 2\|b\|^2$  and the second inequality reuses the bound of the first term. Plugging the above bounds back into  $\sum_{k=1}^{K} \mathbb{E}[\varepsilon_k]$ , we successively deduce
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} [ \varepsilon_ {k} ] = \sum_ {k = 1} ^ {K} \frac {\lambda (1 - \theta)}{2} \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} + \sum_ {k = 1} ^ {K} (\theta + 1 / 2) \lambda \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ \leq \lambda [ (1 - \theta) (\tau^ {2} + 1) + (\theta + 1 / 2) \tau^ {2} ] \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ = \lambda (3 \tau^ {2} / 2 + \beta) \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ], \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \\ \leq \frac {2 \rho \theta}{\rho - \kappa \theta} \left[ \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} + \frac {\rho \lambda (3 \tau^ {2} + 2 \beta)}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \right], \tag {67} \\ \end{array}
$$
and this completes the proof.
Proof of Theorem 8
By Lemma 11, it remains to bound the quantity  $\sum_{k=1}^{K} \mathbb{E}[||x^{k+1} - x^k ||^2]$ . To this end we consider the relation in (66), where we have
$$
\begin{array}{l} \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1} - 3 \lambda \tau^ {2} - 2 \lambda \beta)}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ \leq \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}}. \\ \end{array}
$$
Since we choose  $\beta, \gamma$  such that  $\gamma \theta^2 - 2\theta (\rho + \kappa \beta) - 2\rho \beta^2\theta^{-1} - 3\lambda \tau^2 - 2\lambda \beta > 0$ , then
$$
\frac {\rho \lambda (3 \tau^ {2} + 2 \beta)}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \leq \frac {\lambda (3 \tau^ {2} + 2 \beta) \left\{\frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} \right\}}{\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1} - 3 \lambda \tau^ {2} - 2 \lambda \beta}.
$$
Plugging the bound back, we have
$$
\begin{array}{l} \mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \\ \leq \frac {2 \rho \theta}{\rho - \kappa \theta} \left[ \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} + \frac {\lambda (3 \tau^ {2} + 2 \beta) \left\{\frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} \right\}}{\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1} - 3 \lambda \tau^ {2} - 2 \lambda \beta} \right]. \\ \end{array}
$$
This completes our proof.
G Kernel and subproblems
In this section, we delve into the practical considerations related to Bregman proximal methods. In particular, we discuss how to construct suitable divergence kernels for DSPL so that we can accommodate non-Lipschitzness of the objective function. We also propose efficient subroutines for solving Bregman proximal subproblems, which are applicable to a wide range of loss functions typically encountered in machine learning tasks.
G.1 Choosing the divergence kernel
The selection of the kernel function is a critical aspect of the Bregman proximal method. The choice of kernel has been widely discussed in the literature [3, 6], and we incorporate these findings to construct suitable kernels for the DSPL method. Given the assumptions we have made throughout the paper, it suffices to consider kernels  $d(x) = \mathcal{P}_n^p (\| x\|)\coloneqq \sum_{k = 0}^n p_k\| x\| ^k$ , which are degree- $n$  polynomials in  $\| x\|$ . We recall the following lemma and refer the readers to [3] for its proof.
Lemma 12. Given  $f(x)$  such that  $\frac{f(x) - f(y)}{\|x - y\|} \leq \mathcal{P}_n^p(\|x\|) + \mathcal{P}_n^p(\|y\|) = \sum_{k=0}^{n} p_k(\|x\|^k + \|y\|^2), p \geq 0, \forall x \in \operatorname{dom} h$ , then  $f$  satisfies 1-relative Lipschitzian property with kernel  $d(x) = \sum_{k=0}^{n} \frac{3k+7}{k+2} p_k \|x\|^{k+2}$ .
The above lemma implies that once we establish a polynomial upper bound on the Lipschitzness of  $f_{z}(x,\xi) + \omega (x)$ , a kernel  $d$  is immediately available.
We note that the above kernel covers most of the applications [2] of the prox-linear method, including phase retrieval, blind deconvolution, matrix completion, covariance estimation, robust PCA, and so on. In many machine learning applications,  $c$  has Lipschitz continuous gradient, meaning that  $\nabla c$  is bounded by first-order growth. For example, in the phase retrieval problem, we have  $\omega = 0, h(x) = |x|, c(x,\xi) = \langle a,x\rangle^2 - b$ . Hence, we get
$$
\| \nabla c (z, \xi) \| = 2 | \langle a, z \rangle | \cdot \| a \| \leq 2 \| a \| ^ {2} \| z \| \text {a n d} \frac {f _ {z} (x , \xi) - f _ {z} (y , \xi)}{\| x - y \|} \leq 2 \| a \| ^ {2} \| z \|.
$$
Now assume that  $z = x$ , then for any  $p_0 \geq 0$  and  $p_1 = 2\alpha^2, \alpha = \sup_{a \sim \Xi} \|a\|$  we know that  $2\|a\|^2 \|x\| \leq p_0 + p_1 \|x\|$ . Hence, we can take  $d(x) = \frac{7}{2}\|x\|^2 + \frac{10\alpha^2}{3}\|x\|^3$  as our kernel. Recall that in DSPL  $z = x^{k - \tau_k}, x = x^k$ , as long as there exist some  $M, N$  such that  $\|x^j\| \leq M\|x^k\| + N, k \geq j$  for all  $k$ , our assumptions are satisfied.
Now that we have specified ways to construct kernels for DSPL, we discuss how to solve the Bregman proximal subproblems in the next section. The subproblems' solution determines the practical applicability of our methods, and we show that these subproblems can be efficiently solved for a wide range of problems.
G.2 Bregman proximal subproblem
In this section, we discuss the solution of the proximal subproblems for DSPL when  $\omega = 0$ , where the Bregman proximal subproblem can then be expressed as
$$
\min  _ {x} h (\langle a, x \rangle + b) + V _ {d} (x, y). \tag {68}
$$
Consider a piece-wise linear  $h(x) = \max \{\alpha_1x + \beta_1, \alpha_2x + \beta_2\}$ . This formulation characterizes common nonsmooth loss functions including  $\ell_1$  loss  $|x| = \max \{x, -x\}$  and hinge loss  $\max \{0, x\}$ . Substituting  $h$  in and removing terms irrelevant to  $x$ , we arrive at the following DSPL subproblem.
$$
\min  _ {x} \max  \left\{\langle a _ {1}, x \rangle + b _ {1}, \langle a _ {2}, x \rangle + b _ {2} \right\} + d (x) \tag {69}
$$
The next proposition provides a solution to the above subproblem.
Proposition 3 (Solving the proximal subproblem). The proximal subproblem can be solved by evaluating solutions to the following three problems
$$
\min _ {x} \left\langle a _ {1}, x \right\rangle + d (x),
$$
$$
\min  _ {x} \left\langle a _ {2}, x \right\rangle + d (x),
$$
$$
\min  _ {x} \qquad \qquad \langle a _ {1}, x \rangle + d (x)
$$
$$
\text {s u b j e c t} \quad \left\langle a _ {1} - a _ {2}, x \right\rangle + b _ {1} - b _ {2} = 0,
$$
where the solution to the third problem satisfies  $x = u + \alpha v$ , where  $u = -\frac{(a_1 - a_2)(b_1 - b_2)}{\|a_1 - a_2\|^2}$ ,  $v = \frac{(\|a_1\|^2 - \|a_2\|^2)}{2\|a_1 - a_2\|^2} (a_1 - a_2) - \frac{a_1 + a_2}{2}$  and  $\alpha$  is a positive root of the following equation
$$
\sum_ {k = 0} ^ {n} p _ {k} \alpha \| u + \alpha v \| ^ {k} - 1 = 0.
$$
Proof. Recall that we are solving the following convex optimization problem
$$
\min _ {x} \max  \{\langle a _ {1}, x \rangle + b _ {1}, \langle a _ {2}, x \rangle + b _ {2} \} + d (x),
$$
where  $d$  is strongly convex and thus the problem admits a unique minimizer  $x^{*}$ . Then we do case analysis.
Case 1.  $\langle a_1, x^* \rangle + b_1 > \langle a_2, x^* \rangle + b_2$ . In this case solving  $\min_x \langle a_1, x \rangle + d(x)$  produces  $x^*$ .
Case 2.  $\langle a_1, x^* \rangle + b_1 < \langle a_2, x^* \rangle + b_2$ . In this case solving  $\min_x \langle a_2, x \rangle + d(x)$  produces  $x^*$ .
The above two cases degenerate into the subproblems from stochastic mirror descent and its solution has been discussed in [6]. The last case is a bit more tricky.
Case 3.  $\langle a_1, x^* \rangle + b_1 = \langle a_2, x^* \rangle + b_2$ .
In this case, the proximal subproblem becomes an equality constrained problem.
$$
\text {s u b j e c t} \quad \left\langle a _ {1} - a _ {2}, x \right\rangle + b _ {1} - b _ {2} = 0
$$
Letting  $\lambda$  be the multiplier of the equality constraint and appealing to the optimality condition, we have
$$
a _ {1} + \nabla d (x) + \lambda \left(a _ {1} - a _ {2}\right) = 0 \tag {70}
$$
$$
a _ {2} + \nabla d (x) + \lambda \left(a _ {1} - a _ {2}\right) = 0 \tag {71}
$$
$$
\langle a _ {1} - a _ {2}, x \rangle + b _ {1} - b _ {2} = 0. \tag {72}
$$
Summing (70) (71) and dividing both sides by 2, we have
$$
\frac {a _ {1} + a _ {2}}{2} + \nabla d (x) + \lambda \left(a _ {1} - a _ {2}\right) = 0. \tag {73}
$$
Multiplying both sides of (73) by  $a_1 - a_2$ , we arrive at
$$
\frac {\left\| a _ {1} \right\| ^ {2} - \left\| a _ {2} \right\| ^ {2}}{2} + \langle a _ {1} - a _ {2}, \nabla d (x) \rangle + \lambda \| a _ {1} - a _ {2} \| ^ {2} = 0. \tag {74}
$$
Since  $d(x)$  is a polynomial in  $\| x\|$ ,  $\nabla d(x) = \zeta x, \zeta > 0$  and using (72), we have
$$
\lambda = \frac {b _ {1} - b _ {2}}{\| a _ {1} - a _ {2} \| ^ {2}} \zeta - \frac {\| a _ {1} \| ^ {2} - \| a _ {2} \| ^ {2}}{2 \| a _ {1} - a _ {2} \| ^ {2}}. \tag {75}
$$
Then substituting  $\lambda$  into (74),
$$
\begin{array}{l} 0 = \frac {a _ {1} + a _ {2}}{2} + \nabla d (x) + \lambda \left(a _ {1} - a _ {2}\right) \\ = \frac {a _ {1} + a _ {2}}{2} + \zeta x + \zeta \frac {(a _ {1} - a _ {2}) (b _ {1} - b _ {2})}{\| a _ {1} - a _ {2} \| ^ {2}} - \frac {\| a _ {1} \| ^ {2} - \| a _ {2} \| ^ {2}}{2 \| a _ {1} - a _ {2} \| ^ {2}} \big (a _ {1} - a _ {2} \big) \\ = \zeta \left[ x + \frac {\left(a _ {1} - a _ {2}\right) \left(b _ {1} - b _ {2}\right)}{\| a _ {1} - a _ {2} \| ^ {2}} \right] - \frac {\| a _ {1} \| ^ {2} - \| a _ {2} \| ^ {2}}{2 \| a _ {1} - a _ {2} \| ^ {2}} \left(a _ {1} - a _ {2}\right) + \frac {a _ {1} + a _ {2}}{2} \\ \end{array}
$$
$$
x = \zeta^ {- 1} \left[ \frac {(\| a _ {1} \| ^ {2} - \| a _ {2} \| ^ {2})}{2 \| a _ {1} - a _ {2} \| ^ {2}} (a _ {1} - a _ {2}) - \frac {a _ {1} + a _ {2}}{2} \right] - \frac {(a _ {1} - a _ {2}) (b _ {1} - b _ {2})}{\| a _ {1} - a _ {2} \| ^ {2}}
$$
for some  $\zeta > 0$ . Without loss of generality, we express  $x = u + \alpha v$  for  $u = -\frac{(a_1 - a_2)(b_1 - b_2)}{\|a_1 - a_2\|^2}$  and  $v = \frac{(\|a_1\|^2 - \|a_2\|^2)}{2\|a_1 - a_2\|^2} (a_1 - a_2) - \frac{a_1 + a_2}{2}$  and  $\alpha > 0$ . Substituting back gives  $\zeta = \sum_{k=0}^{n} p_k \|x\|^k$  and
$$
\| \zeta (x - u) - v \| = \| \alpha \zeta v - v \| = | \alpha \zeta - 1 | \cdot \| v \| = 0.
$$
Therefore, we only need to verify the positive roots of  $\sum_{k=0}^{n} p_k \alpha \| u + \alpha v \|^k - 1 = 0$ , which can be done efficiently if  $n$  is not large. This completes the proof.
G.3 Euclidean subproblem with nonsmooth regularizer
In this section, we discuss the solution of the proximal subproblems for DSPL when  $d(x) = \frac{1}{2} \| x\|^2$  and  $\omega$  takes on one of two forms: 1). Indicator function of ball  $\delta_{\{\| x\| \leq R\}}$ . 2).  $\ell_1$  regularizer  $\| x\| _1$ . We still assume  $h(x) = \max \{\alpha_1x + \beta_1,\alpha_2x + \beta_2\}$ , which covers all the popular applications of SPL from [2, 4]. Following the same argument as in the previous section, each DSPL iteration is reduced to
$$
\min _ {x} \max \{\langle a _ {1}, x \rangle + b _ {1}, \langle a _ {2}, x \rangle + b _ {2} \} + \omega (x) + \frac {1}{2} \| x - y \| ^ {2}
$$
and after doing case analysis on  $\langle a_1, x^* \rangle + b_1$  and  $\langle a_2, x^* \rangle + b_2$  as in Proposition 3, it suffices to solve
$$
\min _ {x} \qquad \omega (x) + \frac {1}{2} \| x - y \| ^ {2}
$$
$$
\text {s u b j e c t} \quad \langle a, x \rangle - b = 0
$$
Case 1.  $\omega (x) = \delta_{\{\| x\| \leq R\}}$  . In this case, we need to solve
$$
\min  _ {x} \quad \frac {1}{2} \| x - y \| ^ {2}
$$
subject to  $\langle a,x\rangle = b$
$$
\left\| x \right\| ^ {2} \leq r = R ^ {2},
$$
whose KKT condition is given by
$$
\langle a, x \rangle = b
$$
$$
\lambda \geq 0
$$
$$
\lambda (\| x \| ^ {2} - r) = 0
$$
$$
x - y + \nu a + 2 \lambda x = 0
$$
After simplification, we arrive at
$$
x = \frac {y}{1 + 2 \lambda} - \frac {\langle a , y \rangle a - (1 + 2 \lambda) b a}{(1 + 2 \lambda) \| a \| ^ {2}}.
$$
Then either  $\lambda = 0$  or the solution can be obtained using bisection.
Case 2.  $\omega (x) = \| x\| _1$  . In this case, we need to solve
$$
\min  _ {x} \quad \| x \| _ {1} + \frac {1}{2} \| x - y \| ^ {2}
$$
Given the Lagrangian multiplier  $\nu$  associated with the equality constraint, we have that  $x(\nu) = S(y - \nu a)$ , which is given by the soft-thresholding operator. The residual of the equality constraint is given by:
$$
f (\nu) = \langle a, x \rangle - b = \langle a, x (\nu) \rangle - b.
$$
This function is univariate in  $\nu$  and is semi-smooth. Therefore, semi-smooth Newton method can efficiently find its root. This makes the computation tractable.
H Additional experiments
In this section, we present additional experiments incorporating momentum. Our experiments focus on both robust phase retrieval and blind deconvolution problems.
H.1 Robust phase retrieval
We refer readers to Section 6 for the detailed formulation of phase retrieval. Most of the experiment settings in this section are consistent with Section 6.
H.1.1 Experiment setup
1) Initial point and radius. For the synthetic data, we generate  $x' \sim \mathcal{N}(0, I_n)$  and start from  $x^0 = x^1 = \frac{x'}{\|x'\|}$  and for zipcode data, we generate  $x' \sim \mathcal{N}(\hat{x}, I_n)$  and take  $x^0 = x^1 = x'$ .  $M = 1000 \| x^0 \|$ .
2) Stepsize. We tune the stepsize parameter setting  $\gamma = \sqrt{K / \alpha}$ , where  $\alpha \in \{0.1, 0.5, 1.0\}$  in the asynchronous environment,  $\alpha \in [10^{-2}, 10^{1}]$  for synthetic data in the simulated environment and  $\alpha \in [10^{0}, 10^{1}]$  for the zipcode dataset.
3) Momentum parameter. In the asynchronous environment, we allow  $\beta \in \{0, 0.1, 0.3, 0.6\}$  and in the simulated environment, we test  $\beta \in \{0.6, 0.9\}$  for the synthetic and zipcode data respectively.
4) Others. Other settings are consistent with Section 6.
H.1.2 Asynchronous environment
Figure 6: First: speedup in time and the number of workers. Second: progress of  $\| x^k - \hat{x}\|$  in the first 40 epochs given 16 workers and  $\alpha = 0.5$ . Third: progress of  $f(x^{k}) - f(\hat{x})$  in the first 40 epochs given  $\beta = 0.3$  and different number of workers.
The first figure plots the effect of different momentum parameters in the distributed environment. It can be seen that both DSEGD and DSEPL perform better with momentum. Moreover, the performance of DSEPL dominate DSEGD as we observed in the previous experiments.
H.1.3 Simulated Environment
Figure 7: Left to right:  $(\kappa, p_{\mathrm{fail}}) = (10, 0.3), \alpha = 5.0$ , Geometric and Poisson delays; zipcode data of  $p_{\mathrm{fail}} = 0.2, \alpha = 6.0$ , Geometric and Poisson delays. x-axis represents  $\tau_{\mathrm{max}}$  and y-axis shows average iteration number to reach the stopping criterion over 20 tests.
Figure 7 plots the impact of staleness on the number of iterations each algorithm takes to reach the desired accuracy. It can be seen that with other parameters fixed, DSPL tends to be more robust against delays than the pure subgradient-based methods, which is consistent with the theoretical results. Moreover, we observe that when extrapolation is used, the algorithm converges faster at the cost of less robustness as delay increases.
Our last experiment investigates the robustness of DSPL compared to DSGD and justifies the use of extrapolation in presence of delay. Figure 8 plots the number of iterations for each algorithm to converge with different datasets, extrapolation parameters, and delays. In spite of delays, DSPL and DSEPL still admit a wider range of stepsize parameters ensuring convergence than DSGD. Also, when the stepsize is not large, the use of extrapolation can effectively accelerate the convergence.
H.2 Blind deconvolution problem
In this section, we present the additional experiments on blind deconvolution problem to further illustrate the efficiency of DSPL. The blind deconvolution problem, unlike robust phase retrieval,
Figure 8: First row(Geometric) left:  $(\kappa, p_{\mathrm{fail}}) = (1,0.3), \beta = 0.6, \tau \in \{28,47\}$ , right: zipcode  $p_{\mathrm{fail}} = 0.3, \beta = 0.9, \tau \in \{28,600\}$ . Second row(Poisson) left:  $(\kappa, p_{\mathrm{fail}}) = (10,0.2), \beta = 0.6, \tau \in \{28,47\}$ . right: zipcode  $p_{\mathrm{fail}} = 0.3, \beta = 0.9, \tau \in \{28,600\}$ .
aims to recover two signals from their convolution and its mathematical formulation is given by
$$
\min  _ {x, y \in \mathbb {R} ^ {n}} \frac {1}{m} \sum_ {i = 1} ^ {m} | \langle u _ {i}, x \rangle \langle v _ {i}, y \rangle - b _ {i} | + \mathbb {I} \{\| x \|, \| y \| \leq \Delta \},
$$
where  $\{b_i\}$  are the measurements,  $\{(u_i,v_i)\}$  are the measuring data and  $x,y$  are the optimization variables corresponding to the signals. Similar to phase retrieval, we first present the detailed setup and then inspect the performance of DSPL in both real and simulated asynchronous environments.
H.2.1 Experiment setup
Data generation. We use synthetic data for blind deconvolution problems.
Synthetic data. We take  $m = 300, n = 100$  in the experiments of simulated delay and  $m = 1500, n = 150$  in the asynchronous environment. The data is generated similar to phase retrieval, where, given some conditioning parameter  $\kappa \geq 1$ , we compute  $U = Q_{1}D_{2}, V = Q_{2}D_{2}, Q \in \mathbb{R}^{m \times n}, q_{ij} \sim \mathcal{N}(0,1)$  and  $D = \mathrm{diag}(d), d \in \mathbb{R}^n, d_i \in [1 / \kappa, 1], \forall i$ . Then two true signals are generated like  $\hat{x}$  in phase retrieval and we random corruption is applied.
1) Dataset. In the asynchronous environment, we set  $\kappa = 1$ ,  $p_{\mathrm{fail}} = 0$  and in the simulated environment, we set  $\kappa = 1$  and  $p_{\mathrm{fail}} \in \{0.2, 0.3\}$ .
2) Initial point and radius. We generate  $x', y' \sim \mathcal{N}(0, I_n)$  and start from  $x^0 = x^1 = \frac{x'}{\|x'\|}$ ,  $y^0 = y^1 = \frac{y'}{\|y'\|}$ .  $\Delta = 1000 \| (x^0, y^0) \|$ .
3) Stepsize. We tune the stepsize parameter setting  $\gamma = \sqrt{K / \alpha}$ , where  $\alpha \in \{0.1, 0.5, 1.0\}$  in the asynchronous environment,  $\alpha \in [10^{-2}, 10^{1}]$  for synthetic data in the simulated environment.
The rest of the experiment setup are consistent with in phase retrieval.
H.2.2 Asynchronous environment
The experiments for blind deconvolution again justifies the effect of DSPL in its convergence behavior and robustness to the stochastic delays. By retaining the smooth structure of the inner composite function, DSPL gives a more accurate approximation of the original objective function only at cost of transmitting one more scalar (i.e., the inner objective value). Hence DSPL serves as a competitive alternative to DSGD when the problem enjoys composite structure (2) in the distributed setting.
Figure 9: First: speedup in time and the number of workers. Second: progress of  $\|(x^k, y^k) - (\hat{x}, \hat{y})\|$  in the first 40 epochs given 16 workers and  $\alpha = 0.5$ . Third: progress of  $f(x^k, y^k) - f(\hat{x}, \hat{y})$  in the first 40 epochs given  $\beta = 0.1$  and different number of workers.
Figure 10: From left to right:  $(\kappa, p_{\mathrm{fail}}) = (1, 0.2), \alpha = 4.6$ , Geometric and Poisson delays;  $(\kappa, p_{\mathrm{fail}}) = (1, 0.3), \alpha = 2.1$ , Geometric and Poisson delays. The x-axis represents delay  $\tau$  and the y-axis gives the number of iterations to reach the stopping criterion.
H.2.3 Simulated environment
In the simulated experiments for blind deconvolution, we can see similar robustness of DSPL against delay and stepsize selection, which confirms our previous observations.
H.3 Proximal sub-problems
This subsection shows how the sub-problems from DSPL and DSPL are solved for completeness. The results are a direct adaptation from [1] and interested readers can check [1] for the detailed derivations.
Phase retrieval problem In phase retrieval problem, we denote  $x$  to be the point where the stochastic model function is constructed and  $y$  to be the center of the proximal term. Then the DSPL proximal subproblem is given by
$$
\min  _ {x} \left| \langle a, z \rangle^ {2} + 2 \langle a, z \rangle \langle a, x - z \rangle - b \right| + \frac {\gamma}{2} \| x - y \| ^ {2}
$$
and it admits closed-form solution when away from the boundary
$$
x ^ {+} = y + \operatorname {P r o j} _ {[ - 1, 1 ]} \left(- \frac {\delta}{\| \zeta \| ^ {2}}\right) \zeta ,
$$
where  $\delta = \gamma^{-1}(\langle a,z\rangle^2 +2\langle a,z\rangle \langle a,x - z\rangle -b)$  and  $\zeta = 2\gamma^{-1}\langle a,z\rangle a$  and  $\mathrm{Proj}_{[-1,1]}(\cdot)$  denotes projection onto the box  $[-1,1]$ .
Blind deconvolution problem In the blind deconvolution problem, we use  $z = (z_{x},z_{y})$  to denote the point of model construction and  $w = (w_{x},w_{y})$  to denote the proximal center. Then the subproblem is given by