SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left. \left| \mathbb {E} _ {\xi^ {\prime}} \left[ \mathbb {E} _ {\xi} \left[ f _ {z} \left(\mathcal {A} (z, x, \xi^ {\prime}), \xi\right) - f _ {z} \left(\mathcal {A} (z, x, \xi^ {\prime}), \xi^ {\prime}\right) \right] \right] \right| \right. \\ = \left| \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi) - f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi^ {\prime}) d \mu_ {\xi} d \mu_ {\xi^ {\prime}} \right| \\ = \left| \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi) d \mu_ {\xi} d \mu_ {\xi^ {\prime}} - \int_ {\xi^ {\prime} \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi^ {\prime}) d \mu_ {\xi^ {\prime}} \right| \\ = \left| \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi) d \mu_ {\xi} d \mu_ {\xi^ {\prime}} - \int_ {\xi \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi), \xi) d \mu_ {\xi} \right| \\ = \left| \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi) - f _ {z} (\mathcal {A} (z, x, \xi), \xi) d \mu_ {\xi} d \mu_ {\xi^ {\prime}} \right| \\ \leq \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} \left| f _ {z} \left(\mathcal {A} (z, x, \xi^ {\prime}), \xi\right) - f _ {z} \left(\mathcal {A} (z, x, \xi), \xi\right) \right| d \mu_ {\xi} d \mu_ {\xi^ {\prime}} (29) \\ \leq \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} L _ {f} \cdot \max  \left\{\sqrt {V _ {d} (\mathcal {A} (z , x , \xi) , \mathcal {A} (z , x , \xi^ {\prime}))}, \right. (30) \\ \left. \sqrt {V _ {d} \left(\mathcal {A} (z , x , \xi^ {\prime}) , \mathcal {A} (z , x , \xi)\right)} \right\} d \mu_ {\xi} d \mu_ {\xi^ {\prime}} (31) \\ \leq \int_ {\xi^ {\prime} \sim \Xi} \int_ {\xi \sim \Xi} \frac {2 L _ {f} ^ {2}}{\gamma - \kappa} d \mu_ {\xi} d \mu_ {\xi^ {\prime}} = \frac {2 L _ {f} ^ {2}}{\gamma - \kappa}, (32) \\ \end{array}
$$
where the third equality holds since  $\xi$  and  $\xi^{\prime}$  are from the same distribution; (29) follows from Jensen's inequality and (32) uses (28). Plugging in  $z = x^{k - \tau_k},x = z^k,\xi ' = \xi^{k - \tau_k}$  and  $\gamma = \gamma_{k}$  into (32) completes the proof.
Proof of Lemma 9
First, we have, by the three-point lemma and optimality of  $\hat{x}^k$ , that
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} (x ^ {k + 1}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k + 1}) + \gamma_ {k} V _ {d} (x ^ {k + 1}, x ^ {k}) \\ \leq f _ {x ^ {k} - \tau_ {k}} \left(\hat {x} ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(\hat {x} ^ {k}\right) + \gamma_ {k} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - (\gamma_ {k} - \kappa) V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \\ \end{array}
$$
$$
f \left(\hat {x} ^ {k}\right) + \omega \left(\hat {x} ^ {k}\right) + \rho V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) \leq f \left(x ^ {k + 1}\right) + \omega \left(x ^ {k + 1}\right) + \rho V _ {d} \left(x ^ {k + 1}, x ^ {k}\right)
$$
Summing the above two relations and taking expectations, we have
$$
\begin{array}{l} (\gamma_ {k} - \rho) \mathbb {E} _ {k} \left[ V _ {d} \left(x ^ {k + 1}, x ^ {k}\right) \right] - (\gamma_ {k} - \rho) V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) + (\gamma_ {k} - \kappa) \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] \\ \leq \mathbb {E} _ {\xi^ {k - \tau_ {k}}} \left[ f _ {x ^ {k - \tau_ {k}}} \left(\hat {x} ^ {k}, \xi^ {k - \tau_ {k}}\right) \right] - f (\hat {x} ^ {k}) + \mathbb {E} _ {k} \left[ f \left(x ^ {k + 1}\right) \right] - \mathbb {E} _ {k} \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) \right] \\ = \mathbb {E} _ {\xi^ {k - \tau_ {k}}} \left[ f _ {x ^ {k - \tau_ {k}}} \left(\hat {x} ^ {k}, \xi^ {k - \tau_ {k}}\right) \right] - f (\hat {x} ^ {k}) + \mathbb {E} _ {k} \left[ f (x ^ {k + 1}) \right] - \mathbb {E} _ {k} \left[ \mathbb {E} _ {\xi} \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi\right) \right] \right] \\ + \mathbb {E} _ {k} \left[ \mathbb {E} _ {\xi} \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi\right) \right] \right] - \mathbb {E} _ {k} \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) \right] \\ \leq \frac {\lambda}{2} \| x ^ {k - \tau_ {k}} - \hat {x} ^ {k} \| ^ {2} + \frac {\lambda}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 L _ {f} ^ {2}}{\gamma_ {k} - \kappa}, \tag {33} \\ \end{array}
$$
where (33) uses P2 and Lemma 8. Next we lower-bound the left-hand side using the 1-strong convexity of kernel  $d$
$$
\begin{array}{l} \frac {\gamma_ {k} - \rho}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] - (\gamma_ {k} - \rho) V _ {d} (\hat {x} ^ {k}, x ^ {k}) + (\gamma_ {k} - \kappa) \mathbb {E} _ {k} [ V _ {d} (\hat {x} ^ {k}, x ^ {k + 1}) ] \\ \leq \left(\gamma_ {k} - \rho\right) \mathbb {E} _ {k} \left[ V _ {d} \left(x ^ {k + 1}, x ^ {k}\right) \right] - \left(\gamma_ {k} - \rho\right) V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) + \left(\gamma_ {k} - \kappa\right) \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right]. \tag {34} \\ \end{array}
$$
Re-arranging the terms,
$$
\begin{array}{l} \left(\gamma_ {k} - \kappa\right) \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] + \frac {\gamma_ {k} - \rho}{2} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] (35) \\ \leq (\gamma_ {k} - \rho) V _ {d} (\hat {x} ^ {k}, x ^ {k}) + \frac {\lambda}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {\lambda}{2} \| x ^ {k - \tau_ {k}} - \hat {x} ^ {k} \| ^ {2} + \frac {2 L _ {f} ^ {2}}{\gamma_ {k} - \kappa} \\ \leq \left(\gamma_ {k} - \rho\right) V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) + \frac {3 \lambda}{2} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} \right] + \lambda \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - \hat {x} ^ {k} \right\| ^ {2} \right] + \frac {2 L _ {f} ^ {2}}{\gamma_ {k} - \kappa} (36) \\ = (\gamma_ {k} - \rho) V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) + \frac {3 \lambda}{2} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} \right] + \frac {2 L _ {f} ^ {2}}{\gamma_ {k} - \kappa} + 2 \lambda \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] \\ + \lambda \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - \hat {x} ^ {k} \| ^ {2} - 2 V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] \\ \leq \left(\gamma_ {k} - \rho\right) V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) + \frac {3 \lambda}{2} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} \right] + \frac {2 L _ {f} ^ {2}}{\gamma_ {k} - \kappa} + 2 \lambda \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] (37) \\ \end{array}
$$
where the (36) uses  $\|a + b\|^2 \leq 2\|a\|^2 + 2\|b\|^2$  and (37) again follows by  $V_d(\hat{x}^k, x^{k+1}) \geq \frac{1}{2}\|x^{k+1} - \hat{x}^k\|^2$ . Now we re-arrange the terms and divide both sides by  $\gamma_k - 2\lambda - \kappa$  to get
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] \\ \leq \frac {\gamma_ {k} - \rho}{\gamma_ {k} - 2 \lambda - \kappa} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\gamma_ {k} - \rho}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \\ + \frac {2 L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} + \frac {3 \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ = V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\rho - 2 \lambda - \kappa}{\gamma_ {k} - 2 \lambda - \kappa} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\gamma_ {k} - \rho}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \\ + \frac {2 L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} + \frac {3 \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \tag {38} \\ \end{array}
$$
Now we are ready to evaluate the decrease in the potential function.
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} ^ {d} \left(x ^ {k + 1}\right) \right] \\ = \mathbb {E} _ {k} [ f (\hat {x} ^ {k + 1}) + \omega (\hat {x} ^ {k + 1}) + \rho V _ {d} (\hat {x} ^ {k + 1}, x ^ {k + 1}) ] \\ \leq \mathbb {E} _ {k} \left[ f \left(\hat {x} ^ {k}\right) + \omega \left(\hat {x} ^ {k}\right) + \rho V _ {d} \left(\hat {x} ^ {k}, x ^ {k + 1}\right) \right] \\ \leq f \left(\hat {x} ^ {k}\right) + \omega \left(\hat {x} ^ {k}\right) + \rho V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\rho (\rho - 2 \lambda - \kappa)}{\gamma_ {k} - 2 \lambda - \kappa} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) \tag {39} \\ + \frac {3 \rho \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} \\ - \frac {\rho (\gamma_ {k} - \rho)}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ = \psi_ {1 / \rho} ^ {d} \left(x ^ {k}\right) - \frac {\rho (\rho - 2 \lambda - \kappa)}{\gamma_ {k} - 2 \lambda - \kappa} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\rho \left(\gamma_ {k} - \rho\right)}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {3 \rho \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)}, \\ \end{array}
$$
where (39) plugs in the relation from (38). Another rearrangement completes the proof.
Proof of Theorem 6
Summing the relation from Lemma 9 from  $k = 1,\dots ,K$  and multiplying both sides by  $\gamma -2\lambda -\kappa$
$$
\begin{array}{l} \rho (\rho - 2 \lambda - \kappa) \mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k ^ {*}}, x ^ {k ^ {*}}\right) \right] \\ = \frac {\rho (\rho - 2 \lambda - \kappa)}{K} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) \right] (40) \\ \leq \frac {\gamma - 2 \lambda - \kappa}{K} \bigl \{\psi_ {1 / \rho} ^ {d} (x ^ {1}) - \mathbb {E} [ \psi_ {1 / \rho} ^ {d} (x ^ {K + 1}) ] \bigr \} + \frac {2 \rho L _ {f} ^ {2}}{\gamma - \kappa} \\ + \frac {3 \rho \lambda}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} - \frac {\rho (\gamma - \rho)}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ \leq \frac {(\gamma - 2 \lambda - \kappa) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{\gamma - \kappa} + \frac {\rho}{K} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ - \frac {\gamma - \rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \frac {3 \lambda}{2} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} \right], (41) \\ \end{array}
$$
where (41) uses the relation  $\psi_{1 / \rho}^{d}(x^{K + 1})\geq \inf_{x}\psi^{d}(x)$ . Now it remains to bound the error of delays and recall that we have
$$
\sum_ {k = 1} ^ {K} \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} \leq 2 \sum_ {k = 1} ^ {K} \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} + 4 \gamma^ {- 2} \left(L _ {f} + L _ {\omega}\right) ^ {2} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} \tag {42}
$$
and plugging the bounds back, we have
$$
\begin{array}{l} \rho (\rho - 2 \lambda - \kappa) \mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k ^ {*}}, x ^ {k ^ {*}}\right) \right] \\ \leq \frac {(\gamma - 2 \lambda - \kappa) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{\gamma - \kappa} + \frac {6 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2}}{\gamma^ {2} K} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} \\ \leq \frac {(\rho + 4 \lambda) D}{K} + \frac {D}{\sqrt {K} \alpha} + \frac {2 \rho L _ {f} ^ {2} \alpha}{\sqrt {K}} + \frac {6 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K} \Delta_ {2}, \tag {43} \\ \end{array}
$$
where in (43) we cancel the first summation from (42) since  $\gamma -\rho = \sqrt{K} /\alpha +6\lambda +\kappa \geq 6\lambda$ . Finally we divide both sides by  $\rho (\rho -2\lambda -\kappa)$  to complete the proof.
D.2 Proof of Lemma 2 and Theorem 2
First  $d(x) = \frac{1}{2} \| x\|^2$  satisfies A4'. Since A1, A2, A3, A4 and C1 are equivalent to A1', A2', A3', A5' and C1', noticing that Lemma 2 is a special case of Lemma 9 and Theorem 2 is a special case of Theorem 6, we complete the proof.
D.3 Proof of Theorem 3
Recall that in the proof of DSPL, we actually used A4 and the properties from Proposition 1 that are deduced from A1, A2, A3 and C1. Indeed, it is straightforward to verify that  $f_{z}(x,\xi) = \langle \nabla f(z,\xi),x - z\rangle$  satisfies Proposition 1 given A1, A2, A3 and  $\lambda$ -smoothness of  $f$ .
E Delay-independent analysis
In this section, we further improve our analysis and show that by adopting a simple safeguarding strategy during the iterations.
E.1 Proof of Lemma 3
We show  $\sum_{k=1}^{K} \tau_k \leq mK$  by noticing the following facts: 1). at each iteration there are at most  $m-1$  agents accumulating 1 unit of delay; 2).  $\tau_k$  must come from a machine's previous delay accumulation. 3). once the gradient is used, delay on a machine starts count from 0. Assuming that at  $k = 1$  there are no previously accumulated delays on each agent, then we have  $\sum_{k=1}^{K} \tau_k \leq (m-1)K \leq mK$ . We use a loose bound  $mK$  to simplify notation.
E.2 Proof of Lemma 4
We know that  $T \sum_{k=1}^{K} \mathbb{I}\{\tau_k \geq T\} \leq \sum_{k=1}^{K} \tau_k \leq mK$  and recall that  $T = r^{-1}mK^{\beta}$ :
$$
\sum_ {k = 1} ^ {K} \mathbb {I} \left\{\tau_ {k} \leq T \right\} \geq K - \sum_ {k = 1} ^ {K} \mathbb {I} \left\{\tau_ {k} \geq T \right\} \geq K - \frac {m K}{T} = K - r K ^ {1 - \beta}.
$$
Also we have  $\sum_{k=1}^{K} \tau_k^2 \leq T \sum_{k=1}^{K} \tau_k \leq mKT$ .
E.3 Proof of Theorem 4
First we rewrite Lemma 1 by associating potential reduction with  $I_k \coloneqq \mathbb{I}\{\tau_k \leq T\}$ .
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} \left(x ^ {k + 1}\right) \right] \leq \psi_ {1 / \rho} \left(x ^ {k}\right) - \frac {\rho (\rho - 2 \lambda - \kappa) I _ {k}}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} - \frac {\rho \left(\gamma_ {k} - \rho\right) I _ {k}}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \\ + \frac {\rho \lambda I _ {k}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f} I _ {k}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ], \\ \end{array}
$$
where we have  $x^{k + 1} = x^k$ , thus  $\mathbb{E}_k[\psi_{1 / \rho}(x^{k + 1})] = \psi_{1 / \rho}(x^k)$  if  $I_k = 0$ .
From Lemma 4, we know that  $\sum_{k=1}^{K} I_k \geq K - rK^{1-\beta} = K(1 - rK^{-\beta}) = \eta^{-1}K$ . Then similar telescopic sum over the un-skipped iterations will give
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2} \mathbb {E} [ \| \hat {x} ^ {k ^ {*}} - x ^ {k ^ {*}} \| ^ {2} ] \\ = \frac {\rho (\rho - 2 \lambda - \kappa)}{2 \sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} I _ {k} ] \\ \leq \frac {\gamma - 2 \lambda - \kappa}{\sum_ {k = 1} ^ {K} I _ {k}} \left\{\psi_ {1 / \rho} \left(x ^ {1}\right) - \mathbb {E} \left[ \psi_ {1 / \rho} \left(x ^ {K + 1}\right) \right] \right\} - \frac {\rho (\gamma - \rho)}{2 \sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] I _ {k} (44) \\ + \frac {\rho \lambda}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] I _ {k} + \frac {2 \rho L _ {f}}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] I _ {k} \\ \leq \frac {\rho D}{\sum_ {k = 1} ^ {K} I _ {k}} + \frac {\sqrt {K} D}{\alpha \sqrt {\eta} \sum_ {k = 1} ^ {K} I _ {k}} (45) \\ + \frac {\rho}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ - \frac {\gamma - \rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \lambda \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + 2 L _ {f} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| \right] I _ {k}, \\ \end{array}
$$
where (44) uses the fact that  $\gamma_{k} \equiv \gamma$  is the same for all the iterations; (45) uses the relation  $\gamma = \frac{\sqrt{K}}{\alpha\sqrt{\eta}} + \rho + \kappa + 4\lambda \geq \max \left\{\frac{\sqrt{K}}{\alpha\sqrt{\eta}}, \rho + \kappa + 4\lambda \right\}$ . Then we bound the above terms, respectively, by
$$
\frac {\rho D}{\sum_ {k = 1} ^ {K} I _ {k}} \leq \frac {\rho D}{\eta^ {- 1} K} = \frac {\rho \eta D}{K} \tag {46}
$$
$$
\frac {\sqrt {K} D}{\alpha \sqrt {\eta} \sum_ {k = 1} ^ {K} I _ {k}} \leq \frac {\sqrt {K} D}{\alpha \eta^ {- 1 / 2} K} = \frac {\sqrt {\eta} D}{\sqrt {K} \alpha}, \tag {47}
$$
where we used the fact that  $\sum_{k=1}^{K} I_k \leq K$ . Meanwhile,
$$
\begin{array}{l} \frac {2 \rho L _ {f}}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| I _ {k} \leq \frac {2 \rho L _ {f}}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| \\ \leq \frac {4 \sqrt {\eta} \rho L _ {f} (L _ {f} + L _ {\omega}) \alpha}{\sqrt {K} \eta^ {- 1}} \sum_ {k = 1} ^ {K} (\tau_ {k} + 1) \\ \leq \frac {4 \eta^ {3 / 2} \rho L _ {f} \left(L _ {f} + L _ {\omega}\right) m \alpha}{\sqrt {K}}, \tag {48} \\ \end{array}
$$
where (48) uses a tighter bound  $\sum_{k=1}^{K} \tau_k \leq (m-1)K$ . Finally, we bound the delays using
$$
\frac {2 \rho \lambda}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} ] I _ {k} \leq \frac {8 \eta \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K \sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} I _ {k} \leq \frac {8 \eta^ {2} \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K ^ {2}} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} I _ {k},
$$
and that  $\frac{1}{K^2}\sum_{k = 1}^{K}\tau_k^2 I_k\leq \frac{mKT}{K^2} = \frac{m^2}{rK^{1 - \beta}}$ . Putting the above bounds back, using the same technique as in (18) to cancel the error  $2\lambda \| x^{k + 1} - x^{k}\|^{2}$  with  $-\sum_{k = 1}^{K}\frac{\gamma - \rho}{2}\| x^{k + 1} - x^{k}\|^{2},\gamma -\rho \geq 4\lambda$  we can re-arrange the terms to complete the proof.
E.4 Proof of Theorem 5
We prove Theorem 5 by showing its Bregman version.
Theorem 7 (Safeguarded DSPL in Bregman setting). Under the same conditions as Lemma 9 as well as D1, taking 1)  $\beta >0,K > r^{1 / \beta}$  or 2)  $\beta = 0,r < 1$ , then letting  $\gamma_{k}\equiv \gamma = \frac{\sqrt{K}}{\alpha\sqrt{\eta}} +\rho +\kappa +6\lambda ,\eta = 1 + \frac{r}{K^{\beta} - r}$  for some  $\alpha >0$  and  $k^{*}$  be uniformly chosen between iterations where  $\tau_{k}\leq \sqrt{K}$ , then
$$
\mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k ^ {*}}, x ^ {k ^ {*}}\right) \right] \leq \frac {1}{\rho (\rho - 2 \lambda - \kappa)} \left[ \frac {\eta (\rho + 4 \lambda) D}{K} + \frac {\sqrt {\eta} D}{\sqrt {K} \alpha} + \frac {2 \sqrt {\eta} \rho L _ {f} \alpha}{\sqrt {K}} + \frac {6 \eta^ {2} \rho \lambda \left(L _ {f} + L _ {\omega}\right) ^ {2} m ^ {2} \alpha^ {2}}{K ^ {1 / \beta}} \right],
$$
where  $D = \psi_{1 / \rho}^{d}(x^{1}) - \inf_{x}\psi^{d}(x)$
Proof of Theorem 7
Similar to the proof of Theorem 4, we write the modified potential reduction by
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} ^ {d} \left(x ^ {k + 1}\right) \right] \leq \psi_ {1 / \rho} ^ {d} \left(x ^ {k}\right) - \frac {\rho (\rho - 2 \lambda - \kappa) I _ {k}}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) - \frac {\rho \left(\gamma_ {k} - \rho\right) I _ {k}}{2 \left(\gamma_ {k} - 2 \lambda - \kappa\right)} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \\ + \frac {3 \rho \lambda I _ {k}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f} I _ {k}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} \\ \end{array}
$$
Then telescoping gives
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2} \mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k ^ {*}}, x ^ {k ^ {*}}\right) \right] \\ = \frac {\rho (\rho - 2 \lambda - \kappa)}{2 \sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ V _ {d} \left(\hat {x} ^ {k}, x ^ {k}\right) I _ {k} \right] \\ \leq \frac {\gamma - 2 \lambda - \kappa}{\sum_ {k = 1} ^ {K} I _ {k}} \{\psi_ {1 / \rho} (x ^ {1}) - \mathbb {E} [ \psi_ {1 / \rho} (x ^ {K + 1}) ] \} - \frac {\rho (\gamma - \rho)}{2 \sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] I _ {k} \\ + \frac {3 \rho \lambda}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] I _ {k} + \frac {2 \rho L _ {f}}{\gamma - \kappa} \\ \leq \frac {(\rho + 4 \lambda) D}{\sum_ {k = 1} ^ {K} I _ {k}} + \frac {\sqrt {K} D}{\alpha \sqrt {\eta} \sum_ {k = 1} ^ {K} I _ {k}} + \frac {2 \rho L _ {f}}{\gamma - \kappa} \tag {49} \\ + \frac {\rho}{\sum_ {k = 1} ^ {K} I _ {k}} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ - \frac {\gamma - \rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \frac {3 \lambda}{2} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} \right] I _ {k} \\ \leq \frac {\eta (\rho + 4 \lambda) D}{K} + \frac {\sqrt {\eta} D}{\sqrt {K} \alpha} + \frac {2 \sqrt {\eta} \rho L _ {f} \alpha}{\sqrt {K}} + \frac {6 \eta^ {2} \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} m ^ {2} \alpha^ {2}}{r K ^ {1 - \beta}}, \\ \end{array}
$$
where (49) reuses (46), (47) and (48). A re-arrangement completes the proof of Theorem 7. Taking  $d(x) = \frac{1}{2} \| x\|^2$  completes the proof of Theorem 5.
F DSPL with momentum
F.1 Preliminaries of DSEPL
Momentum has been an important ingredient for a stochastic algorithm to be implemented in practice. In this section, we incorporate the extrapolation technique into the delayed prox-linear algo
rithm. Before the master performs a proximal update, it uses two recent iterates to compute an extrapolated iterate  $y^{k}$ . Then a proximal update is done centered around  $y^{k}$  with delayed information. We summarize the procedure in Algorithm 5. Throughout this section we take  $\gamma_{k} \equiv \gamma$ .
Algorithm 5: Extrapolated DSPL
Input:  $x^0, x^1, \beta$ ;
Let  $c(x^{k - \tau_k},\xi^{k - \tau_k})$  and  $\nabla c(x^{k - \tau_k},\xi^{k - \tau_k})$  be computed by a worker with delay  $\tau_{k}$  In the master node update
$$
y ^ {k} = x ^ {k} + \beta \left(x ^ {k} - x ^ {k - 1}\right),
$$
$$
x ^ {k + 1} = \arg \min  _ {x} \left\{f _ {x ^ {k - \tau_ {k}}} (x, \xi^ {k - \tau_ {k}}) + \omega (x) + \frac {\gamma}{2} \| x - y ^ {k} \| ^ {2} \right\}.
$$
To analyze DSEPL, we extend the framework from [5] to the delayed case and the analysis is based on a virtual iterate
$$
z ^ {k} := x ^ {k} + \beta \theta^ {- 1} \left(x ^ {k} - x ^ {k - 1}\right),
$$
where the extrapolation parameter, also known as momentum, is fixed at some constant  $\beta \in [0,1)$  and  $\theta = 1 - \beta$ . Using the virtual iterate, DSEPL uses a more complicated potential function.
$$
\psi_ {1 / \rho} (z ^ {k}) + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \psi (x ^ {k}) + \frac {\rho (\gamma \beta + 2 \rho \beta^ {2} \theta^ {- 2})}{2 (\gamma - \lambda) \theta} \| x ^ {k} - x ^ {k - 1} \| ^ {2} \tag {50}
$$
As extrapolation increases the instability of the iterations, analysis of DSEPL furthers requires boundedness of the delays.
E1: (Bounded delay) Independent delays are bounded by  $\tau < \infty$ .
The following lemma presents a similar descent property for the potential function.
Lemma 10. Suppose A1, A2, A3 and C1 hold. Given  $0 \leq \beta < 1$ ,  $\rho > 3\lambda + 2\kappa\beta + \kappa$  and  $\gamma \geq \rho$ ,