SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
V _ {d} (\hat {x} ^ {k}, x ^ {k}) = \frac {1}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} = \frac {1}{2 \rho^ {2}} \| \nabla \psi_ {1 / \rho} ^ {d} (x) \| ^ {2}.
$$
When the context is clear, we will use  $\psi = \psi^d$  directly if  $d(x) = \frac{1}{2}\| x\|^2$ . In the next section, we present several useful auxiliary results that appear frequently in the proof.
B.2 Auxiliary lemmas
In this section, we present all the auxiliary lemmas that will be used in the proof of our main results. While some of these lemmas are widely recognized in the field, we reproduce them here to ensure that our work remains self-contained. To begin, we introduce the well-known three-point lemma.
Lemma 5 (Three point lemma). Let  $f$  be a closed convex function and define
$$
z = \underset {x} {\arg \min } \{f (x) + \gamma V _ {d} (x, y) \}.
$$
for some  $y \in \operatorname{dom} d$ . Then we have
$$
f (z) + \gamma V _ {d} (z, y) \leq f (x) + \gamma V _ {d} (x, y) - \gamma V _ {d} (x, z), \forall x \in \operatorname {d o m} d.
$$
The following lemma bounds, for a deterministic function, the proximal step by the regularization term and relative Lipschitzian property.
Lemma 6 (Bounding the proximal step). Given the divergence  $V_{d}$  generated by some 1-strongly convex function  $d$ . Let the function  $f(x)$  satisfy the relative Lipschitzian property (5). For any  $\gamma > 0$  such that  $f + \gamma d$  is strongly convex, if we define
$$
x ^ {+} = \underset {y} {\arg \min } \{f (y) + \gamma V _ {d} (y, x) \},
$$
$$
\sqrt {V _ {d} \left(x ^ {+} , x\right)} \leq \sqrt {2} \gamma^ {- 1} L \tag {6}
$$
$$
\left\| x ^ {+} - x \right\| \leq 2 \gamma^ {- 1} L. \tag {7}
$$
Proof. By the optimality of  $x^{+}$  and relative Lipschizian property, we have
$$
f (x ^ {+}) + \gamma V _ {d} (x ^ {+}, x) \leq f (x)
$$
$$
\gamma V _ {d} \left(x ^ {+}, x\right) \leq f (x) - f \left(x ^ {+}\right) \leq L \sqrt {2 V _ {d} \left(x ^ {+} , x\right)}. \tag {8}
$$
Dividing both sides of (8) by  $\sqrt{V_d(x^+,x)}$  gives (9); (10) uses  $V_{d}(x^{+},x)\geq \frac{1}{2}\| x^{+} - x\|^{2}$ .
Lemma 7 (Bounding the proximal step in expectation). Given the divergence  $V_{d}$  generated by some 1-strongly convex function  $d$ . Let convex stochastic function  $f(x, \xi)$  satisfy the relative Lipschitzian property (5) with  $L = L(\xi)$  for  $\xi \sim \Xi$ . For any  $\gamma > 0$ , if we define
$$
x ^ {+} = \underset {y} {\arg \min } \{f (y, \xi) + \gamma V _ {d} (y, x) \},
$$
$$
\frac {1}{\sqrt {2}} \mathbb {E} _ {\xi \sim \Xi} [ \| x ^ {+} - x \| ] \leq \mathbb {E} _ {\xi \sim \Xi} [ \sqrt {V _ {d} (x ^ {+} , x)} ] \leq \sqrt {2} \gamma^ {- 1} \mathbb {E} _ {\xi \sim \Xi} [ L (\xi) ] \tag {9}
$$
$$
\mathbb {E} _ {\xi \sim \Xi} [ \| x ^ {+} - x \| ^ {2} ] \leq 4 \gamma^ {- 2} \mathbb {E} _ {\xi \sim \Xi} [ L (\xi) ^ {2} ]. \tag {10}
$$
Proof. We can apply Lemma 6. Then we have, for each  $\xi \sim \Xi$ , that
$$
\sqrt {V _ {d} (x ^ {+} , x)} \leq \sqrt {2} \gamma^ {- 1} L (\xi) \quad \text {a n d} \quad V _ {d} (x ^ {+}, x) \leq 2 \gamma^ {- 2} L ^ {2} (\xi).
$$
Taking expectation with respect to  $\xi$ , the fact that  $V_{d}(x^{+},x)\geq \frac{1}{2}\| x^{+} - x\|^{2}$  completes the proof
C Convergence Analysis of DSGD
In this section, we present the convergence analysis of DSGD. The proof follows a standard inexact potential reduction scheme and is done in the Euclidean setup, where Lemma 5 and Lemma 7 hold for  $d = \frac{1}{2} \| \cdot \|^{2}$ .
Since  $\rho >\lambda +\kappa$  and  $\gamma_{k}\geq \rho$ , by three-point lemma we get the following two relations
$$
\begin{array}{l} \langle g ^ {k - \tau_ {k}}, x ^ {k + 1} \rangle + \omega (x ^ {k + 1}) + \frac {\gamma_ {k}}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \\ \leq \left\langle g ^ {k - \tau_ {k}}, \hat {x} ^ {k} \right\rangle + \omega \left(\hat {x} ^ {k}\right) + \frac {\gamma_ {k}}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} - \frac {\gamma_ {k} - \kappa}{2} \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} \tag {11} \\ \end{array}
$$
$$
f \left(\hat {x} ^ {k}\right) + \omega \left(\hat {x} ^ {k}\right) + \frac {\rho}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} \leq f \left(x ^ {k + 1}\right) + \omega \left(x ^ {k + 1}\right) + \frac {\rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \tag {12}
$$
Summing up (11) and (12) and taking expectation, we have
$$
\begin{array}{l} \frac {\gamma_ {k} - \rho}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] - \frac {\gamma_ {k} - \rho}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \frac {\gamma_ {k} - \kappa}{2} \mathbb {E} _ {k} [ \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} ] \\ \leq \mathbb {E} _ {\xi^ {k - \tau_ {k}}} [ \langle g ^ {k - \tau_ {k}}, \hat {x} ^ {k} - x ^ {k - \tau_ {k}} \rangle ] - f (\hat {x} ^ {k}) + \mathbb {E} _ {k} [ f (x ^ {k + 1}) ] - \mathbb {E} _ {k} [ \langle g ^ {k - \tau_ {k}}, x ^ {k + 1} - x ^ {k - \tau_ {k}} \rangle ] \\ = \mathbb {E} _ {\xi^ {k - \tau_ {k}}} \left[ \left\langle g ^ {k - \tau_ {k}}, \hat {x} ^ {k} - x ^ {k - \tau_ {k}} \right\rangle \right] + f \left(x ^ {k - \tau_ {k}}\right) - f \left(\hat {x} ^ {k}\right) \\ + \mathbb {E} _ {k} [ f (x ^ {k + 1}) ] - f (x ^ {k - \tau_ {k}}) - \mathbb {E} _ {k} [ \langle g ^ {k - \tau_ {k}}, x ^ {k + 1} - x ^ {k - \tau_ {k}} \rangle ] \\ \leq \frac {\lambda}{2} \| \hat {x} ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} + 2 L _ {f} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] \tag {13} \\ \end{array}
$$
where (13) follows from  $\lambda$ -weak convexity of  $f(x, \xi)$  and B1 (using [1]). Re-arranging the terms, we have
$$
\begin{array}{l} \frac {\gamma_ {k} - \kappa}{2} \mathbb {E} _ {k} [ \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} ] \\ \leq \frac {\gamma_ {k} - \rho}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \frac {\lambda}{2} \| \hat {x} ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} \\ + 2 L _ {f} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] - \frac {\gamma_ {k} - \rho}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ \leq \frac {\gamma_ {k} - \rho}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \lambda \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \lambda \mathbb {E} _ {k} [ \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} ] \\ + 2 L _ {f} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| - \frac {\gamma_ {k} - \rho}{2} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} \right] \right. \tag {14} \\ \end{array}
$$
where (14) follows by Cauchy's inequality  $\| a + b\| ^2\leq 2\| a\| ^2 +2\| b\| ^2$  . Now we re-arrange the terms and divide both sides by  $\gamma_{k} - 2\lambda -\kappa$  to derive
$$
\begin{array}{l} \mathbb {E} _ {k} [ \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} ] \\ \leq \frac {\gamma_ {k} - \rho}{\gamma_ {k} - 2 \lambda - \kappa} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \frac {2 \lambda}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ + \frac {4 L _ {f}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] - \frac {\gamma_ {k} - \rho}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ = \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} - \frac {\rho - 2 \lambda - \kappa}{\gamma_ {k} - 2 \lambda - \kappa} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \frac {2 \lambda}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ + \frac {4 L _ {f}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] - \frac {\gamma_ {k} - \rho}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ]. \tag {15} \\ \end{array}
$$
Finally, we measure the reduction in potential function  $\psi_{1 / \rho}(x^k)$  and successively deduce that
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} (x ^ {k + 1}) \right] \\ = \mathbb {E} _ {k} \left[ f \left(\hat {x} ^ {k + 1}\right) + \omega \left(\hat {x} ^ {k + 1}\right) + \frac {\rho}{2} \| \hat {x} ^ {k + 1} - x ^ {k + 1} \| ^ {2} \right] \\ \leq \mathbb {E} _ {k} \left[ f \left(\hat {x} ^ {k}\right) + \omega \left(\hat {x} ^ {k}\right) + \frac {\rho}{2} \| \hat {x} ^ {k} - x ^ {k + 1} \| ^ {2} \right] \\ \leq \mathbb {E} _ {k} [ f (\hat {x} ^ {k}) + \omega (\hat {x} ^ {k}) + \frac {\rho}{2} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} ] - \frac {\rho (\rho - 2 \lambda - \kappa)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} \\ + \frac {\rho \lambda}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} \right] + \frac {2 \rho L _ {f}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| \right] \tag {16} \\ - \frac {\rho (\gamma_ {k} - \rho)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ = \psi_ {1 / \rho} (x ^ {k}) - \frac {\rho (\rho - 2 \lambda - \kappa)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} + \frac {\rho \lambda}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ + \frac {2 \rho L _ {f}}{\gamma_ {k} - 2 \lambda - \kappa} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] - \frac {\rho (\gamma_ {k} - \rho)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ]. \\ \end{array}
$$
where relation (16) plugs (15) in. A simple re-arrangement completes the proof.
By the descent property revealed in Lemma 1, we multiply both sides of the inequality by  $\gamma_k - 2\lambda - \kappa$ , telescope over  $k = 1, \ldots, K$  and deduce that
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2} \mathbb {E} [ \| \hat {x} ^ {k ^ {*}} - x ^ {k ^ {*}} \| ^ {2} ] \\ = \frac {\rho (\rho - 2 \lambda - \kappa)}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \hat {x} ^ {k} - x ^ {k} \| ^ {2} ] \\ \leq \frac {\gamma - 2 \lambda - \kappa}{K} \{\psi_ {1 / \rho} (x ^ {1}) - \mathbb {E} [ \psi_ {1 / \rho} (x ^ {K + 1}) ] \} - \frac {\rho (\gamma - \rho)}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {\rho}{K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \lambda \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + 2 L _ {f} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] \\ \leq \frac {(\gamma - 2 \lambda - \kappa) D}{K} - \frac {\rho (\gamma - \rho)}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {\rho}{K} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ - \frac {\gamma - \rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \lambda \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + 2 L _ {f} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| \right], \tag {17} \\ \end{array}
$$
where (17) is due to  $\psi(x^{*}) \leq \mathbb{E}_{k}[\psi_{1 / \rho}(x^{K + 1})]$ . Now it remains to bound the error from the stochastic delays. Let's first consider applying  $\|a + b\|^2 \leq 2\|a\|^2 + 2\|b\|^2$  to get
$$
\sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] \leq 2 \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} ] + 2 \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \tag {18}
$$
and we can bound the first term using
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} ] = \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \left\| \sum_ {l = 1} ^ {\tau_ {k}} x ^ {k + 1 - l} - x ^ {k - l} \right\| ^ {2} \right] \\ \leq \sum_ {k = 1} ^ {K} \tau_ {k} \sum_ {l = 1} ^ {\tau_ {k}} \mathbb {E} [ \| x ^ {k + 1 - l} - x ^ {k - l} \| ^ {2} ] (19) \\ \leq \frac {4 \left(L _ {f} + L _ {\omega}\right) ^ {2}}{\gamma^ {2}} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2}, (20) \\ \end{array}
$$
where (19) uses the fact  $\| \sum_{i=1}^{k} a_i\|^2 \leq k \sum_{i=1}^{k} \|a_i\|^2$ , (20) invokes Lemma 7 since  $\langle g, x \rangle + \omega$  is  $(\|g\| + L_\omega)$  Lipschitz continuous with  $\mathbb{E}_\xi[\|g\|^2] \leq L_f^2$ , and we let  $x^{k-j} = x^1$  if  $k \geq j$ . On the other hand, we bound the first-order terms by
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ] \leq \sum_ {k = 1} ^ {K} \sum_ {l = 0} ^ {\tau_ {k}} \mathbb {E} [ \| x ^ {k - l + 1} - x ^ {k - l} \| ] \\ \leq \sum_ {k = 1} ^ {K} 2 \gamma^ {- 1} \left(L _ {f} + L _ {\omega}\right) \left(\tau_ {k} + 1\right) \tag {21} \\ = 2 \gamma^ {- 1} \left(L _ {f} + L _ {\omega}\right) \left(K + \sum_ {k = 1} ^ {K} \tau_ {k}\right), \\ \end{array}
$$
where (21) again invokes Lemma 7. Plugging the bound back, we deduce that
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{2} \mathbb {E} [ | | \hat {x} ^ {k ^ {*}} - x ^ {k ^ {*}} | | ^ {2} ] \\ \leq \frac {(\gamma - 2 \lambda - \kappa) D}{K} + \frac {\rho}{K} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ - \frac {\gamma - \rho}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \lambda \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + 2 L _ {f} \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| \right] \\ \leq \frac {(\gamma - 2 \lambda - \kappa) D}{K} + \frac {4 \rho L _ {f} (L _ {f} + L _ {\omega})}{\gamma K} \sum_ {k = 1} ^ {K} \left(\tau_ {k} + 1\right) + \frac {8 \rho \lambda \left(L _ {f} + L _ {\omega}\right) ^ {2}}{\gamma^ {2} K} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} \tag {22} \\ \leq \frac {(\rho + 2 \lambda) D}{K} + \frac {D}{\sqrt {K} \alpha} + \frac {4 \rho L _ {f} (L _ {f} + L _ {\omega}) \alpha}{\sqrt {K}} \Big (\frac {1}{K} \sum_ {k = 1} ^ {K} \tau_ {k} + 1 \Big) + \frac {8 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K} \Big (\frac {1}{K} \sum_ {k = 1} ^ {K} \tau_ {k} ^ {2} \Big) \\ \end{array}
$$
where (22) is by  $\gamma = \sqrt{K} / \alpha + \rho + 4\lambda + \kappa$  and we cancel the first summation term in (18) with  $-\sum_{k=1}^{K} \frac{\gamma - \rho}{2} \|x^{k+1} - x^k\|^2$  since  $\gamma \geq \rho + 4\lambda$ . The proof is complete after dividing both sides by  $\rho (\rho - 2\lambda - \kappa)$  and using the fact that  $\frac{1}{2} \|\hat{x}^k - x^k\|^2 = \frac{1}{2\rho^2} \|\nabla \psi_{1/\rho}(x)\|^2$ .
D Convergence analysis of DSPL
In this section, we introduce a more general DSPL associated with Bregman divergence and establish its convergence analysis. After proving the result in our Bregman context, the Euclidean results in Section 4 follow immediately as a special case.
Why use Bregman divergence? Recall that in C1 we assume Lipschitz continuity of both  $c$  and  $\nabla c$ , which does not hold without assuming a bounded set. The main issue here is to expect a Lipschitz smooth function to also be Lipschitz continuous. Using the Bregman context, we can relax such Lipschitz continuity to relative Lipschitzian property, so that we do not necessarily need the bounded set assumption.
D.1 Preliminaries and analysis of Bregman DSPL
In this section, we present the convergence results of DSPL. As we mentioned in Remark 4, we replace  $\frac{1}{2} \| \cdot \|^{2}$  regularization by divergence  $V_{d}$  and summarize the update in Algorithm 4.
Algorithm 4: A delayed stochastic prox-linear method with Bregman proximal updates
$$
x ^ {k + 1} = \underset {x} {\arg \min } \left\{h \left(c \left(x ^ {k - \tau_ {k}}, \xi^ {k - \tau_ {k}}\right) + \left\langle \nabla c \left(x ^ {k - \tau_ {k}}, \xi^ {k - \tau_ {k}}\right), x - x ^ {k} \right\rangle\right) + \omega (x) + \gamma_ {k} V _ {d} \left(x, x _ {k}\right) \right\} \tag {23}
$$
Now we overload the assumptions used in Section 4.
A1': (i.i.d. sample) It is possible to draw i.i.d. samples  $\{\xi^k\}$  from  $\Xi$ .
A2': (Relative Lipschitzian property)  $\omega(x)$  satisfies relatively Lipschitzian property (5)
$$
\omega (x) - \omega (y) \leq L _ {\omega} \sqrt {2 V _ {d} (y , x)}
$$
$\mathbf{A3}^{\prime}$  : (Weak convexity)  $\omega (x)$  is  $\kappa$  -weakly convex.
A4': (Strongly-convex kernel) Kernel function  $d$  is 1-strongly convex.
A5': (Bounded moment) The distribution of the independent stochastic delays  $\{\tau_k\}$  has bounded first and second moments. i.e.,  $\mathbb{E}[\tau_k] \leq \bar{\tau} < \infty$ ,  $\mathbb{E}[\tau_k^2] \leq \tau_{\sigma^2} < \infty$ , for all  $k$ .
C1':  $h(x)$  is convex and  $L_{h}$ -Lipschitz continuous;  $c(x, \xi)$  has  $C(\xi)$ -Lipschitz continuous gradient, for all  $\xi \sim \Xi$ . Moreover, during the algorithm the stochastic function  $f_{z}(x, \xi), z \in \operatorname{dom} d$  is  $L_{f}(\xi)$ -relative Lipschitzian to  $d$  for all  $\xi \sim \Xi$ . Moreover, we assume  $\mathbb{E}_{\xi}[C(\xi)^2] \leq C^2, \mathbb{E}_{\xi}[L_c(\xi)^2] \leq L_c^2$ .
Remark 6. It can be seen that we added extra assumptions on the kernel and relaxed the Lipschitz continuity of  $c$  in  $\mathbf{C}1'$  and  $\omega$  in  $\mathbf{A}2'$ . Specially if  $c$  is  $L_{c}(\xi)$ -Lipschitz continuous, we have  $f_{z}(x,\xi)$  is  $L_{h}L_{c}(\xi)$ -Lipschitz continuous since
$$
\begin{array}{l} f _ {z} (x, \xi) - f _ {z} (y, \xi) \\ = h (c (z, \xi) + \langle \nabla c (z, \xi), x - z \rangle) - h (c (z, \xi) + \langle \nabla c (z, \xi), y - z \rangle) \\ \leq L _ {h} | \langle \nabla c (z, \xi), x - y \rangle | \\ \leq L _ {h} L _ {c} (\xi) \| x - y \| \\ \end{array}
$$
With the above assumptions, we can further derive a more general version of Proposition 1.
Proposition 2 (Proposition 1 in the Bregman setting). Let  $\{f_z(\cdot ,\xi)\}$  be the sequence of stochastic functions queried during the DSPL algorithm, then
$\pmb{P}\pmb{I}^{\prime}$  : (Convexity)  $f_{z}(x,\xi)$  is convex, for all  $x\in \operatorname {dom}d$  and  $\xi \sim \Xi$
$\pmb{P2'}$ : (Two-sided approximation)  $|f_z(x, \xi) - f(x, \xi)| \leq \frac{L_h C(\xi)}{2} \|x - z\|^2$ , for all  $x \in \operatorname{dom} d$  and all  $\xi \sim \Xi$ .
P3': (Relative Lipschitzian property)  $f_{z}(x,\xi) - f_{z}(y,\xi)\leq L_{f}(\xi)\sqrt{2V_{d}(y,x)}$  for all  $x,y\in \operatorname {dom}d$  and all  $\xi \sim \Xi$
As we did in Section 4, we take  $\lambda = L_hC$  and use these constants to present the results. Our analysis adopts the conventional potential reduction framework but a more careful treatment of stochastic noise is needed to improve the convergence result. The next lemma bounds the stochastic noise of the algorithm and is key to our analysis of DSPL.
Lemma 8 (Stability of the stochastic iteration). Assume that the assumptions  $\mathbf{A1}'$  to  $\mathbf{A4}'$  hold. If  $f_{x}(\cdot, \xi)$  is convex and satisfies Proposition 2, then the proximal iteration
$$
x ^ {k + 1} = \arg \min  _ {x} \left\{f _ {x ^ {k - \tau_ {k}}} \left(x, \xi^ {k - \tau_ {k}}\right) + \omega (x) + \gamma_ {k} V _ {d} \left(x, z ^ {k}\right) \right\}
$$
$$
\left| \mathbb {E} _ {k} \left\{\mathbb {E} _ {\xi} \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi\right) \right] - f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) \right\} \right| \leq \frac {2 L _ {f} ^ {2}}{\gamma - \kappa},
$$
for any  $\gamma_{k} > \kappa$
Now we can get the lemmas that match our main results in a Bregman context.
Lemma 9 (Lemma 2 in the Bregman setting). Suppose  $\mathbf{A1}',\mathbf{A2}',\mathbf{A3}',\mathbf{A4}'$  and  $\mathbf{C1}'$  hold, if  $\rho > 2\lambda + \kappa, \gamma_k \geq \rho$ , then
$$
\begin{array}{l} \frac {\rho (\rho - 2 \lambda - \kappa)}{\gamma_ {k} - 2 \lambda - \kappa} V _ {d} (\hat {x} ^ {k}, x ^ {k}) \leq \psi_ {1 / \rho} ^ {d} (x ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} ^ {d} (x ^ {k + 1}) ] - \frac {\rho (\gamma_ {k} - \rho)}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \\ + \frac {3 \rho \lambda}{2 (\gamma_ {k} - 2 \lambda - \kappa)} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} ] + \frac {2 \rho L _ {f} ^ {2}}{(\gamma_ {k} - \kappa) (\gamma_ {k} - 2 \lambda - \kappa)} \\ \end{array}
$$
Theorem 6 (Theorem 2 in the Bregman setting). Under the same conditions as Lemma 9, as well as  $\mathbf{A5}'$ , taking  $\gamma_k \equiv \gamma = \rho + 6\lambda + \kappa + \sqrt{K} / \alpha$  for some  $\alpha > 0$ , letting  $k^*$  be uniformly chosen between 1 and  $K$ , then
$$
\mathbb {E} [ V _ {d} (\hat {x} ^ {k ^ {*}}, x ^ {k ^ {*}}) ] \leq \frac {1}{\rho (\rho - 2 \lambda - \kappa)} \left[ \frac {(\rho + 4 \lambda) D}{K} + \frac {D}{\sqrt {K} \alpha} + \frac {2 \rho L _ {f} ^ {2} \alpha}{\sqrt {K}} + \frac {6 \rho \lambda (L _ {f} + L _ {\omega}) ^ {2} \alpha^ {2}}{K} \Delta_ {2} \right],
$$
where  $D = \psi_{1 / \rho}^{d}(x^{1}) - \inf_{x}\psi (x)$
Proof Proposition 2
$f_{z}(x,\xi)$  inherits convexity from  $h$ . The other properties hold by
$$
\begin{array}{l} | f (x, \xi) - f _ {y} (x, \xi) | \\ = | h (c (x, \xi)) - h (c (y, \xi) + \langle \nabla c (y, \xi), x - y \rangle) | \\ \leq L _ {h} | c (x, \xi) - c (y, \xi) - \langle \nabla c (y, \xi), x - y \rangle | \\ \leq \frac {L _ {h} C}{2} \| x - y \| ^ {2} \\ \end{array}
$$
and  $\mathbf{P3}'$  is by  $\mathbf{C1}'$ .
Proof of Lemma 8
Without loss of generality, we consider the following proximal mapping
$$
\mathcal{A}(z,x,\xi):= \operatorname *{arg  min}_{w}\bigl\{f_{z}(w,\xi) + \omega (w) + \gamma V_{d}(w,x)\bigr \} ,
$$
where  $\mathcal{A}$  denotes the proximal mapping from two past iterates  $z, x$  and a given sample  $\xi \sim \Xi$ . Then we invoke the three-point lemma to get, for any  $u \in \operatorname{dom} d$  that
$$
\begin{array}{l} f _ {z} (\mathcal {A} (z, x, \xi), \xi) + \omega (\mathcal {A} (z, x, \xi)) + \gamma V _ {d} (\mathcal {A} (z, x, \xi), x) \\ \leq f _ {z} (u, \xi) + \omega (u) + \gamma V _ {d} (u, x) - (\gamma - \kappa) V _ {d} (u, \mathcal {A} (z, x, \xi)) \tag {24} \\ \end{array}
$$
Similarly, given some  $\xi' \sim \Xi$ ,  $v \in \operatorname{dom} d$ , we have
$$
\begin{array}{l} f _ {z} \left(\mathcal {A} \left(z, x, \xi^ {\prime}\right), \xi^ {\prime}\right) + \omega \left(\mathcal {A} \left(z, x, \xi^ {\prime}\right)\right) + \gamma V _ {d} \left(\mathcal {A} \left(z, x, \xi^ {\prime}\right), x\right) \\ \leq f _ {z} (v, \xi^ {\prime}) + \omega (v) + \gamma V _ {d} (v, x) - (\gamma - \kappa) V _ {d} (v, \mathcal {A} (z, x, \xi^ {\prime})). \tag {25} \\ \end{array}
$$
Letting  $u = \mathcal{A}(z,x,\xi')$  in (24) and  $v = \mathcal{A}(z,x,\xi)$  in (25), we sum the two relations up and get
$$
\begin{array}{l} (\gamma - \kappa) \left[ V _ {d} \left(\mathcal {A} (z, x, \xi^ {\prime}), \mathcal {A} (z, x, \xi)\right) + V _ {d} \left(\mathcal {A} (z, x, \xi), \mathcal {A} (z, x, \xi^ {\prime})\right) \right] \\ \leq f _ {z} (\mathcal {A} (z, x, \xi), \xi^ {\prime}) - f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi^ {\prime}) + f _ {z} (\mathcal {A} (z, x, \xi^ {\prime}), \xi) - f _ {z} (\mathcal {A} (z, x, \xi), \xi) \\ \leq L _ {f} \left[ \sqrt {2 V _ {d} \left(\mathcal {A} (z , x , \xi^ {\prime}) , \mathcal {A} (z , x , \xi)\right)} + \sqrt {2 V _ {d} \left(\mathcal {A} (z , x , \xi) , \mathcal {A} (z , x , \xi^ {\prime})\right)} \right] (26) \\ \leq 2 L _ {f} \left[ \sqrt {V _ {d} \left(\mathcal {A} (z , x , \xi^ {\prime}) , \mathcal {A} (z , x , \xi)\right) + V _ {d} \left(\mathcal {A} (z , x , \xi) , \mathcal {A} (z , x , \xi^ {\prime})\right)} \right] (27) \\ \end{array}
$$
where (26) is by Lipschitzness from  $\mathbf{A}2^{\prime}$ ,  $\mathbf{P}3^{\prime}$  and (27) uses the relation  $\sqrt{a} +\sqrt{b}\leq \sqrt{2}\sqrt{a + b}$ . Then this implies
$$
\begin{array}{l} \max  \left\{\sqrt {V _ {d} (\mathcal {A} (z , x , \xi^ {\prime}) , \mathcal {A} (z , x , \xi))}, \sqrt {V _ {d} (\mathcal {A} (z , x , \xi) , \mathcal {A} (z , x , \xi^ {\prime}))} \right\} \\ \leq \sqrt {V _ {d} (\mathcal {A} (z , x , \xi^ {\prime}) , \mathcal {A} (z , x , \xi)) + V _ {d} (\mathcal {A} (z , x , \xi) , \mathcal {A} (z , x , \xi^ {\prime}))} \leq \frac {2 L _ {f}}{\gamma - \kappa}. \tag {28} \\ \end{array}
$$
Last we successively deduce that