SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min _ {\| x \|, \| y \| \leq \Delta} | \langle u, z _ {x} \rangle \langle v, z _ {y} \rangle + \langle v, z _ {y} \rangle \langle u, x - z _ {x} \rangle + \langle u, z _ {x} \rangle \langle v, y - z _ {y} \rangle - b | + \frac {\gamma}{2} [ \| x - w _ {x} \| ^ {2} + \| y - w _ {y} \| ^ {2} ].
$$
First we assume that the constraints are inactive, then the solution is available in closed form
$$
w ^ {+} = w + \operatorname {P r o j} _ {[ - 1, 1 ]} (- \frac {\delta}{\| \zeta \| ^ {2}}) \zeta ,
$$
Figure 11: From left to right:  $(\kappa, p_{\mathrm{fail}}) = (1, 0.2)$ , Geometric and Poisson delays with  $\tau = 10$ ;  $(\kappa, p_{\mathrm{fail}}) = (1, 0.3)$ , Geometric and Poisson delays with  $\tau = 216$ .
$$
\delta = \gamma^ {- 1} [ \langle u, z _ {x} \rangle \langle v, z _ {y} \rangle + \langle v, z _ {y} \rangle \langle u, w _ {x} - z _ {x} \rangle + \langle u, z _ {x} \rangle \langle v, w _ {y} - z _ {y} \rangle - b ]
$$
$$
\zeta = \gamma^ {- 1} (\langle v, z _ {y} \rangle u, \langle u, z _ {x} \rangle v).
$$
H.4 Separate figures of Section 6
Figure 12: Figure 2 after splitting DSGD and DSPL. Two plots on the left:  $\| x^{k} - \hat{x}\|$  in the first 40 epochs given  $\alpha = 0.1$ ; two on the right:  $f(x^{k}) - f(\hat{x})$  in the first 40 epochs given  $\alpha = 0.5$ .
References in the appendix
[1] Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization, 29(1):207-239, 2019.
[2] Damek Davis and Dmitriy Drusvyatskiy. Graphical convergence of subgradients in nonconvex optimization and learning. Mathematics of Operations Research, 47(1):209-231, 2022.
[3] Damek Davis, Dmitriy Drusvyatskiy, and Kellie J MacPhee. Stochastic model-based minimization under high-order growth. arXiv preprint arXiv:1807.00255, 2018.
[4] Damek Davis and Benjamin Grimmer. Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems. SIAM Journal on Optimization, 29(3):1908-1930, 2019.
[5] Qi Deng and Wenzhi Gao. Minibatch and momentum model-based methods for stochastic weakly convex optimization. Advances in Neural Information Processing Systems, 34, 2021.
[6] Haihao Lu. "relative continuity" for non-lipschitz nonsmooth convex optimization using stochastic (or deterministic) mirror descent. INFORMS Journal on Optimization, 1(4):288-303, 2019.
[7] R Tyrrell Rockafellar. Convex analysis princeton university press. Princeton, NJ, 1970.
[8] Siqi Zhang and Niao He. On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization. arXiv preprint arXiv:1806.04781, 2018.