SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using this observation, we obtain  $\mu_{\omega ,\alpha}^{r}\coloneqq \frac{rd}{K_{\omega,\alpha}^{r}}\leq \frac{rd}{(1 - r)K}\leq \frac{d}{K}$ . Note that  $\alpha \geq K_{\alpha} / d$  and  $\omega \leq d / K_{\omega} - 1$  for  $\mathrm{Top}K$  and Rand  $K$ . Thus  $\alpha \geq \min \left\{\frac{(1 - r)K}{rd},1\right\}$  and  $\omega \leq \frac{d}{K} -1$ . We substitute the bounds to (16) and obtain
$$
\mathfrak {m} _ {\mathrm {r e a l i s t i c}} ^ {r} = \widetilde {\mathcal {O}} \bigg ((1 - r) K \bigg (\sqrt {\frac {d}{K}} \sqrt {\frac {L}{\mu}} + \frac {d}{K} \sqrt {\frac {r L}{(1 - r) \mu}} + \frac {d}{K} \sqrt {\frac {L _ {\mathrm {m a x}}}{n \mu}} + \frac {r d}{(1 - r) K} + \frac {d}{K} + \frac {r d}{(1 - r) K} \bigg) + d \bigg).
$$
Since  $r \in [0, 1/2]$  and  $K \leq d$ , one can easily show that
$$
\mathfrak {m} _ {\text {r e l i s t i c}} ^ {r} = \widetilde {\mathcal {O}} \left(d \sqrt {\frac {L}{\mu}} + d \sqrt {\frac {L _ {\operatorname* {m a x}}}{n \mu}} + d\right).
$$
It is left to use Lemma 1.4, to get  $\mathfrak{m}_{\mathrm{realistic}}^r = \widetilde{\mathcal{O}}\left(d\sqrt{\frac{L}{\mu}}\right) = \widetilde{\mathcal{O}}\left(\mathfrak{m}_{\mathrm{AGD}}\right)$  for all  $r \in [0,1/2]$ .
Assume that  $r \in (1/2, 1]$ , then  $K_{\omega} = \min \{\lceil r / 1 - rK \rceil, d\}$  and  $K_{\alpha} = K$ . Using the same reasoning, we have
$$
K _ {\omega , \alpha} ^ {r} := (1 - r) K _ {\omega} + r K _ {\alpha} \leq (1 - r) \lceil r / 1 - r K \rceil + r K \leq 3 r K,
$$
$$
K _ {\omega , \alpha} ^ {r} := (1 - r) K _ {\omega} + r K _ {\alpha} \geq r K,
$$
$$
\mu_ {\omega , \alpha} ^ {r} := \frac {r d}{K _ {\omega , \alpha} ^ {r}} \leq \frac {d}{K},
$$
$$
\alpha \geq \frac {K}{d} \text {a n d} \omega \leq \max  \left\{\frac {(1 - r) d}{r K}, 1 \right\} - 1.
$$
By substituting these inequalities to (16), we obtain
$$
\mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = \widetilde {\mathcal {O}} \left(r K \left(\frac {d}{K} \sqrt {\frac {L}{\mu}} + \frac {d}{K} \sqrt {\frac {(1 - r) L _ {\max } \omega}{r n \mu}} + \frac {d}{K} + \frac {(1 - r) d}{r K} + \frac {d}{K}\right) + d\right)
$$
Using Lemma 1.4, one can easily show that  $\mathfrak{m}_{\mathrm{realistic}}^r = \widetilde{\mathcal{O}}\left(d\sqrt{\frac{L}{\mu}}\right) = \widetilde{\mathcal{O}}\left(\mathfrak{m}_{\mathrm{AGD}}\right)$  for all  $r \in (1/2, 1]$ .
F Auxiliary Inequalities For  $\bar{L}$
We now prove useful bounds for  $\bar{L}$ .
Lemma F.1 (Auxillary Inequalities). Assume that the constraint (44) hold, and a constant  $c = 660508$ . Then
$$
\bar {L} \geq c \frac {L _ {\max} \omega p ^ {2}}{\beta^ {2} n} \quad (6 4) \quad \bar {L} \geq c \frac {L _ {\max} \omega p}{\beta n} \quad (6 5) \quad \bar {L} \geq c \frac {L _ {\max} \omega}{n} \tag {66}
$$
$$
\bar {L} \geq c \frac {\sqrt {L L _ {\operatorname* {m a x}}} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}} (6 7) \quad \bar {L} \geq c \frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}} (6 8) \quad \bar {L} \geq c \frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {n}} (6 9)
$$
$$
\bar {L} \geq c \frac {\widehat {L} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}} \quad (7 0) \quad \bar {L} \geq c \frac {\widehat {L} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}} \quad (7 1) \quad \bar {L} \geq c \frac {\widehat {L} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {n}} \quad (7 2)
$$
$$
\bar {L} \geq c \frac {\widehat {L} p \sqrt {\omega}}{\sqrt {\alpha} \beta \sqrt {n}} \quad (7 3) \quad \bar {L} \geq c \frac {\widehat {L} \sqrt {p} \sqrt {\omega}}{\sqrt {\alpha} \sqrt {\beta} \sqrt {n}} \quad (7 4)
$$
$$
\bar {L} \geq c \frac {\widehat {L} p \sqrt {\omega}}{\beta \sqrt {n}} \quad (7 5) \quad \bar {L} \geq c \frac {\widehat {L} \sqrt {p \omega}}{\sqrt {\beta n}} \tag {76}
$$
$$
\bar {L} \geq c \frac {L}{\alpha} \quad (7 7) \quad \bar {L} \geq c \frac {L p}{\alpha \tau} \quad (7 8) \quad \bar {L} \geq c L \tag {79}
$$
$$
\bar {L} \geq c \left(\frac {L \hat {L} ^ {2} \omega p ^ {4}}{\alpha^ {2} \beta^ {2} n \tau^ {2}}\right) ^ {1 / 3} \quad (8 0) \quad \bar {L} \geq c \left(\frac {L \hat {L} ^ {2} \omega p ^ {3}}{\alpha^ {2} \beta n \tau^ {2}}\right) ^ {1 / 3} \tag {81}
$$
Proof. The inequalities (64) and (66) follow from (44). The inequality (65) follows from (64) and (66):
$$
c \frac {L _ {\max} \omega p}{\beta n} \leq c \frac {L _ {\max} \omega}{n} \left(\frac {1}{2} \times \frac {p ^ {2}}{\beta^ {2}} + \frac {1}{2} \times 1 ^ {2}\right) = \frac {c}{2} \times \frac {L _ {\max} \omega p ^ {2}}{\beta^ {2} n} + \frac {c}{2} \times \frac {L _ {\max} \omega}{n} \leq \bar {L}.
$$
The inequalities (67) and (68) follow from (44). The inequality (69) follows from (68) and  $\beta \in (0,1]$ :
$$
\bar {L} \geq c \frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}} \geq c \frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {n}}.
$$
Using Lemma 1.4, (67), (68), and (69), the inequalities (70), (71), and (72) follow from
$$
\bar {L} \geq c \frac {\sqrt {L L _ {\mathrm {m a x}}} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}} \geq c \frac {\hat {L} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}},
$$
$$
\bar {L} \geq c \frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}} \geq c \frac {\hat {L} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}},
$$
$$
\bar {L} \geq c \frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {n}} \geq c \frac {\hat {L} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {n}}.
$$
Next, using Lemma 1.4, and  $\frac{x + y}{2} \geq \sqrt{xy}$  for all  $x, y \geq 0$ , the inequality (73) follows from
$$
c \frac {\widehat {L} p \sqrt {\omega}}{\sqrt {\alpha} \beta \sqrt {n}} \leq c \frac {\sqrt {L L _ {\max}} p \sqrt {\omega}}{\sqrt {\alpha} \beta \sqrt {n}} \leq \frac {c}{2} \times \frac {L}{\alpha} + \frac {c}{2} \times \frac {L _ {\max} p ^ {2} \omega}{\beta^ {2} n} \stackrel {(4 4)} {\leq} \bar {L}.
$$
The inequality (74) follows from
$$
c \frac {\widehat {L} \sqrt {p} \sqrt {\omega}}{\sqrt {\alpha} \sqrt {\beta} \sqrt {n}} \leq c \frac {\sqrt {L L _ {\max}} \sqrt {p} \sqrt {\omega}}{\sqrt {\alpha} \sqrt {\beta} \sqrt {n}} \leq \frac {c}{2} \times \frac {L}{\alpha} + \frac {c}{2} \times \frac {L _ {\max} p \omega}{\beta n} (4 4), (6 5) \bar {L}.
$$
The inequalities (75) and (76) follow from (73), (74), and  $\alpha \in (0,1]$ :
$$
\bar {L} \geq c \frac {\widehat {L} p \sqrt {\omega}}{\sqrt {\alpha} \beta \sqrt {n}} \geq c \frac {\widehat {L} p \sqrt {\omega}}{\beta \sqrt {n}},
$$
$$
\bar {L} \geq c \frac {\widehat {L} \sqrt {p} \sqrt {\omega}}{\sqrt {\alpha} \sqrt {\beta} \sqrt {n}} \geq c \frac {\widehat {L} \sqrt {p} \sqrt {\omega}}{\sqrt {\beta} \sqrt {n}}.
$$
The inequalities (77) and (78) follow from (44), and (79) follows from (77) and  $\alpha \in (0,1]$ . Using Lemma 1.4, and  $\frac{x + y + z}{3} \geq (xyz)^{1/3}$  for all  $x,y,z \geq 0$ , the inequalities (80) and (81) follow from
$$
c \left(\frac {L \widehat {L} ^ {2} \omega p ^ {4}}{\alpha^ {2} \beta^ {2} n \tau^ {2}}\right) ^ {1 / 3} \leq c \left(\frac {L ^ {2} L _ {\max} \omega p ^ {4}}{\alpha^ {2} \beta^ {2} n \tau^ {2}}\right) ^ {1 / 3} \leq \frac {c}{3} \times \frac {L p}{\alpha \tau} + \frac {c}{3} \times \frac {L p}{\alpha \tau} + \frac {c}{3} \times \frac {L _ {\max} \omega p ^ {2}}{\beta^ {2} n} (4 4) \leq \bar {L},
$$
$$
c \left(\frac {L \widehat {L} ^ {2} \omega p ^ {3}}{\alpha^ {2} \beta n \tau^ {2}}\right) ^ {1 / 3} \leq c \left(\frac {L ^ {2} L _ {\max } \omega p ^ {3}}{\alpha^ {2} \beta n \tau^ {2}}\right) ^ {1 / 3} \leq \frac {c}{3} \times \frac {L p}{\alpha \tau} + \frac {c}{3} \times \frac {L p}{\alpha \tau} + \frac {c}{3} \times \frac {L _ {\max } \omega p}{\beta n} (4 4), (6 5) \leq \bar {L}.
$$
G Proof of Lemma E.10 (First Symbolically Computed)
We use the notations from the proof of Theorem E.9.
Lemma E.10 (First Symbolically Computed). Assume that for the parameter  $\bar{L}$ , the inequalities from Sections I and J hold. Then, for all  $t \geq 0$ , exists  $\rho$  in (90),  $\kappa$  in (88),  $\lambda$  in (82), and  $\nu_{t}$  in (83) such that (46) holds.
Proof. The inequalities (46) are equivalent to
$$
\frac {8 \theta_ {t + 1} ^ {2}}{\alpha} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \leq \nu_ {t},
$$
$$
\nu_ {t} \frac {8}{\alpha p} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \leq \rho ,
$$
$$
\rho \frac {8 p}{\tau} \leq \lambda ,
$$
$$
\frac {8 p \omega}{n \bar {L} \beta} + \nu_ {t} \frac {8 \omega}{n \beta} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \lambda \frac {8 \tau^ {2} \omega}{n \beta} \leq \kappa .
$$
Let us take
$$
\lambda := \rho \frac {8 p}{\tau} \tag {82}
$$
to ensure that the third inequality holds. It left to find the parameters such that
$$
\frac {8 \theta_ {t + 1} ^ {2}}{\alpha} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \rho \frac {8 p}{\tau} \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \leq \nu_ {t},
$$
$$
\frac {8 p \omega}{n \bar {L} \beta} + \nu_ {t} \frac {8 \omega}{n \beta} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho \frac {8 p}{\tau} \cdot \frac {8 \tau^ {2} \omega}{n \beta} \leq \kappa .
$$
$$
\nu_ {t} := \theta_ {t + 1} ^ {2} \widehat {\nu} (\kappa , \rho), \tag {83}
$$
where we additionally define
$$
\widehat {\nu} \equiv \widehat {\nu} (\kappa , \rho) := \frac {8}{\alpha} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \rho \frac {8 p}{\tau} \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right), \tag {84}
$$
to ensure that the first inequality holds. It left to find the parameters  $\kappa$  and  $\rho$  such that
$$
\widehat {\nu} (\kappa , \rho) \frac {8}{\alpha p} \left(\frac {\gamma_ {t + 1} \theta_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \leq \rho ,
$$
$$
\frac {8 p \omega}{n \bar {L} \beta} + \widehat {\nu} (\kappa , \rho) \frac {8 \omega}{n \beta} \left(\frac {\gamma_ {t + 1} \theta_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho \frac {8 p}{\tau} \cdot \frac {8 \tau^ {2} \omega}{n \beta} \leq \kappa .
$$
Using Lemma E.1, we have  $\frac{\gamma_{t + 1}\theta_{t + 1}}{\overline{L} + \Gamma_{t + 1}\mu} \leq \frac{\gamma_{t + 1}\theta_{t + 1}}{\overline{L} + \Gamma_{t}\mu} \leq \frac{1}{\overline{L}}$ , so it is sufficient to show that stronger inequalities hold:
$$
\widehat {\nu} (\kappa , \rho) \frac {8}{\alpha p \bar {L} ^ {2}} \leq \rho , \tag {85}
$$
$$
\frac {8 p \omega}{n \bar {L} \beta} + \hat {\nu} (\kappa , \rho) \frac {8 \omega}{n \beta \bar {L} ^ {2}} + \rho \frac {8 p}{\tau} \cdot \frac {8 \tau^ {2} \omega}{n \beta} \leq \kappa . \tag {86}
$$
From this point all formulas in this lemma are generated by the script from Section P (see Section 4 in Section P). We use the SymPy library (Meurer et al., 2017).
Using the definition of  $\widehat{\nu}$ , the left hand side of (86) equals
$$
\begin{array}{l} \frac {2 0 4 8 L ^ {2} \omega p ^ {3} \rho}{\bar {L} ^ {2} \alpha \beta n \tau^ {2}} + \frac {1 0 2 4 L ^ {2} \omega p ^ {2} \rho}{\bar {L} ^ {2} \alpha \beta n \tau} + \frac {2 5 6 L ^ {2} \omega p \rho}{\bar {L} ^ {2} \alpha \beta n} + \frac {3 2 L \omega p}{\bar {L} ^ {2} \alpha \beta n} \tag {87} \\ + \kappa \left(\frac {2 5 6 \hat {L} ^ {2} \omega p}{\bar {L} ^ {2} \alpha \beta n} + \frac {2 5 6 \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {2} \alpha \beta^ {2} n}\right) + \frac {6 4 \omega p \rho \tau}{\beta n} + \frac {8 \omega p}{\bar {L} \beta n} + \frac {2 0 4 8 \hat {L} ^ {2} \omega^ {2} p ^ {2} \rho \tau}{\bar {L} ^ {2} \alpha \beta n ^ {2}}, \\ \end{array}
$$
where we grouped the terms w.r.t.  $\kappa$ . Let us take  $\bar{L}$  such that the bracket is less or equal to  $1/2$ . We define the constraints in Section I. Therefore, (86) holds if
$$
\kappa := \frac {4 0 9 6 L ^ {2} \omega p ^ {3} \rho}{\bar {L} ^ {2} \alpha \beta n \tau^ {2}} + \frac {2 0 4 8 L ^ {2} \omega p ^ {2} \rho}{\bar {L} ^ {2} \alpha \beta n \tau} + \frac {5 1 2 L ^ {2} \omega p \rho}{\bar {L} ^ {2} \alpha \beta n} + \frac {6 4 L \omega p}{\bar {L} ^ {2} \alpha \beta n} + \frac {1 2 8 \omega p \rho \tau}{\beta n} + \frac {1 6 \omega p}{\bar {L} \beta n} + \frac {4 0 9 6 \hat {L} ^ {2} \omega^ {2} p ^ {2} \rho \tau}{\bar {L} ^ {2} \alpha \beta n ^ {2}}.
$$
Using the definition of  $\widehat{\nu}$  and  $\kappa$ , the left hand side of (85) equals
$$
\begin{array}{l} \frac {3 2 L}{\bar {L} ^ {2} \alpha^ {2}} + \frac {1 6 3 8 4 L \hat {L} ^ {2} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} + \frac {1 6 3 8 4 L \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} \\ + \rho \left(\frac {2 0 4 8 L ^ {2} p ^ {2}}{\bar {L} ^ {2} \alpha^ {2} \tau^ {2}} + \frac {1 0 2 4 L ^ {2} p}{\bar {L} ^ {2} \alpha^ {2} \tau} + \frac {2 5 6 L ^ {2}}{\bar {L} ^ {2} \alpha^ {2}} + \frac {1 0 4 8 5 7 6 L ^ {2} \hat {L} ^ {2} \omega p ^ {3}}{\bar {L} ^ {4} \alpha^ {3} \beta n \tau^ {2}} + \frac {5 2 4 2 8 8 L ^ {2} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta n \tau}\right) \\ + \frac {1 3 1 0 7 2 L ^ {2} \hat {L} ^ {2} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} + \frac {1 0 4 8 5 7 6 L ^ {2} \hat {L} ^ {2} \omega p ^ {4}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n \tau^ {2}} + \frac {5 2 4 2 8 8 L ^ {2} \hat {L} ^ {2} \omega p ^ {3}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n \tau} \tag {89} \\ + \frac {1 3 1 0 7 2 L ^ {2} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} + \frac {2 0 4 8 \hat {L} ^ {2} \omega p \tau}{\bar {L} ^ {2} \alpha^ {2} n} + \frac {3 2 7 6 8 \hat {L} ^ {2} \omega p \tau}{\bar {L} ^ {2} \alpha^ {2} \beta n} + \frac {3 2 7 6 8 \hat {L} ^ {2} \omega p ^ {2} \tau}{\bar {L} ^ {2} \alpha^ {2} \beta^ {2} n} \\ \left. + \frac {1 0 4 8 5 7 6 \hat {L} ^ {4} \omega^ {2} p ^ {2} \tau}{\bar {L} ^ {4} \alpha^ {3} \beta n ^ {2}} + \frac {1 0 4 8 5 7 6 \hat {L} ^ {4} \omega^ {2} p ^ {3} \tau}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n ^ {2}}\right) + \frac {4 0 9 6 \hat {L} ^ {2} \omega p}{\bar {L} ^ {3} \alpha^ {2} \beta n} + \frac {4 0 9 6 \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {3} \alpha^ {2} \beta^ {2} n}, \\ \end{array}
$$
where we grouped the terms w.r.t.  $\rho$ . Let us take  $\bar{L}$  such that the bracket is less or equal to  $1/2$ . We define the constraints in Section J. Therefore, (85) holds if
$$
\rho := \frac {6 4 L}{\bar {L} ^ {2} \alpha^ {2}} + \frac {3 2 7 6 8 L \hat {L} ^ {2} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} + \frac {3 2 7 6 8 L \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} + \frac {8 1 9 2 \hat {L} ^ {2} \omega p}{\bar {L} ^ {3} \alpha^ {2} \beta n} + \frac {8 1 9 2 \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {3} \alpha^ {2} \beta^ {2} n}. \tag {90}
$$
Finally, under the constraints from Sections I and J on  $\bar{L}$ , the choices of parameters (90), (88), (83) and (82) insure that (46) holds.
H Proof of Lemma E.11 (Second Symbolically Computed)
Proof. Since  $p \geq 0$  and  $D_{f}(z^{t},y^{t + 1}) \geq 0$  for all  $t \geq 0$ , the inequality (47) is satisfied if
$$
\begin{array}{l} \frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max}}{n}\right) + \theta_ {t + 1} - 1 \leq 0. \\ \end{array}
$$
Note that  $\theta_{t + 1}\leq \frac{1}{4}$  for all  $t\geq 0$  . Therefore, it is sufficient to show that
$$
\begin{array}{l} \frac {4 \omega L _ {\operatorname* {m a x}}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\operatorname * {m a x}} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\operatorname* {m a x}}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\mathrm {m a x}}}{n}\right) \leq \frac {3}{4}. \\ \end{array}
$$
In the view of (83), we have to show that
$$
\begin{array}{l} \frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max } + \widehat {\nu} \left(\frac {\gamma_ {t + 1} \theta_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max }}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\mathrm {m a x}}}{n}\right) \leq \frac {3}{4}. \\ \end{array}
$$
Using Lemma E.1, we have  $\frac{\gamma_{t + 1}\theta_{t + 1}}{L + \Gamma_{t + 1}\mu} \leq \frac{\gamma_{t + 1}\theta_{t + 1}}{L + \Gamma_t\mu} \leq \frac{1}{L}$ , so it is sufficient to show that
$$
\begin{array}{l} \frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max } + \frac {\widehat {\nu}}{\bar {L} ^ {2}} \left(\frac {4 \omega L _ {\max }}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \tag {91} \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max}}{n}\right) \leq \frac {3}{4}. \\ \end{array}
$$
From this point all formulas in this lemma are generated by the script from Section P (see Section 5 in Section P).
Let us substitute (90), (88), (82), and (84) to the last inequality and obtain the inequality from Section K. The conditions from Section L insure that the inequality from Section K holds. It left to prove (48). Since  $p \geq 0$ ,  $\mathbb{E}_t\left[\| u^{t + 1} - u^t\|^2\right] \geq 0$  and  $\theta_{t + 1}^2 \geq 0$  for all  $t \geq 0$ , the inequality (48) holds if
$$
\frac {4}{\bar {L}} \left(\frac {L}{2} + \kappa 4 \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 L ^ {2} + \lambda \left(2 \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \leq 1. \tag {92}
$$
Let us substitute (90), (88) and (82) to the last inequality and obtain the inequality from Section M. The inequality from Section M holds if  $\bar{L}$  satisfy the inequalities from Section N.
I Symbolically Computed Constraints for  $\bar{L}$  Such That The Term w.r.t.  $\kappa$  is less or equal  $1/2$  in (87)
$$
\frac {2 5 6 \hat {L} ^ {2} \omega p}{\bar {L} ^ {2} \alpha \beta n} \leq \frac {1}{4} \tag {94}
$$
J Symbolically Computed Constraints for  $\bar{L}$  Such That The Term w.r.t.  $\rho$  is less or equal  $1/2$  in (89)
$$
\frac {2 5 6 L ^ {2}}{\bar {L} ^ {2} \alpha^ {2}} \leq \frac {1}{2 8} \quad (9 5) \quad \frac {1 0 2 4 L ^ {2} p}{\bar {L} ^ {2} \alpha^ {2} \tau} \leq \frac {1}{2 8} \quad (9 6) \quad \frac {2 0 4 8 L ^ {2} p ^ {2}}{\bar {L} ^ {2} \alpha^ {2} \tau^ {2}} \leq \frac {1}{2 8} \tag {97}
$$
$$
\frac {2 0 4 8 \hat {L} ^ {2} \omega p \tau}{\bar {L} ^ {2} \alpha^ {2} n} \leq \frac {1}{2 8} \quad (9 8) \quad \frac {3 2 7 6 8 \hat {L} ^ {2} \omega p \tau}{\bar {L} ^ {2} \alpha^ {2} \beta n} \leq \frac {1}{2 8} \quad (9 9) \quad \frac {3 2 7 6 8 \hat {L} ^ {2} \omega p ^ {2} \tau}{\bar {L} ^ {2} \alpha^ {2} \beta^ {2} n} \leq \frac {1}{2 8} \tag {100}
$$
$$
\frac {1 3 1 0 7 2 L ^ {2} \hat {L} ^ {2} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} \leq \frac {1}{2 8} \quad (1 0 1) \quad \frac {1 3 1 0 7 2 L ^ {2} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} \leq \frac {1}{2 8} \quad (1 0 2) \quad \frac {1 0 4 8 5 7 6 \hat {L} ^ {4} \omega^ {2} p ^ {2} \tau}{\bar {L} ^ {4} \alpha^ {3} \beta n ^ {2}} \leq \frac {1}{2 8} \quad (1 0 3)
$$
$$
\frac {1 0 4 8 5 7 6 \hat {L} ^ {4} \omega^ {2} p ^ {3} \tau}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n ^ {2}} \leq \frac {1}{2 8} \quad (1 0 4) \quad \frac {5 2 4 2 8 8 L ^ {2} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta n \tau} \leq \frac {1}{2 8} \quad (1 0 5) \quad \frac {5 2 4 2 8 8 L ^ {2} \hat {L} ^ {2} \omega p ^ {3}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n \tau} \leq \frac {1}{2 8} \quad (1 0 6)
$$
$$
\frac {1 0 4 8 5 7 6 L ^ {2} \hat {L} ^ {2} \omega p ^ {3}}{\bar {L} ^ {4} \alpha^ {3} \beta n \tau^ {2}} \leq \frac {1}{2 8} \quad (1 0 7) \quad \frac {1 0 4 8 5 7 6 L ^ {2} \hat {L} ^ {2} \omega p ^ {4}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n \tau^ {2}} \leq \frac {1}{2 8} \quad (1 0 8)
$$
K Symbolically Computed Expression (91)
$$
\begin{array}{l} \frac {5 4 4 L ^ {2}}{\bar {L} ^ {2} \alpha^ {2}} + \frac {1 6 3 8 4 L ^ {4}}{\bar {L} ^ {4} \alpha^ {4}} + \frac {4 L _ {\max} \omega}{\bar {L} n} \\ + \frac {2 0 4 8 L ^ {2} p}{\bar {L} ^ {2} \alpha^ {2} \tau} + \frac {4 0 9 6 L ^ {2} p ^ {2}}{\bar {L} ^ {2} \alpha^ {2} \tau^ {2}} + \frac {6 5 5 3 6 L ^ {4} p}{\bar {L} ^ {4} \alpha^ {4} \tau} \\ + \frac {1 3 1 0 7 2 L ^ {4} p ^ {2}}{\bar {L} ^ {4} \alpha^ {4} \tau^ {2}} + \frac {1 6 L L _ {\max} \omega}{\bar {L} ^ {2} \alpha n} + \frac {1 2 8 L _ {\max} \omega p}{\bar {L} \beta n} \\ + \frac {1 2 8 L _ {\max} \omega p ^ {2}}{\bar {L} \beta^ {2} n} + \frac {8 1 9 2 L ^ {3} L _ {\max} \omega}{\bar {L} ^ {4} \alpha^ {3} n} + \frac {5 1 2 L L _ {\max} \omega p}{\bar {L} ^ {2} \alpha \beta n} \\ + \frac {5 1 2 L L _ {\max} \omega p ^ {2}}{\bar {L} ^ {2} \alpha \beta^ {2} n} + \frac {2 0 4 8 L _ {\max} \hat {L} ^ {2} \omega^ {2} p}{\bar {L} ^ {3} \alpha \beta n ^ {2}} + \frac {2 0 4 8 L _ {\max} \hat {L} ^ {2} \omega^ {2} p ^ {2}}{\bar {L} ^ {3} \alpha \beta^ {2} n ^ {2}} \\ + \frac {4 0 9 6 L L _ {\operatorname* {m a x}} \omega p \tau}{\bar {L} ^ {2} \alpha^ {2} n} + \frac {3 2 7 6 8 L ^ {3} L _ {\operatorname* {m a x}} \omega p}{\bar {L} ^ {4} \alpha^ {3} n \tau} + \frac {6 5 5 3 6 L ^ {3} L _ {\operatorname* {m a x}} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} n \tau^ {2}} \\ + \frac {6 9 6 3 2 L \hat {L} ^ {2} \omega p}{\bar {L} ^ {3} \alpha^ {2} \beta n} + \frac {6 9 6 3 2 L \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {3} \alpha^ {2} \beta^ {2} n} + \frac {1 3 1 0 7 2 L ^ {2} \hat {L} ^ {2} \omega p \tau}{\bar {L} ^ {4} \alpha^ {4} n} \\ + \frac {2 6 2 1 4 4 L ^ {3} L _ {\max} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} + \frac {2 6 2 1 4 4 L ^ {3} L _ {\max} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} + \frac {2 7 8 5 2 8 L ^ {2} \hat {L} ^ {2} \omega p}{\bar {L} ^ {4} \alpha^ {3} \beta n} \\ + \frac {2 7 8 5 2 8 L ^ {2} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {4} \alpha^ {3} \beta^ {2} n} + \frac {2 0 9 7 1 5 2 L ^ {3} \hat {L} ^ {2} \omega p}{\bar {L} ^ {5} \alpha^ {4} \beta n} + \frac {2 0 9 7 1 5 2 L ^ {3} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {5} \alpha^ {4} \beta^ {2} n} \\ + \frac {1 6 7 7 7 2 1 6 L ^ {4} \hat {L} ^ {2} \omega p}{\bar {L} ^ {6} \alpha^ {5} \beta n} + \frac {1 6 7 7 7 2 1 6 L ^ {4} \hat {L} ^ {2} \omega p ^ {2}}{\bar {L} ^ {6} \alpha^ {5} \beta^ {2} n} + \frac {1 0 7 3 7 4 1 8 2 4 L ^ {3} \hat {L} ^ {4} \omega^ {2} p ^ {4}}{\bar {L} ^ {7} \alpha^ {5} \beta^ {4} n ^ {2}} \\ + \frac {1 0 7 3 7 4 1 8 2 4 L ^ {3} \hat {L} ^ {4} \omega^ {2} p ^ {2}}{\bar {L} ^ {7} \alpha^ {5} \beta^ {2} n ^ {2}} + \frac {2 1 4 7 4 8 3 6 4 8 L ^ {3} \hat {L} ^ {4} \omega^ {2} p ^ {3}}{\bar {L} ^ {7} \alpha^ {5} \beta^ {3} n ^ {2}} + \frac {4 2 9 4 9 6 7 2 9 6 L ^ {4} \hat {L} ^ {4} \omega^ {2} p ^ {4}}{\bar {L} ^ {8} \alpha^ {6} \beta^ {4} n ^ {2}} \\ \end{array}
$$
L Symbolically Computed Constraints for  $\bar{L}$  Such That The Inequality from Section K Holds
$$
\frac {5 4 4 L ^ {2}}{\bar {L} ^ {2} \alpha^ {2}} \leq \frac {1}{3 2 6} \quad (1 0 9) \quad \frac {1 6 3 8 4 L ^ {4}}{\bar {L} ^ {4} \alpha^ {4}} \leq \frac {1}{3 2 6} \quad (1 1 0) \quad \frac {4 L _ {\max } \omega}{\bar {L} n} \leq \frac {1}{3 2 6} \tag {111}
$$