SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \leq \Gamma_ {0} \left(\mathbb {E} \left[ f (z ^ {0}) - f (x ^ {*}) \right] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (z ^ {0}) \right\| ^ {2} \right] + \nu_ {0} \mathbb {E} \left[ \| w ^ {0} - u ^ {0} \| ^ {2} \right] \right. \\ \left. + \rho \mathbb {E} \left[ \left\| k ^ {0} - \nabla f (z ^ {0}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {0} - \nabla f (z ^ {0}) \right\| ^ {2} \right]\right) \\ \left. \right. + \left(\frac {\bar {L} + \Gamma_ {0} \mu}{2} \mathbb {E} \left[\left\| u ^ {0} - x ^ {*} \right\| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {T} \mu}{2} \mathbb {E} \left[\left\| u ^ {T} - x ^ {*} \right\| ^ {2} \right]\right). \\ \end{array}
$$
Using the initial conditions and the non-negativity of the terms, we get
$$
\Gamma_ {T} \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] + \frac {\Gamma_ {T} \mu}{2} \mathbb {E} \left[ \| u ^ {T} - x ^ {*} \| ^ {2} \right] \leq \Gamma_ {0} \left(f (z ^ {0}) - f (x ^ {*})\right) + \frac {\bar {L} + \Gamma_ {0} \mu}{2} \| u ^ {0} - x ^ {*} \| ^ {2}.
$$
Using Lemma E.1, we have
$$
\begin{array}{l} \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] + \frac {\mu}{2} \mathbb {E} \left[ \left\| u ^ {T} - x ^ {*} \right\| ^ {2} \right] \\ \leq \exp \left(- T \min  \left\{\sqrt {\frac {p \mu}{4 \bar {L}}}, p \theta_ {\min } \right\}\right) \left(2 \left(f (z ^ {0}) - f (x ^ {*})\right) + \left(\frac {\bar {L}}{\Gamma_ {0}} + \mu\right) \| u ^ {0} - x ^ {*} \| ^ {2}\right). \\ \end{array}
$$
It is left to use the definitions of  $\bar{L}$  and  $\theta_{\mathrm{min}}$
E.6 General convex case
Let us use Theorem E.9 to analyze the general convex case ( $\mu$  can possibly be equal to zero):
Theorem E.14. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let
$$
\bar {L} = 6 6 0 5 0 8 \times \max \left\{\frac {L}{\alpha}, \frac {L p}{\alpha \tau}, \frac {\sqrt {L L _ {\mathrm {m a x}}} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}}, \frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}}, \frac {L _ {\mathrm {m a x}} \omega p ^ {2}}{\beta^ {2} n}, \frac {L _ {\mathrm {m a x}} \omega}{n} \right\}, (5 1)
$$
$$
\begin{array}{l} \beta = \frac {1}{\omega + 1},   \theta_ {\min } = \frac {1}{4} \min  \left\{1, \frac {\alpha}{p}, \frac {\tau}{p}, \frac {\beta}{p} \right\},   h _ {i} ^ {0} = \nabla f _ {i} (z ^ {0}) f o r a l l i \in [ n ],   w ^ {0} = u ^ {0},   k ^ {0} = \nabla f (z ^ {0}), \\ v ^ {0} = \nabla f (z ^ {0}), a n d \Gamma_ {0} \in [ 1, \bar {L} / L ]. T h e n A l g o r i t h m 1 r e t u r n s \varepsilon - s o l u t i o n, i . e ., \mathbb {E} \left[ f (z ^ {T}) \right] - f (x ^ {*}) \leq \varepsilon , \\ a f t e r \end{array}
$$
$$
T = \left\{ \begin{array}{l l} \Theta \left(\frac {1}{p \theta_ {\operatorname* {m i n}}} \log \frac {\bar {L} \left\| z ^ {0} - x ^ {*} \right\| ^ {2}}{\Gamma_ {0} \varepsilon}\right), & \frac {\bar {L} \left\| z ^ {0} - x ^ {*} \right\| ^ {2}}{\varepsilon} <   \frac {1}{p \theta_ {\operatorname* {m i n}} ^ {2}} \\ \Theta \left(\max  \left\{\frac {1}{p \theta_ {\operatorname* {m i n}}} \log \frac {1}{\Gamma_ {0} p \theta_ {\operatorname* {m i n}} ^ {2}}, 0 \right\} + Q \sqrt {\frac {\left\| z ^ {0} - x ^ {*} \right\| ^ {2}}{\varepsilon}}\right), & \text {o t h e r w i s e} \end{array} \right. \tag {52}
$$
iterations, where
$$
Q := \Theta \left(\max \left\{\sqrt {\frac {L}{\alpha p}}, \sqrt {\frac {L}{\alpha \tau}}, \sqrt {\frac {\sqrt {L L _ {\mathrm {m a x}}} (\omega + 1) \sqrt {\omega \tau}}{\alpha \sqrt {n}}}, \sqrt {\frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \sqrt {n}}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega (\omega + 1) ^ {2} p}{n}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega}{n p}} \right\}\right).
$$
Remark E.15. One can see that the optimal  $\Gamma_0$  in (52) from Theorem E.14 equals  $\Gamma_0 = \bar{L} / L$ . But the dependence on  $\Gamma_0$  is under the logarithm, so if the dependence on the logarithm is not critical, one can take any  $\Gamma_0 \in [1, \bar{L} / L]$ .
$$
\begin{array}{l} \Gamma_ {T} \left(\mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {T} - \nabla f _ {i} (z ^ {T}) \| ^ {2} \right] + \nu_ {T} \mathbb {E} \left[ \| w ^ {T} - u ^ {T} \| ^ {2} \right] \right. \\ \left. + \rho \mathbb {E} \left[ \left\| k ^ {T} - \nabla f (z ^ {T}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {T} - \nabla f (z ^ {T}) \right\| ^ {2} \right]\right) \\ \leq \Gamma_ {0} \left(\mathbb {E} \left[ f (z ^ {0}) - f (x ^ {*}) \right] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (z ^ {0}) \right\| ^ {2} \right] + \nu_ {0} \mathbb {E} \left[ \| w ^ {0} - u ^ {0} \| ^ {2} \right] \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. + \rho \mathbb {E} \left[ \left\| k ^ {0} - \nabla f (z ^ {0}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {0} - \nabla f (z ^ {0}) \right\| ^ {2} \right]\right) \\ \left. \right. + \left(\frac {\bar {L} + \Gamma_ {0} \mu}{2} \mathbb {E} \left[\left\| u ^ {0} - x ^ {*} \right\| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {T} \mu}{2} \mathbb {E} \left[\left\| u ^ {T} - x ^ {*} \right\| ^ {2} \right]\right). \\ \end{array}
$$
Using the initial conditions, the non-negativity of the terms, and  $\Gamma_0\leq \bar{L} /L$  , we get
$$
\begin{array}{l} \Gamma_ {T} \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] \leq \Gamma_ {0} \left(f (z ^ {0}) - f (x ^ {*})\right) + \frac {\bar {L} + \Gamma_ {0} \mu}{2} \| u ^ {0} - x ^ {*} \| ^ {2} \\ \leq \frac {\bar {L}}{L} \left(f (z ^ {0}) - f (x ^ {*})\right) + \bar {L} \| z ^ {0} - x ^ {*} \| ^ {2}. \\ \end{array}
$$
Using the  $L$ -smoothness and  $\mu \leq L \leq \bar{L}$ , we have
$$
\Gamma_ {T} \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] \leq \bar {L} \left\| z ^ {0} - x ^ {*} \right\| ^ {2} + \bar {L} \left\| z ^ {0} - x ^ {*} \right\| ^ {2} \leq 2 \bar {L} \left\| z ^ {0} - x ^ {*} \right\| ^ {2}.
$$
$$
\left\{ \begin{array}{l l} \frac {\Gamma_ {0}}{2} \exp \left(t p \theta_ {\min }\right)   \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] \leq 2 \bar {L}   \left\| z ^ {0} - x ^ {*} \right\| ^ {2}, & t <   \bar {t} \\ \frac {p (t - \bar {t}) ^ {2}}{1 6}   \mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] \leq 2 \bar {L}   \left\| z ^ {0} - x ^ {*} \right\| ^ {2}, & t \geq \bar {t}, \end{array} \right.
$$
where  $\bar{t} := \max \left\{\left\lceil \frac{1}{p\theta_{\min}}\log \frac{1}{2\Gamma_0p\theta_{\min}^2}\right\rceil ,0\right\}$ . The last inequalities guarantees that Algorithm 1 returns  $\varepsilon$ -solution after (52) iterations.
E.7 Choosing optimal parameters
Proof. We implicitly assume that  $p \in (0,1]$  and  $\tau \in (0,1]$ . Using (12) and (4), we have
$$
\operatorname *{arg  min}_{p,\tau}\max_{L_{\max }\in [L,nL]}\mathfrak{m}^{r}_{\text{new}} = \operatorname *{arg  min}_{p,\tau}\max_{L_{\max }\in [L,nL]}\widetilde{\Theta}\left((1 - r)K_{\omega}T + r\left(K_{\alpha} + pd\right)T\right).
$$
Note that only  $T$  depends on  $L_{\mathrm{max}}$  and  $\tau$ . We have
$$
\min_{\tau}\max_{L_{\max}\in [L,nL]}T
$$
$$
\begin{array}{l} \stackrel {(1 0)} {=} \min _ {\tau} \max _ {L _ {\max } \in [ L, n L ]} \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L}{\alpha \tau \mu}}, \sqrt {\frac {\sqrt {L L _ {\max}} (\omega + 1) \sqrt {\omega \tau}}{\alpha \sqrt {n} \mu}}, \right. \right. \\ \sqrt {\frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \sqrt {n} \mu}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \Biggr \}) \\ \end{array}
$$
$$
\begin{array}{l} = \min _ {\tau} \widetilde {\Theta} \left(\max \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L}{\alpha \tau \mu}}, \sqrt {\frac {L (\omega + 1) \sqrt {\omega \tau}}{\alpha \mu}}, \sqrt {\frac {L \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \mu}}, \sqrt {\frac {L \omega (\omega + 1) ^ {2} p}{\mu}}, \sqrt {\frac {L \omega}{p \mu}}, \right. \right. \\ \left. \left. \frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \right\}\right) \\ \end{array}
$$
The last term attains the minimum when
$$
\tau = \min  \left\{\frac {1}{\omega + 1}, \frac {p ^ {1 / 3}}{(\omega + 1) ^ {2 / 3}} \right\}. \tag {53}
$$
Therefore, we get
$$
T^{\prime}:= \min_{\tau}\max_{L_{\max}\in [L,nL]}T
$$
$$
= \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L (\omega + 1)}{\alpha \mu}}, \sqrt {\frac {L (\omega + 1) ^ {2 / 3}}{\alpha p ^ {1 / 3} \mu}}, \sqrt {\frac {L \omega (\omega + 1) ^ {2} p}{\mu}}, \sqrt {\frac {L \omega}{p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\}\right),
$$
where we use that
$$
\frac {1}{\tau} = \max \left\{\omega + 1, \frac {(\omega + 1) ^ {2 / 3}}{p ^ {1 / 3}} \right\} \leq \max \left\{\omega + 1, \frac {2}{3} (\omega + 1) + \frac {1}{3 p} \right\} = \Theta \left(\max \left\{(\omega + 1), \frac {1}{p} \right\}\right).
$$
It is left to find
$$
\begin{array}{l} \underset {p} {\arg \min } \left(\underset {\tau} {\min } \underset {L _ {\max } \in [ L, n L ]} {\max } \mathfrak {m} _ {\text {n e w}} ^ {r}\right) = \underset {p} {\arg \min } \widetilde {\Theta} \left((1 - r) K _ {\omega} T ^ {\prime} + r \left(K _ {\alpha} + p d\right) T ^ {\prime}\right) \\ = \underset {p} {\arg \min } \widetilde {\Theta} \left(A \times T ^ {\prime} + B \times p T ^ {\prime}\right), \tag {55} \\ \end{array}
$$
where  $A \coloneqq (1 - r)K_{\omega} + rK_{\alpha} \geq 0$  and  $B \coloneqq rd \geq 0$ . Note that  $A$  and  $B$  do not depend on  $p$ . If  $p \geq {}^A /B$ , then  $\widetilde{\Theta}\left(AT^{\prime} + BpT^{\prime}\right) = \widetilde{\Theta}\left(BpT^{\prime}\right)$ . The term  $pT^{\prime}$  is non-decreasing function w.r.t.  $p$ . For  $p \geq {}^A /B$ , it means that an optimal point  $p = {}^A /B$ . Thus the argmin (55) is equivalent to
$$
\operatorname * {a r g   m i n} _ {p \in Q _ {0}} \widetilde {\Theta} \left(A \times T ^ {\prime} + B \times p T ^ {\prime}\right) = \operatorname * {a r g   m i n} _ {p \in Q _ {0}} \widetilde {\Theta} \left(T ^ {\prime}\right),
$$
where  $Q_0 \coloneqq \left\{p \mid p \leq \frac{A}{B}\right\}$ . Thus, we have
$$
\underset {p \in Q _ {0}} {\arg \min } \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L (\omega + 1)}{\alpha \mu}}, \sqrt {\frac {L (\omega + 1) ^ {2 / 3}}{\alpha p ^ {1 / 3} \mu}}, \sqrt {\frac {L \omega (\omega + 1) ^ {2} p}{\mu}}, \sqrt {\frac {L \omega}{p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\}\right).
$$
The next observation is that this argmin is non-decreasing when  $p \geq 1 / \omega + 1$ . It means that the minimum attains at some point  $p \in Q_1 := \{p \mid p \leq 1 / \omega + 1, p \in Q_0\}$ . Using this information, we can eliminate the redundant terms (for instance,  $A \sqrt{\frac{L(\omega + 1)}{\alpha \mu}} \leq A \sqrt{\frac{L}{\alpha p \mu}}$  for  $p \in Q_1$ ) and get an equivalent argmin
$$
\underset {p \in Q _ {1}} {\arg \min } \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L \omega}{p \mu}}, \frac {1}{\alpha}, \frac {1}{p} \right\}\right),
$$
The last term attains the minimum at
$$
p = \min  \left\{\frac {1}{\omega + 1}, \frac {A}{B} \right\}. \tag {56}
$$
Using (56) and (53), an optimal  $\tau$  is
$$
\tau = \frac {p ^ {1 / 3}}{(\omega + 1) ^ {2 / 3}}. \tag {57}
$$
We substitute (56) and (57) to (10) and obtain (13). We use Lemma 1.4 to eliminate the redundant terms. Note that  $\mu_{\omega, \alpha}^{r} \coloneqq B / A$ .
Using (12) and (56), we have
$$
\mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = (1 - r) K _ {\omega} T _ {\text {r e a l i s t i c}} + r \left(K _ {\alpha} + p d\right) T _ {\text {r e a l i s t i c}} + d
$$
$$
\begin{array}{l} \leq (1 - r) K _ {\omega} T _ {\text {r e a l i s t i c}} + r \left(K _ {\alpha} + \frac {(1 - r) K _ {\omega} + r K _ {\alpha}}{r d} d\right) T _ {\text {r e a l i s t i c}} + d \\ = 2 (1 - r) K _ {\omega} T _ {\text {r e a l i s t i c}} + 2 r K _ {\alpha} T _ {\text {r e a l i s t i c}} + d. \\ \end{array}
$$
Note that
$$
\begin{array}{l} \mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = (1 - r) K _ {\omega} T _ {\text {r e a l i s t i c}} + r \left(K _ {\alpha} + p d\right) T _ {\text {r e a l i s t i c}} + d \\ \geq (1 - r) K _ {\omega} T _ {\text {r e a l i s t i c}} + r K _ {\alpha} T _ {\text {r e a l i s t i c}} + d. \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\text {o p t i m i s t i c}} = \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L \operatorname* {m a x} \left\{1 , \mu_ {\omega , \alpha} ^ {r} \right\}}{\alpha \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\max } ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \sqrt {\frac {L ^ {1 / 2} L _ {\max } ^ {1 / 2} (\omega + 1) ^ {3 / 2}}{\sqrt {\alpha n} \mu}}, \right. \right. \tag {14} \\ \left. \sqrt {\frac {L _ {\operatorname* {m a x}} \omega \operatorname* {m a x} \{\omega + 1 , \mu_ {\omega , \alpha} ^ {r} \}}{n \mu}}, \frac {1}{\alpha}, (\omega + 1), \mu_ {\omega , \alpha} ^ {r} \right\}\Biggr). \\ \end{array}
$$
The total communication complexity thus equals  $\mathfrak{m}_{\mathrm{optimistic}}^r = \widetilde{\Theta} \left( ((1 - r) K_\omega + r K_\alpha) T_{\mathrm{optimistic}} + d \right)$ .
Proof. We implicitly assume that  $p \in (0,1]$  and  $\tau \in (0,1]$ . We start the proof as in Theorem 5.2. Using (12) and (4), we have
$$
\operatorname *{arg  min}_{p,\tau}\mathfrak{m}^{r}_{\mathrm{new}} = \operatorname *{arg  min}_{p,\tau}\widetilde{\Theta}\left((1 - r)K_{\omega}T + r\left(K_{\alpha} + pd\right)T\right).
$$
Unlike Theorem 5.2, we know the ratio  $L_{\mathrm{max}} / L$ , thus
min T
$$
\begin{array}{l} \stackrel {(1 0)} {=} \min _ {\tau} \widetilde {\Theta} \bigg (\max \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L}{\alpha \tau \mu}}, \sqrt {\frac {\sqrt {L L _ {\max}} (\omega + 1) \sqrt {\omega \tau}}{\alpha \sqrt {n} \mu}}, \right. \\ \sqrt {\frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \sqrt {n \mu}}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \Biggr \}) \\ \end{array}
$$
$$
\tau = \min  \left\{1, \left(\frac {L n}{L _ {\max }}\right) ^ {1 / 3} \min  \left\{\frac {1}{\omega + 1}, \frac {p ^ {1 / 3}}{(\omega + 1) ^ {2 / 3}} \right\} \right\}. \tag {58}
$$
$$
\begin{array}{l} T ^ {\prime} := \min  _ {\tau} T \\ = \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\operatorname* {m a x}} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\operatorname* {m a x}} ^ {1 / 3} (\omega + 1) ^ {2 / 3}}{\alpha n ^ {1 / 3} p ^ {1 / 3} \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, \omega + 1, \frac {1}{p} \right\}\right). \tag {59} \\ \end{array}
$$
$$
\begin{array}{l} \arg \min  _ {p} \left(\min  _ {\tau} \mathfrak {m} _ {\mathrm {n e w}} ^ {r}\right) = \arg \min  _ {p} \widetilde {\Theta} \left((1 - r) K _ {\omega} T ^ {\prime} + r \left(K _ {\alpha} + p d\right) T ^ {\prime}\right) \\ = \underset {p} {\arg \min } \widetilde {\Theta} (A \times T ^ {\prime} + B \times p T ^ {\prime}), \tag {60} \\ \end{array}
$$
where  $A \coloneqq (1 - r)K_{\omega} + rK_{\alpha} \geq 0$  and  $B \coloneqq rd \geq 0$ . Note that  $A$  and  $B$  do not depend on  $p$ . If  $p \geq {}^A /B$ , then  $\widetilde{\Theta}\left(AT^{\prime} + BpT^{\prime}\right) = \widetilde{\Theta}\left(BpT^{\prime}\right)$ . The term  $pT^{\prime}$  is non-decreasing function w.r.t.  $p$ . For  $p \geq {}^A /B$ , it means that an optimal point  $p = {}^A /B$ . Thus the argmin (60) is equivalent to
$$
\operatorname *{arg  min}_{p\in Q_{0}}\widetilde{\Theta}\left(A\times T^{\prime} + B\times pT^{\prime}\right) = \operatorname *{arg  min}_{p\in Q_{0}}\widetilde{\Theta}\left(T^{\prime}\right),
$$
where  $Q_0 \coloneqq \left\{p \mid p \leq \frac{A}{B}\right\}$ . Next, we have
$$
\begin{array}{l} \operatorname *{arg  min}_{p\in Q_{0}}\widetilde{\Theta}\left(T^{\prime}\right) \\ = \operatorname *{arg  min}_{p\in Q_{0}} \\ \end{array}
$$
$$
\widetilde {\Theta} \left(\max \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\max} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\max} ^ {1 / 3} (\omega + 1) ^ {2 / 3}}{\alpha n ^ {1 / 3} p ^ {1 / 3} \mu}}, \sqrt {\frac {L _ {\max} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\max} \omega}{n p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\}\right).
$$
For  $p \geq \left(\frac{Ln}{L_{\max}}\right)^{1/3} \frac{1}{\omega + 1}$ , using Lemma 1.4, we have  $p \geq \frac{1}{\omega + 1}$
$$
\begin{array}{l} \sqrt {\frac {L}{\alpha p \mu}} \leq \sqrt {\frac {L ^ {2 / 3} L _ {\max } ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \\ \sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1) ^ {2 / 3}}{\alpha n ^ {1 / 3} p ^ {1 / 3} \mu}} \leq \sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}} \\ \sqrt {\frac {L _ {\mathrm {m a x}} \omega}{n p \mu}} \leq \sqrt {\frac {L _ {\mathrm {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}} \\ \frac {1}{p} \leq \left(\frac {L _ {\operatorname* {m a x}}}{L n}\right) ^ {1 / 3} (\omega + 1) \leq (\omega + 1). \\ \end{array}
$$
It means that for  $p \geq \left( \frac{Ln}{L_{\max}} \right)^{1/3} \frac{1}{\omega + 1}$ , the argmin (61) is equivalent to
$$
\underset {p \in Q _ {0}} {\arg \min} \widetilde {\Theta} \bigg (\max \left\{\sqrt {\frac {L ^ {2 / 3} L _ {\max} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \sqrt {\frac {L _ {\max} \omega (\omega + 1) ^ {2} p}{n \mu}}, \frac {1}{\alpha}, (\omega + 1) \right\} \bigg).
$$
Since all terms are non-increasing functions of  $p$ , the minimum is attained at a point  $p = \left(\frac{Ln}{L_{\max}}\right)^{1/3} \frac{1}{\omega + 1}$  for all  $p \geq \left(\frac{Ln}{L_{\max}}\right)^{1/3} \frac{1}{\omega + 1}$ . Let us define
$$
Q _ {1} := \left\{p \middle | p \leq \left(\frac {L n}{L _ {\operatorname* {m a x}}}\right) ^ {1 / 3} \frac {1}{\omega + 1}, p \in Q _ {0} \right\}.
$$
The last observation means that the argmin (61) is equivalent to
arg min  $p\in Q_1$
$$
\begin{array}{l} \widetilde {\Theta} \bigg (\max \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}}, \sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1) ^ {2 / 3}}{\alpha n ^ {1 / 3} p ^ {1 / 3} \mu}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\mathrm {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\} \bigg) \\ = \underset {p \in Q _ {1}} {\arg \min } \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\}\right), \tag {62} \\ \end{array}
$$
where we eliminate the redundant terms using the additional information  $p \leq \left( \frac{Ln}{L_{\max}} \right)^{1/3} \frac{1}{\omega + 1}$ . In particular,
$$
\sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1)}{\alpha n ^ {1 / 3} \mu}} \leq \sqrt {\frac {L}{\alpha p \mu}} \mathrm {a n d} \sqrt {\frac {L ^ {2 / 3} L _ {\mathrm {m a x}} ^ {1 / 3} (\omega + 1) ^ {2 / 3}}{\alpha n ^ {1 / 3} p ^ {1 / 3} \mu}} \leq \sqrt {\frac {L}{\alpha p \mu}}.
$$
for all  $p \leq \left( \frac{Ln}{L_{\max}} \right)^{1/3} \frac{1}{\omega + 1}$ . Without the condition  $p \in Q_1$ , the argmin (62) attains the minimum at a point
$$
p = \max \left\{\frac {1}{\omega + 1}, \left(\frac {L n}{L _ {\mathrm {m a x}}}\right) ^ {1 / 2} \frac {1}{\sqrt {\alpha} (\omega + 1) ^ {3 / 2}} \right\}.
$$
Considering the condition  $p \in Q_1$  and  $\mu_{\omega, \alpha}^r \coloneqq B / A$ , we have
$$
p = \min  \left\{1, \frac {1}{\mu_ {\omega , \alpha} ^ {r}}, \left(\frac {L n}{L _ {\operatorname* {m a x}}}\right) ^ {1 / 3} \frac {1}{\omega + 1}, \max  \left\{\frac {1}{\omega + 1}, \left(\frac {L n}{L _ {\operatorname* {m a x}}}\right) ^ {1 / 2} \frac {1}{\sqrt {\alpha} (\omega + 1) ^ {3 / 2}} \right\} \right\}. \tag {63}
$$
It is left carefully to substitute (63) to
$$
\widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega}{n p \mu}}, \frac {1}{\alpha}, (\omega + 1), \frac {1}{p} \right\}\right)
$$
and obtain (14). The proof of  $\mathfrak{m}_{\mathrm{realistic}}^r = \widetilde{\Theta}\left((1 - r)K_\omega +rK_\alpha\right)T_{\mathrm{realistic}})$  is the same as in Theorem 5.2.
E.8 Comparison with EF21 + DIANA
Proof. Using the inequality of arithmetic and geometric means, i.e.,  $\sqrt{xy} \leq \frac{x + y}{2}$  for all  $x, y \geq 0$ , and  $L \geq \mu$ , we have
$$
\begin{array}{l} \mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = \widetilde {\Theta} \left(K _ {\omega , \alpha} ^ {r} \left(\sqrt {\frac {L (\omega + 1)}{\alpha \mu}} + \sqrt {\frac {L _ {\operatorname* {m a x}} \omega (\omega + 1)}{n \mu}} + \sqrt {\frac {L \mu_ {\omega , \alpha} ^ {r}}{\alpha \mu}} + \sqrt {\frac {L _ {\operatorname* {m a x}} \omega \mu_ {\omega , \alpha} ^ {r}}{n \mu}} + \frac {1}{\alpha} + \omega + \mu_ {\omega , \alpha} ^ {r}\right) + d\right) \\ = \widetilde {\mathcal {O}} \left(K _ {\omega , \alpha} ^ {r} \left(\frac {L}{\alpha \mu} + \frac {L _ {\mathrm {m a x}} \omega}{n \mu} + \omega + \sqrt {\frac {L \mu_ {\omega , \alpha} ^ {r}}{\alpha \mu}} + \sqrt {\frac {L _ {\mathrm {m a x}} \omega \mu_ {\omega , \alpha} ^ {r}}{n \mu}} + \mu_ {\omega , \alpha} ^ {r}\right) + d\right). \\ \end{array}
$$
From the definition of  $\mu_{\omega ,\alpha}^{r}\coloneqq {}^{rd} / K_{\omega ,\alpha}^{r}$  , we get
$$
\mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = \widetilde {\mathcal {O}} \left(K _ {\omega , \alpha} ^ {r} \left(\frac {L}{\alpha \mu} + \frac {L _ {\operatorname* {m a x}} \omega}{n \mu} + \omega\right) + \left(\sqrt {\frac {L K _ {\omega , \alpha} ^ {r} \times r d}{\alpha \mu}} + \sqrt {\frac {L _ {\operatorname* {m a x}} \omega K _ {\omega , \alpha} ^ {r} \times r d}{n \mu}} + r d\right) + d\right).
$$
Using the inequality of arithmetic and geometric means again and  $r \leq 1$ , we obtain
$$
\mathfrak {m} _ {\text {r e a l i s t i c}} ^ {r} = \widetilde {\mathcal {O}} \left(K _ {\omega , \alpha} ^ {r} \left(\frac {L}{\alpha \mu} + \frac {L _ {\max} \omega}{n \mu} + \omega\right) + K _ {\omega , \alpha} ^ {r} \left(\frac {L}{\alpha \mu} + \frac {L _ {\max} \omega}{n \mu}\right) + d\right).
$$
The last equality means that  $\mathfrak{m}_{\mathrm{realistic}}^r = \tilde{\mathcal{O}}\left(\mathfrak{m}_{\mathrm{EF21 - P + DIANA}}^r\right)$  for all  $r\in [0,1]$ .
E.9 Comparison with AGD
Theorem 6.2. For all  $r \in [0,1]$  and for all  $K \in [d]$ , let us take the RandK and TopK compressors with the parameters (expected densities) i)  $K_{\omega} = K$  and  $K_{\alpha} = \min \{\lceil 1 - r / rK\rceil ,d\}$  for  $r \in [0,1 / 2]$ , ii)  $K_{\omega} = \min \{\lceil r / 1 - rK\rceil ,d\}$  and  $K_{\alpha} = K$  for  $r \in (1 / 2,1]$ . Then we have  $\mathfrak{m}_{\mathrm{realistic}}^r = \tilde{\mathcal{O}} (\mathfrak{m}_{\mathrm{AGD}})$ .
Proof. Consider that  $r \in [0, 1/2]$ , then  $K_{\omega} = K$ ,  $K_{\alpha} = \min \{ \lceil 1^{-r} / rK \rceil, d \}$ . Therefore, we have
$$
K _ {\omega , \alpha} ^ {r} := (1 - r) K _ {\omega} + r K _ {\alpha} \leq (1 - r) K + r \lceil^ {1 - r} / r K \rceil \leq 3 (1 - r) K
$$
$$
K _ {\omega , \alpha} ^ {r} := (1 - r) K _ {\omega} + r K _ {\alpha} \geq (1 - r) K.
$$