SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Li, Z. and Richtárik, P. (2021). CANITA: Faster rates for distributed convex optimization with communication compression. Advances in Neural Information Processing Systems, 34:13770-13781.
Liu, X., Li, Y., Tang, J., and Yan, M. (2020). A double residual compression algorithm for efficient distributed learning. In International Conference on Artificial Intelligence and Statistics, pages 133-143. PMLR.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR.
Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel, A. R., Roučka, v., Saboo, A., Fernando, I., Kulal, S., Cirmrman, R., and Scopatz, A. (2017). Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103.
Mishchenko, K., Gorbunov, E., Takáč, M., and Richtárik, P. (2019). Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269.
Nesterov, Y. (2018). Lectures on convex optimization, volume 137. Springer.
Philippenko, C. and Dieuleveut, A. (2020). Artemis: tight convergence guarantees for bidirectional compression in federated learning. arXiv preprint arXiv:2006.14591.
Philippenko, C. and Dieuleveut, A. (2021). Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems, 34:2387-2399.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092.
Richtárik, P., Sokolov, I., and Fatkhullin, I. (2021). EF21: A new, simpler, theoretically better, and practically faster error feedback. In Neural Information Processing Systems, 2021.
Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. (2014). 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Fifteenth Annual Conference of the International Speech Communication Association.
Stonyakin, F., Tyurin, A., Gasnikov, A., Dvurechensky, P., Agafonov, A., Dvinskikh, D., Alkousa, M., Pasechnyuk, D., Artamonov, S., and Piskunova, V. (2021). Inexact model: a framework for optimization and variational inequalities. Optimization Methods and Software, 36(6):1155-1201.
Tang, H., Lian, X., Yu, C., Zhang, T., and Liu, J. (2020). DoubleSqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In Proceedings of the 36th International Conference on Machine Learning (ICML).
Tyurin, A. and Richtárik, P. (2023). DASHA: Distributed nonconvex optimization with communication compression and optimal oracle complexity. International Conference on Learning Representations (ICLR).
1 Introduction 1
1.1 The problem 1
1.2Assumptions 2
2 Motivation: From Unidirectional to Bidirectional Compression 2
2.1 Compression mappings 2
2.2 Unidirectional (i.e., w2s) compression 3
2.3 Bidirectional (i.e., w2s and s2w) compression 4
3 Contributions 4
4 New Method: 2Direction 5
5 Theoretical Communication Complexity of 2Direction 7
5.1 The ratio  $L_{\mathrm{max}} / L$  is not known 8
5.2 The ratio  $L_{\mathrm{max}} / L$  is known 8
6 Theoretical Comparison with Previous State of the Art 8
7 Proof Sketch 9
8 Limitations and Future Work 9
A Table of Notations 14
B The Original ADIANA Algorithm 14
C The Original EF21-P + DIANA Algorithm 15
D Useful Identities and Inequalities 15
E Proofs of Theorems 15
E.1 Analysis of learning rates 15
E.2 Generic lemmas 19
E.3 Construction of the Lyapunov function 24
E.4 Main theorem 29
E.5 Strongly-convex case 34
E.6 General convex case 35
E.7 Choosing optimal parameters 36
E.8 Comparison with EF21 + DIANA 40
E.9 Comparison with AGD 40
F Auxiliary Inequalities For  $\bar{L}$  41
G Proof of Lemma E.10 (First Symbolically Computed) 43
H Proof of Lemma E.11 (Second Symbolically Computed) 44
I Symbolically Computed Constraints for  $\bar{L}$  Such That The Term w.r.t.  $\kappa$  is less or equal  $1 / 2$  in (87) 45
J Symbolically Computed Constraints for  $\bar{L}$  Such That The Term w.r.t.  $\rho$  is less or equal  $1 / 2$  in (89) 46
K Symbolically Computed Expression (91) 46
L Symbolically Computed Constraints for  $\bar{L}$  Such That The Inequality from Section K Holds 49
M Symbolically Computed Expression (92) 54
N Symbolically Computed Constraints for  $\bar{L}$  Such That The Inequality from Section M Holds 55
O Symbolical Check That The Constraints from Sections I, J, L and N Follow From The Constraint (44) 57
P Jupyter Notebook for Symbolic Computations 63
P.1 Fileutils.py 68
Q Experiments 71
Q.1 Setup 71
Q.2 Results 71
R Convergence Rate of CANITA obtained by Li and Richtárik (2021) 72
S Comparison with ADIANA 72
A Table of Notations
Notation	Meaning
g = O(f)	Exist C > 0 such that g(z) ≤ C × f(z) for all z ∈ Z
g = Ω(f)	Exist C > 0 such that g(z) ≥ C × f(z) for all z ∈ Z
g = Θ(f)	g = O(f) and g = Ω(f)
g = O(f)	Exist C > 0 such that g(z) ≤ C × f(z) × log(poly(z)) for all z ∈ Z
g = Ω(f)	Exist C > 0 such that g(z) ≥ C × f(z) × log(poly(z)) for all z ∈ Z
g = Θ(f)	g = O(f) and g = Ω(f)
{a, ..., b}	Set {i ∈ Z | a ≤ i ≤ b}
[n]	{1, ..., n}
B The Original ADIANA Algorithm
In this section, we present the ADIANA algorithm from (Li et al., 2020). In the following method, the notations, parameterization, and order of steps can be slightly different, but the general idea is the same.
Algorithm 3 Accelerated DIANA (ADIANA) by Li et al. (2020)
1: Parameters: Lipschitz-like parameter  $\bar{L} >0$  strong-convexity parameter  $\mu \geq 0$  probability  $p$  ,parameter  $\Gamma_0$  initial point  $x^0\in \mathbb{R}^d$  , initial gradient shifts  $h_1^0,\dots,h_n^0\in \mathbb{R}^d$    
2: Initialize  $\beta = 1 / \omega +1$  and  $z^0 = u^0 = x^0$    
3: Initialize  $h^0 = \frac{1}{n}\sum_{i = 1}^{n}h_i^0$    
4: for  $t = 0,1,\ldots ,T - 1$  do   
5:  $\Gamma_{t + 1},\gamma_{t + 1},\theta_{t + 1} =$  CalculateOriginalLearningRates(..)   
6: for  $i = 1,\dots ,n$  in parallel do   
7:  $y^{t + 1} = \theta_{t + 1}u^{t} + (1 - \theta_{t + 1})z^{t}$    
8:  $m_i^{t,y} = \mathcal{C}_i^{D,y}(\nabla f_i(y^{t + 1}) - h_i^t)$  Worker i compresses the shifted gradient via the compressor  $\mathcal{C}_i^{D,y}\in \mathbb{U}(\omega)$    
9: Send compressed message  $m_i^{t,y}$  to the server   
10: end for   
11:  $m^{t,y} = \frac{1}{n}\sum_{i = 1}^{n}m_i^{t,y}$  Server averages the messages   
12:  $g^{t + 1} = h^t +m^{t,y}\equiv \frac{1}{n}\sum_{i = 1}^{n}(h_i^t +m_i^{t,y})$    
13:  $u^{t + 1} = \arg \min_{x\in \mathbb{R}^d}\langle g^{t + 1},x\rangle +\frac{\bar{L} + \Gamma_t\mu}{2\gamma_{t + 1}}\left\| x - u^t\right\|^2 +\frac{\mu}{2}\left\| x - y^{t + 1}\right\|^2$  Server does a gradient-like descent step   
14: Flip a coin  $c^t\sim$  Bernoulli(p)   
15: Broadcast non-compressed messages  $u^{t + 1}$  to all n workers   
16: for  $i = 1,\dots ,n$  in parallel do   
17:  $x^{t + 1} = \theta_{t + 1}u^{t + 1} + (1 - \theta_{t + 1})z^t$    
18:  $z^{t + 1} = \left\{ \begin{array}{ll}x^{t + 1}, & c^t = 1\\ z^t, & c^t = 0 \end{array} \right.$    
19:  $m_i^{t,z} = \mathcal{C}_i^{D,z}(\nabla f_i(z^{t + 1}) - h_i^t)$  Worker i compresses the shifted gradient via the compressor  $\mathcal{C}_i^{D,z}\in \mathbb{U}(\omega)$    
20:  $h_i^{t + 1} = h_i^t +\beta m_i^{t,z}$  Update the control variables   
21: Send compressed message  $m_i^{t,z}$  to the server   
22: end for   
23:  $h^{t + 1} = h^t +\beta \frac{1}{n}\sum_{i = 1}^{n}m_i^{t,z}$  Server averages the messages
C The Original EF21-P + DIANA Algorithm
In this section, we present the EF21-P + DIANA algorithm from (Gruntkowska et al., 2023). In the following method, the notations, parameterization, and order of steps can be slightly different, but the general idea is the same.
Algorithm 4 EF21-P + DIANA by Gruntkowska et al. (2023)
1: Parameters: learning rates  $\gamma >0$  and  $\beta >0$  , initial model  $u^0\in \mathbb{R}^d$  , initial gradient shifts  $h_1^0,\dots,h_n^0\in$ $\mathbb{R}^d$  , average of the initial gradient shifts  $h^0 = \frac{1}{n}\sum_{i = 1}^{n}h_i^0$  , initial model shift  $w^{0} = u^{0}\in \mathbb{R}^{d}$    
2: for  $t = 0,1,\ldots ,T - 1$  do   
3: for  $i = 1,\dots ,n$  in parallel do   
4:  $m_i^t = \mathcal{C}_i^D (\nabla f_i(w^t) - h_i^t)$  Worker i compresses the shifted gradient via the dual compressor  $\mathcal{C}_i^D\in \mathbb{U}(\omega)$    
5: Send compressed message  $m_i^t$  to the server   
6:  $h_i^{t + 1} = h_i^t +\beta m_i^t$  Worker i updates its local gradient shift with stepsize  $\beta$    
7: end for   
8:  $m^t = \frac{1}{n}\sum_{i = 1}^n m_i^t$  Server averages the n messages received from the workers   
9:  $h^{t + 1} = h^t +\beta m^t$  Server updates the average gradient shift so that  $h^t = \frac{1}{n}\sum_{i = 1}^n h_i^t$    
10:  $g^{t} = h^{t} + m^{t}$  Server computes the gradient estimator   
11:  $u^{t + 1} = u^{t} - \gamma g^{t}$  Server takes a gradient-type step with stepsize  $\gamma$    
12:  $p^{t + 1} = \mathcal{C}^P (u^{t + 1} - w^t)$  Server compresses the shifted model via the primal compressor  $\mathcal{C}^P\in \mathbb{B}(\alpha)$    
13:  $w^{t + 1} = w^{t} + p^{t + 1}$  Server updates the model shift   
14: Broadcast compressed message  $p^{t + 1}$  to all n workers   
15: for  $i = 1,\dots ,n$  in parallel do   
16:  $w^{t + 1} = w^{t} + p^{t + 1}$  Worker i updates its local copy of the model shift   
17: end for   
18: end for
D Useful Identities and Inequalities
For all  $x,y,x_1,\ldots ,x_n\in \mathbb{R}^d$ $s > 0$  and  $\alpha \in (0,1]$  we have:
$$
\left\| x + y \right\| ^ {2} \leq (1 + s) \left\| x \right\| ^ {2} + \left(1 + s ^ {- 1}\right) \| y \| ^ {2}, \tag {17}
$$
$$
\left\| x + y \right\| ^ {2} \leq 2 \left\| x \right\| ^ {2} + 2 \left\| y \right\| ^ {2}, \tag {18}
$$
$$
\langle x, y \rangle \leq \frac {\| x \| ^ {2}}{2 s} + \frac {s \| y \| ^ {2}}{2}, \tag {19}
$$
$$
(1 - \alpha) \left(1 + \frac {\alpha}{2}\right) \leq 1 - \frac {\alpha}{2}, \tag {20}
$$
$$
(1 - \alpha) \left(1 + \frac {2}{\alpha}\right) \leq \frac {2}{\alpha}, \tag {21}
$$
$$
\langle a, b \rangle = \frac {1}{2} \left(\| a \| ^ {2} + \| b \| ^ {2} - \| a - b \| ^ {2}\right). \tag {22}
$$
Variance decomposition: For any random vector  $X \in \mathbb{R}^d$  and any non-random  $c \in \mathbb{R}^d$ , we have
$$
\mathbb {E} \left[ \| X - c \| ^ {2} \right] = \mathbb {E} \left[ \| X - \mathbb {E} [ X ] \| ^ {2} \right] + \| \mathbb {E} [ X ] - c \| ^ {2}. \tag {23}
$$
Lemma D.1 (Nesterov (2018)). Let  $f: \mathbb{R}^d \to \mathbb{R}$  be a function for which Assumptions 1.2 and 1.3 are satisfied. Then for all  $x, y \in \mathbb{R}^d$  we have:
$$
\left\| \nabla f (x) - \nabla f (y) \right\| ^ {2} \leq 2 L (f (x) - f (y) - \langle \nabla f (y), x - y \rangle). \tag {24}
$$
E Proofs of Theorems
E.1 Analysis of learning rates
In this section, we establish inequalities for the sequences from Algorithm 2.
Lemma E.1. Suppose that parameter  $\bar{L} > 0$ , strong-convexity parameter  $\mu \geq 0$ , probability  $p \in (0,1]$ ,  $\bar{L} \geq \mu$ , and  $\Gamma_0 \geq 1$ . Then the sequences generated by Algorithm 2 have the following properties:
1. The quantities  $\theta_{t + 1},\gamma_{t + 1}$  , and  $\Gamma_{t + 1}$  are well-defined and  $\theta_{t + 1},\gamma_{t + 1}\geq 0$  for all  $t\geq 0$
2.  $\gamma_{t + 1} = p\theta_{t + 1}\Gamma_{t + 1}$  for all  $t\geq 0$
3.  $\bar{L}\theta_{t + 1}\gamma_{t + 1}\leq (\bar{L} +\Gamma_t\mu)$  for all  $t\geq 0$
$$
\Gamma_ {t} \geq \frac {\Gamma_ {0}}{2} \exp \left(t \min  \left\{\sqrt {\frac {p \mu}{4 \bar {L}}}, p \theta_ {\min } \right\}\right)
$$
for all  $t\geq 0$
$$
\Gamma_ {t} \geq \left\{ \begin{array}{l l} \frac {\Gamma_ {0}}{2} \exp \left(t p \theta_ {\min}\right), & t <   \bar {t} \\ \frac {1}{4 p \theta_ {\min} ^ {2}} + \frac {p (t - \bar {t}) ^ {2}}{1 6}, & t \geq \bar {t}. \end{array} \right.
$$
where  $\bar{t} := \max \left\{\left\lceil \frac{1}{p\theta_{\min}}\log \frac{1}{2\Gamma_0p\theta_{\min}^2}\right\rceil ,0\right\}$ .
6.  $\{\theta_{t + 1}\}_{t = 0}^{\infty}$  is a non-increasing sequence.
1. Note that  $\bar{\theta}_{t + 1}$  is the largest root of
$$
p \bar {L} \Gamma_ {t} \bar {\theta} _ {t + 1} ^ {2} + p (\bar {L} + \Gamma_ {t} \mu) \bar {\theta} _ {t + 1} - (\bar {L} + \Gamma_ {t} \mu) = 0. \tag {25}
$$
We fix  $t \geq 0$ . Assume that  $\Gamma_t > 0$ .
$$
p \bar {L} \Gamma_ {t} \times 0 + p (\bar {L} + \Gamma_ {t} \mu) \times 0 - (\bar {L} + \Gamma_ {t} \mu) <   0.
$$
Therefore, the largest root  $\bar{\theta}_{t + 1}$  is well-defined and  $\bar{\theta}_{t + 1} \geq 0$ , and  $\theta_{t + 1} = \min \{\bar{\theta}_{t + 1}, \theta_{\min}\} \geq 0$ . Next,  $\gamma_{t + 1}$  is well-defined and
$$
\gamma_ {t + 1} = p \theta_ {t + 1} \Gamma_ {t} / (1 - p \theta_ {t + 1}) \geq 0
$$
since  $p\theta_{t + 1} \in [0,1 / 4]$ . Finally,  $\Gamma_{t + 1} = \Gamma_t + \gamma_{t + 1} > 0$ . We showed that, for all  $t \geq 0$ , if  $\Gamma_t > 0$ , then  $\theta_{t + 1}, \gamma_{t + 1} \geq 0$  and  $\Gamma_{t + 1} > 0$ . Note that  $\Gamma_0 > 0$ , thus  $\theta_{t + 1}, \gamma_{t + 1} \geq 0$  and  $\Gamma_{t + 1} > 0$  for all  $t \geq 0$ .
2. From the definition of  $\gamma_{t + 1}$  and  $\Gamma_{t + 1}$ , we have
$$
\left(1 - p \theta_ {t + 1}\right) \gamma_ {t + 1} = p \theta_ {t + 1} \Gamma_ {t},
$$
which is equivalent to
$$
\gamma_ {t + 1} = p \theta_ {t + 1} (\Gamma_ {t} + \gamma_ {t + 1}) = p \theta_ {t + 1} \Gamma_ {t + 1}.
$$
3. Recall again that  $\bar{\theta}_{t + 1}\geq 0$  is the largest root of
$$
p \bar {L} \Gamma_ {t} \bar {\theta} _ {t + 1} ^ {2} + p (\bar {L} + \Gamma_ {t} \mu) \bar {\theta} _ {t + 1} - (\bar {L} + \Gamma_ {t} \mu) = 0.
$$
If  $\bar{\theta}_{t + 1}\leq \theta_{\mathrm{min}}$  , then
$$
p \bar {L} \Gamma_ {t} \theta_ {t + 1} ^ {2} + p (\bar {L} + \Gamma_ {t} \mu) \theta_ {t + 1} - (\bar {L} + \Gamma_ {t} \mu) = 0.
$$
Otherwise, if  $\bar{\theta}_{t + 1} > \theta_{\mathrm{min}}$  , since
$$
p \bar {L} \Gamma_ {t} \times 0 + p (\bar {L} + \Gamma_ {t} \mu) \times 0 - (\bar {L} + \Gamma_ {t} \mu) <   0,
$$
and  $\theta_{t + 1} = \theta_{\min} <   \bar{\theta}_{t + 1}$  , then
$$
p \bar {L} \Gamma_ {t} \theta_ {t + 1} ^ {2} + p (\bar {L} + \Gamma_ {t} \mu) \theta_ {t + 1} - (\bar {L} + \Gamma_ {t} \mu) \leq 0. \tag {26}
$$
In all cases, the inequality (26) holds. From this inequality, we can get
$$
p \bar {L} \Gamma_ {t} \theta_ {t + 1} ^ {2} \leq (\bar {L} + \Gamma_ {t} \mu) (1 - p \theta_ {t + 1})
$$
$$
\bar {L} \theta_ {t + 1} \frac {p \theta_ {t + 1} \Gamma_ {t}}{(1 - p \theta_ {t + 1})} \leq (\bar {L} + \Gamma_ {t} \mu).
$$
Using the definition of  $\gamma_{t + 1}$ , we obtain
$$
\bar {L} \theta_ {t + 1} \gamma_ {t + 1} \leq (\bar {L} + \Gamma_ {t} \mu)
$$
4. Let us find the largest root of the quadratic equation (25):
$$
\bar {\theta} _ {t + 1} := \frac {- p (\bar {L} + \Gamma_ {t} \mu) + \sqrt {p ^ {2} (\bar {L} + \Gamma_ {t} \mu) ^ {2} + 4 p \bar {L} \Gamma_ {t} (\bar {L} + \Gamma_ {t} \mu)}}{2 p \bar {L} \Gamma_ {t}}. \tag {27}
$$
Let us define  $a \coloneqq p^2 (\bar{L} + \Gamma_t \mu)^2$  and  $b \coloneqq 4\bar{L} p(\bar{L} + \Gamma_t \mu) \Gamma_t$ , then
$$
\bar {\theta} _ {t + 1} = \frac {- \sqrt {a} + \sqrt {a + b}}{2 \bar {L} p \Gamma_ {t}}
$$
Since  $\Gamma_t \geq 1$  for all  $t \geq 0$ , and  $\bar{L} \geq \mu$ , we have
$$
a = p ^ {2} \left(\bar {L} + \Gamma_ {t} \mu\right) ^ {2} \leq p \left(\bar {L} + \Gamma_ {t} \mu\right) ^ {2} \leq p \left(\bar {L} + \Gamma_ {t} \mu\right) \left(\Gamma_ {t} \bar {L} + \Gamma_ {t} \mu\right) \leq 2 \bar {L} p \left(\bar {L} + \Gamma_ {t} \mu\right) \Gamma_ {t} = \frac {b}{2}.
$$
Using  $\sqrt{x + y} \geq \left(\sqrt{x} + \sqrt{y}\right) / \sqrt{2}$  for all  $x, y \geq 0$ , and  $\sqrt{b} \geq \sqrt{2}\sqrt{a}$  we have
$$
\begin{array}{l} \bar {\theta} _ {t + 1} = \frac {- \sqrt {a} + \sqrt {a + b}}{2 \bar {L} p \Gamma_ {t}} \geq \frac {- \sqrt {a} + \frac {1}{\sqrt {2}} \sqrt {a} + \frac {1}{\sqrt {2}} \sqrt {b}}{2 \bar {L} p \Gamma_ {t}} \\ = \frac {\left(\frac {1}{\sqrt {2}} - 1\right) \sqrt {a} + \frac {1}{\sqrt {2}} \left(\frac {1}{\sqrt {2}} + 1 - \frac {1}{\sqrt {2}}\right) \sqrt {b}}{2 \bar {L} p \Gamma_ {t}} \geq \frac {\frac {1}{\sqrt {2}} \left(\frac {1}{\sqrt {2}}\right) \sqrt {b}}{2 \bar {L} p \Gamma_ {t}} = \frac {\sqrt {b}}{4 \bar {L} p \Gamma_ {t}}. \\ \end{array}
$$
Therefore
$$
\bar {\theta} _ {t + 1} \geq \frac {\sqrt {4 \bar {L} p (\bar {L} + \Gamma_ {t} \mu) \Gamma_ {t}}}{4 \bar {L} p \Gamma_ {t}} = \sqrt {\frac {(\bar {L} + \Gamma_ {t} \mu)}{4 \bar {L} p \Gamma_ {t}}} \geq \max \left\{\sqrt {\frac {1}{4 p \Gamma_ {t}}}, \sqrt {\frac {\mu}{4 p \bar {L}}} \right\}
$$
$$
\theta_ {t + 1} \geq \min  \left\{\max  \left\{\sqrt {\frac {1}{4 p \Gamma_ {t}}}, \sqrt {\frac {\mu}{4 p \bar {L}}} \right\}, \theta_ {\min } \right\}. \tag {28}
$$
Next, since  $p\theta_{t + 1}\in [0,1 / 4]$ , we have
$$
\gamma_ {t + 1} := p \theta_ {t + 1} \Gamma_ {t} / (1 - p \theta_ {t + 1}) \geq p \theta_ {t + 1} \Gamma_ {t} (1 + p \theta_ {t + 1}) \tag {29}
$$
$$
\Gamma_ {t + 1} := \Gamma_ {t} + \gamma_ {t + 1} \geq \left(1 + p \theta_ {t + 1} + p ^ {2} \theta_ {t + 1} ^ {2}\right) \Gamma_ {t}.
$$
Using (28) and (29), we obtain
$$
\begin{array}{l} \Gamma_ {t + 1} \geq \left(1 + p \min  \left\{\sqrt {\frac {\mu}{4 p \bar {L}}}, \theta_ {\min } \right\}\right) \Gamma_ {t} = \left(1 + \min  \left\{\sqrt {\frac {p \mu}{4 \bar {L}}}, p \theta_ {\min } \right\}\right) \Gamma_ {t} \\ \geq \Gamma_ {0} \left(1 + \min  \left\{\sqrt {\frac {p \mu}{4 \bar {L}}}, p \theta_ {\min } \right\}\right) ^ {t + 1} \geq \frac {\Gamma_ {0}}{2} \exp \left((t + 1) \min  \left\{\sqrt {\frac {p \mu}{4 \bar {L}}}, p \theta_ {\min } \right\}\right), \tag {30} \\ \end{array}
$$
where we use that  $1 + x \geq e^{x} / 2$  for all  $x \in [0,1]$ .
5. Using (28) and (29), we have
$$
\begin{array}{l} \Gamma_ {t + 1} \geq \left(1 + p \theta_ {t + 1} + p ^ {2} \theta_ {t + 1} ^ {2}\right) \Gamma_ {t} \\ \geq \Gamma_ {t} + p \min  \left\{\sqrt {\frac {1}{4 p \Gamma_ {t}}}, \theta_ {\min } \right\} \Gamma_ {t} + p ^ {2} \min  \left\{\sqrt {\frac {1}{4 p \Gamma_ {t}}}, \theta_ {\min } \right\} ^ {2} \Gamma_ {t}. \\ \end{array}
$$
The sequence  $\Gamma_{t + 1}$  is strongly increasing. Thus, exists the minimal  $\widehat{t} \geq 0$  such that  $\Gamma_{\widehat{t}} \geq (4p\theta_{\min}^2)^{-1}$ . For all  $0 \leq t < \widehat{t}$ , it holds that  $\Gamma_t < (4p\theta_{\min}^2)^{-1}$ ,  $\sqrt{\frac{1}{4p\Gamma_t}} > \theta_{\min}$ , and
$$
\Gamma_ {t + 1} \geq \Gamma_ {t} + p \theta_ {\min } \Gamma_ {t} + p ^ {2} \theta_ {\min } ^ {2} \Gamma_ {t} \geq \Gamma_ {t} + p \theta_ {\min } \Gamma_ {t} \geq \Gamma_ {0} (1 + p \theta_ {\min }) ^ {t + 1} \geq \frac {\Gamma_ {0}}{2} \exp ((t + 1) p \theta_ {\min }) \tag {31}
$$
Therefore, if  $\widehat{t} > 0$ , then
$$
\frac {1}{4 p \theta_ {\operatorname* {m i n}} ^ {2}} > \Gamma_ {\widehat {t} - 1} \geq \frac {\Gamma_ {0}}{2} \exp \left((\widehat {t} - 1) p \theta_ {\min }\right).
$$
Thus, we have the following bound for  $\widehat{t}$ :
$$
\widehat {t} \leq \bar {t} := \max  \left\{\left\lceil \frac {1}{p \theta_ {\operatorname* {m i n}}} \log \frac {1}{2 \Gamma_ {0} p \theta_ {\operatorname* {m i n}} ^ {2}} \right\rceil , 0 \right\}.
$$
For all  $t \geq \widehat{t}$ , we have  $\sqrt{\frac{1}{4p\Gamma_t}} \leq \theta_{\min}$  and
$$
\Gamma_ {t + 1} \geq \Gamma_ {t} + p \sqrt {\frac {1}{4 p \Gamma_ {t}}} \Gamma_ {t} + p ^ {2} \frac {1}{4 p \Gamma_ {t}} \Gamma_ {t} = \Gamma_ {t} + \sqrt {\frac {p}{4}} \sqrt {\Gamma_ {t}} + \frac {p}{4}.
$$
Using mathematical induction, let us show that
$$
\Gamma_ {t} \geq \frac {1}{4 p \theta_ {\min } ^ {2}} + \frac {p (t - \widehat {t}) ^ {2}}{1 6} \tag {32}
$$
for all  $t \geq \widehat{t}$ . For  $t = \widehat{t}$  it is true since  $\Gamma_{\widehat{t}} \geq \left(4p\theta_{\min}^2\right)^{-1}$  by the definition of  $\widehat{t}$ . Next, for some  $t \geq \widehat{t}$ , assume that (32) holds, then
$$
\begin{array}{l} \Gamma_ {t + 1} \geq \Gamma_ {t} + \frac {\sqrt {p}}{2} \sqrt {\Gamma_ {t}} + \frac {p}{4} \\ \geq \frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p (t - \widehat {t}) ^ {2}}{1 6} + \frac {\sqrt {p}}{2} \sqrt {\frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p (t - \widehat {t}) ^ {2}}{1 6}} + \frac {p}{4} \\ \geq \frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p (t - \widehat {t}) ^ {2}}{1 6} + \frac {p (t - \widehat {t})}{8} + \frac {p}{4} \\ \geq \frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p (t - \widehat {t} + 1) ^ {2}}{1 6}. \\ \end{array}
$$
We proved the inequality using mathematical induction. Combining (31) and (32), we obtain the unified inequality for  $\Gamma_t$ :
$$
\Gamma_ {t} \geq \min  \left\{\frac {\Gamma_ {0}}{2} \exp (t p \theta_ {\min }) , \frac {1}{4 p \theta_ {\min } ^ {2}} + \frac {p (t - \widehat {t}) ^ {2}}{1 6} \right\}
$$
for all  $t \geq 0$ . Also, if  $t < \bar{t}$ , then the first term in the minimum is less or equal than the second one. Therefore,
$$
\Gamma_ {t} \geq \left\{ \begin{array}{l l} \frac {\Gamma_ {0}}{2} \exp (t p \theta_ {\min }) , & t <   \bar {t} \\ \min  \left\{\frac {\Gamma_ {0}}{2} \exp (t p \theta_ {\min }) , \frac {1}{4 p \theta_ {\min } ^ {2}} + \frac {p (t - \hat {t}) ^ {2}}{1 6} \right\}, & t \geq \bar {t}. \end{array} \right. \tag {33}
$$
Let us bound the second term. Recall that  $\bar{t} \geq \widehat{t}$ , thus, if  $t \geq \bar{t}$ , then  $(t - \widehat{t})^2 \geq (t - \bar{t})^2$ . In (33), we can change  $\widehat{t}$  to  $\bar{t}$  and get
$$
\Gamma_ {t} \geq \left\{ \begin{array}{l l} \frac {\Gamma_ {0}}{2} \exp \left(t p \theta_ {\min }\right), & t <   \bar {t} \\ \min  \left\{\frac {\Gamma_ {0}}{2} \exp \left(t p \theta_ {\min }\right), \frac {1}{4 p \theta_ {\min } ^ {2}} + \frac {p (t - \bar {t}) ^ {2}}{1 6} \right\}, & t \geq \bar {t}. \end{array} \right.
$$
Using the Taylor expansion of the exponent at the point  $\bar{t}$ , we get
$$
\begin{array}{l} \frac {\Gamma_ {0}}{2} \exp (t p \theta_ {\min }) \geq \frac {\Gamma_ {0}}{2} \exp (\bar {t} p \theta_ {\min }) + \frac {\Gamma_ {0}}{2} p ^ {2} \theta_ {\min } ^ {2} \exp (\bar {t} p \theta_ {\min }) \frac {(t - \bar {t}) ^ {2}}{2} \\ \geq \frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p}{8} (t - \bar {t}) ^ {2} \geq \frac {1}{4 p \theta_ {\mathrm {m i n}} ^ {2}} + \frac {p}{1 6} (t - \bar {t}) ^ {2}. \\ \end{array}
$$
for all  $t\geq \bar{t}$  . Finally, we can conclude that
$$
\Gamma_ {t} \geq \left\{ \begin{array}{l l} \frac {\Gamma_ {0}}{2} \exp \left(t p \theta_ {\min }\right), & t <   \bar {t} \\ \frac {1}{4 p \theta_ {\min } ^ {2}} + \frac {p (t - \bar {t}) ^ {2}}{1 6}, & t \geq \bar {t}. \end{array} \right.
$$
6. Let us rewrite (27):
$$
\begin{array}{l} \bar {\theta} _ {t + 1} = \frac {- p (\bar {L} + \Gamma_ {t} \mu) + \sqrt {p ^ {2} (\bar {L} + \Gamma_ {t} \mu) ^ {2} + 4 p \bar {L} \Gamma_ {t} (\bar {L} + \Gamma_ {t} \mu)}}{2 p \bar {L} \Gamma_ {t}} \\ = - \frac {1}{2} \left(\frac {1}{\Gamma_ {t}} + \frac {\mu}{\bar {L}}\right) + \sqrt {\frac {1}{4} \left(\frac {1}{\Gamma_ {t}} + \frac {\mu}{\bar {L}}\right) ^ {2} + \frac {1}{p} \left(\frac {1}{\Gamma_ {t}} + \frac {\mu}{\bar {L}}\right)}. \\ \end{array}
$$
Let us temporarily denote  $a_{t} \coloneqq \left(\frac{1}{\Gamma_{t}} + \frac{\mu}{L}\right)$  for all  $t \geq 0$ . Note that  $a_{t}$  is a non-increasing sequence, since  $\Gamma_{t+1} \geq \Gamma_{t}$  for all  $t \geq 0$ . Therefore,
$$
\bar {\theta} _ {t + 1} = - \frac {1}{2} a _ {t} + \sqrt {\frac {1}{4} a _ {t} ^ {2} + \frac {1}{p} a _ {t}}.
$$
Let us take the derivative of the last term w.r.t.  $a_{t}$  and compare it to zero:
$$
\begin{array}{l} - \frac {1}{2} + \frac {\frac {1}{2} a _ {t} + \frac {1}{p}}{2 \sqrt {\frac {1}{4} a _ {t} ^ {2} + \frac {1}{p} a _ {t}}} \geq 0 \Leftrightarrow \frac {1}{2} a _ {t} + \frac {1}{p} \geq \sqrt {\frac {1}{4} a _ {t} ^ {2} + \frac {1}{p} a _ {t}} \\ \Leftrightarrow \frac {1}{4} a _ {t} ^ {2} + \frac {1}{p} a _ {t} + \frac {1}{p ^ {2}} \geq \frac {1}{4} a _ {t} ^ {2} + \frac {1}{p} a _ {t} \Leftrightarrow \frac {1}{p ^ {2}} \geq 0. \\ \end{array}
$$