SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (k ^ {p} + \gamma) k ^ {p} \left\langle \tilde {\mathbb {A}} x ^ {k + 1} - \tilde {\mathbb {A}} x ^ {k}, x ^ {k + 1} - x ^ {k} \right\rangle \\ = (k ^ {p} + \gamma) k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {k} \right\rangle + (k ^ {p} + \gamma) k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, \tilde {\mathbf {A}} x ^ {k} \right\rangle \\ - (k ^ {p} + \gamma) k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k + 1} - x ^ {k} \right\rangle - (k ^ {p} + \gamma) k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, \tilde {\mathbf {A}} x ^ {k} \right\rangle \\ = - \left(k ^ {p} + \gamma\right) \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, - k ^ {p} \left(\tilde {\mathbf {A}} x ^ {k} + \left(x ^ {k + 1} - x ^ {k}\right)\right) \right\rangle + k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k}, - \left(k ^ {p} + \gamma\right) \left(\tilde {\mathbf {A}} x ^ {k + 1} + \left(x ^ {k + 1} - x ^ {k}\right)\right) \right\rangle \\ = - \left(k ^ {p} + \gamma\right) \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, \left(k ^ {p} + \gamma\right) \tilde {\mathbf {A}} x ^ {k + 1} + \gamma \left(x ^ {k + 1} - x ^ {0}\right) \right\rangle + k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k}, k ^ {p} \tilde {\mathbf {A}} x ^ {k} + \gamma \left(x ^ {k} - x ^ {0}\right) \right\rangle \\ = - \left(k ^ {p} + \gamma\right) \left(\left(k ^ {p} + \gamma\right) \| \tilde {\mathbf {A}} x ^ {k + 1} \| ^ {2} + \gamma \langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \rangle\right) + k ^ {p} \left(\| \tilde {\mathbf {A}} x ^ {k} \| ^ {2} + \gamma \langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \rangle\right). \\ \end{array}
$$
Now dividing both sides by  $k^p (k^p +\gamma)$  , we get the desired result.
- Proof for (44)
This can be checked by just expanding the inner product of left hand side.
- Proof for (45)
$$
\begin{array}{l} c _ {k + 1} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - c _ {k} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = c _ {k + 1} \left(\left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - \left\| x ^ {k} - x ^ {0} \right\| ^ {2}\right) + \left(c _ {k + 1} - c _ {k}\right) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = c _ {k + 1} \left\langle x ^ {k + 1} - x ^ {k}, \left(x ^ {k + 1} - x ^ {0}\right) + \left(x ^ {k} - x ^ {0}\right) \right\rangle + \left(c _ {k + 1} - c _ {k}\right) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = c _ {k + 1} \left(\left\langle x ^ {k + 1} - x ^ {k}, x ^ {k + 1} - x ^ {0} \right\rangle + \left\langle x ^ {k + 1} - x ^ {k}, x ^ {k} - x ^ {0} \right\rangle\right) + \left(c _ {k + 1} - c _ {k}\right) \left\| x ^ {k} - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
Reorganizing (40), we get two different expressions for  $x^{k + 1} - x^k$ .
$$
x ^ {k + 1} - x ^ {k} = - \left(\tilde {\mathbb {A}} x ^ {k + 1} + \frac {k ^ {p}}{k ^ {p} + \gamma} \tilde {\mathbb {A}} x ^ {k}\right) - \frac {\gamma}{k ^ {p} + \gamma} \left(x ^ {k} - x ^ {0}\right)
$$
$$
x ^ {k + 1} - x ^ {k} = - \left(\frac {k ^ {p} + \gamma}{k ^ {p}} \tilde {\mathbb {A}} x ^ {k + 1} + \tilde {\mathbb {A}} x ^ {k}\right) - \frac {\gamma}{k ^ {p}} \left(x ^ {k + 1} - x ^ {0}\right).
$$
Now plugging these to previous equality, be get the desired result.
$$
\begin{array}{l} c _ {k + 1} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - c _ {k} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = - c _ {k + 1} \left(\left\langle \frac {k ^ {p} + \gamma}{k ^ {p}} \tilde {\mathbf {A}} x ^ {k + 1} + \tilde {\mathbf {A}} x ^ {k} + \frac {\gamma}{k ^ {p}} \left(x ^ {k + 1} - x ^ {0}\right), x ^ {k + 1} - x ^ {0} \right\rangle \right. \\ \left. \left. + \left\langle \tilde {\mathbf {A}} x ^ {k + 1} + \frac {k ^ {p}}{k ^ {p} + \gamma} \tilde {\mathbf {A}} x ^ {k} + \frac {\gamma}{k ^ {p} + \gamma} \left(x ^ {k} - x ^ {0}\right), x ^ {k} - x ^ {0} \right\rangle\right) + \left(c _ {k + 1} - c _ {k}\right) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \right. \\ = - c _ {k + 1} \frac {k ^ {p} + \gamma}{k ^ {p}} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \right\rangle - c _ {k + 1} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k} - x ^ {0} \right\rangle \\ - c _ {k + 1} \left\langle \tilde {\mathbb {A}} x ^ {k}, x ^ {k + 1} - x ^ {0} \right\rangle - c _ {k + 1} \frac {k ^ {p}}{k ^ {p} + \gamma} \left\langle \tilde {\mathbb {A}} x ^ {k}, x ^ {k} - x ^ {0} \right\rangle \\ - c _ {k + 1} \frac {\gamma}{k ^ {p}} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - c _ {k + 1} \frac {\gamma}{k ^ {p} + \gamma} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} + (c _ {k + 1} - c _ {k}) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ \end{array}
$$
(i)  $\| \tilde{\mathbf{A}} (x^k)\| ^2 = \mathcal{O}(1)$  for  $p > 0,\gamma >0$
Plugging
$$
a _ {k} = 1 + \frac {\gamma}{2 k ^ {p}}, \qquad \qquad b _ {k} = \frac {\gamma}{k ^ {p}}, \qquad \qquad c _ {k + 1} = \frac {\gamma k ^ {p}}{4 \left(k ^ {p} + \frac {\gamma}{2}\right)} \left(\frac {\gamma}{k ^ {2 p}} - \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right)
$$
$$
\lambda_ {k} = 1 + \frac {\gamma}{2 k ^ {p}}, \quad \tau_ {k} = c _ {k + 1}
$$
to (41) and (42), we obtain
$$
\begin{array}{l} U ^ {k + 1} - U ^ {k} + \left(1 + \frac {\gamma}{2 k ^ {p}} - \frac {\gamma k ^ {p}}{4 (k ^ {p} + \frac {\gamma}{2})} \left(\frac {\gamma}{k ^ {2 p}} - \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right)\right) \langle \tilde {\mathbb {A}} x ^ {k + 1} - \tilde {\mathbb {A}} x ^ {k}, x ^ {k + 1} - x ^ {k} \rangle \\ = - \left(\frac {\gamma (2 k ^ {p} + \gamma)}{2 k ^ {2 p}} + \frac {\gamma}{2} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \| \tilde {\mathbf {A}} x ^ {k + 1} \| ^ {2} - \frac {\gamma (2 k ^ {p} + \gamma)}{2 k ^ {p} (k ^ {p} + \gamma)} \| \tilde {\mathbf {A}} x ^ {k} \| ^ {2} \\ \left. - \left(\frac {\gamma^ {2}}{k ^ {2 p}} + \frac {\gamma}{2} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \rangle \right. \\ \left. - \left(\frac {\gamma^ {2}}{k ^ {p} (k ^ {p} + \gamma)} + \frac {\gamma k ^ {p}}{2 (k ^ {p} + \gamma)} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \rangle \right. \\ \left. - \frac {\gamma^ {2}}{2 (2 k ^ {p} + \gamma)} \left(\frac {\gamma}{k ^ {2 p}} - \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \| x ^ {k + 1} - x ^ {0} \| ^ {2} \right. \\ \end{array}
$$
$$
\begin{array}{l} - \frac {\gamma^ {2} k ^ {p}}{2 (k ^ {p} + \gamma) (2 k ^ {p} + \gamma)} \left(\frac {\gamma}{k ^ {2 p}} - \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \| x ^ {k} - x ^ {0} \| ^ {2} + \left(c _ {k + 1} - c _ {k}\right) \| x ^ {k} - x ^ {0} \| ^ {2} \\ = - \frac {\gamma (2 k ^ {p} + \gamma)}{2 k ^ {2 p}} \left\| \tilde {\mathbb {A}} x ^ {k + 1} + \frac {1}{2 k ^ {p} + \gamma} \left(\gamma + \frac {k ^ {2 p}}{2} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \left(x ^ {k + 1} - x ^ {0}\right) \right\| ^ {2} \\ \left. - \frac {\gamma (2 k ^ {p} + \gamma)}{2 k ^ {p} (k ^ {p} + \gamma)} \left\| \tilde {\mathbf {A}} x ^ {k} + \frac {1}{2 k ^ {p} + \gamma} \left(\gamma + \frac {k ^ {2 p}}{2} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right)\right) \left(x ^ {k} - x ^ {0}\right) \right\| ^ {2} \right. \\ - \frac {1}{2} \left(\frac {\gamma}{k ^ {p}} - \frac {\gamma}{(k + 1) ^ {p}}\right) \| \tilde {\mathbf {A}} x ^ {k + 1} \| ^ {2} \\ - \underbrace {\frac {\gamma}{2 (2 k ^ {p} + \gamma)} \left(2 \gamma \left(\frac {1}{(k + 1) ^ {p}} - \frac {1}{k ^ {p}}\right) + \frac {k ^ {2 p}}{4} \left(\frac {1}{k ^ {p}} - \frac {1}{(k + 1) ^ {p}}\right) ^ {2}\right)} _ {s _ {k, 1}} \| x ^ {k + 1} - x ^ {0} \| ^ {2} \\ - \underbrace {\frac {\gamma k ^ {p}}{2 (k ^ {p} + \gamma) (2 k ^ {p} + \gamma)} \left(2 \gamma \left(\frac {1}{(k + 1) ^ {p}} - \frac {1}{k ^ {p}}\right) + \frac {k ^ {2 p}}{4} \left(\frac {1}{k ^ {p}}\right) ^ {2} - \frac {1}{(k + 1) ^ {p}}\right)} _ {s _ {k, 0}} \| x ^ {k} - x ^ {0} \| ^ {2} \\ + \left(c _ {k + 1} - c _ {k}\right) \left\| x ^ {k} - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
The continuous counterpart of above equality is
$$
\begin{array}{l} \dot {U} _ {1} (t) + \left\langle \frac {d}{d t} \tilde {\mathbf {A}} (X (t)), \dot {X} (t) \right\rangle \\ = - \frac {\gamma}{t ^ {p}} \left\| \tilde {\mathbf {A}} (X (t)) + \left(\frac {\gamma}{t ^ {p}} + \frac {p}{t}\right) (X (t) - X _ {0}) \right\| ^ {2} - \frac {1}{2} \frac {\gamma p (p - 1)}{t ^ {p + 2}} \| X (t) - X _ {0} \| ^ {2}, \\ \end{array}
$$
which can be obtained by differentiating and reorganizing the conservation law for  $E_{1}$ . Note, as  $\frac{k^{2p}}{2}\left(\frac{1}{k^p} -\frac{1}{(k + 1)^p}\right) = \mathcal{O}\left(k^{2p}\right)\mathcal{O}\left(\frac{1}{k^{p + 1}}\right) = \mathcal{O}\left(\frac{1}{k^{1 - p}}\right)$ , the order of the term  $\frac{1}{2k^p + \gamma}\left(\gamma +\frac{k^{2p}}{2}\left(\frac{1}{k^p} -\frac{1}{(k + 1)^p}\right)\right)$  corresponds to  $\left(\frac{\gamma}{t^p} +\frac{p}{t}\right)$ . Thus we can see the sum of first two terms on the right hand side of the equality for discrete setting corresponds to the first term of the right hand side of the equality for continuous setting.
We can observe that  $s_{k,1}, s_{k,0} = \mathcal{O}\left(\frac{1}{k^{2p + 1}}\right) + \mathcal{O}\left(\frac{1}{k^{p + 2}}\right)$ . And since  $c_k = \mathcal{O}\left(\frac{1}{k^{2p}}\right) + \mathcal{O}\left(\frac{1}{k^{p + 1}}\right)$ , we have  $c_{k + 1} - c_k = \mathcal{O}\left(\frac{1}{k^{2p + 1}}\right) + \mathcal{O}\left(\frac{1}{k^{p + 2}}\right)$  as well. Therefore the coefficients of  $\| x^{k + 1} - x^0\|^2$  and  $\| x^k -x^0\|^2$  are summable for  $p > 0$ . From Lemma G.1, we know  $\| x^{k + 1} - x^0\|^2$  and  $\| x^k -x^0\|^2$  are bounded by  $4\| x^0 -x^\star \| ^2$ .
The term  $\left\langle \tilde{\mathbb{A}} x^{k + 1} - \tilde{\mathbb{A}} x^k,x^{k + 1} - x^k\right\rangle$  on the left hand side is nonnegative from monotonicity of  $\tilde{\mathbf{A}}$ , and the coefficient is nonnegative as well. The coefficient of  $\left\| \tilde{\mathbf{A}} x^{k + 1}\right\| ^2$  in the right hand side,  $-\frac{1}{2}\left(\frac{\gamma}{k^p} -\frac{\gamma}{(k + 1)^p}\right)$ , is nonpositive. As a result, we get below inequality.
$$
\begin{array}{l} U ^ {k + 1} \leq U ^ {k} + \frac {1}{2} \left(\frac {\gamma}{k ^ {p}} - \frac {\gamma}{(k + 1) ^ {p}}\right) \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} + (c _ {k + 1} - c _ {k}) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ \leq U ^ {1} + 2 \left\| x ^ {0} - x ^ {\star} \right\| ^ {2} \sum_ {m = 1} ^ {\infty} \left(\left(\frac {\gamma}{k ^ {p}} - \frac {\gamma}{(k + 1) ^ {p}}\right) + 2 \left(c _ {k + 1} - c _ {k}\right)\right) = M. \\ \end{array}
$$
As done in (8), by monotonicity of  $\mathbb{A}$  and Young's inequality
$$
\begin{array}{l} M \geq U ^ {k} = a _ {k} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} + b _ {k} \left\langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \right\rangle + c _ {k} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ \geq a _ {k} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} + b _ {k} \left\langle \tilde {\mathbf {A}} x ^ {k}, x ^ {\star} - x ^ {0} \right\rangle \\ \geq a _ {k} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} - \frac {1}{2} \left(\left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} + b _ {k} ^ {2} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}\right) = \frac {1}{2} \left(1 + \frac {\gamma}{k ^ {p}}\right) \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} + \frac {\gamma^ {2}}{2 k ^ {2 p}} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
Reorganizing, we get the desired result
$$
\begin{array}{l} \left\| \tilde {\mathbb {A}} x ^ {k} \right\| ^ {2} \leq 2 \left(1 - \frac {\gamma}{k ^ {p}}\right) ^ {- 1} \left(M + \frac {\gamma^ {2}}{2 k ^ {2 p}} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}\right) \\ = 2 \left(1 + \frac {\gamma}{k ^ {p} - \gamma}\right) \left(M + \frac {\gamma^ {2}}{2 k ^ {2 p}} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}\right) = \mathcal {O} (1). \\ \end{array}
$$
(ii)  $\left\| \tilde{\mathbf{A}} (x^k)\right\| ^2 = \mathcal{O}\left(\frac{1}{k^{2p}}\right)$  for  $0 <   p <   1,\gamma >0$  Plugging
$$
\begin{array}{c c c} a _ {k + 1} = k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right), & b _ {k + 1} = \gamma k ^ {p}, & c _ {k + 1} = \frac {k ^ {p} \gamma^ {2}}{4 \left(k ^ {p} + \frac {\gamma}{2}\right)} \\ \lambda_ {k} = a _ {k + 1}, & \tau_ {k} = c _ {k + 1} & \end{array}
$$
to (41) and (42) with  $p = 1$ , we obtain
$$
\begin{array}{l} U ^ {k + 1} - U ^ {k} + \left(k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right) - \frac {k ^ {p} \gamma^ {2}}{4 \left(k ^ {p} + \frac {\gamma}{2}\right)}\right) \langle \tilde {\mathbb {A}} x ^ {k + 1} - \tilde {\mathbb {A}} x ^ {k}, x ^ {k + 1} - x ^ {k} \rangle \\ = - \gamma \left(k ^ {p} + \frac {\gamma}{2}\right) \| \tilde {\mathbf {A}} x ^ {k + 1} \| ^ {2} - \gamma^ {2} \langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \rangle - \frac {\gamma^ {3}}{4 \left(k ^ {p} + \frac {\gamma}{2}\right)} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} \\ + \underbrace {\left(\frac {k ^ {2 p} \left(k ^ {p} + \frac {\gamma}{2}\right)}{k ^ {p} + \gamma} - (k - 1) ^ {p} \left((k - 1) ^ {p} + \frac {\gamma}{2}\right)\right)} _ {= q _ {k}} \| \tilde {\mathbf {A}} x ^ {k} \| ^ {2} + \underbrace {\frac {\gamma \left(- (k - 1) ^ {p} k ^ {p} + k ^ {2 p} - \gamma (k - 1) ^ {p}\right)}{k ^ {p} + \gamma}} _ {= r _ {k}} \langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \rangle \\ - \frac {\gamma^ {3} k ^ {p}}{4 (k ^ {p} + \gamma) (k ^ {p} + \frac {\gamma}{2})} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} + \frac {\gamma^ {3} (k ^ {p} - (k - 1) ^ {p})}{8 ((k - 1) ^ {p} + \frac {\gamma}{2}) (k ^ {p} + \frac {\gamma}{2})} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = - \gamma \left(k ^ {p} + \frac {\gamma}{2}\right) \left\| \tilde {\mathbf {A}} x ^ {k + 1} + \frac {\gamma}{2 k ^ {p} + \gamma} \left(x ^ {k + 1} - x ^ {0}\right) \right\| ^ {2} + q _ {k} \left\| \tilde {\mathbf {A}} x ^ {k} + \frac {r _ {k}}{2 q _ {k}} \left(x ^ {k} - x ^ {0}\right) \right\| ^ {2} \\ - \underbrace {\left(\frac {r _ {k} ^ {2}}{4 q _ {k}} + \frac {\gamma^ {3} k ^ {p}}{4 (k ^ {p} + \gamma) (k ^ {p} + \frac {\gamma}{2})} + \frac {\gamma^ {3} (k ^ {p} - (k - 1) ^ {p})}{8 ((k - 1) ^ {p} + \frac {\gamma}{2}) (k ^ {p} + \frac {\gamma}{2})}\right)} _ {s _ {k}} \| x ^ {k} - x ^ {0} \| ^ {2}. \\ \end{array}
$$
The continuous counterpart of this equality is
$$
\begin{array}{l} \dot {U} _ {2} (t) + 2 t ^ {2 p} \left\langle \frac {d}{d t} \tilde {\mathbf {A}} (X (t)), \dot {X} (t) \right\rangle \\ = - 2 t ^ {p} (\gamma - p t ^ {p - 1}) \left\| \tilde {\mathbf {A}} (X (t)) + \frac {\gamma}{t ^ {p}} (X (t) - X _ {0}) \right\| ^ {2} - \gamma p (p - 1) t ^ {p - 2} \| X (t) - X _ {0} \| ^ {2} \\ \end{array}
$$
which can be obtained by differentiating the conservation law for  $E_{2}$ . The first term on the right hand side corresponds to the sum of first two terms in the right hand side of discrete equality. Thus we may expect the order of the coefficients for the terms would match, and we will check the expectation is indeed true.
With some calculation, we can observe
$$
s _ {k} = \frac {2 \gamma^ {2} (k - 1) ^ {p} k ^ {2 p} \left((k - 1) ^ {p} - k ^ {p}\right) ^ {2}}{4 \left((k - 1) ^ {p} + \frac {\gamma}{2}\right) \left(k ^ {p} + \frac {\gamma}{2}\right) d _ {k}},
$$
$$
d _ {k} = 2 k ^ {p} (k - 1) ^ {p} \left((k - 1) ^ {p} + \frac {\gamma}{2}\right) - 2 k ^ {3 p} - \gamma k ^ {2 p} + 2 \gamma (k - 1) ^ {p} \left((k - 1) ^ {p} + \frac {\gamma}{2}\right).
$$
By considering Newton expansion and from  $0 < p < 1$ , we see
$$
\begin{array}{l} d _ {k} = 2 k ^ {3 p} + \gamma k ^ {2 p} + \mathcal {O} (k ^ {3 p - 1}) - 2 k ^ {3 p} - \gamma k ^ {2 p} + 2 \gamma k ^ {2 p} + \mathcal {O} (k ^ {p}) \\ = 2 \gamma k ^ {2 p} + \mathcal {O} \left(k ^ {3 p - 1}\right) + \mathcal {O} \left(k ^ {p}\right) = \mathcal {O} \left(k ^ {2 p}\right). \\ \end{array}
$$
Thus we can check the leading order of numerator is  $p + 2p + (2p - 2) = 5p - 2$ , and leading order of the denominator is  $p + p + 2p = 4p$ . As  $5p - 2 - 4p = p - 2$ , we have  $s_k \in \mathcal{O}\left(k^{p-2}\right)$ , which matches with the continuous counterpart. Therefore  $\sum_{k=1}^{\infty} s_k < \infty$ .
On the other hand, we see
$$
\begin{array}{l} q _ {k} = \frac {k ^ {2 p} \left(k ^ {p} + \frac {\gamma}{2}\right)}{k ^ {p} + \gamma} - (k - 1) ^ {p} \left((k - 1) ^ {p} + \frac {\gamma}{2}\right) \\ \leq k ^ {2 p} - \left(k ^ {p} - p k ^ {p - 1} + \mathcal {O} \left(k ^ {p - 2}\right)\right) \left(\left(k ^ {p} - p k ^ {p - 1} + \mathcal {O} \left(k ^ {p - 2}\right)\right) + \frac {\gamma}{2}\right) \\ = - \frac {\gamma}{2} k ^ {p} + 2 p k ^ {2 p - 1} + \mathcal {O} \left(k ^ {2 p - 2}\right). \\ \end{array}
$$
Since  $p > 2p - 1$ , we have  $\lim_{k\to \infty}q_k = -\infty$ . Note this matches with the continuous counterpart as well.
Therefore there is  $N > 0$  such that for  $k > N$ ,  $q_{k} < 0$ . Now for  $k > N$  we have
$$
U _ {k + 1} \leq U _ {k} + s _ {k} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \leq U _ {N} + 4 \left(\sum_ {m = N} ^ {\infty} s _ {m}\right) \left\| x ^ {0} - x ^ {\star} \right\| ^ {2} = M.
$$
Thus for  $k > N$ , by monotonicity of  $\mathbb{A}$  and Young's inequality
$$
\begin{array}{l} M \geq U ^ {k + 1} = k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right) \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} + \gamma k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \right\rangle + \frac {k ^ {p} \gamma^ {2}}{4 \left(k ^ {p} + \frac {\gamma}{2}\right)} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} \\ \geq k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right) \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} + \gamma k ^ {p} \left\langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {\star} - x ^ {0} \right\rangle \\ \geq k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right) \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} - \frac {k ^ {p}}{2} \left(\left(k ^ {p} + \frac {\gamma}{2}\right) \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} + \frac {\gamma^ {2}}{k ^ {p} + \frac {\gamma}{2}} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}\right) \\ = \frac {k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right)}{2} \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} - \frac {k ^ {p} \gamma^ {2}}{2 k ^ {p} + \gamma} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
Reorganizing, we get the desired result.
$$
\left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} \leq \frac {2}{k ^ {p} \left(k ^ {p} + \frac {\gamma}{2}\right)} \left(M + \frac {\gamma^ {2}}{2} \left\| x ^ {0} - x ^ {\star} \right\| ^ {2}\right) = \mathcal {O} \left(\frac {1}{k ^ {2 p}}\right).
$$
(iii)  $\left\| \tilde{\mathbf{A}} (x^k)\right\| ^2 = \mathcal{O}\left(\frac{1}{k^2}\right)$  for  $p = 1,\gamma \geq 1$  Plugging
$$
a _ {k + 1} = k ^ {2}, \quad b _ {k + 1} = \gamma \left(k - \frac {1}{2} (\gamma - 1)\right), \quad c _ {k + 1} = \frac {1}{4} \gamma (\gamma - 1)
$$
$$
\lambda_ {k} = k (k + 1), \quad \tau_ {k} = c _ {k + 1}
$$
$$
\begin{array}{l} U ^ {k + 1} - U ^ {k} + \left(k (k + 1) - \frac {1}{4} \gamma (\gamma - 1)\right) \left\langle \tilde {\mathbb {A}} x ^ {k + 1} - \tilde {\mathbb {A}} x ^ {k}, x ^ {k + 1} - x ^ {k} \right\rangle \\ = - (\gamma - 1) (k + 1) \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} - \frac {k ^ {2} (\gamma - 1)}{k + \gamma} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} \\ - \frac {\gamma (\gamma - 1) (k + \frac {\gamma}{4})}{k} \langle \tilde {\mathbf {A}} x ^ {k + 1}, x ^ {k + 1} - x ^ {0} \rangle - \frac {\gamma (\gamma - 1) (k - \frac {\gamma}{4})}{k + \gamma} \langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \rangle \\ - \frac {\gamma^ {2} (\gamma - 1)}{4 k} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - \frac {\gamma^ {2} (\gamma - 1)}{4 (k + \gamma)} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = - (\gamma - 1) (k + 1) \left\| \tilde {\mathbf {A}} x ^ {k + 1} + \frac {\gamma}{2 (k + 1)} \left(1 + \frac {\gamma}{4 k}\right) \left(x ^ {k + 1} - x ^ {0}\right) \right\| ^ {2} \\ - \frac {k ^ {2} (\gamma - 1)}{k + \gamma} \left\| \tilde {\mathbf {A}} x ^ {k} + \frac {\gamma}{2 k} \left(1 - \frac {\gamma}{4 k}\right) \left(x ^ {k} - x ^ {0}\right) \right\| ^ {2} \\ -\underbrace{\frac{\gamma^{2}(\gamma - 1)}{64k(k + 1)}\left(8(\gamma - 2) - \frac{\gamma^{2}}{k}\right)}_{= s_{k,1} = \mathcal{O}\bigl(\frac{1}{k^{2}}\bigr)}\bigl {\| x^{k + 1} - x^{0}\bigr \|^{2} - \underbrace{\frac{\gamma^{3}(\gamma - 1)}{64k(k + \gamma)}\left(8 - \frac{\gamma}{k}\right)}_{= s_{k,0} = \mathcal{O}\bigl(\frac{1}{k^{2}}\bigr)}\bigl {\| x^{k} - x^{0}\bigr \|^{2}. \\ \end{array}
$$
$$
\dot {U} _ {3} (t) + 2 t ^ {2} \left\langle \frac {d}{d t} \tilde {\mathbf {A}} (X (t)), \dot {X} (t) \right\rangle = - 2 t (\gamma - 1) \left\| \tilde {\mathbf {A}} (X (t)) + \frac {\gamma}{t} (X (t) - X _ {0}) \right\| ^ {2}
$$
which can be obtained by differentiating the conservation law for  $E_{3}$ . Note the terms match with same order of coefficients, except two  $\left\| x^{k + 1} - x^0\right\|^2$  and  $\left\| x^k -x^0\right\|^2$ , while these terms are summable as  $s_{k,1},s_{k,0} = \mathcal{O}\left(\frac{1}{k^2}\right)$ . Therefore we have
$$
\begin{array}{l} U ^ {k + 1} \leq U ^ {k} - s _ {k, 1} \left\| x ^ {k + 1} - x ^ {0} \right\| ^ {2} - s _ {k, 0} \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ \leq U ^ {1} + 4 \left\| x ^ {0} - x ^ {\star} \right\| ^ {2} \sum_ {m = 1} ^ {\infty} \left(s _ {m, 1} + s _ {m, 0}\right) = M. \\ \end{array}
$$
Thus by monotonicity of  $\mathbb{A}$  and Young's inequality
$$
\begin{array}{l} M \geq U ^ {k} = k ^ {2} \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} + \gamma \left(k - \frac {1}{2} (\gamma - 1)\right) \left\langle \tilde {\mathbf {A}} x ^ {k}, x ^ {k} - x ^ {0} \right\rangle + \frac {1}{4} \gamma (\gamma - 1) \left\| x ^ {k} - x ^ {0} \right\| ^ {2} \\ = k ^ {2} \left\| \tilde {\mathbf {A}} x ^ {k + 1} \right\| ^ {2} + \gamma \left(k - \frac {1}{2} (\gamma - 1)\right) \langle \tilde {\mathbf {A}} x ^ {k}, x ^ {\star} - x ^ {0} \rangle \\ \geq k ^ {2} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} - \frac {1}{2} \left(\left(k - \frac {1}{2} (\gamma - 1)\right) ^ {2} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} + \gamma^ {2} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}\right) \\ \geq \frac {k ^ {2}}{2} \left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} - \frac {\gamma^ {2}}{2} \left\| x ^ {\star} - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
$$
\left\| \tilde {\mathbf {A}} x ^ {k} \right\| ^ {2} \leq \frac {2}{k ^ {2}} \left(M + \frac {\gamma^ {2}}{2} \left\| x ^ {0} - x ^ {\star} \right\| ^ {2}\right) = \mathcal {O} \left(\frac {1}{k ^ {2}}\right).
$$
(iv)  $\left\| \tilde{\mathbf{A}} (x^k)\right\| ^2 = \mathcal{O}\left(\frac{1}{k^{2\gamma}}\right)$  for  $p = 1,0 <   \gamma <  1$  Plugging
$$
a _ {k} = k ^ {2 \gamma}, \qquad \qquad \qquad b _ {k} = \gamma k \left(k - \frac {1}{4}\right) ^ {2 \gamma - 2}, \qquad \qquad c _ {k + 1} = \frac {1}{4} \gamma (\gamma - 1) k ^ {2 \gamma - 2}
$$
$$
\lambda_ {k} = k ^ {2 \gamma - 1} (k + \gamma), \quad \tau_ {k} = c _ {k + 1}
$$