SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I Shamir. Reproducibility in optimization: Theoretical framework and limits. Advances in Neural Information Processing Systems, 35, 2022.
[2] Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. Advances in Neural Information Processing Systems, 31, 2018.
[3] Zeyuan Allen-Zhu and Elad Hazan. Optimal black-box reductions between optimization objectives. Advances in Neural Information Processing Systems, 29, 2016.
[4] Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
[5] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-223. PMLR, 2017.
[6] Amit Attia and Tomer Koren. Algorithmic instabilities of accelerated gradient descent. Advances in Neural Information Processing Systems, 34:1204-1214, 2021.
[7] Amit Attia and Tomer Koren. Uniform stability for first-order empirical risk minimization. In Conference on Learning Theory, pages 3313-3332. PMLR, 2022.
[8] Hedy Attouch, Zaki Chbani, and Hassan Riahi. Combining fast inertial dynamics for convex optimization with Tikhonov regularization. Journal of Mathematical Analysis and Applications, 457(2):1065-1094, 2018.
[9] James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the ACM Conference on Economics and Computation, pages 321-338, 2018.
[10] Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533:452-454, 2016.
[11] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33:4381-4391, 2020.
[12] Heinz H Bauschke, Patrick L Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011.
[13] Richard Bellman. Stability theory of differential equations. Courier Corporation, 2008.
[14] Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. arXiv preprint arXiv:2002.12410, 2020.
[15] Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
[16] Digvijay Boob and Cristóbal Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. Mathematical Programming, pages 1-43, 2023.
[17] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[18] Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference on Machine Learning, pages 725-734. PMLR, 2019.
[72] Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252, 1995.
[73] John Von Neumann. On the theory of games of strategy. Contributions to the Theory of Games, 4:13-42, 1959.
[74] Junchi Yang, Siqi Zhang, Negar Kiyavash, and Niao He. A catalyst framework for minimax optimization. Advances in Neural Information Processing Systems, 33:5667-5678, 2020.
[75] TaeHo Yoon and Ernest K Ryu. Accelerated algorithms for smooth convex-concave minimax problems with  $O(1 / k^2)$  rate on squared gradient norm. In International Conference on Machine Learning, pages 12098-12109. PMLR, 2021.
[76] Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics, pages 568-576. PMLR, 2021.
[77] Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194(1-2):901-935, 2022.
[78] Liang Zhang, Kiran K Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. Advances in Neural Information Processing Systems, 35:35174-35187, 2022.
[79] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer:  $k$  steps forward, 1 step back. Advances in Neural Information Processing Systems, 32, 2019.
[80] Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex-strongly-concave minimax optimization. In Uncertainty in Artificial Intelligence, pages 482-492. PMLR, 2021.
1.1 Our Contributions 2
1.2 Related Works 3
2 Preliminaries in Algorithmic Reproducibility 4
3 Deterministic Gradient Oracle for Minimization Problems 5
3.1 Inexact Initialization Oracle 6
3.2 Inexact Deterministic Gradient Oracle 6
4 Deterministic Gradient Oracle for Minimax Problems 7
4.1 Regularization Helps! 8
4.2 Inexact Proximal Point Method 9
5 Stochastic Gradient Oracle for Minimax Problems 10
A Near-optimal Guarantees in the Minimization Case 17
A.1 Inexact Initialization Oracle 17
A.2 Inexact Deterministic Gradient Oracle 18
B Preliminary Results in the Minimax Case 22
B.1 Lower-bounds for Reproducibility 23
B.2 Guarantees of Gradient Descent Ascent 23
B.2.1 General Analysis 23
B.2.2 Inexact Initialization Oracle 24
B.2.3 Inexact Deterministic Gradient Oracle 25
B.2.4 Stochastic Gradient Oracle 26
B.3 Guarantees of Extragradient 27
B.3.1 General Analysis 27
B.3.2 Inexact Initialization Oracle 29
B.3.3 Inexact Deterministic Gradient Oracle 29
B.3.4 More Discussions 31
C Near-optimal Guarantees in the Minimax Case 32
C.1 Useful Lemmas 32
C.2 Regularization Helps! 35
C.2.1 Inexact Initialization Oracle 35
C.2.2 Inexact Deterministic Gradient Oracle 36
C.3 Inexact Proximal Point Method 36
C.3.1 General Analysis 37
C.3.2 Inexact Initialization Oracle 37
C.3.3 Inexact Deterministic Gradient Oracle 38
D Numerical Experiments
A Near-optimal Guarantees in the Minimization Case
This section provides proof for the near-optimal guarantees of Algorithm 1 in the minimization case. We start with some commonly-used facts that follow from basic algebraic calculations. See Bauschke et al. [12] for an example.
Lemma A.1. The following facts will be used in the analysis. For any vectors  $a, b \in \mathbb{R}^d$ , it holds that
(i)  $2a^{\top}b = \| a\|^{2} + \| b\|^{2} - \| a - b\|^{2},$
(ii)  $2a^{\top}b = \| a + b\|^{2} - \| a\|^{2} - \| a\|^{2},$
(iii)  $-\gamma \| a\| ^2 -\frac{1}{\gamma}\| b\| ^2\leq 2a^\top b\leq \gamma \| a\| ^2 +\frac{1}{\gamma}\| b\| ^2,\quad \forall \gamma >0,$
$(iv)\| \eta a + (1 - \eta)b\| ^2 +\eta (1 - \eta)\| a - b\| ^2 = \eta \| a\| ^2 +(1 - \eta)\| b\| ^2,\quad \forall \eta \in \mathbb{R}.$
A.1 Inexact Initialization Oracle
This section contains proof of Lemma 3.2 and Theorem 3.3 for the near-optimal guarantees of Algorithm 1 in the inexact initialization case.
Proof of Lemma 3.2. By the optimality conditions of  $x_{r}^{*}$  and  $(x_{r}^{*})'$ , we have that for any  $x, x' \in \mathcal{X}$ ,
$$
\left(\nabla F \left(x _ {r} ^ {*}\right) + r \left(x _ {r} ^ {*} - x _ {0}\right)\right) ^ {\top} \left(x - x _ {r} ^ {*}\right) \geq 0,
$$
$$
\left(\nabla F \left(\left(x _ {r} ^ {*}\right) ^ {\prime}\right) + r \left(\left(x _ {r} ^ {*}\right) ^ {\prime} - x _ {0} ^ {\prime}\right)\right) ^ {\top} \left(x ^ {\prime} - \left(x _ {r} ^ {*}\right) ^ {\prime}\right) \geq 0.
$$
Taking  $x' = x_r^*$  and  $x = (x_r^*)'$  in the above equation, we obtain that
$$
\left(x _ {r} ^ {*} - (x _ {r} ^ {*}) ^ {\prime}\right) ^ {\top} \Big ((\nabla F (x _ {r} ^ {*}) + r (x _ {r} ^ {*} - x _ {0})) - (\nabla F ((x _ {r} ^ {*}) ^ {\prime}) + r ((x _ {r} ^ {*}) ^ {\prime} - x _ {0} ^ {\prime})) \Big) \leq 0.
$$
Since  $\nabla F$  is monotone when  $F$  is convex, rearranging terms, we get
$$
\begin{array}{l} 0 \geq (x _ {r} ^ {*} - (x _ {r} ^ {*}) ^ {\prime}) ^ {\top} (\nabla F (x _ {r} ^ {*}) - \nabla F ((x _ {r} ^ {*}) ^ {\prime})) + r \| x _ {r} ^ {*} - (x _ {r} ^ {*}) ^ {\prime} \| ^ {2} - r (x _ {r} ^ {*} - (x _ {r} ^ {*}) ^ {\prime}) ^ {\top} (x _ {0} - x _ {0} ^ {\prime}) \\ \geq r \left\| x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime} \right\| ^ {2} - r \left(x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime}\right) ^ {\top} \left(x _ {0} - x _ {0} ^ {\prime}\right). \\ \end{array}
$$
Given  $r > 0$ , this means
$$
\begin{array}{l} \left\| x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime} \right\| ^ {2} \leq \left(x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime}\right) ^ {\top} \left(x _ {0} - x _ {0} ^ {\prime}\right) \\ \leq \left\| x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime} \right\| \left\| x _ {0} - x _ {0} ^ {\prime} \right\|. \\ \end{array}
$$
Dividing both sides by  $\| x_r^* - (x_r^*)' \|$ , the proof is complete.
By converging sufficiently close to the optimal solution, we can ensure Algorithm 1 is reproducible. The near-optimal convergence rate is achieved using AGD [60] as the base algorithm.
Proof of Theorem 3.3. We first analyze the convergence guarantee. Let  $x^{*} \in \arg \min_{x \in \mathcal{X}} F(x)$  be one minimizer of  $F(x)$ , and  $x_{r}^{*} = \arg \min_{x \in \mathcal{X}} F_{r}(x)$  be the unique minimizer of  $F_{r}(x)$ . By the definition of  $F_{r}(x)$ , we have that
$$
\begin{array}{l} F (x _ {r}) - F (x ^ {*}) = F _ {r} (x _ {r}) - \frac {r}{2} \| x _ {r} - x _ {0} \| ^ {2} - F _ {r} (x ^ {*}) + \frac {r}{2} \| x ^ {*} - x _ {0} \| ^ {2} \\ \leq F _ {r} \left(x _ {r}\right) - F _ {r} \left(x ^ {*}\right) + \frac {r}{2} \left\| x ^ {*} - x _ {0} \right\| ^ {2} \\ \leq F _ {r} \left(x _ {r}\right) - F _ {r} \left(x _ {r} ^ {*}\right) + \frac {\bar {r}}{2} \| x ^ {*} - x _ {0} \| ^ {2} \tag {4} \\ \leq \epsilon_ {r} + \frac {r D ^ {2}}{2}. \\ \end{array}
$$
$\epsilon_r$  and  $r$  will be selected later. For reproducibility, we proceed as
$$
\begin{array}{l} \left\| x _ {r} - x _ {r} ^ {\prime} \right\| \leq \left\| x _ {r} - x _ {r} ^ {*} \right\| + \left\| x _ {r} ^ {*} - \left(x _ {r} ^ {*}\right) ^ {\prime} \right\| + \left\| \left(x _ {r} ^ {*}\right) ^ {\prime} - x _ {r} ^ {\prime} \right\| \\ \leq \delta + 2 \sqrt {\frac {2 \epsilon_ {r}}{r}}. \\ \end{array}
$$
where we use the optimality condition of  $x_r^*$  by  $r$ -strong-convexity of  $F_{r}(x)$ :
$$
\begin{array}{l} \frac {r}{2} \left\| x _ {r} - x _ {r} ^ {*} \right\| ^ {2} \leq F _ {r} \left(x _ {r}\right) - F _ {r} \left(x _ {r} ^ {*}\right) \\ \leq \epsilon_ {r}, \\ \end{array}
$$
the optimality condition of  $(x_r^*)'$  and Lemma 3.2. Setting
$$
r = \frac {\epsilon}{D ^ {2}}, \quad \epsilon_ {r} = \frac {\epsilon}{2} \min  \left\{1, \frac {\delta^ {2}}{4 D ^ {2}} \right\},
$$
we guarantee that  $F(x_{r}) - F(x^{*}) \leq \epsilon$  and  $\| x_{r} - x_{r}^{\prime} \| \leq 2\delta$ . The gradient complexity of AGD to achieve  $\epsilon_{r}$  approximation error on the function value gap of an  $\ell$ -smooth and  $(\ell + r)$ -strongly convex function is  $\mathcal{O}(\sqrt{(\ell + r) / r} \log(1 / \epsilon_{r})) = \tilde{\mathcal{O}}(\sqrt{\ell D^{2} / \epsilon})$ , where  $\tilde{\mathcal{O}}$  hides logarithmic terms.
A.2 Inexact Deterministic Gradient Oracle
This section contains proof of Lemma 3.4 and Theorem 3.5 for the guarantees in the inexact deterministic gradient case. We first study the convergence behavior of AGD [60] for smooth and strongly-convex functions under the inexact gradient oracle. For the sake of simplicity and to enable a general analysis, we slightly abuse notation here to consider the optimization problem
$$
\min  _ {x \in \mathcal {X}} f (x),
$$
where  $f:\mathcal{X}\to \mathbb{R}$  satisfies the following assumption.
Assumption A.2.  $f(x)$  is  $\ell$ -smooth and  $\mu$ -strongly convex on the closed convex domain  $\mathcal{X}$ .
We consider the inexact gradient oracle defined below (referred to as  $\delta$ -oracle in this section).
Definition 7. (δ-oracle) At any querying point  $x\in \mathcal{X}$ , the  $\delta$ -oracle returns a vector  $g(x)\in \mathbb{R}^d$  such that  $\| g(x) - \nabla f(x)\| ^2\leq \delta^2$ , where  $\nabla f(x)$  is the true gradient of  $f(x)$ .
In previous work, Devolder et al. [27] define a different inexact oracle that is motivated by the exact first-order oracle and study the convergence behavior of first-order algorithms including AGD.
Definition 8.  $((\delta, \ell, \mu)$ -oracle [27]) At any querying point  $x \in \mathcal{X}$ , the  $(\delta, \ell, \mu)$ -oracle returns approximate first-order information  $(f_{\delta, \ell, \mu}(x), g_{\delta, \ell, \mu}(x))$  such that for any  $y \in \mathcal{X}$
$$
\frac {\mu}{2} \| x - y \| ^ {2} \leq f (y) - \left(f _ {\delta , \ell , \mu} (x) + g _ {\delta , \ell , \mu} (x) ^ {\top} (y - x)\right) \leq \frac {\ell}{2} \| x - y \| ^ {2} + \delta .
$$
The lemma below characterizes that the two oracles can be transformed into each other (adapted from Devolder et al. [27, 28]).
Lemma A.3. Under Assumption A.2. A  $\delta$ -oracle can be transformed to a  $(\delta', \ell', \mu')$ -oracle with  $\delta' = (1/(2\ell) + 1/\mu)\delta^2$ ,  $\ell' = 2\ell$ , and  $\mu' = \mu/2$ . A  $(\delta, \ell, \mu)$ -oracle can be transformed to a  $\delta'$ -oracle for  $\delta'$  defined in (7).
Proof. Given a  $\delta$ -oracle that returns  $g(x)$  at any point  $x \in \mathcal{X}$ , we construct a  $(\delta', \ell', \mu')$ -oracle as
$$
f _ {\delta^ {\prime}, \ell^ {\prime}, \mu^ {\prime}} (x) = f (x) - \frac {\delta^ {2}}{\mu}, \quad g _ {\delta^ {\prime}, \ell^ {\prime}, \mu^ {\prime}} (x) = g (x).
$$
By  $\ell$ -smoothness of  $f(x)$  and fact (iii) in Lemma A.1, we have that
$$
\begin{array}{l} f (y) \leq f (x) + \nabla f (x) ^ {\top} (y - x) + \frac {\ell}{2} \| x - y \| ^ {2} \\ = f (x) + g (x) ^ {\top} (y - x) + (\nabla f (x) - g (x)) ^ {\top} (y - x) + \frac {\ell}{2} \| x - y \| ^ {2} \tag {5} \\ \leq f (x) + g (x) ^ {\top} (y - x) + \ell \| x - y \| ^ {2} + \frac {\delta^ {2}}{2 \ell}. \\ \end{array}
$$
Similarly by  $\mu$ -strong convexity of  $f(x)$  and fact (iii) in Lemma A.1, we have that
$$
\begin{array}{l} f (y) \geq f (x) + \nabla f (x) ^ {\top} (y - x) + \frac {\mu}{2} \| x - y \| ^ {2} \\ = f (x) + g (x) ^ {\top} (y - x) + (\nabla f (x) - g (x)) ^ {\top} (y - x) + \frac {\mu}{2} \| x - y \| ^ {2} \tag {6} \\ \geq f (x) + g (x) ^ {\top} (y - x) + \frac {\mu}{4} \| x - y \| ^ {2} - \frac {\delta^ {2}}{\mu}. \\ \end{array}
$$
Combined the above two equations together, we obtain that
$$
\frac {\mu}{4} \| x - y \| ^ {2} \leq f (y) - \left(f (x) - \frac {\delta^ {2}}{\mu} + g (x) ^ {\top} (y - x)\right) \leq \ell \| x - y \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}.
$$
This concludes the proof of the first part. For the second part, given a  $(\delta, \ell, \mu)$ -oracle in Definition 8, we construct a  $\delta'$ -oracle as follows:  $g(x) = g_{\delta, \ell, \mu}(x)$ . Taking  $y = x$  in Definition 8, we obtain  $\forall x$ ,
$$
f _ {\delta , \ell , \mu} (x) \leq f (x) \leq f _ {\delta , \ell , \mu} (x) + \delta .
$$
Therefore, by strong-convexity of  $f(x)$ , we have that  $\forall x, y$
$$
\begin{array}{l} f (y) \geq f (x) + \nabla f (x) ^ {\top} (y - x) + \frac {\mu}{2} \| x - y \| ^ {2} \\ \geq f _ {\delta , \ell , \mu} (x) + \nabla f (x) ^ {\top} (y - x) + \frac {\mu}{2} \| x - y \| ^ {2}. \\ \end{array}
$$
Combined with the second part of Definition 8, we obtain that  $\forall x,y$
$$
(\nabla f (x) - g _ {\delta , \ell , \mu} (x)) ^ {\top} (y - x) \leq \frac {\ell - \mu}{2} \| x - y \| ^ {2} + \delta .
$$
Then by a similar proof as for the convex case in Devolder et al. [28]. Let  $\Delta(x) = \nabla f(x) - g_{\delta, \ell, \mu}(x)$  and  $y = x + \min \{\sqrt{2\delta / (\ell - \mu)}, r(x)\} \Delta(x) / \| \Delta(x) \|$  for  $r(x) = \max \{r \in \mathbb{R} \mid (x + r\Delta(x) / \| \Delta(x) \|) \in \mathcal{X}\}$ . We have that
$$
\left\| \nabla f (x) - g _ {\delta , \ell , \mu} (x) \right\| \leq \left\{ \begin{array}{l l} \sqrt {2 \delta (\ell - \mu)}, & \text {w h e n} \sqrt {\frac {2 \delta}{\ell - \mu}} \leq r (x), \\ \frac {\ell - \mu}{2} r (x) + \frac {\delta}{r (x)}, & \text {o t h e r w i s e .} \end{array} \right. \tag {7}
$$
Since we use  $g(x) = g_{\delta ,\ell ,\mu}(x)$ , the proof is complete.
Devolder et al. [27] prove that AGD equipped with  $(\delta, \ell, \mu)$ -oracle in Definition 8 converges to a  $\mathcal{O}(\delta \sqrt{\ell / \mu})$ -neighborhood of the optimal solution with accelerated rate  $T = \tilde{\mathcal{O}}(\sqrt{\ell / \mu})$ :
$$
f (x _ {T}) - f ^ {*} \leq \mathcal {O} \left(\exp \left(- T \sqrt {\frac {\mu}{\ell}}\right) + \delta \sqrt {\frac {\ell}{\mu}}\right),
$$
where  $x_{T}$  is the output of  $T$ -step AGD and  $f^{*}$  is the optimal value. They further establish a lower-bound showing tightness of the  $\mathcal{O}(\delta \sqrt{\ell / \mu})$  error for any first-order methods with accelerated rate.
Here, we are interested in the performance of AGD under the  $\delta$ -oracle in Definition 7. Motivated by the transformation in Lemma A.3, we choose the parameters in AGD as follows:
$$
\begin{array}{l} x _ {t + 1} = \Pi_ {\mathcal {X}} \left(y _ {t} - \frac {1}{2 \ell} g (y _ {t})\right), \\ y _ {t + 1} = x _ {t + 1} + \frac {2 - \sqrt {\mu / \ell}}{2 + \sqrt {\mu / \ell}} \left(x _ {t + 1} - x _ {t}\right). \tag {8} \\ \end{array}
$$
The results can be implied by Devolder et al. [27] together with Lemma A.3. We provide detailed proof in the following for completeness of the paper.
Lemma A.4. Under Assumption A.2. Let  $x^{*}$  be the unique minimizer of  $f(x)$  and  $\kappa = \ell / \mu$  be the condition number. Given an inexact  $\delta$ -oracle in Definition 7. Starting from  $y_{0} = x_{0}$ , AGD with updates (8) for  $t = 0, 1, \dots, T - 1$  converges with
$$
f (x _ {T}) - f (x ^ {*}) \leq \exp \left(- \frac {T}{2 \sqrt {\kappa}}\right) \left(f (x _ {0}) - f (x ^ {*}) + \frac {\mu}{4} \| x _ {0} - x ^ {*} \| ^ {2}\right) + \sqrt {\kappa} \left(\frac {1}{\ell} + \frac {2}{\mu}\right) \delta^ {2}.
$$
Proof. By (5) in the proof of Lemma A.3, we have that
$$
f (x _ {t + 1}) \leq f (y _ {t}) + g (y _ {t}) ^ {\top} (x _ {t + 1} - y _ {t}) + \ell \| x _ {t + 1} - y _ {t} \| ^ {2} + \frac {\delta^ {2}}{2 \ell}.
$$
Similarly by (6), we know for any  $x \in \mathcal{X}$
$$
f (x) \geq f (y _ {t}) + g (y _ {t}) ^ {\top} (x - y _ {t}) + \frac {\mu}{4} \| x - y _ {t} \| ^ {2} - \frac {\delta^ {2}}{\mu}.
$$
Combining the above two results, for any  $x \in \mathcal{X}$ , we have
$$
\begin{array}{l} f \left(x _ {t + 1}\right) - f (x) = f \left(x _ {t + 1}\right) - f \left(y _ {t}\right) + f \left(y _ {t}\right) - f (x) \\ \leq g \left(y _ {t}\right) ^ {\top} \left(x _ {t + 1} - x\right) + \ell \| x _ {t + 1} - y _ {t} \| ^ {2} - \frac {\mu}{4} \| x - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ \leq - 2 \ell \left(x _ {t + 1} - y _ {t}\right) ^ {\top} \left(x _ {t + 1} - x\right) + \ell \| x _ {t + 1} - y _ {t} \| ^ {2} - \frac {\mu}{4} \| x - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ = - \ell \| x _ {t + 1} - y _ {t} \| ^ {2} + 2 \ell (x _ {t + 1} - y _ {t}) ^ {\top} (x - y _ {t}) - \frac {\mu}{4} \| x - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}, \\ \end{array}
$$
where in the last inequality we use the optimality condition of the projection step such that  $\forall x\in \mathcal{X}$
$$
\left(x _ {t + 1} - y _ {t} + \frac {1}{2 \ell} g (y _ {t})\right) ^ {\top} (x - x _ {t + 1}) \geq 0.
$$
Let  $\theta := 1 / (2\sqrt{\kappa}) = \sqrt{\mu / (4\ell)}$ . Setting  $x = x_{t}$  and  $x = x^{*}$  in the above equation, we get
$$
\begin{array}{l} (1 - \theta) (f (x _ {t + 1}) - f (x _ {t})) \leq - \ell (1 - \theta) \| x _ {t + 1} - y _ {t} \| ^ {2} + 2 \ell (1 - \theta) (x _ {t + 1} - y _ {t}) ^ {\top} (x _ {t} - y _ {t}) \\ - \frac {\mu}{4} (1 - \theta) \| x _ {t} - y _ {t} \| ^ {2} + (1 - \theta) \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \theta \left(f \left(x _ {t + 1}\right) - f \left(x ^ {*}\right)\right) \leq - \ell \theta \| x _ {t + 1} - y _ {t} \| ^ {2} + 2 \ell \theta \left(x _ {t + 1} - y _ {t}\right) ^ {\top} \left(x ^ {*} - y _ {t}\right) \\ - \frac {\mu}{4} \theta \| x ^ {*} - y _ {t} \| ^ {2} + \theta \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}. \\ \end{array}
$$
Let  $\Delta_t \coloneqq f(x_t) - f(x^*) \geq 0$ . Summing the above two up, by fact (i) in Lemma A.1, we obtain
$$
\begin{array}{l} \Delta_ {t + 1} - (1 - \theta) \Delta_ {t} \leq - \ell \| x _ {t + 1} - y _ {t} \| ^ {2} + 2 \ell \left(x _ {t + 1} - y _ {t}\right) ^ {\top} ((1 - \theta) x _ {t} + \theta x ^ {*} - y _ {t}) - \frac {\mu}{4} \theta \| x ^ {*} - y _ {t} \| ^ {2} \\ - \frac {\mu}{4} (1 - \theta) \| x _ {t} - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}, \\ = \ell \| y _ {t} - (1 - \theta) x _ {t} - \theta x ^ {*} \| ^ {2} - \ell \| x _ {t + 1} - (1 - \theta) x _ {t} - \theta x ^ {*} \| ^ {2} - \frac {\mu}{4} \theta \| x ^ {*} - y _ {t} \| ^ {2} \\ - \frac {\mu}{4} (1 - \theta) \| x _ {t} - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}. \\ \end{array}
$$
Let  $\theta u_{t} := x_{t} - (1 - \theta)x_{t-1}$  for  $t \geq 1$ . From the update (8) of AGD, we observe
$$
\begin{array}{l} (1 + \theta) y _ {t} = (1 + \theta) x _ {t} + (1 - \theta) \left(x _ {t} - x _ {t - 1}\right) \\ = 2 x _ {t} - (1 - \theta) x _ {t - 1} \\ = x _ {t} + \theta u _ {t}. \\ \end{array}
$$
Rearranging terms, we can get  $x_{t} = (1 + \theta)y_{t} - \theta u_{t}$  and thus
$$
\begin{array}{l} y _ {t} - (1 - \theta) x _ {t} = y _ {t} - (1 - \theta) ((1 + \theta) y _ {t} - \theta u _ {t}) \\ = y _ {t} - \left((1 - \theta^ {2}) y _ {t} - (1 - \theta) \theta u _ {t}\right) \\ = \theta \left(\theta y _ {t} + (1 - \theta) u _ {t}\right). \\ \end{array}
$$
It is easy to verify that the above also holds when  $u_0 = x_0 = y_0$ . Since  $\ell \theta^2 = \mu / 4$ , we have that
$$
\begin{array}{l} \Delta_ {t + 1} - (1 - \theta) \Delta_ {t} \\ \leq \frac {\mu}{4} \| \theta y _ {t} + (1 - \theta) u _ {t} - x ^ {*} \| ^ {2} - \frac {\mu}{4} \| u _ {t + 1} - x ^ {*} \| ^ {2} - \frac {\mu}{4} \theta \| x ^ {*} - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ = \frac {\mu}{4} \| (1 - \theta) \left(u _ {t} - x ^ {*}\right) + \theta \left(y _ {t} - x ^ {*}\right) \| ^ {2} - \frac {\mu}{4} \| u _ {t + 1} - x ^ {*} \| ^ {2} - \frac {\mu}{4} \theta \| x ^ {*} - y _ {t} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ \leq \frac {\mu}{4} (1 - \theta) \| u _ {t} - x ^ {*} \| ^ {2} - \frac {\mu}{4} \| u _ {t + 1} - x ^ {*} \| ^ {2} + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}, \\ \end{array}
$$
where we use fact  $(iv)$  in Lemma A.1. Rearranging terms, we then obtain
$$
\Delta_ {t + 1} + \frac {\mu}{4} \| u _ {t + 1} - x ^ {*} \| ^ {2} \leq (1 - \theta) \left(\Delta_ {t} + \frac {\mu}{4} \| u _ {t} - x ^ {*} \| ^ {2}\right) + \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2}.
$$
Unrolling the recursion, we have that
$$
\begin{array}{l} f (x _ {T}) - f (x ^ {*}) \leq \Delta_ {T} + \frac {\mu}{4} \| u _ {T} - x ^ {*} \| ^ {2} \\ \leq (1 - \theta) ^ {T} \left(\Delta_ {0} + \frac {\mu}{4} \| u _ {0} - x ^ {*} \| ^ {2}\right) + ((1 - \theta) ^ {T - 1} + \dots + (1 - \theta) + 1) \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ \leq \exp (- \theta T) \left(f (x _ {0}) - f (x ^ {*}) + \frac {\mu}{4} \| u _ {0} - x ^ {*} \| ^ {2}\right) + \frac {1}{\theta} \left(\frac {1}{2 \ell} + \frac {1}{\mu}\right) \delta^ {2} \\ = \exp \left(- \frac {T}{2 \sqrt {\kappa}}\right) \left(f (x _ {0}) - f (x ^ {*}) + \frac {\mu}{4} \| x _ {0} - x ^ {*} \| ^ {2}\right) + \sqrt {\kappa} \left(\frac {1}{\ell} + \frac {2}{\mu}\right) \delta^ {2}, \\ \end{array}
$$
where we use the fact that  $1 + \eta \leq e^{\eta},\forall \eta \in \mathbb{R}$
Lemma 3.4 immediately follows from Lemma A.4. With the above results at hand, we are ready to show proof of Theorem 3.5 below.