SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\max  _ {y \in \mathcal {Y}} F (x _ {r}, y) - \min  _ {x \in \mathcal {X}} F (x, y _ {r}) \leq \left(4 (\sqrt {2} + 7) \frac {\ell D ^ {2}}{\epsilon} + 3 \sqrt {2}\right) \delta D + \epsilon .
$$
The reproducibility is  $\| z_r - z_r' \|^2 \leq 36(9 + 4\sqrt{2})D^4(\delta^2/\epsilon^2)$ .
C.3 Inexact Proximal Point Method
Proof of Theorem 4.7 and 4.8 for the guarantees of Algorithm 3 is provided in this section.
C.3.1 General Analysis
We first analyze the convergence of the inexact proximal point method (Inexact-PPM). Given initialization  $(x_0, y_0)$  and  $\alpha > 0$ , for  $t = 0, 1, \dots, T - 1$ , each step of Inexact-PPM is
$$
(x_{t + 1},y_{t + 1})\text{is an inexact solution to}\min_{x\in \mathcal{X}}\max_{y\in \mathcal{Y}}\hat{F}_{t}(x,y) = F(x,y) + \frac{1}{2\alpha}\| x - x_{t}\|^{2} - \frac{1}{2\alpha}\| y - y_{t}\|^{2}.
$$
Lemma C.6. If we run Inexact-PPM and make sure that for each sub-problem  $\tilde{\nabla}\hat{F}_t(z_{t + 1})^\top (z_{t + 1} - z)$ $\leq \hat{\epsilon}$  for all  $z = (x,y)\in \mathcal{X}\times \mathcal{Y},$  where  $z_{t + 1} = (x_{t + 1},y_{t + 1})$  and  $\tilde{\nabla}\hat{F}_t(z_{t + 1}) = (\nabla_x\hat{F}_t(x_{t + 1},y_{t + 1}), - \nabla_y\hat{F}_t(x_{t + 1},y_{t + 1}))$ , then we have  $\forall z\in \mathcal{X}\times \mathcal{Y}$
$$
\max _ {y \in \mathcal {Y}} F (\bar {x} _ {T + 1}, y) - \min _ {x \in \mathcal {X}} F (x, \bar {y} _ {T + 1}) \leq \frac {\| z _ {0} - z \| ^ {2}}{2 \alpha T} + \hat {\epsilon}.
$$
Proof. The proof is similar to Proposition 7 in Mokhtari et al. [58]. The same as (25), for any  $z = (x,y) \in \mathcal{X} \times \mathcal{Y}$  and any  $t = 0,1,\dots ,T - 1$ , we have that
$$
\begin{array}{l} \tilde {\nabla} F (z _ {t + 1}) ^ {T} (z _ {t + 1} - z) = \left(\tilde {\nabla} \hat {F} _ {t} (z _ {t + 1}) - \frac {1}{\alpha} (z _ {t + 1} - z _ {t})\right) ^ {\top} (z _ {t + 1} - z) \\ = \frac {1}{2 \alpha} \| z _ {t} - z \| ^ {2} - \frac {1}{2 \alpha} \| z _ {t + 1} - z \| ^ {2} - \frac {1}{2 \alpha} \| z _ {t + 1} - z _ {t} \| ^ {2} + \tilde {\nabla} \hat {F} _ {t} (z _ {t + 1}) ^ {\top} (z _ {t + 1} - z) \\ \leq \frac {1}{2 \alpha} \| z _ {t} - z \| ^ {2} - \frac {1}{2 \alpha} \| z _ {t + 1} - z \| ^ {2} + \hat {\epsilon}. \\ \end{array}
$$
Taking summation from  $t = 0$  to  $T - 1$  and dividing both sides by  $T$ , we conclude that
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \tilde {\nabla} F (z _ {t + 1}) ^ {\top} (z _ {t + 1} - z) \leq \frac {\| z _ {0} - z \| ^ {2}}{2 \alpha T} + \hat {\epsilon}.
$$
The proof is completed by Lemma B.2.
C.3.2 Inexact Initialization Oracle
This section provides proof of Theorem 4.7.
Proof of Theorem 4.7. Let  $\bar{z}_{T + 1} = (\bar{x}_{T + 1},\bar{y}_{T + 1}) = (1 / T)\sum_{t = 0}^{T - 1}(x_{t + 1},y_{t + 1})$ . By Lemma C.6 and the choice that  $\alpha = 1 / \ell, \hat{\epsilon} = \delta^2 /(2\alpha T^2)$ , we immediately have
$$
\max  _ {y \in \mathcal {Y}} F (\bar {x} _ {T + 1}, y) - \min  _ {x \in \mathcal {X}} F (x, \bar {y} _ {T + 1}) \leq \frac {\ell D ^ {2}}{T} + \frac {\ell \delta^ {2}}{2 T ^ {2}}.
$$
$\mathcal{O}(1 / T)$  convergence rate is guaranteed for  $\delta \leq \mathcal{O}(\sqrt{T})$ . Note that the condition number of  $\hat{F}_t(x,y)$  is  $\mathcal{O}(1)$  when  $\alpha = 1 / \ell$ . Therefore, to guarantee an  $\epsilon$ -saddle point of  $F(x,y)$ , a total complexity of  $\mathcal{O}(T\log (1 / \hat{\epsilon})) = \mathcal{O}((1 / \epsilon)\log (1 / (\epsilon \delta)))$  is sufficient for various algorithms including GDA [32] and EG [72] applying Lemma C.5 with  $\delta = 0$ .
Let  $z_{t}^{*} = (x_{t}^{*},y_{t}^{*})$  be the unique saddle point of  $\hat{F}_t(x,y)$  with proximal center  $z_{t}$ , and  $(z_t^*)'$  be the saddle point when the proximal center is  $z_{t}'$ . For the reproducibility guarantee, similarly to Section C.2.1, we can obtain that
$$
\begin{array}{l} \left\| z _ {t + 1} - z _ {t + 1} ^ {\prime} \right\| \leq \left\| z _ {t + 1} - z _ {t} ^ {*} \right\| + \left\| z _ {t} ^ {*} - \left(z _ {t} ^ {*}\right) ^ {\prime} \right\| + \left\| \left(z _ {t} ^ {*}\right) ^ {\prime} - z _ {t + 1} ^ {\prime} \right\| \\ \leq \left\| z _ {t} - z _ {t} ^ {\prime} \right\| + 2 \sqrt {\frac {2 \hat {\epsilon}}{\ell}}, \tag {26} \\ \end{array}
$$
where we use Lemma B.4 in Zhang et al. [78] and  $(1 / \alpha)$ -SC-SC of  $\hat{F}_t(x,y)$ :
$$
\begin{array}{l} \tilde {\nabla} \hat {F} _ {t} \left(z _ {t + 1}\right) ^ {\top} \left(z _ {t + 1} - z _ {t} ^ {*}\right) \geq \hat {F} _ {t} \left(x _ {t + 1}, y _ {t} ^ {*}\right) - \hat {F} _ {t} \left(x _ {t} ^ {*}, y _ {t + 1}\right) + \frac {1}{2 \alpha} \| z _ {t + 1} - z _ {t} ^ {*} \| ^ {2} \\ \geq \frac {\ell}{2} \| z _ {t + 1} - z _ {t} ^ {*} \| ^ {2}. \\ \end{array}
$$
Therefore, by induction, we have that for any  $t = 1,2,\dots ,T$
$$
\begin{array}{l} \left\| z _ {t} - z _ {t} ^ {\prime} \right\| \leq \left\| z _ {0} - z _ {0} ^ {\prime} \right\| + 2 t \sqrt {\frac {2 \hat {\epsilon}}{\ell}} \\ \leq \delta + 2 \delta \frac {t}{T} \\ \leq 3 \delta . \\ \end{array}
$$
The reproducibility is then  $\| \bar{z}_{T + 1} - \bar{z}_{T + 1}' \|^2 \leq 9\delta^2$  using Jensen's inequality.
C.3.3 Inexact Deterministic Gradient Oracle
For Theorem 4.8, we provide proof when using GDA as the base algorithm. According to Lemma C.4, EG can also be applied here with a similar argument.
Proof of Theorem 4.8. When setting  $\alpha = 1 / \ell$ , the auxiliary problem is  $\ell$ -strongly-convex-strongly-concave and  $2\ell$ -smooth. Let  $z_{t}^{K}$  be the output of  $K$ -step GDA with initialization  $z_{t}^{0}$  on the minimax problem  $\min_{x\in \mathcal{X}}\max_{y\in \mathcal{Y}}\hat{F}_{t}(x,y)$  at iteration  $t$ . Denote its saddle point as  $z_{t}^{*}$ . By Lemma C.3, if  $K\geq 32\log (8\ell^{2}D^{2} / (3\delta^{2}))$ , we have that
$$
\begin{array}{l} \left\| z _ {t} ^ {K} - z _ {t} ^ {*} \right\| ^ {2} \leq \exp \left(- \frac {K}{3 2}\right) \left\| z _ {t} ^ {0} - z _ {t} ^ {*} \right\| ^ {2} + \frac {9 \delta^ {2}}{4 \ell^ {2}} \\ \leq \frac {3 \delta^ {2}}{\ell^ {2}}. \\ \end{array}
$$
By Lemma C.5, we can thus set  $z_{t + 1} = [z_t^K ]_{2\ell}$  and guarantee that
$$
\tilde {\nabla} \hat {F} _ {t} (z _ {t + 1}) ^ {\top} (z _ {t + 1} - z) \leq (4 \sqrt {6} + 5 \sqrt {2} + 4) \delta D, \quad \forall z \in \mathcal {X} \times \mathcal {Y}.
$$
According to Lemma C.6, we then have
$$
\max  _ {y \in \mathcal {Y}} F (\bar {x} _ {T + 1}, y) - \min  _ {x \in \mathcal {X}} F (x, \bar {y} _ {T + 1}) \leq \frac {\ell D ^ {2}}{T} + 2 1 \delta D.
$$
When  $\delta \leq \epsilon /(42D)$ ,  $T\geq 2\ell D^{2} / \epsilon$  is required to obtain an  $\epsilon$ -saddle point, and the total gradient complexity is  $TK = (64\ell D^2 /\epsilon)\log (8\ell^2 D^2 /(3\delta^2)) = \tilde{\mathcal{O}} (1 / \epsilon)$  with  $\tilde{\mathcal{O}}$  hiding logarithmic terms.
We then show the reproducibility guarantee. From Lemma C.5, we know that  $\| z_{t + 1} - z_t^*\| \leq (1 + \sqrt{2} /2)\| z_t^K -z_t^*\| +\sqrt{2}\delta /\ell \leq 4.5\delta /\ell$ . By (26), we have that
$$
\left\| z _ {t + 1} - z _ {t + 1} ^ {\prime} \right\| \leq \left\| z _ {t} - z _ {t} ^ {\prime} \right\| + \frac {9 \delta}{\ell}.
$$
By induction, we conclude that  $\| z_t - z_t'\| \leq 9t(\delta /\ell)$ , and thus the reproducibility is  $\| \bar{z}_{T + 1} - \bar{z}_{T + 1}'\| ^2\leq$ $81\delta^{2}T^{2} / \ell^{2} = 324D^{4}(\delta^{2} / \epsilon^{2})$
Some numerical experiments that demonstrate the effectiveness of regularization to improve reproducibility are provided in this section. We test the algorithms on two problems: a minimization problem with a quadratic objective and a minimax problem with a bilinear objective. The experiments are conducted on a single local machine.
Minimization. We first compare the performance of gradient descent (GD), accelerated gradient descent (AGD), Algorithm 1 with GD as the base algorithm (Reg-GD), and Algorithm 1 with AGD as the base algorithm (Reg-AGD) on a quadratic minimization problem
$$
\min  _ {x \in \mathbb {R} ^ {d}} \frac {1}{2} \| A x - b \| ^ {2}.
$$
Here,  $b \in \mathbb{R}^d$  with each entry sampled from the Gaussian distribution with mean 0 and standard deviation 10 and  $A \in \mathbb{R}^{d \times d}$  is a random positive semi-definite matrix with rank  $d - 1$  that makes
Figure 1: Comparisons among GD, AGD, and their regularized version on the quadratic minimization problem with  $\delta$ -inexact gradients. The left figure plots the convergence behavior and the right shows the reproducibility. Both axes are plotted utilizing a logarithmic scale.
sure the problem is convex but not strongly-convex. To be specific, we let  $A = U\Sigma U^{\top}$  where  $U$  is a random orthogonal matrix drawn from the Haar distribution, and  $\Sigma$  is a diagonal matrix with 1 entry being 0 and the others uniformly sampled from [0.1, 10]. This ensures that the problem is smooth with a parameter smaller than 100.
We implement an inexact gradient oracle that returns  $A^\top (Ax - b) + \delta e$  where  $e \in \mathbb{R}^d$  is an all-one vector and  $\delta \in \mathbb{R}$  controls the inexactness level. We test the aforementioned four algorithms with this inexact gradient oracle on both convergence performance measured by function value and reproducibility performance measured by the deviation compared to the trajectory obtained from using the true gradient when  $\delta = 0$ . In the experiments, we let  $d = 100$  and  $\delta = 0.1$ . For all four algorithms, we set the number of iterations to be  $T = 10000$ , and the stepsize to be 0.01 based on the fact that the smoothness parameter is at most 100. For the regularization-based methods, we set the regularization parameter of the auxiliary problem to 0.05. All other parameters are set according to the theoretically suggested values. The results are illustrated in Figure 1.
In Figure 1, we see AGD converges faster than GD, but the deviation in iterates is much larger. When introducing regularization, i.e., Reg-AGD, the reproducibility guarantee is greatly improved with only a small degradation in the convergence performance. It is worth mentioning that Reg-GD also has a smaller deviation bound compared to GD. All the results align with our theoretical analysis. Changing the inexactness level  $\delta$  or the random seed for sampling the matrix  $A$  and the vector  $b$  does not influence the phenomenon too much, so we do not report the results with different selections.
Minimax. We also test the performance of gradient descent ascent (GDA), Extragradient (EG), and their regularized counterparts (Reg-GDA and Reg-EG) in Algorithm 2 on a bilinear matrix game
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} x ^ {\top} A y.
$$
Here,  $A \in \mathbb{R}^{d \times d}$  is generated the same as in the quadratic minimization example,  $\mathcal{X} = \{x \in \mathbb{R}^d \mid \| x\| \leq D\}$  and  $\mathcal{Y} = \{y \in \mathbb{R}^d \mid \| y\| \leq D\}$  are  $d$ -dimensional balls with diameter  $2D$  measured by the Euclidean norm. The projection onto these balls can be easily achieved. We implement an inexact gradient oracle that returns  $Ay + \delta e$  and  $A^\top x + \delta e$  for the partial gradients w.r.t.  $x$  and  $y$  respectively, where  $e \in \mathbb{R}^d$  is an all-one vector and  $\delta \in \mathbb{R}$  controls the inexactness level.
We test the aforementioned four algorithms with this inexact gradient oracle on both convergence performance measured by the duality gap (computable due to bounded domain) and reproducibility performance measured by the deviation compared to the trajectory obtained from using the true
Figure 2: Comparisons among GDA, EG, and their regularized version on the bilinear matrix game with  $\delta$ -inexact gradients. The left figure plots the convergence behavior and the right shows the reproducibility. Both axes are plotted utilizing a logarithmic scale.
gradient when  $\delta = 0$ . In the experiments, we let  $d = 500$  and  $\delta = 0.1$ . For all four algorithms, we set the number of iterations to be  $T = 10000$ , and the stepsize is 0.1 for EG, 0.05 for Reg-EG, 0.001 for GDA, and 0.0001 for Reg-GDA. The choice of stepsizes here adheres to our theoretical analysis noticing that the smoothness parameter is no larger than 10. Larger stepsize for GDA will make the trajectory easily diverge. For the regularization-based methods, we set the regularization parameter of the auxiliary problem to 0.05. The results are plotted in Figure 2. We see again the effectiveness of regularization to improve the reproducibility of the algorithms. Reg-EG largely reduces the deviation of EG even with respect to magnitude (note the figure is logarithmically scaled), with only a small degradation in terms of the convergence speed.