SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that our choice of the step size is maximal and is independent of the noise. In the deterministic setting, letting  $\sigma_{\mathrm{Bil}} \asymp \varepsilon \sqrt[4]{\lambda_{\min}(\mathbf{BB}^{\top})\lambda_{\max}(\mathbf{B}^{\top}\mathbf{B})}$ , the complexity bound in Corollary 3.3 reduces to  $\mathcal{O}\left(\sqrt{\frac{\lambda_{\max}(\mathbf{B}^{\top}\mathbf{B})}{\lambda_{\min}(\mathbf{BB}^{\top})}}\log\left(\frac{1}{\varepsilon}\right)\right)$ , which matches the lower bound in Ibrahim et al. [2020]. Notably, Azizian et al. [2020b] proposed an algorithm achieving an upper bound that matches the lower bound in Ibrahim et al. [2020], Li et al. [2022a] also proposed a lower-bound matching SEG algorithm that uses a shared sample in both steps under an unbounded noise assumption. In contrast, our algorithm is in the independent-sample setting with bounded noise variance.
Remark 3.4 Standard acceleration techniques do not attain the optimal nonasymptotic convergence rate for bilinear games [Gidel et al. 2019b]. This limitation applies to various algorithms, including the direct approach [§2.3], as well as several other acceleration techniques [Thekumparamil et al. 2022, Kovalev et al. 2022, Jin et al. 2022], all of which fall short of achieving optimal acceleration for bilinear games. Therefore, matching both lower bounds in a single algorithm in the general stochastic setting has been an open problem. While [Li et al. 2022b] present an algorithm that achieves both lower bounds in a single algorithm, it relies on the use of optimistic gradients rather than extragradients on the bilinear coupling function. Furthermore, our algorithm and analysis is more general than those in [Li et al. 2022b] as we can handle the general variational inequality with proximal operators.
We have presented a stochastic extragradient-based acceleration algorithm, AG-EG, for solving stochastic monotone variational inequalities with separable structure. The iteration complexity of our algorithm matches the lower bound and is independent of the size of the feasible set. When specialized to solving the bilinearly coupled saddle-point problem (3), our AG-EG algorithm simultaneously matches lower bounds due to Zhang et al. [2022] and Ibrahim et al. [2020] for strongly-convex-strongly-concave and bilinear games, respectively. To the best of our knowledge, this is the first time that all three lower bounds have been met by a single algorithm. There are some remaining issues to be addressed, however, including the case of one-sided nonstrong convexity, the setting of unbounded noise variance, and the characterization of the full parameter regime dependency on  $\lambda_{\mathrm{min}}(\mathbf{BB}^{\top})$ . These are left as important directions for future research.
This work is supported in part by Canada CIFAR AI Chair to GG, by the Mathematical Data Science program of the Office of Naval Research under grant number N00014-18-1-2764 and also the Vannevar Bush Faculty Fellowship program under grant number N00014-21-1-2941 and National Science Foundation (NSF) grant IIS-1901252 to MIJ. This work is also supported in part by the NSF CAREER Award 1906169 to QG and by NSF Award's IIS-2110170 and DMS-2134106 to SSD.
Othmane Sebbouh, Marco Cuturi, and Gabriel Peyre. Randomized stochastic gradient descent ascent. In International Conference on Artificial Intelligence and Statistics, pages 2941-2969. PMLR, 2022.
Kiran K Thekumparamgil, Niao He, and Sewoong Oh. Lifted primal-dual method for bilinearly coupled smooth minimax optimization. In International Conference on Artificial Intelligence and Statistics, pages 4281-4308. PMLR, 2022.
Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252, 1995.
Paul Tseng. On accelerated proximal gradient methods for convex-concave optimization. submitted to SIAM Journal on Optimization, 2(3), 2008.
Jialei Wang and Lin Xiao. Exploiting strong convexity from data with primal-dual first-order algorithms. In International Conference on Machine Learning, pages 3694-3702, 2017.
Yuanhao Wang and Jian Li. Improved algorithms for convex-concave minimax optimization. Advances in Neural Information Processing Systems, 33:4800-4810, 2020.
Lin Xiao, Adams Wei Yu, Qihang Lin, and Weizhu Chen. Dscovr: Randomized primal-dual block coordinate algorithms for asynchronous distributed optimization. Journal of Machine Learning Research, 20(1):1634-1691, 2019.
Guangzeng Xie, Yuze Han, and Zhihua Zhang. Dippa: An improved method for bilinear saddle point problems. arXiv preprint arXiv:2103.08270, 2021.
Pan Xu, Tianhao Wang, and Quanquan Gu. Continuous and discrete-time accelerated stochastic mirror descent for strongly convex functions. In International Conference on Machine Learning, pages 5492-5501. PMLR, 2018.
Yan Yan, Yi Xu, Qihang Lin, Lijun Zhang, and Tianbao Yang. Stochastic primal-dual algorithms with faster convergence than  $o(1 / \sqrt{T})$  for problems without bilinear structure. arXiv preprint arXiv:1904.10112, 2019.
Yan Yan, Yi Xu, Qihang Lin, Wei Liu, and Tianbao Yang. Optimal epoch stochastic gradient descent ascent methods for min-max optimization. Advances in Neural Information Processing Systems, 33:5789-5800, 2020.
Junchi Yang, Antonio Orvieto, Aurelien Lucchi, and Niao He. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics, pages 5485-5517. PMLR, 2022.
Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194(1-2):901-935, 2022.
Yuchen Zhang and Lin Xiao. Stochastic primal-dual coordinate method for regularized empirical risk minimization. Journal of Machine Learning Research, 18:1-42, 2017.