Li, Y., Ju, C., Fang, E. X., and Zhao, T. Implicit regularization of bregman proximal point algorithm and mirror descent on separable data. arXiv preprint arXiv:2108.06808, 2021.
Lyu, K. and Li, J. Gradient descent maximizes the margin of homogeneous neural networks. In the 8th International Conference on Learning Representations, 2020.
Nacson, M. S., Lee, J., Gunasekar, S., Savarese, P. H. P., Srebro, N., and Soudry, D. Convergence of gradient descent on separable data. In Proceeding of the 22nd International Conference on Artificial Intelligence and Statistics, pp. 3420-3428, 2019.
Nesterov, Y. On an approach to the construction of optimal methods of minimization of smooth convex functions. *Ekonomika i Mateaticheskie Metody*, 24(3):509-517, 1988.
Neyshabur, B., Tomioka, R., and Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
Orabona, F. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019.
Orabona, F. and Pál, D. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems 29, 2016.
Rakhlin, S. and Sridharan, K. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems 26, pp. 3066-3074, 2013.
Ramdas, A. and Pena, J. Towards a deeper geometric, analytic and algorithmic understanding of margins. Optimization Methods and Software, 31(2):377-391, 2016.
Soheili, N. and Pena, J. A smooth perceptron algorithm. SIAM Journal on Optimization, 22(2): 728-737, 2012.
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-2878, 2018.
Sun, H., Ahn, K., Thrampoulidis, C., and Azizan, N. Mirror descent maximizes generalized margin and can be implemented efficiently. In Advances in Neural Information Processing Systems 35, 2022.
Telgarsky, M. Margins, shrinkage, and boosting. In International Conference on Machine Learning, pp. 307-315. PMLR, 2013.
Tseng, P. On accelerated proximal gradient methods for convex-concave optimization. submitted to SIAM Journal on Optimization, 2(3), 2008.
van Erven, T. and Koolen, W. M. MetaGrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29, pp. 3666-3674, 2016.
Vardi, G. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591, 2022.
Wang, B., Meng, Q., Zhang, H., Sun, R., Chen, W., Ma, Z.-M., and Liu, T.-Y. Does momentum change the implicit regularization on separable data? Advances in Neural Information Processing Systems 35, 2022a.
Wang, G., Lu, S., and Zhang, L. Adaptivity and optimality: A universal algorithm for online convex optimization. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, 2019.
Wang, G., Wan, Y., Yang, T., and Zhang, L. Online convex optimization with continuous switching constraint. Advances in Neural Information Processing Systems, 34:28636-28647, 2021a.
Wang, G., Hanashiro, R., Guha, E., and Abernethy, J. On accelerated perceptrons and beyond. arXiv preprint arXiv:2210.09371, 2022b.
Wang, J.-K. and Abernethy, J. D. Acceleration through optimistic no-regret dynamics. In Advances in Neural Information Processing Systems 32, pp. 3824-3834, 2018.