Introduction
Many important problems in supervised learning can be expressed by the following population risk minimization problem:

\min_{x\in\mathbb{R}^{d}}\Bigl{\{}\mathcal{R}(x):=\mathbb{E}_{z\sim\mathcal{D}}[\ell(x,z)]\Bigr{\}}, (1)

where x\in\mathbb{R}^{d} denotes the learnable parameter, z\in\mathcal{Z} denotes a data sample coming from an unknown data distribution \mathcal{D} , \mathcal{Z} denotes the space of data points, and \ell:\mathbb{R}^{d}\times\mathcal{Z}\to\mathbb{R}_{+} is the loss function, measuring the quality of the parameter x .
Since the data distribution \mathcal{D} is not known, as a proxy for solving ( 1 ), we consider the empirical risk minimization (ERM) problem, given as follows:

\min_{x\in\mathbb{R}^{d}}\Bigl{\{}\hat{\mathcal{R}}(x,S_{n}):=\frac{1}{n}\sum_{i=1}^{n}\ell(x,z_{i})\Bigr{\}}, (2)

where S_{n}:=\{z_{1},\dots,z_{n}\} denotes a training set of n points that are independently and identically distributed (i.i.d.) and sampled from \mathcal{D} . To solve the ERM problem, typically gradient-based optimization algorithms are being used in practice.

In this study, we will consider the gradient descent (GD) algorithm [ 4 ] , which is based on the following deterministic recursion:

 x_{k+1}=x_{k}-\eta_{k} abla\hat{\mathcal{R}}(x_{k},S_{n}),\ \ \ k=0,1,2,\dots

Here, k represents the iteration number, \eta_{k}>0 is the step-size (learning rate), and x_{0}\in\mathbb{R}^{d} is the initialization. Even though stochastic counterparts of GD (e.g., stochastic GD [ 3 ] ) have been more popular due to their reduced computational requirements, the GD algorithm has attracted increasing attention in the past few years, as it has been illustrated that GD can achieve similar generalization performance, as measured by the generalization error |\hat{\mathcal{R}}(x_{k},S_{n})-\mathcal{R}(x_{k})| , as long as certain design choices are implemented [ 13 ] . These design choices mainly include using large and/or oscillating learning rates and using explicit regularization of the empirical risk.

Contrary to the wisdom of optimization theory, which typically suggests using smaller learning rates, the fact that large learning rates often provide better generalization performance in modern non-convex optimization problems has opened up several research directions. In this context, [ 7 ] showed that large learning rates drive the GD dynamics towards the ‘edge of stability’, meaning that the learning rate should be chosen large enough to make the algorithm diverge. Recently, the large learning rates have been extended to exotic learning rate schedules (i.e., non-trivial evolution of \eta_{k} ), such as fractal scheduling [ 1 ] and cyclic scheduling [ 28 , 32 ] .

It has been shown that the use of large learning rates introduces a chaotic behavior in the GD dynamics [ 23 ] . Hence, a possible explanation for the improvements brought by large learning rates can be partially attributed to such chaotic behavior, since chaotic systems exhibit a stochastic-like evolutions (even though they are fully deterministic), which might be an underlying cause of the improvements in generalization obtained in non-convex settings. However, achieving the desired chaotic dynamics is a problem-dependent task and can be often difficult in practice.

In this paper, we develop a novel optimization framework, coined multiscale perturbed gradient descent (MPGD), where a chaotic component is introduced to GD in a controlled manner. Our approach is based on extending the GD recursion via generic regularization terms, where the regularization coefficients are modulated by using external deterministic dynamical systems that exhibit chaotic characteristics. Our main contributions are as follows.

We build up on recent advances in rough paths theory and homogenization techniques [ 15 , 5 , 6 ] , and show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process (Theorem 4.1 ; see also Theorem C.1 in Appendix).

We draw connections to the recent theoretical links between heavy-tailed processes and generalization [ 34 , 33 , 18 ] , and under certain topological regularity assumptions, we derive a generalization bound (Theorem 4.2 ) for the limiting SDE and relate the worst-case generalization error over its trajectories to the parameters of MPGD.

We show that MPGD exhibits a natural form of implicit regularization by deriving an appropriate explicit regularizer in the weak perturbation regime (Theorem 4.3 ). Our analysis shows that, in this regime, the derived regularizer introduces terms that penalize the Hessian of the loss function.

We empirically demonstrate the advantages of MPGD on different settings and provide further support for the developed framework and the theory (see Section 5 ).

We emphasize that our primary goal is to understand the behavior of deterministic GD in a chaotic setting. In this respect, we shall underline that, at this stage, our goal is not to develop a competitive algorithm that would outperform SGD, but rather to provide a solid theoretical analysis for how stochastic behavior can emerge from GD with suitable deterministic components, as well as their implication in terms of generalization error.