Multiscale Perturbed Gradient Descent (MPGD)
In full generality, we start by considering the following family (parametrized by m>0 ) of deterministic fast-slow dynamical systems on \mathbb{R}^{d}\times Y , with Y a bounded metric space:

\left\{\begin{aligned} x^{(m)}_{k+1}&=x^{(m)}_{k}+\frac{1}{m}a_{m}(x^{(m)}_{k})+\sum_{i=1}^{q}\frac{1}{m^{1/\alpha_{i}}}b_{m}^{(i)}(x^{(m)}_{k})v_{i}(y_{k}^{(i)}),\ y_{k+1}^{(i)}&=T_{i}(y_{k}^{(i)}),\ \ i=1,2,\ldots,q,\end{aligned}\right

for k=0,1,2,\dots , where the \alpha_{i}\in(1,2) , a_{m}:\mathbb{R}^{d}\to\mathbb{R}^{d} , b_{m}^{(i)}:\mathbb{R}^{d}\to\mathbb{R}^{d\times r_{i}} , v_{i}:Y\to\mathbb{R}^{r_{i}} , and T_{i}:Y\to Y . The initial conditions x_{0}^{(m)},y_{0}^{(i)} are independent random variables. One important example of recursions of the form ( 7 ) is the GD defined in ( 3 ).
Other than GD, the above family of recursions ( 7 ) includes various variants of GD as special cases upon specifying suitable choices of the a_{m} , b_{m}^{(i)} , v_{i} , T_{i} , and the dimensions.

Of particular interest are new algorithms that can be studied within the above framework. To this end, we introduce multiscale perturbed gradient descent (MPGD), whose dynamics are described by the following recursion:

\left\{\begin{aligned} x^{(m)}_{k+1}&=x^{(m)}_{k}-\frac{1}{m} abla\hat{\mathcal{R}}(x^{(m)}_{k},S_{n})-\frac{\mu}{m^{\frac{1}{\alpha_{1}}}}v_{1}(y_{k}^{(1)})\odot x^{(m)}_{k}+\frac{\sigma}{m^{\frac{1}{\alpha_{2}}}}v_{2}(y_{k}^{(2)}),\ y^{(1)}_{k+1}&=T(y_{k}^{(1)}),\ \ y^{(2)}_{k+1}=T(y_{k}^{(2)}),\ \ k=0,1,2,\dots,\end{aligned}\right

where T is the Thaler map ( 5 ) with \gamma=\alpha^{-1}\in(1/2,1) , the v_{1},v_{2} are observable maps, \mu\geq 0 and \sigma\in\mathbb{R} are tunable parameters.
The above recursion is of the form ( 7 ), with q=2 , r_{1}=r_{2}=d , a_{m}(x)=-\nabla\hat{\mathcal{R}}(x,S_{n}) , b_{m}^{(1)}(x)=-\mu\ \rm{diag}(x) , and b_{m}^{(2)}(x)=\sigma I independent of m . Another class of algorithms that falls within this framework is when m is set to n , the number of training data points, in which case the empirical risk converges to the population risk as m=n\to\infty .

To complete the description of MPGD, it remains to specify the choice for the observables v_{i}(y_{k}^{(i)}) in ( 8 ) whose rescaled version can be shown to generate stable laws X_{\alpha,\beta} , with \alpha\in(1,2) and \beta\in[-1,1] , in the limit m\to\infty . Following [ 16 ] , we take them to be independent realizations of the observable v^{(k)} described as follows. Let d_{\alpha}=\frac{\alpha^{\alpha}(1-\gamma)\Gamma(1-\alpha)\cos(\alpha\pi/2)}{2^{1-\gamma}-1}, and \delta_{0},\delta_{1},\dots be independent copies of the random variable \delta where \mathbb{P}(\delta=\pm 1)=\frac{1}{2}(1\pm\beta) .
The observables v^{(k)} are then defined to be: v^{(k)}=\chi^{(k)}v\circ T^{k} , where v:[0,1]\to\mathbb{R} is the mean zero observable given by v(y)=d_{\alpha}^{-\gamma}(1-2^{\gamma-1})^{-\gamma}\tilde{v}(y) , with

\tilde{v}(y)=\begin{cases}1&if $y\leq y^{*$},\ (1-2^{1-\gamma})^{-1}&if $y>y^{*$},\end{cases}\hskip 8.5359pt\chi^{(k)}=\chi_{k-1}\cdots\chi_{0}\in\{\pm 1\}\hskip 2.84544pt with \ \chi_{j}=\begin{cases}1&if $T^{jy\leq y^{*}$},\ \delta_{j}&if $T^{jy>y^{*}$}.\end{cases}

In particular, the random variables \chi^{(k)} get updated only when the trajectory visits (y^{*},1] and are not changed during the phase in [0,y^{*}] .

The initial condition can be, in theory, equally well chosen using the invariant probability measure \nu or the uniform Lebesgue measure. Empirically, convergence of the probability density is faster if it is drawn using \nu . Hence, we consider initial conditions drawn using \nu . Therefore, for MPGD we propagate uniformly distributed initial conditions y_{0}^{\prime}\in[0,1] under 10,000 iterations of the Thaler map and take the initial condition as y_{0}=T^{10000}y_{0}^{\prime} .
Figure 1 illustrates the dynamics of a realization of the Thaler iterates and the observable (perturbations used in MPGD), as well as the asymmetry and heavy tail nature of the distribution of the corresponding Birkhoff sum.