Generalization Properties of MPGD
In this section, we shift our focus to the generalization properties of MPGD. Our main roadmap will be to consider the limiting SDE arising from Theorem 4.1 (with \beta=0 ) and relate it to the recently developed generalization bounds for heavy-tailed random processes [ 33 , 18 ] .

For mathematical convenience, we consider the following special case of ( 7 ) 1 1 1 We believe a similar result would hold for ( 9 ) without symmetrization, which we leave for future work. :

\left\{\begin{aligned} x^{(m)}_{k+1}&=x^{(m)}_{k}+m^{-1}a_{m}(x^{(m)}_{k})+m^{-\frac{1}{\alpha}}b_{m}(x^{(m)}_{k})\left(v(y^{(1)}_{k})-v(y^{(2)}_{k})\right),\ y^{(1)}_{k+1}&=T(y^{(1)}_{k}),\quad y^{(2)}_{k+1}=T(y^{(2)}_{k})\end{aligned}\right

In the light of Theorem 4.1 , the recursion ( 11 ) converges weakly to the following SDE:

\mathrm{d}X_{t}=a(X_{t})\mathrm{d}t+b(X_{t})\diamond\mathrm{d}\mathrm{L}_{t}^{\alpha}, (12)

where \mathrm{L}^{\alpha}_{t} denotes the symmetric \alpha -stable process in \mathbb{R}^{d} (i.e., \beta=0 ).

Following the previous work [ 33 , 18 ] , we are interested in bounding the worst-case generalization error over the trajectories of ( 12 ). More precisely, let (X_{t})_{0\leq t\leq 1} denote the solution of ( 12 ) 2 2 2 Here, the range [0,1] is arbitrary and could be replaced with any range [0,T] for 0<T<\infty . , and define the set \mathcal{X} so that it contains all the points visited by the trajectory:

\mathcal{X}:=\{x\in\mathbb{R}^{d}:\exists t\in[0,1],X_{t}=x\}

Then, our goal is to analyze \sup_{x\in\mathcal{X}}|\hat{\mathcal{R}}(x,S_{n})-\mathcal{R}(x)| . Since we are mainly interested in the case when a(x)=-\nabla\hat{\mathcal{R}}(x,S_{n}) , the SDE ( 12 ) and hence the trajectory \mathcal{X} will depend on the dataset S_{n} . Therefore, in the generalization bound, we will need to control the statistical dependence of \mathcal{X} on S_{n} , through the notion of \rho -mutual information, defined as follows. Let X,Y be two random elements, let \mathbb{P}_{X,Y} denote their joint distribution, and let \mathbb{P}_{X},\mathbb{P}_{Y} denote their respective marginal distributions. Then the \rho -mutual information between X and Y is defined as: I_{\rho}(X,Y)=D_{\rho}(\mathbb{P}_{X,Y}\|\mathbb{P}_{X}\otimes\mathbb{P}_{Y}) , where D_{\rho} is the \rho -Renyi divergence: 3 3 3 As \rho\to 1 , D_{\rho} tends to the Kullback-Leibler divergence. D_{\rho}(\mu,\nu)=\frac{1}{\rho-1}\log\mathbb{E}\bigg{[}\frac{\mathrm{d}\mu}{\mathrm{d}\lambda}(Z)^{\rho}\frac{\mathrm{d}\nu}{\mathrm{d}\lambda}(Z)^{1-\rho}\bigg{]} ,
and \frac{\mathrm{d}\mu}{\mathrm{d}\lambda} denotes the Radon-Nikodym derivative.

Finally, we require two technical regularity assumptions (Assumptions 2 and 3 , given in Appendix D ) for our generalization bound. Informally, Assumption 2 is a topological regularity condition over the trajectory \mathcal{X} , whereas Assumption 3 is a statistical regularity assumption and imposes that in the neighborhood of a local minimum x^{\star} of \hat{\mathcal{R}}(x,S_{n}) , the statistical behavior of the process X_{t} can be approximated by the process solving the SDE \mathrm{d}\tilde{X}_{t}=b(x^{\star})\diamond\mathrm{d}\mathrm{L}_{t}^{\alpha} . This requires b to be regular in the neighborhood of x^{\star} and further imposes that a(x^{\star})=0 , which is natural when a(x)=-\nabla\hat{\mathcal{R}}(x,S_{n}) .
We now ready to present our generalization bound.

Assume that Assumptions 2 and 3 given in Section D hold, and that \ell is bounded by B>0 and L -Lipschitz continuous. There exists a constant K_{1}>0 such that with probability at least 1-\delta ,

\sup olimits_{x\in\mathcal{X}}|\hat{\mathcal{R}}(x,S_{n})-\mathcal{R}(x)|\leq K_{1}\max\{B,L\}\left(\sqrt{\frac{\alpha}{n}}+\sqrt{\frac{\log(1/\delta)+I_{\infty}(S_{n},\mathcal{X})}{n}}\right), (14)

where I_{\infty}(X,Y):=\lim_{\rho\to\infty}I_{\rho}(X,Y) . Furthermore, there exists K_{2}>0 such that

\mathbb{E}\sup olimits_{x\in\mathcal{X}}|\hat{\mathcal{R}}(x,S_{n})-\mathcal{R}(x)|\leq K_{2}\max\{B,L\}\left(\sqrt{\frac{\alpha}{n}}+\sqrt{\frac{I_{1}(S_{n},\mathcal{X})}{n}}\right)

This result shows that the generalization error is mainly bounded by two terms: the tail exponent of the limiting Lévy process \mathrm{L}^{\alpha}_{t} , and the statistical dependence of \mathcal{X} on S_{n} . The bound suggests that, provided the mutual information between \mathcal{X} and S_{n} is fixed, the generalization error can be reduced by using a smaller \alpha in MPGD. However, it has been illustrated that a smaller \alpha can induce significant error on \hat{\mathcal{R}}(x,S_{n}) [ 35 , 9 ] , hence a reasonable value of \alpha should be chosen to balance both the empirical risk and the generalization error. The algorithm parameters \mu,\sigma interact with the bound via the constant terms K_{1} , K_{2} and the mutual information terms I_{1} , I_{\infty} . Unfortunately, their dependence is quite implicit and we are not able to provide quantitative estimations.

Finally, the main difference between Theorem 4.2 and [ 33 , 18 ] is that, in our bounds we have explicit access to the characteristic exponent of \mathrm{L}_{t}^{\alpha} since we construct it manually through our dynamics; whereas in the prior work generic processes were used as approximations for SGD, hence their results are not as explicit.