Implicit Regularization of MPGD
To understand MPGD and its connection to the vanilla GD ( 3 ) better, we provide additional analysis for the behavior of loss functions optimized under the recursion ( 8 ) via the lens of implicit regularization [ 25 , 24 ] . By this, we mean regularization imposed implicitly by the learning strategy, without explicitly modifying the loss. We shall achieve this by deriving an appropriate explicit regularizer through a perturbation analysis in the weak perturbation regime.

To this end, we work in the regime where the parameters \mu,\sigma are small, keeping m fixed. We set \mu=\mu_{0}\epsilon and \sigma=\sigma_{0}\epsilon , where \epsilon>0 is a small parameter, and analyze the loss optimized using the recursion ( 8 ) in the small \epsilon regime. In the sequel, we let \overline{x}_{k}^{(m)} denote the states of the unperturbed GD, satisfying the recursion \overline{x}_{k+1}^{(m)}=\overline{x}_{k}^{(m)}-\frac{1}{m}\hat{\mathcal{R}}(\overline{x}_{k}^{(m)},S_{n}) , for k=0,1,\dots .
To simplify notation, we shall denote \hat{\mathcal{R}}(x):=\hat{\mathcal{R}}(x,S_{n}) .

The following result relates the loss function, averaged over realizations of the
injected perturbations, evolved under training with MPGD to that of unperturbed GD in the weak perturbation regime.

Let m>0 , k\in\mathbb{N} , and S_{n} be given. Assume that r_{1}=1 and r_{2}=d in Eq. ( 8 ), and \mathbb{E}[v_{1}(y^{(1)}_{k})]=\mathbb{E}[v_{2}(y^{(2)}_{k})]=0 for all k . Then, for a scalar-valued loss function \hat{\mathcal{R}} with \nabla\hat{\mathcal{R}} having Lipschitz continuous partial derivatives in each coordinate up to order three (inclusive),

\mathbb{E}[\hat{\mathcal{R}}(x_{k}^{(m)})] =\hat{\mathcal{R}}(\overline{x}_{k}^{(m)})+\frac{\epsilon^{2}}{2}\left(tr\left(C_{k}^{(m)} abla^{2}\hat{\mathcal{R}}(\overline{x}_{k}^{(m)})\right)- abla\hat{\mathcal{R}}(\overline{x}_{k}^{(m)})^{T}\lambda_{k}^{(m)}\right)+\mathcal{O}(\epsilon^{3}), (16)

as \epsilon\to 0 , where the expectation is with respect to the realization of the y^{(i)} , the \lambda_{k}^{(m)} are the vector with the l th component:

[\lambda_{k}^{(m)}]^{l}=\frac{1}{m}\sum_{i=1}^{k}\sum_{j=1}^{d}[\Phi_{i}^{(m)}]^{lj}tr\left(C_{i-1}^{(m)} abla^{2}[ abla\hat{\mathcal{R}}(\overline{x}_{i-1}^{(m)})]^{j}\right),

the \Phi_{i}^{(m)}:=\prod_{j=i}^{k-1}(I-\frac{1}{m}\nabla^{2}\hat{\mathcal{R}}(\overline{x}_{j}^{(m)})) , with the empty product taken to be the identity,
and the C_{k}^{(m)} are covariance matrices with the (p,q) -entry of \sum_{i_{1},i_{2}=1}^{k}\sum_{r,s=1}^{d}[\Phi_{i_{1}}^{(m)}]^{pr}[\Phi_{i_{2}}^{(m)}]^{qs}\theta_{i_{1},i_{2},r,s}^{(m)} , where

\theta_{i_{1},i_{2},r,s}^{(m)} =\frac{\mu_{0}^{2}}{m^{2/\alpha_{1}}}\mathbb{E}[v_{1}(y_{i_{1}-1}^{(1)})v_{1}(y_{i_{2}-1}^{(1)})]\cdot[x_{i_{1}-1}^{(m)}]^{r}[x_{i_{2}-1}^{(m)}]^{s}+\frac{\sigma_{0}^{2}}{m^{2/\alpha_{2}}}\mathbb{E}[[v_{2}]^{r}(y_{i_{1}-1}^{(2)})[v_{2}]^{s}(y_{i_{2}-1}^{(2)})].

Theorem 4.3 implies that the loss function optimized under MPGD at a given iteration and m is, on average, approximately equivalent to a regularized objective functional. Moreover, \frac{\nabla\hat{\mathcal{R}}(\overline{x}_{k}^{(m)})^{T}v_{k}^{(m)}}{tr\left(C_{k}^{(m)}\nabla^{2}\hat{\mathcal{R}}(\overline{x}_{k}^{(m)})\right)}\leq\frac{C}{m} for some constant C independent of m , suggesting that the trace term is the dominant explicit regularizer for large m . This explicit regularizer is solely determined by the discrete-time flow generated by the Jacobians I-\frac{1}{m}\nabla^{2}\hat{\mathcal{R}}(\overline{x}_{j}^{(m)}) , the covariances C_{k}^{(m)} , and the Hessian of the loss function, all evaluated along the dynamics of the unperturbed GD. We can therefore expect the use of the perturbations in MPGD as a regularization mechanism should reduce the Hessian of the loss function according to the perturbation levels ( \mu_{0},\sigma_{0} ) and the correlations \frac{1}{m^{2/\alpha_{1}}}\mathbb{E}[v_{1}(y_{i_{1}-1}^{(1)})v_{1}(y_{i_{2}-1}^{(1)})] , \frac{1}{m^{2/\alpha_{2}}}\mathbb{E}[v_{2}(y_{i_{1}-1}^{(2)})v_{2}(y_{i_{2}-1}^{(2)})] . Reducing the Hessian of the loss function can lead to flatter minima in the loss landscape, which is widely believed to associate with better generalization in deep learning [ 21 , 20 ] .

Lastly, we observe that in the weak perturbation regime and for non-convex losses, MPGD is more beneficial than GD in the sense that it reduces the Hessian of the loss, thereby promoting flatter minima. Hence, in order to observe the benefits of our scheme, we suspect that the optimization problem should be non-convex and local minima should have different curvatures.