CIFAR-10 Classification
We consider training a ResNet-18 classifier [ 19 ] on the CIFAR-10 data set [ 22 ] .
We follow the setup used in [ 13 ] , where a ResNet-18 is trained using batch size of 50K (the entire training dataset). As in [ 13 ] , we consider a standard ResNet-18 with the parameters in the linear layer randomly initialized and the parameters in the batch normalization initialized with mean zero and unit variance, except for the last in each residual branch which is initialized to zero. The default random CIFAR-10 data ordering is kept as is, and every image is randomly augmented by horizontal flips and random crops after padding by 4 pixels.

Using the setup of [ 13 ] , the reference mini-batch SGD is trained using a batch size of 128 (sampling without replacement), Nesterov momentum of 0.9 and weight decay of 0.0005. The learning rate is warmed up from 0.0 to 0.1 over the first 5 epochs and then reduced via cosine annealing to 0 over the course of training for 300 epochs (resulting in 390x300=117,000 update steps). The validation accuracy obtained in [ 13 ] is 95.7%. For the full-batch GDs, we replace the mini-batch updates by full batches and accumulate the gradients over all mini-batches.
For the Gaussian scheme and MPGD, we inject the perturbations into the GD recursion for parameters of the input layer of ResNet-18, choosing \mu=0.03 , \sigma=0.01 and \beta=0.5 (for MPGD). In our experiments, we fix the learning rate to 0.1 throughout training, and do not add momentum and weight decay. This is in constrast to [ 13 ] , where the additional tricks applied in the mini-batch SGD are also used for the full-batch GD, leading to higher test accuracies (see Table 2 in [ 13 ] ).

Table 2 shows that MPGD can lead to better test performance and lower accuracy gap when compared to the baseline (full batch GD) and Gaussian noise-perturbed GD. Using the perturbations in ( 11 ) instead can also achieve lower accuracy gap. Therefore, adding perturbations of MPGD can be potentially a useful trick to improve training of deep architectures on benchmark data sets. Lastly, we note that additional tricks, such as gradient clipping and tuning of optimization hyperparameters, can be applied to improve our test accuracies and close the gap to results obtained with the reference mini-batch SGD. However, our main focus here is not on competitive performance, but rather on demonstrating the effects of the perturbations in MPGD when compared to full batch GDs and Gaussian noise-perturbed full-batch GD to support our theory.

Lastly, we remark that we did not tune the step-size for any of the algorithms. Since our generalization bound applies to the asymptotic SDE, which is obtained when the step-size goes to zero, in order to stay close to the theory, we chose a small enough step-size to be both not far from the continuous dynamics, and large-enough so that the algorithm converges in a reasonable amount of time. That being said, we have tried a range of step-sizes for both algorithms, and we observed that the proposed scheme consistently outperforms vanilla GD for smaller step-sizes. Whereas if we use a large step-size, both algorithms perform similarly. In this regime, vanilla GD as well potentially emits a chaotic behavior, which might also indicate the importance of the "implicit randomness". However, as we indicated in the introduction, this regime is not easily controllable, and our purpose is to introduce a controlled chaotic component, with a clear theoretical understanding.