On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Authors: Vincent Szolnoky
Viktor Andersson
Balazs Kulcsar
Rebecka Jörnsten
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural network
regularization
gradient descent
gradient similarity
noisy labels
generalization

Abstract: Most complex machine learning and modelling techniques are prone to over-fitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having a level of implicit regularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce a new framework, Model Gradient Similarity (MGS), that (1) serves as a metric of regularisation, which can be used to monitor neural network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for a new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high levels of label noise or limited sample sizes.

On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Vincent Szolnoky
Department of Mathematical Sciences
Chalmers University of Technology
Göteborg, Chalmers Tvärgata 3, 41296, Sweden
szolnoky&&calmmers.ee
Viktor Andersson
Balázs Kulesár
Department of Electrical Engineering
Department of Electrical Engineering
Chalmers University of Technology
Chalmers University of Technology
Göteborg, Chalmersplatsen 4, 41296, Sweden vikta@chalmers.sst
Göteborg, Chalmersplatsen 4, 41296, Sweden
kullcarr@chllmers.ce
Rebecka Jörnsten
Department of Mathematical Sciences
Chalmers University of Technology
Göteborg, Chalmers Tvärgata 41296, Sweden jornsten@chalmers..o
Abstract
Most complex machine learning and modelling techniques are prone to over- fitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having level of implicit reg- ularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce new framework, Model Gradient Similarity (MGS), that (I) serves as metric of regularisation, which can be used to monitor neu- ral network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high lev- els of label noise or limited sample sizes.
Introduction
Since the inception modern neural network architecture and training, many efforts have been made understand their generalisation properties. When neural networks are trained with Gradi- ent Descent (GD) this induces implicit regularisation in the network, causing it to attain surprising generalisation performance with no extra intervention. Despite this, neural networks will in many instances overfit in the presence of noise and have been shown to have the capacity to completely memorise data [Zhang et al, 2016). Therefore, alongside the desire understand how neural net- works are implicitly regularised, explicit regularisation has been an ongoing pursuit to improve generalisation even further.
broad range of explicit regularisers now exist that attack the problem from many different angles. Below, provide short overview of some of the most commonly used methods. Weight decay
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

[Krogh and Hertz [991 places a L2-penalty on the parameters to encourage a minimum-non norm solution. Normalisation schemes, that normalise the data as it flows through the network, have been shown to act as explicit regularisers |Joffe and Szegedy, 2015). Double back-propagation incorporates the gradients with respect to the loss itself as part of the loss, hence the name, as step [Drucker and Le Cun, [99I]. Although these methods were initially used as a part of a energy minimisation principle, they are now believed to help finding "flat minima", which have been shown to produce solutions that generalise better [Zhao et all 2022]. Dropout turns off connections between neurons during training and thus forces the network not to rely on any one given connection [Srivastava et all 2014]. Another way to view the learning problem is to ignore the network itself and instead focus on the optimisation scheme. Here, multitude of different learning rate and optimisers exist that offer explicit regularisation Some standout examples include Cyclical Learning Rate and Yang, 2020] and the Adam optimisation algorithm [Kingma and Ba, 2015]. More recently, especially in works related to the Neural Tangent Kernel (NTK) [a et all 20IN], focus has been put onto functional regularisation. The core idea is view the neural network architecture being an opaque function approximator and instead regularise it as it is viewed from function space rather than the parameter space. This is usually achieved via a form of kernel ridge regression or an approximation thereof [Bietti et all 2019 Hollman el all 2019].
the gradient of the gradient will be evaluated in the complete
Our contribution. In this article, new framework called Model Gradient Similarity (MGS) is introduced. MGS builds on the idea that regularisation and, to a large extent, generalisation, is tightly connected to how the model evolves during training. We show that MGS can be used to analyse neural network training, monitoring how model gradients evolve together. It is revealed that explicit regularisers, despite their fundamentally different construction, operate via the same underlying mechanisms in terms of increasing model gradient similarity. This insight lays the foundation for new class of regularisers aimed at directly boosting model gradient similarity. We derive two simple MGS regularisation techniques from first principles and compare performance to a wide range of regularisation methods. In majority of the cases, the MGS regularisation achieves top performance and exhibits robustness qualities. This suggests that the MGS framework opens the door for the further development of both novel regularisers and the improvement of existing ones.
2 Model Gradient Similarity
Let x E X denote an input observation in a data batch and Y the target. We will train the model fe. parameterised by 6, using loss function L via gradient descent (GD). Thus, the standard single- observation batch update rule using GD at time is f/+1 = 0, 17 o (f(x), y). where is the learning rate, When training a neural network, fo shall represent the raw output from the network before any additional normalisation is performed (e.g. softmax as in the case of classification).
The loss gradients with respect to the parameters VoL puss-parameter gradient) are thus the main contributing factor to the update of the model. In the general case for common losses, such as mean-squared error or cross-entropy, we apply the chain rule to the loss-parameter gradient: VeL- = Vofa V VVC. We denote the two components as: the loss gradients with respect to the model output v (loss-model gradients) and the model gradients with respect to the model parameters Vofo (model-parameter gradients).
Studying the influence that certain input data has other data points during training is common technique from robust statistics. The use of influence functions to study model behaviour has been explored for neural networks to, for example, help identify important training samples and detect mislahelled data [Koh and Liang 2017]- How learning one data point influences another has also been used in algorithms for clustering data and 2020]. Recently this idea has been broadened under the general concept of gradient similarity/coherncce and has been suggested as an explanation for a neural network's 's ability to generalise [[aahhn et aIj 2020, Chatterjee and Zielinki, 2022].. is here, however, mainly considered from the perspective of the poss-parmettrr gradient VoL and not the maral-parameter gradients Vofe. Contrary to the Joss-parameter gradients, which diminish as any minima is approached, the model-parameter gradients display different behaviour and describe the evolution of the function realised by the model. To help understand why they can be useful in understanding neural network generalisation and regularisation thereof, we will borrow some inspiration about gradient similarity from Charpiat et al. [201%] and use it to define the core ideas behind MGS.
2

---

Good generalisation
Poor generalisation
An illustration on the connection between MGS and generalisation. The arrows represent the model gradients, Vaa for each data point. Left: model that exhibits low similarity between its gradients. The model is free adapt the gradients for each point and therefore learns them individually. The most likely outcome then is that will overfit the data and generalise poorly. Right: A model that is required maintain level of similarity between its gradients. To lower the overall loss, while maintaining model gradient similarity, the model forced to learn from the data in groups defined by similar gradients. Thus, the model will have to learn separations in the data based on such groupings instead of each point individually.
Figure 1:
2.1 Defining MGS and its Relation to Generalisation
When judging the similarity of two points x and x' from the model's point of view, one may be inclined to simply compare the output fa(r) to the other f=()). Unfortunately, due the non-linear nature of neural networks, the same output might be produced for a number of different reasons and is therefore difficult to draw conclusions about similarity directly from the output.
An alternative notion of similarity between x and x' can be defined by how much changing the model output for æ (by training on this sample) changes the output for x'. the points are dissimilar from the model's standpoint, then changing fe(r) should have little influence over fe fz()) and vice-versa. That after one step using GD for single input sample æ the parameters are changed by:
80 = @8=1No/V,C((f))..
This induces the following change in the output of the model for that specific
s0+50(x) ff(x) 50 ((1(0 = f (Vef(()) vef(=)
(f))))))
On the other hand, this update will also yield a change the model for another given by:
fo+50(x)) = fo(x') Vof(x') 80 + ((158113)
2 ( V.f(t))
Therefore, the kernel ke(r, Vef(x) Vef(z') (derived from the model-paramedt gradients) crucial understanding how the model evolves after one step of GD. determines how much of the loss-model gradient V (f(x), y) is used to update the model, up to scaling by the learning rate. Another way to look at it is that (o describes the influence learning x has over x'. If ko (z, æ²) is large, which occurs when the model gradients are similar, then an update for f(x) will also move f((²) the same direction. Likewise, if k(x,, x²) is small, meaning the model gradients are dissimilar, then the update for f(x) will have little affect on f(z'). The cartoon example in figure illustrates how this property can useful understand how model generalises better if exhibits higher MGS.

---

2.2 The contribution of MGS to Gradient Descent
equations and above are combined, the complete update for single step of gradient descent with training data X, is:
60 X) fx( + Vof(X) 50 + 0(100011
fx( (Ko(X) v,((f(X))
[
x(
xx)
V=L(((1)))))
-1
So(mn)
(1), 21)
yyL(f(
Here Kx(X) i(X, X) is the kernel matrix for data X. The update for each model output fo can thus be seen as weighted average of the loss model gradients v (f(x)).
f0+ fo(xx) - Σ (yC((())))) an E
(3)
From equation B. follows that the kernel ultimately controls the weighting of the loss-model gra- dients: the greater the similarity exhibited between the model gradients, the greater the averaging effect will be. This partially explains how GD induces implicit regularisation as, unless Kg (X) is dominated its diagonal, model update for observation 25 will also utilise the loss gradients of other observations.
Furthermore, it has been observed from the Neural Tangent Theory perspective that (x will of- ten align with the ideal kernel YY in classification problems, which perfeetly discriminates targets. This phenomenon has already been investigated as of the reasons behind implicit generalisation for neural networks when trained with SGD [Kopitkov and Indelman, 2020, Baratin et all 2021]]. Here, also see that the alignment will naturally engage the averaging effect, grouping gradients according to their target classes. It is still unclear what the cause of the alignment is. However, MGS provides explanation as why such an alignment is useful. More detail on the connection to Neural Tangent Theory is presented in the supplementary material.
To summarise, apart from capturing to what extent the model gradients are aligned with each other, MGS also determines how loss-model gradients for observations that have high MGS with each other are averaged in the GD step.
2.3 Metrics to Quantify MGS
Since the kernel (x(X) captures coordinated learning between similar observations (equation B), is desirable to identify metrics which can summarise the overall model gradient similarity in a batch of data.
To find relevant metrics Kg we note that the spectrum (X) = (Tef(X))"Vaf(X) is the same as that of Ō (V. (X)) the non-centered version of the sample covariance matrix C' = (X) - Vef(X)) (Vef()) Vef(X))². The spectrum of C and Ö are interlaced as their corresponding eigenvalues A S S A
An < Xn.
We also note that the trace and determinant of C' are commonly used measures of the overall variance and covariance for a given sample of data. Here, the model-parameter gradients V fo constitute the data. Moreover, the trace and determinant of Ö give a rough approximation of the same metrics of C. Proofs of these two properties are given in the supplementary material.
Thus, the summary statistics tr Kg and det Kş can be seen as approximations of the model gradient variance and covariance. Smaller values for the trace and determinant of the kemnel, Kg, reflect higher degree of model gradient similarity and thus a larger averaging effect in equation B
These metrics also serve proxies for measuring the magnitude of the diagonal elements and rel- ative magnitude of the diagonal-to-on cocoonal----of-ddggeeoo -fffddiggnnal elements of Kg. From this perspective, smaller values for the trace and determinant of the Ko reflect how well the kernel can be summarised by low-rank representation. Low-rank Ko can be viewed indicative of coordinated learning in that restricts the number of independent directions in which the functions can evolve in equation R

---

From a practical standpoint, the exact values of these metrics are not important. Rather, we wish to monitor how they evolve during training. Utilising the kernel also constitutes a pragmatic solution since calculating C, of size p x p. is near infeasible for larger networks, whereas Ko is only of size n x n, where p and n are the number of parameters and data batch respectively, and generally < p.
3 Tracking MGS Metrics During Training
Let us now investigate how the metrics tr K₂ and det K₂ evolve during training. Using these metrics, a wide range regularisers, representing the most commonly used ones, are compared. Addition- ally. two new regularisation methods based on MGS are plotted alongside as reference. These are introduced in the next section.
The results are shown in figures D. BL and EL In each example, all methods have their hyper- parameters tuned, using grid search and cross-validation. Additionally the results are aggregated after using different training splits and network initialisations. Figure 0 presents results from a simple funly-con network (FCN) architecture trained on a toy problem generated from two concentric cirles perturbed with noise. Figure BI stems from training large convolution network (AlexNet) on a corrupted version of MNIST with label noise and restricted training size. Figure is created in a similar way but instead uses the more difficult Fashion MNIST dataset [Xiao el all 2017 and a network with residual connections (ResNer20) to verify if the phenomenon still occurs in more complex scenarios. In each example, despite being different problems and different architec- tures, all regularisation methods exhibit the same behaviour: when regularisation applied the rate of MGS metrie growth is decreased, indicating higher model gradient similarity. Furthermore, the test accuracy or generalisation performance (figures B & E. is strongly correlated with the evolution of the MGS metrics. The slower the rate of MGS growth, the better the final generalisation, and when the growth plateaus so does the test accuracy.
We list some interesting observations from the MNIST example in figure B
The unregularised network (gray) goes through a rapid boost in accuracy and then quickly overfits. When test accuracy peaks, there is small plateau in the MGS metries. However, once it starts to decline in accuracy, the MGS metries again grow, indicating model gradient similarity is decreased.
Weight penalty (bluc) is able to regularise the network initially, but also eventually overfits. This is reflected in the MGS metrics as they plateau when test accuracy peaks and then start to increase at a seemingly proportional rate to the decline in performance.
All methods that achieve high test accuracy control the growth of the MGS metrics. Still, many them begin to overfit towards the end of training which coincides with the MGS metrics also gradually increasing.
Only the MGS penalty (to be introduced in the next section) is able maintain stable test accuracy
Dropout (brown) has gaps in its determinant metric. This means that it is causing Ke to be singular which could have ramifications on the stability of the training.
More extensive test bench experiments are provided in section
4 MGS regularisation
In the previous section, we saw a clear connection between boosting MGS (reflected by lower MGS metrics) and the ability of the network to generalise. The effect on MGS by the explicit regularisers was common to all the methods despite their widely differing design. Thus, we may think of each regularisation method as acting as proxy for enforcing model gradient similarity. This motivates the construction of a new regularisation scheme that directly optimises MGS.
We thus modify the original loss include new penalty term (X))(X)) which acts on the MGS kernel:
= ((f()))))))))))))))))))))))))))) - g(Ko(X))
5

---

Unregularised
Weight penaaly iy
penally (del)
10
1.0
log(tr Kol
logidetKa)
14
48-
Unrogubrsed
Dropout We ght panalty
gradient penalty (dots) gradient penalty (parameter)
norm penslty MGSS penally (trace)
peralty (det)
Leaming (lwveree decay!
Figure 2: FCN on two-cirele classification. Top: Decision boundaries are shown to visualise how well the network has generalised for each method. Bottom: MGS metrics vs. epochs for each regulariser are tracked during the training period. The connection between network generalisation and the MGS metrics is clear. For unregularised network which overfits the data, MGS metrics grow rapidly. Once any explicit regularisation is used, MGS metric growth is slowed. Coupled with this, the decision boundaries approach the true model for methods that constrain the MGS metrics better. (Complete set decision boundaries are provided in the supplementary material.)
Aggregated test accuracy
log(tr Ko)
logidet Kg)
10.
arised
peralty (trace)
Learning (ADAM)
Weight ponalty
gracient penalty (parameter)
Test accuracy vs. training epochs. Middle and Right: MGS metrics vs. training epochs. Test accuracy is clearly coupled to MGS evolution. All methods that yield high final test accuracy exhibit small and slow-increasing MGS metrics, indicat- ing higher degree of model gradient similarity.
Figure 3: AlexNet on corrupted MNIST. Left:

---

Aggregated test accuracy
lng(tr
logidet
222
24
24
Unregu arised
Dropout
Loss grodient penalty (data) Loss gradlent penalty (parameter)
MG5 penalty (trace)
Learning rate (ADAM)
Weight ponalty
Figure 4: ResNet20 corrupted Fashion MNIST. Left: Test accuracy vs. training epochs. Middle and Right: MGS metrics vs. training epochs. Even this scenario using a more complex network, with residual connections, and harder dataset, the coupling between test accuracy and MGS evolution is clear. Regularisers that achieve higher test accuracy also minimise the MGS metrics. Of note is the ADAM optimiser which first achieves good test accuracy but then quickly overfits, this is shown directly in the MGS metrics as well.
We already have two prime candidates for namely the previously investigated MGS metries:
(Ir(Ko(X)) a tr Ko
E
Idet (Ko(X)) a det Ko(X) log
II A
where a is a penalty factor and D are the eigenvalues from the spectral dccomposition of Kg. As previously mentioned in section MK smaller values for these penalty metrics reflect higher model gradient similarity. Adding an explicit penalty on these metrics thus forces the network to learn data in groups and preventing it from learning points individually (over-fitting memorization).
The evolution of the two were shown figures E and The metries seem to behave similarly. However, yor (K)) is more efficient to compute it does not require either the complete spectral decomposition nor the full gradient similarity kernel to be computed. However, since both metrics gradient based, they are, of course, computationally more burdensome than commonly used regularisers such as weight decay. For calculations the penalties we make use of Neural Tangents [Novak et all 201%] and Fast Finite Width NTK [Novak et all 2022] libraries which are built on JAX Bradbury et all [00X]]]
We note that the penalties are equivalent to penalising the arithmetic and geometric mean of the eigenvalues of the kernel, respectively. The former will mainly be affected by large leading eigen- values. The latter essentially penalises the eigenvalues of the kernel on log-scale and may thus take more of the overall structure of the kernel into account. Supplemental figure A. depicts results on the toy two-circle data from using both penalties. The differences are subtle. For pragmatic consid- crations, on real data we therefore utilise only the trace penalty for explicit regularisation, but both metrics can be used for monitoring neural network training.
Details on how Kg is calculated with regards mini-batches, vector eector-outputs and large data sets are provided in the supplementary material.
Experiments
We compare the performance of optimising MGS directly versus the most common, explicit regu- larisers. Although there exists plethora of explicit regularisers and modifications thereof, the ones chosen are a representative of the main types of regularisers with the most widespread use in prac- tice. As the aim is to compare performance between regularisers, a systematic investigation is done where the regularisers are rigorously tested against common overfitting scenarios caused by target noise and training size. Two main tests are done, one a classification problem and the other on a regression problem. For each test two different types of architectures are chosen. This test how universal each method is in handling vastly different problems and networks. Finally, test
7

---

Table 1:
Test aceuracy for corrupted MNIST dataset using LeNet-5 architecture. Final test accuracy and one standard deviation is shown with the maximum test accuracy in parenthesis un- derneath. Noise column represents percentage of training labels that have been randomly flipped. Note the ability of MGS to handle large amounts of noise. Comparing max accuracy (in paren- thesis) during training to final aceuracy, MGS also the most consistent and not susceptible to overfitting,
Noise
Unregularised
Dropout
Weight
Loss grad.
MGS
89.6 11.3
95.8 +0.5
83.1 +9.6
95.1 +0.5
95.2 #0.3
0%
(90.2)
(95.9)
(87.1)
(95.1)
(95.2)
62.0 +2.
80.9 +2.9
72.2 +4.2
88.7 +2.0
93.1 +0.7
30%
(85.2)
(92.0)
(79.2)
(92.0)
(93.4)
37.3 +2.9
59.2 #4.1
10.0 #0.4
80.7 #5.9
88.4 #
60%
(73.7)
(83.7)
(62.3)
(81.4)
(89.1)
23.9 +2.4 (62.4)
39.5 +4.7
10.1 #0.5
15.1 #5.4
74.5 +4.0
80%
(56.6)
(23.5)
(17.2)
(75.1)
of robustness to variability found in common training setups is performed. Complete details on the experiments, complete results and code to run them can be found in the supplementary material.
5.1 Classification: corrupted MNIST
We generate a corrupted version of the popular MNIST dataset by applying a motion blur to each training image. The test set is kept uncorrupted, which ascertains how well the network generalises: the network overfits the training set, will learn the corrupted version and perform poorly on the clean data. A testhench of challenging scenarios training size. Fach regularisation method is tuned in an identical setting, of medium difficulty, and then retains the chosen hyper-parameter for the entire testbench. This ensures an even playing field and that each method is given the same starting point. For each testing scenario, multiple runs are performed using a new network initialisation and resampling of the training data. No early stopping is used as we purely want to test how well each regularisation method can prevent over-fitting on its own.
created by varying the amount of label noise and
Table M shows the results for scenarios from the 143 scenario large testbench (see supplemental). In this scenario 3000 samples are used for training at 4 noise levels. It is immediately clear that MGS achieves the best performance overall in many aspects. Not only does it reach higher accuracy levels than the other methods, it is also robust in its performance. This can be seen both by the standard deviation of its final test accuracy, but also when comparing the max test accuracy to its final one. When comparing the maximum values, is also evident that early stopping would not have resulted in better performance for any of the methods compared to the final MGS result. Additionally in supplemental figures A.2 and A.3, the evolution of test accuracy during training is depicted, visually showing that early stopping techniques do not exhibit improved performance over MGS. Indeed, MGS max and final test accuracy essentially coincide, indicating that MGS regularised networks do not overfit. The same conclusions hold true for a fully connected network (see supplemental) where performance overall worse for all methods but MGS outperforms the others.
5.2 Regression: Facial Keypoints
A similar setup to the classification is created using corrupted version of the Facial Keypoints data set was used regression benchmark. Based on images of human faces, the problem is predict the coordinates of 15 key facial features. Each image is first corrupted using a motion blur. Noise is then introduced by adding normally distributed values, using different scale parameters, to the target variables. Here, we fixed the number of training samples while the amount of target noise was varied. Each method tuned using the same scenario first. Once again no early stopping is used. Similar conclusions can be drawn as from the classification results. The results are visible in table 2 where the noise column corresponds to the scale parameter of the normally distributed noise. Interestingly, only Dropout and MGS were actually able to converge to satisfactory performance level. However,

---

Table 2:
Test loss for the corrupted Facial Keypoints dataset using a LeNet-5 architecture. Final test loss and one standard deviation is shown with the minimum test loss in parenthesis underneath. Only Dropout and MGS achieve an acceptable level of final test loss. The other methods also provide marginal increase in performance compared to the unregularised network. In the supplemental we compare results on FCN architecture for which only MGS able to attain good performance.
Noise
Unregularised
Dropout
Weight
Loss grad.
MGS
212.4
68.2 +36.7
14.9 +2.2
113.2 #53.2
14.1 #1.0
0
117411
(54.2)
(13.9)
(90.3)
(13.8)
(104.0)
235.2
54.8 +24.0 (45.2)
15.4 +2.2 (14.4)
92.6+46.5
14.3 +1.1
10
#188.6
(69.0)
(13.6)
(140.5)
102.0
266.1
16.6 #2.2 (15.3)
83.8 #52.7
15.3 +1.3
20
#116.2 (48.2)
+233.1
(49.3)
(14.7)
(115.9)
229.4
94.8 113.5 (39.0)
19.7 +2.8 (17.7)
70.6 +58.0
17.8 #2.0
30
+225.7
(41.9)
(16.9)
(151.5)
Loss grad.
Weight Training size
Dropout
100% (teat
Label noise
Epochs
60%
20%
Quarti
Learning rate
Weight
Dropout
MGS
Figure 5: Test accuracy quantiles after varying different parameters controlling training. MGS plotted in red against the other methods. MGS outperforms the other regularisers, and is also the most robust with regards to changes in the training setup, in all but one case. The only area which it shows degradation in performance is when the learning rate changed. However this seems affect all methods, but weight penalty to a lesser degree.
when results are compared for a FCN architecture (table A.Z. supplemental), Dropout falls into the same category as the other methods while MGS performance remains high. This shows weakness in method such Dropout: that it is inevitably architecture dependent.
5.3 Training parameter robustness
Finally, we test the robustness of each regularisation method by changing: amount of training size, label noise, batch size, learning rate, and epochs. A siddle--groud scenario was chosen from this MNIST testbench as starting point to tune each method. Then, the five training variables were changed individually to both higher and lower values, tracking the performance of each method after training. Consistent with the current findings, MGS outperforms each method substantially. In figure we use radar charts to summarise the test bench results.
We note that MGS is the least affected by change in all but one of the test parameters conditions, and exhibits superior performance compared to the other regularisers. Performance only affected by learning rate to some extent. This is not unexpected as the same can be seen for the other methods apart from weight penalty. Also, as is evident from the GD update step (equation [I), the one directly contributing training parameter in MGS is the learning rate.

---

Conclusion
We introduced the concept of Model Gradient Similarity (MGS) and discussed its connections to regularisation for models trained with gradient descent and the generalisation properties of neural networks. We proposed metrics that can be used summarise MGS for a model and track the training of different neural network architectures for various learning problems.
was shown that wide range of explicit regularisers all appeared attempt to enforce higher model gradient similarity, i.e. lower MGS metrics. Moreover, higher test accuracy performance was shown be reflected in lower MGS metric values.
Based on these findings, a new type of regulariser, geared toward direct control of MGS, was de- signed and found to achieve top performance in several rigorous test bench experiments. Its overall robustness to label noise and training parameter settings was also an indication that directly optimis- ing MGS comes closer to more holistic approach to regularisation.
Taken together, these results provided insight into the underlying mechanisms of neural network regularisation. Due to the higher computational costs for gradient based regularisers, such as the MGS metric penalities introduced here or loss-gradient penalties, their use for direct optimisation is not always efficient. To scale for use in larger networks and in more complex settings, additional work is needed to obtain more efficient ways to compute or approximate the MGS kernel or metrics thereof. Future work could thus focus both how MGS can be used to design as well as to improve upon existing ones. The MGS metrics can be useful as KPI's for measur- ing network's sss current capacity for under/over-fitting Finally, the grouping effect of regularised neural network training, where model gradient similarity encourages coordinated learning across observations, suggests that MGS regularisation be explored for joint prediction modeling and clustering.
regularisers
Acknowledgments and Disclosure of Funding
This research was supported by funding from Centiro Solutions, the Swedish Research Council (VR). the Swedish Foundation for Strategic Research, and the Wallenberg AI, Autonomous Systems and Software Program (WASP).
10

---

References
Aristide Baratin, Thomas George, Csar Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment, 2021.
David Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference Learning Reprexentations, 2021.
Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal. A kemnel perspective for regu- larizing deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 674. PMLR, 09-15 Jun 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- MMiln,,,, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. hwetthuu.oooopeee goom/google/jxx
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Satrajit Chatterjee and Piotr Zielinski. On the generalization mystery in deep learning, 2022. URL wwwprr//b////aa https://arddv.or//aabb/
10036.
H. Drucker and Y. Le Cun. Double backpropagation increasing generalization performance. In SICNN-9I-Seatie International Joint Conference on Neural Networks, volume ii, pages 145-150 vol.2, 1991. doi: 10. 11W.1///NN......11.....
Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. study of gradient variance in deep learning, 2020. URL https: // arxiv org/ htsp/bbs/2000/ 2007. 04532.
Hangfeng He and Weijie Su. The local elasticity neural networks. Proceedings of the 8th International Conference on Learning Representations, 2020.
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust learning with jacobian regularization, 2019.
Paul Honeine. An eigenanalysis of data centering in machine learning, 2014. URL https://axxxi org/abs/14/ 2904
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Interna- tional Conference on Machine Learning Volume 37, ICML' 15, page 448456. JMLR.org. 2015.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Pang Wei Koh and Percy Liang Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' page 18851894, JMLR.org, 2017.
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Igor Farkaš, Paolo Masulli, and Stefan Wermter, editors, Artificial Neural Networks and Machine Learning - ICANN 2020, pages 168-179, Cham, 2020. Springer International Publishing.
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In Proceed- ings of the 4th International Conference on Neural Information Processing Systems, NIPS*91, page 950957, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc
11

---

Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 3572-8583. Curran Associates, Inc., 2019.
Jiaqi Li and Xiaodong Yang. A cyclical learning rate method in deep learning training. In 2020 In- ternational Conference on Computer, Information and Telecommunication Systems (CITS), pages 1-5, 2020. doi: 10.1099CCITSA9457220002222222
Jörg Martin and Clemens Elster. Inspecting adversarial examples using the fisher information. Neu- rucomputing, 382:80-86, 2020. doi: https://doi.org/ org//0. /016///.eecomm21111111 1..09.1..05.....
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision, 2019. URL
https://arxiv. wttp://.rrggbbb///bggeb 02337.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schocnholz. Neural tangents: Fast and easy infinite neural networks in python, 2019. URL
02803.
Roman Novak, Jascha Sohl-Dickstein, and Samuel Stern Schoenholz. Fast finite width neural tan- gent kernel. In Fourth Symposium on Advances in Approximate Bayesian Inference, 2022.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' d'Alché-But Buc, E. Fox, and Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Mach. Learn. Res., 15(1): 19291958, jan 2014.
Yoshio Takane and Tadashi Shibayama. Principal component analysis with external information both subjects and variables. Psychometrika, 56(1)))())))))) (6(1:)trriia.66199777711: 56(1):97-100, Mar 1991. doi: 10.1007/BF0229558.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- manist novel image dataset for benchmark- ing machine learning algorithms, 2017.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2016. URL https://arxiv.. org/ hbs///b///// 1611. 03530
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving gener- alization in deep learning, 2022. URL https://arxxx org/ ab///2bb/222 03599.
12


Reviews:
[{"review_id": "xbVvYpgeskn", "reviewer": "Reviewer_1j14", "summary": "In this paper, the authors proposed a new concept called \"Model Gradient Similarity\" (MGS), which can be used to analyze the gradient descent algorithm and the effect of explicit regularizations, and can be used to design new regularization schemes. The main contributions of this paper are as follows:\n\n(1) The authors defined MGS as the dot product between two model-parameter gradient vectors from two different training inputs -- this can be viewed as a kernel. The authors did an analysis based on first-order Taylor decomposition, and derived a first-order approximation of how the model behavior would change after one update of gradient descent. In essence, the (approximate) amount of change is (the negative of) the learning rate, multiplied by the matrix vector product between the MGS kernel matrix and the loss-model gradient vector.\n\n(2) The authors further developed two metrics to summarize the MGS, by using the trace and the determinant of the MGS kernel matrix computed from training data. These can be used to approximate variance and covariance of model gradients on training data, and provide summary statistics of MGS -- the smaller these metrics, the larger the MGS is.\n\n(3) The authors tracked the MGS metrics during training for several regularization schemes, and discovered that all of them to some extent function by trying to inhibit the growth of MGS metrics (i.e., encouraging higher MGS among training samples).\n\n(4) Finally, the authors designed explicit regularization schemes, one based on the trace and the other the determinant of the MGS kernel matrix on training data and showed that (1) both worked well on a toy example, and (2) the trace regularization can be applied to real-world datasets to improve both accuracy and robustness -- for computational reasons only trace is applied to real datasets.", "questions": "I do not have questions -- the paper is very well written and everything is explained clearly.", "limitations": "The authors have adequately addressed the limitations of their work, namely, the computationally intensive nature of gradient-based regularization schemes, and also in particular, of the proposed MGS determinant regularization.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The first-order Taylor decomposition analysis on gradient descent to show the effect of MGS on model outputs is very neat. Given that the learning rate is usually small in practice, the change in parameters after one update is generally small in norm, so the first-order analysis generally holds. Great work!\n\n+ The summary metrics to quantify MGS are theoretically grounded and sound.\n\n+ The observation that all explicit regularization schemes inhibit MGS metrics (trace and determinant) growth is also insightful, and sheds light on why regularization works.\n\n+ The proposed MGS trace regularization is simple and effective.\n\nWeaknesses:\n\nThere are not many weaknesses. If possible, I would love to see a mathematical analysis of how MGS is linked to regularization schemes such as weight decay -- but this can be future work.\n\nThere is one small typo that needs to be fixed:\n\nEquation (2) contains an error after the \"approximately equal\" sign. It should be f_\\theta(x'), rather than f_\\theta(x).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fMsBE2S-b6x", "reviewer": "Reviewer_k97d", "summary": "This paper presents a novel view on regularization for neural networks. The core idea focuses on the similarity of model gradients when evaluating data points. The paper proposes a kernel that conveys the influence of one training sample has over the other. Two global metrics, the trace and the determinant, are used as proxy metrics to track the model generalization. They are also used as regularizers by including them in the optimization objective. Experimental results show that tracking and optimizing these metrics leads to models that retain a high generalization, even in the presence of noisy annotations for classification and regression problems.", "questions": "1. Figure 2: regularization methods are dependent on hyperparameters. Dropout depends on the probability of dropping out, weight decay is controlled by a small coefficient $\\alpha_{\\theta}$, etc. Are these results robust to the choice of these hyperparameters?\n2. In general: the different regularizations depend on hyperparameters that are proper of each method. Those that are included in the loss (loss grad and weight decay) are modulated by a hyperparameter $\\lambda$, similar to the way $\\alpha$ is defined for $\\widehat{\\mathcal{L}}(f(X), Y)$ (equations between lines 175-177). Dropout depends on the probability of an activation being dropped (usually $p=0.5$). What protocol was followed to guarantee that those hyperparameters were optimal? This is particularly relevant for section 6.3 where the other hyperparameters were analyzed.\n3. How dependent are these results on the batch size?\n4. Corrupted MNIST: Section 6.1 mentions the use of motion blur, but table 1 only talks about flipped labels. What is considered a corruption in this case and what is the motivation behind adding motion blur?\n5. Could MGS (i.e., $\\text{tr}~K_{\\theta}$) be used as an early stopping criterion?", "limitations": "There is little discussion of the limitations of this work, even though there are a few points that could be addressed:\n\n- Even though the method is well supported by theoretical findings, a discussion on the effects of hyperparameters is lacking. In particular, effects of the batch size or assumptions about the data distribution e.g., the test set exhibits no covariance shift.\n- From the empirical results, the model architecture lacks representative (and relevant) components that are often found in modern ANN architectures (see second point on “Significance”).\n- A discussion of the experimental hyperparameters (how they were found and how they affect the results) is also missing in the main paper.\n- Finally, an account on the computational costs of MGS metrics and the other baselines should be presented.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- *Originality:* \n    - Most of the elements in the paper, in isolation, are not new: the topic of regularization in ANNs has been studied for the past few decades. Tracking or including gradients (or other statistics) of data-batches to add constraints to the optimization process has also been done in the past. The idea of measuring the influence between samples during learning has also been addressed before. (In this regard, I was missing a discussion to the work on influence functions from Koh and Percy [1].) However, the originality from this work comes from the unique way to combine these ideas. In particular, the use of the determinant and the trace as global metrics.\n\n- *Quality*:\n    - In general, the main claims of the paper are well supported. A weak point can be found e.g., in the motivation (L39-40) where, the fact that one thing (regularization) can be achieved in multiple ways, doesn’t immediately implies that it is not well understood. One could argue that the more understood a problem gets, the more ways one could find to make it work.\n\n- *Clarity*:\n    - MGS was never defined nor tied to any metric or formula. (Saying that it is a “framework” is still vague, and needs a more concrete definition.) Something like “any measure of similarity that expresses the influence that one sample has on the model” will make MGS more tangible. Alternatively, MGS could be represented by the kernel $K_{\\theta}$, while $\\text{tr}$ and $\\text{det}$ are the proposed metrics to track MGS.\n    - Plots are small and most axes are not labeled.\n    - Introducing MGS regularizers in section 5, while already presenting results based on those in section 4 is rather confusing. I strongly suggest restructuring the development of the paper so that it follows a more sequential timeline.\n    - For the abstract, please use `\\begin{abstract}...\\end{abstract}`.\n    - L143: does FCN stand for fully-connected network? Make sure it is explained somewhere to prevent confusions with “fully *convolutional* networks.”\n\n- *Significance:*\n    - Experiments using AlexNet (Figure 3): how representative is AlexNet w.r.t. state-of-the-art DL? Can architectural components such as residual connections, batch-normalization or initialization methods (He initialization vs. truncated gaussians) that are not part of AlexNet, affect the outcome of these experiments?\n    - For a better overview of the viability of MGS metrics, it is important to report the computational cost (memory and FLOPs) of using MGS compared to dropout, weight-decay, etc.\n\n## References\n\n1. Koh, Pang Wei, and Percy Liang. \"Understanding black-box predictions via influence functions.\" *International conference on machine learning*\n. PMLR, 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aArb2Mx8Ck", "reviewer": "Reviewer_TsWy", "summary": "the submission proposed to use the trace of the kernel matrix, which is constructed by taking the inner product of a matrix that has gradients of function outputs w.r.t. parameters and the transpose of the matrix itself, to regularise the training of a neural network. The submission through experiments showed that their proposed gradient-based regulariser provided better generalisation than other existing regularisation approaches. ", "questions": "See above. ", "limitations": "See above.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. the title of the submission doesn't seem to reflect the content of the submission. \n\nThe analysis and the proposed regulariser eventually don't convey much information on the similarity of gradients computed on a pair of data samples, since it is purely based on the l2 norm of gradient vectors on individual data samples. Even though the determinant of a matrix utilises the pair-wise similarity, as the submission mentioned as well, the difference between the effect of trace and that of the determinant is subtle.\n\n2. I am not sure how the gradient is used when there are multiple outputs, for example, multi-class classification problems.\n\nFor classification problems with more than two output classes, the gradient of the function w.r.t. the parameter on each data sample is a matrix. It doesn't seem clear to me how the proposed approach uses the matrix. \n\n3. The proposed regulariser is similar to the Lipschitz constraint that has been used in Wasserstein GANs, and other regularisation schemes. \n\nThe idea of Lipschitz constraint is to make sure that a small change in the input data sample only leads to a tiny change in the output, and the proposed approach in this paper, which computes the trace of the kernel matrix, regularises / penalises the norm of a gradient vector, which is essentially regularises the Lipschitz constant of the model locally.\n\nA relatively thorough theoretical comparison can be found here: https://proceedings.neurips.cc/paper/2020/file/8929c70f8d710e412d38da624b21c3c8-Paper.pdf\n\n4. In all experiments, I wonder if any early stopping is applied, and if not, I think early stopping could alleviate demonstrated overfitting issues. \n\nIt is a little wild to me that weight regularisation and dropout performed very poorly, but I think realistically, early stopping is well-adopted on top of various regularisation schemes, and it could reduce the performance between the proposed approach and all other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept