Stiffness: A New Perspective on Generalization in Neural Networks
Authors: Stanislav Fort
Paweł Krzysztof Nowak
Stanisław Jastrzebski
Srini Narayanan
Venue:  (2020)
Keywords: stiffness
gradient alignment
critical scale

Abstract: We investigate neural network training and generalization using the concept of stiffness. We measure how stiff a network is by looking at how a small gradient step on one example affects the loss on another example. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate. We experiment on MNIST, FASHION MNIST, and CIFAR-10 using fully-connected and convolutional neural networks. Our results demonstrate that stiffness is a useful concept for diagnosing and characterizing generalization. We observe that small learning rates reliably lead to higher stiffness at a given epoch as well as at a given training loss. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of a dynamical critical length that characterizes the distance over which datapoints react similarly to gradient updates. The dynamical critical length decreases with training and the higher the learning rate, the smaller the critical length.

Under review as a conference paper at ICLR 2020
STIFFNESS: A NEW PERSPECTIVE ON GENERALIZA- TION IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate neural network training and generalization using the concept of stiffness. We measure how stiff a network is by looking at how a small gradient step example affects the loss another example. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, training iteration, and learning rate. We experiment on MNIST, FASHION MNIST, and CIFAR-10 using fully-connected and convolu- tional neural networks. Our results demonstrate that stiffness for diagnosing and characterizing generalization. We observe that small learning rates reliably lead to higher stiffness at a given epoch as well as at given training loss. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of dynamical criti- cal length that characterizes the distance over which datapoints react similarly gradient updates. The dynamical critical length decreases with training and the higher the learning rate, the smaller the critical length.
useful concept
INTRODUCTION
Neural networks are a class highly expressive function approximators that proved to be successful in approximating solutions to complex tasks across many domains such vision, natural language understanding, and game-play. They have long been recognized as universal function approximators (Homik et al. 1989 Cybenko 1989 Leshno al. 1993). The specific details that lead to their expressive power have recently been studied in Montúfar et al. (2014); Raghu et al.] (2017); |Poole| al.) (2016). Empirically, neural networks have been extremely successful generalizing to new data despite their over-parameerrzzatoo for the task at hand, as well as their proven ability to fit arbitrary random data perfectly Zhang al. (2016);
Arpit (rrpt al. (2017).
The fact that gradient descent is able to find good solutions given the highly over-parametrized family of functions has been studied theoretically in Arora et al. (2018) and explored empirically in Li et al. (2018), where the effective low-dimensional nature of many common learning problems shown. Fort Scherlis (2019) extends the analysis in [L al. (2018) to demonstrate the role of initialization on the effective dimensionality. and Fort & Jastrzebski (2019) use the result to build
phenomenological model of the loss landscape.
Du et al. 201 8a) and Du et al. (2018b) use Gram matrix study convergence in neural network Worah (2017) study the concentration properties of a similar covari- ance matrix formed from the output of the network. Ghorbani et al. (2019) investigate the Hessian eigenspectrum and Papyan| (2019) show how it is related to the gradient covariance. Both concepts closely related to our definition of stiffness.
empirical loss. Pennington
To explain the remarkable generalization properties of neural networks, it has been proposed (Ra- haman et al.) 2018) that the function family is biased towards low-frequency functions. The role of similarity between the neural network outputs to similar inputs has been studied in SSchonhhllz et al. (016) for random initializations and explored empirically in Novak et al. 2018.
1.1 OUR CONTRIBUTION
In this paper, we study generalization through the lens of stiffness. We measure how stiff a neural network is by analyzing how small gradient step based one input example affects the loss

---

Under review as a conference paper at ICLR 2020
on another input example. Mathematically, if the gradient of the loss at point X1 with respect to the network weights is [(X1 = I1. and the gradient at point X₂ is T2. we define stiffness 91 -2. We specifically focus on the sign of 91 92 well as the cosine between the two vectors co (2) = タ1 f2. where 9 = // ø/lgl. which both capture the resistance of the functional approximation learned to deformation by gradient steps. We find the concept of stiffness useful in diagnosing and characterizing generalization. As a corollary, we use stiffinss to characterize the regularization power of learning rate, and show that higher learning rates bias the functions learned towards lower stiffness.
We show that stiffness is directly related generalization, and in particular that it starts dropping sharply at the moment training and validation set losses stop evolving together. We explore the concept of stiffness for fully-connected (FC) and convolutional neural networks (CNN) on classi- fication datasets (MNIST, FASHION MNIST, CIFAR-10). We focus on how stiffness between data points depends on their 1) class membership. 2) distance between each other in the space of inputs, 3) training epoch, and 4) the choice of learning rate. We study stiffness between two images in the training set, one in the training and one in the validation set, and both images the validation set.
We observed the stiffness based on class membership and noticed clear evolution towards higher stiffness for images between different classes and towards lower stiffness within the same class. We diagnose and characterize the class-depentee neural networks on the datasets mentioned above in different stages of training. We observe the stiffness between inputs regress zero with the onset of overfitting. demonstrating the clear connection to generalization.
stiffness matrix for fully-connected and convolutional
The choice of learning rate effects the stiffness properties of the learned function significantly, High learning rates induce functional approximations that are less stiff over larger distances (i.e. data points further apart stop responding similarly to gradient updates). We define the concept of dynam- ical critical length to capture the phenomenon.
This paper is structured as follows: we introduce the concept of stiffness and the relevant theory in We describe our experimental setup in Section 3 and discuss their results in Section We conclude with Section
Section 2
THEORETICAL BACKGROUND
2.1 STIFFNESS DEFINITIONS
Let functional approximation (e.g. neural network) f be parametrized by tunable parameters W. Let us assume a classification task and data point have the ground truth label y. A loss L(fw(X) y) gives the amount of mismatch between the function's output at input X and the ground truth y. The gradient of the loss with respect to the parameters
V Www.C(/w(Wc (X),,))
is the direction in which, if were to change the parameters W, the loss would change the most rapidly (for infinitesimal step sizes). Gradient descent uses this step (the negative of it) to update the weights and gradually tune the functional approximation better correspond to the desired outputs on the training dataset inputs. Let us consider two data points with their ground truth labels (X1, V1)
Stillars
Sefffnnee
StiTioee
C. Ca (Na 0
Figure I: A diagram illustrating the concept of stiffness. It can be viewed in two equivalent ways: a) as the change in loss at datapoint induced by the application of gradient update based on another datapoint, and b) the alignment of loss gradients computed at the two datapoints.
and (X2, y2) We construct gradient with respect to example 1 as zi Vw (wf((w(X1)) and
2

---

Under review as a conference paper at ICLR 2020
ask how do the losses on data points 1 and 2 change the result of small change of W in the direction -91, i.c. what is
ACI = L(fw-est (X1), yn) ((fw((())))))) vn),
(2)
which is equivalent to
AC = -EV
(3)
The change in loss on input 2 due to the same gradient step from input becomes equivalently A2 = -VV <fW- (X2), y2) -Eğ1 02 We are interested in the correlation in loss changes AL1 and 2 We know that A AL < O since we constructed the gradient update accordingly. We define positive stiffness mean AC2
as well, i.e. that losses both inputs went down. We assign the stiffness of O for AL2 0. If AC2 > O, the two inputs would be anti-stiff (negative stiffness). The equations above show that this can equivalently be thought of the overlap between the two gradients Ø1 T2 being positive for positive stiffness, and negative for negative stiffness. We illustrate this in Figure
The above indicate that what we initially conceived of as change in loss due to the application of small gradient update from one input another is in fact equivalent to analyzing gradient alignment between different datapoints.
We will be using different definitions of stiffness: the sign stiffness and the cosine stiffness. We define the sign stiffness to be the expected sign of gi 92 (or equivalently the expected sign of AC1 222) as
Saga ((X,, 1, 3/1 (X2, 92); f) E Isign (gi 92)]
where stiffness depends on the dataset from which X₁ and X2 are drawn. The cosine stiffness is
Sens (x,, v1), (X2, 92): /) = E [cos (gi -2)]
(5)
where cos (gT 92) - (g/ I ()) We use both versions of stiffness as they are suitable to highlight different phenomena the sign stiffness shows the stiffness between classes clearer, while the cosines stiffness is more useful for within-class stiffness.
2.2 TRAIN-TRAIN, TRAIN-VAL, AND VAL-VAL
When measuring stiffness between two datapoints, we have options: 1) choosing both datapoints from the training set (we call this train-train), 2) choosing one from the training set and the other from the validation set (train-val), and 3) choosing both from the validation set (val-val). The train- val stiffness is directly related to generalization, it corresponds the amount of improvement the training set transferring to the improvement of the validation set. We empirical observe that all 3 options are empirically remarkably similar, which gives us confidence that they all track general- ization. This is further supported by observing their behavior is function of epoch in Figure
2.3 STIFFNESS BASED CLASS MEMBERSHIP
A natural question to ask is whether a gradient taken with respect to an input X1 in class will also decrease the loss for example X₂ with true class c2. particular, we define the class stiffness matrix
C(ca, cb. =
E XxEr,
[S((11, (X2, 92))]
(6)
The on-diagonal elements of this matrix correspond the suitability of the current gradient update to the members of a class itself. In particular, they correspond to within class generalizability. The off off-diagonal elements, the other hand, express the amount of improvement transferred from one class to another. They therefore directly diagnose the amount of generality the currently improved features have.
consistent summary of generalization between classes is the off-diagonal sum of the class stiffness matrix
Shewwen classes
Σ
Σ C(c1,
Nc(Ne 1)
3

---

Under review as a conference paper at ICLR 2020
In experiments, track this value function of learning rate The quantity related to how generally applicable the learned features are, i.c. how well they transfer from class to another. For example, for CNNs learning good edge detectors in initial layers typically benefits all downstream tasks, regardless of the particular class in question. We do the equivalent for the within-class stiffness (= diagonal elements). When the within-class stiffness starts going < 1, the generality of the features improved does not extend to even the class itself.
reached fixed loss.
2.4 STIFFNESS AS A FUNCTION OF DISTANCE
We investigate how stiff two inputs are based on how far away from each other they are. We can think of neural networks as form of kemnel learning and here we are investigating the particular form of the learned kernel. This links our results to the work on spectral bias (towards slowly-varying, low frequency functions) in Rahaman et al. (2018). We are able to directly measure the characteristic size of the stiff regions in neural networks trained on real tasks which we call dynamical critical length Ç. We work with data normalized to the unit sphere |X| = and use their mutual cosine to define their distance
X1 X₂
distance X1, x2)=11111
X ||21
which has the advantage of being bounded between 0 and 2. We track this threshold distance & as function of training and learning rate to estimate the characteristic size of the stiff regions of a neural net.
METHODS
1.0
0.2
trein
10a
diagonal
(1oos Bell
0.6
geiffness =
Off diagenal (
yal - val
Loss
triin
cal
0.2
Val
0.0
10"
-0.2
10
10'
10
10
10"
107
10'
10"
10'
10'
Epoch
Epoch
The evolution of training and validation loss (left panel), within-class stiffness (central panel) and netween-classee stiffness (right panel) during training. The onset of over- over-fitting is marked orange. After that, both within-class and between-classes stiffness regress to O. The same effect is visible in stiffness measured between two training set datapoints, one training and one validation datapoint, and two validation set datapoints.
Figure/ 2:
3.1 EXPERIMENTAL SETUP
We ran a large number of experiments with fully-connectdd (FC) and convolutional neural networks (CNN) on 3 classification datasets: MNIST (LeCun & Cortes 2010), FASHION MNIST Xiao et al. (017), and CIFAR- Krizhevsky (2009). Using those experiments, we investigated the behavior of stiffness as a function of 1) training epoch, 2) the choice of learning rate, 3) class membership, and 4) the input space distance between images.
For experiments with fully- fully-connected neural networks, we used a 3 layer ReLU network of the form X + 500 + 300 - 100 - y. For experiments with convolutional neural networks, we used a layer network with filter size and the numbers of channels being 16, 32, and 32 after the respective convolutional layer, each followed by 2 x max pooling. The final layer was fully-connected. No batch normalization used.
4

---

Under review as a conference paper at ICLR 2020
We pre- reepprocsssed the network inputs have zero mean and unit variance, and normalized all data to the unit sphere |X| 1. We used Adam with different (constant) learning rates optimizer and the default batch size of 32.
3.2 TRAINING AND STIFFNESS EVALUATION
We evaluated stiffness properties between data points from the training set, one from the training and one from the validation set, and both from the validation set. We used the training set to train our model. The procedure was as follows: 1) Train for a number of steps on the training set and update the network weights accordingly. 2) For each of the modes { train-train train-val, and val- val], go through tuples of images coming from the respective datasets. 3) For each tuple calculate the loss gradients ÿ1 and 92, and compute check (ğg F2) and cos 02). 4) Log the input space distance between the images as well as other relevant features. In our experiments, we used fixed subset (lypically of & 500 images for experiments with 10 classes) of the training and validation sets to evaluate the stiffness properties on. We convinced ourselves that such subset is sufficiently large to provide measurements with small enough statistical uncertainties, which we overlay in our figures.
3.3 LEARNING RATE DEPENDENCE
We investigated how stiffness properties depend on the learning rate used training. To be able to do that, first looked at the dynamical critical scale & for each training step of our network, and then compared those based on the epoch and training loss at the time in order to be able to compare training runs with different learning rates fairly. The results are shown in Figures 4 and 6
4 RESULTS
4.1 STIFFNESS PROPERTIES BASED ON CLASS MEMBERSHIP
SemplocWW FASHION MMST atcsccle cotrasaionl
Ove
Owe
Owe
Owe
StmppaCCNN MMST #tecchh110
SimpleCua FASHION BRAST wosth-20 mannte
Cless
Cress
Gass
Figure 3: Class membership dependence of stiffness for CNN on FASHION MNIST at 4 different stages of training. The figure shows stiffness between train-train, train-val and val-val pairs of images, as well as sign and cosine metrics.
5

---

Under review as a conference paper at ICLR 2020
We explored the stiffness properties based class membership function of training epoch different stages of training at initialization, very early, around epoch 1, and at the late stage. Our results are summarized in Figures 3 and Figures and g The within-class (on diagonal) and between-classes results are summarized in Figure 4 which as an example of equivalent plots we generated for all our experiments. Initially, improvement based input from particular
Stiffness vs epoch for CNN FASHION MNIST
train,triag
train-val
val.val
1.0
1.0
1.0
0.0
(008 ..1))
0.4
0.4
0.4
0.2
0.2
0.2
-0.-
-0.2
-0.4
-0.4
-0.4
2.0
1.5
1.0
0.5
0.0
0.5
1.5
2.0
-20
1.5
10
0.5
0.0
0.
10
2.0
-20
1.5
1.0
0.5
0.0
0.5
1.0
2.0
3.0
1.0
1.0
(1. Bb))))
0.5
0.0
0.0
0.0
-0.5
-10
-2.
-15
-1.0
-1.0
-20 -.5 11--11-55
-1.0
2.001.
1.0
005
0.0
1.0
log:o (Fpoch)
(Epoch))))))
1ogno (Epoch)
Figure 4: Stiffness as function of epoch. The plots summarize the evolution of within-class and between-classes stiffness measures as function of epoch of training for a CNN on FASHION MNIST.
class benefits only members of the same class. Intuitively, this could be due some crude features shared within a class (such as the typical overall intensity, or the average color) being learned. There no consistent stiffness between different classes at initialization. As training progresses, within- class stiffness stays high. In addition, stiffness between classes increases as well, given the model were is powerful enough for the dataset. With the onset of overfitting, as shown in Figure 2 the model becomes increasingly less stiff until even stiffness for inputs within the same class is lost.
4.2 STIFFNESS AS A FUNCTION OF DISTANCE BETWEEN DATAPOINTS
CNN FASHION MNIST LR
CNN FASHION HNIST
train train
tain-vll
valvval
trgin brain
tran
clacs
Same class
X.))
oni XX)
X3
X..
FASHION MNIST
CVN FASHION
train
traloval
yalve
train
bran
valvvl
Same class
68
Same class
oost X²)
Xoo Xz)
Xont Xz)
Xos Xz)
Xo Xa)
Figure 5: Stiffness between images of the same class as function of their input space distance for 4 different stages of training of a CNN on FASHION MNIST.
We investigated stiffness between two inputs as a function of their distance in the input space in order measure how large the patches of the learned function that move together under gradient

---

Under review as a conference paper at ICLR 2020
updates We focused examples from the same class. Examples of our results shown in Figure and Figures 9 and 10 Fitting a linear function to the data, we estimate the distance at which stiffness goes to 0, and called it the dynamical critical length Equivalent plots were generated for each epoch of training in each of our experiments in order to measure used in Figure 6
4.3 THE DYNAMICAL CRITICAL LENGTH Ç. AND THE ROLE OF LEARNING RATE
Critical length CNN FASHION MNIST
train-train..
train-val...
val-val
Ohtical longth
0.6
Epoch
Epoch
Epoch
Critical length for CNN on FASHION MNIST
train-train
train-val
val-val
1.8
1.6
Criticl length
10
17-10-10
10*
10a
10
Train loss
Train loss
Train loss
Figure 6: The effect of learning rate the stiffness dynamical critical scale & the input space distance between images over which stiffness disappears. The upper part shows & for different learning rates as a function of epoch, while the lower part shows it as function of training loss in order be able to compare different learning rates fairly. The larger the learning rate, the smaller the stiff domains.
At each epoch of training for each of our experiments, we analyzed the distribution of within-class stiffness between images based on their distance, and extracted the zero crossing which we call the critical dynamical scale ç. In Figures and [1 we summarize the dependence of on the epoch of training as well as the training loss for different learning rates. We use the training loss to make sure are comparing runs with different learning rates the equivalent stages of training. We that the bigger the learning rate, the smaller the domain size g.
4.4 STIFF DOMAIN SIZE THE CHARACTERISTIC LENGTH SCALE?
natural question arises as whether the characteristic distance between two input points which stiffness reaches zero defines the typical scale of spatial variation of the learned function. Unfor- tunately, that is not necessarily the case, though can be for some families of functions. The stiff domain sizes visible in c.g. Figure represent the typical length scale over which neural networks react similarly to gradient inputs, rather than the typical length scale of variation of the function value itself.
To illustrate the difference, imagine a function that varies rapidly over input data, however, whose losses over the same data move in the same direction application of gradient step based any of the data points. This function would have a small characteristic length scale of value variation, yet large stiff domain size.

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION AND CONCLUSION
We explored the concept of neural network stiffness and used it to diagnose and characterize gener- alization. We studied stiffnness for models trained on real datasets, and measured its variation with training iteration, class membership, distance between data points, and the choice of learning rate. We explored stiffness between pairs of data points coming both from the training set, one from the training and one from the validation set, and both from the validation set. stiffness is directly related to the transfer of improvements on the training set to the validation set. We used two different stiffness metrics the sign stiffness and the cosine stiffness to highlight different phenomena.
On real data, we explored models trained on MNIST, FASHION MNIST, and CIFAR-10 through the lens of stiffness. In essence, stiffness measures the alignment gradients taken at different input data points, which we show is equivalent to asking whether a weight update based on one input will benefit the loss on another. We demonstrate the connection between stiffness and generalization and show that with the onset of overfitting to the training data, stiffness decreases and eventually reaches O, where even gradient updates taken with respect images of a particular class stop benefiting other members of the same class. This happens even within the training set itself, and therefore could potentially be used as a diagnostic for early stopping to prevent overfitting.
Having established the usefulness of stiffness dependence on class membership. We find that in general gradient updates with respect to member of class help to improve loss on data points in the same class, i.e. that members of the same class have high stiffness with respect to each other. This holds at initialization as well as throughout most of the training. The pattern breaks when the model starts overfitting to the training set, after which within-class stiffness eventually reaches O. We observe this behavior with fully-connected and convolutional neural networks on MNIST, FASHION MNIST, and CIFAR-10.
diagnostic tool for generalization, we explored its
Stiffness between inputs from different classes relates to the generality of the features being learned and within-task transfer of improvement from class to class. With the onset of overfitting, the stiff- ness between different classes regresses to 0, as does within-class stiffness.
We also investigated the characteristic size of stiff regions in our trained networks at different stages of training. By studying stiffness between two inputs and measuring their distance in the input space, we observed that the farther the datapoints and the higher the epoch of training, the less stiffness exists between them on average. This allowed us to define the dynamical critical scale an input space distance over which stiffness between input points decays to 0. & corresponds to the size of stiff regions- patches of the data space that can move together when gradient update is applied, provided it were an infinitesimal gradient step. For finite step sizes, the matter becomes more complicated, the linear regime in which we operate ceases apply.
We investigated the effect of learning rate on stiffness by observing how & changes as a function of epoch and the training loss for different learning rates. We show that the higher the learning rate, the smaller the &. i.c. for high learning rates the patches of input space that are improved together are smaller. This holds both as a function of epoch and training loss, which we used in order to compare runs with different learning rates fairly. This points towards the regularization role of learning rate on the kind of function we learn. We observe significant differences in the characteristic size of regions of input space that react jointly to gradient updates based on the learning rate used to train them.
In this paper, all the experiments were conducted with two fixed architectures. One obvious exten- sion to the concept of stiffness would be to ascertain the role stiffness might play in architecture search. For instance, we expect locality (as in CNN) to reflect in higher stiffness properties. It is quite possible that stiffness could be a guiding parameter for meta-learning and explorations in the space of architectures, however, this is beyond the scope of this paper and potential avenue for future work.
In summary, we defined the concept of stiffness, showed its utility in providing perspective to better understand generalization characteristics in a neural network and observed its variation with learning rate.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. CoRR, abs/1802.06509, 2018. URL httpp://axxv.
org/abs/ 1802. 06509
Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th In- ternational Conference on Machine Learning, ICMI, 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 233-242, 2017. URL http::/proceedd ngs ml http:/gg..........aaaa..pppp..ee wwtppp/////arrpppp//ppp html
George Cybenko. Approximation by superpositions of sigmoidal function. MCSS, 2:303-314, 1989.
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient Descent Finds Global Minima of Deep Neural Networks. arXiv:1811.03804 [cs, math, stat), November 2018a. URL http:: wwtp:iivvoorraaioooooooo awaaaaass//// 03814 03804
arXiv: 181 1.00044.4.
Simon S. Du, Xiyu Zhai, Barnabas Poezos, and Aarti Singh. Gradient Descent Provably Optimizes Over-parameteriized Neural Networks. arXiv: 1810.02054 [cs, math, stat], October 2018b. 8b.. URL
arXiv: 1810.02054.
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes, 2019.
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural network loss landscapes. Proceedings of the AAAI Conference on Artificial Intelligence, 33: 35743581, Jul 2019. ISSN 2159-5399, doi: 10.
URL http:://xx
doi 1ei.org/111 10. 1609 aaai. v3310lvv3311...0 330135577733
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density, 2019.
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approxi- mators. Neural Nenw., 2(5):359 366, July 1989 ISSN 0893-6080, doi: 10.1016/0993-6000)88) 90020-8. hRLhtt:: dx. doi org 10 10011111100 10111/11111110 6080 (89) (0000000000010010001000 90020-8
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL httpp://aaan
lecun cwm eeeeeeddmmmmm st.
Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolynomial activation function can approximate any function. Neural Networks, 6:861-867, 1993.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimensioon of objective landscapes. CoRR, fbs//804.08888, 2018. URL http://azxdv.org/aazs 1804. 08838
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. CoRR, 0bss/882.00866 2018.
Vardan Papyan. Measurements three-level hicrarchical structure in the outliers in the spectrum of deepnet hessians. In ICMI, 2019.

---

Under review as a conference paper at ICLR 2020
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2637-2646. Curran Asso-
ciates, Inc., 2017.
Ben Poole, Subhaneil Lahiri, Maithreyi Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Ex- ponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Sys- Tems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3360-3368, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha fohl-Dickstein. the ex- pressive power of deep neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1847-2854, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL /t/./////o////ooo/oooo/r//////// htmml
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. arXiv e-prints, art. arXiv: 1..iv:1806... 806.08734, June 2018.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. CoRR, abs//611.002233 2016. /ttt::/
01232
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/ 2017
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/161 (11.611...00 1.0550,, 2016.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ADDITIONAL CLASS STIFFNESS MATRIX RESULTS
SinakcWN MRE coochhe0.3 pitr
POUN DESTER
Fon mahunai
ClAA
CLAA
SinoleCH HRSST eooch=1.0
Figure 7: Class-membership dependence of stiffness for a CNN on MNIST different stages of training.
SimpleCWW OPARICD aiintellaaton patra valga zat-qua
Singleckk Chak 10 st epochee) cutrastion
Due
SimpleCA on CIFR-10 apech-33
Ons
Omn
Figure 8: Class-membership dependence of stiffness for CNN CIFAR-10 at 4 different stages of training.
11

---

Under review as a conference paper at ICLR 2020
A.2
ADDITIONAL STIFFNESS AS A FUNCTION OF DATAPOINT SEPARATION RESULTS
FASHION MNS intti trainvval
10
FASHION MNIST
1.01.00 Lit 1i.d trainvval
yal-val
trgincralo
yalival
Same cless
Same
00
1.0 (C Nol
0.5
5.5
0.0
1.0
05
Xx)
1- No, X5)
Xx, Xe)
1- X5.
FC FASHION MNIST - to LR- 1o o 10
FC on FASHION MMST o LB 16 ID trgintrelo
tqui traio
voln
val-val
vrannvol
yalvvl
Same class
5000 3601
Same
0-2
3.0
0.0
0.5
1.0 X21
05
3.5
0.5
0.0
0.5
1.0
06
25
1- 000 X1
1- Xo
1- cool
X2
1-
-o, Xg]
1- Ng
Figure 9: Stiffness between images of the same class function of their input space distance for different stages of training of a fully-connected network (FC) on FASHION MNIST.
CNN CITAR-10 00 UR- 22 trainivll
CNN CITAR-10 Lit 12:0 10
valvvl
traincrralo
trainval
valvvl
Same cless
00
055
3.0
0.0
0.5
1.0
055
5.5
0.0
1.0
0.0
05
Xx)
1- No. X5)
coef Xg)
0o- X,, Xa)
o0d N1.
CNN on CIAR-I0 L- L 82 O 10
CNN on CIFAR-10 o- di LR-12.0 10-1
trann
tain val
yal-val
Ergin-craio
train-vl
yal-vll
Same 3611
Same clacs
1000 3631
60
65
0.0
0.5
1.0
65
10
0.5
0.0
0.5
1.0
0.0
05
3.0
1 000
Xg
xg)
Xg]
1- Xx
1- Xooo
Xg)
Ko.0oos,, X2)
1-
Figure 10: Stiffness between images of the same class as function of their input space distance for 4 different stages of training of a CNN on CIFAR-10.
12

---

Under review as a conference paper at ICLR 2020
Critical length for FC on FASHION MNIST
train-train..
train-val...
valval
Critical Tengsh
1.2
0.6
10
10'
10"
Epach
Epoch
Epach
Critical length for CNN on CIFAR-10 train-val..
train-train
Chtical Tongth x
1.2
1.0
0.8
10
10a
10a
10a
10
10
10a
20
10
10a
10a
Epoch
Epoch
Epoch
Critical length for a FC on FASHION MNIST
train-train
train-val
0.9
0.7
Oritical
0.6
0.4
10
10"
10
10a
10
10
Train
Critical length for CNN on CIFAR-10
train,train
train-val
val-val
Critical longth V
1.4
1.2
1.0
0.6
2--10
10
10a
10a
Train loss
Train
Train
The effect of learning rate on the stiffness dynamical critical scale & the input space distance between images over which stiffness disappears. The upper part shows & for different learning rates as a function of epoch, while the lower part shows it as a function of training loss in order to be able to compare different learning rates fairly. The larger the learning rate, the smaller stiff domains.
Figure 11:
13


Reviews:
[{"review_id": "SyxlE4TRcH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the concept of stiffness: a measure of the change in the loss of sample A due to a gradient step based on sample B. It analyses the expected dynamic for A, B samples from the same and different classes, as well as, samples from the train and test sets.\n\nTo better understand the dynamics of optimization in neural networks is an open and important problem and the paper is clearly motivated in this regard. The proposed method is straight forward and I am not aware of a similar method. \n\nIn addition to that, the paper also introduces \"dynamical critical length ξ\" which is the stiffness of A, B samples based on the cosine similarity of the respective inputs (section 2.4). A linear estimator of when this length becomes 0 is also introduced. Confusingly this is also called the \"dynamical critical length ξ\" in section 4.2. Later on the term \"dynamical scale ξ\" and \"dynamical critical scale ξ\" seem to be used interchangeably. Figure 6 mentions the \"critical length χ\" on the y-axis which seems to be a typo as no such measure was introduced.\n\nThe equivalence between eq. 2 and the two parts of eq. 3 is not obvious. We'd appreciate if the authors would provide a proof of such. \n\nOverall, the paper is written in a simple language but paragraphs remain surprisingly hard to understand. An example of such is e.g. section 4.4: What do the authors mean by \"characteristic distance\" between two input points? What is \"the typical scale of spatial variation\" of a function? etc.\n\nThe paper concludes that:\n\n1.) there is a link between generalization and stiffness\n2.) stiffness decreases with the onset of overfitting\n3.) \"general gradient updates with respect to a member of a class help to improve loss on data points in the same class\"\n4.) \"The pattern breaks when the model starts overfitting to the training set, after\nwhich within-class stiffness eventually reaches 0\"\n5.) This is observed for different models on different datasets\n6.) \"we observed that the farther the datapoints and the higher the epoch of training, the less\nstiffness exists between them on average\"\n7.) \"the higher the learning rate, the smaller the ξ\"\n\nVerdict: Reject\n\nThe conclusions are self-evident. The paper fails to demonstrate the usefulness of stiffness and most results are expected and provide little to no insights into the optimization dynamics of deep neural networks. In fact, the reasoning in this paper is almost tautological (conclusions 1-6).\n\nE.g. if the A, B samples used to compute stiffness are separately drawn from the train and test set then stiffness is a proxy for the difference between the train error and the test error after another gradient step. The authors then compute stiffness at different points of the optimization procedure and conclude that stiffness decreases when the network starts to overfit. Since overfitting is the point in training where train error and test error diverge it is obvious that this can also be observed with regards to \"stiffness\". Hence, the reasoning is circular.\n\nConclusion 7 is slightly different in that it observes that larger learning rates result in smaller ξ which, given the previous paragraph, we can rewrite into the statement \"larger learning rates generalise better\". This is a well known empirical observation and has been discussed thoroughly (e.g. on connection with flat and sharp minima or learning rate decay schedules). \n\nDisclaimer: This review was done on short notice. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xhRSnvsB", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces “stiffness”, a new metric to characterize generalization in neural networks. Stiffness is a pretty simple concept and is relatively straightforward to compute. The authors evaluate this metric on standard datasets using two relatively small neural networks. On the whole, the paper is written clearly and explains its methodology in simple language.\n\nI have a few observations:\n1. The equivalence between equation 2 and equation 3 is mentioned in passing but no explanation is provided. Th equivalence is not clear so I would encourage the authors to provide a short proof.\n2.  Since stiffness depends on the gradients obtained on points in the input space, which in turn depends on the loss, why would a practitioner training a neural network turn to stiffness to diagnose overfitting instead of just looking at the values of the training and validation losses? Indeed, the authors themselves say that a network has overfitted when training and validation losses diverge. The paper fails to motivate why stiffness is better than just looking at losses during training. \n3. The authors mention “The train-val stiffness is directly related to generalization, as it corresponds to the amount of improvement on the training set transferring to the improvement of the validation set. ”. Typically, generalization is evaluated on a held out test set so I fail to understand what the authors mean by this statement. We would expect validation error to underestimate test error  so while they are related, train-val stiffness would not necessarily characterize generalization. It would be interesting to see a train-test stiffness graph to test the authors claim.\n4. The paper fails to motivate the the utility of the concept of “Dynamical Critical distance”. Since the primary goal of  paper is to understand generalization, I would like the authors to clarify the motivation to study this quantity. What additional insight does this provide with respect to generalization?\n5. The term “dynamical critical distance” is not used uniformly. For example, it is mentioned as “dynamical critical scale” in section 3.3 and “dynamical critical length” in section 4.2.\n6. While the paper on the whole is written in a clear fashion, I found section 4.4 to be particularly confusing. The authors should consider rewriting that section to make it clearer.\n\nIn summary, the concept of stiffness seems to closely follow training and validation losses and any problem diagnosed using stiffness would therefore be also diagnosed via examining the loss values. This along with other concerns mentioned above mean that I cannot recommend this paper for publication.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgSdPqotS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission introduces a metric, termed stiffness, to evaluate the generalization capability of neural networks. The metric is novel and straightforward, it measures how stiff a network is by looking at how a small gradient step on one example affects the loss on another example. The authors study several configurations  on three small datasets. They demonstrate that stiffness is a useful concept for diagnosing and characterizing generalization. \n\nI give an initial rating of weak accept because (1) The paper is well motivated and well written. Studying generalization is important for neural networks. (2) It seems from experiments that stiffness is a useful metric to indicate models' generalization capability. However, I have a few concerns. \n\nFirst, the authors study several configurations like train-train, train-val and val-val. However, these configurations are still in-domain analysis, the data distribution is quite similar. It can not support author's claims well. Adding an experiment where domain gap is large will make the submission stronger, such as train-test, cross-dataset or challenging tasks like semantic segmentation. \n\nSecond, the datasets being used are very small. I understand that for theoretically analysis, small datasets are quick to converge and easy to demonstrate. However, this submission focuses on generalization problem during transfer learning. Hence, it needs at least a bigger dataset, like ImageNet, to show it really works. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was some support for the ideas presented, the majority of reviewers felt that this submission is not ready for publication at ICLR in its present form.

Concerns raised include lack of sufficient motivation for the approach, and problems with clarity of the exposition.
Decision: Reject