Extreme Memorization via Scale of Initialization
Authors: Harsh Mehta
Ashok Cutkosky
Behnam Neyshabur
Venue: ICLR 2021 Poster (2021)
Keywords: Scale of initialization
Memorization
Overfitting
Generalization
Generalization Measure
Understanding Deep Learning

Abstract: We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which generalization ability is affected depends on the activation and loss function used, with sin activation being the most extreme. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function. Our empirical investigation reveals that increasing the scale of initialization correlates with misalignment of representations and gradients across examples in the same class. This insight allows us to device an alignment measure over gradients and representations which can capture this phenomenon. We demonstrate that our alignment measure correlates with generalization of deep models trained on image classification tasks.

Published as a conference paper at ICLR 2021
EXTREME MEMORIZATION VIA SCALE OF INITIALIZA- TION
Harsh Mehta Google Research ha rrshmggoogle.coom com
Ashok Cutkosky Boston University
Behnam Neyshabur
Blueshift, Alphabet
ashok@cutkosky..oo
neyshaburrggoggee.com
ABSTRACT
We construct experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which generalization ability is affected depends the activation and loss function used, with sin activation demonstrating extreme memorization. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function. Our empirical investigation reveals that increasing the scale of initialization correlates with misalignment of representations and gradients across examples in the same class. This insight allows us to devise an alignment measure over gradients and representations which can capture this phenomenon. We demonstrate that our alignment measure correlates with generalization of deep models trained on image classification tasks.
INTRODUCTION
Training highly overparametrized deep neural nets large datasets has been very successful modern recipe for building machine learning systems. As a result, there has been a significant interest in explaining some of the counter-intuitive behaviors seen in practice, with the end-goal of further empirical success.
One such counter-intuitio trend is that the number of parameters in models being trained have increased considerably over time, and yet these models continue to increase in accuracy without loss be observed even after the point where
of generalization performance. In practice, improvements the number of parameters far exceed the number of examples in the dataset, i.c., when the network is overparametrizeed (Zhang al., 2016 Arpit al. 2017). These wildly over over-parame parammeetttee networks avoid overfitting even without explicit regularization techniques such weight decay dropout, suggesting that the training procedure (usually SGD) has an implicit bias which encourages the net to generalize (Caruana et al., 2000; Neyshabur et al. 2014 2019 Belkin et al. 2018a Soudry et al. 2018
Contributions In order to understand the interplay between training and generalization, we in- vestigate situations which the network can be made induce scenario in which the accuracy on the test set drops to random chance while maintaining perfect accuracy on the training set. We refer to this behavior as extreme memorization, distinguished from the more general category of memorization where either test set performance is higher than random chance or the net fails to attain perfect training set accuracy. In this paper, we examine the effect of scale of initialization on generalization performance of SGD. We found that it is possible to construct experimental setup in which simply changing the scale of the initial weights allows for a continuum of generalization ability, from very little overfitting to perfectly memorizing the training set. It is our hope that these observations provide fodder for further advancements in both theoretical and empirical understanding of generalization.
'The code used for experiments is open-sourced https: //g thub. researchh google wtt.-reeeccttteee///ettteeeeeeee reme_memor zation

---

Published as a conference paper at ICLR 2021
CIEAR-10
ca
AocuraCh
0.4
Test Accuracy
Alignment
e.
CusoNat
0.3
16:1 TeUl
0.7
0.8
MLP
DenseNes
0.2
lel
0.5
RosNett 50
0.
0.0
20
60
100
0.g
20
60
00
1do
120
140
1K training steps
Alignment.
1k training steps
Test accuracy
(b) Changing scale of initialization
(c) Alignment on deep models
Figure 1: (a) Results when using sin activation function in 2-layer MI.. We initialize the first layer using random normal distribution with mean zero and vary the standard deviation shown in the plots. Initialization scheme the top layer is kept unchanged and uses glorot uniform initializer (Glorot Bengio 2010). The plot shows the drastic changes in generalization ability solely due the changes in scaling on CIFAR-IO dataset. Plot (b) shows the correlation between best test accuracy and gradient alignment values across different datasets, CIFAR-10 Krizhevsky 2009) CIFAR- 100 and SVHN illustrates that the alignment measure can also capture differences generalization across model architectures. Note that, order to fair comparison, all hyperparameters (e.g. learning rate, optimizer) are kept constant.
change scale of initialization. Finally, plot (c)
We construct two-layer feed forward network using sin activation and observe that in- creasing the scale of initialization of the first layer has strong effect on the implicit regularization induced by SGD, approaching extreme memorization of the training scale is increased. We observe this phenomenon on different image classification datasets: CIFAR-10, CIFAR-100 and SVHN.
the
For the popular ReLU activation, one might expect that changing the scale should not affect the predictions of network, due to its homogeneity property. Nevertheless, even with ReLU activation we see a similar drop in generalization performance. We demonstrate that generalization behavior can be attributed further up in the network to variety of common loss functions (softmax cross cross-entrope entropy, hinge and squared loss).
Gaining insight from these phenomena, we devise an empirical "gradient alignment" measure which quantifies the agreement between gradients for examples corresponding a class. We observe that this measure correlates well with the generalization performance as the scale of initialization increased. Moreover, we formulate a similar notion for representations as well.
Finally, provide evidence that alignment measure is able capture generalization performance across architectural differences of deep models on image classification tasks.
2 RELATED WORK
Understanding the generalization performance of neural networks is a topic of widespread interest. While overparametrized nets generalize well when trained via SGD real datasets, they can just as easily fit the training data when the labels are completely shuffled (Zhang et al. 2016). In fact, Belkin al. (2018b) show that the perfect overfitting phenomenon seen in deep nets can also be observed in kernel methods. Further studies like Neyshabur et al. (217); Arpit et al. (2017) expose the qualitative differences between
trained with real vs random data. Generalization performance has been shown to depend on many factors including model family, number of parameters, learning rate schedule, explicit regularization techniques, batch size, (Keskar et al.) 2016 Wilson et al., 2017). Xiao et al. 2019) further characterize regions of hyperparameter spaces where the net memorizes the training set but fails to generalize completely.
Interestingly, there has been recent work showing that werr-araametrization aids not just with generalization but optimization too (Du et al., 2019 2018 Allen-Zhu et al.) 2018 Zou et al.) 2019). Du et al. (2018) show that for sufficiently gradients induced by ReLU activation remains positive definite throughout training due to parameters staying close to initialization. Moreover, in the infinite width limit the network behaves like its linearized version of the same net around initialization (Lee et al. 2019). Jacot et al. (2018) explicitly characterize the solution obtained by SGD in terms of Neural Tangent Kernel which, in the infinite
peraaeereetee nets, the gram matrix of the
2

---

Published as a conference paper at ICLR 2021
width limit, stays fixed through the training iterations and deterministic at initialization. Finally, Frankle & Carbin (2018); Frankle et al. (2019) hypothesize that overparametrized neural nets contain much smaller sub-networks, called "lottery tickets", which when trained in isolation can match the performance of the original net.
From optimization standpoint, several initialization schemes have been proposed in order to facilitate neural network training (Glorot & Bengio [2010) He et al. 2015a). Balduzzi et al. (2017) identify the shattered gradient problem where the correlation between gradients w.r.t the input in feedforward networks decays exponentially with depth. They further introduce looks linear initialization in order to prevent shattering. Recent work explores some intriguing behavior induced by changing just the scaling of the net at initialization. Building on observation made by others (Li & Liang 2018 8 Du et al., 2019 2018 Zou et al. 2019 Allen-Zhu et al. 2018). Chizat Bach (2018) formally ly introduce the notion of lazy training, phenomenon in which an over wver-parametrized net can converge to zero training loss even as parameters barely change. [Chizat & Bach (2018) further observe that any model can be pushed to this regime by scaling the initialization by certain factor, assuming the output is close to zero at initialization. Moreover, Woodworth et al. (2020) expand on how scale of initialization acts regimes, called the kernel and rich regimes. In the kernel regime, the behavior of the net is equivalent to learning using kernel methods, while in the rich regime, gradient descent shows richer inductive biases which are not captured by RKHS norms. In practice, the transition from rich regime to kernel regime also comes with a drop in generalization performance. Geiger al. (2019) further explore the interplay between hidden layer size and scale of initialization in disentangling both regimes, specifically that the scale of initialization, which separates kernel and rich regime, is function of the hidden size.
controlling quantity for transitioning between two very different
On somewhat orthogonal direction, from a theoretical perspective, several studies attempt to bound the generalization eтror of the network based on VC-dimension (Vapnik 1971, sharpness
based measures such as PAC-Bayes bounds (McAllester) 1999: Dziugaite Roy 2017 Neyshabur et al. (017), or norms of the weights (Bartlett 1998] Neyshabur et al. 2015b Bartlett et al.) 2017]
Neyshabur et al. 2019 Golowich et al., 2019). Further works explore generalization from an empirical standpoint such sharpness based measures (Keskar et al. 2016), path norm (Neyshabur
ctt al. 2015a) and Fisher-Raa Rao metric (Liang et al. (017). A few have also emphasized the role of distance from initialization in capturing generalization behavior (Dziugaite & Roy 2017 Nagarajan Long & Sedghi 2019).
& Kolter 2019 Neyshabur et al. 2019
Li & Liang 2018) study 2-layer ReLU net and points out that final learned weights accumulated gradients added to the random initialization and these accumulated gradients have low rank when trained structured datasets. |Wii & Ma (2019) obtain tighter bounds by considering data-dependent properties of the network such as norm of the Jacobians of each layer with respect to the previous layers. More recently. Chatterjee (2020) hypothesize that similar examples lead similar gradients, reinforcing each other in making the the overall gradient stronger in these directions and biasing the net make changes in parameters which benefit multiple examples.
EXTREME MEMORIZATION
In this section, we discuss the experimental setup which leads extreme memorization due to increase in scale of initialization. To reiterate, we refer to the scenario where the net obtains perfect training accuracy but random chance performance on the test set as extreme memorization.
In order to investigate this in the simplest setup possible, we consider a 2-layer feed-forward network trained using stochastic gradient descent (SGD):
(अ)) (Wix)
where è is the chosen activation function, x E RP, W, e Riop, W2 E Rexh and z E R* is the output of the net. The aim is to find parameters [Wi, W2] which minimizes the empirical loss L. == n Ex (xx(x;).)) given i.i.d draws of n data points {(x,y,)) from some unknown joint distribution over x E R and y E RR We focus on multi-class classification problems, in which each y is restricted to be one of the standard basis vectors in R². We use the notations E, = ((z(x)),))) and r; = of ()) is shorthand for the hidden layer representation for input x,. Also, for any E (1, (1,...,))) k)), we use the shorthand y = c say that y is the eth standard basis vector.
3

---

Published as a conference paper at ICLR 2021
In experiments, choose large hidden size that the net is very over wer-parameterizer and always gets perfect accuracy on the training set. Also, since we are only interested in studying the implicit regularization induced by SGD, we do not use explicit regularizers like weight decay, dropout, etc. More details on the exact setup, datasets used and hyper-parameter are in the appendix.
3.1
SIN ACTIVATION
Relative Dorms
Gred Aligrmment
Besp Alignment
Accuraty
20
X
12
1k training steps
x 1k training steps
1k training stegs
x 1k training steps
(a) Training accuracy
(b) Relative norms
(c) Gradient alignment
(d) Rep alignment
Figure 2: Results when using sin activation function in a 2-layer MLP applied on CIFAR- 10 dataset Krizhevsky
2009). We initialize W1 using random normal distribution with mean and vary the standard deviation shown in the plots. The initialization scheme for W2 kept unchanged, using a Glorot uniform initializer
GGlorot & Bengio 2010). (a) shows the the evolution and rate of attaining perfect training accuracy. (h) plots the norm of the gradients of Wi over norm of Wi. As clucidated in (Chizat & Bach) 2018). increasing the scale initialization leads to gradients being increasingly smaller than the weights and thus weights not being able move very far from initialization. (c) shows how example gradient alignment can capture differences generalization ability case of sin activation as the scale of initialization is increased. Plot (d) shows that representation alignment also able to discriminate generalization ability induced at high scale of initialization. We obtain similar results on CIFAR- 100 and SVHN datascts as well, which are included in the appendix.
As shown in Figure setting e to sin function results a degradation of generalization performance to the point of extreme memorization just by increasing the scale of initialization of the hidden layer W. Intuitively, when using sin activations, if W, remains close to its initial value, then a single hidden layer can be approximated by kernel machine with specific shift-invariant kernel K, where K determined by the initializing distribution (Rahimi Recht 2008). For example, when the initializing distribution is a Gaussian with standard deviation σ, K IS a Gaussian kernel with width 1/a. Formally, consider a network architecture of =(x) = W2 + b), where W1 is a matrix whose entries are initialized via a Gaussian distribution with variance a² and b e IR is a bias vector whose coordinates are initialized uniformly from [0, 2T). Then (Rahimi & Recht 2008) showed
[00llx (-
x'112
IL Ro(Wix + b), 6(Wi2' + b)}] x exp
2
Thus, when holding Wi and b fixed, the network approximates a kernel machine with a Gaussian kernel whose width decreases W1 is scaled up (which corresponds increasing the variance parameter in its initialization). In this scenario, one expects that the classifier will obtain near-perfect accuracy on the train data, but have no signal elsewhere because all points are nearly orthogonal in the kernel space. We did not specify bias vector in our architecture, but intuitively one expects similar behavior. In fact, we have the following analogous observation (proved in Appendix
Theorem 1. Suppose each entry of Wi is initialized via a Gaussian with mean O and variance o². Then for any and x', we have
(_o²11z
2'1
(o(W2)) o(Waa')) (X] S h hexp
2
This suggests that for large enough a, the vectors (dW1z) will be nearly uncorrelated in expectation at initialization. Further, for any loss function and label y, we have that the columns of Ww, (=(()) y) proportional to (()))) and so these gradients should also display lack of correlation as increases. We argue that this lack of correlation leads to memorization behavior, By memorization, mean that our trained model will have wear-perfect accuracy on the training set, while having very low or even near-random performance on the testing set, indicating that the model has "memorized" the training set without learning anything about the testing set.
To gain some intuition for why we might expect poor correlation among features or gradients to produce memorization, let us take look an extreme case where the gradients for all the examples
4

---

Published as a conference paper at ICLR 2021
orthogonal to each other. More concretely, suppose the true data distribution is such that for all independent samples (01, 3/1 ), (I2, y2) with (x1, V1) * (22, V2), we have (V&1, Vl2) < e for all W1. W2 for some small e. Then we should expect that taking a gradient step along any given example gradient should have negligible (()) effect on the loss for any other example. As a result, the final trained model may achieve very small loss the training set, but should learn essentially nothing about the test set it will be perfectly memorizing model.
3.2 MEASURING ALIGNMENT
Motivated by this orthogonality intuition, we wish develop an empirical metric that can measure the degree to which training points are well-aligned with each other. We begin with a review of related metries in the existing literature and suggest improvements order better capture generalization.
0.24
1.0
20
1.0
0.9
1.0
9900
0.5
0.08
0.16
0.5
0.5
0.6
0.5
600
300
0.0
0.00
0.0
0.0
0.0
0.5
-0.16
-0.00
-0.5
2
-0.5
-0.5
600
300
-1.0
-0.24
-1.
10
0
10
-1010
0
10
-0.9
-1.0
10
-900
log(toll/ ID
(a) Alignment (Ours)
(b) Diversity
(c) Stiffness
(d) Confusion
Figure 3: Comparing different gradient-based measures for the simple case of having two samples from the same class where a = V4, and b Vf2.
Related statistics Other relevant gradient-based based measures have been suggested for understanding optimization or generalization. One such measure Gradient diversity (Yin et al. 2018), that quantifies the extent to which individual gradients are orthogonal to each other and is defined as E ||2/1 Σn= 1e. Unfortunately, Gradient Diversity did not correlate with generalization in our experiments in Section Moreover, as shown in Figure 3 Gradient Diversity is most sensitive when the cosine of the angle between two gradient highly negative, a scenario which is rare in high dimensional spaces. Furthermore, this notation does not take the class information into account and treats all pairs of samples equally. Cosine Gradient Stiffness (Fort et al. 2019) is another measure to capture the similarity of gradients and can be calculated Eeze) cos VL, VG,), Fort et al. (2019) also define a modified version of Cosine Gradient Stiffness that allows this calculation within classes. Although it measures quantity which is close to what want, shown in Figure 3 this measure is invariant to the scale of the gradient. That means that samples with very small gradients would be weighted much as samples with large gradients, thus discarding valuable information. Finally, we also consider Gradient Confusion (Sankararaman et al.) 2019), which can be calculated as min (,,, Vl,). We note that, as shown in Figure 3] gradient confusion is sensitive the norm of gradients and is most affected by the ratio of the norms. Also, similar to Gradient Diversity, this measure does not take the class information into account.
With these observations mind, we formulate our measure of alignment 5 between gradient vectors and compare it with other measures in Figure Note that we normalize our alignment measure by the mean gradient norm in order to avoid discarding magnitude information from individual gradients:
Ω KVe, V
(V4,v))) Assuming n vectors, we can further expand since Eg4 (VC, v(,) m(n-1) and E|| [ Σg= |v
n Eisg 444, VG,) Ω = (n- 1) |),
(3)
Efficient computation of alignment Note that Eij; (V, Vl,) may appear to require O(n²) time to compute, but in fact it can be computed in (n) time by reformulating the expression as:
Σ (VE, Ve,)
Σ Ve
|vee, 2
(4)
5

---

Published as a conference paper at ICLR 2021
For comparison, Gradient Diversity also can be computed in (n)) time but Cosine Gradient Stiffness and Gradient Confusion appears to require (n time.
Alignment within a class In order to compute alignment for a specific class, we propose selecting the examples of class and compute the alignment for that subset. More specifically, we formulate specific alignment for each class -,....;
,...,,,, k, follows:
noe i ( VVe, Ve)
=c]
(n. DE [V c])²
where nee is the number of training examples with label y and 11 [r] is the indicator of the proposition p- it is one if p is true and zero otherwise. We further take the mean of Q. over all classes for overall view of how in-class alignment behaves.
Onn class
(6)
As shown Figure [ -la-ss correlates well with generalization ability of the net when scale of initialization is increased. All of our gradient alignment plots report the average in-class alignment -
0.03
Accuracy
3:55
Aligmment:
Alignment
8:40
16]
0.33
1e-1
9.35
1633
162
1c1
035
-0.0
1k training steps
x 1k training steps
k training steps
(a) Test accuracy
(b) Representation alignment
(c) Gradient alignment
Figure 4: Results when using RcLU activation in 2-layer MLP with Softmax cross-entropy loss function when trained on CIFAR-10 dataset. Similar to Figure 2 W1 is initialized with random normal distribution with mcan zero and varying standard deviation scale a as shown in the plots. (a) shows how the test accuracy drops and saturates as is increased. (c) shows how gradients start to show misalignment as the scale is increased. (b) shows similar misalignment trend for hidden layer representations. Note that, in contrast the extreme memorization phenomenon we observed for sin activation, here we observe more limited decrease in both generalization performance and alignment. Similar results on CIFAR-10D SVHN and additional plots for CIFAR-I0 with all the loss functions discussed in Section 3.2 can be found in the appendix.
Representation Alignment Since gradients are the sole contributor to changes in the the weights of the net, they play a crucial part in capturing generalization performance. However, calculating the gradient for every example in the batch can incur a significant compute and memory overhead. Fortunately, the gradients for W2 are functions of the intermediate representations Considering example representations instead of example gradients has practical advantage that representations can be obtained for free with the forward pass. Also, representation alignment, defined below as Ω",, at any training step, accounts for the cumulative changes made by the gradients since the beginning of the training whereas gradient alignment only accounts for the current step. We show comparison with gradient alignment in Figure 2 For completeness, gradient and representation alignment for all the experiments where useful do so. We formulate representation alignment in the same way we formulated gradient alignment below.
((W,r)) (x;).
provide plots for both
Eiey (Ts, [y+ = = c]
S class
2 := (n. ( 1) Ir.l c])²
4 WHY SHOULD THE SCALING AFFECT HOMOGENEOUS ACTIVATIONS ?
For sin activations, extreme memorization phenomenon may be explainable through the lens of random Fourier features and kernel machines, which suggests that large initialization leads to very poorly aligned examples. In this Section, investigate what happens when use more typical

---

Published as a conference paper at ICLR 2021
activations such as ReLU. We find that even for ReLU, increasing the scale of the initialization leads to a drop in generalization performance, and a similar downward movement in alignment as the initialization scale increases (see Figure 4. This might feel counter- is
the ReLU activation which means that scale factors from the input can be pulled out of the function altogether and it only changes the scale of the output. However, this does not take into account the effect of the loss function L, which is typically not homogeneous. We study 3 commonly used loss functions, namely softmax cross-entropy, multi-class hinge loss and squared loss, and show their effect on gradients when weights are close to their initialization. The result we present for ReLU holds for linear activation too. Even though with linear activations we don't expect perfect training accuracy, we do see the same trend in alignment measures and the drops in generalization performance that goes with it. Due to space constraints, we refer the reader to appendix Section [F for the plots.
As shown in F increasing the scale of initialization also leads to the scale of the gradients being much smaller than the scale of the parameters at initialization (Chizat & Bach 2018 Woodworth cct al. 2020). Thus it was high enough in the beginning, SGD should not be able to correct the scale of the weights during the course of the training.
Softmax cross entropy Typically, the softmax layer consists of a weight vector st for every class, which is used to compute the logits 2;. These logits then are used to compute the probability pt for each class using the softmax function g: R* + Rt:
P: (=)===
E, e5/T for i I....,k k and (ai...., zx) E R*
Assuming T is 1, which is typically the case, the derivative of the loss with respect to 2; is da. (L(((()),)) y) = P: yi where is the negative log-likelihood and g(z) = (PI...., is the Softmax function. Let us consider the limiting behavior of this gradient when we increase the seale of the network, which causes the values to become arbitrarily high. In this case, all the ps except the one corresponding to the largest 2 value become zero, so that the gradient is 0 if the prediction is correct, and otherwise is -1 in the coordinate of the correct class and 1 in the coordinate of the predicted class. Contrasting this with the case where the scale of the network is arbitrarily close to O, the gradient in the coordinate of the correct class will be 1/k 1.0 and 1 in the incorrect class coordinates, so that all the gradients are the same and the alignment is 1, which is the maximum possible alignment. Therefore the gradients with respect the logits will on average be more orthogonal in the former case. Since the gradients for parameters will be multiplied by gradient with respect to the logits due to the chain rule, they will be more orthogonal as well. We corroborate this intuition with empirical evidence as shown in Figure
In practice, since the initialization scheme chosen carefully, weight scaling is less of concern in the beginning, but it can become an issue during the course of the training if the magnitude of the weights starts to increase. In either scenario, one simple strategy to counteract the effect of scaling of the net is to increase the temperature term T with it such that magnitude of the input to the Softmax can stay the same and consequently there will be no relative change to alignment in the gradients coming from the loss function. Moreover, this observation also brings some clarity into why tuning hyper-parameters parameters that affects the scale of the network is sometimes helpful in practice, either by explicitly tuning the temperature term or appling weight decay which favors parameters of low norm and implicitly controls the scale of the network throughout the training run.
hyper-
The arguments made for Softmax can also be adapted Sigmoid for binary classification. Moreover, since Sigmoid is also used occasionally as an activation function, it is valuable to see how it behaves with changes in scale of initialization in that capacity. We do in fact observe similar degradation in generalization performance, although in this case, there is an extra complication that increasing the scale of the input to Sigmoid also affects the training accuracy since gradients for the hidden layer starts to saturate beyond a certain scale. More details on this can be found in the appendix.
Hinge loss is defined by: ((2,g) = Σ max(0 A+
(8)
where A is the target margin. In practice A is typically set to 1.0. However, if the network outputs scaled by factor of a, this will have the same effect as scaling the margin to be and then
7

---

Published as a conference paper at ICLR 2021
scaling the loss by a: Eiey max(0, A + az, - azy) a Eiff max(O, 4/a + 2, -2y) With this in mind, let us calculate the gradient:
de dz. 1
i z 1 1(A + 2, - 2v > 20) i i
- By
is instructive to take look at what happens when the effective margin is arbitrarily close zero. At initialization, we can treat each 1(z; Ey > 0) as independently 0 or uniformly at random, so we can expect half of the gradient coordinates for incorrect classes to be On the other extreme, the effective margin becomes large 1(4 + 2, Zy > 0) will always be 1, and the gradients for all incorrect classes will be 1, Again, the latter case will lead to the maximum alignment value of 1, so that the gradients more aligned across examples.
In this case, misalignment can be fixed by scaling the margin A with the scale of the network. Intuitively, want to change the loss function such that the scale factor can be pulled out of the loss entirely so that scaling of the loss by a constant doesn't It change the minimizer.
Myy ApPu Stattod
Acnuracy
Algnment
Accuracy
Alggmment
HLF Rocia 1g
Souffled lobcl
Condies
1K training steps
1k training steps
1k training
1k training steps
(a) Test accuracy
(b) Alignment
(c) Test accuracy
(d) Alignment
Figure 5: Plots (a) and (b) shows how representation alignment increases with generalization performance as the architecture is improved from 2-layer MLP with ReLU activation to a ConvNet architecture (exact details in the appendix) on CIFAR-10 datasct. We see further increase when even bigger and widely used architectures like 5e and DenseNet-122 121 are employed the same task. Note that keep all the hyperparameters same across architectures in this experiment. Plots (c) and (d) elucidates the drop in representation alignment when the labels are shuffled the case 2-layer MLP.
Squared loss is defined as v) (== 1) y and de 1 in the coordinate of the correct class. In the extreme case where scale of the net close to 0, will also be close to zero the y term will dominate in gradients for all the examples. On the other hand, when the scale is high, the z term dominates. Again, since y is a constant in our training and will essentially be a random vector at initialization, to be more aligned when the scale of the network is lower. Similar to the argument presented for hinge loss, the effect of scaling on generalization performance in this case can be fixed by scaling the one-hot vector y appropriately with it so that the scale factor can be pulled out of the loss function.
de = y. where y is one hot vector with
can expect the gradients across examples
IS ALIGNMENT RELEVANT MORE BROADLY?
In this Section, we explore whether the alignment metric is useful in capturing generalization performance more generally. More specifically, is alignment relevant when we make changes to architecture data distribution rather than the initialization scheme? We provide empirical evidence which suggests an optimistic answer.
Introduction of new architecture changes has been very successful recipe in advancing performance of deep learning models. In the task of image recognition, addition of convolutional layers and pooling layers (Lecun et al., 1998) and, more recently, residual layers (He al. 2015b 2016) have caused significant jumps in generalization performance. Moreover, several theoretical studies show how convolution and pooling operations can significantly affect the implicit bias of SGD, in favor of better generalization in the image domain (Cohen & Shashua, 2016 Gunasekar et al. 2018). In Figure 5, we investigate the architectural change of extending our standard 2-layer MLP with preceding convolutional layers. Unsurprisingly, performance. Moreover, the plots show that the addition of convolutional layers leads the last layer representations to be significantly more aligned, suggesting that these architecture changes cause the
observe substantial improvement in generalization
8

---

Published as a conference paper at ICLR 2021
net to discard irrelevant variations in the input more effectively across examples and ultimately leads to better generalization. Finally, we experiment with popular large scale image recognition models like ResNet-50 and DenseNet-121 (Huang et al. 2017) and also observe a similar trend.
Another way to impact generalization performance is to shuffle the labels in the training set (Zhagg al., 2016; Arpit et al., 2017). If completely shuffle the labels in the dataset, don't 't expect the model to generalize on the test set at all, i.e., random chance performance. S shows that shuffling the labels also leads to a drop in representation alignment.
CONCLUSION AND FUTURE WORK
In this work, investigated how increasing the scale of initialization can degrade the generalization ability of neural nets. We observed extreme case of this phenomenon in the case of sin activation, making it particularly interesting given a recent rise in the use of sin activation in practical setting (Sitzmann et al. 2020; Tancik al. 2020), since our work shows how sensitive the generalization performance can be to scale of ization in those case. This phenomena is also quite conspicuous with more popular activations like ReLU and Sigmoid. In the case of ReLU, the drop in generalization performance can be attributed to the loss function since the rest of the net unaffected by scaling due homogeneity, We complement these observations by defining an alignment measure that correlates empirically well with generalization in a variety of settings, including the inductive bias introduced by architectural changes like convolution layers, indicating its broader importance.
argue that
Our formulation of alignment measure suggests some intriguing avenues for future research. For example, as shown in Figure I even though our experiments suggest that low scale of initialization leads to increased representational alignment, there seems to be sweet spot below which its affect on generalization ability no longer holds true. Exploring generalization behavior in this case of ultra-low scale of initialization also an interesting direction of future research.
7 ACKNOWLEDGEMENTS
We are grateful to Eugene Ic, Walid Krichene and Jascha Sohl-dickstein for reading earlier drafts of this paper and providing valuable feedback.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterizaaion, 2018
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Y Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. 06 2017.
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what is the question? In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 342-350, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:
//proceedings m.r. www.sresspppe http////0///aaaaaaaa htm.mmmmmm.mmm.m....mmmmm..mm...
The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Transactions on Information Theory, 44(2):525-536, 1998.
L. Bartlett.
Peter I. Bartlett, Dylan Foster, and Matus J Telgarsky. Spectrally-normalizeed margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp. 6240-6249, 2017.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off, 2018a.

---

Published as a conference paper at ICLR 2021
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning need understand kernel learning. 02 2018b.
Rich Caruana, Steve Lawrence, and Lee Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'00, pp. 381-387, Cambridge, MA, USA, 2000. MIT Press.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization, 2020.
Lénaïi Chizat and Francis Bach. A note on lazy training supervised differentiable programming. ArXiv, abs/18/ 0tt//812..000 12.07955, 2018.
Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry, 2016.
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks, 2018.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes parameterizze neural networks. In International Conference on Learning Representations, 2019.
n wwtpeeeoeeeeeeeeeeeeeeeeeeeeeeee for.mp1-=S11e310000
Gintare Karolina Dziugaite and Daniel Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 1703. 1008, 2017.
Stanislav Fort, Pawel Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A perspective generalization in neural networks, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis. Training prumed neural networks. CoRR, abs/ 100.00333,, 2018. /tt.tpp://axxx/..00/// wttp://v.rggga///////o//////// 1803. 03635
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis scale. arXiv preprint (Xiv::9903001111 8, 2019.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy learning in deep neural networks: empirical study. ArXiv, abs/1996.08033, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Information and Inference; A Journal of the IMA, May 2019. ISSN 2049-8764. doi: 10. 10.10093imaiaa/iz2007. htRl:/ dwi.dooiooooooooo www.ddi.....o.o..
ima iiwiiiaaiiiiiiiiiiiiiiiiiiiiiiii aaiaaiiiii000
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Implicit bias of gradient descent on linear convolutional networks, 2018.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 201 doi: 10. 10.1109/icw.22015.22. URL http: /x // doi
erg/10.1100 www.:/CcV.... 2015 123
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10. 10%/vvpr.2016.90. URL http::///dk......rr//1.1....
cvpr 2016 90
10

---

Published as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1 10.1109/cyp:.211..25. URL http::/ct.......g////////o/ wtt.doi..rg//1..11033 /x
CVPR. 2017.20
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 100 (canadian institute for advanced research).
html
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8572-8583. Curran Associates, Inc., 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data, 2018.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks, 2017.
Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks, 2019.
David A. McAllester. Pac-bayesian model averaging. In Proceedings of the Twelfth Annual Confer- ence on Computational Learning Theory, COLT '99, pp. 164-170, New York, NY, USA, 1999. Association for Computing Machinery. ISBN 1581131674. doi: 10.114533774000307433. URL
/tttps://doi.rg//// 111.1....115330 10500/00001100770003000000 307435
Vaishnavh Nagarajan and J. Zico Kolter. Generalization in deep networks: The role of distance from initialization, 2019.
Yuval Netzer, Tiejie Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning, 2014.
Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: nath-normalized optimiza- tion in deep neural networks, 2015a.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015b.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring general- ization in deep learning, 2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of
pocerearametiiztiion in generalization of neural networks. In International Conference on Learning Representations, 2019, URL /tttttts///peerreeeeee http://wwwww.......oeo.oooaaaaaa
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, pp. 1177-1184, 184, 2008.
11

---

Published as a conference paper at ICLR 2021
Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization on gradient confusion and stochastic gradient descent. ArXiv, abs//104.069000 2019.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In arXiv, 2020.
Daniel Soudry, Elad Hoffer, and Nathan Srebro. The implicit bias of gradient descent on separable data. In International Conference on Learning Representations, 2018. URL /ttpps////peeereeleee net / Eorum ?idmrlg g/
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint 2rXiv:2006.107399 2020.
Vladimir Vapnik. Chervonenkis: On the uniform convergence relative frequencies of events to their probabilities. 1971.
Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation, 2019.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4148-4158. Curran Associates, Inc., 2017.
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models, 2020.
Lechao Xiao, Jeffrey Pennington, and Samuel S. Schoenholz. Disentangling trainability and general- ization deep learning. ArXiv, 13000333 2019.
Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: key ingredient for scalable distributed learning. In Amos Storkey and Fernando Perez-Cruz (eds.). Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research,
pp. 2098--007, Playa Blanca, Lanzarote, Canary Islands, 09-11 Apr 2018. PMLR. URLhttp
//proceedings. ml wttps/p///s...eeeeeee
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/161 1.03530, 2016. |
//axxiv.or//as wttp://o/abbballl.1l 03530
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109())467-492, Oct 2019. ISSN 1573-0565. doi: 1010007/10994-09--0000
URL http::/://d...ollooo///////////o
e10994-019-05839-8
12

---

Published as a conference paper at ICLR 2021
A ORGANIZATION OF THE APPENDIX
We describe the training procedure and datasets used throughout the paper in detail in appendix C
In appendix D we reproduce the extreme memorization phenomenon from Figure 2 on
CIFAR-100 and SVHN.
Fig shows how changing the scale of activation leads to drop in gener- alization performance in the case of ReLU activation and softmax cross-entropy loss. In appendix
In Section 3.2
B we include results with multi-class hinge and squared losses.
We include results when using linear activation with softmax cross-entropy loss in appendix
In appendix G we discuss how Sigmoid function, when used as activation, responds to scaling of initialization.
Appendix includes details on the exact architecture and hyperparameters used in Section
B PROOF OF THEOREM
In this section we provide the missing proof of Theorem Theorem 1. Suppose each eniry of Wi is initialized via a Gaussian with mean O and variance a². Then for any and x', we have
| restated below:
((o(W2)) o(W12 s hexp (- (_o2l0x =²1
2
Proof. Since each individual row of W1 is independent, it suffices to prove the statement for h (a'_z) and let A x cr. Notice If x 0 the statement is trivially true, suppose +f O. Let e T
that (A, ..) and || S || ar-=al) x/|.. We also have
Wix' cWiæ WAA
Notice that W, x is normally distributed with mean 0 and variance a² |=]1= Further, W, A is normally distributed with mean 0 and variance Al Let A be mean 0 random variable with variance
a I21 and B be a mean random variable with variance a²| A|| Notice that since (A, xx) = the joint distribution (Wiæ, Wix') is the same that of (A, cA + B). Therefore have:
E ((6(W==).0W3)) W;
E [sin( (A) sin(cA B)] A,
[[(sin(A) (in( cos B) + sin(A) sos(c))sin(B))
[E(sin(A) sin cos [B]]
[[ssin(A) sin(ccA) [[cos(B)]]
< |E(cos(B)) B)] < exp
C DESCRIPTION OF THE TRAINING PROCEDURE AND DATASETS
We use the Tensorflow framework for conducting our empirical study and all of our code is included as part of supplementary material. In every experiment, we train using SGD, without momentum, with constant learning rate of 0.01 and batch size of 256. We employ p100 single-instance GPU for each training run. For most of the experiments, the model trained until it obtains perfect accuracy on the training set, with only a few exceptions which either unavoidable requires extravagant training iterations. For example, the experiments involving linear activation, since none of the datasets use completely linearly separable, do expect the net to get 100% accuracy
13

---

Published as a conference paper at ICLR 2021
the training set. Another interesting case is the Sigmoid activation, for which the gradients starts to saturate as the scale of the input to the Sigmoid function increases. Thus, we stop the training at a point when at least one of the model in the study achieves perfect accuracy on the training set.
In our 2-layer MLP model, in almost all cases we use 1024 units for the hidden layer with exceptions of 1) experiments with Sigmoid activation and 2) ReLU activation with squared loss. In both of these cases, we increase the number of hidden units to 2048 in order to increase their training speed. Number of units for the softmax layer depends the number of output classes, which is 10 for CIFAR-10 / SVHN, and 100 for CIFAR-100. The details of the ConvNet architecture are included in appendix H For any layer that doesn't involve changing the initialization scale. for instance the top layer in all our models. defaults to using Glorot uniform initializer (Glorot & Bengio 2010). For experiments corresponding to Sections and 3.2 we refrain from employing bias variables order to match the setup exactly. For experiments in Section 5 all biases are initialized to zero.
We employ image classification datasets each having 32x32 pixels color image as input. CIFAR-10 dataset (Krizhevsky) 2009) consists of 60000 images with 10 classes. Classes are balanced with 6000 images per class. Training set consists 50000 images and 10000 test images. CIFAR-I00 (Krizhevsky et al.) is very similar to CIFAR-10 10 except that it has 100 classes with 600 images per class. Finally, The Street View House Numbers (SVHN) Dataset (Netzer et al. 2011) has images of digits from house numbers obtained from Google Street View with total of 10 classes. Training set contains 73257 images and 26032 test images.
D SIN ACTIVATION
Figure 2 shows that increasing the scale of initialization for hidden layer weights W1 in a 2-layer MLP model leads to extreme memorization CIFAR-10 dataset. Keeping everything else the same, reproduce the same phenomenon on two other datasets, namely, CIFAR-100 and SVHN respectively.
0.30
6.35
Alignment
0.35
1x0
16 1
1e.3
1w:2
Alignment.
6.25
0.30
6.20
leD
1e.3
1m-2
le
Accuracy
6.15
6.10
0.19
1c0
1el
6.85
0.00
0.33
6.00
-0.00
20
30
-840
10
30
0.00
10
20
40
1K training steps
X 1K training steps
X 1K training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.33
le3
0.30
Accuracy
Ie
0.4
0.05
0.2
100
1e]
0.
10
70
30
10
10
1K training steps
1k training steps
(d) Train accuracy
(c) Hidden layer activations
Figure 6: Results when using sin activation function on CIFAR-100 dataset.
14

---

Published as a conference paper at ICLR 2021
0.99
1e-3
0,4
1e:2
le2
Alignment
0.010
16-1
lell
Ic1
0.255
16-1
Iell
Accuracy
6.3
0,4
l62
163
0015
0.000
X 1k training steps
training steps
X 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.06
Alignment
Accuracy
1eB
361
1e-3
le.
0.0
0.00
x 1k training steps
X 1k training steps
(d) Train accuracy
(c) Hidden layer activations
Figure 7: Results when using sin activation function on SVHN dataset.
D.1 VARYING LEARNING RATES
Figure illustrates the lazy training phenomenon, in which gradients for the first layer are smaller compared the weights of the net. Thus, one might worry that the lack of generalization behavior is due to some scaling mismatch between the weights and the gradients essentially due to poor learning rate choice rather than some modified implicit bias due modified scaling. To investigate this possibility, we conduct the following study where we selectively alter the learning rate for the first layer. shown in the figure below, we observe extreme memorization phenomenon when the scale of the net is high across wide range of learning rates. Note that increasing the learning rate after certain point induces the net to stop learning altogether i.e. it no longer achieves perfect training accuracy.
As special case, we also experiment with frozen first layer weights. Even though, this case, net fails to achieve perfect training accuracy, still see the extreme memorization phenomenon when the scale initialization is high enough.
1.0
1.2
1e-1
1e:3
Accuracy
1e-2
1e-1
Accuracy
0.8
1e-4
1e:3
le0
1c:2
lel
0.6
1e-1
0.4
0.0
0.0
100
150
200
300
100
200
250
300
X 1k training steps
X 1k training steps
(a) Test accuracy
(b) Train accuracy
Figure 8: Results when using sin activation function CIFAR-10 dataset with scale of initialization to 1.0 and varying learning rates for the first layer shown plot.
15

---

Published as a conference paper at ICLR 2021
1.0
Accuracy
0.6
1e-4
1e-3
Accuracy
0.8
1e-4
1e-3
1e-2
1e-2
0.6
le-1
1e-1
0.4
le0
1e0
lel
le1
0.2
0.2
0.0
0.0
100
150
200
250
300
50
100
150
200
250
300
1k training steps
1k training steps
(a) Test accuracy
(b) Train accuracy
Figure 9: Results when using sin activation function on CIFAR-10 10 dataset with scale of initialization set to 1c-2 and varying learning rates for the first layer shown in the plot.
0.45
0.6
0.40
0.5
M
Accuracy
0.35
1e-3
Accuracy
0.4
1e-3
1e-2
1e:2
0.25
le:1
1e0
0.20
lel
0.2
0.15
0.1
0.05
0.0
50
100
150
200
0
50
100
150
200
1 training steps
X 1 training steps
(a) Test accuracy
(b) Train accuracy
Figure 10:
Results when using sin activation function on CIFAR-10 dataset when the first layer weights are frozen and varying scale of initialization as shown in the plot.
D.2 DOES DEPTH MATTER?
also experiment with increasing the depth of the MLP from 2-layers to 4-layers. We find the extremem memorization is present even in this case. We also similar decrease in generalization performance in the case of ReLU.
0.10
0.6
8.40
Aocurace
0.35
1e-3
Accuracy
0.0
Alignment
0.4
Teal
0.30
1e:2
le-2
162
0.
03
0.30
0.4
0:3
0.0
training steps
1k training steps
X training steps
(a) Test accuracy
(b) Train accuracy
(c) Hidden layer activations
Figure 11: Results when using siu activation function CIFAR-10 dataset with 4-layer MLP.
16

---

Published as a conference paper at ICLR 2021
E RELU ACTIVATION
E.1 SOFTMAX CROSS ENTROPY
1.
0.35
68
Alignment
0.26
Accuracy
Alignment
66
0.99
0.35
1e.5
ea
64
1n22
10:3
Le
1eb
100
Tel
120
150
1k training steps
1k training steps
1 training steps
(a) Top layer gradients
(b) Train accuracy
(c) Logit gradients
Figure 12: Additional plots when using ReLU activation function with softmax cross-entropy CIFAR-10
datasct.
0.31
Alignment
Aligmment
6,3
Accuracy
1et
0.20
n.c
1ex3
1o.3
1e.2
1e22
16-1
16-
0.00
x 1k training steps
training steps
x 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
1.2
1e-3
1.0
Accuracy
0.5
Alignment:
0.3
Aligmment
0.8
16:3
0.35
16:3
16:2
10
1e
1ex1
0.2
0.0
0.20
-4.3
x 1k training steps
x 1k training steps
X
1k training steps
(d) Train accuracy
(e) Hidden layer activations
(f) Logit gradients
Figure 13: Results when using ReLU activation function with softmax cross-entropy CIFAR-100 dataset.
17

---

Published as a conference paper at ICLR 2021
0.093
0:30
0.023
1m2
1o3
0.25
Aligmment
0.009
Alignment
Accuracy
0.15
0.bb
0.10
1c
0.005
162
00000
0.00
-0005-
0.2
1c1
-0.010
50
100
-0.0
03
00
102
0.1
100
1k training steps
1k training steps
1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.38
1.0
0.37
1e-3
1ee2
0.5
Accuracy
0.9
Alggmment
0.36
Alignment
0.G
0.34
0.7
16-3
1e-2
Ie:1
855
0.1
0.30
X 1K training steps
X 1k training steps
X 1K training steps
(d) Train accuracy
(c) Hidden layer activations
(f) Logit gradients
Figure 14: Results when using ReLU activation function with softmax cross-entropy on SVHN dataset.
E.2 HINGE LOSS
0.040
1e.3
Alignment
0.000
1e-2
1
Alignment
622
le/2
Incl
1a0
Le0
6.1
lel
160
162
8.05
0.000
0.00
-0.000
32
-0.0%
0.15
x 1K training steps
x 1k training steps
x 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
3.0
1c2
Accuracy
Alignment
0.33
Alignment
16-1
1c0
le]
1e1
1o-3
0.30
0.39
0.2
0.0
0.3
16
20
026
10
16
26
25
30
13
25
x 1k training steps
1k training steps
X 1k training steps
(d) Train accuracy
(c) Hidden layer activations
(f) Logit gradients
Figure 15: Results when using ReLU activation function with hinge loss on CIFAR-10 datasel.
18

---

Published as a conference paper at ICLR 2021
0.085
0:2
103
36
1m22
0.20
16-2
Aligmment
0010
Alignment
0:39
1e-1
lef
Accuracy
0.30
0.
1c
0.001
0.0
0.1
162
8000
1c1
-0.00
30
10
50
-0.0
10
0.2
10
50
1k training steps
1k training steps
1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.35
1e-3
1e.3
1.0
0.34
ls2
1e-2
Accuracy
0.9
Alggmeett
0.23
Alignment
as
0.0
0.32
1e-2
Ie-]
0.5
x 1K training steps
X 1k training steps
X 1K training steps
(d) Train accuracy
(c) Hidden layer activations
(f) Logit gradients
Figure 16: Results when using ReLU activation function with hinge loss on SVHN dataset.
0.35
1e.3
1e-2
6.15
0.20
Allgnment
Alignment
Accuracy
0.15
1e0
1e1
8.10
lel
0.05
16-2
1e-3
0:000
16-1
160
-0.000
training steps
x 1k training steps
X 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
12
1.0
Accuracy
1..
Alignment
0.25
Lec3
Alignment
100
1el
0.6
Le
0.20
1e-2
16-1
0.0
X 1k training steps
x 1k training steps
X 1 training steps
(d) Train accuracy
(c) Hidden layer activations
(f) Logit gradients
Figure 17: Results when using ReLU activation function with hinge loss on CIFAR-100 dataset.
E.3 DOES DEPTH MATTER?
Similar to sin activation, we a decrease in generalization performance even in the case of ReLU, despite increasing the depth.
19

---

Published as a conference paper at ICLR 2021
1.0
Rep Aligmment
0.
Accuracy
1633
Accuracy
0.
Ie-3
1e3
1e-2
1ex2
64
1e-2
16-1
68
0.4
0.
0.0
000
x 1k training steps
x 1k training steps
1k training steps
(a) Test accuracy
(b) Train accuracy
(c) Hidden layer activations
Figure 18: Results when using ReLU activation function on CIFAR-10 10 dataset with 4-layer MLP.
E.4 SQUARED LOSS
Note that when employing squared loss, we increase the number of hidden units from the usual 1024 units 2048 in order to compensate for very low training speed. Also, observed that increasing the scale of initialization for W1 beyond a certain scale leads to divergence in training after a few iterations. Thus, we recover the phenomenon of interest with much less aggresive increase scale of initialization i.e. we double the standard deviation instead of increasing it by ten times as done in other experiments.
0:00
0.07
0.003
Allgmment
0:06
0:66
Alignment
0.29
Accuracy
0.016
0.016
0:04
2.00
0.15
0.002
0.001
0.00
0.004
0.5
0.00
training steps
x 1k training steps
x 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.9
0.8
0.00L
0.002
Accuracy
0.6
0,00a
0.004
Alignment
c.6
0.016
ca
0.001
0.002
0.4
0.004
03
0.000
6.1
0.2
0.335
x 1k training steps
x 1k training steps
X 1K training steps
(d) Train accuracy
(e) Hidden layer activations
(I) Logit gradients
Figure 19: Results when using ReLU activation function with squared loss CIFAR-10 dataset.
20

---

Published as a conference paper at ICLR 2021
0.10
0.49
0.35
0.001
Allgnment
0.50
D:000
0.004
0.016
0.002
Alignment
0.29
0.39
0.001
0.002
Accuracy
0.13
0.20
0.002
0.001
2:00
0,004
0.004
0.000
2.04
0.00
0.00
150
250
350
200
300
50
200
301
350
training steps
X 1K training steps
1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
1.0
1.0
Accuracy
0.001
Aligmment:
Alignment
0.7
0.002
6.
0.004
0.00
0,001
0.001
0.066
01002
0,002
0.304
0.064
0.000
0.008
0.0
00
200
200
250
300
100
1.0
300
x
1 training steps
X 1k training steps
training steps
(d) Train accuracy
(c) Hidden layer activations
(f) Logit gradients
Figure 20: Results when using ReL. activation function with squared loss CIFAR-100 dataset.
0.005
0.25
0.80
Alignment
0.000
0.001
Alignment
0,004
0.002
0,001
0.85
0.80
0.015
0,008
0.75
0.70
0.002
0.004
0.008
0.16
150
training steps
training steps
training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.3
0.5
0.001
0.001
0.002
0.5
0.002
Accuracy
119
Alignment
0.356
0.260
0.008
0.001
Aligmment
0.5
0.016
0.016
0.001
0.355
0.6
0.340
0.3
0.008
822
0.015
0.330
0.4
20
300
130
200
0,300
50
100
150
300
9-1
100
150
200
1 training steps
1k training steps
1 training steps
(d) Train accuracy
(e) Hidden layer activations
(f) Logit gradients
Figure 21: Results when using ReLU activation function with squared loss on SVHN dataset.
F LINEAR ACTIVATION
21

---

Published as a conference paper at ICLR 2021
0.25
0.6
Allgnment
2:00
Alignment
022
10.3
Accuracy
0.33
0.13
1
0.33
16-1
100
2c.2
0.04
Tel1
1c0
100
Tml
100
150
200
250
training steps
X 1k training steps
1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.60
0.23
0.55
www P
Accuracy
0.50
0.25
Tmx3
0.45
1c.2
0.60
1e-1
0:25
160
0.30
1o:1
1ef1
0.25
1ol
0.20
09
106
150
200
100
150
200
1K training steps
1 training steps
(d) Train accuracy
(c) Hidden layer activations
Figure 22: Results when using linear activation function with softmax cross entropy loss on CIFAR-10 dataset.
0.30
Allgnment
0.34
0.10
0:13
Jufl
Alignment
0.27
1e33
Iel
1e-2
1s0
Accuracy
0.16
1w0
te:
0:00
1c1
6,15
0.12
tel
2:00
0.04
c.s5
0.08
n.00
-0.00
0.01
x 1k training steps
x 1k training steps
x 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
n.
Te-3
Ie-
Accuracy
Alignment
Se-2
1c0
1o0
1e1
o.t
00
300
-0.05
200
x 1 training steps
1K training steps
(d) Train accuracy
(e) Hidden layer activations
Figure 23: Results when using linear activation function with sofumax cross entropy loss on CIFAR-100 dataset.
22

---

Published as a conference paper at ICLR 2021
0.36
0.030
W
0.24
Alignment
3/225
Alignment
M
MW
163
Accuracy
0.20
0.22
1ex2
1
0.13
10-2
3/000
3.00
156
250
X 1K training steps
1K training steps
X 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
0.45
Www
Accuracy
0.30
Alignment
0.08
1
1s-1
1c0
0.20
1e:1
0.3
Tef
1o1
0.10
106
150
200
100
150
200
1K training steps
1k training steps
(d) Train accuracy
(c) Hidden layer activations
Figure 24: Results when using linear activation function with softmax cross entropy loss on SVHN dataset.
G
SIGMOID ACTIVATION
Note that when employing Sigmoid activation, after certain scale the hidden layer gradients start to vanish. We try to compensate for this by increasing the number of hidden units to 2048.
0.00
0.0
163
0.9
Alignment
0.06
Aligmment
1e0
Ie:
Accuracy
0.7
le]
0.00
0.00
0.3
0.01
0.33
-2.01
50
100
130
200
0.3
60
100
150
200
0.30
100
230
x 1k training steps
1K training steps
X 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
1.0
16:
te
Accuracy
Alignment
1e0
16-1
lel
0.
0.2
50
100
150
200
0.
50
100
100
200
training steps
1 training steps
Train accuracy
(c) Hidden layer activations
Figure 25; Results when using Sigmoid activation function on CIFAR10 dataset.
23

---

Published as a conference paper at ICLR 2021
0.20
16:3
10.3
1o-2
Alignment
1n-1
Alignment
1c0
Acouracy
1el
Tel
98
1c
0.09
1.00
04
250
0.0
0.10
X 1k training steps
X 1k training stops
x 1k training stops
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
Ie-3
Accuracy
le1
1e-3
1e-2
0.7
0.
350
x 1k training steps
1k training steps
(d) Train accuracy
(c) Hidden layer activations
Figure 26: Results when using Sigmoid activation function on CIFAR dataset.
Alignment
0:19
Alggmmett
1e0
lel
0.33
6.86
0.
0.30
0.000
0.35
-6.600
-0.000
50
100
150
200
0.4
100
0.15
50
100
150
250
250
1K training steps
1k training steps
X 1k training steps
(a) Hidden layer gradients
(b) Top layer gradients
(c) Test accuracy
le:2
Accuracy
1c
1ell
1e1
1e-3
1el
0.0
50
100
150
2000
200
0.5
100
150
2010
2010
1k training steps
1k training steps
(d) Train accuracy
(e) Hidden layer activations
Figure 27: Results when using Sigmoid activation function on SVHN dataset.
H CONVNET ARCHITECTURE
Our goal to recover the phenomenon that convolution and pooling operation leads to more aligned representations. In order to do this, construct simple ConvNet architecture. We start with two consecutive convolution and max pool operations, followed by a fully- fully-connected and softmax layers. Note that the last two layers fully connected and softmax operationally the same our 2-layer MLP.
We keep all the other hyper parameters the same between training runs for all the architectures. All the models are trained with SGD without momentum with learning rate set to 0.01 and batch size to 256.
24

---

Published as a conference paper at ICLR 2021
Convolution layer 1: 32 filters with 5x5 kernel size followed by ReLU activation.
Max pooling layer 1: pool size 3x3 with stride of 2x2
Convolution layer 2: 64 filters with 5x5 kernel size followed by ReLU activation.
Max pooling layer 2: pool size 3x3 with stride of 2x2
Fully connected layer with 1024 hidden units followed by ReLU activation.
Softmax layer with 10 output logits.
25


Reviews:
[{"review_id": "rgGm2Jn30l", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper:** A series of empirical observations are made about the influence of scale of init on generalization (in particular, that a continuum of generalization performance from random to very good can be generated by varying only the scale of init) , and these effects are explained in detail for different activation functions. The authors also propose a measure of gradient alignment which they show correlates with generalization performance\n\n**Pros/strong points:**\n - detailed explanations for each activation function provide nice insights\n - solid experiments\n\n**Cons/weak points:**\n - overall clarity and presentation of information is the largest weakness in my opinion, although the writing is generally good. I think it just needs a few more passes, with an eye to making sure things are accessible/understandable/flow. Could be improved substantially just with formatting/subsections or something, e.g. italicizing key insights or making sub paragraphs where each gives a particular insight\n - some small things in related work\n\n**Summary of review + recommendation:** Overall I think this is a good paper, and could be a very good paper with some \"tightening up\" and clarifications. The combination of things is too much for me to recommend acceptance out of the box, but the things are relatively small and I think easy to address, and I'd be happy to increase my score.\n\n**Detailed review and Specific questions/recommendations:**\n - unclear what \"large scale training\" means\n - \"engendering\" is an unnecessary word there\n - observations about overparameterized models should be cited, e.g. Zhang et al.  and Arpit et al.\n - Chizat & Bach further observe (not observes), same incorrect pluralization with many citations (suggest checking the whole document)\n - background work portion of the intro misses works, some poorly explained / relationship to current work not discussed, and overall feels rushed. The Related Work section does a good job mostly though. I suggest moving the 3rd p of the intro into the related work, moving the first section of it about scale of init to the first sentence of Contributions. Merging them should get you some extra space for more experiments/larger figs.\n - Related work on inits should cite lottery ticket works (e.g. Frankle et al)\n - Geiger et al reference you describe what they do but not what to take away from it\n - extreme memorization should be bolded since it's a term you're defining (and make clear if you're proposing this term and if not, where it is from), but you then  define memorization the same way you define extreme memorization, making this term (\"extreme\") seem unnecessary. \n - \"from verylittle overfitting to perfectly memorizing the training set while making zero progress on test error\" this sentence is unclear, makes it sound like \"while...\" applies to both of the 2 extremes. Suggest rephrasing.\n - I find 2nd bullet of contributions unclear about what is expected vs. what happens and what we learn from that \n - in 3rd bullet briefly summarize the alignment measure. I suggest using a different/more precise term for this measure (e.g. gradient alignment) since just \"alignment\" means so many things already\n - In \"related statistics\" you don't mention if Chatterjee has a measure for coherence of gradients (I skimmed that paper and it seems not, but I'm not sure). If not, then maybe calling this a measure of gradient coherence would be appropriate? I googled it quickly and it seems that coherence means something specific in linear algebra and signal processing: (https://en.wikipedia.org/wiki/Mutual_coherence_(linear_algebra), https://en.wikipedia.org/wiki/Coherence_(signal_processing)) - maybe not necessary to comment on in the paper, but if the authors are familiar with this use of the term I'd appreciate a clarification of how it relates to the mentioned measures of alignment I'd be interested\n - \"related statistics\" should also mention Arpit et al critical sample ratio (which does take class information into account) and comment on differences (or if it's too different to include here, I'd appreciate an explanation of why)\n - Mention computational cost of the different measures of alignment\n - homogeneity is repeatedly mentioned without explanation (just a brief 1-line would do)\n - \"fix the scale\" ambiguous whether this means fix in place (at a particular value) or fix as in correct\n - seems obvious to me that the scaling would affect relus (especially in the absence of bias as your experiments say); if the scale is larger, fewer values are initialized near the non-linear region of the relus, meaning there are more 'dead relus' near the beginning (which I would guess up to a point could provide regularization, but past that point would just make learning slow and even unstable). Could the authors comment on this; do you think it's correct/relevant? How does it fit with the argument about homogeneity?\n - State important equations in words as well as math for clarity and ease of reading (as is done for eq.6; make sure this is done consistently, especially important for your proposed measures of alignment).\n - Conclusion discusses the results strangely, without mentioning the actual results (e.g. \"making it particularly interesting\" - why/how is it interesting, what are the implications for people using sin?, \"the loss function plays a crucial role\" what role? what things are good for what, what should I look out for?). The conclusion should stand on its own and summarize results, not reference them in a way that requires me to have read the whole paper to understand.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YJ8awi6jIv2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###\nSummary:\nThis paper investigates the role of scale in generalization of neural networks. It shows that the initial scale of a 2-layers MLP, with sinusoid or ReLU activation, can control the memorization behavior of the network, from very little overfitting to complete memorization.\nIt then proposes an alignment measure which correlates with generalization for different initial scale. It shows that this alignment measure can capture generalization performances for other architecture such as ResNet or DenseNet on the CIFAR-10 dataset.\n\n###\nReasons for score: \nOverall, I find the paper to be a bit borderline. \nThe observation regarding the scale impacting generalization is novel and interesting as I would have assumed that large initial scale would lead to bad optimization rather than a lack of generalization.\nHowever, all the experiments regarding the scale are carried out on a two-layers MLP models and it is not clear to me if similar conclusion would be true for deeper architecture.\n \n \n###\n Pros:\n- interesting observation regarding the impact of the initial scale on generalization\n- clearly show the effect in a two-layers MLP with various activation and loss functions\n- propose an alignment measure which have some promising correlation with generalization\n \n###\nCons: \n- experiments investigating the impact of the initialization scales only for two layers MLPs\n- Alignment is not compared with other generalization metrics in section 5.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EHVIZ3OAzcn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Overview \n\nThe paper studies how the generalization of the neural network trained with SGD is affected by the scale of the random initialization. Specifically, that when the scale of the initialization is big, the network overfits to the training set with bad performance on the test set. \nThe paper also provides a hypothesis why does it happen -- that the graidents of difference examples are orthogonal in the \"bad\" mode and proposes a measure called \"alignment\" to predict the generalization reghime of the network.\n\nThe problem, tackled in paper is interesting and the paper itself is thought-provoking. \n\n\nThe central model studied is 2 layer fully-connected neural network with {sin, ReLU} activation, where the 1st layer is initialized with variable (studies) scale and the 2nd - with Xavier init.\nBoth layers are without biases. The model is optimized with SGD w/o momentum and constant learning rate and the loss function is one of the classification losses: CE, hinge loss.\n\n## Strong points\n\nThe paper formulates hypothesis and verifies it via series of controlled experiments. Besides \"toy\" 2-layer model, the similar results are get with more powerful architectures, like CNN, DenseNet and so on on the set of middle-sized datasets like SVHN and CIFAR.  \nThe paper clearly states its place among related works and proposes a useful metric for diagnosing model training. \n\n## Questions (and possible weak points\n\n\n1. Figure 2 shows that the norm(grad)/norm(weight) decreases significantly with the scaling-up the initalization. This is hypothesized to be one of the problems with large scale initialization: the weights do not go far from the original (random) values. If this is the case and the problem, may be scaling up the learning rate with the initalization to keep the norm constant, would help?\nWhy is this an important question? Because the problem with large init is the bad gradient direction (as hypothesized in paper), then scaling the learning rate would not help. If, otherwise, the problem is the learning rate scale, then the work could be seen as indirect confirmation of the works about large-vs-small learning rate reghimes, e.g.\nLewkowycz et.al.  https://arxiv.org/pdf/2003.02218.pdf\nMoreover, may be then we should study not the scale of init, but the ratio (init scale/lr). \n\n\n2. Following Q1, one also could use different learning rates for different layers depending on the scale of weights and/or activation. Yes, that would significantly complicate the experimental setup, but may lead to the different conclusions. E.g. it is commonly know that deep sigmoid networks are hard to train because of vanishing gradient issue. However, as it was shown in \"Revise Saturated Activation Functions\" Xu et.al (https://openreview.net/forum?id=D1VDjyJjXF5jEJ1zfE53), one could perfectly fine train such networks. The only thing, which is needed, is the proper rescaling of the learning rate per layer, up to ridiculous values such as 4^11.\n\n\n3. (minor) why do you don't use bias? Bias-less NNs are less common and give worse results. \n\n4.  (just curious) Where is the actual learning with large scale init take place - in layer 1 or 2? If only one layer is actually trained, can we obtain the same results, when the non-training layer weights are frozen?\n\n\n5. When examining Figures in appendix (e.g. 17, 20, etc), it looks like that proposed \"alignment\" measure can be used as alarm -- if it is low, the generalization is low, but the ranking of (test accuracy) and (alignment) is not really aligned. Any comments on that?\n\nOverall, I like the paper, although would also like to have answers to my questions.\n\n## Update after rebuttal\n\nAll my concerns have been addressed, including the possible alternative explanation of the experimental results. I strongly recommend the paper to be accepted.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a clear and useful empirical study of how the initialization scale and activations function affects the generalization capability of neural networks. Previous works showing the effect of the initialization scale (Chizat and Bach (2018), Geiger et al. (2019), Woodorth et al. (2020)) had a more limited set of experiments. Moreover, here an extreme case is shown, wherewith sin activation function no generalization is possible at a large init scale (there the kernel regime is useless for generalization since the hidden layer output becomes very sensitive to any small perturbation in the input). Lastly, two alignment measures are suggested, which are correlated with the generalization across several architectures and initialization scales.

All the reviewers argued for acceptance, and one strongly so. I agree that the paper is sufficiently interesting and clear to be accepted. However, despite the high scores, I only recommend a poster and not spotlight/oral: I think the novelty of the empirical study is not groundbreaking, given the experiments in previous works, and the usefulness of the suggested measures are not completely clear without a thorough comparison against previously suggested measures.
Decision: Accept (Poster)