Understanding Why Neural Networks Generalize Well Through GSNR of Parameters
Authors: Jinlong Liu
Yunzhi Bai
Guoqing Jiang
Ting Chen
Huayan Wang
Venue:  (2020)
Keywords: DNN
generalization
GSNR
gradient descent

Abstract: As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalize well. In this paper, we provide a novel perspective on these issues using the gradient signal to noise ratio (GSNR) of parameters during training process of DNNs. The GSNR of a parameter is simply defined as the ratio between its gradient's squared mean and variance, over the data distribution. Based on several approximations, we establish a quantitative relationship between model parameters' GSNR and the generalization gap. This relationship indicates that larger GSNR during training process leads to better generalization performance. Futher, we show that, different from that of shallow models (e.g. logistic regression, support vector machines), the gradient descent optimization dynamics of DNNs naturally produces large GSNR during training, which is probably the key to DNNs’ remarkable generalization ability.

Published as a conference paper at ICLR 2020
UNDERSTANDING WHY NEURAL NETWORKS GENER- ALIZE WELL THROUGH GSNR OF PARAMETERS
Jinlong Liu', Guo-qing Jiang'. Yunzhi Bai', Ting Chen?, and Huayan Wang'
Ytech KWAT incorporation
jaggiioong..ingguuoiig....g baiyunzhi, wanghuayan| (arnghaaann@kuuiiiliii @kuaishou.com
*Samsung Research China Beijing (SRC-B)
ting I. I.chen samsung.com
ABSTRACT
As deep neural networks (DNNs) achieve tremendous success across many appli- cation domains, researchers tried to explore many aspects on why they gener- alize well. In this paper, we provide a novel perspective on these issues using the gradient signal to noise ratio (GSNR) of parameters during training process of DNNs. The GSNR of parameter is defined as the ratio between its gradi- ent's squared mean and variance, over the data distribution. Based on several ap- proximations, we establish a quantitative relationship between model parameters' GSNR and the generalization gap. This relationship indicates that larger GSNR during training process leads to better generalization performance. Moreover, we show that, different from that of shallow models (e.g. logistic regression, support vector machines), the gradient descent optimization dynamies of DNNs naturally produces large GSNR during training, which is probably the key to DNNs' re- markable generalization ability.
INTRODUCTION
Deep neural networks typically contain far more trainable parameters than training samples, which seems to easily cause a poor generalization performance. However, in fact they usually exhibit
remarkably small generalization gaps. Traditional generalization theories such VC dimension
(Vapmık & Chervonenkis, (199]) or Rademacher complexity (Bartlett & Mendelson son, 2002) cannot
researrch focuses on the generalization ability of DNNs (Neyshabur explain its mechanism. Extensive research
2 all. 2017 Arora et all, 2018: Keskar et all, 2016, Dinh et all 2017: Hofer et all 2017: Novak
2018: Dziugarte Roy. 2017 Iakubovitz et all 2019: Kawaguchi et all 2017 Advani
Saxe
2017)
Unlike that of shallow models such as logistic regression or support vector machines, the global minimum of high-dimensional and non-convex DNNs cannot be found analytically, but can only be approximated by gradient descent and its variants (Zeilei 2012; Kingma & Ba, 2014 Graves 20E).. Previous work (Zhang all 2016: Hardi all 2015: Dziugatte & Roy, 2017) suggests that the generalization ability of DNNs is closely related to gradient descent optimization. For example, Harde (2015) claims that any model trained with stochastic gradient descent (SGD) for reason- able epochs would exhibit small generalization error. Their analysis is based on the smoothness of loss function. In this work, attempt understand the generalization behavior of DNNs through GSNR and reveal how GSNR affects the training dynamics of gradient descent. Stanislay Fori (201%) studied new gradient alignment measure called stiffness order understand generaliza- tion better and stiffness is related to our work.
The GSNR of a parameter defined as the ratio between its gradient's squared mean and variance over the data distribution. Previous work tried to use GSNR to conduct theoretical analysis on deep learning. For example, Rainforth et all (201) used GSNR analyze variational bounds in
"corresponding author

---

Published as a conference paper at ICLR 2020
unsupervised DNNS such variational auto-encoder (VAE). Here we focus analyzing the relation between GSNR and the generalization gap.
Intuitively, GSNR measures the similarity of a parameter's gradients among different training sam- ples. Large GSNR implies that most training samples agree on the optimization direction of this parameter, thus the parameter is more likely to be associated with meaningful "pattern" and assume its update could lead to a better generalization. In this work, we prove that the GSNR is strongly related the generalization performance, and larger GSNR means a better generalization.
To reveal the mechanism of DNNs' good generalization ability, we show that the gradient descent optimization dynamics of DNN naturally leads to large GSNR of model parameters and therefore good generalization. Furthermore, we give complete analysis and detailed interpretation to this phenomenon. We believe this probably the key to DNNs remarkable generalization ability.
In the remainder of this paper we first analyze the relation between GSNR and generalization (Sec- tion (). We then show how the training dynamics lead large GSNR of model parameters experi- mentally and analytically in Section B.
2 LARGER GSNR LEADS TO BETTER GENERALIZATION
In this section, we establish a quantitative relation between the GSNR of model parameters and generalization gap, showing that larger GSNR during training leads better generalization.
2.1 GRADIENTS SIGNAL TO NOISE RATIO
Consider a data distribution Z = X x V, from which each sample (z, y) is drawn; a model ú (1r,, 0) parameterized by 0: and loss function T..
The parameters' gradient w.r.t. and sample (x,, yt) is denoted by
g Vi, 0) or (0) == (T ( 0))
(1)
00
whose j-th element is g, (0,). Note that throughout this paper we always use to index data examples and j index model parameters.
Given the data distribution 2, we have the (sample-wise) mean and variance of (0). We denote them as g() E(n,y)=z( (g(z, y. 8)) and (²) (A)())))((((())) y. 0)). respectively.
The gradient signal to noise ratio (GSNR) of one model parameter 0, is defined as:
r(0,) g² (0,)
(,(
At particular point of the parameter space, GSNR measures the consistency of a parameter's gra- dients across different data samples. Figure M intuitively shows that if GSNR is large, the parameter gradient space tends be distributed in the similar direction and if GSNR is small, the gradient vectors are then scatteredly distributed.
2.2 ONE-STEP GENERALIZATION RATIO
In this section introduce concept help measure the generalization performance dur- ing gradient descent optimization, which we call one-step generalization ratio (OSGR). Consider training D (I1,81); ....,,,, (n))
2" with n samples drawn from Z, and test set D' {(24 v/),,... (I',. ) z In practice we use the loss on D' measure generalization. For simplicity. we assume the sizes of training and test datasets are equal, i.e. = n'. We denote the empirical training and test loss as:
L[D] Σ L(v. 0)), LID'
Σ L(y/. f(z,, 1=1
respectively. Then the empirical generalization gap is given by L T.
2

---

Published as a conference paper at ICLR 2020
Greater GSNR
Smaller GSNR
Gradient())
Gradien(())
Gradien vectorad single
Graaionn of sumple
Gradiont
Gradient
(radient(B))
Gradient(0))
Figure 1: Schematic diagram of the sample-wise parameter gradient distribution corresponding to greater (Left) and smaller (Right) GSNR. Pink arrows denote the gradient vectors for each sample while the blue arrow indicates their mean.
OSGR()
< OSGR() < (most commonly happens)
OSGR()
Test loss
Loss
Loas
Loxs
Test loss
Test loss
Train
Train loss
Train
Epochs
Epocks
Epochs
Schematic diagram of the training behavior satisfies OSGR(() 0 (Left), 0 OSGR(L) (Middle) and OSGR(L) (Right). Note that the Middle scenario most com- monly happens in regular tasks.
Figure 2:
In gradient descent optimization, both the training and test loss would decrease step by step. We use [L[D] and AL to denote the one-step training and test loss decrease during training, re- spectively. Let's consider the ratio between the expectations of AL[D'] and AL|D] of one single training step, which we denote as n).
R(2,n) n) = ED,D' (OD=2=(AL(D")
(4)
ED~Z.(AL(D)
Note that this ratio also depends on current model parameters 0 and learning rate X. We are not including them in the above notation as we will not explicitly model these dependencies, but rather try to quantitatively characterize R for very small À and for at the early stage of training (satisfying Assumption [))
Also note that the expectation of AL|D') is over D and D'. This because the optimization step is performed D. We refer to R(2,n) n) OSGR of gradient descent optimization. Statistically the training loss decreases faster than the test loss and 0 < OSG R(tt) < 1 (Middle panel of Figure D). which usually results in a non-zero generalization gap at the end of training. If OSGR(t) is large (w 1) in the whole training process (Right panel of Figure D) generalization gap would be small when training completes, implying good generalization ability of the model. If OSGR(t) is small (= 0), the test loss will not decrease while the training loss normally drops (Left panel of Figure D)).
corresponding to large generalization gap.
2.3 RELATION BETWEEN GSNR AND OSGR
In this section, we derive a relation between the OSGR during training and the GSNR of model pa- rameters. This relation indicates that, for the first time far know, the sample- wise gradient distribution parameters related to the generalization performance of gradient descent optimiza- tion.
3

---

Published as a conference paper at ICLR 2020
In gradient descent optimization, take the average gradient over training set D, which we denote as (f). Note that we have used (f) to denote gradient evaluated on one data sample and g()) to denote its expectation over the entire data distribution. Similarly we define gDr(0) to be the average gradient over test set D'
[L[D]
R
g(d aL(D1] a0
ED (D(
71 Σ g( Vr.
00 gD
71 Σ
(5)
Both the training and test dataset randomly generated from the same distribution 2" so can treat gn(0) and gD (0) as random variables. At the beginning of the optimization process, is randomly initialized thus independent of D, so gD( and g (0) would obey the same distribu- tion. After period of training, the model parameters begin to fit the training dataset and become a function of D, i.e. A ((D), therefore distributions of gD g D)) and gD gD(()( become differ- ent. However we choose not to model this dependency and make the following assumption for our analysis:
Assumption 2.3.1 (Non-overfitinn limit approximation) The average gradient over the training dataset and test dataset (0) and gD (0) obey the same distribution.
Obviously the mean of (+) and gD (0) is just the mean gradient over the data distribution g(0).
Ep-zz (gD (0)] = Ep, 2 (g) (0))
g(()
(6)
We denote their variance 0² (2) i.e
Varp (gD (0) 2Br (0) = (0)
It is straightforward to show that:
o² d²( = VarD=
Σ
g: ()) p()
(8)
n1
where (0) is the variance of the average gradient over the dataset of size n, and (0) is the variance of the gradient of a single data sample.
In one gradient descent step, the model parameter is updated by A0 0c+1 O2 ~g (0) where À is the learning rate. A is small enough, the one-step training and test loss decrease can be approximated by
0L|D]
ALDD] ~ -A-
0(2²) AgD (0) 002)
(9)
00
D
ALID' AA
+ 0(13) (g) gD (0) + 0(2))
(10)
00
Usually there are some differences between the directions of gD and (0), so statistically A tends to be larger than AL(D') and the generalization gap would increase during training. When À + O, one single training step the empirical generalization gap increases by AL(D) [D))",, for simplicity we denote this quantity as V:
ALD ALID'] R
(g) gD (g) (0)
(11)
A(g(0) (g(())+))(8)) (g(()))))))))))))))))))))))))))) + g())
(12)
= A(g(O) + +e)(c-e)
(13)
Here we replaced the random variables by (0) = g(0) + and gp (0) = (() + é' where and
é' are random variables with zero mean and variance o² (0). Since E(c') = F((c) = 0. and d' are independent, the expectation of v is
ED. nn (V) E(de c) +
Σ 0² (2,)) = >
O(22)
(14)
where o² (0(;) is the variance the of average gradient of the parameter 0z-
4

---

Published as a conference paper at ICLR 2020
For simplicity, when it involves single model parameter 8,, we will use only subscript instead of the full notation. For example, we use oj. Yj, and gD,j to denote (²())) r(0z), and gD (03)
respectively.
Consider the expectation of AL(D] and AL[D" when A ++0
ED (AL(D) x xEn-z~ gn(0)) = Σ
ED (BD
(15)
3
1o.0'~2-(A1DD)) = ED, (AL[D)- -v)
(16)
"(ED (Bb.)) - -03)
(17)
= E (EDwzn (Bn - p3//n)
(18)
Substituting (18) and (L5) into ( we have:
Σ
R(Z,n) =1- T1 "E, E (gD
(19)
Although we derived eq. (l9) from simplified assumptions, we can empirically verify it by estimating two sides of the equation on real data. We will elaborate on this estimation method in section 244
We can rewrite eq. (I9) as:
2r (gD
R R(2,n) 1--
n Σ
(20)
y E E (BD,)') EpI (gb.))
Enuz (gD
1
(21)
n
Eg E En (DD r1
where EDa2" (g VarD (gD.s)
EZ (g..)) 1/33 + g]
We define AL [D] to be the training loss decrease caused by updating 0y. We can show that when A is very small AL, [D] Ag. O(2)). Therefore when À - O, we have
T where Wj = EDN2D AL,[D] E DI)
R(2,n) = Σ
W ry
with Iw, - =1 (22)
Eq. (22) shows that the GSNR ry plays crucial role in the model's generalization ability the one- step generalization ratio in gradient descent equals one minus the weighted average of ry+ all model parameters divided by n. The weight is proportional to the expectation of the training loss decrease resulted from updating that parameter. This implies that larger GSNR of model parameters during training leads to smaller generalization gap growth thus better generalization performance of the trained model. Also note when n + oc, we have RZZ,n) 1, meaning that training on more data helps generalization.
over
2.4 EXPERIMENTAL VERIFICATION OF THE RELATION BETWEEN GSNR AND OSGR
The relation between GSNR and OSGR, i.e. eq. (19) (22) be empirically verified using any dataset if: (1) The dataset includes enough samples to construct many training sets and a large enough test set that
reliably estimate p3: Ep-zz (gD.,) and OSGR. (2) The learning rate is small enough. (3) In the early training stage of gradient descent.
To empirically verify eq. (I9), show how estimate its left and right hand sides, i.e. OSGR by definition and OSGR as a function of GSNR. Suppose we have M training sets each with size n, and test of size n'. We initialize model and train it separately on the M training sets and test it with the same test set. For the 1-th training iteration, we denote the training loss and test loss of
the model trained the m-th training dataset [mm)) and I. (m(m)) respectively. Then the left hand
5

---

Published as a conference paper at ICLR 2020
KC50h=20
100
ch_6
1000
ch_8
2000
ch_10
4000
to RIS cfee, 19 idSGe fandiion CONR)
ch d__2
6000
of j0SGR fanction CSMR)
opoch--c0 0-0.936
epoch-2500
CH0.967
ch_14
8000
x
ch_16
10000
15000
ch_18
dh_20
Figure 3: Left hand (LHS or OSGR by definition) and right side (RHS or OSGR as a function of GSNR) of eq. (I9). Points are drawn under different experiment settings. Left: LHS vs RHS at epoch 20, 100, 500, 2500. Each point is drawn by LHS and RHS computed at the given epoch under different model structure (number of channels) or training data size; red dotted line is the line of best fit computed by least squares; blue dotted line is the line of reference representing LHS = RHS; the value of e in each title represents the Pearson correlation coefficient between LHS and RHS computed by points in figure. Right: The legend. Different symbols and colors stand for different number channels and training data size. Different random noise levels are not distinguished.
side, i.e. OSGR by definition, of the 1-th iteration can be estimated by
Σ
[((m)
Lm(m)
R2, RJ m=
(23)
M (m))
L, (m)
For the model trained on the m-th training set, we can compute the t-th step average gradient and sample-wise gradient variance of f on the corresponding training set, denoted as gm,j.a and Pm respectively. Therefore the right hand side of eq. (I9)) can be estimated by
M
M
1
=
Pm.
(24)
M
M
We performed the above estimations on MNIST with a simple CNN structure consists of Conv-Relu-MaxPooling blocks and 2 fully-connected layers. First, estimate eq. ((2)) with
M = 10, we randomly sample 10 training sets with size n and a test set with size 10,000. To cover different conditions, we (1) choose E {1000, 2000, 4000, 6000, 8000, 10000, 15000]. respectively; (2) inject noise by randomly changing the labels with probability Prandom E (0.0,0.1, 0.3, 0.5]; (3) change the model structure by varying number of channels the lay- ers, ch E [6, 8, 10, 12, 14, 16, 18, 20). See Appendix A for more details of the setup. We use the gradient descent training (not SGD), with small learning rate 0.001. The left and right hand sides of [ at different epochs are shown in Figure E where each point represents one specific choice of the above settings.
At the beginning of training, the data points are closely distributed along the dashed line correspond- ing to LHS=RHS. This shows that eq. (IS) fits quite well under a variety of different settings. As training proceeds, the points become more scattered as the non-overfitinng limit approximation no longer holds, but correlation between the LHS and RHS remains high even when the training con- verges (at epoch 2,500). We also conducted the same experiment on CIFAR10 and toy dataset A. observed the same behavior. See Appendix for these experiments.
The empirical evidence together with our previous derivation of eq. (199) clearly show the relation between GSNR and OSGR and its implication in the model's generalization ability.
TRAINING DYNAMICS OF DNNs NATURALLY LEADS TO LARGE GSNR
In this section, we analyze and explain one interesting phenomenon: the parameters' GSNR of DNNs rises in the early stages of training, whereas the GSNR of shallow models such logistic regression or support vector machines declines during the entire training process. This difference gives rise to GSNR's 's large practical values during training, which in turn is associated with good

---

Published as a conference paper at ICLR 2020
generalization. We analyze the dynamics behind this phenomenon both experimentally and theoret- ically.
3.1 GSNR BEHAVIOR OF DNNS TRAINING
For shallow models, the GSNR of parameters decreases in the whole training process because gra- dients become small as learning converges. But for DNNs it is not the case. We trained DNNs on the CIFAR datasets and computed the GSNR averaged over all model parameters. Because E (BD,)) = # + 87 and we assume n is large, E (BD,)) = gy. In the case of only one large training datasets, we estimate GSNR of I-th iteration by
Tj R3 BD.,.// Pb..,
(25)
As shown in Figure E. the GSNR starts out low with randomly initialized parameters. As learning progresses, the GSNR increases in the early training stage and stays at a high level in the whole learning process. For each model parameter, we also computed the proportion of the samples with the same gradient sign, denoted as Psame_ sign. Figure He, we plot the mean time series of this proportion for all the parameters. This value increases from about 50% (half positive half negetive due to random initialization) to about 56% finally, which indicates that for most parameters, the gradient signs different samples become more consistent. This is because meaningful features begin to emerge in the learning process and the gradients of the weights on these features tend to have the same sign among different samples.
Previous research (Zhang et al,, 20TG) showed that DNNs achieved zero training loss by memorizing training samples even if the labels were randomized. We also plot the average GSNR for model trained using data with randomized labels in Figure g and find that the GSNR stays at a low level throughout the training process. Although the training loss of both the original and randomized labels go to zero (not shown), the GSNR curves clearly distinguish between these two cases and reveal the lack of meaningful patterns the latter one. We believe this is the reason why DNNs trained on real and random data lead to completely different generalization behaviors.
[n]
[4]
Laican Moall
2008 ansae Now
855
5
0:50
0.02
130
120
100
200
300 500 800
Figure 4: (a): GSNR curves generated by a simple network based real and random data. An obvious upward process in the early training stage was observed for real data only. (b): Same plot for R8,Net18. (c): Average of Psame_sign for the same model in (a).
3.2 TRAINING DYNAMICS BEHIND THE GSNR BEHAVIOR
In this section we show that the feature learning ability of DNNs is the key reason why the GSNR curve behavior of DNNs is different from that shallow models during the gradient descent training. To demonstrate this, a simple two-layer perceptron regression model is constructed. A synthetic dataset generated as following. Each data point constructed i.i.d. using y ToF1 + e, where Tt and 21 are drawn from uniform distribution [-1,1] and is drawn from uniform distribution [-0.01, 0.01]. The training set and test set sizes are 200 and 10,000, respectively. We use a very simple two-layer MLP structure with 2 inputs, 20 hidden neurons and I output.
We randomly initialized the model parameters and trained the model on the synthetic training dataset. As control setup we also tried to freeze model weights in the first layer to prevent it from learning features. Note that a two layer MLP with the first layer frozen is equivalent to a linear regression model. That is, regression weights are learned the first layer. We plot the average GSNR of the second layer parameters for both the frozen and non-frozen cases. Figure 5 shows that in the non-frozen case, the average GSNR parameters of
the second layer using fixed features extracted by
7

---

Published as a conference paper at ICLR 2020
a.00.00
(b)
010
Undinege firz Freaze liss: tayer
Fram tas sebreete
Preace
Setre3
0.025
0.2
0.05
0.020
0.15
Averge 65NA
3.01%
0.05
CONN
0.05
0.04
0.01
0.02
0.000
0.00
0.00
500
750
1000
500
750
1000
900
1000
Oooch
Figure 5: Average GSNR (a) and loss (b) curves for the frozen and non-frozen case. (c): GSNR curves of individual parameters for the non-frozen case.
the second layer shows significant upward process, whereas in the frozen case the average GSNR decreases in the beginning and remains at low level during the whole training process.
In the non-frozen case, GSNR curve of individual parameters of the second layer shown in Figure The GSNR for some parameters show a significant upward process. To measure the quality of these features, computed the Pearson correlation between them and the target output y. both the beginning of training and at the maximum point of their GSNR curves. We can see that the learning process learns "good" features (high correlation value, i.e. with stronger correlation with y) from random initialized ones, as shown in Table This shows that the GSNR increasing process is related to feature learning.
3.3 ANALYSIS OF TRAINING DYNAMICS BEHIND DNNs' GSNR BEHAVIOR
In this section, will investigate the training dynamics behind the GSNR curve behavior. In the case of fully connected network structure, we can analytically show that the numerator of GSNR, iee. the squared gradient mean of model parameters, tends to increase the early training stage through feature learning.
Consider fully connected network, whose parameters are 0 = (W()), b(1) ..... W(maaz) (imnea))) b(lma)) where (1)), ((2) are the weight matrix and bias of the first layer, and so on. We denote the ac- tivations of the I-th layer as (1) {a. (g(())), where is the index for nodes/channels of this layer, and ((--) is the collection of model parameters in the layers before 1, i.e. gil-)
(W(1), (1)) .... W([11) b([-)). In the forward pass data sample i, (a, is multiplied by the weight matrix W():
0,c = Σ Wll (0
(26)
where ( ] is the output of the matrix multiplication, for the i-th data sample, on the I-th
( layer, c = 21, C} is the index of nodes/channels in the (I + 1)-th layer. We use gD to denote
the average gradient of weights of the 1-th layer W(), i.e. (!) = 4 Σ Sh. where L; is the loss of the i-th sample.
Here we show that the feature learning ability of DNNs plays a crucial role in the GSNR increasing process. More precisely, we show that the learning of features (Q(t-)) i.e. the learning of pa- rameters g(l-) tends to increase the absolute value of gD Consider the one-step change of gradient (4)
(i)
mean Ago gp.++ gp, with the learning rate À + one training step, is updated by AO = 0 0: = (f). Using linear approximation with X + 0, we have
(1)
(1)
0g (1)
Ago
(1)
AO,
Σ
AO,
dg..s.c
AO,
(27)
002
002
00,
8, C
e, E
where ((1-) and ((++) denote model parameters before and after the 1-the layer (including the (-th), respectively.
8

---

Published as a conference paper at ICLR 2020
(1) We focus the first term of eq. (27)), i.e. the one-step change of gD caused by learning 0(--) Substituting BD
!!) 1 E==1
aL; awas and Adj = 2-> Ei=1 8L, into eq. (27), we have
80,
ÖL, da 1² other terms ao 00;
11) X
Ag DD,sc
W 8g E
Σ
(28)
2
The detailed derivation of eq. (2S)) can be found in Appendix B We can see the first term (which is summation over parameters in ((l-) in eq. (28) has opposite sign with wi!.)
s.c. This term will make Agn negatively correlated with W s.c. We plot the correlation between Ag,,s. with w 8.c for model trained on MNIST for 200 epochs in Figure Ea. In the early training stage, they are indeed negatively correlated. For top-10% weights with larger absolute values, the negative correlation is even more significant.
8,C tends to increase the ab- Here show that this negative correlation between AgD and W!!
(1)
solute value of gD through an interesting mechanism. Consider the weights !! with {w! > (!) O, gD,.s..c < 0}. Learning 6 would decrease gD,,,c and thus increase its absolute value because the first term in eq. (28) is negative. On the other hand, learning w!! would increase w and its absolute value because Aw!!) -Ag is positive. This will form positive feedback process,
(1)
(8)
in which the numerator of GSNR, (g) (gh,..))² would increase and so the GSNR. Similar analysis (1)
can be done for the case with {W!) < 0, 0.gDD..c> 03.
On the other hand, when [W!) D.e.c > 0}, we show that the weights tend to change into the earlier
case, i.e. [W!!) s.c.D,s.c < 0} during training. Consider the case of (W > 0, gD,..c > 0), the first term in eq. (28)) is negative, learning (((-) tends to decrease (1) gD,,,e or even change its sign. Another posibility is that learning ( s,e changes the sign of w!! s,c because A AW!!) 8,0 = ->gD,s,c is negative.
In both cases the weights change into the earlier case with (W!)g e. < 0}. Similar analysis can be done for the case of (W!!) 0, < 0).
Therefore (W!)), (1)
s,,.D < 0} is a more stable state in the training process. For a simple model trained on MNIST, We plot the proportion of weights satisfying {W (1) (1)
W....... 0} in Figure B s,s,,,, 0} than the opposite. Because weights with small absolute value easily change sign during training, we also plot this proportion for the top-10% weights with larger absolute values. We can see that for the weights with large absolute values, nearly 80% of them have opposite signs with their gradient mean, confirming earlier analysis. For these weights, the numerator of GSNR, (B)),.)) tends increase through the positive feedback process discussed above.
and find that there are indeed more weights with {w!. (1)
Table 1: Pearson correlation between features and target output y, where cr and Crman are correlations at the begin- ning of training and maxi- mum of GSNR curve respec- tively.
AB amglis
2.4
60-
0.4
10% Mawatnnnnnn WT
0
29
50
350
322
230
175
200
30
220
130
125
200
feature id
Ctg
Ca
0
-0.11
0.47
Figure 6: MNIST experiments. Left: Correlation between Agh and (: their gradient mean.
5
0.11
0.44
13
0.07
0.40
s,c. Right: Ratio of weights that have opposite signs with
14
-0.21
-0.27
17
-0.33
0.53

---

Published as a conference paper at ICLR 2020
4 SUMMARY
In this paper, we performed a series of analysis on the role of model parameters' GSNR in deep neural networks' generalization ability. We showed that large GSNR is key small generalization gap, and gradient descent training naturally incurs and exploits large GSNR as the model discovers useful features in learning.
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv: 2017.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach, 2018. 1rX.v:1802.05296.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463-482, 2002.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference Machine Learning-Volle Vollum 70, pp. 1019-1028. JMLR. org, 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 178. 2017.
Alex Graves. Agenerating sequences with recurrent neural networks, 2013. Xrv: 1.w:1330..800x
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240 2015.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in Neural Information Processing Systems, pp- 1731-1741. 2017.
Daniel Jakubovitz, Raja Giryes, and Miguel RD Rodrigues. Generalization error in deep learning, 2019.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint / 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-bater training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 048960 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6904 2014
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947-5956.
2017.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: An empirical study. arXiv: 100:::0002200 08760. 2018.
Tom Rainforth, Adam R Kosiorek, Tuan Anh Le, Chris J Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. arXiv preprint arXiv:1802.045577 2018.
Stanislaw Jastrzebski Srini Narayanan Stanislav Fort, Pawe Krzysztof Nowak. Stiffness: A new perspective on generalization in neural networks, 2019. arXiv: (0t.:19991.999911
Vladimir N Vapnik and Ja Chervonenkis. The necessary and sufficient conditions for consistency of the method of empirical risk. Pattern Recognition and Image Analysis, 1(3):284-305, 1991.
10

---

Published as a conference paper at ICLR 2020
Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012. arXiv:1212.5701.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint wrxii:1611.035555 2016.

---

Published as a conference paper at ICLR 2020
A APPENDIX A
A.1 MODEL STRUCTURE IN SECTION 2.4
As shown in Table 2 all models in the experiment consist of 2 Conv-Retu-MaxxPolling blocks and 2 fully-connected layers, but they are different in the number of channels. We choose the number of channels from {6, 8, 10, 12, 14, 16, 18, 20).
Table 2: Model structure On MNIST in Section
p is the number of channels and q = (2.5
Layer
input #channels
output #channels
conv + relu maxpooling conv + relu maxpooling
p
p
q
flatten relu
16 *
10 * 10
relu
10 * q
softmax
A.2 EXPERIMENT ON CIFAR1O
Different from the experiment on MNIST, we use deeper network on CIFAR1O. We also include the Batch Normalization (BN) layer, because we find that it's difficult for the network converge
in the absence of it. The network consists of 4
BeN-Relo-Com--B--Reee
blocks
and fully-connected layers. More details are shown in Table B.
Table 3: Model structure CIFAR10. is the number of channels.
Layer
input #channels
output #channels
conv + bn + relu 3 conv + relu maxpooling
p
p
bn relu conv + bn + relu 2p maxpooling
p
2p
2p
conv + bn + relu
2p
4p
bn relu maxpooling
4p
4p
conv + bn relu
4p
8p
conv + bn + relu
8p
8p
maxpooling
flatten
fc relu fc relu
32 *
8 *
8
8 *
fc softmax
8 * q
10
The experiment is conducted under similar setting as that of MNIST in section ME! We choose n E {2000, 4000, 6000, 8000, 10000), Prandom E {0.0, 0.2, 0.4), ch {6,8, 8, 10, 12, 14, 16, 18). We use the gradient descent training (Not SGD), with small learning rate of 0.001. The left and right hand sides [ at different epochs are shown in Figure M where each point represents one specific combination of the above settings. Note that at the evaluation step of every epoch,
12

---

Published as a conference paper at ICLR 2020
KESCh-26
spoch-100
ch_6
ch_18
ch_8
2000
23 of 19 I0SGI fandtio CORR)
dh_10
4000
10SG8 hancton CSMR)
2910000 setvierr
epoch--c0
c-0.93
spect-IECO
CH0.350
ch_12
6000
1h_14
8000
ch_16
-0.
Figure 7: Left hand (LHS) and right side (RHS) of eq. (IL)) Points are drawn under different
experiment settings. Left figure: LHS RHS relation epoch 20, 100, 500, 1000.
N 6
N_18
8
N_20
N_10
50
N_12
100
300
X N_
600
Figure 8: Similar with Fig. but for a toy regression model discussed in Appendix A.3.
the same mean and variance inside the BN layers the training dataset. That's to ensure that the network and loss function are consistent between training and test.
At the beginning of training, compared to that of MNIST, the data points no longer perfectly resides on the diagonal dashed line. We suppose that's beacuse of the presence of BN layer, whose internal parameters, i.e. running mean and running variance, optimization process, but change their values in a different way. Their change affects the OSGR, yet could not include them in the estimation of OSGR. However, the strong positive correlation between the left and right hand sides of eq. (I9) can always be observed until the training begins to converge.
not regular Icarnable parameters in the
A.3 EXPERIMENT ON TOY DATASET
In this section we show a simple two-layer regression model consists of FC-Relu structure with only 2 inputs, hidden layer with N neurons and output. similar synthetic dataset with the training data used in the experiment of Section 8.2 is generated as follows. Each data point is constructed i.i.d. using y = To + e, where vp and xı are drawn from uniform distribution of []1, and is drawn from uniform distribution of [-Mmaise; Thoise].
To estimate eq. ((4)), we randomly generate 100 training sets with n samples each, ie. M=100. and test with 20,000 samples. To cover different conditions, we (1) choose
(50, 100, 300, 600, 1000, 2000, 6000]; (2) inject noise with Mhoise E {0.2, 4, 6, 8}; (3) perturb
model structures by choosing N (6, 8, 10, 12, 14, 16, 18, 20). We gradient descent with leam- ing rate 0.001.
Figure shows similar behavior Fig. B. During the early training stages, the LHS and RHS of eq. (19) are very close. Their highly correlated relation remains until training converges, whereas the RHS of eq. (IT) decreases significantly.
13

---

Published as a conference paper at ICLR 2020
B APPENDIX B
Derivation of eq.
(28)
(1)
(1)
Σ
A0j + other terms
(29)
303
0g E00
0 1 Σ i=1
BL.
n2
aw!!
1
aL;
+ olher terms
(30)
00,
(ycan-)
1=1
0o (1)
0 1 Σ
00.
aL,
aw!
do..
dd
(1)
+ other terms
(31)
003
do do
da
a0;
0g EB(-)
0 O a
aL. (
1=1
da
da
W
+ other terms
(32)
n
00j
Io
o'
30,
Öhg d
LL,
Will
oL,
0a
Do!0
00,
;0 00,
(1)
0o (I) 00,
8SEA0-) iml
+ other terms
(33)
()
Above we used the first term of eq.
W( oer and
0o.
a that can both be derived from eq. (26) Consider
(1)
(a
ow!!)
When 5' =, C. have
OL, da
Ag.
2 + other terms
(34)
n²
do 00,
(,E0()
Note that the term related to
aLL. do 00,
i,
and the terms when s' * sor + cn eq. H are merged
into other terms of eq. ((24).
14

---

Published as a conference paper at ICLR 2020
C
APPENDIX C
Notations
2
A data distribution satisfies A' x y
or (z,
A single data sample
D
Training set consists of n samples drawn from Z
D'
Test set consists of n' samples drawn from
0
Model parameters, whose components denoted 0z
(0) or (8)
Parameters' gradient w.r.f. single data sample or (24, 9i)
8(0)
Mean values of parameters' gradient over a total data distribu- tion, i.e., (g.z(g((0))
(D)
Average gradient over the training dataset, i.e., Ei= (.)
(0)
Average gradient over the test dataset, i.e., 1 n² Ei=1g/ Note that, in eq. (G)), we assume n' n
gD,j
Same as gz
(0)
Variance of parameters' gradient of a single sample, i.e. Var (0)
Same p²(z))
(0)
Variance of the average gradient training dataset of size n., i.e., Varp (a [0)
of
Same o²(0)
ry (ror((0))
Gradient signal noise ratio (GSNR) of model parameter 0,
L[D]
Empirical training loss, i.e., 1 Ei= Lyy, f(2,, 0))
T(D')
Empirical test loss, i.e,, 1 n7 Em T.y,, (x),
[T[D]
One-step training loss decrease
AL
One-step training loss decrease caused by updating one param- eter 0j
R(3,n)
One-step generalization ratio (OSGR) for the training and test sets of size n sampled from data distribution 2, i.e. Ep.p'>gn (AL(D") =
A
Learning rate
V
One-step generalization gap increment, i.e., (T AT.
6
Random variables with zero mean and variance x²(
W0 and (0)
Model parameters (weight matrix and bias) of the I-th layer
((-)
Collection of model parameters over all the layers before the [-th layer
gD
(1)
Average gradient of (!)) over the training dataset
g(++)
Collection of model parameters over all the layers after the (-th layer, including the (-th layer
a()) {a! (A(-)))
Activations of the 1-th layer, where s (1, is the index of nodes/channels in the 1-th layer.
( (o0
Outputs of matrix multiplication of the 1-th layer, where c {1, 2, ..., C} is index of nodes/channels in the (1 + 1)-th layer.
(0) and of evaluated on data sample
0, and o
15


Reviews:
[{"review_id": "rye6IOi42B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary\n\nThis paper introduces a quantity termed the \"one-step generalization ratio\" . They derive approximate relations between OSGR and GSNR then show experimental results demonstrating the validity of these approximations, thus linking GSNR and a quantity related to generalization. They investigate the empirical value of GSNR during training of a neural network on Cifar10 with real labels vs. random labels. The final section derives a relation which attempts to explain the correlation between the size of the expected gradient and the learning of features.\n\nReview\n\nFirst, I should note that I am not well-versed in this specific line of work (GSNR ,as other papers were cited with this concept), and thus I cannot readily assess whether the contributions are \"well-placed in the literature\".\n\nFor the derivations and theory, I checked as much of the derivations as possible, although I'm not familiar with all the assumptions used throughout, but they seem relatively reasonable and supported by the experiments. I.e. 2.3.1 seems reasonable at the start of training. But I'm not sure how to evaluate some of the limiting arguments, like taking the step size to zero. \n\nThe paper addresses an important topic and is interesting. The arguments seem well reasoned and logical. The experiments addressed the veracity of of the approximate relations derived in the first section as well as some interesting trends related to the GSNR quantity during training of some simple models. \n\nSmall Concerns \n\nOn line 188 it is mentioned that GSNR is the \"reason\" for a particular phenomenon or explains a phenomenon. It certainly seems reasonable that it is correlated or an indicator of the underlying phenomenon, but it doesn't seem (at least intuitively) that GSNR is the cause or reason for the generalization concepts being examined.  \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxvaNRhFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors suggest a new point of view on generalization through the lens of the distribution of the per-sample gradients. The authors consider the variance and mean of the per-sample gradients for each parameter of the model and define for each parameter the Gradient Signal to Noise ratio (GSNR). The GSNR of a parameter is the ratio between the mean squared of the gradient per parameter per sample (computed over the samples) and the variance of the gradient per parameter per sample (also computed over the samples). The GSNR is promising as a measure of generalization and the authors provide a nice leading order derivation of the GSNR as a proxy for the measure of the generalization gap in the model. After the derivation, experimental results on MNIST are presented and suggest that empirically there is a relation between the generalization gap of neural network trained by gradient descent and the GSNR quantity given in the paper. Next the author analyze the GNSR of DNNs as opposed with shallow models or other learning techniques and observe that the GSNR differs when using random labels (lower GSNR) as compared with true labels and exhibits different behavior along training for DNNs and gradient descent.\n\n Assumption 2.3.1 is not necessarily the most realistic in the current training paradigm since multi-epoch training indeed separates the training and testing per-sample gradient distributions significantly. \n\nOverall the paper gives a fresh (as far as I know) and nice idea on generalization of neural networks. The derivation requires quite a few stringent assumption (the leading order analysis, on the step size, assumption 2.3.1 on the test and train gradient distributions) the experiments do suggest that the theory is valid to an extent, especially during the early parts of training. In contrast, the presentation of the paper distracts from the work and needs additional cleaning up. Also, the experimental section 2.4 would benefit additional empirical analysis on other datasets  and additional experiments, as well as more thorough explanation of the experiments. The writing of the paper needs additional proofreading as currently it is easy to spot typos and grammar errors along the paper. I currently vote weak reject, for solid content and not-quite-ready presentation.\n\nAdditional experiments and careful proofreading should definitely enhance the paper and get it to the level of publications, so I am willing to change my decision if the authors improve the writing and the overall presentation. I like the idea presented in the paper and encourage the authors to resubmit a more tidy draft.\n\nsmaller presentation issues and typos/grammar issues:\n\nFigure 1: name X and Y labels with more meaningful names instead of RHS,LHS\nFigure 3(c): add legend\nline 64: consists >>> which consists\nline 73: we refer R(z) >>> we refer to R(z)\nline 79 futher >>> further overall sentence needs more work\nline 92 distribution becomes >>> distributions become \nline 100: using common notation summing over n samples, the sum should possibly start from i=1 to n instead of i = 0 to n\nline 132 include >>> includes (also possibly rephrase sentence)\nline 136 vefity >>> verify \nline 137: M number >>> M  \nline 157 the data points closely distributed >>> the data points are closely distributed\nline 162 thorough analytically >>> thorough analytical \n\n\n\n\n-------- Update After Rebuttal --------\n\nI have read the comments the authors made and reviewed the paper's revision. \nThe presentation has improved a lot (Even though there is still room to go). \nNonetheless at this time I choose to update my score to a weak accept, as I think the authors bring a fresh (as far as I know) and interesting idea with regards to empirical generalization.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lyj_jm5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper defines the quantity of \"gradient SNR\" (GSNR), shows that larger GSNR leads to better generalization, and shows that SGD training of deep networks has large GSNR. It tells a great story on why SGD-trained DNNs have good generalization.\n\nThis topic is highly relevant to this conference.\n\nHowever, I struggle to rate this paper, since I feel swamped with math. It is hard work to read this paper, and I can honestly say that I could semi-confidently follow until about Eq. (8). To even get there, I had to scroll back and forth to remember the definitions of the various symbols. The math may be very well correct, but it is infeasible to verify (or follow) it fully. It does not make it easier that one cannot really search a PDF for greek symbols with indices etc. Someone who reads theoretical papers all day long might do better here.\n\nThis is the reason I rate the paper Weak Reject.\n\nSome feedback points:\n\nSection 2.1:\n\nEq. (1): It seems the common definition of SNR is the ratio of mean standard deviation. Your SNR is its square. This should be explained.\n\nI think it would help the reader a lot to give some intuitive meaning to the GSNR value. Can you, in Section 2.1, explain with examples what typical (or extreme) values would be?\n\nAssumption 2.3.1:\n\nThis is dropped on the reader without any motivation. It is also confusing: \"we will make our derivation under the non-overfitting limit approximation\" conflicts with \"In the early training stage,...\" So is this whole derivation only true in the early stages?\n\nAssumption 2.3.1 seems to address a thought I had when reading this: At the end of the training, I would expect mu_q(theta) to be zero (the definition of convergence). At the start, it is arbitrary as it entirely depends on the initial values. So this paper must look at some part between the two extremes to make sense. Is it? Is this assumption related?\n\nWhat is the difference between \\sigma and \\rho? Seems one is on the data distribution and one on a sampled set. But then why is \\mu the same in both cases (Eq. (1) vs. Eq. (5))?\n\nAll plots:\n\nThe plot labels are far too small to be readable.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Quoting a reviewer for a very nice summary:

"In this work, the authors suggest a new point of view on generalization through the lens of the distribution of the per-sample gradients. The authors consider the variance and mean of the per-sample gradients for each parameter of the model and define for each parameter the Gradient Signal to Noise ratio (GSNR). The GSNR of a parameter is the ratio between the mean squared of the gradient per parameter per sample (computed over the samples) and the variance of the gradient per parameter per sample (also computed over the samples). The GSNR is promising as a measure of generalization and the authors provide a nice leading order derivation of the GSNR as a proxy for the measure of the generalization gap in the model."

The majority of the reviewers vote to accept this paper. We can view the 3 as a weak signal as that reviewer stated in his review that he struggled to rate the paper because it contained a lot of math.
Decision: Accept (Spotlight)