How gradient estimator variance and bias impact learning in neural networks
Authors: Arna Ghosh
Yuhan Helena Liu
Guillaume Lajoie
Konrad Kording
Blake Aaron Richards
Venue: ICLR 2023 poster (2023)
Keywords: Computational Neuroscience
learning and plasticity
Credit assignment
Imperfect gradient descent
Gradient approximation
Biologically-plausible learning
Neuromorphic computing
Neural networks

Abstract: There is growing interest in understanding how real brains may approximate gradients and how gradients can be used to train neuromorphic chips. However, neither real brains nor neuromorphic chips can perfectly follow the loss gradient, so parameter updates would necessarily use gradient estimators that have some variance and/or bias. Therefore, there is a need to understand better how variance and bias in gradient estimators impact learning dependent on network and task properties. Here, we show that variance and bias can impair learning on the training data, but some degree of variance and bias in a gradient estimator can be beneficial for generalization. We find that the ideal amount of variance and bias in a gradient estimator are dependent on several properties of the network and task: the size and activity sparsity of the network, the norm of the gradient, and the curvature of the loss landscape. As such, whether considering biologically-plausible learning algorithms or algorithms for training neuromorphic chips, researchers can analyze these properties to determine whether their approximation to gradient descent will be effective for learning given their network and task properties.

Published as a conference paper at ICLR 2023
HOW GRADIENT ESTIMATOR VARIANCE AND BIAS COULD IMPACT LEARNING IN NEURAL CIRCUITS
Arna Ghosh
Yuhan Helena Liu University of Washington Seattle, WA, USA Mila-Quebec AI Institute Montréal, QC, Canada
Guillaume Lajoie
McGill University &
Unversite de Montréal & Mila-Quebar AI Institute Montréal, QC, Canada
Mila- Mila-Quebec AI Institute
Montréal, QC, Canada
gho.h @ mail.mcgill.ca
Konrad Körding
Blake Richards
University of Pennsylvania Philadelphia. PA, USA CIFAR Learning in Machines & Brains Toronto, ON, Canada
McGill University, Mila-Quebec AI Institute &
Montreal Neurological Institute
Montréal, QC, Canada
CIFAR Learning in Machines & Brains
Toronto, ON, Canada
blake.riee richards @ megill.ca
ABSTRACT
There is growing interest in understanding how real brains may approximate gradients and how gradients can be used to train neuromorphic chips. However, neither real brains nor neuromorphic chips can perfectly follow the loss gradient, parameter updates would necessarily variance and/or bias. Therefore, there is a need to understand better how variance and bias in gradient estimators impact learning dependent network and task properties. Here, we show that variance and bias can impair learning on the training data, but some degree of variance and bias gradient estimator can be beneficial for generalization. We find that the ideal amount of variance and bias in gradient estimator are dependent on several properties of the network and task: the size and activity sparsity of the network, the norm of the gradient, and the curvature of the loss landscape. As such, whether considering algorithms or algorithms for training neuromorphic chips, researchers can analyze these properties to determine whether their approximation to gradient descent will be effective for learning given their network and task properties.
gradient estimators that have
weaggiially-laassibeee learning
INTRODUCTION
Artificial neural networks (ANNs) typically use gradient descent and its variants to update their parameters in order to optimize loss function (LeCun et al. 2015) Rumelhart et al. 1986). Impor- tantly, gradient descent works well, in part, because when making small updates to the parameters, the loss function's 's gradient is along the direction of greatest reduction Motivated by these facts, longstanding question in computational neuroscience is, does the brain approximate gradient descent (Liliicrap et 2020 Whittingoon & Bogacz 2019h Over the last few years, many papers show that, in principle, the brain could approximate gradients of some loss function (Murray 2019: Liu
al. 2021 Payeur et 2021 Lillicrap al. 2016 Scellier & Bengio 2017). Also inspired by the brain, neuromorphic computing has engineered unique materials and circuits that emulate biological networks in order to improve efficiency of computation (Roy et al. 2019 Li al. 2018b). But, unlike ANNs, both real neural circuits and neuromorphic chips must rely on approximations to the true gradient. This due to noise biological synapses and memristors, non-differentiable operations such spiking, and the requirement for weight updates that do not use non-local information (which can lead to bias) (Cramer Benjamin et al. 2022 M. Payvand et al. 2020b; Laborieux et al. 202T
Shimizu et al. 2021 Nettei et al. | N. R. Shanbhag et al. 2019). Thus, both areas of research
If we assume Fuclidean metric in weight space.

---

Published as a conference paper at ICLR 2023
could benefit from principled analysis of how learning is impacted by loss gradient variance and bias.
In this work, we ask how different amounts of noise and/or bias in estimates of the loss gradient affect learning performance. As shown in a simple example in Fig. 2 learning performance can be insensitive some degree of variance and bias in the gradient estimate, and even benefit from it, but excessive amounts of variance and/or bias clearly hinder learning performance. Results from optimization theory shed light on why imperfectly following the gradient-e.g. via stochastic gradient descent (SGD) or other noisy GD settings-Cue improve generalization in ANNs (Foret et al. 2020 Chaudhari et al. 2019 Yao al. 2018, Ghorbani al., 2019). However, most of these results treat unbiased gradient estimators. In contrast, in this work, we are concerned with the specific case of weight updates with intrinsic but known variance and bias, as often the case in computational neuroscience and neuromorphic engineering. Moreover, we also examine how variance and bias can hinder training. because the amount of variance and bias biologically-plausble and neuromorphic learning algorithms is often at levels that impair, rather than improve, learning (Laborieux et al. 202T. and sits a different regime than that typically considered in optimization theory.
A
Train
B
Test
1.03
103
10
Better than GD
GD Accuracy
Variance
Variance
Accuracy
1.0
1.0
Worse than GD
10
10
0.0
0.0
0.97
0.0
10 Bias
1
0.0
10
10- Bias
10
Figure 1: Train and test accuracy of VGG-16 network trained for 50 epochs (to convergence) on CIFAR-10 using full-batch gradient descent (with no learning rate schedule) with varying amount of variance and bias (as a fraction of the gradient norm) added to the gradient estimates. These results (avg of 20 seeds) indicate that excessive noise and bias harms learning, but a small amount can aid it.
The observations in Fig. give rise to an important question for computational neuroscientists and neuromorphic chip designers alike: what amount of variance and bias in a loss gradient estimate tolerable, or even desirable? To answer this question, we first observe how variance and bias in gradient approximations impact the loss function in single parameter update step on the training data. (We also extend this to multiple updates in Appendix A.) We utilize an analytical and empirical framework that is agnostic to the actual learning rule, and derive the factors that affect performance in an imperfect gradient setting. Specifically, we assume that each update is comprised of the contribution of the true gradient of the loss function with respect to the parameter, fixed amount of bias, and some noise. Similar to Raman et al. (2019), we derive an expression for the change in the loss function after a discrete update step in parameter space: w(t + At) w(t) + (ttt) where At is akin to learning rate in standard gradient descent algorithms. We then characterize the impact on learning using the decrease in loss function under an approximate gradient setting as compared to the decrease in loss function when following the true gradient.
Our analysis demonstrates that the impact of variance and bias are independent of each other. Furthermore, we empirically validate our inferences in ANNs, both toy networks, and various VGG configurations (Vedaldi & Zisserman 2016) trained on CIFAR-10 (Krizhevsky & Hinton 2009). Our findings can be summarized as follows:
1. The impact of variance is second order in nature. It is lower for networks with a threshold non- -ineeriiy, as compared to parameter matched linear networks. Under specific conditions, variance matters lesser for wider and deeper networks.
2. The impact of bias increases linearly with the norm of the gradient of the loss function, and depends on the direction of the gradient as well as the eigenvectors of the loss Hessian.
3. Both variance and bias can help to prevent the system from converging to sharp minima, which may improve generalization performance.
2

---

Published as a conference paper at ICLR 2023
Altogether, results provide guidelines for what network and task properties computational neuroscientists and neuromorphic engineers need to consider when designing and using noisy and/or biased gradient estimators for learning.
ESTIMATING THE IMPACT OF VARIANCE AND BIAS ON TRAINING
Our analysis focuses on situations where the synaptic weights of a network, w, are trained to optimize a loss, C(w). We use an auxiliary variable, i, to denote different points in this optimization trajectory and At to denote single step in this trajectory. (We extend to the multi-step case in Corollary A.2 of the Appendix.) Compared to standard optimization protocols in the ANN literature, At is akin to the learning rate. Therefore, the loss as measured at one particular point in the trajectory is denoted as [[)( and the loss at the next point in the trajectory, ie., after a weight update step, is denoted by C[w(t+ + Δt)]. In the rest of this work. characterize C[w(( At]] [[w(()] function of the variance and bias in the weight update.
We assume that L is twice differentiable and At is small enough such that Liw((t + At]] can be effectively approximated around [[)( using second order Taylor series, i.e., higher order terms beyond the second order can be ignored. Furthermore, we define weight update equation that consists of the gradient, a fixed bias, and some noise. We denote the bias vector as b b,3. where 3 is norm N vector that indicates the direction of bias the N-dimensional parameter space. A special case of B would be B I when all parameters have the same bias. Similarly, we assume white noise in the gradient estimates. With these assumptions, we define the weight update equation to be:
Aw(() == = [[ +b3
+ + oo /o
(1)
where (()((( denotes the first derivative of the gradient of L evaluated at wi(t) and is a unit vector in the N-dimensional parameter space whose elements are zero-mean i.i.d. (see Appendix for generality of this assumption), such that the total variance of Aw(() is o² ((V). where f(N denotes the dependence of variance the total number of network parameters, N. Finally, we define (b,, o²) as the change in [[w((t) when performing the aforementioned weight update as compared to the change in (tw( when the weight update step follows the true gradient, i.e.
Ali(b,02) = [E(uu(( + Ar) [(w((1)( - [L(wo(t - A0) - ((w((())
(2)
where [[[w(t + - (lwr(()) is the change in loss with bias b and variance o². For brevity, we drop the t is the subscript in the rest of the paper. Formally, we present the following Lemma:
Lemma 2.1. Second order Taylor series expansion. Assuming a small learning rafe, the bias, b and variance, a², in gradient estimates can be linked to changes in L upon one weight update step as compared to the change in L under true gradient descent weight update step:
Ea [AC 03)] = Ea [1 + A0] L(w((())))))))) ([10(()](( [f]oo(1 + At) - (11000))
(V (1)) B) At 1
B ((un((()) Ar2
1 (Vo (V Vz
1 ((N At + N
T
(3)
where (:--) denotes the dot product, Tr denotes the Trace operator of square matrix, orange terms indicate the impact of bias, and the green term indicates the impact of variance.
In the next section, we present an in-depth analysis of the impact of variance when f(N) is sublinear, as is the case for some learning algorithms, e.g. Equilibrium Propagation (Laborieux et al. 202T).
We present more general analysis along with the proofs of all lemmas and theorems in Appendix A.
3

---

Published as a conference paper at ICLR 2023
2.1 ANALYSIS OF THE IMPACT OF VARIANCE AND BIAS ON TRAINING LOSS
One of the corollaries that follow from Lemma 2. is that the impact of variance and bias on the expected (b, o²) are independent of each other. Moreover, it is clear that the impact of variance is directly proportional to the trace of V (()) i.e., the loss Hessian, and inversely proportional to .N, i... the number of trainable parameters in the network.
Previous work investigating the object categorization loss landscape for feedforward ANNs demon- strated pronounced effect of width, i.c. increasing width leads to smoother loss landscapes (Li et al., 2018a), thereby leading to lower trace of Hessian (Hochreiter & Schmidhuber 1994). This empirical observation in deep ANNs implies that increasing the width both lowers the trace of the Hessian in addition to its obviously increasing the number of trainable network parameters. Therefore, we can say directly from Lemma 2.1 that the impact of variance is lower for wider networks. However, the role of depth is more complicated. Notably, increasing depth (with no skip connections) could theoretically make the loss landscape less smooth (Li et al. 2018a). Thus, we begin by analyzing the case of increasing depth in linear feedforward networks. We prove that, in fact, increasing depth
leads to lower values for Tr thereby implying a lower impact of variance.
N
Theorem 2.2. Increasing depth lowers impact of variance For linear feedforward ANNs, the impact of variance (0, o²) for a (L + 1) layer network is less than that of L layer network, i.e.,
(O, d² -lye)) S AL(0,, o² (CC(0,2)),-taa t.-tayer
(4)
where each layer has d units and weights initialized by the Xavier initialization strategy and the total variance in gradient estimates does not grow with the size of the network, i.e., f(N) is some constan
Although these assumptions on layer weights may not strictly hold over the course of training, empirical experience suggests that there exists loose correspondence among these quantities if the conditions hold true during initialization. Taken together, overparameterization in either width or depth leads to a lower impact of variance in gradient estimates on the network's training.
Besides the network architecture, the non-linearity used in the network also impacts the function implemented by the network, and therefore affects the trace of the loss Hessian. Specifically, we demonstrate that the trace of the loss Hessian is lower for networks with a threshold non-linearity with gain less than or equal to 1, such as a ReLU operation, as compared to a linear network. Formally,
Theorem 2.3. ReLU lowers impact of variance Let o(.) denote a threshold non-linearity function. The impact of variance on (0, o²) for a network with such non- non-irirassstes is less than that of an equivalent linear network, i.e.
AC,(0,0") S (O, ) Lânear
where the gain of ((.) is less than or equal to (e.g. ReLU).
Summarily, the standard practices in deep neural networks, like overparameterized networks or the use of ReLU non-linean oo-liinerr, lead to lower impact of variance the network's training. Interestingly, brains are also arguably overparameterized and characterized by threshold non-linearities that lead to sparse activations. These properties could offer potential solution to countering any negative impact of variance in gradient estimates in the brain and allow biological agents to reliably train complex tasks despite not being able to actually conduct true gradient descent (Lillicrap al. 2020). It also suggests neuromorphic chips could tolerate more noise in gradient estimators if they are made larger.
In contrast to the impact of variance, the impact bias is more nuanced and depends on the norm and direction of gradient, the direction of the bias vector, as well as the eigenvectors of the loss Hessian. Using a few algebraic manipulations, Lemma 2. I can be extended show that:
Theorem 2.4. The impact of bias on (L, 0) grows linearly with the norm of the gradient.
ACr(b, 0) - = + CblV(CCu(()l)lP
(6)
2
where P, Q are terms that are independent of the norm of the gradient, V C(w(()]
"Check Appendix A for extension the general case where variance changes with number of parameters.
4

---

Published as a conference paper at ICLR 2023
This result implies that initialization and architectural considerations could play important roles in mitigating the impact of bias on training by limiting the norm of the gradient. In the next section, we will empirically validate these results in linear and shallow networks trained on toy datasets well as in VGG networks trained on CIFAR-10.
2.2 EMPIRICAL. VERIFICATION OF THE VARIANCE AND BIAS RESULTS
Our experimental setup (illustrated in Fig. is motivated by Raman et al. 2019), and follows the standard student-teacher framework. A "teacher" ANN is initialized with fixed parameters. The task is for a randomly initialized "student" network to learn the input-output mapping defined by the teacher. We defer the reader the Appendix B for experimental details.
A
Teacher Network
B
weights
Approx gradient
Teacher
True gradient
Output
Loss (C)
Error
Input
AC,(b,o')
Student
Output
AW
Learnnng Rule
update
Student Network
Figure Experimental setup for empirical validation of our results, focusing on AL, (b, o²) to understand how gradient approximations impact learning in one update step. (A) The student-teacher framework motivated by Raman et al. 2019). (B) Illustration of A o²) (defined in Eq.
First, we validate our analytical results pertaining to the impact of variance, i.e., Theorems 2.2 and 2.3
in relatively shallow (1-8 hidden layers) fully connected ANNs. We fix the teacher network archi- tecture and use ReLU non-linearity to threshold the intermediate layer activations. Subsequently, vary the depth and width of the student network with no non-linearity and observe AL, o²) at different points in the loss landscape. For each such setting, we add a ReLU non-linearity to the student network to validate the effect of threshold non-linearity on (0, o²). We plot the mean (O, o²) across different points in the loss landscape in Fig. Note that in order to use a sample accurately reflecting the loss landscape that would be encountered during learning trajectory, we train a control network with the same architecture and non- lineariy as the student network using the true gradient and evaluate A., (0, o²) for each update step along the learning trajectory. In doing so, we are able to observe the desired quantities across wide range of gradient norm values.
+.
+ 2 layers
A 10
B
0'2022
4 layers
0'=033
+ 6 layers
o'=0.4
10-10
8 layers 10-10
(..(0.a²)
-
10
10-11
10
10-12
10
100
10a
10
100
Network width
Number of parameters
Network width
Figure 3: Impact of variance is lower for wider, deeper, and ReL.U networks. Solid lines indicate linear networks, dashed lines indicate networks with ReI.U non-linearity, and dotted line indicate the teacher network architecture. (A) Varying hidden layer width in one hidden layer linear ANNs, keeping input/output dimensions same. (B) Linear and ReLU networks with varying width and depth plotted with number of parameters in x-axis. (C) Same B but plotted with network width in x-axis.
5

---

Published as a conference paper at ICLR 2023
VGG networks trained on the CIFAR-10 dataset. Specifically, we maintain the student-teacher framework and fix the teacher network to be a VGG-19 network trained on CIFAR-10 to 92.6% test accuracy, and family as student networks (with variance set to x² 20). For each network, we use both random weight initialization and imagenet-pretrained weights determine whether initialization has an effect on the impact of variance, Fig. demmosttrass that our results pertaining to depth and width hold for deep convolutional neural networks (CNNs).
Furthermore, we validate Theorems 2.2 and 2.3
different networks from the VGG
Similarly, we validate Theorem 2.4, first in shallow fully connected ANNs and subsequently in deep CNNs trained on CIFAR-10 10. Specifically, we demonstrate in Fig. 4 that the impact of bias (with b = 0.02) on performance of a linear shallow network grows with the norm of the gradient. Note that the sign duality of (b, 0) in this figure follows from Theorem 2.4 where the quantities P and Q depend on the direction of the bias vector with respect to the direction of gradient and the eigenvectors of the loss Hessian (see proof in Appendix for more details). Therefore, bias can help or hinder learning depending on its direction. This behaviour is in contrast to variance, which always hinders learning for convex loss (Lemma
Nonetheless, is worth noting that when the gradient norm is small bias increases it always hinders training (see Fig. B)).
A
B 0.06
02
0.06
0E,lb00
4.1-
0.02
+blaz 0.20
*biee 0.35
<ias 0.50
-0.04
7
0.5
19C1
[VC)]
Figure 4: Impact of bias grows linearly with gradient norm and has quadratic relationship with the amount of bias when the gradient norm tends i.c., validating expression from Theorem 2.4 (6, 0) = + with the slope being the amount of bias (see second term in equation). quadratically with amount of bias when gradient norm is small (see first term in equation).
(A) Impact of bias grows linearly with gradient norm,
(B) Impact of bias grows
Following this, measure the absolute value of AL, (b, 0) and plot its mean across multiple update steps in VGG networks trained on CIFAR-10. In Fig. 6 we show that the relation stated in Theorem 2.4 holds across different VGG architectures, irrespective of the weight initialization. Furthermore, deeper VGG networks have lower impact of bias owing to a lower gradient norm.
A
two-way vep configuration "p- 1.87e-11
B
one-way "p-
ac,(0,0'=20)
*ggof vgy13 vgg16 vgg19 vggft vag13 vggfé 1g
16
untrained
pretrained
factor of downsampling on vgg19
Figure Impact of variance lower for deeper and wider VGG networks when training on CIFAR-10. 10. (A) Deeper VGG configurations, both imagenct--reeraiin
untrained, exhibit lower impact of variance. (B) Wider VGG-19 configuration networks exhibit lower impact of variance. Factor of downsampling indicates the reduction in the number of filters in convolutional layers and dimensionality of intermediate fully connected layers compared to the original VGG- configuration.

---

Published as a conference paper at ICLR 2023
twc-way ANOVA vag confii "p 2.05e-5
16-6
6.5
[ac,(b=0.02.00 011
5-
3
4.5
vgg11
vgg pretrained
vgg11 untrained
vggt1 13
untrained
pretrained
Figure 6: Impact of bias is lower for (A) deeper VGG networks when training on CIFAR-10. (B)
Deeper VGG networks exhibit lower gradient norm, thereby mitigating the impact bias in training.
ESTIMATING THE IMPACT VARIANCE AND BIAS ON GENERALIZATION
In the previous section, studied how variance and bias in gradient estimates impacts training. However, from a machine learning perspective, an important question is to understand the impact on the system's generalization. Specifically, we ask the question: under what conditions could variance and bias in gradient estimates aid generalization performance? Interestingly, current deep learning practices rely on not following the exact gradient of the loss function train models, which have been demonstrated to help generalization performance. Some common examples of such practices include stochastic gradient descent (SGD) and dropou
In this section, we use our framework to understand how variance and bias in gradient estimates could alter the learning trajectory, thereby impacting generalization. Specifically, we investigate the conditions under which parameter update would necessarily lead to descent the loss landscape. This matters for understanding generalization because the flatness the loss landscape a good proxy for generalization (Baldassi et al. 2020 Jiang al. 2019 Sankar et al. 2021 Tsuzuku et al., (020,Peezze et al., 2021:: : flatter minima tend to have better generalization performance than sharper ones. We leverage this viewpoint understand the impact of variance and bias on generalization. Specifically, we investigate the conditions under which gradient approximations could help to avoid sharp minima, thereby promoting convergence wide flat minima.
3.1 ANALYSIS OF THE IMPACT OF VARIANCE AND BIAS DESCENT OF NARROW MINIMA
To quantify the flatness of loss landscape, use eigenvalues of the loss Hessian. Without loss of generality, we assume that the loss Hessian is a normal square matrix, i.e., its eigenvectors form an orthogonal basis of the W-dimensional parameter space. In order to understand the properties of loss minima that the network could potentially converge to, we derive the sufficiency conditions under which parameter update ascends (or descends) the loss landscape. This strategy indicates that noise in gradient estimates would lead to the system not descending the loss landscape when the minima is too sharp, which can aid generalization.
Theorem 3.1. Noise helps avoid sharp minima The sufficient condition under which noise prevents the system from descending the loss landscape, i.e., L Llur(1 Ar) 2 Lluo(t)), is
2
X 2
Al
(TV
where >1 2 Az... 2 AN denote the eigenvalues of 4 L around the minima.
Interestingly, the condition presented in Theorem presents a lower bound for the smallest cigenvalue, i.c., the eigenvalue associated with the direction of least curvature, and connects the variance to gradient norm ratio. This ratio can be thought of as the inverse noise ratio when there is
Discussion the relationship between our analysis and SGD and dropout in the Appendix C.2
7

---

Published as a conference paper at ICLR 2023
no bias in the approximation. Furthermore, we also show in Appendix A that the sufficient condition for descending the loss landscape is similar to the one presented in Theorem 3.1 but provides an Note that we get closer to upper bound for the largest eigenvalue, i... A I 27 (+(
the minimum, the gradient norm would decrease (property of smooth L), and therefore, the inverse noise ratio would increase. Thus, the upper bound on curvature of loss minima that the system can potentially converge to decreases. Taken together, this shows that variance helps the system avoid converging to sharp minima and instead promotes converging wide flat minima.
Bias, on the other hand, has a more nuanced effect on the learning trajectory. This complexity is unsurprising given the dependence of the impact of bias on direction of the bias vector, the direction of the gradient, and the cigenvectors of the loss Hessian, Once again, we derive the sufficiency condition for when a parameter update step would produce a decrease the loss function. It turns out this condition depends on both minima flatness and amount of bias relative to the gradient norm.
Theorem 3.2. Bias prevents descent into local minima dependent on Hessian spectrum. Bias in gradient estimates could prevent converging to minima. Specifically, the sufficient condition for the weight updates to produce a decrease in L when using a biased estimate of the
theegrand gradient is:
2
(8)
|wCll VNNI+ vat VI+ 4b A
V (1+22AM EEAI)
where v/2² Ej X2, i.e., the Frobenius norm of the loss Hessian, and À 2 A2... 2 AN denote the eigenvalues of Ve around the minima and 4 f. a normal matrix.
The condition in Theorem 3.2 indicates that a system that follows a biased gradient approximation will descend the loss landscape until some neighbourhood around minimum (characterized by the minima flatness and relative bias). Inside this neighbourhood, the system may not descend the loss landscape and therefore not converge to the minimum. The condition depends on minima flatness due to the tl term (Eq. (D): sharper (higher curvature) minima should have higher t/s which makes it less likely to satisfy the condition. The condition also depends on the amount of bias relative to gradient norm, i.e., higher relative bias makes it less likely to satisfy the condition for descent, which we also support through simulation (see Fig. (C). Taken together, variance and bias in gradient estimates promote gradient descent dynamics that converge to wide flat minima, which can aid generalization. This effect is empirically demonstrated by our first example of VGG-16 networks trained on CIFAR-10 (Fig.
3.2 EMPIRICAL VERIFICATION OF THE IMPACT OF VARIANCE AND BIAS ON GENERALIZATION
A
B
gradient descent
biased approximation
Trailing eigenvalue
2881
Trailing eigenvalue
288
Empirical likellhood descent
1.0
90
Empirical
20
0.8
0.5
0.5
0.5
20
288
0.5
90
288
0.6 [P|2]]
0.9
1.2
Leading eigenvalue
Leading eigenvalue
0
Empirical likelihood descent
Figure 7: Empirical verification of Theorems 3.1 and 3.2 in linear ANNs. Dashed lines indicate the
theoretical limit and colours indicate the empirical probability of descent. (A) True gradient descent. (B) Noise in gradient estimates, a = ||V LlI. (C) Bias in gradient estimates, ratio VV
b is varied.
We follow the same experimental setting as described in Section 2.2 for verifying Theorems and 3.2 Owing to computational bottlenecks in loss Hessian estimation for high-dimensional nonlinear settings, we restrict our empirical validation to training linear networks for optimizing MSE loss. In doing we gain full control over the loss Hessian via the input covariance statistic Fig. A & demonstrate that our inferences from Theorem 3.I hold empirically. When the leading
for MSE loss minima, the Hessian is equal the input covariance matrix harring constant factor
8

---

Published as a conference paper at ICLR 2023
eigenvalue is less than the theoretical limit, weight update step always leads to decrease in C; when the trailing eigenvalue is greater than the theoretical limit, a weight update step always leads to an increase in L. Furthermore, the theoretical limit is higher for the case when there is no variance than with variance. These observations demonstrate that following a noisy version of the gradient helps the network avoid sharp local minima. Similarly, Fig. C demonstrates that our inferences from Theorem 3.2 hold empirically. Specifically, when the ratio of bias to gradient norm is below the theoretical threshold, a weight update step always leads to a decrease in L.
DISCUSSION
In both computational neuroscience and neuromorphic computing, gradient estimators are known to have variance and bias due to both algorithmic and hardware constraints (Laborieux et al. 202T
M. Payvand et al., 2020a Lillicrp et al.; 2020, Pozzi al. 2018; Sacramento et al. 2018: Rubin
et al.
2021: Roelisema & Holmaat 2018 Bellec et al. 20200 Neftci et al. 2019 Huh & Sejnowski
2018 Zenke& & Neftci (021). p. Using mathematical analysis and empirical verification, we found that
the impact of variance and bias on learning is determined by the network size, activation function, gradient norm, and loss Hessian, such that the amount of variance and bias that are permissible or even desirable depends on these properties. Though our empirical verification was done using feedforward ANNs, our mathematical analysis was formulated to be as general as possible, and we believe that our assumptions are reasonable for most cases that computational neuroscientists and neuromorphic chip engineers face. Furthermore, we add a corollary to Lemma 2.1 and discuss extensions of our analysis to multi-step optimization settings in Appendix A (see Corollary A.2). Thus, our work can inform research in these areas by providing a guide for how much variance and bias relative to the gradient norm is reasonable for given network.
4.1 RELATED WORK
Our analysis was inspired in part by recent work by Raman et al. (2019) characterizing bounds on learning performance neural circuits. They demonstrated that the absence of noise the gradient estimator, larger networks are always better at training. but with noise added, there is size limit beyond which training is impaired. Our work was also informed by research into generalization in deep ANNs, which provided the rationale for analyses examining the potential for a learning algorithm to descend into sharp minima or not (Foret et al., 2020 Chaudhari et al.) 2019 Yao et al. 2018 Ghorbani et al. 2019 Smith et al. 2021). As well, our work has some clear relationship to other work examining the variance of gradient estimators (e.g. Werrel et al. (2003), but here, we are asking novel question, namely, how does given amount of variance and bias impact performance for different network parameters?
More broadly, our work relates the deep learning literature because modern ANNs rarely the true loss gradient over the entire dataset to make weight updates. Instead, it is common practice to add noise in different forms, e.g. SGD (Bottou 2012). dropout (Srivastava et al., 2014). dropconnect
WWa et al., 2013) and label noise (Blanc et al.) 2020 HaoChen et al.) 2021 Damian et al., 2021). But, the noise structure in these scenarios may differ from assumptions in our work. For example, SGD does not exhibit white noise structure (Xie et al. 2021 Zhu et al. 2019). Likewise, dropout can be thought of as adding noise, but for truly unbiased estimate of the gradient, one would have to consider all possible configurations of dropped neurons, something quite uncommon in practice.
4.2 LIMITATIONS AND FUTURE WORK
This work focused on characterizing the impact of variance and bias in gradient estimates, but it lacks any in-depth analysis of specific bio-plausible or neuromorphic learning algorithms. Similarly, our work only characterizes learning performance, but does not provide a concrete proposal to mitigate excessive noise or bias in gradient approximations. As such, our analyses can be used by other researchers to assess their learning algorithms but they do not directly speak to any specific learning algorithm nor provide mechanisms for improving existing algorithms. Nonetheless, we believe that our work provides framework to help develop such strategies, and we leave it to future work.
TThough several analytical studies have used white noise to model SGD dynamics, and concluded that the SGD noise acts regularizer against converging to sharp minima (Li al. 202T) similar to our analyses.
9

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We strongly believe that reproducibility is critical for research progress in both machine learning and computational neuroscience. To this end, we have provided thorough experimental details in the appendix, and in the supplementary materials, we have included the code to run all of the experiments and generate the figures. Our code can also be accessed from the project's github repo We believe that this information will allow the community to validate/replicate our results and further build on our work.
ACKNOWLEDGEMENT
The authors would like to thank Abhishek Banerjee, Jonathan Cornford, and Shahab Bakhtiari for insightful discussions and helpful feedback over the course of the project, as well as Zahraa Chorghay and Colleen Gillon for their aesthetic contribution to the manuscript and figures. This research was generously supported by Vanier Canada Graduate scholarship and Healthy Brains for Healthy Lives fellowship (A.G.); NSERC PGS-D and NSF AccelNet IN-BI N-BIC program, Grant No. DISE-2019976 AM02 (Y.H.L.): NSERC Discovery Grant RGPIN-2018. 1---20811-441,, FRQS Research Scholar Award, Junior LAJGU0401 -253111--888, Canada Research Chair in Neural Computations and Interfacing (G.L..;; NSERC (Discovery Grant: RGPIN-2020-05 105; Discovery Accelerator Supplement: RGPAS- 2020-00031), Healthy Brains, Healthy Lives (New Investigator Award: 2b- 2b-NISU-8) (B.A. CIFAR Learning in Machines and Brains Program (K.K. B.A.R.); Canada CIFAR AI Chair program (G.L. & B.A.R.). This work was enabled by the material support of NVIDIA in the form of computational resources and support provided by Mila.
REFERENCES
Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1): 161-170, 2020.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj. Robert Leg- enstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent net- works of spiking neurons. Nature Communications, 11(1):3625, July 2020. ISSN 2041- 1723. doi: 10. 100-0000-44177-00--11-2-666 6+1467-020-17236->
htttss::::: / www nature. www.articlestiieeoceoooooo..oo.e Number: Publisher: Nature Publishing Group.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an omstein-uhlenheck like process. In Conference on learning theory, pp- 483-513. PMLR, 2020.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp. 421 421-436. Springer, 2012.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):12400 2019.
Cramer Benjamin, Billaudelle Sebastian, Kanya Simeon, Leibfried Aron, Grübl Andreas, Karasenko Vitali, Pehle Christian, Schreiber Korbinian, Stradmann Yannik, Weis Johannes, Schemmel Johannes, and Zenke Friedemann. Surrogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, 119(4))22101191119, January 2022. doi: 10. 2173 21091941011 19. lisher: Proceedings of the National Academy of Sciences.
http::////oi..r///.....o 00211..177.33 1073/ 21091 54119 Pub-
Alex Damian, Tengyu Ma, and Jason Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412. 2020.
10

---

Published as a conference paper at ICLR 2023
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning. pp. 2232-2241 PMLR. 2019.
Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory, pp. 2315-2357. PMLR. 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks, Advances in neural information processing systems, 31, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where find them. International Conference on Learning Representations, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, 2009.
Axel Laboricux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias. Frontiers in Neuroscience, 15, 2021. ISSN 1662-453X. URL https:://////..... //Ww:
Erontiersin org http://atieeeeeeee wtt.://ssi......aattte.eeeee/eee 3389/ fnins 2021. 633674
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553) 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018a.
Yibo Li, Zhongrui Wang, Rivu Midya, Qiangfei Xia, and J Joshua Yang. Review of memristor devices in neuromorphic computing: materials sciences and device challenges. Journal of Physics D: Applied Physies, 500500 September 2018b. ISSN 0022-3727. doi: 10. 101188/6161-666// aade3f. URL httpp://ds doi org. 1088 36 64 23/44/333///33333 aade3f Publishing.
Publisher: IOP
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems, 34, 2021.
Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 13276, November 2016. /tLp:tt::/ddx httpt:::/dxx..i..o.... wwt.:.oorgg100/ 10. 1038//cooms132333
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(6))335-346; 2020.
Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. Cell- type-specific neuromodulation guides synaptic credit assignment in spiking neural network. Proceedings of the National Academy of Sciences, 1 118(51), 2021.
M. Payvand. M. E. Fouda, F. Kurdahi, A. Eltawil, and E. O. Neftci. Error- triggeed Three- Factor Learning Dynamics for Crossbar Arrays. In 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp. 218-222, September 2020a. doi: 10. 1 109/AICAS48895... 2220000009999. Journal Abbreviation: 2020 2nd IEEE International Confer- ence on Artificial Intelligence Circuits and Systems (AICAS).
M. Payvand, M. F. Fouda, F. Kurdahi, A. M. Eltawil, and E. O. Neftci. On-Chip Error- -riigeeri Learning of Multi-Layer Memristive Spiking Neural Networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10(4):522-533 December 2020b. ISSN 2156-3365. doi: 10. WAA.EECCCSS.2000040000000
Alan Murray and Peter Edwards. Synaptic weight noise during mlp learning enhances fault-tolerance. generalization and learning trajectory. Advances in neural information processing systems, 5, 1992.
11

---

Published as a conference paper at ICLR 2023
James M Murray. Local online learning in recurrent networks with random feedback. ELife, 8: e43299, 2019.
N. R. Shanbhag, N. Verma, Kim, A. D. Patil, and L. R. Varshney. Shannon-Inspired Statistical Computing for the Nanoscale Era. Proceedings of the IEEE, 107(1))90-107, January 2019. ISSN 1558-2256. doi: 10. 1090///R...001..... 28699677.
Emre O. Neftci, Charles Augustine. Somnath Paul, and Georgios Detorakis, Event-Driven Ran- dom Back Prop-Prropgttion:: Enabling Neuromorphic Deep Learning Machines. Frontiers in Neuro- science, 11, 2017. ISSN 1662-453X. URL https:::: /wwp//www... front wtw.:rooooorrssorg//oeaeoaaaee ott.s.i.r...//..iicleeee
10. 3389 fnins.nin..... 2017 1.. 00324
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.
IEEE Signal Processing Magazine, 36(6):51-66) 2019.
Antonio Orvieto, Hans Kersting. Frank Proske, Francis Bach, and Aurelien Lucchi. Anticorrelated noise injection for improved generalization. arXiv preprint (ixiv:2202.0288. 2022.
Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst- dependent synaptic plasticity can coordinate learning in hierarchical eircuits. Nature Neuroscience, doi: 10. 00.10388/411933-01-0005555 021-00857-
24(7):1000-0019, July 2021. ISSN 144--7766
URL https
//doi.org/10.10333 103.:/10..338//111111133
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. Advances in Neural Information Processing Systems, 34, 2021.
Isabella Pozzi, Sander Bohté, and Pieter Roelfsema. A biologically plausible learning rule for deep 2018.
learning in the brain. arXiv preprint 181iv:1811..11
Dhruva Venkita Raman, Adriana Perez Rotondo, and Timothy O'Leary. Fundamental bounds learning performance in neural circuits. Proceedings of the National Academy of Sciences, 116 (21):10537-10546, 2019. ISSN 0027-8424.
Pieter R. Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroxcience, 19(3):166-18)) feb 2018. ISSN 14710048. doi: 10. 1038//mm..0118..mm
Thorsteinn Rögnvaldsson. On langevin updating in multilayer perceptrons. Neural computation, (5):916-926, 1994.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based based machine intel- ligence with neuromorphic computing. Nature, 575(7784):607-617,, November 2019. ISSN doi: 10. 1011039941388--011.17727 URL
1476-4687.
1038/
s41586 1177-677-21
Jonathan E Rubin, Catalina Vich, Matthew Clapp, Kendra Noneman, and Timothy Verstynen. The credit assignment problem cortico-basal cunglia-thalamic networks: A review, a problem and a possible solution. European Journal of Neuroscience, 53(7):2234-2255 2021.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(9);333-536, 1986.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Scnn. Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018.
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35:11, pp. 9481-9488, 2021.
Benjamin Scellier and Yoshua Bengio. Equilibrium Propagation: Bridging the Gap between Energy- Based Models and Backpropagation. Frontiers in Computational Neuroscience, 11:24, 2017. ISSN 1662-5188, doi: 10.3389/fmccom.2017..
201..000020.44.4 URL http: http:///wwwwww..... ncbi. nlm. nih. giv.
pmc/ amc/artics wtt.tiiceeee//551555555
12

---

Published as a conference paper at ICLR 2023
Genki Shimizu, Kensuke Yoshida, Haruo Kasai, and Taro Toyoizumi. Computational roles of intrinsic synaptic dynamics. Computational Neuroscience, 70:34 42, October 2021. ISSN 0959-4388. doi:
000.00.. URL https:://wwwtt sci.eneeieee.rerccccc.cmm... com/ seenm/ccincee//
Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations. 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955) 2014.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis. In International Conference on Machine Learning, pp. 9636-9647. PMLR, 2020.
Andrea Vedaldi and Andrew Zisserman. Vgg convolutional neural networks practical. Department of Engineering Science, University of Oxford, 66, 2016.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference machine learning, pp. 1058-1066. PMLR, 2013.
Justin Werfel, Xiaohui Xie, and H Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Advances in neural information processing systems, 16, 2003.
James C.R. Whittington and Rafal Bogacz. Theories of Error Back- Paok-Pronngttion in the Brain. Trends in Cognitive Sciences, 2019. ISSN 1364-6613, doi: ti1s..... 12.005. URL
https: /00 1016/j ticc............................ 2018 005
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference Learning Representations, 2021.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness adversaries. Advances in Neural Information Processing Systems, 31, 2018.
Friedemann Zenke and Emre O Neftci. Brain-inspired learning neuromorphic substrates. Proceed- ings of the IEEE, 109(5):935-950, 2021.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning, pp. 7654-7663. PMLR, 2019.
13


Reviews:
[{"review_id": "v2PoOHsJPb", "reviewer": "Reviewer_27vS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Many methods rely on approximations of the true gradient, which have intrinsic bias and variance. This paper shows how bias and variance of gradient estimators can hinder or improve learning. Although bias and variance can potentially impair learning, the right amount can be helpful for learning and for generalization. The effect of bias and variance seems to be independent of each other, so this paper studies each separately. The paper provides a framework for studying the effect of adding a fixed bias and white noise in the update rule. Given the bias and variance of a learning algorithm, one can use these guidelines to find the network design that facilitates learning. ", "summary_of_the_review": "The work is well-written and well-motivated. The paper tackles a fundamental problem and provides a principled way of analyzing gradient estimators. The proposed framework has substantial results explaining many of the best/standard design practices in deep learning in a grounded way. However, the paper is lacking some mathematical rigor when it comes to notations. The proposed framework has an unclear impact on giving guidelines for the targeted fields, namely computational neuroscience, and neuromorphic engineering, and some arguments are needed to justify how it serves the intended purpose. More arguments and examples are needed to justify how the assumptions are not limiting. In addition, the paper is not well-placed in the literature, which makes the novelty of the work unclear. More careful work is needed to distinguish this work from others. There is also some missing evidence for some claims made by the authors and some imprecisions. Finally, the paper has some experimental inconsistencies that contradict other results but can be fixed easily.\n\nThat being said, I think substantially improving the quality of this paper seems well possible by addressing this feedback. All my comments can be addressed by the authors since they are not affecting the main arguments of the paper. This paper has potential, and I would be willing to change my scores, subject to the authors’ clarifications and incorporating the requested changes.\n", "strength_and_weaknesses": "**Strengths**:\n1. The problem tackled by this paper is fundamental and important. The paper shows a promising direction that can lead to more impactful works.\n2. The paper provides a principled way of analyzing how bias and variance impact learning using a careful setup based on the change in loss with an update having intrinsic known bias and variance compared to the change of the loss using the update based on true gradient.\n3. The paper has substantial results explaining many of the standard design practices in deep learning in a more grounded way. \n4. The paper is well-written, and the problem is well-motivated.\n\n**Weaknesses**:\n1. Unclear notations. The authors used the same notations to write vectors and scalars. Reading these notations would be challenging to follow for many readers. Please consider updating your notations and refer to the notation section in the Formatting Instructions template for ICLR 23.\n2. The framework impact is unclear. The authors mentioned that the case of intrinsic but known bias and variance is often the case in computational neuroscience and neuromorphic engineering. This is the main motivation for their approach. However, the framework provided is limited to specific cases, namely, white noise and fixed bias.\nThe authors argue that their assumptions are reasonable for most cases computational scientists and neuromorphic engineers face, but they don’t provide evidence for their claims. Clearly, this framework provides an important way for analyzing methods such as perturbed gradient descent methods with Gaussian noise, but it’s unclear how it can help analyze other cases. This suggests that the framework is quite limited.\n\t* The authors need to show that their choices and assumption are still useful for computational neuroscience and neuromorphic engineering. This can happen by referring to contributing and important works from these fields having known bias and variance with Gaussian noise.\n\t* In the experiments, the used bias is restricted to having the same magnitude for all weights ($b \\vec{1}$). Can we reproduce the results if we use arbitrary biases? It would be better if the authors tried a number of arbitrary biases and averaged the results.\n3. The paper is not well-placed in the literature. The authors didn’t describe the related works fully (e.g., stochastic gradient Langevin dynamics). This makes the work novelty unclear since the authors didn’t mention how analyzing the gradient estimator was done in earlier works and how their contribution is discernible from the earlier contributions. Mentioning earlier contributions increases the quality of your work and makes it distinguishable from other work. Please also refer to my comment in the novelty section.\n4. Missing evidence of some claims and missing details. Here, I mention a few:\n\t* It’s not clear how increasing the width and/or depth can lower the trace of the Hessian (Section 2.1). If this comes from a known result/theory, please mention it. Otherwise, please show how it lowers the trace.\n\t* The authors mentioned that they use an analytical and empirical framework that is agnostic to the actual learning rule. However, the framework is built on top of a specific learning rule. It’s unclear what is meant by agnostic in this context (Section 1).\n\t* The authors mentioned in the abstract that the ideal amount of variance depends on the size and sparsity of the network, the norm of the gradient, and the curvature of the loss landscape. However, the authors didn’t mention the sparsity dependence anywhere in the paper.\n\t* The authors mentioned in a note after the proof of Theorem A.5 that it is also valid for Tanh but not Sigmoid. However, the proof assumed that the second derivative is zero. It’s unclear whether a similar derivation can be developed without this assumption. However, the authors only mention the relationship with the gain of $\\phi(.)$.\n\t* More information is needed on how the empirical likelihood of descent is computed (Fig. 7).\n\t* The use of MSE should be mentioned in Theorem A.3 since it’s not proven for any loss function. In addition, the dataset notation is wrong. It should be $\\mathcal{D}=\\\\{(x_1, y_1), ..., (x_M, y_M)\\\\}$, where $M$ is the number of examples since it’s a set containing input-output pairs, not just a single pair.\n\t* The argument in Section 2.1 that increasing depth could theoretically make the loss less smooth is not related to the argument being made about variance. It is unclear how this is related to the analyses of how increasing depth affects the impact of the variance. I think it needs to be moved in the discussion on generalization instead.\n5. A misplaced experiment that does not provide convincing evidence to support the theorems and lemmas developed in the paper with less experimental rigor (Fig. 1).\n\t* The experiment is misplaced being at the introduction section. This hurts the introduction and makes the reader less focused on your logic to motivate your work. \n\t* It’s not clear from the figure what the experiment is. The reader has to read appendix B2 to be able to continue reading your introduction, which is unnecessary. \n\t* The results are shown with only three seeds. This is not enough and cannot create any statistical significance in your experiment. I suggest increasing the number of runs to 20 or 30.\n\t* It’s unclear why batch gradient descent is used instead of gradient descent with varying bias and variance. Using batch gradient descent might undesirably add to the bias and variance.\n\t* The experiment results are not consistent with the rest of the paper. We cannot see the relationship when varying the bias or variance similar to other experiments. Looking at Fig.1B where bias=0, for example, we find that adding a small amount of variance reduces performance, but adding more improves performance up to a limit. This is not the case with the other experiments, though. I suggest following the previous two points to make the results aligned with the rest of your results.\n6. Alternative hypotheses can be made with some experiments. The experiment in Fig. 3.A  needs improvement. The authors mention that excessive amounts of variance and/or bias can hinder learning performance. In Fig. 3. A, they only show levels of variance that help decrease loss. An alternative explanation from their figure is that by increasing the variance, the performance improves. This is not the case, of course, so I think the authors need to add more variance curves that hinder performance to avoid alternative interpretations.\n\n**Minor issues that didn’t impact the score**:\n* There are nine arXiv references. If they are published, please add this information instead of citing arXiv.\n* What is a norm $\\sqrt{N}$ vector? Can you please add the definition to the paper?\n* You mentioned that the step size has to be very small. However, in Fig. 1, the step size used is large (0.02). Can you please explain why? Can this be an additional reason why there is no smooth relationship between the values of the variance and performance?\n* No error bars are added in Fig. 4 or Fig. 7. Can you please add them?\n* In experiments shown in Fig. 3 and Fig. 5, the number of runs used to create the error bars is not mentioned in Appendix B.2. \n* A missing $\\frac{2}{|\\mathcal{D}|}$ in Eq. 27.\n* In Theorem A.3 proof, how the input $\\mathbf{x}$ has two indices? The input is a vector, not a matrix. Moreover, shouldn’t $\\sum_k (W_k^{(2)})^2 = 1/d$, not $d$?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W2t7jeLPHS", "reviewer": "Reviewer_Yx5o", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how bias and variance of gradient noise can impact the generalization performance. Specifically, the paper proposes a theory which connects bias and variance to network size, activation functions, gradient norm and Hessian. This theory is shown to coincide with empirical results. ", "summary_of_the_review": "The paper provides an interesting and useful analysis of noise and variance of learning algorithms. The theory is shown to match practice on some learning problems.  I believe the paper makes an interesting and useful contribution and I therefore recommend acceptance.", "strength_and_weaknesses": "Strengths:\n---\n* The theoretical results seem interesting and, importantly, are validated in a series of experiments. \n* The proposed theory may be useful to find neural architectures which work well for a given learning algorithm. \n\nWeaknesses:\n---\n* Figure 1 was not very illuminating except for the sharp phase transition occuring when bias and/or variance get too large. As a red/green-blind person, it was very difficult to tell apart the areas which improve upon GD and it looked more or less random. It is important that these experiments are conducted over many different random seeds and then averaged, to make sure the differences statistically significant. Perhaps the image in Fig. 1 will look less \"noisy\" if more random seeds are considered? If the computational resources are available, it would be great to have a higher-resolution plot. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RT2eMgu9l-9", "reviewer": "Reviewer_HbzZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors in this paper discuss the pros and cons of how variance and bias affect gradient estimator, to better understand how gradients are approximated by brains. The authors argue that while bias and variance generally affect training data learning negatively, some amount of bias, variance can be beneficial for generalization. The authors try to quantify bias/variance in relation to several factors such as size and sparsity of  the network, gradient norm and loss landscape curvature. The authors feel some of these factors can constructively help guide training of biologically inspired networks or neuro chips, especially for designing gradient estimators. The authors propose that bias & variance prevent a network from converging to a sharp minima. This leads to better generalization. THey also show that the imapct of bias has a linear relationship with the norm of the gradient of the loss function, and is influenced by the gradient direction and the loss Hessian's eigen vectors. \nThe authors also show that the impact of variance is to a lesser degree and low for those networks with a threshold non-linearity. They also show that variance is least influential for deep/narrow networks. \n\n", "summary_of_the_review": "My recommendation is to not accept the paper, unless a few revisions are made which are highlighted in the weakness section. This paper illustrates the importance of the impact of variance and bias on gradient estimators and how it impacts learning. While it definitely improves the understanding by proposing mathematical theorems and providing explanations via analysis, it is not clear to me how it is novel and advances the field and how bias or noise and their detrimental impact can be reduced. I think if some of these problems are addressed via additional work, it could be a great paper. ", "strength_and_weaknesses": "Strengths: \n1. Mathematic basis and experimental setup is solid, supported by strong analysis  to demonstrate impact of variance on wider deeper networks which is low\n2. Sound empirical verification of proposed theorems to assess the impact of bias/variance  generalization. \n3. Great summarization of relevant work \n\n\nWeakness: \n1. Empirical evaluation done only on artificial neural networks, could be extended to more architectures. \n2. Does not have any in depth analysis of any biologically focussed learning algorithms (acknowledged by author)\n3. While it acknowledges the impact of bias/noise on learning performance, it does not provide any plans to eliminate noise/variance \n4. There isnt any plan to improve learning algorithms as well in the advent of bias/noise impact on them (acknowledged by author)\n5. There is an assumption that Loss L is twice differentiable but no justification for this assumption. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9eANviFYs6H", "reviewer": "Reviewer_Z5BT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study effects of bias and variance of gradient estimators for training neural networks that is typically done via gradient descent and its variants. In particular, the authors provide novel and intersting theoretical results on the effects of gradient estimators bias and variance on changes in loss function. They further confirm these results by numerical experiments. ", "summary_of_the_review": "Overall, the paper presents novel and strong results on the effects of bias and variance of gradient estimators on training deep networks via gradient methods. The main weaknesses of this paper are (a) missing details as explained under the weaknesses section) (b) slightly overstating the importnace of bias and variance in better generalization results. ", "strength_and_weaknesses": "Strengths:\n(a) The authors study an interesting and generic problem, which will have wide influence on the community, given that training neural networks via stochastic gradient descent methods are at the center of attention in machine learning. \n(b) The authors present strong and insightful theoretical results.   \n(c) The authors validate their results via numerical experiments. \n\nWeaknesses:\n(a) I had difficulty understanding and interpreting some key parts in the paper, in particular:\n      (a1) Early on the authors define delta L in Eq. (2) as difference between loss changes when performing the weights with bias and variance (via E1. (1)) versus when the weights are updated using the full gradient. It is not clear to me whether the delta L reported in Figures 3 to 6 is this quantity or simply the changes on the loss function at each step. If the former is the case, did you update the weights via Eq. (1) with and without bias and variance and then computed the difference in loss functions at every step? \n     (a2)  In Figures 5 and 6 I was not able to find which bias (b) and variance (sigma) these results correspond to. \n     (a3) In Sec. 3.1, the authors claim that updating weights by adding bias and variance could lead to \"flatter\" points w.r.t. the loss function and hence better generalization. In particular, in Theorems 3.1 and 3.2 they provide some conditions for the decrease in loss functions and then they show both theoretically and empirically that by adding bias and variance the decrease in loss function is less likely. However, I am not convinced that this is sufficient for showing that adding bias and variance would lead to better generalization. Perhaps reporting some metrics or the loss value for test sets would strengthen these claims. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster