Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study effects of bias and variance of gradient estimators for training neural networks that is typically done via gradient descent and its variants. In particular, the authors provide novel and intersting theoretical results on the effects of gradient estimators bias and variance on changes in loss function. They further confirm these results by numerical experiments. 
strength_and_weaknesses: Strengths:
(a) The authors study an interesting and generic problem, which will have wide influence on the community, given that training neural networks via stochastic gradient descent methods are at the center of attention in machine learning. 
(b) The authors present strong and insightful theoretical results.   
(c) The authors validate their results via numerical experiments. 

Weaknesses:
(a) I had difficulty understanding and interpreting some key parts in the paper, in particular:
      (a1) Early on the authors define delta L in Eq. (2) as difference between loss changes when performing the weights with bias and variance (via E1. (1)) versus when the weights are updated using the full gradient. It is not clear to me whether the delta L reported in Figures 3 to 6 is this quantity or simply the changes on the loss function at each step. If the former is the case, did you update the weights via Eq. (1) with and without bias and variance and then computed the difference in loss functions at every step? 
     (a2)  In Figures 5 and 6 I was not able to find which bias (b) and variance (sigma) these results correspond to. 
     (a3) In Sec. 3.1, the authors claim that updating weights by adding bias and variance could lead to "flatter" points w.r.t. the loss function and hence better generalization. In particular, in Theorems 3.1 and 3.2 they provide some conditions for the decrease in loss functions and then they show both theoretically and empirically that by adding bias and variance the decrease in loss function is less likely. However, I am not convinced that this is sufficient for showing that adding bias and variance would lead to better generalization. Perhaps reporting some metrics or the loss value for test sets would strengthen these claims. 




clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easy to follow. The authors provide novel and strong results, they further validate these results by empirical experiments. Unfortunately, some missing information regarding the experiments (as mentioned in (a1) under weaknesses above) makes the results irreproducible.  
summary_of_the_review: Overall, the paper presents novel and strong results on the effects of bias and variance of gradient estimators on training deep networks via gradient methods. The main weaknesses of this paper are (a) missing details as explained under the weaknesses section) (b) slightly overstating the importnace of bias and variance in better generalization results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies how bias and variance of gradient noise can impact the generalization performance. Specifically, the paper proposes a theory which connects bias and variance to network size, activation functions, gradient norm and Hessian. This theory is shown to coincide with empirical results. 
strength_and_weaknesses: Strengths:
---
* The theoretical results seem interesting and, importantly, are validated in a series of experiments. 
* The proposed theory may be useful to find neural architectures which work well for a given learning algorithm. 

Weaknesses:
---
* Figure 1 was not very illuminating except for the sharp phase transition occuring when bias and/or variance get too large. As a red/green-blind person, it was very difficult to tell apart the areas which improve upon GD and it looked more or less random. It is important that these experiments are conducted over many different random seeds and then averaged, to make sure the differences statistically significant. Perhaps the image in Fig. 1 will look less "noisy" if more random seeds are considered? If the computational resources are available, it would be great to have a higher-resolution plot. 


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and was easy to read. 
Quality & Novelty: While some related analysis may exist in related literature, the confirmation of the theory with experiments makes an overall compelling study.
Reproducibility: It is not clear if the results are reproducible and it seems important for the code to be released.

Minor comment:
- \hat n is a unit-length vector? unit-vectors are of referred to as basis elements

summary_of_the_review: The paper provides an interesting and useful analysis of noise and variance of learning algorithms. The theory is shown to match practice on some learning problems.  I believe the paper makes an interesting and useful contribution and I therefore recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors in this paper discuss the pros and cons of how variance and bias affect gradient estimator, to better understand how gradients are approximated by brains. The authors argue that while bias and variance generally affect training data learning negatively, some amount of bias, variance can be beneficial for generalization. The authors try to quantify bias/variance in relation to several factors such as size and sparsity of  the network, gradient norm and loss landscape curvature. The authors feel some of these factors can constructively help guide training of biologically inspired networks or neuro chips, especially for designing gradient estimators. The authors propose that bias & variance prevent a network from converging to a sharp minima. This leads to better generalization. THey also show that the imapct of bias has a linear relationship with the norm of the gradient of the loss function, and is influenced by the gradient direction and the loss Hessian's eigen vectors. 
The authors also show that the impact of variance is to a lesser degree and low for those networks with a threshold non-linearity. They also show that variance is least influential for deep/narrow networks. 


strength_and_weaknesses: Strengths: 
1. Mathematic basis and experimental setup is solid, supported by strong analysis  to demonstrate impact of variance on wider deeper networks which is low
2. Sound empirical verification of proposed theorems to assess the impact of bias/variance  generalization. 
3. Great summarization of relevant work 


Weakness: 
1. Empirical evaluation done only on artificial neural networks, could be extended to more architectures. 
2. Does not have any in depth analysis of any biologically focussed learning algorithms (acknowledged by author)
3. While it acknowledges the impact of bias/noise on learning performance, it does not provide any plans to eliminate noise/variance 
4. There isnt any plan to improve learning algorithms as well in the advent of bias/noise impact on them (acknowledged by author)
5. There is an assumption that Loss L is twice differentiable but no justification for this assumption. 

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: There is good clarity in terms of mathematical equations for examining loss, variance and bias and how it affects learning . These are well explained followed up experimental backing and data analysis to support the proofs/theorems for impact on training, loss and across different ranges of networks and generalization. Clear arguments of why proxy metric L is necessary which is differentiable given fairness constraints are non-convex

Novelty: From a novelty perspective this paper does not have much to offer as existing concepts such as bias variance, loss and their impact is revisited across different learning paradigms, explained rigorously with math proof. 

Reproducibility: There isn't a section made available on how to reproduce the experimental setup to quantify measurements and validate the results proposed in the paper. 
summary_of_the_review: My recommendation is to not accept the paper, unless a few revisions are made which are highlighted in the weakness section. This paper illustrates the importance of the impact of variance and bias on gradient estimators and how it impacts learning. While it definitely improves the understanding by proposing mathematical theorems and providing explanations via analysis, it is not clear to me how it is novel and advances the field and how bias or noise and their detrimental impact can be reduced. I think if some of these problems are addressed via additional work, it could be a great paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Many methods rely on approximations of the true gradient, which have intrinsic bias and variance. This paper shows how bias and variance of gradient estimators can hinder or improve learning. Although bias and variance can potentially impair learning, the right amount can be helpful for learning and for generalization. The effect of bias and variance seems to be independent of each other, so this paper studies each separately. The paper provides a framework for studying the effect of adding a fixed bias and white noise in the update rule. Given the bias and variance of a learning algorithm, one can use these guidelines to find the network design that facilitates learning. 
strength_and_weaknesses: **Strengths**:
1. The problem tackled by this paper is fundamental and important. The paper shows a promising direction that can lead to more impactful works.
2. The paper provides a principled way of analyzing how bias and variance impact learning using a careful setup based on the change in loss with an update having intrinsic known bias and variance compared to the change of the loss using the update based on true gradient.
3. The paper has substantial results explaining many of the standard design practices in deep learning in a more grounded way. 
4. The paper is well-written, and the problem is well-motivated.

**Weaknesses**:
1. Unclear notations. The authors used the same notations to write vectors and scalars. Reading these notations would be challenging to follow for many readers. Please consider updating your notations and refer to the notation section in the Formatting Instructions template for ICLR 23.
2. The framework impact is unclear. The authors mentioned that the case of intrinsic but known bias and variance is often the case in computational neuroscience and neuromorphic engineering. This is the main motivation for their approach. However, the framework provided is limited to specific cases, namely, white noise and fixed bias.
The authors argue that their assumptions are reasonable for most cases computational scientists and neuromorphic engineers face, but they don’t provide evidence for their claims. Clearly, this framework provides an important way for analyzing methods such as perturbed gradient descent methods with Gaussian noise, but it’s unclear how it can help analyze other cases. This suggests that the framework is quite limited.
	* The authors need to show that their choices and assumption are still useful for computational neuroscience and neuromorphic engineering. This can happen by referring to contributing and important works from these fields having known bias and variance with Gaussian noise.
	* In the experiments, the used bias is restricted to having the same magnitude for all weights ($b \vec{1}$). Can we reproduce the results if we use arbitrary biases? It would be better if the authors tried a number of arbitrary biases and averaged the results.
3. The paper is not well-placed in the literature. The authors didn’t describe the related works fully (e.g., stochastic gradient Langevin dynamics). This makes the work novelty unclear since the authors didn’t mention how analyzing the gradient estimator was done in earlier works and how their contribution is discernible from the earlier contributions. Mentioning earlier contributions increases the quality of your work and makes it distinguishable from other work. Please also refer to my comment in the novelty section.
4. Missing evidence of some claims and missing details. Here, I mention a few:
	* It’s not clear how increasing the width and/or depth can lower the trace of the Hessian (Section 2.1). If this comes from a known result/theory, please mention it. Otherwise, please show how it lowers the trace.
	* The authors mentioned that they use an analytical and empirical framework that is agnostic to the actual learning rule. However, the framework is built on top of a specific learning rule. It’s unclear what is meant by agnostic in this context (Section 1).
	* The authors mentioned in the abstract that the ideal amount of variance depends on the size and sparsity of the network, the norm of the gradient, and the curvature of the loss landscape. However, the authors didn’t mention the sparsity dependence anywhere in the paper.
	* The authors mentioned in a note after the proof of Theorem A.5 that it is also valid for Tanh but not Sigmoid. However, the proof assumed that the second derivative is zero. It’s unclear whether a similar derivation can be developed without this assumption. However, the authors only mention the relationship with the gain of $\phi(.)$.
	* More information is needed on how the empirical likelihood of descent is computed (Fig. 7).
	* The use of MSE should be mentioned in Theorem A.3 since it’s not proven for any loss function. In addition, the dataset notation is wrong. It should be $\mathcal{D}=\\{(x_1, y_1), ..., (x_M, y_M)\\}$, where $M$ is the number of examples since it’s a set containing input-output pairs, not just a single pair.
	* The argument in Section 2.1 that increasing depth could theoretically make the loss less smooth is not related to the argument being made about variance. It is unclear how this is related to the analyses of how increasing depth affects the impact of the variance. I think it needs to be moved in the discussion on generalization instead.
5. A misplaced experiment that does not provide convincing evidence to support the theorems and lemmas developed in the paper with less experimental rigor (Fig. 1).
	* The experiment is misplaced being at the introduction section. This hurts the introduction and makes the reader less focused on your logic to motivate your work. 
	* It’s not clear from the figure what the experiment is. The reader has to read appendix B2 to be able to continue reading your introduction, which is unnecessary. 
	* The results are shown with only three seeds. This is not enough and cannot create any statistical significance in your experiment. I suggest increasing the number of runs to 20 or 30.
	* It’s unclear why batch gradient descent is used instead of gradient descent with varying bias and variance. Using batch gradient descent might undesirably add to the bias and variance.
	* The experiment results are not consistent with the rest of the paper. We cannot see the relationship when varying the bias or variance similar to other experiments. Looking at Fig.1B where bias=0, for example, we find that adding a small amount of variance reduces performance, but adding more improves performance up to a limit. This is not the case with the other experiments, though. I suggest following the previous two points to make the results aligned with the rest of your results.
6. Alternative hypotheses can be made with some experiments. The experiment in Fig. 3.A  needs improvement. The authors mention that excessive amounts of variance and/or bias can hinder learning performance. In Fig. 3. A, they only show levels of variance that help decrease loss. An alternative explanation from their figure is that by increasing the variance, the performance improves. This is not the case, of course, so I think the authors need to add more variance curves that hinder performance to avoid alternative interpretations.

**Minor issues that didn’t impact the score**:
* There are nine arXiv references. If they are published, please add this information instead of citing arXiv.
* What is a norm $\sqrt{N}$ vector? Can you please add the definition to the paper?
* You mentioned that the step size has to be very small. However, in Fig. 1, the step size used is large (0.02). Can you please explain why? Can this be an additional reason why there is no smooth relationship between the values of the variance and performance?
* No error bars are added in Fig. 4 or Fig. 7. Can you please add them?
* In experiments shown in Fig. 3 and Fig. 5, the number of runs used to create the error bars is not mentioned in Appendix B.2. 
* A missing $\frac{2}{|\mathcal{D}|}$ in Eq. 27.
* In Theorem A.3 proof, how the input $\mathbf{x}$ has two indices? The input is a vector, not a matrix. Moreover, shouldn’t $\sum_k (W_k^{(2)})^2 = 1/d$, not $d$?

clarity,_quality,_novelty_and_reproducibility: The paper is clear, except for the used mathematical notations, which need to be improved. The experiments are reproducible since the details are mentioned in the appendix. The degree of novelty of this work is unclear since the authors didn’t address most of the related works (e.g., perturbed gradient descent and stochastic gradient Langevin dynamics) or the larger history of research on adding noise to improve learning (e.g., Murray and Edwards 1992). This paper needs to be well-distinguished from other works, which can happen by carefully assigning credit to the earlier papers in addition to mentioning how your paper’s contributions are different from earlier contributions.
summary_of_the_review: The work is well-written and well-motivated. The paper tackles a fundamental problem and provides a principled way of analyzing gradient estimators. The proposed framework has substantial results explaining many of the best/standard design practices in deep learning in a grounded way. However, the paper is lacking some mathematical rigor when it comes to notations. The proposed framework has an unclear impact on giving guidelines for the targeted fields, namely computational neuroscience, and neuromorphic engineering, and some arguments are needed to justify how it serves the intended purpose. More arguments and examples are needed to justify how the assumptions are not limiting. In addition, the paper is not well-placed in the literature, which makes the novelty of the work unclear. More careful work is needed to distinguish this work from others. There is also some missing evidence for some claims made by the authors and some imprecisions. Finally, the paper has some experimental inconsistencies that contradict other results but can be fixed easily.

That being said, I think substantially improving the quality of this paper seems well possible by addressing this feedback. All my comments can be addressed by the authors since they are not affecting the main arguments of the paper. This paper has potential, and I would be willing to change my scores, subject to the authors’ clarifications and incorporating the requested changes.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper examines how variance and bias in gradient estimators can affect learning depending on properties such as network size and activity sparsity, the gradient's norm, and the curvature of the loss landscape. The authors show that while variance and bias can impair learning on training data, they may also be beneficial for generalization. They suggest that researchers analyze these properties to determine if their approximation to gradient descent will be effective for learning given their network's task properties.

The reviewers appreciated the theoretical and empirical support provided for the claims. The paper is of high quality, giving a clear value and message, meeting the bar of this venue. Hence, I recommend accepting this paper. I strongly encourage the authors to strengthen the paper's motivation and connect how the gradient estimation is related to implementations with neuromorphic chips. Concrete examples of how true gradients cannot be calculated in those implementations would help readers appreciate the relevance and importance of this work. Moreover, the authors should discuss the relevance of studying the effect of fixed bias and white noise in the context of methods where they occur.



justification_for_why_not_higher_score: Albeit a good work, it falls short of giving a clear case where gradient estimation involves white noise in implementations for neuromorphic chips, the opening motivation of the paper.
justification_for_why_not_lower_score: The work constitutes a good contribution toward the understanding of gradient estimation.
Decision: Accept: poster