Title: A Spectral Approach to Generalization and Optimization in Neural Networks
Authors: Farzan Farnia
Jesse Zhang
David Tse
Venue: ICLR 2018
Keywords: Generalization
Neural Networks
Fourier Analysis

Abstract: The recent success of deep neural networks stems from their ability to generalize well on real data; however, Zhang et al. have observed that neural networks can easily overfit random labels. This observation demonstrates that with the existing theory, we cannot adequately explain why gradient methods can find generalizable solutions for neural networks. In this work, we use a Fourier-based approach to study the generalization properties of gradient-based methods over 2-layer neural networks with sinusoidal activation functions. We prove that if the underlying distribution of data has nice spectral properties such as bandlimitedness, then the gradient descent method will converge to generalizable local minima. We also establish a Fourier-based generalization bound for bandlimited spaces, which generalizes to other activation functions. Our generalization bound motivates a grouped version of path norms for measuring the complexity of 2-layer neural networks with ReLU activation functions. We demonstrate numerically that regularization of this group path norm results in neural network solutions that can fit true labels without losing test accuracy while not overfitting random labels.


Review discussion:
Deep neural networks have found great success in various applications. This paper presents a theoretical analysis for 2-layer neural networks (NNs) through a spectral approach.
Specifically, the authors develop a Fourier-based generalization bound. Based on this, the authors show that the bandwidth, Fourier l_1 norm and the gradient for local minima of the population risk can be controlled for 2-layer NNs with SINE activation functions. Numerical experimental results are also presented to verify the theory.
(1) The scope is a bit limited. The paper only considers 2-layer NNs. Is there an essential difficulty in extending the result here to NNs with more layers?
Also, the analysis for gradient-based method in section 6  is only for squared-error loss, SINE activation and a deterministic target variable. What would happen if Y is random or the activation is ReLU?
(2) The generalization bound in Corollary 3 is only for the gradient w.r.t. \alpha_j. Perhaps, an object of more interest is the gradient w.r.t. W. It would be intersting to present some analysis regarding the gradient w.r.t. W.
(3) It is claimed that the bound is tighter than that obtained using only the Lipschitz property of the activation function.
However, no comparison is clearly made. It would be better if the authors could explain this more?
In summary, the application domain of the theoretical results seems a bit restricted.
Minor comments:
Eq. (1): d\xi should be dx
Lemma 2: one \hat{g} should be \hat{f}
Thank you for your feedback.
First let us note that analyzing the generalization performance of gradient methods is an open problem, even in the specific case of 2-layer neural nets and squared-error loss function. Our focus in this work is to address this open problem in the special case of 2-layer NNs with sine activation. We have chosen and analyzed this simple neural network model via Fourier analysis, because as shown in our numerical experiments (section 7) this simple model can still easily overfit random labels. Here is our response to the other questions and comments:
1)  A possible way of extending our Fourier-based analysis to multi hidden layer neural nets is through the analysis of the Fourier representation of composite of sine functions. To apply our Fourier-based analysis of gradient methods for ReLU function, one possible way is to approximate ReLU by its Fourier series after assuming the input X is properly bounded. For a random Y(x), we need to perform our Fourier analysis for the resulting stochastic process instead of a deterministic function.
2)  A sufficient condition for Theorem 4’s bounds is the generalization of the gradients w.r.t. \alpha. Generalization of the gradients only w.r.t. W does not provide a sufficient condition for applying Theorem 4.
Also, establishing generalization w.r.t. both \alpha and W leads to slower rates of convergence compared to our result in Corollary 3.
3)  For a two layer neural net f(x)=a*phi(Wx) with activation phi, the Lipschitz-based bounds are linear in the square root of W’s norm. Our Corollary 2 improves this dependency to the square root of the log of W’s norm. The improvement holds for bandlimited phi’s such as sine or Gaussian activations. We will make this point clearer in the text.
4)  Thanks for pointing out the typos. We will correct them in our draft.
This work proposes to study the generalization of learning neural networks via the Fourier-based method. It first gives a Fourier-based generalization bound, showing that Rademacher complexity of functions with small bandwidth and Fourier l_1 norm will be small. This leads to generalization for 2-layer networks with appropriate bounded size. For 2-layer networks with sine activation functions, assuming that the data distribution has nice spectral property (ie bounded bandwidth), it shows that the local minimum of the population risk (if with isolated component condition) will have small size, and also shows that the gradient of the empirical risk is close to that of the population risk. Empirical results show that the size of the networks learned on random labels are larger than those learned on true labels, and shows that a regularizer implied by their Fourier-based generalization bound can effectively reduce the generalization gap on random labels.
The idea of applying the Fourier-based method to generalization is interesting.
However, the theoretical results are not very satisfactory.
-- How do the bounds here compared to those obtained by directly applying Rademacher complexity to the neural network functions?
-- How to interpret the isolated components condition in Theorem 4? Basically, it means that B(P_X) should be a small constant. What type of distributions of X will be a good example?
-- It is not easy to put together the conclusions in Section 6.1 and 6.2. Suppose SGD leads to a local minimum of the empirical loss. One can claim that this is an approximate local minimum (ie, small gradient) by Corollary 3.
But to apply Theorem 4, one will need a version of Theorem 4 for approximate local minima.
Also, one needs to argue that the local minimum obtained by SGD will satisfy the isolated component condition. The argument in Section 8.6 is not convincing, ie, there is potentially a large approximation error in (41) and one cannot claim that Lemma 1 and Theorem 4 are still valid without the isolated component condition.
Thank you for your feedback. Here is our response to the three points raised in this review:
1)  The existing Rademacher complexity bounds for neural nets are based on the activation function’s Lipschitz constant. For a two layer neural net f(x)=a*phi(Wx) with activation phi, those Lipschitz-based bounds are linear in the square root of W’s norm. Our Corollary 2 improves this dependency to the square root of the log of W’s norm. The improvement holds for bandlimited phi’s such as sine or Gaussian activations.
Although the ReLU function does not satisfy the above condition, our Fourier-based generalization result is still applicable to ReLU-based networks (Theorem 3) and motivates new capacity norms (group path norms) for these networks.
2)  For example, consider a multivariate Gaussian X ~ N( 0 , sigma^2*I). The Fourier transform of P_X in this case, which is  exp{ - (sigma*||w||)^2 / 2 }, becomes sufficiently small in Eq. (39) if the components are O( 1/sigma ) apart.
Therefore, we need the standard deviation of X to be large enough so that the components are O( 1/sigma ) apart. We will add this example to section 6.
3)  For an approximate local minimum where the population gradient is epsilon instead of zero, the upper-bound in Lemma 1 (Eq. (12)) changes by at most epsilon and hence the Fourier L1_norm bound in Theorem 4 changes by at most d*epsilon (d is the size of the hidden layer).
We agree that the isolated components assumption provides a barrier for applying our theory to the general case.
However, the condition holds given that Y(x)’s Fourier transform has distant local extrema (at least B(P_X) apart).
For example, the condition holds if Y(x)=a^T sin(Wx) where each two different rows w_i,w_j of W satisfy || w_i - w_j || > B(P_X). We will include this discussion in section 6.
This paper studies the generalization properties of 2-layer neural networks based on Fourier analysis. Studying the generalization property of neural network is an important problem and Fourier-based analysis is a promising direction, as shown in (Lee et al., 2017).
However, I am not satisfied with the results in the current version.
1) The main theoretical results are on the sin activation functions instead of commonly used ReLU functions.
2) Even if for sin activation functions, the analysis is NOT complete. The authors claimed in the abstract that gradient-based methods will converge to generalizable local minima.
However, Corollary 3 is only a concentration bound on the gradient. There is a gap that how this corollary implies generalization. The paragraph below this corollary is only a high level intuition.
Thanks for your feedback. Here is our response to the two comments:
1)  The Fourier-based generalization bound in Theorem 2 is applicable to a general activation function. By applying this generalization result, Theorem 3 motivates new capacity norms (group path norms) for ReLU-based networks. Supporting this, our numerical experiments in sections 7.2,7.3 indicate that regularizing group path norms can close the generalization gap without compromising test accuracy over neural nets with ReLU activation.
In section 6, we apply our Fourier-based generalization result to address the open problem of analyzing generalization performance of gradient methods in the special case of 2-layer neural nets with sine activation. Here we have chosen and analyzed this simple neural network model via Fourier analysis, because as shown in our numerical experiments this simple neural network can still easily overfit random labels.
2)  Corollary 3 is not the final generalization bound, but it should be applied together with Theorem 4. More specifically, Corollary 3 provides a sufficient condition to apply the bounds in Theorem 4 on the spectral properties of the local minima found, which in turn bounds the generalization error of those local minima.
The generalization result holds for the local minima found by large-batch gradient descent, and the paragraph after Corollary 3 gives high level intuition why we expect the bound to also hold for the local minima found by small-batch gradient descent. We will make this explanation clearer in the text.
We thank all the reviewers for their detailed comments and feedback which greatly helped us to improve this work. We have posted a revision which includes changes addressing the main comments.
In summary,
1) We have clarified the explanation after Corollary 2 about the comparison between the Fourier-based Rademacher complexity bound and the existing Lipschitz-based bounds for 2-layer neural nets (asked by AnonReviewer2 and AnonReviewer3).
2) We have added Remark 1 after Theorem 4 explaining what this result implies when X has a multivariate Gaussian distribution (asked by AnonReviewer3).
3) We have made the explanation after Corollary 3 clearer (asked by AnonReviewer1) by directly applying an approximate version of Theorem 4 included in section 8.6 (asked by AnonReviewer3) to the local minima of the empirical risk.

Meta-review:
role: PC
data: comment: Understanding the generalization behavior of deep networks is certainly an open problem. While this paper appears to develop some interesting new Fourier-based methods in this direction, the analysis in its current form is currently too restrictive, with somewhat limited empirical support, to broadly appeal to the ICLR community. Please see the reviews for more details.  

Decision: Reject