Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper mainly focuses on experimental results on real data to verify the so-called Frequency Principle: DNNs often fit target functions from low to high frequencies during the training process. Some theoretical analyses are also provided to backup the empirical observation. 

This paper is very well-written. The methods are explained very clearly, and the logic is easy to follow. However I think this paper also has some weak points, as listed below:

(1) The frequency principle lacks a rigorous definition. In Section 2, the authors provide very inspiring explanations and examples, however no rigorous definitions are given. Is there a way to directly quantitatively define the response frequency?

(2) In Section 3.1, it is not explained why the frequencies are calculated based on the samples. Probably I have missed something, but based on the description, can’t the frequencies be directly calculated on a grid along the 1-dimensional subspace defined by the mean and first principle component of the data? In other words, even after training the predictor function with real data samples for several steps, the frequency of a predictor function should still be its own property and  should be independent of the distribution of data inputs. In fact, are the vectors $n^{-1/2} (cos( 2\pi k x_{p_1,1}), cos( 2\pi k x_{p_1,2}), \ldots, cos( 2\pi k x_{p_1,n}) )$ for different $k$ even orthonormal vectors? I think unless $x_{p_1,i}$ follows certain specific distributions, these vectors are not even close to orthonormal. Therefore using them to calculate the frequencies is very weird.

(3) In fact, are Sections 3,4 and 6 studying the same kind of frequency? It is not very clear due to the vague definitions.

Because of these concerns, I think this paper is on the borderline. For now I tend to recommend a weak reject.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper studies the training process of NNs through the lens of Fourier analysis. The authors argue that during the training process, NNs will first learn low frequencies part of the function first and then the high frequency part. To verify this claim empirically, the author propose two methods: 1. examine the convergence of different frequencies in a pre-selected direction in the frequency space during training; 2. examine the convergence rate of the 2-norm of low v.s. high frequencies during training.  Through the experimental results of these two methods, the authors conclude that NNs learn the low  frequency components before the high frequency components. The authors also discuss a potential application of this observation to solving high dimensional PDEs: coupling DNNs training (good at learning low frequency components) with the Jacobi method (good at learning high frequency components). Finally, the authors also provide some theoretical intuition (Thm 1., 2.) why low frequency components are learned faster and an explanation why NNs could generalize well on images but perform poorly on tasks like learning parity functions. 


Other comments: 
1. It seems the filtering method is a better (might be a sufficient) way to justify the F-Principle than the projection method, given the projection method examines only one direction (also appointed out in the paper).  
2. When talking about Fourier transform, would you specify what is the domain of the functions and how the functions are defined (section 3.1) The notation there is somewhat confusing (which makes the rest of the paper difficult to follow) since you are mentioning the Fourier transform of the set {(x_i, y_i)}. It will be helpful to define the function before defining its Fourier transform.  Please also mention what is the domain of the function, {x_i}_i or R^d? 
3. According to equation (4), it seems the domain of the functions is {x_i}_i, otherwise equation (4) should be a function of x\in R^d, not x_i.  
4. Could you elaborate why (4) is a good approximation of the low frequency energy rather than the L2 norm (over x\in R^d) of (4) with x_i replaced by x\in R^d. 
5. It might be useful to refine the related work section. It is not clear what are the previous contributions prior to this paper, and it seems [1] shares some similar results/observation with the this paper. 

Overall, I lean to a weak rejection. The key findings (and similar results, e.g. NNs learn simple functions first), i.e. F-Principle seems to have already appeared in previous works, e.g. [1] and the theoretical results of this paper are limited to an idealized setting (results of more general setting appear in another work, mentioned in the paper.)


[1]Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht,
Yoshua Bengio, and Aaron Courville. On the spectral bias of deep neural networks. arXiv preprint
arXiv:1806.08734, 2018. 1, 8, A 



review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes to analyze the loss of neural networks in the Fourier domain. Since this is computationally expensive for larger-dimensional datasets, the analysis instead first projects the data onto the principal component of the data, and then using a Gaussian kernel estimation (which has nice properties in the Fourier domain). The analysis finds that DNNs tend to learn low-frequency components before high-frequency ones.

Overall I quite like the analysis of this paper. I think it could be clearer and contain more experiments but it is otherwise rather convincing proof that DNNs learn low-frequency patterns first.


- More experiments: in particular, analyzing this phenomenon over more than a single principal component or through non-linear transformations of the data.
- It's not always clear how $\mathbf{k}$ is calculated or where it comes from, whether it is implicit through the Gaussian metric or chosen randomly. The paper would benefit from always making this clear in the text and figure captions.
- It's well known that different optimizers seem to learn differently both in terms of speed and features that end up being learned (thus generalization), repeating this analysis for Adam, RMSprop & friends would be great.

The paper is mostly easy to read, but there are a few mistakes here and there that slow down reading. Here are a few:
- "variation problems" you mean "variational"?
- "This difference implicates" -> "implies"
- "by the Parseval's theorem" -> "by Parseval's theorem" (occurs multiple times)
- "difference of" -> "difference between"
- "in previous section" -> "in the previous section"
- "to verify F-Principle" -> "to verify the F-Principle"
review_title: Official Blind Review #1

Meta-review: Borderline decision.  The idea is nice, but the theory is not completely convincing.  That makes the results in this paper not be significant enough.
Decision: Reject