Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The papers proposes an interesting analysis that links several aspects of architectural design in Deep NNs to the spectral analysis and observed roughness. Different activations functions are considered in the study, mainly centered on deep CNN with or without skip connections (in the framework of ResNet v1 and v2). The starting point, which is not novel, actually, but relevant, is that specific types of non-linearities introduce harmonic distortions, and the effect is potentially amplified when multiple non-linearities are stacked. Theoretically, the paper shows that there is a concrete link between architectural choices in the network design and the blueshift in the frequency domain. Experimentally, the observations support the mathematical analysis. All in all, some of the conclusions regarding trainability of CNN architectures with skip connections have been already noted and do not seem greatly new, but the paper introduces a nice perspective to see this phenomenon in another light. 
The paper is generally well written and I appreciated reading it.

The major downside I see in the current form of the manuscript is given by some aspects of the presentation. For instance, Fig. 1 is clearly misplaced (it should be in Section 4.4). Similarly, Figure 6 should be in Section 5. Moreover, abbreviations would be better used in a more uniform manner (e.g., SDFA, FDSA, SDSA). Regarding the reported experiments: are the given plots achieved by averaging over multiple runs? (only in one of the many experimental settings this information is given in the paper). Finally, the link between the left and the right sides in Figure 6 is not really straightforward, perhaps grouping together the short and the noshort results could be of help for the reader.

-- EDIT: 
Thanks for the nice feedback during the rebuttal. I am happy to stay with my rating of clear acceptance.


review_title: Hanrmonic distorsion in deep neural networks 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper proposes a new approach for how to analyze the ruggedness of the surface of the neural network loss. Specifically, the paper proposes to apply harmonic distortion on the weight-to-output (w-o) maps. That is, the method casts the w-o functions in the Fourier domain and then aggregate the surface characteristics by virtue of averaging the different order Fourier coefficients. The  paper shows that non-linearities are responsible for blueshifting with deeper layers, that is for "transferring more energy" on the higher frequencies. The consequence is rougher surfaces, as well as higher frequencies for gradients, which can lead to exploding gradients in the deeper layers. The remedy is with skip connections and feature averaging, which although are methods already known to improve trainability, the paper corroborates that they also make sense in terms of said approach. The paper conducts various empirical and ablation studies, providing evidence of the claims.

The strengths of the paper:
- I believe that at the core the paper offers some really nice preliminary ideas and intuitions. It is very intuitive that deeper layers generate higher frequencies to the loss and thus can make the optimization harder. On the other hand, higher frequencies are necessary for expressivity. This is another instantiation of the bias-variance tradeoff. By finding ways to balance the relative strength of high and low frequencies, one could throttle how much expressivity is necessary for the task at hand.
- The paper goes to great lengths to support some of the claims empirically. There is lots of different experiments and it seems the claims are overall supported by the findings. Different nonlinaerities and architectures (maybe too much, in that there is less focus) are explored, which is admirable.
- I like the quality of the visualizations. It is clear that the authors have spent quite some time in generating their figures.

The weaknesses of the paper:

- Sure, if we focus only on the low-order frequencies trainability is better. Also, I dare say that the insight is not exactly surprising although very intriguing. However, what exactly is the message? That we should have only low frequencies? Or that we should have some high frequencies? That skip connections are good for better training? I believe that in many ways, the message is incomplete if one leaves out expressivity and it would be nice to extend the theory to say something about the potency of the neural network on learning patterns and generalizing. The authors already comment on this in the 'future work' lines. I think that this should become current work, otherwise the work is incomplete, at least from the current perspective.
- I have the feeling there are places where the analysis is imprecise, although it could be that I also misunderstood.
 - For one, the crux of the analysis is that the neural network nonlinearities are expressed in Fourier series (sec 3.2). Then, in the next section 3.3 the paper says that in practice nonlinearities are not polynomial and might not have a convergent Taylor expansion. So, instead a Chebyshev approximation is opted for. However, it is not clear if the Chebyshev approximation suffices or what are the limits of it? but I think this must be elaborated further.
 - Also, what are these Chebyshev approximations per nonlinearity? I think it is quite important to clarify this, considering there are nonlinearities that all but very similar, e.g., the ReLU and the leaky ReLU. What is the big difference between the two in terms of the described analysis?
- I find it hard to understand often what the analysis tries to say, either the analysis is incomplete, the writing generally unclear or I simply don't understand some details. I list my comments by order of reading (not importance).
 - Throughout the paper there is a clear desire to connect rougness with layer depth. However, in all equations and analysis the depth is not explicitly present. For instance, in equations 4-6 there is only the degree of the polynomial K, but no layer variable or index. From what I gather, the (implicit) argument is that by the successive stacking of layers, the corresponding low/high order frequencies get stronger or weaker, relatively. Then, the objective is to compare the corresponding low and high frequencies for different layers, showing that for deeper layers the higher frequencies get stronger because of the recursion. This is how depth is 'qualitatively' introduced as a variable. Is this indeed the intention? If yes, I think it can be written more explicitly.
- I find figure 1 a bit perplexing. Again, I understand what is the message, but it is hard for me to connect it to the theory, since the theory makes only indirect references to the specific nonlinearities. Also, what is 'ReLU->ResNet' supposed to stand for? ResNet is a ReLU when including skip connections? And what is 'Linear->ReLU'? To put it otherwise, adding skip connections or a ReLU nonlinearity are discrete design choices. However, the figure has continuous axis. So, what exactly is illustrated? The 'vertical' axis corresponds to the t variable in the Fourier coefficient. What about the other axis?
 - The related work points to Li et al and their spectral analysis to ground the proposed research. However, it is not explained what these observations are and how they relate to the current paper. It would be nice for the reader to add a short explanation.
 - Do we expect a difference by considering 1D slides, instead of 2D slides as motivated by Li et al? Why yes, why no?
 - It is not explained why are the mean path are empirically zero-functions. I infer that this is the case because at any location of the loss surface, if we take a small ball around it there will be an equal amount of parameters for which there is a higher or lower loss value? However, wouldn't this imply already a strong gradient (about 1, if I am not mistaken)?
 - What I find a bit confusing is that in equation 4 and 5 we apply the nonlinearity \phi on p(t). However, p(t) are the 1-D slices of our neural network f defined in the preamble of section 3. I would assume that the nonlinearities would then already be inside p(t). In fact, in the preamble of 3 there is also a mention of \phi and how p(t) is a polynomial when \phi is the identity function. Maybe I have misunderstood something here. 
 - There is an attempt to connect exploding gradients to blueshifting. However, this is not entirely clear to me. Indeed, one can say that simultaneously we have blueshifting in the gradient and at the same time exploding gradients. Does this mean that one cause the other, however? Couldn't one have explosions by having disproportionately large low order frequencies (not that it is the case, just wondering)? Or some other phenomenon.
 - There is a connection to exploding gradients, however, in deep networks vanishing gradients also important (maybe more so). Can the analysis address also vanishing gradients?
 - It is not exactly clear why the frequency dependent signal averaging is wekaner than exponetial downweights. The explanation is very brief and a bit vague (law of large numbers, exponential decay). Is there a more precise qualitative or quantitative argument here?
 Can one still call the method as more 'global' in saying something about roughness, given that all coefficients are computed per layer? Of course, each layer's coefficient are influenced by all previous layers, but is this enough to paint the method 'global'?
 - Is somehow K (polynomial order) connected to L (number of layers)? Or is this relation the way I described above?
 - Perhaps relevant to the previous point, and taking the position of the devil's advocate, in a way what is put forward by the paper is a re-interpretation of existing knowledge. While certainly very intriguing, is there a new insight on trainability for a new type of method/technique that can improve trainability? What about wide layers and them being easier/harder to train?
 - The text in p. 6 on Fig 5 (In figure 5, we use the power law ... ) is unclear to me. 
 - In p. 7 there is a great misalignment between the figure references and the figure locations in the paper.
 - How is leaky ReLU connected to other nonlinearities? How precisely does it make a difference? How difference is the Chebyshev polynomial?
 - What does it mean 'making the networking more linear'? Do you mean increasing the \alpha hyperparameter till it becomes 1, in which case you have a linear function?

In general, I find the paper quite interesting and with valuable potential contributions, but incomplete and not ready for publication at this stage. I believe it would worth it if the authors took the time to revisit the crispness of the message as well as the writing. Of course, I am more than happy to revisit my recommendation if the authors produce a convincing argument.
review_title: Official review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper applies harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. This gives global view of the network output roughness.

Strong points: The paper introduces an interesting measure "roughness" of deep neural network via harmonic distortion analysis. It evaluates the blueshift near the initialization for various nonlinearity (ReLU, LReLU, TanH, Sigmoid,…) and architecture choices (no skip, skip, depth, width). The blueshift fits people's intuition of the architectural choices.  

Weak points: 
1. Current presentation mainly focus on explaining existing networks via blueshift measure. It does not "predict" new choices of the nonlinearity and architectures. This prediction could be related with the data: find the one nonlinearity and architectures that could best fit the data complexity.  
2. The hypothesis that spectral blueshift impedes training fit the observations that architectural choices with  good harmonics generation control are easier to train to good performance. However this hypothesis is not consistent with the Sigmoid +NS case in Figure 16. I suppose it is hard to optimize Sigmoid+NS for a deep network (50 layers). 
3. The paper claims this can give a global view of the roughness. But the measurements are near the initialization. In this sense, it would be better to look at the roughness measure out of the initialization neighborhood.

I would not recommend the acceptance for now because of the above weak points.

After the rebuttal

I thank the authors for the detailed feedback. I am a bit with AnonReviewer3 about the concerns on the descriptive languages of the paper "less nonlinear", "more expressiveness". Moreover, the behaviors of different nonlinearities in the Fig.3 and Fig.17 are related to the specific initialization and batch normalization, which is rather not a global view of the landscape. I would keep the score unchanged.
review_title: An interesting measure and view of the deep neural network

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ***Summary***

I would firstly like to thank the authors for an interesting read. I enjoyed going through the submission very much.

The authors propose to understand the qualitative effects of nonlinearities by studying the impact they have on the Fourier spectrum of deep neural networks. The central hypothesis is that nonlinearities with a lot of energy in their side lobes (high frequencies), lead to neural networks that have a rougher mapping and that are consequently tougher to train because the derivative landscape is also rougher. They back this hypothesis up with some mathematical arguments from the area of harmonic distortion analysis and with empirical experiments to support the qualitative predictions of this theory.

***Pros***

I found the submission very readable. I think the balance of text to mathematics in the main submission was about right, reserving the appendix for a more in depth discussion.

I think that while the central finding that deep mappings are smoother is, in itself, not particularly novel, the chain of reasoning to get to this fact is new. I like the use of the Fourier spectrum to show this and the analysis behind how the spectra of various nonlinearities affect overall network smoothness..

The choice of experiments, which sequentially back up the claims, makes for a good paper. I particularly enjoyed the results in Figure 2, which were very instructive and gave good insight into the predictions of the theory.


***Cons and constructive feedback***

In order from start to finish.

In the abstract should differential be differentiable?

I think a good paper to cite would be “Avoiding Pathologies in Very Deep Networks” (Duvenaud et al., 2014) who analyze deep kernels in Gaussian processes. While the underlying models are different, the kinds of qualitative results in this paper are very similar to the submission.

I am concerned about the use of the Fourier spectrum to model the ReLU nonlinearity. Will there not be issues with the Gibb’s phenomenon? The discontinuous gradient will mean that a spectrum exists, but reconstructions are poor.

Paragraph below equation 3: uniformely -> uniformly

Equation 4: using t_j is confusing given that you use t in eqn 1. Please change to another symbol

Eqn 6: Please define the autocorrelation symbol in the main text.

Eqn 6: Please define z versus z_j

Section 3.2 discussion: I would assume that while higher order autocorrelations would broaden the spectrum they would also smooth it out. For high orders it would like Gaussian-like in shape. This would not necessarily lead to blue-shifting.

Section 3.3: therfore -> therefore

Section 3.4 trivial -> trivially

Section 3.5: Exponential downweighting. ResNets have combinatorially more medium length paths than short or long ones. So the average weight of a medium path is far higher than short or long ones. I would have liked to have seen a deeper analysis of this effect.

Experiments: I found these very interesting. What is the motivation for only focussing on networks at initialization? I would have loved to have seen what a pertained network looks like.

Are ensembles covered within the scope of this theory? They seem to have good performance but since each member is trained individually there is no smoothing of the training function, although the test loss function is smoother when all member models are combined.


***Post rebuttal review***

Having read the rebuttal, I am very happy with the author responses. My main concerns about the Gibb's phenomenon and the choice to consider blueshifting at initialization have been thoroughly addressed. It is clear to me that the authors have thought long and hard about the rebuttal and used it to improve their submission. Therefore I maintain that this is still a clear accept.
 
review_title: Initial review

Meta-review: The paper presents an analysis of the spectral impact of non-linearities in a neural network, using harmonic distortion analysis as a means to quantify the effect they have in the spectral domain, linking a blue-shift phenomenon to architectural choices. This is an interesting analysis, that could be strengthened by a more thorough exploration of how this analysis relates to other properties, such as generalization, as well as through the impact of the blueshift effect through the training process.
Decision: Accept (Poster)