Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks
Authors: Christian H.X. Ali Mehmeti-Göpel
David Hartmann
Michael Wand
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning theory
loss landscape
harmonic distortion analysis
network trainability

Abstract: In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. Each nonlinear layer creates higher-frequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape. Unlike differential models (such as vanishing gradients, sharpness), this provides a more global view of how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.

Published as a conference paper at ICLR 2021
RINGING RELUs: HARMONIC DISTORTION ANALYSIS OF NONLINEAR FEEDFORWARD NETWORKS
Christian H.X. Ali Meheeti-Göpel Institute of Computer Science
David Hartmann
Institute of Computer Science
University Mainz Staudingerweg 9, 55122 Mainz, Germany chal imehê uni-mains
Johannes Guttenber-Uniivesiin of Mainz
Staudingerweg 9, 55128 Mainz, Germany
dahartma @ uuni -ma i nz de
Michael Wand
Institute of Computer Science
Johannes Guteenber-Univvrsitty of Mainz Staudingerweg 9, 55128 Mainz, Germany mwand@uni. mz.
ABSTRACT
In this paper, apply harmonic distortion analysis to understand the effect nonlinearities the spectral domain. Each nonlinear layer creates higher- frequency harmonics, which call "blueshift", whose magnitude increases with network depth, thereby increasing the "roughness" of the output landscape. Unlike differential models (such vanishing gradients, sharpness), this provides global view how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for connection with trainability.
INTRODUCTION
ReLU
ResNet
linear
(00))
ReLU ResNet
parameter
Figure Continous transition of a loss path between linear feedforward ("lincar"), nonlinear feedforward ("ReL.U") and nonlinear residual ("ResNet") regimes. Graph: loss path near initialization of a ResNet56 v2 with LReLUs with negative slope a E [0, 1] and residual branch weight v E [o, 1]. 1, 0, Right: 1,
Left: 0, 0, Middle:
In the past decade, the emergence of practical deep neural networks arguably has had disruptive impact on applications of machine learning. Depth as such appears to be key expressive models (Raghu et al. (017).. However, depth also comes with challenges concerning training stability. Theoretical problems include vanishing and exploding gradients (Hochreiter) 1991), chaotic feed- forward dynamics (Poole et al. (016)) or decorrelation of gradients (Balduzzi et al.) (017). In practice, a number of "recipes"" are widely used, such as specific nonlinearities (Glorot al. 2011 He et al. 2015), normalization methods such batch normalization (Ioffe & Szegedy 2015), shortcut architectures (Srivastava et al. 2015] He et al. 2016a b), or multi multi-path architecture with (Huang et al. 2017) and without shortcuts (Szegedy al. 2016). Broadly speaking, a key research question is to understand how the shape of the network function, i.c., the map from inputs and parameters to outputs, is affected by architectural choices.

---

Published as a conference paper at ICLR 2021
Our paper considers specifically the roughness of the weighttsto--uung function ("w-o functtion") of nonlinear feed-for forward networks. Motivated by the recent visualizations of (Li et al. 2018), which show how depth increases roughness and residual connections smoothen the output again, our goal is to provide an analytical explanation of this effect, and study its implications on network design and trainability. To this end, first formalize "roughness" as the decay-rate of the expected power spectrum of a function class. Our main contribution is to then apply harmonic distortion analysis to nonlinear feedforward networks, which predicts the creation of high-frequency "harmonics" (thereby "blueshifting" the power spectrum) by polynomial nonlinearities with large higher-order coefficients. Based on this model, we discuss how network depth increases blueshift and thus roughness, while shortcut connections, low-degree nonlinearities and parallel computation paths dampen it. In relation to trainability, we show an analytic link between blueshift and exploding gradients. Unlike the former model, the spectral view describes more global behavior of the w-o function over regions in the paramcter domain.
Experiments confirm the theoretical predictions: We observe the predicted effects of depth, shortcuts and parallel computation on blueshift, and are able to differentiate different types of nonlinearities by the decay rate of coefficients of a polynomial approximation. The findings are in-line with known advantages in trainability of the different architectures. We further strengthen the evidence by training large set of networks with a different amount of nonlinearity and depth, which shows clear correlation between blueshift and training-problemss, as well as a trade-off with expressivity.
In summary, our paper explains how network architecture affects roughness, shows connection to trainability, and thereby provides new tool for analyzing the design of deep networks.
RELATED WORK
Vanishing or exploding gradients are a central numerical problem (Hochreiter, 1991 Pascanu et al. 2013 Yang et al. 2019): : If the the magnitudes of the singular values of layer Jacobians deviates from onc, subspaces are attenuated ((o] < 1) or amplified ((or) > 1), potentially cascading exponentially over multiple layers (Pennington et al. 2017a). Formally, the behavior of stacks of matrices and nonlinear functions can be modeled by random matrix theory or Gaussian mean-field approximations (Poole et 2016 Pennington et al., 2017b 2018). The gist is that initialization, orthogonal weight matrices are needed, which is challenging for convolutional architectures. A solution for lanh-networks given by Xiao al. (2018); for ReLU, there is negative result (Pennington et al. 20176). Using mean-field theory, it can be shown that batch normalization (loffe & Szegedy (2015) leads to exploding gradients at initialization (Yang al. 2019) (which equalize after few steps, but that might be too late (Frankle et al. 2020)).
different route is taken by Balduzzi et al. 2017) who observe an increasing decorrelation of gradients in the input space. Similar to our paper, they show that deeper networks lead to spectral whitening (starting from brown noise); however, the analysis is performed with respect to the inputs x. not weights W. The scale-space structure shown by our model might give further hints on the mechanisms behind training difficulties.
By visualizing random slices of the loss surface, Li ct al. (0018) observe that the loss surface of deep feedforward networks transitions between nearly convex to chaotic with increasing depth; our work explains these observations by spectral analysis. Duvenaud et al. (2014) visualize pathologies on the landscape of deep gaussian processes that model deep wide-lima nonlinear networks. Fourier analysis of network functions (Candès) 1999) wrt. input (Rahaman al.) 2019 Xu et al.) 2019: Xu] 2018; [Basri et al., 2019) Yang & Salman 2019) has been used to show an inductive bias towards low-frequency functions (wrt. input x), as well as strong anisotropy of this spectrum, Waang et al. (2020) prove under some assumptions that all "bad" local minima of a deep residual network are very shallow.
3 HARMONIC DISTORTION
We now analyze the effect of nonlinearity by relating the Fourier spectrum of preactivation with that of its postactivation. Let of denote the preactivation of a single neuron of a neural network consisting of I.-layers. We use to denote the input to the whole network and thus f. W to denote
2

---

Published as a conference paper at ICLR 2021
the weights, and & to denote the employed nonlinearity. et al. (2018) visualize "roughness" using random 2D-slices in weight space. We follow their basic idea and consider 1D slices
p(() = f(x, W + Dt
(I)
for random directions D and t [0, 1]. D is initialized with entries from No,1 and normalized to a determines the path length. By varying D, this samples a ball of radius a around a
F = [DD[r point W in parameter space. For id, the network f is multi-linear in W and thus p is polynomial in t; empirically, this yields rather smooth functions (Fig. D To understand general nonlinearities better, we represent p by complex Fourier series (Appendix B.I discusses convergence and approximation quality):
p(1)= = Σ
2n exp (2tikl), 2k E C.
(2)
k=-oc
Here, the sequence :: + C contains the Fourier coefficients for p: [0, IR. As is real, x is symmetric in the sense of 2 2-k-
3.1 FORMALIZING ROUGHNESS
The roughness of a class of random functions can be characterized by the statistics of their power- spectrum [Musgrave| [993): Given a random process that generates functions p. we consider mean pa and variance o of the Fourier coefficients žg. For paths with short length as used in our experiments, the means μıd are empirically very close to zero except for 2g ("DC" coefficient), which is excluded in all experiments. Therefore, we can focus variance: In general, functions where the variance of nigh-frequency components drop off more quickly will appear smoother. common model, which often fits natural data well, is fractal Brownian motion (FBM-) noise, where the σr drop off according to power law:
ok 0 for some h > 0.
(3)
The so-called "fractal coefficient" describes the roughness of the noise function. similar approach has been taken by Hoffer et al. (2017), who modeled the loss surface by analyzing the dynamics of random walk on a random potential. In our experiments, we estimate the average power-spectrum (1x (by sampling D uniformly to quantify the roughness in a single number h.
unit sphere) and fit power-law to these spectra in order
Experiments (Section 4 and analytical arguments (Appendix B.2). show that the FBM/power-law model is a realistic model of the functions computed by the lower layers of a neural network. For higher layers the fit becomes worse, phenomenon we will explore in the next chapter using harmonic distortion analysis.
3.2
WHY THE OUTPUT FUNCTION GETTING ROUGHER?
Intuitively, applying ReLU to function p is reminiscent of clipping audio signal in amplitude, which is known to produce high-frequent ringing artifacts. We describe the effect of single nonlinearity on the spectrum of preactivation pt Inductively, this describes the spectral shift of the whole network. For the analysis, we assume that Ø is a K-th order polynomial:
K
((x)
a,
The effect of polynomial nonlinear maps the spectrum of function can be understood by harmonic distortion analysis (see e.g. Feynman et al.] (1965), Ch. 50.8). We can simply (see Appendices B.
B.3 for details) plug the Fourier expansion of p into the polynomial representation of Φz:
s(p(t))
Σd
aj 1=0
Σ
2h exp (2Trkt))
(5)
3

---

Published as a conference paper at ICLR 2021
The convolution theorem tells that j-th power of functions corresponds convolving the spectrum 2 of the function j-times with itself. We designate by & the vector containing all Fourier coefficients
2g and by X the convolution operator. We then write the spectrum of the output 2' as:
K
== =()(()
(6)
j=0
Discussion: We can make three important observations: First, each repeated auto-convolutions terms. We call this broadening effect blueshift. The exact magnitude is hard to quantify and thus left to the experiments (Appendix B.44 informal arguments for a growth of O()1/2 rather than the trivial upper bound of O())). Second and correspondingly. larger coefficients aj for larger orders j, increase the blueshift. Third, j-fold convolutions correspond to j-th powers of the žy- Hence, larger magnitudes | also increase the blueshift (the nonlinearity becomes more visible with larger magnitude).
in Eq. [ broadens the spectrum by adding
3.3 COMMON NONLINEARITIE.
relu
In practice, nonlinearities are usually not polynomial, and might not even have globally convergent Taylor series. It is possible to approximate any continuous function by polynomial (Stone-Werd Weieessrsssssssss theorem). We conjecture that a close polynomial approximation will be sufficient for a qualitative prediction of the blueshift effect (our theoretical model does not guarantee convergence we therefore validate this claim experimentally). We employ Chebyshef approximation (which is reasonably stable and non-oscillatory) on the interval [-5, 5] and compare the speed at which higher order coefficients drop off:
0.1
tanh softplus
0.001
0.0001
0.00001
5
10
Figure 2: Absolute values of polynomial coefficients: [tyl over jt see also Fig. 19
Fig. shows the coefficient magnitudes for ReLU, tanh and softplus (which we focus on in our experiments). softplus has the strongest drop-off, followed by tanh and ReLU. Appendix F.Igives more details and covers several additional popular nonlinearities (Figures 18 that this correlates with blueshift, as expected.
givees
19 Figure show
3.4 CONNECTION TO EXPLODING GRADIENTS
The creation higher- frequency harmonics directly affects gradient-based training methods because the gradient operator is high-pas, figh-ppss filter in the spectral domain: Taking derivatives multiplies the Fourier coefficients by 2Tik, amplifying coefficients linearly with frequency & (which is trivial seen by taking the derivatives of Eq. 2). The reciprocal fractal exponent r == h-1 has exponential influence on the average gradient magnitude: With | (1:-") = Okk"), ), we obtain gradients k [=x2]] E (21++)2) Blueshift thus causes exploding gradients: Weights lower layers are shifted more often, creating higher-order harmonics, which correspond to larger norm of the gradient function which means that there must be, average, larger gradients along the
[)) = 2T / 1:2 12|2z1², path considered. Independently of this, both exploding and vanishing gradients can be caused by other effects than blueshift: [Saxe et al. (2014) show that even deep linear networks can suffer from exploding gradients caused by stacking matrices with non-uniform singular spectrum.
The blueshift effect itself is independent of the condition number of the weight matrices: blueshift would occur even when only stacking nonlinearities. Although it is possible to find a locally linear region of the loss surface (e.g. CReLU + Looks Linear initialization (Balduzzi et al.) (017]) and locally achieve dynamic isometry (Xiao et al.) 2018), blueshift effects become relevant again when this region is eventually left during the training process (ref. Appendix Figure [2]. In contrast, residual connections affect the entire loss surface and our model is able to capture this effect.
3.5 RESIDUAL AND MULTIPATH NETWORKS
Residual Networks exhibit two different dampening effects on the loss surface that can be experimen- tally isolated:
4

---

Published as a conference paper at ICLR 2021
layer
(a) All layer walk power spectrum k
(b) First layer walk derivative pow. spectr. k & Ixxd
50
ReL.U
TanH
SoftPlus
ReLU
TanH
SoftPlus
no-short In(pow)
-
short cuts In(pow)
0
2
4
0
0
2
0
2
4
2
4
0
2
In(freg)
In(freq)
In(freq)
In(freq)
+
In(freq)
In(freq)
+
Figure 3: Normalized power spectra per layer (red is deeper) in 50-layer CNN at initialization, averaged over multiple neurons, for various Φ. (a) the left, weights are varied and spectrum of p(t) is shown. (b) On the right, only the first layer is varied and the spectrum of (t) is shown.
Exponential downweighting (ED): Residual networks repeatedly compute the sum of the result of a small feedforward network with its input. By construction, this reduces the relative weight of the nonlinear output of the residual block by mixing it with the unprocessed signal. Doing this repeatedly creates an ensemble (Huang et al., 2016) in which the weight of contributions shrink exponentially with the number of nonlinear processing steps. As detailed e.g. (Veit et al.] (0166), each nonlinear block only contributes partially to the overall result. The weight of the signal component passing through nonlinearities shrinks exponentially due to vanishing gradient effects (at initialization, non- aligned non-uniform singular value spectra lead to exponential dampening: after training, a similar dampening effect is observed empirically by Veit et al.).
rreueeny-deeeendee signal-averaging (FDSA): According the blueshift model, the outputs of the residual blocks will contain more additional harmonics depend the weights in the residual block, which are statistically independent of the input (see Appendix [B.5] for details). Therefore, we expect to see a reduction in the expected linear correlation between the main and residual branch for higher frequencies. As an effect, when adding the main and residual branch, higher frequencies get dampened even more.
Fourier coefficients (from harmonics). The
The second effect is weaker than the first, governed by the law of large numbers (i.e., n decay for averaging independent values), but is also more broadly applicable: It should generally occur when averaging over multiple computational paths with independent weights, while residual connections require an identity in one path.
EXPERIMENTAL RESULTS: MEASURED SPECTRA
The model presented in the previous section gives only qualitative hints the magnitude of the blueshift, the behavior of non-polynomial nonlinearities, and the magnitude of ED and FDSA effects. We therefore now validate these qualitative predictions experimentally.
We consider two architectures: a basic Toy-CNN model with a constant number of features in each layer for fair per-layer statistics and ResNet v1 variants for more "realistic" model. We used Cifar as input data, but all figures in this section look qualitatively similar on other datasets; A more detailed network description, parameters and experiments on MNIST can be found in the Appendix. section, we always measure the effect of blueshift in the region around initialization (i.e., W is the initialization point); During training, the loss surface tends get smoother but the effect persists (ref. Appendix Figure]
this
4.1 EFFECTS OF BLUESHIFT ON LOSS PATH
To show the effect of nonlincarity and residual connections on the loss surface, we sample a single loss path ResNet56 with Leaky ReLU activations. We can continuously tune the amount of
5

---

Published as a conference paper at ICLR 2021
NoShort Relu NoShort + SoftPlus
NoShort None
Short Relu
Short SoftPlus
Avg. path
2.5
2.1
100 Step
200
100 Step
200
100 Step
200
100 Step
200
100 Step
200
Mag.
0
50 Frequency
100
50
100
0
50
100
50
100
50 Frequency
100
Frequency
Frequency
Frequency
Figure 4: "Average loss path" and its respective magnitude spectrum of 500 random loss paths for different variants of ResNet56 at initialization.
nonlinearity by adjusting its negative slope a. We can equally continuously tune the relative weight of the residual branch by multiplying it with factor in all residual blocks. Figure [ shows transition from a linear feedforward to a nonlinear feedforward to a nonlinear residual regime; We can observe the apparition of harmonics with increasing nonlinearity that get dampened when switching on residual connections.
4.2 EFFECTS OF BLUESHIFT ON THE AVERAGE POWER SPECTRUM OF THE W-O SURFACE
We now want to view the effects of blueshift on the expected power-spectrum of the w-o surface, We "Toy-CNN" with depth L 50 and track the surfaces of each layer while walk in single random direction in weight space. We transform the resulting DD-paths with discrete FFT and average all power spectra over all neurons in a layer, all input receptive fields and all batch images. Fig-3h (left side) shows the resulting average power spectrum, normalized by function norm. The right hand side (b) shows the same plot, but with D restricted only changing the weights of the first layer of the network, and taking the derivative of the resulting function (by weighting by frequency, k22 |2 before normalization. As we use only 100 samples without prior bandlimiting, aliasing phenomena occur as slight upward slant is visible at the high-frequency end of all plots.
The results Fig. 3a confirm our predictions quite well: A spectral blueshift is clearly visible and, as expected, ReL. shows strong blueshift, tanh slightly weaker blueshift, and softplus only a small effect. Harmonics show up as a bump in the spectrum that travels towards higher frequencies with increasing layer. This extends other nonlinearties well (see Appendix, Fig. [7). The plot also confirms that the power spectrum of the lower layers is well described by 1 law, as predicted.
Spectral shift over depth:
Scale shift of single layer parametrization: By only changing the weights in single layer and taking the derivative (Fig. (3p), we see that blueshift is responsible for a gradient scale mismatch between earlier and deeper layers in networks without 8kip-connections: the more nonlinearities between the modified weight and the output, the higher the gradient magnitude.
Residual connections: Our model predicts that residual connections will reduce the blueshift strongly by ED and FDSA (ref. Section 3.5). This is also consistent with our observations: higher layers show almost the same 1 1/k-spectrum as the initial layer. Looking at the first layer weights, weighted by frequency, the response is almost flat, with small emphasis low-frequency weights.
4.3 QUANTIFYING AND MEASURING SPECTRAL SHIFT
Utilizing the pawer-law model presented in section 3.1 we measure blueshift by estimating the fractal exponent å of a loss path via a power-law fit of the magnitude spectrum. Figure represents what the "average loss path" looks like for ResNet 56 at initialization with different activation functions, with and without shortcuts. For each architecture, we sample paths in 500 different random directions D and assess the fractal exponent of every sampled path, in order to visualize the path

---

Published as a conference paper at ICLR 2021
|ReLU SoftPls I Tinil
sotoother
3.0
Averge abs. correlation
Joss suzfaoo smoodhees
2.5
0.6
0.4
reha ibst rela to thatt
0.2
roughter
slot pddipta deet
tash shoer
Frequency
0.5
15
20
25
30
35
45
Number of Layets
Figure Average absolute path correlation per frequency bin of the main branch and the resid- ual branch of a ResNet56 v2 with different ac- tivation functions at initialization. Lower bin number indicates lower frequencies.
Figure 5: Smoothness of the loss surface at ini- tialization for a Toy-CNN of varying depth with and without shortcuts.
with median h value and its associated spectrum. We clearly see that nonlinear architectures without skip-connections present most blueshift, the linear architecture has no blueshift and the architectures with skip ciipecoonectioos do also exhibit, albeit dampened, blueshift.
In Figure 5 we assess the average loss path smoothness of a Toy-CNN with varying depth at initialization. For averaging, initialize each network 50 times and sample 20 paths in different random directions for each initialization. The results confirm quantitatively the visual results Li al. (2018) that the loss surface becomes rougher with increasing network depth and that shortcut connections have strong smoothing effect. Since our roughness measure is a non-pointwise view, we that even the delta-initialized network that does not suffer from vanishing gradients initialization still gets rougher with increasing depth. The power-law fit still has some limitations: As seen in Fig. 3] power-law fit imperfect as the higher layers are not straight lines anymore. This shows in curves of Fig. as apparent increase of smoothness in the first layers for the "NoShort" architectures.
4.4
RESIDUAL NETWORKS CONTROL HARMONICS VIA ED AND FDSA
In Section 3.5
we described two different mechanisms that allow ResNets to dampen figh-frequency modes. We now want to verify the existence of FDSA and further individuate the two effects.
FDSA predicts that skip-connections have dampening effect high frequencies in the loss surface because the correlation between the main branch and the residual branch is decreasing with higher frequencies. We empirically verify this prediction 50 random directions and measuring the respective w-o function (we only sample 10% of the outputs for computational reasons) of the two summands in each residual block just before addition. Using Gaussian filters in Fourier domain, we can compute the linear correlation of each pair of paths (block input/output) per frequency bin and average In Figure frequency, with ReLU showing the strongest effect, again followed by tanh and softplus in that order, confirming our prediction. This property seems to allow residual network to automatically regulate the higher- frequency harmonics content of the output.
ResNet 56 v2 initialization by walking in
all batch images, neurons and blocks in the network. 6 we see that the absolute linear correlation is decaying for all architectures with increasing
We want to individuate FDSA in setting where ED is not present. We modify our Toy-CNN and ResNet "NoShort" architectures such that we replace every n-feature convolution / batch normaliza- tion / /activation sequence in the network with a feature sequence where the a feature groups get averaged after the activation (in reality we only use summation, since normalizing taken care of by batch normalization). Since high frequencies added by the nonlincarity are less correlated between features (ref. Fig. (), we expect that averaging feature groups smoothen the w-o function; we see on the right of Figure that this is indeed the case.
Finally, we want to demonstrate that the positioning of the activation function is crucial. Our model predicts that since ResNet v2 blocks (He al. (016bb use summation after ReLU, we will see stronger averaging effect than in ResNet vl. Indeed. looking at their respective average power spectra in random direction in Figure 7 see this confirmed.
7

---

In this section, we want to demonstrate a correlation between blueshift and network trainability. Again, use variants of the ResNet v1/v2 network, exact network parameters and training hyperparameters can be found in tabular form in Appendix A We use Cifar 10 as training data, training results for Published as a conference paper at ICLR 2021
Accuracy-
Blodo vn Bldda
Pow: spectrum
0m3
0,5
0,5
(an))
RVT NS
NS
6
0
200
400
600
800
Prequency
Step
logefres)
Figure 7: Normalized power spectrum of the last layer of a ResNet 94 v1/v2 (post/pre- activation) at initialization.
Figure 8: Left: Comparing the training performance on Cifarl0 of a feature-averaging ResNet56 vI "NoShort" to its vanilla, wide and residual equivalent. Right: Spectrum shift of averaging Toy-CNN 50 layers at initialization, varying averaging factor a.
5 EMPIRICAL RESULTS: TRAINING
Cifar100 can be found in Appendix [
Controlling Blueshift via feature averaging: We want to investigate the impact of the smoothing of the loss surface via FDSA (ref. Section 3.5) on training speeds. We choose ResNet56 "NoShort" as our architecture, since at this depth simple feedforward networks with batch-normalizatiio start become difficult to train. We experiment on the averaging network (ref. Section 4.4) with a since we experimentally determined that for a >> 3 the performance stops increasing. For faimess, we included a Wide ResNet56 "NoShort" with approximately the same parameter count than the averaging network. We see the averaged results over five runs in Figure FigureB(ssaded (shaded areas represent the standard deviation). We observe that the averaged network outperforms both the original network and the wide network while still performing worse than the network with skip-connections. This is consistent with our theory since ED has a bigger smoothing effect than FDSA.
As demonstrated in Figure the amount of blueshift in a network can be controlled by tuning Leaky ReLU's negative slope a. We now want to show that networks with very strong blueshift are hard train but conversely needed for expressivity. For this, we train a ResNet56 v1/v2 with and without shortcuts for 30 epochs once on Cifar10 for different values of and visualize the training accuracy (average the last 25 values) and compare it to the blueshift of the network at initialization. On Figure 9 we that for the networks without skip-connections, training becomes more difficult with regard to network depth, which correlates with increased blueshift. We see that this effect can be alleviated by making the network more linear (by increasing a) and thus reducing blueshift. For networks with skip-conecctions, we see that blueshift is greatly reduced and that no deterioration of trainability with regard to network depth is noticeable. Conversely, networks that near línear regime for high values of a tend to train worse since they lack expressivity. Interestingly. this effect is stronger for ResNet v2 blocks than for ResNet v1 blocks because the former make the loss surface smoother than vI blocks (ref. Figure? 7).
Controlling Blueshift with Leaky ReLU:
amount of nonlinearity
6 DISCUSSION
Our experimental findings validate the predictions of our model with respect to the roughness of the w-o function: Nonlinearities with larger higher-order polynomial coeffieents create larger variances in the high-frequency part of the spectrum. While related results increasing complexity with depth have been given earlier (Poole et al. 2016 Schoenholz et al. 2017). the harmonics model offers simple and more global view of the roughness of the loss surface and how it is affected by nonlinearity choice and residual connections. Our observations on training speed are consistent with the hypothesis that spectral blueshift impedes training: Architectural choices (residual connections,
8

---

Published as a conference paper at ICLR 2021
RecNet Short
NoShort
0.8
0.4
0.4
0.2
0.2
3.0
2.5
2.0
3.4
0.2
0.5
20
32
64
56
68
180
02
104
20
52
44
56
6.5
80
92
304
20
44
56
68
80
104
Network depth
Figure 9: Comparing w-o smoothness and trainability for ResNets with varying depth and amount of nonlinearity. Upper row: training accuracy after 30 epochs of training Cifarl0. Bottom row: blueshift at initialization indicated by the fractal coefficient of the network's last layer.
ResNet v2 blocks, LReLU) that control harmonics generation are also those that are easier to train to the their full performance. Our model also consistent with the empirical observations Han et al. (2017). who found that ResNet performance varies with the location and number of ReLUs. Further, the results in Figure 9 give strong hints for correlation between high-frequency content and problems in trainability for deep but very nonlinear networks (small a). From analytical perspective, architectures that are prone to blueshift are prone to vanishing/exploding gradients: We have shown that high magnitudes of high-frequencies in the functions lead to increase in the expected gradient magnitude. Further, in simple multi-layer network without shortcuts, we expect more high-frequency content lower than on higher layers as soon as the nonlinearity of the activation function is actually is utilized (and therefore must also creates harmonics). A rescaling of layers can reduce these discrepancies, but the normalization will still necessarily depend on the position in parameter space. As far as accuracy is concerned, there is an obvious trade-off between blueshift and expressivity: networks with low-degree polynomials as nonlinearities contain less high- frequent harmonics but cannot approximate complex functions (Kidger & Lyons 2020). Future work could consist in further exploring this trade-off. Further, it would be interesting to examine potential architectural alternatives to residual connections with comparable performance and trainability.
CONCLUSION AND FUTURE WORK
In this paper, we have analyzed the effect of different architectures of nonlinear networks on the smoothness of the computed function. Specifically, we have linked polynomial nonlinearity to harmonics ereation and validated experimentally that the power spectrum shifts towards higher frequencies ("blueshift"). Further, we have described two distinct effects that explain the smoothing effect of shortcut connections, which also confirmed experimentally. Finally, empirically linked reduced blueshift to increased trainability and training speeds.
In future work, hope derive explicit closed-form equations for how architectural designs like network depth, residual connections and nonlinearity choice affect the w-o function. It could be also interesting to further explore the roughness roughness-expressaleee expressability trade-off that we described earlier.
ACKNOWLEDGEMENNTS
This work has been partially supported by the RMU Network for Deep Sictinuouu-Discreedd Machine Learning (DeCoDeML project). We wish to thank Jan Disselhoff for valuable discussions and the annonymous reviewers for their helpful feedback.

---

Published as a conference paper at ICLR 2021
REFERENCES
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 342-350. PMLR, 2017.
Ronen Basri, David W. Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8- 14 December 2019, Vancouver, BC, Canada, pp. 4763 47722 2019.
Emmanuel Candès. Harmonic analysis of neural networks. Applied and Computational Harmonic Analysis, ((2):197-218) 1999.
David Duvenaud, Oren Rippel, Ryan P. Adams, and Zoubin Ghahramani. Avoiding pathologies in very deep networks. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014, volume 33 of JMLR Workshop and Conference Proceedings, pp. 202-210. JMLR.org. 2014.
Richard P Feynman, Robert B Leighton, and Matthew Sands. The feynman lectures on physies; vol. i. American Journal of Physics. 50(8), 1965.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. CoRR, abs/2002.10365, 2020.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, 2011.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. CoRR, abs/ 1l1.04444,4,4 2018
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6307-6315. IEEE Computer Society, 2017. doi: 10. 10.1109/CVPR..2177..o
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034. IEEE Computer Society, 2015. doi: 10. 10.1109/CCV.2015.1222
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016a. doi: 10.1 (009CCPR.2006..99
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jini Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 630-645, Springer, 2016b. doi:
0111///788-331-44492-
S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma thesis, TU Munich, 1991.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the general- ization gap in large batch training of neural networks. In Isabelle Guyon, Ulike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1731-1741, 2017.
10

---

Published as a conference paper at ICLR 2021
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646 661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Pattem Recognition, CVPR 2017, Honolulu. HI, USA, July 21-26, 2017, pp. 2261-2269. IEEE Computer Society, 2017. doi: 10.1009/VVR.2207...4..
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org, 2015.
Patrick Kidger and Terry J. Lyons. Universal approximation with deep narrow networks. In Jacob D. Abernethy and Shivani Agarwal (eds.), Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event |Graz, Austria], volume 125 of Proceedings of Machine Learning Research, pp. 2306 2327. PMLR, 2020.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Desa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 6391-6401, 2018.
F.K. Musgrave. Methods for Realistic Landscape Imaging. PhD thesis, Yale University, 1993.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 1310-1318. JMLR.org, 2013.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through theory and practice. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4785-4795. Curran Associates, Inc., 2017a.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4785-4795, 2017b.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In Amos J. Storkey and Fernando Pérez-Cruz (eds.), International Conference on Artificial Intelligence and Statistics. AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pp. 1924-1932. PMLR, 2018.
Ben Poole.
Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3360-3368, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- -oanning-Volumee 70, pp. 2847-2854 JMLR.org, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5301-53 1301-5310. PMLR, 2019.

---

Published as a conference paper at ICLR 2021
Andrew M. Saxe. James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations. ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In Proceedings of the 5th International Conference Learning Representations, 2017.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, /es//1555008877 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826. 2016.
Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Daniel D. Lee, Masashi Sugiyama, Ulike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 550-558, 2016.
Lifu Wang, Bo Shen, Ning Zhao, and Zhiyuan Zhang. Is the skip connection provable to reform the neural network loss landscape? In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI 2020, pp. 2792-2798. ijeai.org, 2020. doi: 22244966jaai.
Lechao Xiao, Yasaman Bahri, Jaseha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla con- volutional neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5393-5402, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In Tom Gedeon, Kok Wai Wong, and Minho Lee (eds.), Neural Information Processing 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12- 15, 2019, Proceedings, Part I, volume 11953 of Lecture Notes in Computer Science, pp. 264-274. Springer, 2019. doi:
22.
Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis. CoRR, abs/1808. 1ess0988 2018.
Greg Yang and Hadi Salman. A fine-grained spectral perspective neural networks. CORR, abs/1907. 2019.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel Schoenholz. A mean field theory of batch normalization. In 7th International Conference on Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
A NETWORK DETAILS
A.1 NETWORK ARCHITECTURES
The implementation for our experiments is based on PyTorch 1.5 and are provided as supplementary material.
We create a simple convolutional network consisting of units of 3x3-convolutions with 16 feature channels on all layers (16 for the averaging networks), followed by batch normalization and non-linear activation, stacked-up L-times. Optional short-cut connections are added (only explicitely stated) between each unit (i.e., residual blocks of depth 1). No pooling striding is used. The input layer uses a stride of 5 to reduce memory demands. The classification layer is composed of fully-connected layer, softmax and cross-yntropy loss.
Toy-CNN:
12

---

Published as a conference paper at ICLR 2021
As "real-world" example, we standard Residual Network v1 (BasicBlock) He et al. (2016a) with varying depth, with ("Short/S") and without ("NoShort/NS") ciip-connections. ResNet v2 blocks
Residual Network:
used in Figure T (v2 blocks show slower transitions) 6 (correlations decaying faster in ResNet v2, ref. 3.5). We use the standard number of planes for Cifar-10 training 16/32/64 (28/56/1 12 for the wide network).
A.2 FOURIER WALK HYPERPARMETERS
We sample paths consisting of 100 samples each and with path length a = 1 (Eq. in all experiments, except for Figures D and 6 where double both values for higher frequency resolution and Figure where we want to walk in different radius (a E {0.1, 1, 10)). All experiments are performed training mode and with data augmentation at initialization. Datapoints for each architecture are measured on different mini-batch to exclude bias by batch selection.
12
A.3
TRAINING HYPERPARMETERS
The Syper-parameters below usually reach the standard test-accuracy of approximately 92-93% for ResNet56 on Cifar10.
Dataset
Cifar10 (Cifar100 for Figure 10
Epochs
200
Scheduler
Multistep (y = 0.1)
Milestones
100, 150
Learning rate
0.1
Batch size
128
Optimizer
SGD + Momentum
Momentum
0.9
Weight decay
0.0001
Augmentation
Random Flip
B FORMAL DETAILS CONCERNING HARMONIC NETWORK ANALYSIS
B. SERIES REPRESENTATION AND CONVERGENCE
In our paper, consider feed-forward networks consisting of L layers. Each layer I 1...L can be one of the following:
An affine map
- W (t)
that transforms the input using linear map and an offset vector.
A nonlinearity
x (x)
(8)
that applies (potentially layer-specific) nonlinear function to each input.
All of the discussed architectures (including convolutional networks, striding/average pooling, resid- ual connections, parallel computations, and layer-dependent nonlinearitiy) can be expressed as a composition of these layer types. We currently do not address max-pooling, and understand batch-normalizzttion layers statically as a fixed scaling and shifting (not analyzing their training dynamies). The loss function itself (such sequence of softmax and cross-entropy) is also not included in our analysis we restrict our consideration to layer outputs before any classification. As logistic regression similar problems) numerically well-understood already, this is not major obstacle. All our experiments show layer outputs before classification (softmax).
The main restriction of our formal model that we assume that all nonlinearities are polynomials of finite degree Under this condition, all outputs and intermediate results of the whole network can be
'Tis means that the analytical results are currently not established for practical non- can-pooyyooia nonlinearities. The results in our paper obtained from polynomial approximations should therefore be considered experimental results at this point. An analysis of infinite series approximations covering more general class of nonlincarities is still left for future work.
13

---

Published as a conference paper at ICLR 2021
represented by large multi-variate polynomial, and any linear section p(() 1/0 (W alD,x) x) of any output of any layer is univariate polynomial of finite degree.
Polynomial functions finite domain are Lipschitz-continuuus and have finite variation; both conditions are sufficient for the convergence of the Fourier series in an Lg-sense. Further, as the Fourier-basis forms Schauder-basis of L2(0, 1]. the series expansion is unique.
Fourier analysis:
In our derivation (Equation 5p. we plug the series representation into the polynomial and conclude that it can be represented in the spectral domain as sum of convolutions (Equation 6 To make this more rigorous, we proceed in smaller steps: Let again R" - IR denote the linear section along linear direction in parameter space of any layer output. Our nonlinearity expands
K o(p(t)) Easptyy.
As p(t) is a polynomial of finite degree, p()) for any finite is also a polynomial of finite degree. This implies that there is a unique Fourier series that represents characterized by the coefficients sequence 2 + C. According to the convolution theorem for the complex Fourier series, the Fourier series of p p is given by @ where
[2 ula
Zm
Ug 1m-
(10)
Because the series must exist (it converges for both p and p²) and it is unique, the product of the series must be represented by the auto-convolution. Inductively, we obtain p² corresponding to the j-fold auto-convolution
X 4=1
Note: We use the notation [x]: == Hú denote the indexing of vectors and sequences that result from
computation.
Approximation quality and finite transforms: The Fourier representation is accurate in L2- sense; converges L2-norm, which an integral measure. This still permits any finite series expansion to have large point-wist errors (no convergence in infinity-nore large eπror regions rather shrink to a zero set in the limit). A common example of such problems is Gibb's ' phenomenon: A discontinuous function (such as the gradient function of ReLU) will show "overshooting" of constant magnitude near the discontinuities; only the area affected shrink with the frequency order of the Fourier series. Further, even higher-order discontinuities lead to significant high-frequency harmonics (i.e., show bad approximation behavior for truncated series).
While the theory transformation (DFT) we use in our experiments. The DFT operates at finite frequency order, using a matching set of regular samples of the continuous function. Here, it is important note that the DFT, as a mapping from Cd to Cd will reproduce the function values faithfully at each sample point for any frequency order (the DFT bijective and even unitary, thus even allowing numerically accurate and stable reconstructions). Evaluating the obtained series in between sampling points, however, would reveal issues at discontinuities; as in most applications of DFT, our experiments do not perform such continuous evaluations.
affected by this, this might appear be concern for the discrete Fourier
An issues that does remain aliasing: Nonlinearities in general and in particular discontinuous ones (even with higher order discontinuities) broaden the spectrum (this effect is particularly strong for first-order discontinuous functions such as ReLU). Therefore, the sampling frequency has to be chosen sufficiently high to capture these effects. Otherwise, the high-frequency harmonies will alias as lower-frequency signal components, which might lead to an underestimation of the blueshift effect.
B.2 INPUTS TO THE SECOND LAYER NONLINEARITIES ARE FBM-NOISE
For the first layer of ReLU network, the following calculation also supports the experimental findings of an ((1/k)-power law from an analytical perspective. Let p!)) be a w-o path the output of given neuron from the first layer. It follows that:
14

---

Published as a conference paper at ICLR 2021
p (t)
ReLU Σ w + +a o zz
(11)
ReLU [m - + n²], for some T2 E R.
(12)
This is a piecewise linear function that is constant zero on one interval and linear with slope m on another interval. With < b E [0, bounding the activation interval, the Fourier series becomes:
/."
(1,4) 2k
exp exp (-2wik (mt + n)) dt.
(13)
By the chain rule of derivations, this implies
E
(14)
*
1:
Correspondingly, the preactivations of the second layer will be random linear combinations of FBM- noise with h = 1. which again forms FBM-noise with that spectral variance (every Fourier coefficient a sum of independent coefficients; therefore, variance is additive).
Similar findings where also previously shown in input space by Balduzzi et al. (017)) which, for the first layer, is just the dual to varying the weights (and therefore must yield the same result).
B.3 BLUESHIFT AND DEPTH
We now aim formally understanding the blueshift effect in multi-layer network. For simplicity, we consider a simple stack of linear and nonlinear layers with a fixed nonlinearity:
[xx w) W(L) (w(L-1)... (WW()))..))
(15)
We now consider a single output of layer
1 x W) o w/ ( (
(16)
Next, we consider a linear section + W + iD with & E (]
p (1)
(( w!
(1V0) -ID) (wil-)
(DD-))) ((D)) + ". (D(")))..)
By replacing with finite polynomials (Equation 4 obtain multi-variate polynomials outputs of all of these function.
Varying parameters in single layer: we now select layer & by setting all ( for all
i + k and only using the k-th matrix as direction vector, the parameter will pass through / - k + nonlinearities:
w!!h 10 . pru--)....((w) +((()) ( (g (o................... )).
p (1)
(17)
with a constant vector y (R-1))
Correspondingly. each nonlinearity will act the spectrum of (*) as blueshift operator (forming the sum of repeated auto-convolutions of the spectrum, weighted by the polynomial coefficients). Therefore, see that parameters of earlier layers (small layer index k closer to the input layer) blueshifted more frequently than later layers.
The equation also shows that the singular value spectrum of W(() affects the results as well, in addition the spectral shifts caused by the nonlinearities: the values rescale the input domain
15

---

Published as a conference paper at ICLR 2021
(shrink/expand space wrt. parameter t), thereby changing the frequency reciprocal to the scale factor. However, a linear transformation scales the Fourier spectrum as whole while nonlinearities spread the spectrum irreversibly by applying blueshifts (weighted sums of auto-convolutions of the spectrum). The magnitude of this nonlinear effect is, nontheless, dependent on the signal magnitude and thus affected by choices W
Varying parameters in layer 0..l: Similar arguments hold if we vary only parameters in the first / layers of the network. However, we obtain a mix spectra that have been blueshifted (and scaled by weight matrices) different number of times.
B.4 SPECTRAL BROADENING THROUGH AUTO-COUT AUTO-CONVOLUTIONS
The exact amount of broadening of a spectrum z : + IR by an auto-convolttion is non-trivial to quantify; we therefore resort to experiments for determining the effect in practice.
In special cases, we can analyze the effect:
Band limited functions: Assuming that z is bandlimited, specifically 2 z 0 for all k K, and |z = | > 0 for & S. K it is trivial to see that an auto-convolution extends the support by entries to each side (2k overall). This means, after j convolutions, we obtain non-zero spectrum within k jj.... The proof is obtained by considering the left-most and right-most terms in definition of the discrete convolution.
Central limits:
If we assume that z is real and positive, and has finite second moments, the central limit theorem guarantees convergence towards Gaussian function with standard deviation O (vJ).
The first assumption is obviously unrealistic in practice.
the general case the j-th power of a Fourier series
K
Σ
2 exp (2Tikt)
(18)
expands to the sum
2k 22, exp (2mi(k) + k2 .... k,)).
(19)
K
k
K
If only count the number of terms of the same frequency k occurrences of each frequency will tend towards normal distribution with variance j. However, as the complex coefficients can cancel out, the limit distribution is only an upper bound.
ky. the number of
B.5 FREQUENCY DEPENDENCE IN AVERAGING OF MULTIPLE COMPUTATION PATHS
We observe that averaging functions after nonlinearities leads to smoothing of the result by decreasing the correlation between higher-frequency Fourier coefficients. In order to understand the effect, we look simple ID model case: We consider the composite functions
(posr(x) = o6 [1 g(x) + g(x)
(20)
=thy'
and
(pre(x) = s(wi g(x))
d(u2 g(a))
(21)
=ths
=tha
where W1, We random numbers, drawn from the normal distribution.
Let 2g be the sequence Fourier coefficients g. and ug that of the result. In the first case, fposts a random linear combintation of the input spectra performed, which is then blueshifted:
w1z z + W22)
(22)
16

---

Published as a conference paper at ICLR 2021
Training
Validation
0.8
0.8
Accuracy
0.1
0.4
0.2
RN56 (a=1)
0.2
RN56 NS (a=1)
(a-3)
Wide RN56 NS (am1)
Step
Step
Figure 10: Repeating the experiments of Figure for single run on the Cifar100 dataset.
In the second case,
W
(W2x),
(23)
first blueshift the randomly-scale spectra and then form the sum. This creates frequency dependence because larger powers (larger values of j. corresponding to more repeated convolutions, and thus j-fold products of input Fourier coefficients) behave strongly to the weight scaling: While a linearly weighted average will be linearly correlated with the original, averages of nonlinearly transformed functions will loose linear correlation (and thus appear more random to a simple averaging operation). The harmonic distortion analysis that expresses the powers cuto-convolutions of the spectrum shows that higher- greeueffr components (created by the blueshift) are also the ones that behave more nonlinearly.
nonlinearly and thus react more
C TRAINING ON CIFAR100
We repeat the experiment on averaging- networks for the Cifar 100 dataset, holding out 1% of the training data for validation. We leave all hyperparameters untouched. In Figure T0, we still the advantage the neeraging-networkk over the regular network, especially validation accuracy.
D OTHER SPECTRAL SHIFT FIGURES
D.1 SPECTRAL SIIFT DURING TRAINING
To show that the blueshift effect is not limited to the initialization, show the spectral shift of a Toy-CNN 50 at 0 and 20 epochs of training in Figure [I] ResNets still do not show any major blueshift at any time. The ReLU and TanH architectures without skip-coonections show less show blueshift after 20 epochs of training, indicating that blueshift is more of a problem in early phases of training (Frankle al. 2020). When increasing the sampling radius
10, blueshift is clearly visible again for ReLU and TanH activations in the networks without skip-connections.
D.2 SPECTRAL SHIFT WITH
INITIALIZATION
The experiments of Figure 3 with batch normalization are subject exploding gradients. Xiao et al. (2018) describe an initialization scheme that fixes exploding gradients for TanH activations. Fig 12] shows that for very small region near the orthogonal initialization point, blueshift vanishes. As soon walk further from the initialization point, blueshift becomes visible again.
D.3 SPECTRAL SHIFT IN GRADIENT DIRECTION
The subspace the loss surface where gradient descent operates has been studied (Gur-Ari et al. 2018) and attributed specific behavior. We want to how blueshift behaves when slicing the
17

---

Published as a conference paper at ICLR 2021
All layer walk during training
layer walk delta-orthagonal
Ep. [al
Relu
TanH + SoftPlus + NS
Relu
Tanh + NS (Delta)
log(ing)
0.1
log(maag)
(og(g)
logefrog)
log(freg)
logefreg)
log(frog)
Ep. 20
Relu NS
Tanll NS
SoftPlus NS
Relu
Tanh NS (Delta)
(or
log((aag)
log(nnag)
(freg))
(fg((re))
log(freq)
log(fres)
log(freq)
Ep. 20 10)
Relu NS
TanH NS
SoftPlus + NS
Relu S
Tanh + NS (Delta)
log(mag)
10 log((mag)
log(Ireq)
log(ireg)
logg/req)
log(Ireq)
log(freq)
Figure 11: Repeating the experiments of Figure 3 (all layers, unscaled) during training.
Repeating the experi- ments of Figure 3 (all layers, un- scaled) for TanH activations with delta-orthogonal initialization for dif- ferent path lengths
Figure 12:
All layer walk in gradient direction
All layer walk with varying width
Relu
TanH SoftPlus
NS (wm10)
NS (w=50)
NS (w=90)
log(mag)
1og Codmme
(og(Fres)
Ing(freu)
(reeq)
logí free))
log(free)
log(freq)
Relu
TanH+ S
SoftPlus
S (w=10)
S (w=50)
S (w=90)
logtmagg
log(nagg
log(treq)
log(Ireq)
log(Ireg)
log frog))
log(Ireq)
log(Ereq)
Figure 13: Repeating the experiments of Figure (all layers, unscaled) at initialization gradi- ent direction.
Figure 14: Repeating the experiments of Fig- ure (all layers, unscaled) for variable network width.
network in the direction of gradient descent. For this, we repeat the experiments of Figure 3 but compute the gradient initialization, normalize it and use it as direction of equation ! In Figure 13 we see that the effect is still visible although dampened for ReLU and tanh activations. The effect seems a little more visible for softplus activations.
D.4 VARYING WIDTH
To see if layer width has any influence on blueshift, repeat the experiment of Figure 3 for Toy-CNN 50 with number of filters w. We can see on Figure 14] layer width doesn't seem have notable impact blueshift strength.
18

---

Published as a conference paper at ICLR 2021
Loss paths (single layer walk)
4.9
Short + TanH
Short + ReLU
Short + SoftPlus
NoShort + TanH
NoShort ReLU
NoShort + SoftPlus
4.7
NoShort + TanH (Delta init.)
3
step
Power spectra
Fractal coefficient
2
log[pow)
-10
-10
10
20
30
40
50
log(freg)
Number of layers
Figure 15: Resulting loss paths and respective path-normalized power spectra when varying parameters in only one layer of Toy-CNN 50 at initialization. The color path indicates in which layer parameters were varied (red is deeper).
Figure 16: Smoothness of the loss surface initialization for a Toy-CNN of varying depth with and without shorteuts MNIST for batch size of 256.
E WALK PER LAYER
To the contribution of each layer to loss path, sample random direction D and measure the resulting loss path when only modifying weights in single layer. We realize this for a Toy-CNN with 50 layers initialization in Figure We that varying weights in higher layers (red) results in smooth paths whereas varying weights in lower layers (blue) results in rougher paths. In the normalized power-spectra, we can see the increased blueshift of paths resulting from varying earlier layers.
F SMOOTHNESS MEASUREMENTS ON MNIST
To show that smoothness measurements mostly independent of batch size and dataset, we revisit Figure 5 This time, we use the MNIST dataset and a batch size of 256. All other hyperparameters are maintained. We see that although the absolute values slightly differ, the relative behavior of the nonlinearities is qualitatively the same. The results are shown in Figure 16
F.1 MORE ACTIVATION FUNCTIONS
To demonstrate that blueshift occurs with every nonlinear activation function, repeat the experiment of Figure 3 for exponential linear units (ELU), gaussian error linear units (GELU), hard tanh (HTANH). leaky relu (LReLU), scaled exponential linear unit (SELU) and sigmoid activations.
In order to demonstrate the that spectral spread depends on the magnitude higher-order polynomial coefficients, we show the plot of the magnitude of the coefficients of polynomial approximation
with Chebyshev nodes of degree 25 in Figure 19 The fit is performed for the interval [-5, 5] (slightly
different from Figure 2 in the main paper, which shows fit for [-1,1]). Figure 19 a) also shows again
19

---

Published as a conference paper at ICLR 2021
All layer walk with more activation functions
ELU NS
GELU NS
HTANH + NS LRELU + NS SELU NS SIGMOID NS
-2
2
-2
log(freq)
log(freq)
log(freq)
log((ree)
log(freq)
log(freg)
ELU + S
GELU +
HTANH S
LRELU+ +
SELU +
SIGMOID +
-2
-4
-4
1
4
0
0
2
log(freq)
log(freq)
log(freq)
log(freq)
log(freq)
log(freq)
Figure 17: Repeating the experiments of Figure 3 (all layers, unscaled) for more nonlinearities.
Polynomial Chebyshev approximation of various nonlinearities of degree 25 within [-5,5]
relu
tanh
softplus
clu
gelu
htanh
sigmoid
selu
Irelu
Irelu (a 0.25)
Irelu (a 0.75)
Figure 18: Various Nonlinearities with their Chebysher-Polynomia Approximation within the interval 5).
the comparison between ReLU,, tanh, and softplus, uniformely with this Chebyshef approximation; this shows the differences more clearly (please also note the logarithmic scale of the y-Axis).
The approximations are quite tight within the interval, see Figure 18 for reference.
20

---

Published as a conference paper at ICLR 2021
Comparison of the magnitude of the polynomial coefficients [ay] (for the degree 25 approximation of Figure 18)
0.01
0.1
15:08
16:11
0.01
0.001
10:17
relu
1P-31
softplus
softplus
15:25
000001
10
(a) comparison of relu, tanh, softplus
(b) close-up of (a)
0.01
0.01
18:03
12-43
16-11
1E-17
1F-17
PCU
SoNu
Sath
cigroid
10-23
1F:06
15:25
(c) comparison of relu.tan .hartanhh (relu for reference)
hardtanh
(d) comparison of relu, elu, gelu, selu
0.01
10:08
1843
18:4
18:7
rreu
x Ielud.0
Xisluc.::
18-23
15:26
10
(d) different variants of leaky-relu (default is a = 1F 2)
(e) closeup of (d)
Absolute value of the polynomial coeffients the Chebyshev polynomial approximation. All fits are done within the interval [-5, 5). Note: Different interval from Fig. 2 in the main paper! (to capture the shape of all nonlinearities well).
Figure 19:
21

---

Published as a conference paper at ICLR 2021
2
8
relu
6,251221 RE-02
XIIIXX00-001
5.08969TE-0 E-101
119240155..4
2NR28407E-01
tnn
5.18050028 16
9.9999266-00
2.76P886E-11 15
3.207043EE
4621522N8- 15
LO8S1699B OE
3. 3,22245422- 15
6.9314722F-01
8.0003221-115
6.6211234-15
2.85547855-11
8542686F-12
L677S199E-00
gelu
1.0220576-06
858921821-01
198601246115
7,1426855115
9.9256157EE
haanh
-64522701--17
D,524831 TE-O1
S. A1101720E- 15
69176966641
4.5580022-01
LI6001 08F-15
6.1890911-02
3.644477--01
2. SHAASOZEE-O
-22879677F-02
361900145:-01
708969465-0
signoid
5,00000IF-01
4999980-001
4.5774799-11
2.06306176-02
066100255-115
107891796-00
DO6GN T4R-15
telu -0.25
468841661-42
6. 25200000--00
606447688-01
94213822-15
4,29124522-01
4,58012538-15
2. 21126300E--0
belo
16780535-02
401711125-15
-14364151F-01
-591618677-15
7.0821018F-00
210
21
e22
1,901099
1655978E--0
11229566-00
unnh
2.9902288E--22
1578462E-15
99914500E--0
2.4554185B-1
3.888 4351-43
3.3129449--17
8.8697176E-06
softplus
3.27648371 15
2.5656019E 05
102625008 15
99069549 06
2. 1-246631E-11
16
1,29945291L 00
3,040666 LE: 17
elu
-4.5051680E-00 02
4,3845220E
1.3337777E
7,8485900--
10136655-41
9.6036723E 0S
2.1 1.176866EE
pelu
2,0301054E-1
-116600006-03
47447800-
1880443E--0
5.4550867E-17
7,94441906-00
L66608236-
hunn
3.37381738-01
L.5872645E
99 5223816E-02
3.6951250B-18
L88671588
3.8818441 17
-L L.987481
Irelu
-L.3304598EE--
4.48111233-02
7. 66069561E 16
16394181E-02
2. 1268001E-16
2.1017260E--
3. 3,5009736E: 17
selu
-7992053288-02
S 2914448T--0
01344500--0
3. 34276777-41
L33293326-00
3,8300446-4E 04
sigmold
2.073982HE
Indu 0.25
1.6622433-11
6. 6.6273806E-02
6. 55057758-17
12419EME-02
19999355-17
-159222667B-00
L6274033E-
Inlu -0.75
56E-15
143460022-02
-7. 1728855--16
4.1399466--03
L02114788-16
5.3073880E-04
-8. 19623066-118
15
16
2
relu
848495799-84
21695827
43183633-007
DR 20120008-30
-I (996 F54SE-08
176683177-19
499792278-21
4.20572568-008
L7666NGE--
softplus
2.96267288
elu
8.0662081 FE-06
1.72992235-05
-4 677705945-877
445642655-077
L820
-..50947768-00
-. 59984277-100
gelu
4.5000588--077
2.2272244-118 18
1.74814082
6,155829998 10
8.0667968-21
+L. ALBUSZS 198-11
1522-152-11
826685000-00
LS1174446-20
7F-22
1.5320901--19
solu
-.1625322.04
9.03610091-05
689108990000
166255377-00
2273472022-477
6.16995758-08
6,9523220E-O 09
signoid
-13305631--18
7,2716001F-09
11384477-200
119742088-10
-1 110999779-211
1.00055200-11
7.6307576F-24
lilla
1.3822644--04
-20580252--18
17792600-06
1.5112244-119
3.2386755-07
3.48614108-21
J. 4.22100000-009
46108947E-05
292043318-19
-272577535-66
0.29918998-22
L87956888-07
-1141173688-1
-2.74033438-00
[21
22
22
24
2G
relu
3.0296993
166 1360E- 10
5 5,0295283--2 24
1.0129154E
3.6122190E
unh
2. 2.7966355E-1 10
-2 2.566760E-22
-3 3.81312800-11
164103228-26
2.31
softllu
2.4 2,1004255-22
94645005--15
-3 3.87667577--4
..7796077--11 17
2. 2225501
gelu
2,54291 10E
66296233-113
-4 442790160E-24
961170035--1
3.10743366-06
hranh
-7 000894537-00
4866408T-22
1. 11077240-100
4445173622F22
-6 668651 2677-11
Itelu
3,2080:18E
5952472E
5.17167715B-2
-L0224672E
3.6319817E:
AS008777E-09
1222263E-10
1.0005331--11
128824856-11
signoid
3841516F-25
-2 15
27.8476055-227
1,57777
Irelu 0.25
2. 2.1839993--22 22
20882206 10
3 3.6215891E-24
-7.82187268-11
2.5987741B22
Irela -0.75
2,02122119-23
02844000E-11
-1 21.0223122-244
13
8.490951 1E:27
Table 1: Numerical values for the polynomial coefficients used in Figure 18 and 19
22


Reviews:
[{"review_id": "U9OXDu9-nAB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a new approach for how to analyze the ruggedness of the surface of the neural network loss. Specifically, the paper proposes to apply harmonic distortion on the weight-to-output (w-o) maps. That is, the method casts the w-o functions in the Fourier domain and then aggregate the surface characteristics by virtue of averaging the different order Fourier coefficients. The  paper shows that non-linearities are responsible for blueshifting with deeper layers, that is for \"transferring more energy\" on the higher frequencies. The consequence is rougher surfaces, as well as higher frequencies for gradients, which can lead to exploding gradients in the deeper layers. The remedy is with skip connections and feature averaging, which although are methods already known to improve trainability, the paper corroborates that they also make sense in terms of said approach. The paper conducts various empirical and ablation studies, providing evidence of the claims.\n\nThe strengths of the paper:\n- I believe that at the core the paper offers some really nice preliminary ideas and intuitions. It is very intuitive that deeper layers generate higher frequencies to the loss and thus can make the optimization harder. On the other hand, higher frequencies are necessary for expressivity. This is another instantiation of the bias-variance tradeoff. By finding ways to balance the relative strength of high and low frequencies, one could throttle how much expressivity is necessary for the task at hand.\n- The paper goes to great lengths to support some of the claims empirically. There is lots of different experiments and it seems the claims are overall supported by the findings. Different nonlinaerities and architectures (maybe too much, in that there is less focus) are explored, which is admirable.\n- I like the quality of the visualizations. It is clear that the authors have spent quite some time in generating their figures.\n\nThe weaknesses of the paper:\n\n- Sure, if we focus only on the low-order frequencies trainability is better. Also, I dare say that the insight is not exactly surprising although very intriguing. However, what exactly is the message? That we should have only low frequencies? Or that we should have some high frequencies? That skip connections are good for better training? I believe that in many ways, the message is incomplete if one leaves out expressivity and it would be nice to extend the theory to say something about the potency of the neural network on learning patterns and generalizing. The authors already comment on this in the 'future work' lines. I think that this should become current work, otherwise the work is incomplete, at least from the current perspective.\n- I have the feeling there are places where the analysis is imprecise, although it could be that I also misunderstood.\n - For one, the crux of the analysis is that the neural network nonlinearities are expressed in Fourier series (sec 3.2). Then, in the next section 3.3 the paper says that in practice nonlinearities are not polynomial and might not have a convergent Taylor expansion. So, instead a Chebyshev approximation is opted for. However, it is not clear if the Chebyshev approximation suffices or what are the limits of it? but I think this must be elaborated further.\n - Also, what are these Chebyshev approximations per nonlinearity? I think it is quite important to clarify this, considering there are nonlinearities that all but very similar, e.g., the ReLU and the leaky ReLU. What is the big difference between the two in terms of the described analysis?\n- I find it hard to understand often what the analysis tries to say, either the analysis is incomplete, the writing generally unclear or I simply don't understand some details. I list my comments by order of reading (not importance).\n - Throughout the paper there is a clear desire to connect rougness with layer depth. However, in all equations and analysis the depth is not explicitly present. For instance, in equations 4-6 there is only the degree of the polynomial K, but no layer variable or index. From what I gather, the (implicit) argument is that by the successive stacking of layers, the corresponding low/high order frequencies get stronger or weaker, relatively. Then, the objective is to compare the corresponding low and high frequencies for different layers, showing that for deeper layers the higher frequencies get stronger because of the recursion. This is how depth is 'qualitatively' introduced as a variable. Is this indeed the intention? If yes, I think it can be written more explicitly.\n- I find figure 1 a bit perplexing. Again, I understand what is the message, but it is hard for me to connect it to the theory, since the theory makes only indirect references to the specific nonlinearities. Also, what is 'ReLU->ResNet' supposed to stand for? ResNet is a ReLU when including skip connections? And what is 'Linear->ReLU'? To put it otherwise, adding skip connections or a ReLU nonlinearity are discrete design choices. However, the figure has continuous axis. So, what exactly is illustrated? The 'vertical' axis corresponds to the t variable in the Fourier coefficient. What about the other axis?\n - The related work points to Li et al and their spectral analysis to ground the proposed research. However, it is not explained what these observations are and how they relate to the current paper. It would be nice for the reader to add a short explanation.\n - Do we expect a difference by considering 1D slides, instead of 2D slides as motivated by Li et al? Why yes, why no?\n - It is not explained why are the mean path are empirically zero-functions. I infer that this is the case because at any location of the loss surface, if we take a small ball around it there will be an equal amount of parameters for which there is a higher or lower loss value? However, wouldn't this imply already a strong gradient (about 1, if I am not mistaken)?\n - What I find a bit confusing is that in equation 4 and 5 we apply the nonlinearity \\phi on p(t). However, p(t) are the 1-D slices of our neural network f defined in the preamble of section 3. I would assume that the nonlinearities would then already be inside p(t). In fact, in the preamble of 3 there is also a mention of \\phi and how p(t) is a polynomial when \\phi is the identity function. Maybe I have misunderstood something here. \n - There is an attempt to connect exploding gradients to blueshifting. However, this is not entirely clear to me. Indeed, one can say that simultaneously we have blueshifting in the gradient and at the same time exploding gradients. Does this mean that one cause the other, however? Couldn't one have explosions by having disproportionately large low order frequencies (not that it is the case, just wondering)? Or some other phenomenon.\n - There is a connection to exploding gradients, however, in deep networks vanishing gradients also important (maybe more so). Can the analysis address also vanishing gradients?\n - It is not exactly clear why the frequency dependent signal averaging is wekaner than exponetial downweights. The explanation is very brief and a bit vague (law of large numbers, exponential decay). Is there a more precise qualitative or quantitative argument here?\n Can one still call the method as more 'global' in saying something about roughness, given that all coefficients are computed per layer? Of course, each layer's coefficient are influenced by all previous layers, but is this enough to paint the method 'global'?\n - Is somehow K (polynomial order) connected to L (number of layers)? Or is this relation the way I described above?\n - Perhaps relevant to the previous point, and taking the position of the devil's advocate, in a way what is put forward by the paper is a re-interpretation of existing knowledge. While certainly very intriguing, is there a new insight on trainability for a new type of method/technique that can improve trainability? What about wide layers and them being easier/harder to train?\n - The text in p. 6 on Fig 5 (In figure 5, we use the power law ... ) is unclear to me. \n - In p. 7 there is a great misalignment between the figure references and the figure locations in the paper.\n - How is leaky ReLU connected to other nonlinearities? How precisely does it make a difference? How difference is the Chebyshev polynomial?\n - What does it mean 'making the networking more linear'? Do you mean increasing the \\alpha hyperparameter till it becomes 1, in which case you have a linear function?\n\nIn general, I find the paper quite interesting and with valuable potential contributions, but incomplete and not ready for publication at this stage. I believe it would worth it if the authors took the time to revisit the crispness of the message as well as the writing. Of course, I am more than happy to revisit my recommendation if the authors produce a convincing argument.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDQrTQJAIRf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The papers proposes an interesting analysis that links several aspects of architectural design in Deep NNs to the spectral analysis and observed roughness. Different activations functions are considered in the study, mainly centered on deep CNN with or without skip connections (in the framework of ResNet v1 and v2). The starting point, which is not novel, actually, but relevant, is that specific types of non-linearities introduce harmonic distortions, and the effect is potentially amplified when multiple non-linearities are stacked. Theoretically, the paper shows that there is a concrete link between architectural choices in the network design and the blueshift in the frequency domain. Experimentally, the observations support the mathematical analysis. All in all, some of the conclusions regarding trainability of CNN architectures with skip connections have been already noted and do not seem greatly new, but the paper introduces a nice perspective to see this phenomenon in another light. \nThe paper is generally well written and I appreciated reading it.\n\nThe major downside I see in the current form of the manuscript is given by some aspects of the presentation. For instance, Fig. 1 is clearly misplaced (it should be in Section 4.4). Similarly, Figure 6 should be in Section 5. Moreover, abbreviations would be better used in a more uniform manner (e.g., SDFA, FDSA, SDSA). Regarding the reported experiments: are the given plots achieved by averaging over multiple runs? (only in one of the many experimental settings this information is given in the paper). Finally, the link between the left and the right sides in Figure 6 is not really straightforward, perhaps grouping together the short and the noshort results could be of help for the reader.\n\n-- EDIT: \nThanks for the nice feedback during the rebuttal. I am happy to stay with my rating of clear acceptance.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "52APBAuFo49", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "***Summary***\n\nI would firstly like to thank the authors for an interesting read. I enjoyed going through the submission very much.\n\nThe authors propose to understand the qualitative effects of nonlinearities by studying the impact they have on the Fourier spectrum of deep neural networks. The central hypothesis is that nonlinearities with a lot of energy in their side lobes (high frequencies), lead to neural networks that have a rougher mapping and that are consequently tougher to train because the derivative landscape is also rougher. They back this hypothesis up with some mathematical arguments from the area of harmonic distortion analysis and with empirical experiments to support the qualitative predictions of this theory.\n\n***Pros***\n\nI found the submission very readable. I think the balance of text to mathematics in the main submission was about right, reserving the appendix for a more in depth discussion.\n\nI think that while the central finding that deep mappings are smoother is, in itself, not particularly novel, the chain of reasoning to get to this fact is new. I like the use of the Fourier spectrum to show this and the analysis behind how the spectra of various nonlinearities affect overall network smoothness..\n\nThe choice of experiments, which sequentially back up the claims, makes for a good paper. I particularly enjoyed the results in Figure 2, which were very instructive and gave good insight into the predictions of the theory.\n\n\n***Cons and constructive feedback***\n\nIn order from start to finish.\n\nIn the abstract should differential be differentiable?\n\nI think a good paper to cite would be “Avoiding Pathologies in Very Deep Networks” (Duvenaud et al., 2014) who analyze deep kernels in Gaussian processes. While the underlying models are different, the kinds of qualitative results in this paper are very similar to the submission.\n\nI am concerned about the use of the Fourier spectrum to model the ReLU nonlinearity. Will there not be issues with the Gibb’s phenomenon? The discontinuous gradient will mean that a spectrum exists, but reconstructions are poor.\n\nParagraph below equation 3: uniformely -> uniformly\n\nEquation 4: using t_j is confusing given that you use t in eqn 1. Please change to another symbol\n\nEqn 6: Please define the autocorrelation symbol in the main text.\n\nEqn 6: Please define z versus z_j\n\nSection 3.2 discussion: I would assume that while higher order autocorrelations would broaden the spectrum they would also smooth it out. For high orders it would like Gaussian-like in shape. This would not necessarily lead to blue-shifting.\n\nSection 3.3: therfore -> therefore\n\nSection 3.4 trivial -> trivially\n\nSection 3.5: Exponential downweighting. ResNets have combinatorially more medium length paths than short or long ones. So the average weight of a medium path is far higher than short or long ones. I would have liked to have seen a deeper analysis of this effect.\n\nExperiments: I found these very interesting. What is the motivation for only focussing on networks at initialization? I would have loved to have seen what a pertained network looks like.\n\nAre ensembles covered within the scope of this theory? They seem to have good performance but since each member is trained individually there is no smoothing of the training function, although the test loss function is smoother when all member models are combined.\n\n\n***Post rebuttal review***\n\nHaving read the rebuttal, I am very happy with the author responses. My main concerns about the Gibb's phenomenon and the choice to consider blueshifting at initialization have been thoroughly addressed. It is clear to me that the authors have thought long and hard about the rebuttal and used it to improve their submission. Therefore I maintain that this is still a clear accept.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "38RJoz66etD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper applies harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. This gives global view of the network output roughness.\n\nStrong points: The paper introduces an interesting measure \"roughness\" of deep neural network via harmonic distortion analysis. It evaluates the blueshift near the initialization for various nonlinearity (ReLU, LReLU, TanH, Sigmoid,…) and architecture choices (no skip, skip, depth, width). The blueshift fits people's intuition of the architectural choices.  \n\nWeak points: \n1. Current presentation mainly focus on explaining existing networks via blueshift measure. It does not \"predict\" new choices of the nonlinearity and architectures. This prediction could be related with the data: find the one nonlinearity and architectures that could best fit the data complexity.  \n2. The hypothesis that spectral blueshift impedes training fit the observations that architectural choices with  good harmonics generation control are easier to train to good performance. However this hypothesis is not consistent with the Sigmoid +NS case in Figure 16. I suppose it is hard to optimize Sigmoid+NS for a deep network (50 layers). \n3. The paper claims this can give a global view of the roughness. But the measurements are near the initialization. In this sense, it would be better to look at the roughness measure out of the initialization neighborhood.\n\nI would not recommend the acceptance for now because of the above weak points.\n\nAfter the rebuttal\n\nI thank the authors for the detailed feedback. I am a bit with AnonReviewer3 about the concerns on the descriptive languages of the paper \"less nonlinear\", \"more expressiveness\". Moreover, the behaviors of different nonlinearities in the Fig.3 and Fig.17 are related to the specific initialization and batch normalization, which is rather not a global view of the landscape. I would keep the score unchanged.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an analysis of the spectral impact of non-linearities in a neural network, using harmonic distortion analysis as a means to quantify the effect they have in the spectral domain, linking a blue-shift phenomenon to architectural choices. This is an interesting analysis, that could be strengthened by a more thorough exploration of how this analysis relates to other properties, such as generalization, as well as through the impact of the blueshift effect through the training process.
Decision: Accept (Poster)