Explaining Representation Bottlenecks of Convolutional Decoder Networks
Authors: Ling Tang
Wen Shen
Zhanpeng Zhou
YueFeng Chen
Quanshi Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Fourier transform
Deep Learning Theory
Representation Learning

Abstract: In this paper, we prove representation bottlenecks of a cascaded convolutional decoder network, considering the capacity of representing different frequency components of an input sample. We conduct the discrete Fourier transform on each channel of the feature map in an intermediate layer of the decoder network. Then, we introduce the rule of the forward propagation of such intermediate-layer spectrum maps, which is equivalent to the forward propagation of feature maps through a convolutional layer. Based on this, we find that each frequency component in the spectrum map is forward propagated independently with other frequency components. Furthermore, we prove two bottlenecks in representing feature spectrums. First, we prove that the convolution operation, the zero-padding operation, and a set of other settings all make a convolutional decoder network more likely to weaken high-frequency components. Second, we prove that the upsampling operation generates a feature spectrum, in which strong signals repetitively appears at certain frequencies. We will release all codes when this paper is accepted.

Under review as a conference paper at ICLR 2023
EXPLAINING REPRESENTATION BOTTLENECKS OF CONVOLUTIONAL DECODER NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, prove representation bottlenecks of cascaded convolutional de- coder network, considering the capacity of representing different frequency com- ponents of input sample. We conduct the discrete Fourier transform each channel of the feature map in an intermediate layer of the decoder network. Then, we introduce the rule of the forward propagation of such intermediate- spec- trum maps, which is equivalent to the forward propagation of feature maps through a convolutional layer. Based on this, we find that each frequency component in the spectrum map is forward propagated independently with other frequency compo- nents. Furthermore, we prove two bottlenecks in representing feature spectrums. First, we prove that the convolution operation, the zero-padde peroppadding operation, and a set of other settings all make a convolutional decoder network more likely to weaken high-frequency components. Second, we prove that the upsampling oper- ation generates a feature spectrum, in which strong signals repetitively appears at certain frequencies. We will release all codes when this paper is accepted.
INTRODUCTION
Deep neural networks (DNNs) have exhibited superior performance in many tasks. However, in recent years, many studies discovered some theoretical defects of DNNs, e.g., the vulnerability to adversarial attacks (Goodfellow et al.] 2014). and the difficulty of learning middle-complex inter- actions (Deng al. 2022 Besides, other studies explained typical phenomena during the training of DNNs, e.g., the double-descent phenomenon (Nakkiran et al. 2019), the information bottleneck hypothesis (Tishby & Zaslavsky 2015), and the lottery ticket hypothesis (Frankle Carbin 2018).
In comparison, in this study, we propose a new perspective to investigate how a cascaded convolu- tional decoder' network represents features at different frequencies. L.e., when we apply the discrete Fourier transform (DFT) to each channel of the feature map or the input sample, we try to prove which frequency components of each input channel is usually strengthened/weakkned by the net- work. To this end, previous studies (Xu et al., 2019a; Rahaman et al.) 2019) claimed that DNNs were less likely to encode high-frequency components. However, these studies focused on a specific frequency that took the landscape of the loss function all input samples the time domain. In comparison, we focus on a fully different type of frequency. i.e., the frequency w.z.. the DFT on an input image feature map.
Reformulating forward propagation in the frequency domain. As the basis for subsequent theoret- ical proof, discover that as a new forward propagation on the feature spectrum. We derive the rule that forward propagates spectrums of different channels through a cascaded convolutional network, which is mathematically equivalent to the forward propagation on feature maps through a cascaded convolutional network.
reformulate the traditional forward propagation of feature maps
Based the reformulation of the forward propagation, we prove the following conclusions.
(1) The layerwise forward propagation of each frequency component of the spectrum map is in- dependent with other frequency components. In the forward propagation process, each frequency component of the feature spectrum is forward propagated independently with other frequency com- ponents, if the convolutionn operation does not change the size of the feature map in each channel. In
Here, the decoder represents typical network, whose feature map size is non-decreasing during the for- ward propagation.

---

Under review as a conference paper at ICLR 2023
Epoch: 30
100
200
300
400
Map size: 4x4
Sxc8 16 32x33
64x64
(8)
(b)
Figure 1: Two representation bottlenecks of cascaded convolutional decoder network. (a) The convolution operation and the zero-padding operation make the decoder usually learn low- frequecuy components first and then gradually learn higher frequencies. (b) For cascaded upconvolutional layers, the upsampling operation in the decoder repeats strong frequency components of the input to generate spectrums of upper layers. We visualize the magnitude map of the feature spectrum, which is averaged over all channels. For clarity, we move low frequencies to the center of the spectrum map, and move high frequencies corners of the spectrum map. High frequency components in magnitude maps in (b) are also weakened by the convolution operation after upsampling.
this way, we analyze three classic operations, including the convolution, the zero-padding, and the upsampling operations, and prove two representation bottlenecks, as follows.
(2) Representation bottleneck We prove that both the convolution operation and the zero-padding operation make cascaded convolutional decoder network more likely to weaken the high-frequency components of the input sample, as shown in Figure aa), ) if the convolution operation with a padding operation does not change the size of the feature map in a channel.
Besides, we also prove that the following three conditions further strengthen the above representa- tion bottleneck, including (I) deep network architecture; (2) small convolutional kernel size; and (3) a large absolute value of the mean value of convolutional weights.
(3) Representation bottleneck 2. We porve that the upsampling operation makes cascaded convo- lutional decoder network generate feature spectrum, in which strong signals repetitively appears at certain frequencies, as shown in Figure Ib).
Note that all above findings can explain general trends of neural networks with convolution, zero- padding, and upsampling operations, instead of deriving the deterministic property of specific network. Besides, we have not derived the property max max-pooling operations, so this paper, is difficult to extend such findings to neural networks for image classification.
2 RULES OF PROPAGATING FEATURE SPECTRUMS
In this section, we aim to reformulate the forward propagation of a cascaded convolutional decoder! network in the frequency domain. To this end, first introduce the rule of convolutional layer propagating the feature spectrum from lower layer to an upper layer.
Convolution operation. Given a convolutional layer, let W(r==1), w(lm=)) W(fer:)) denote convolutional kernels of this layer, and let [[w==1], g ..... (lle== E R denote D bias terms. Each d-th kernel [(rr=d] E RC*K* K is of the kernel size K x K. and C denotes the channel number. Accordingly, we apply the kernel on feature F RRC.C RCXMXN with C channels, and obtain the output feature F E DXM xN' as follows.
F (Fn S.L. F (ilr
F Rer=d) Ip, d 1,2....,
(I)
where (()) M'''N' denotes the feature map of the d-th channel. & denotes the convolution operation. 1p E R is an all-ones vector.
Discrete Fourier transform. Given the e-th channel of the feature F C i.e., ple)
X we use the discrete Fourier transform (DFT) (Sundararajan) 2001) to compute the frequency spectrum of this channel, which is termed glo) E Ca follows. C denotes the algebra of complex numbers.
G(e) av
> M-1 N-1
m== n=
moc
0 M 1; -....N..
(2)
2

---

Under review as a conference paper at ICLR 2023
Input Be
Input spectrums G EexANA
Vu 4 i w T guw A (ax)
Output spectrums co.MKN
Ouiput feature RoxMEN
pusn g (000 A
Omn
(a)
prren C
DFT
Inverse
(b)
Convolutional operarions
Figure 2: (a) Forward propagation in the frequency domain and (b) forward propagation in the time domain. The convolution operation on an input feature F is essentially equivalent to matrix multiplication on spectrums of the feature.
Each frequency component at the frequency ju is represented as complex number, i.e. Gle E C. Let [G ...., G() E CCXMxN denote the tensor of frequency spectrums of the C channels of F. We take the C-dimensional vector at the frequency [u, u] of the tensor G, i.e, g(ue) = [G G ........ 1 E c to represent the frequency component (u, ej. Frequency com- ponents closed to [0, 00.0].00 N [] 1. 0). or [M N, 1] represent low-frequency signals, whereas frequency components closed to [M/2, N/2] represent high-frequency signals.
Reformulating the layerwise forward propagation process. For a specific convolutional layer (the stride size of the convolution operation is 1), the rule of propagating spectrums of input features into spectrums of output features is given as follows, which well represents the traditional forward propagation of features in Equation
Theorem 1, (proven in Appendix A.1.
Let H [H( H ...... HDD) E CDXM'XN' denote spec-
trums of the output feature R bxM xx Then, H can be computed as follows.
1, u' ='= 0=:00==' =0
('''o = .. + Σ
Σ
aute'u Thus) (ue)
g
s.. d
(3)
10, otherwise
Ou'v'ue 1 sin((M - K)A.. sin((N K) H(( KD) +NN- -K--
(4)
MN sin(Awe'T)
sin(?
where (1(v') ( H33) "........
HT E C.;; (b")) b ,..., b(D)) E T denotes the vector
of bias terms; Ou'e'ne E C is a coefficient; Aun- (u-u - MM-Ki Yov'
N(N- ((--))- T(w))
M(M-K-1
CD is matrix of complex numbers, which is exclusively determined by convolutional kernels I 2 Wi
Tr(w)) => L-O
W Ber e 30
cta
d 1, d
(5)
In Equation 3), the T(ue) term corresponds to the interference process (Beaver 2018) in physics, and the Ctufutue term corresponds to the diffraction process. According to Equation (4). for most "o],"'), [u, v). law'w'ue! close to O. |ou,''' is relatively large only when [u', v'] close to [u, v].
We notice that in most real implementations, the convolution operation does not change the size of the feature map in channel by applying the padding operation. Decoder networks usually use the upsampling operation to increase the dimension of features. Therefore, we limit our research to the scope of convolution operations without changing the size of the feature map channel. Thus, we propose the following assumption, which is used in all subsequential proofs.
Assumption 1. To simplify subsequential proofs, we assume that before each convolution operation, there exists circular padding operation (Londono) 1982), and set the stride size of the convolution operation to I. The circular padding operation is used to extend the last row and the last column of the feature map in each channel, so as to avoid the convolution changing the size of the feature map.
Thus, keeping the feature map size unchanged removes the diffraction process term of theory, and derives Theorem 2. In fact, because w'v is small in most cases, the diffraction process is actually ignorable, even when the convolution operation changes the size of the feature map.
Theorem 2. (proven in Appendix A.2). Based on Assumption spectrums the frequency domain can be simplified as follows.
X the layerwise dynamics of feature
(uv) (wv) (wv) 8 M Nb
(6)

---

Under review as a conference paper at ICLR 2023
Understanding the convolution operation in the frequency domain. As Figure shows, Theorem means that conducting the convolution operation on an input feature F is essentially equivalent to conducting matrix multiplication spectrums of F. For example, for all frequencies except for the fundamental frequency, we have the output spectrum h ((x)) = T(we) g g(m)).
(Conclusion 1) Each frequency component of the feature spectrum is propagated independently with other frequencies, h h(#*) = T(ee)g("") where T(ee) is exclusively determined by convolutional weights. Therefore, g(n)) is propagated independently with other frequency components g!n'(")
Reformulating the entire propagation process of cascaded convolutional network. To sim- plify the further proof, we temporarily investigate the spectrum propagation of a network with L. cascaded convolutional layers, but without activation functions. Nevertheless, we have conducted various experiments, and experimental results in Figure show that all our theorems can well reflect properties of an ordinary cascaded convolutional network with ReLU layers.
Let a convolutional network contain L cascaded convolutional layers. Each 1-th layer contains Ct convolutional kernels, ((]]kr=1), (y()) ..... w(] E _2--1xXxK, with Ct bias terms glt.)),04..) ...., ,((CCr) E R. Let E Roxx denote the input sample. The network generates the output sample F = net(r) E RCLXN Then, we derive the forward propagation of spectrums of to spectrums of F in the frequency domain as follows.
Corollary 1. (proven in Appendix A.3 Let G |G (G..,...,GGG G.C,)) CCoXMXN denote fre- quency spectrums of the Co channets of the input x. Then, based on Assumption 7
spectrums of the image 2 generated by L cascaded convolutional layers, i.e. H [H()) H H( C N is given as
h (uv) = T(w))(2:1) f(u)) + fmn13
(7)
where ((w))L:1) = T(L.w) (( E cCexCo; (wo) ((v)) [GW, Gun...., G. E C and
(nw)) = [H H H E ect. denote vectors at the frequency [u. v) in tensors and H, respectively. = (b Eiz T(n)(-)))-))) CCL; b(0) (b g0.2) ....., bu..TT R denotes the vector of bias terms of Ct convolutional kernels in the I-th layer.
Besides, the learning of parameters wilh affects the matrix T(lw,) Therefore, we further reformulate the change of (t.ue) during the learning process, follows.
Corollary 2. (proven in Appendix A.4) Based Assumption 4 the change of each frequency components T(..u) during the learning process is reformulated as follows.
BTT))) MMN Σ
M-IN-1
Σ Xureuue ( (T
1.1) 8
o' + B
Øosss
- ('
(8)
ai
sin(K (( / sin(K(e v' /T/N/N)
(N 11 (K- (K-110-00))
S.L. Xu'v'ue MN sin((u (u--u'u'j//pp /) sin((v - V')
where n is the learning rate; Xu'v'ee E 70-1,a/')) (aa'"")(()aaa'") E CC--100o; T('4)))a = T(lndd) ...Tt, E COLXCI; 8' = LMM((b--)) E (10000111)6--1 E CC--1 ((u0)(1111) is the conjugate of (())
a coefficient; Tho 11-
Verifying the forward propagation in Corollary and the change of Th(lao) in Corollary 2
We computed the similarity between real spectrums H' = [ H (2) HH((),.... ....] measured by applying the DFT to the real network output, and spectrums H [H()) ((,,,((() H(.)....
derived in Corollary | in order to verify the correctness of the forward propagation in the frequency domain. Specifically, we measured the cosine similarity similariyyH"' H) E. (vo((ve(noorm( H (+)),, vec(norm (H)))))) where vec(-) represents the vectorization of matrix, and norm(-) represents computing the norm of each complex number in a matrix.
To this end, we constructed the following three baseline networks to verify whether Corollary [ derived from specific assumptions could also objectively reflect real forward propagations in real neural networks. Specifically, the first baseline network contained 10 convolutional layers. Each convolutional layer applied zero- paddings and was followed by an ReLU layer. Each convolutional layer contained 16 convolutional kernels (kernel size was 3 x 3) with 16 bias terms. We set the stride size of the convolution operation to 1. The second baseline network was constructed by removing all ReLU layers from the first baseline network, which was closer to the assumption in Corollary 1
4

---

Under review as a conference paper at ICLR 2023
CIEAR--O datoeet
Tiny Nee.eeeteeeeaaacooooooomoooo dataset
Uroden dataect
CHEAR- 10 cocaset
Limy www.-lmmgeeeee dataset
Broden cocasrt
6.55
5:46
(b)
3.44
busilind.
1.22
0.99
0.02
0.37
3.00
1.00
Network layer number
Netwek apat Fumber
Natwork lxper Furborr
Netwok byyer
AatAar aget number
layer number
Figure 3: (a) Fitness between the derived feature spectrums H in Corollary 1 and the real feature spectrums H. measured in real DNN. (b) Fitness between the derived change of ((i,e)) in Corol- lary and the real T(lm)) measured in real DNN. The shaded area represents the standard deviation.
The third baseline network was constructed by replacing all aero-paddings with circular paddings from the second baseline network, which was exactly the same with the assumption in Corollary []
FigureBar a) reports similarity(H" H) that was measured on spectrums in different layers and averaged over all samples. The similarity between real spectrums and derived spectrums was large for all the three baseline networks, which verified Corollary I Note that the cosine similarity was computed based on high-dimensional vectors with as many as 32 or 64 or 224² dimensions (determined by the dataset), in which case tiny noises were accumulated significantly. Therefore, the similarity greater than 0.8 was already significant enough to verify the practicality of our theory.
Besides, we also measured the similarity between the real change of T(,.))) computed by mea- suring real network parameters, termed A ATT(ie))) and the change of T( derived with certain assumptions in Corollary 2 i.e. AT((,,))) in order to verify Corollary 2 The similarity was also computed as similarity( A (T(..we), AT((.)) = E. (e *T(L.er) D), vec(norm( (ATT()))))))))
The verification was also conducted the above three baseline networks. Figure B]b reports vl, similarity( A "T((,,)), Ar(..)) averaged over all samples. The similarity was greater than 0.88 for all the three baseline networks, which verified Corollary 2
3 REPRESENTATION BOTTLENECKS
We further analyze the effects of three classic operations on representing different frequency com- ponents of an input sample, including the convolution operation, the zero-padding operation, and the upsampling operation, and discover two representation bottlenecks.
Effects of the convolution operation. Given an initialized, cascaded, convolutional decoder network with L convolutional layers, let us focus on the behavior of the decoder network in early epochs of training. We notice that each element in the matrix T((.we) is exclusively determined by the e-th channel of the d-th kenel w L...K-1.0.K E R k according to Equation (5). Because parameters (' in the decoder network are set to random noises, we can consider that all elements in T(C,u)) irrelevant to each other, ie.,, Yd * d',c c # d', TT((.w))) is irrelevant to T((?"") Similarly, since different layers' parameters W/( are irrelevant to each other in the initialized decoder network, we can consider that elements in different layers' Tr(.w0) irrelevant to each other, i.e., AI & I', elements in (((lue)) and elements in (T(', are irrelevant to each other. Moreover, since the early training of a DNN mainly modifies a few parameters according to the lottery ticket hypothesis (Frankle & Carbin 2018), we can still assume such irrelevant relationships in early epochs, as follows.
Assumption 2. (proven in Appendix A.5 We assume that all elements in T(l,uv) are irrelevant to each other, and vI + I, elements in T (hav) and T' pav) are irrelevant to each other in early epochs. Vd 4 d'; + d',
E E (Tlliw)) d' (
= E.pt, (Til.ae)
EE [Taee
(10)
Vl,,c,,',e Eyyc [T (pec) Ewer [I (Bll [[ w).....w.)) wit) [Tee
(f(o)(1-1))) (11)
Besides, according experimental experience. the mean value of all parameters in w(t usually has a small bias during the training process, instead of being exactly zero. Therefore, let us assume that in early epochs, each parameter in w( is sampled from a Gaussian distribution N(p, of).
According to (wv) ((e)) ((ue)) + Nbb in Corollary each frequency component (ue) of the output spectrum is exclusively determined by the component g(we) of the input sample and the matrix ((=0)(:1) (L..uv ... w............ (2.v) since 8.. = 0 on all frequencies other than the fun- damental frequency. Therefore, the magnitude of ((s))(ta) reflects the strength of the network encoding this specific frequency component ((se))
5

---

Under review as a conference paper at ICLR 2023
Based on Assumption Fand Assumption Thu follows Gaussian distribution of complex numbers, as follows.
Theorem 3. (proven in Appendix A.5)
2
can prove that
dd, c T(haw)) -Complex = -u,Ru, = K2 oK"o,r r- of Raw,2.)
(12)
s.f. Ru (unuKT/MM) sin(uK=/N) ed( 11-111 [K- [K-1)=)
(13)
sin(ur/M) sin(uT/N)
Different from the Gaussian distribution of real numbers, the Gaussian distribution of complex numbers has three parameters f C, E and C, which control the mean value, the variance, and the diversity of the phase of the sampled complex number, respectively. Specifically, a large value of |y| indicates that the sampled complex number T(llw)) is less likely to have diverse phases. Ruv C is a complex coefficient, S |Ravl S K².
For low-frequency component [u v Ruulo is relatively large. Therefore, the second-order moment of TT0... ("")) i.e. 1pg Raglor + K2 Rogee is large, which indicates that the sampled (i")
de
de ((t) is more likely to have large norm. Besides, the parameter |r| = |of Ra 20 is large Till is less likely to have diverse phases. In contrast, for a high- frequncyyy component [u oliih), the sampled Tek: glith is less likely to have a large norm and is more likely to have diverse phases.
for low frequencies, which means that the sampled de
Theorem 4. (proven in Appendix A. For the simplest case that each convolutional layer only contains a feature map with a single channel, iee. VI, Ct =. Then, based on Theorem 3 and follows distribution, which is the product of I, complex numbers, where each complex number follows a Gaussian distribution. The mean value of T(o)LA:) is II piRuv E C. The logarithm of the second-order moment is given as log (T(Ww()))) Et. log([R)Ruaol + TR.
Assumption 2 T(m)((t))) -T(..wo) ...pp....))
For the more general case that each convolutional kernel contains more than one channel, i.e., VI, G > 1, the (OM(T(W)LL)) also approximately exponentially increases along with the depth of the network with a quite complicated analytic solution. Please see Appendix 6 the proof.
(Conclusion 2) Therefore, according to above proof, the convolution operation makes a cascaded convolutional decoder network more likely weaken the high-frequency components of the input sample, if the convolution operation does not change the feature map size. Specifically, we obtain the following five remarks specify detailed mechanisms of weakening tigh-freeuency components.
Remark According to Theorem 4 for each frequency component [u, v], the second-order moment (OM(T( will exponentially increase along with the depth 7. of the network. We can consider that each layer' 7((,.0)) has independent effects log(Iı Rae! 12 + K2 K²a on log SOM((T EL. log(1n R k²a)).
We admit that the conclusion in Remark is derived from the second-order moment of (L:) instead of deterministic claim for a specific neural network. Nevertheless, according to the Law of Large Numbers, ((w)(1))) is still convincing metric to reflect the significance of ((w))
Remark 2. If the decoder network is deep, then the decoder network is less likely to learn high- frequency components. It is because RReel is relatively large for low-frequency components. In this way, the large effect a single layer's ((ftu))) of wwwfrequency components on log SOM(T((o))La)) o²oo), can be accumulated through different layers according to the Law of
i.e., log( 1 + Large Numbers and the independence between different layers in Remark I
Therefore, the large |Rat,,in value for low-frequency component low ko0 makes The (mttat) more likely to have a large norm, whereas the small ,ta value for high-frequency component (e wiai makes ((u:"a, less likely to have large norm. This indicates that deep decoder network will almost certainly strengthen the encoding of low-frequency components of the input sample, while weaken the encoding of high-frequency components.
Remark 3. If the expectation μr of convolutional weights in each i-th layer has large absolute value |ar), then the decoder network is less likely to learn high-frequency components. is because according Theorem 4 large absolute value |aa| boosts the imbalance effects |arR....1 among different frequency components, thereby strengthening the trend of encoding www-ureue com- ponents of the input sample.

---

Under review as a conference paper at ICLR 2023
Remark 4. If the convolutional kernel size K is small, then the decoder network is less likely to learn high-frequency components, It is because according to Theorem 4 a large K value alleviates imbalance of the second-order moment SOM(T(w>)(())) between low frequencies and high frequen- cies caused by the imbalance of Ruol- Thus, a small K value strengthens the trend of encoding low-frequency components of the input sample.
Remark 5. If the cascaded convolutional decoder network is trained on natural images, then the decoder network is less likely to learn high-frequenyy components. Previous studies (Ruderman 1994] have empirically found that natural images were dominated by low-frequency components. Specifically. frequency spectrums of natural images follow a Power-law distribution. I.e., low- frequency components (e.g., the frequency component [u, closed to [0,0), [0, N 1), [M 1, 1,)
and [M N,N 1)) have much larger length ll ||g 2| VV. |G(s) than other frequency com- ponents. Besides, according to rules of the forward propagation in Equation (7) and the change of T(.a) in Equation (8), if the frequency component gl of the input image has a large magnitude, then h (wr) of the output image also has a large magnitude. This means that using natural images as the input strengthens the trend of encoding low- wwwerruuency components.
These five remarks tell us different ways to strengthen or weaken the capacity of a decoder of modeling specific frequency components. Experiments in Section have verified Remarks T in the general case that each convolutional layer contains more than one feature maps.
Effects of the zero-padding operation. To simplify the proof, let us consider the following one- size zero-pate perroppdding. Given each e-th channel rr(o) E MMN of the feature map, the zero pero-padding puts zero values at the edge of ple), so as to obtain a new feature Fite) E gM'"N"
xx'" as follows.
0 < M, M, OSn<N
Vm,
Fany
(14)
0,
MSm M', N En
We have proven that the pero-padding operation boosts magnitudes of low-frequency components of feature spectrums of the feature map, as shown Theorem
Theorem 5. (proven in Appendix A.7p Let each element in each e-th channel plo) of the feature map follows the Gaussian distribution o²), glo E MMx denotes the frequency spectrum of ple), and H()) C
xN' denotes the frequency spectrum of the output feature F(ee) after applying zero-padding on Fle), Then, the zero-padding on Flo) brings in additional signals at each frequency [u, as follows, whose strength is measured by averaging over different sampled features.
G 11 [all0|aa| (al(
siin( (imnMurz) /Mr Not/N' sin(ut/M) UT/N
0A S u < M. 0 020 < N, Epco [[((])
-M
(15)
VM SMSU< S N CUSCNN Epc 11 sin sin(Mun/MM) Nuw/ ((M-11" K? sin(ut/M') /
e
(Conclusion 3) According to rules of the forward propagation in Equation and the change of T(,,,))) in Equation (8. the zero-padding operation strengthens the trend of encoding low-frequency components of the input sample, because Epte H G( [] is large for low frequencies [u, v].
Effects of the upsampling operation. Let the /-th intermediate-layer feature map F E RR Mox pass through an upsampling layer to extend its width and height to A xN, subject to M Mo-ratio, N No ratio as follows.
ple).
m m /ratio mod(m"" raaii) O; mod(ni" ratio)) s.t. otherwise
Ve, mn" .n.
Flo) mant
(16)
0.
n n= ni n
Theorem 6. (proven in Appendix A.8) Let G ( (G.))..., trums of the Ct channels of feature F. Then, spectrums H = (1 H(²) ..... H(i e CXXXMN of the oulput feature F can be computed as follows.
CoiMooxoo denote spec-
Ve,u,u v.
H(e)
-G(e) s.t. = 1,,..,,, M/Mo: t= N/No
(17)
Theorem 6 shows that the upsampling operation repeats the strong magnitude of the fundamental frequency G of the lower layer to different frequency components Ve, H( of the higher layer, where u" 0, Mo, 23. 0, No, 2No, ... Such phenomenon is shown in Appendix C.2
7

---

Under review as a conference paper at ICLR 2023
CIFAR-10 10 dataset
CIFAR-10 datasel
Baa--lnageNet dataset
Broden dataset
10.0
12.5
15
15
15
(a)
10
5.0
7.5
randomly selected
(b)
0.7
OPAR margeitt
0.0
Freguemenies
20
40
20
40
20 40
20
40
Network byeer number
Network: byyer number
Network layer number
Network boyer number
Figure 4: (a) The exponential increase of the second-order moment of feature spectrums, (h (the linear increase of logSOM(h (*)) along with the layer number linearly). (b) A small kernel size K usually made the network learn a higher proportion n of low-frequency components.
(Conclusion 4) The upsampling operation makes the upconvolution operation generate a feature spectrum, in which strong signals of the input repetitively appears at certain frequencies. Such unexpected strong signals hurt the representation capacity of the network.
More crucially, according the spectrum propagation in Corollary | such unexpected frequency components can be further propagated to upper layers. Thus, Corollary [ may provide some clues to differentiate real samples and the generated samples.
4 EXPERIMENTS
Verifying that a neural network usually learned low-frequent components first. Our theorems prove that a cascaded convolutional decoder network weakens the encoding of high-frequency com- ponents. In this experiment, we visualized spectrums of the image generated by decoder network, which showed that the decoder usually learned low- oow-freuenccy components in early epochs and then shifted its attention to gigh-frequency components. To this end, we constructed cascaded convolu- tional auto-en by using the VGG-16 (Simonyan & Zisserman, 2015) as the encoder network. The decoder network contained four upconvolutional layers. Each convolutiona/lpccooolltiii layer in the auto-encoder applied zero-padding and was followed by a batch normalization layer and an ReLU layer. The auto-encoder was trained on the Tiny-ImageNet dataset (Le & Yang 2015) using the mean squared error (MSE) loss for image reconstruction. Our theorem was verified by the well-known phenomenon in Figure Ma), i.e., an auto-encoder usually first generated images with wwwfrreuency components, and then gradually generated more high-frequency components. In addition, Appendix C.I shows results on more datasets, which also yielded similar conclusions.
Verifying that the upsampling operation made decoder network repeat strong signals at certain frequencies of the generated image in Theorem 6 To this end, we compared feature spectrums between the input spectrum and the output spectrum of the upsampling layer. We also conducted experiments on the auto-encoder introduced above. Figure [bb) shows that the decoder network repeated strong signals at certain frequencies of the generated image. In addition, Ap- pendix C.2 shows results on more datasets, which also yielded similar conclusions.
Verifying that the zero- prro-padding operation strengthened the encoding of low-frequency com- ponents. To this end, we compared feature spectrums between the network with zero zero-padding op- erations and the network without zero-padding operations. Therefore, we constructed the following two baseline networks. The first baseline network contained 5 convolutional layers, and each layer applied cero-paddings. Each convolutional layer contained 16 convolutional kernels (kernel size was 7)7), except for the last layer containing 3 convolutional kernels. The second baseline network was constructed by replacing all zero-padding operations with circular padding operations. Results on the Broden dataset in Figure 5ic) show that the network with zero prrdd-ddding operations encoded more significant fveeueeyynyyy components than the network with cireular padding operations. In addition, Appendix C.3 shows results on more datasets, which also yielded similar conclusions.
Verifying factors that strengthened low-frequency components. (/) Verifying that a deep net- work strengthened low-freuuenc components in Remark / and Remark 2 To this end, we con- structed a network with 50 convolutional layers. Each convolutional layer applied zero sero-paddings to avoid changing the size of feature maps, and was followed by an ReLU layer. We conducted this experiment on three datasets, including CIFAR-10 (Krizhevsky et al. 2009). Tiny- ImageNet, and Broden (Bau et al.) 2017) datasets, respectively. The exponential increase of ((u))(L.)) along with the network depth L in Remark indicates that frequency component h (wr) of the network output also increases exponentially along with T.. Therefore, for the frequency component ((m) generated
8

---

Under review as a conference paper at ICLR 2023
Input
Layer
Layer Layer 10
Input
0 y 0.001 0.01
Input
circular
(a)
Figure 5: (a) A higher layer of network usually generated features with more oow-rrquenccy com- ponents, but with less high-frequency components. (b) A network whose convolutional weights have a mean value significantly biased from 0 usually strengthened low-frequene wwwfrequency components, but weak- ened high-frequency components. (c) A network with zero- proo-paddin operations usually strengthened more low-frequency components than a network with circular padding operations. Here, each mag- nitude map of the feature spectrum was averaged over all channels. For clarity, we moved low frequencies to the center of the spectrum map, and moved high frequencies to corners the spec- trum map. Besides, we only visualized components in the center of the spectrum map with the range of relatively low frequencies E {u|o S < M/8) U [u/7M/8 s < M3; E {u/0 s v N/8) U [v]7N/8 < N] for clarity.
by each 1-th layer in a real decoder network, we measured its second-order moment SOM(h Figure 4 shows that (O)() increased along with the layer number in an exponential manner.
Besides, we visualized feature spectrums of different convolutional layers, which verified the claim in Remark 2 that deep decoder network strengthens the encoding of low-frequency components of the input sample. Results on the Broden dataset in Figure 55a) show that magnitudes of low- frequency components increased along with the network layer number. In addition, Appendix C.4 shows results on more datasets, which also yielded similar conclusions.
(2) Verifying that larger absolute mean value ir of each 1-th layer's parameters strengthened low-frequency components in Remark 3 To this end, we compared feature spectrums of the same network architecture with different mean values μu of parameters. Therefore, we applied the network architecture used in the verification of the effects of the pero-padding. but we changed the kernel size to 9x9. Based on this architecture, we constructed three networks, whose parameters were sampled from Gaussian distributions. ([r 0, x² 0.01²), N 0.001. o² = 0.012), and W(p 0.01, o² 0.01²), respectively. Results on the Broden dataset Figure ()) show that magnitudes of low- frequency components increased along with the absolute mean value of parameters. In addition, Appendix C.5 shows results on more datasets, which also yielded similar conclusions.
(3) Verifying that small kernel size K strengthened low-frequency components in Remark 4 To this end, we compared feature spectrums of networks with different kernel sizes. Therefore, we constructed three networks with kernel sizes of 1 1x 3x3, and 555. Each network contained 5 con- volutional layers, each layer contained 16 convolutional kernels, except for the last layer containing 3 kernels. We used the metric d Eju.-Jesmm (1) / Σ.. E.[ E.[ [2] to measure the ratio of low-frequency components to all frequencies, where n0e = 00 < < M/8,0 v < N/8 M/8, 7N/8 I v < N] [7M/8 S < M. O S v < N/8] U [7M/8 S M, 7N/8 s << NJ. Fig- ure 4 b) show that the network with a small kernel size encoded more low-frequency components.
CONCLUSION
In this paper, we have reformulate the rule for the forward propagation of a cascaded convolutional decoder network in the frequency domain. Based on such propagation rules, we have discovered and theoretically proven that both the convolution operation and the rero-padding operation strengthen low-frequency components in the decoder. Besides, the upsampling operation repeats the strong magnitude of the fundamental frequency in the input feature to different frequencies of the spectrum of the output feature map. Such properties may hurt the representation capacity of convolutional decoder network. Experiments have verified our theoretical proofs. Note that our findings can explain general trends of networks with above three operations, but cannot derive deterministic property of a specific network, and cannot be extended to networks for image classification, because have not derived the property of the max-pooling operation, which will be derived in the future.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT STATEMENT
As fundamental research in machine learning, this paper does not introduce any new ethical or societal concerns. The results in this paper do not include misleading claims; their correctness is theoretically verified. Related work accurately represented. Though in theory any technique can be misused, it is not likely to happen at the current stage.
REPRODUCIBILITY STATEMENT
This research discovered and theoretically explained two bottlenecks of cascaded convolutional decoder network in representing feature spectrums. For our theoretical results, formal statements and the complete proofs of all Theorems, Corollaries, and the Assumption in Section and Section 3 are provided in Appendix A. We have discribed additional experimental details in Appendix C, including various model architectures and benchmark datasets, which ensure the reproducibility. We will release all the codes and datasets when this paper accepted.
REFERENCES
Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based clas- sification using the information bottleneck principle. IEEE transactions on paltern analysis and machine intelligence, 42(9):2225-2239) 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 3319-3327, 2017.
John Beaver. The Physics and Art of Photography, Volume I: Geometry and the Nature of Light.
Morgan & Claypool Publishers, 2018.
Huiqi Deng, Qihan Ren, Xu Chen, Hao Zhang, Jie Ren, and Quanshi Zhang. Discovering and explaining the representation bottleneck of dnns. In International Conference on Learning Rep- resentations, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 18X/vv:803.00363,, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1/265:1412.6572, 2014.
Ken Kreutz-Delgado. The complex gradient operator and the cr-calculus. arXiv preprint
4XXY:00906.4835, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
Xiang Li. Shuo Chen, and Jian Yang. Understanding the disharmony between weight normalization family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Yuping Lin, Hui Jiang, et al. Bandlimiting neural networks against adversarial attacks. arXiv preprint arxiv: 12797, 2019.
Luis Eduardo Londono. Real time digital signal processing. 1982.
Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang. Theory of the frequency principle for general deep neural networks. arXiv preprint arXiv: 1906.0923, 2019.
Chao Ma, Lei Wu, et al. Machine learning from a continuous viewpoint, i. Science China Mathe- matics, (1111):223-22666 2020.
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2023
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- on Machine Learning, pp. 5301-5310. PMLR, 2019.
Heckel Reinhard and Furkan Yilmaz Fatih. Early stopping in deep networks: Double descent and how to eliminate it. In International Conference on Learning Representations, 2020.
Daniel L Ruderman. The statistics of natural images. Network: computation in neural systems, (4):517, 1994.
Ravid Shwartz-Ziv -iv and Naftali Tishby, Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 1703.00000.00 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Duraisamy Sundararajan. World Scientific, 2001.
The discrete Fourier transform: theory, algorithms and applications.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruma, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6699 2013.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
David Tse and Pramod Viswanath. Fundamentals of wireless communication. Cambridge university press, 2005.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350. 2017.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequee component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684-8694, 2020.
Natalie Wolchover and Lucy Reading. New theory cracks open the black box of deep learning. Quanta Magazine, 3, 2017.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: 06523, 2019a.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference Neural Information Processing, pp. 264-274. Springer, 2019b.
Dong Yin, Raphael Gontijo Lopes. Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances Neural Information Processing Systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
A PROOFS OF OUR THEORETICAL FINDINGS
We first introduce important equation, which is widely used in the following proofs.
Lemma 1. Given complex numbers, eine = 0, 1, 1,1,...,N' 1, the sum of these N complex numbers given follows.
sin No sin
4A IR,
Σ
ino
NN-10
Specifically, when NO = 2k A: E Z,
-N < k < N, we have
N sin(
No
AB E R
ing
2
(8-138 2
NôØ;
s.l. No = 2kT, k E Z, - < <,,,
a=0 sin(
2
(2)
1,
0
0
where
89
0,
otherwise
We prove Lemma
I as follows.
Proof. First, let us use the letter C to denote the term of NN-1
ein
N
in8
e
n=00
Therefore, S is formulated as follows.
N
10
e ini
E
C
n=1
Then, S can be computed S g e -1
Therefore, we have
s S
S
eill
2n
eino
N
enn
6:0
6
1
No 2
(1-11
e
No 1/2i
(N-11)
(et4
e -18)
1/22
sin( No
13-
sin @
7
Therefore, we prove that N-1 sind
sin( N2
1N-1)0
sin(?]
Then, prove the special case that when No 2km, k E Z, - <k EN-1 eind = =N66
[N, 0=
otherwise" follows.
0,
12

---

Under review as a conference paper at ICLR 2023
When 8 = O, we have
lim
N-1 1
ino
lim 0->0 sin( (n))
sin
CN 13e
0-+0
7=0
sin( 20)
lim
0 sin
=
When 0 + 0, and NO 2kT,, h E, -N < < N, have
sin( No
Σ n=0
ine
IN
11
sin
sin(kT)
sin(
kx
N
In the following proofs, the following two equations are widely used, which are derived based on Lemma
M N Σ m=0 n=0
M -
N (27)
22
e in(-
m=0
n=0 MN,
(M6
)N6 "3 var
//According to Equation
u ====
0,
otherwise
To simplify the representation, let owe be the simplification 6 wax ò_ =2x in the following proofs. Therefore, we have
Σ
MN8
MNN, 0,
==== 0 otherwise
n=0
Similarly, we derive the second equation as follows.
am
M
imm
(u-u (o-00)))))
N-1 in(
MN8
w'2e 8
//According to Equation 2
(4)
MN8
MN,, v'=vv'== =
0, otherwise
A. PROOF OF THEOREM 1
In this section, we prove Theorem | in Section of the main paper.
Theorem 1. Let H (H() H ......!(")) [T E CDXM'XN' denote spectrums of the output feature P PER PERDXM' DXM' Then, H can be computed as follows.
--1N-1
1, n' 0, =
10 (1/)) MNb
au'o'u T (wo)
d
0, otherwise
4=0 M=0
13

---

Under review as a conference paper at ICLR 2023
sin((M K))uT) sin( K ()w( ((M
Oudoouw MN
sin sin(Xuw))
sin(Tve: TT)
where h(w'e' [H H
atuls W HHI CD; (60)) ((0) BIDiJT E R. denotes the vector
of bias terms: aw E C is a coefficient; Aus' = M(M-K-I) (u-w'MMo(KK] Too' (o-d')N-o(-- Tiw)
NIN-R+1)
CDxC is matrix of complex numbers, which is exclusively determined by convolutional kernels W Whtr= Wh
T
E Exe 1nD
W Wl *
d d= D; c= 1.2....,C.
Proof. Given each c-th channel of the feature spectrum G( the corresponding feature F(e) in the time domain can be computed as follows.
M-IN-1 ΣΣ 1=0 0=0
Flo)
MN
Then, let us conduct the convlution operation (in Equation ( in the main paper) on feature [F(1) F(2) ..... F(O)], in order to obtain the output feature E R Dx''
AA = 1,2, D;0.. m < M1; M
K-1K-1 1-1
F(a) b(4) Σ Σ
Wkknmdd ple)
cm1 [-u
K-1 K-1
M-IN-1
Σ
Wker
ΣΣ
Gle)
ulm 12 vin vialal )2
MN
bh
Σ
Σ
(
+ )2 Σ
Σ
Wher=d JC 22
e
MN
M-IN-1
= <
Σ
Thu
(c)
MN
4E0 V=D
Then, let us conduct the DFT on each channel of F, in order to obtain feature spectrums (())) of F.
dd 1,2....
D; 05
M 050 N';
M'
(0.d)) mn
-
+ )2
u'v''
m=0
M' M'-1 N'
M-1N-1
Σ
Σ
)25 (b)
Σ Tuue) Tuue + 12 / /Equation
MN
m=0
v=0
M-IN-1
M M'-IN'-
M (' 8
Σ ΣΣD
Σ
Thuel Gle)
MN Σ
Σ
< - ) - m022
cml
ma= NmU
M' W'-IN-1
1/ Let Ou
Σ
de mm+ 3 m322
MN
=
Σ
Σ
T (u)) (cc)
((c)
Ou'e'ue
14

---

Under review as a conference paper at ICLR 2023
When the convlution operation does not apply paddings, and its stride size is 1, M' M K 1, N K K+L In this way, Ou'v'uv can be rewritten follows.
M
N'
Σ m==
Al
Ne
m & & )n)2
MN
n=0
1/M'
M
+
+1,N== = K1
N-K
Σ
Σ
Jm x-x 1n)2
MN
(5)
N-K K
MT
12
"l N x---7 12
MN
m=
n=
// According Equation
sin((M K) ())
sin((N - K Tow! ((MM-K)An -NN-KMuuth Krou))
MN
sin ()
sin (Tve!
where Aun'
(a 14--- a JM [(K-
7om! (o-K)N-((KK1)
MMM K+1)
N(N-K+1)
Therefore, we prove that the vector h (w'e'') follows.
[H w/rr 11
w.....
E CP be computed as
M-1N-1
=d= = 1,2... D; h (w'd') ww M'
Σ
Σ
au'o 7 g(w)
A.2 PROOF OF THEOREM 2
In this section, we prove Theorem in Section of the main paper.
Theorem 2. Based on Assumption the layerwise dynamics of feature spectrums in the frequency domain can be simplified as follows.
h (uv) ( g fuv) + Nb
(6)
Proof. Based Assumption 1 the convolution operation does not change the size of the feature map, i.e. M' = M. N' = N. In this case, Ou'e'uv can be computed as follows.
M' 1 N'
e c
mm ( * mm]n
Olyy Cufee
MN
m==
n=0
M-
N-1
1
(w-w') 11
[-- 22
1/M' M. N
MN
==0
M
N-1
22
cil (000
//According to Equation
4
MN
du-u 8'pppp' 1, u' = u
1, 11 =
where 5
0. otherwise" 8,_
0. otherwise
Therefore, (a'v') can be computed as follows.
M' N'-1
(u'o')
awe T(u'd') (u''' + M'N
(ll'')g((u'o')
a=0
==0
M-1N-1
Σ
8 8 T. ol + Eer MNb
= (p''v')
0
(n'v')
+ M NN
15

---

Under review as a conference paper at ICLR 2023
Then, prove that h (wv) ((un)l()) + MMNbu..
A.3 PROOF OF COROLLARY
In this section, we prove Corollary I Section of of the main paper.
..... !C..) C CoxMM denote frequency spectrums of the Co channels of Then, based on Assumption 7 spectrums of the generated image E, i.e., = |H H H(..), E CMMNN can be computed as follows.
Corollary 1. Let G =G d(
(8) where T(u))(L:1) = T(,,uv)... ... ((2.)))((..)))) E CCLXCo; gluv) = [G Guv,.... Gue E C and ((v) = [H H respectively. 3= (b + Σ T(00)(:0)) denotes the vector of bias terms of Ci convolutional kernels in the 1-th layer.
h (wv) = =((w)((::1)
8 ((w)) durB3
HH(C (C
E COL denote vectory at the frequency [u, v] in tensors G and H.
CC bt = [b4.1), b(t.) 3.... BU.CT) R
Proof. Let G [G((.1), ((l.2),... ... G(,CC)) E CCIXMXN denote feature spectrums of the 1-th layer. Let g(l.uo) = [G (lL2) GlO E C denote the frequency component at the frequency [u, v]. When I 0, g (0,we) denotes the frequency component of the input sample. When 1 I., g (L)) denotes the frequency component of the network output.
Based Theorem g(l.uw) can be computed follows.
11 1,2, L, g g(l.av) (1,w0) g (1-1,.)) + Nb
Then, the frequency component g g(L,av) of the network output can be computed as follows. g T(..wv) (T (1 (L-2,wv) + Sy BN. (1N((L111) + dN (L) glt (1- p(lno) (t
g (L,ux)
-(( ( + S...)N(
2,00) + (L,)))) 8 MNb(L-)) + 8 MN luo)
-T(l.w) ((..)) (0,00) MNT((..)) ....")............ T.B.u (1) +...+ MND (8)
+ MN ((. 11(3,00) ( (1) +... MMD
Let (L.)( = ". (1..e) and B = (N (1) + E/ 1 ()- Let (ue) = g(l,,r)) denote the frequency component of the network output, and let g (uv) = g (0 denote the frequency component the input sample. Then, we prove that (wv) can be computed as follows.
(un) = (T:))( (wv) + S
A.4 PROOF OF COROLLARY
In this section, we prove Corollary in Section 2 of the main paper.
Corollary 2. Based Assumption 7 the change of each frequency components T(.w) during the learning process reformulated follows.
A7 MMN
M-1 N-1
Σ
Xute'us (T (1 1:1) (od (
+ bwt
Masss
T (1
a a(h
=0=0 v'=0
sin( (K(u sin(K(u
af / sin - V'/ (N- (x- 1o0-00
3.1. Xu'v'ue MN sin((-u() w')x/M) sin((v- (n((u-v')x/N)
(10)
where " the learning rate; Xu'v'ev
a coefficient; p01)
70-11'/ -...'// (') CC-1XCo; Tro' ((04))) -Tthdd "a..:/)...((t....... ((tt), CCLXCr 3' ==M((b"-)) 11-1 11)0)111)))6--111 E C9-1, ((vv)( is the conjugate of ((10)(-111)
16

---

Under review as a conference paper at ICLR 2023
Proof. First, we focus single convolutional layer.
According the DFT and the inverse DFT, we can obtain the mathematical relationship between
G and F(l.c) and the mathematical relationship between (1,u0) and ( [er=d]] as follows.
de
cts
M M-118 N-1
R-1
K-1
Fhhe - 1( +7 12
T((.u))
Σ
Σ
W:00 e
122
de
m=0 n=0
1=0
==0
(11)
pillc)
Σ
(hh e
+
1
MN
c6a
MN
Based on Equation [1] and the derivation rule for complex numbers (Kreutz-Delgado 2009). we can obtain the mathematical relationship between 8Lost and 8lass Dc!!.e
aF and the mathematical rela- mm
tionship between
BLoss 0T (1ao) and ÖLoss aw [i] as follows. Note that when we use gradient descent to
optimize a real-valued loss function Loss with complex variables, people usually treat the real and imaginary values, a C and b C, of complex variable (z + bi)
separate real-valued
variables, and separately update these two real-valued variables. In this way, the exact optimization step of computed based such technology is equivalent to 2Lass Since Fll.e) and W [[(eeerrdd) are real numbers, aLass aLoss and aLoss aF
cts
ŌLoss
apihe)
Loss (e(le)
M- -
Loss
33 N 12
aLoss
K-1K-1
Σ
Loss
)2
MN
OF
(.))
MN
Ow())
()) ber-d]
de
LLsss
Losss
12
ôLoss
aosss
)2
aw [kere
87ul.wo
mn
(12)
Let us conduct the convolution operation (based on Assumption on the feature map F(l-1) [1(1-111), p(1-1.2) ..... p(1-1,Ci] E CCMMN, and obtain the output feature map F())
(P(.)) p(..),... ..... p(.b)) E DXMNN of the I-th layer as follows.
C K-IK-1
F(ld)) ((d)
Σ
W
( cta
(13)
1=0
Based on Equation
II and Equation 12 and the derivation rule for complex numbers (Keutzz-
Delgado 2009 the exact optimization step of Th
(ll.vv) in real implementations can be computed as
17

---

Under review as a conference paper at ICLR 2023
follows.
Öooss
OT!h.n))
R- K-IK-1
E
Σ
LLsss (D
97 )au
//Equation 12
MN
d
K-
Σ
Σ
Loss
e cl 22)
//Equation 13
MN
aFt.d)
//B //Equation 11
K-1
M-1N-11
Lass
N-1N-11
(1110))
-1( -1("" (1mm
s'cmpio 12x el + #2)2
MN
MN P.d
m=D
v'=0
Σ
12
N-1
Öooss
i0 "aa + 12 " 12m
MN
MN
FF.laa
K-1K-1
MM-1N-1
E Σ
Σ
1.) oLoss
-st + +
e 12
//Equation
MN
aGu'd
(.d)
1-U s
w'=0
K-1K-1M-1N
Σ
Σ
ÖLsss
+ (0- 12
MN
Σ
(1-1c) ŌLass
Σ
(u-u/se (u- 125
MN
K-IK-1
1/ Let Xa'wtur
MN
L=D 2
M-1N-1
Σ
Σ
Xu Go (G(
(l-1.e) ÖLoss
where Xw'v'ue can be rewritten as follows.
K-1K-1
(u-e/it (o-e'1a 1n y2a
Xutorue
MN
K-1
MN
100
K(u-
sin
K K(v-v')
(K- 1i--- (K- -1--)x-o))))
// //According Equation 1
MN sin
(w-
sin
18

---

Under review as a conference paper at ICLR 2023
Similarly, we computed the gradient of the loss function w.tt the spectrum map T
(110
as follows.
Looss
M
Loss
//Equation 12
MN
OF
1.c)
W U|
aLoss
o( u/m 4 y2a
//Equation
13
MN
cts
0F
OF
U.d)
m-
Eqqution 12
M-IN-1
1
Σ
Σ
W W
ΣΣ
Loss clm vim 2) 127 e
MN
00 (la)
M-1N-1
Σ
Σ
M
oLoss
i M + )2
K-1
W (ppor - +3 32327
Sp y2m
MN
a0 um
m=0
8=0
M-1N-1
Σ
M
ÔLoss (1.n)) il um )2T
10 12/27 o
//Equation
MN
00 (id)
de
m=0 m=0
u=0
u=o
M-IN-1
M-1N N-11
ôLoss 7ll..))
MN
n=o
Loss uv)
8 8u-w
//Equation
aG!l.a
ØLoss (l.''')))
de
Based on the derived
Loss (u
E C and
BLoss a0
C, we can further write gradients
aLoss
a
CDxC and
C as follows.
0lg
WIyT
M-1 N-1
OLoss
Σ
Σ Xa
ÔLoss
(14)
a(Tilio))T
0 (gl)
n/=0w- w00
ØLoss
oLoss
7lli'oo))
(15)
0 01 1
G(g ()
Furthermore, we extend the above proof of single convolutional layer to network with L. cas- caded convolutional layers. Let gllaav') denote the frequency component at the frequency [u', o'] of the 1-th layer's output feature, and let T(l.v)) the matrix computed by the I-th layer's convolu- tional weights. Then, according to Equation 15). the gradient w.r.l. g g(l,a'e') be computed as follows.
LLsss
ØLoss L.n''
0 Und
0
(16)
ØLoss
19

---

Under review as a conference paper at ICLR 2023
According to Equation 14).
the gradient w.r.
can be computed as follows.
aLoss
M-IN-1
Xu (,)))
Loss
a(Tiliue)
0(g (g'laav
1/Corollary
1
M-1N-1
Σ
Xu (T (:1 w'))
Oooss
p(olo()aa1)) (17)
O(g (g
// Let (ac)
(u (wv) = (L.)))
g
M-1N-1
Σ v'=0
Σ
Xure'uo (T un
+ (( (1i
aLoss
0(h (
Let us use the gradient descent algorithm to update the convlutional weight [:(D[(ardd]] In of the n-th epoch, the updated frequency spectrum W [[]][kerdd]] n++1 can be computed as follows.
[[ d] cte
(0/eerr In 7) cts
ÔLoss
Vt,
h+1
OW [erdd
where 17 the learning rate. Then, the updated frequency spectrum ((.ue) n+1 computed based on Equation 12 is given as follows.
Arrllau)
y(ll.uv)
I++1
12
K-1
K
w(ll(ker=d] cta
T(lluo)) In / //Equation In + )2x
do
t=
[kardd]
Loss
)e 3/ 12 -T((.w))
cte
11
ww [[Kerrd]]
cts
K-1K-1
K-1K-1 ÖLoss
Wete
ne " )2 )2
T((luv) In) 17
o + y2
aw [0feerl (0]errd cts
K-IK-1
oooss
Σ
5 )m
//Equation
ow U[kerd
1=
2=0
Loss
N
//Equation
12
Lwo)
Therefore, we prove that any step on W [[]eeed equals MN step on T In this way, pull Equation 17 in the change ((,w)) can be computed as follows.
(h.u)) = -
M-1N-
Σ
of (Td (od)) o. B²) a why
ØLoss T (1244)
(18)
w/=D
A.5 PROOFS OF ASSUMPTION 2 AND THEOREM 3
In this section, we prove Assumption 2 and Theorem 3 in the main paper.
Assumption 2. We assume that all elements in Tillww) are irrelevant to each other, and vI + 1' elements in Thaae) and we) are irrelevant to each other in early epochs.
vd # d': of d, Baa | (Tll (p(t
Ey (T(l.ar) Ey (()))
d'd
20

---

Under review as a conference paper at ICLR 2023
VI, e, c, EweD... w(y) (T ( 1011 Buth (Til. 1) was
Tre (T
Besides, according to experimental experience. the mean value of all parameters in w usually has small bias during the training process, instead of being exactly zero. Therefore, let us assume that in early epochs, each parameter in W is sampled from a Gaussian distribution N(u.of)
Proof. Given an initialized, cascaded. convolutional decoder' network with L convolutional layers, let us focus on the behavior of the decoder network in early epochs of training. We notice that each element in the matrix T(t.u)) is exclusively determined by the e-th channel of the d-th kenel W. ",IKK.1:K E IR Tgx*K according to Equation 5). Because parameters in w in the decoder network set to random noises,
consider that all elements in ((llvv) irrelevant to each other, i.e. dd o d', o + d' (1(.sv)) is irrelevant to 1 2((,,00)_ Similarly, since different layers' parameters w are irrelevant to each other in the initialized decoder network, we can consider that elements in different layers' T(llao) irrelevant to each other, ie., vA F I', elements in T'(l.s)) and elements in 7 .e)) irreeevnt to each other. Moreover, since the early training of DNN mainly modifies few parameters according to the lottery ticket hypothesis (Frankle & Carbin 2018), we can still assume such irrelevant relationships in early epochs, follows.
Then, prove Theorem
Theorem 3. Based on Assumption and Assumption distribution of complex numbers, as follows.
2 we can prove that T((,uo) follows a Gaussian
Vd.e
do
-oomplexX (p p4 Ru., 0² K²o, of Ram.2w)
sin(uKT/MM sin( (K 11u (8- 110 y
st. Run sin(um/A MM) sin( Su(uU//)
According to Assumption each convolutional weight follows Gaussian distribution, i.e., Wker=d ~ N(u, of). For the convenience of proving, let us extend Wherrd into an complex cte number. this way, Wike=dd follows a Gaussian distribution of complex numbers, Le., Wker=d ~ cta
Proof.
cts
Complex (f4 of,0) 0).
Previous studies Tse & Viswanath (2055) proved that given N complex numbers, if each complex number follows Gaussian distribution, then the linear summation of these N complex numbers also follows a Gaussian distribution of complex numbers. Since (r(.ur)) is a linear combination of de
Vt, s, (0][k]r)dd] gp(.uv) also follows Gaussian distribution complex numbers follows.
de
Ad, c T(lluw) Coomplex 0f,,02,1
where
1 B(T((uu))
//By definetion of μ
1-
E
w w0Ol
[kke 2 (( )2m]
Equation
1 1/v ors
E (Kker w (fterrd
B 01 ker
E W ker-d
ct's'
K-1K-1
B
E
(0 [tr (e +
K- K-
1/E /E (1 cts
P
t= 0=
K 1
/flet
Rus
edd uc 22
t=0
==0
21

---

Under review as a conference paper at ICLR 2023
a² B[(TTe
E(T
de (TT(
BB definetion of o²
Var(T
K-1K-1
-aal
Σ
W Ofkerrdd e +
cts
/Equation
11
1/v + t' or s'
E W [err=
W ())kerd
B[W
(1) ker=d]
E W (
cts
ct's'
K-1K
VarlW
Ofker cx( (2
cts
K-1K-1
(arlwe(0)eerr cto
//VarlaX]= = VaalVVr(XX]
K-IK-1
Σ
Σ of
//Var //Var(Wadt (W 1 oî
cto
1=
==
Ka
Bl(T(hiw)
B(Thhwo
DJ
//By definetion of
de
//Define C[X] E[(X ( E
K-11-1
C
Σ
W ker 1 ( * )2m]
/Equation
11
1/V x or o s' B [k)r W (terdd
E w
[lker d) cts
[E W
(-())eeeer)) kered)
C[ har -
K
C[W (0 je (x( 2 22
1/0|ax|= = -
K-1K-1
of
e 2n ym 2m
//Var[W(() (1)kkr) o7
==0
K-1K-1
R2
1/ Ruv
Σ 1=
Σ
1
Finally, let consider the value of Ruv.
Ruw
22x
==0
Suz
< 20z
sin( Ku
M
sin sin( m))
K 4) N
" (K Die
(K 10 //According to Equation
sin M x)
Therefore, we prove that
Ad, c
(hlav) Complex pe Ruee 8² 12t x of R2n.2o) de sin(uKT/MM sin(vKT/NN) "1 (K-1)) + (K 1)-
s.t. Rue sin(ut/M) sin(uT/N)
22

---

Under review as a conference paper at ICLR 2023
A.6 PROOF OF THEOREM 4
Theorem 4. (proven in Appendix A.5) Based on Assumption and Assumption we can prove that T(lav) follows Gaussian distribution of complex numbers, as follows.
Ad,
T(l.u)) Complex =14 Ru,, 0² 1r = =o R2m.20)
de
Ruv sin(uKT/MM) /inuuKK/// sin(uKK/N)
e: 1K 1 ,1, 1K 110
sin(ut/M) sin(vT/N)
Proof. According Theorem 4 Vd, e. 1ET pyuuv V Var (rr 1²007
SOM
= EE(T((lu 121 de
|B(T(l ( + aarl I (T
(19)
1 Ruv 2 K20
Then, we have
log (og(SMM( (T(W))(())))
log( (To)
log (4. ((
Assumption and C =
Log(EE (,00) (1 11
log(()/z Rav 2 //F /Equation 19
log
pe Ruvl k2
1=1
log( (n Ruv! 2 K2a))
For the more general case that each convolutional kernel contains more than one channel, ie., VI, C. >> the (eeo)t)))) also approximately exponentially increases along with the depth of the network with quite complicated analytic solution, as proved below. Note that the fol- lowing proof is based Assumption Besides, we further assume that all element in ((=)) are in- dependent with each other. Le., Vd * d': c. + d' (())))))))))
E ((v))) ( (((()))))))))))
Proof. According Theorem all element (t.uw)) follow the same Gaussian distribution. There- fore, we have
B[T(l.ue)) E E ((xxxrL)) ui 1 1(xxx
(20)
and we have
(O.(.()).o)) =SOM(T(()an)) Maxa 1)
(21)
-luy 1 K2 22 1)
Let us first consider the expectation of (L:)) follows.
23

---

Under review as a conference paper at ICLR 2023
B[T(
(CC.-1IB(T1) (1 E (C //Assumption
Equation 20
Ro (Cxx..)) /
//Theorem
//
Assumption 2
Ct
1=1
(22)
Then, we have
SOM (T(uxaa))
(CL-ISOM(THL L.w)) ( CI-1 1) (0 / 1 (12200))
//According Assumption and Equation 21
//we further Assume vd * d';c o' (11)( ( = (1:)
de
(IAr Ruol + K2 =ko,))5OM (Too)) Cz-1 1
E (w) (()
Ct.-1
// //According to Equation
19
Equation
22
C1 II
Ci (Koi)?)
Ct-1 Rupu 2 II Cj- Blry
C
C
[=1
k=
5=1+1
+ (Kos)²)) 1cxx0o
(23)
Therefore, we prove that for the more general case that 1I, C. 1, the second-order moment SOM(T(w))(L))) also approximately exponentially increases along with the depth of the network.
A.7 PROOF THEOREM 5
In this section, we prove Theorem in the main paper.
Theorem 5. Let each element in each c-th channel p(r) of the feature map follows the Gaussian distribution (Na, o²). G(o) MxN denotes the frequency spectrum of glo) and H(S) e CM' denotes the frequency spectrum of the output feature Fll) after applying zero-padding ple), Then, the gero-padding on ple) brings in additional signals at each frequency [u, v] as follows, whose strength is measured by averaging over differrntt sampled features.
10500 0A < M,0KM Epto [ - G( = |al (1 (sin(MuT//M NUT/N
MN8
(in(ux/M)) UT/N
WA
S M N50K Bpt !
sin (MM) NeT/N (aluu//M) vn/N
IN De
24

---

Under review as a conference paper at ICLR 2023
Proof.
M-14 N
E Epic)
| = E
Σ Σ
F -1 27 + )mm)
//Equation 11
0 n
-
(24)
M- N-1
Σ
Σ
2
1/F -N(a, mn
a
m==0 n=
aMMo
0 S u<
SUC A.....SS
//Equation
Epte: [Hle) uv
M' 1-
= =ppool
Fle) un + 3 12m]
Eqqation 11
M-1 N-1
Eptol Σ
Fle) -
um
+ yz 2
mx e
n=0
M N-
M
// mn
~ NN
172 0
0=
sin
Mu M
sin
No No
i (M-1)= (N-11" 5=,
/ //Equation
sin
sin
N7 π
(25)
When 0 Lu M,0
Epte [H -G(oll = Epee (1 G
Epe (1())
Epto [G
/ According to Equation 24
Equation 25
No No Mu T) sin sin( M
π
(M D N N6
sin M T) sin( N π))
No sin( Mi x) sin
Mu
No π.. MN8
jal(
sin Mr n
sin ())
When M S M' N SE<N N'
Eprol [H = pr [H(1)]
Mu sin
Mr π sin NF
(2---1 (8-19 ys| // //Equation 25
la sin( ()) sin N
sin( Mr π sin N# m
|a||
sin म
sin N
Therefore, we prove that
sin( Mu 27
π) sin( Nu
Nr T
VO < u <
(
- Givll |a|(1
MN5
sin( Mi π) sin( x. TT)
sin m) sin( No T))
N
sin u σ) sin( / T)
25

---

Under review as a conference paper at ICLR 2023
A.8 PROOF OF THEOREM 6
In this section, we prove Theorem 6 i the main paper.
Theorem 6. Let G (G G ..... G ( CCxMMoXNo denote spectrums of the Ci channels of feature F. Then, spectrums H = ( H HDCo e CCIMMNN of the output feature F can be computed as follows.
U,,u,U u+ (0M0
H
Gue
,..... M/Mo: ,.... N/No
(26)
Proof.
Mo-1 No-1
Glo Σ
Σ
F(e) e
i{ + No 12r
Ma
//Equation 11
(27)
M-1N-1
Σ
Σ
p(c) -1( 1)) (u=(t- y2n
Eqqution
Mp-1 Na-1
Σ
(a MomaXmmtttio) te 11 rantiol j2a
mn
mad mm0
Mo-1NNo1
Flo) -i( (c-Maam)) (o+((e-))N)) )2a
Σ
Af/ratio
N/ratio
m=0
n=0
//M = Mo . ratio; N No ratio
(28)
Mo No
-(( (1-
(o+ No) j2n
mne
Mo
No
m=0 ==0
Mo 1l--1am No-1
um
on
e 1((( (1- 12n
Mp Mo-1
Σ
Σ
Ele) muL
My
Ng
1/7,1EZ
Gic)
//Equation 27
Therefore we prove that:
ue,u,u, U, H(s)
Hut (0---11 G(c)
s.t. = I,..., 1,...,,MMM;; t= Iy..., N/No
(29)
B RELATED WORK
Although few previous studies directly prove DNN's bottleneck from the perspective of represent- ing specific feature components, still make survey on research on the representation capacity of DNN.
Some studies focused on specific frequency that took the landscape of the loss function on all input samples as the time domain (Xu et al.) 22099; Rahaman et al. 2019 Xu et al.) 2019a Luo al. 2019). Based on such specífic frequency, they observed and proved a phenomenon namely Frequency Principle (F-Principle) that a DNN first qucikly learned low-frequency components, and then relatively slowly learned the high-frequency ones, which might shed new light on understanding the representation capacity of DNN. For example, Lin et al. (2019) empirically proposed to smooth out high- gigh-freeuency components to improve the adversarial robustness. Besides, Ma et al. (2020) explored the boundary of the F-Principle, beyond which the F-Principle did not hold any more. In comparison, we focus on fully different type of frequency. i.e., the frequency w.E.. the DFT on an input image feature map.
26

---

Under review as a conference paper at ICLR 2023
In this direction. previous studies mainly experimentally analyzed the relationship between the learn- ing of different frequencies and the robustness of DNN, Yin et al. (2019) conducted lot of experi- ments to analyze the robustness of DNN w.r.r different frequencies of the image. They discovered that both adversarial training and Gaussian data augmentation improved the DNN's robustness to higher frequencies. Waang al. (2020) empirically proposed to remove high frequency components of convolutional weights to improve the adversarial robustness. In comparison, we theoretically prove representations bottleneck of DNNs in the frequency domain.
Besides, many studies explained the representation capacity of a DNN in the time domain. The information bottleneck hypothesis shows that the learning process of DNNs is to retain the task- relevant imput information and discard the irseliirelevant input information (Tishby & Zaslavsky/
2015 Shwartz-Ziv & Tishby 2017 Wolchover & Reading, 2017, Amjad & Geiger 2019). The lottery ticket hypothesis shows that some initial parameters of DNNs inherently contribute more to the network output (Frankle Carbin 2018). The double-descent phenomenon describes the specific training process of DNNs that the loss first declines, then rises, and then declines again (Nakkiran et al. 2019 Reinhard & Fatih 2020). DNNs with the batch normalization may sometimes conflicted with the weight decay (Van Laarhoven 2017 Li et al. 2020). DNNs vulnerable to adversarial examples (Szegedy et 22013] Goodfellow et al. 20144) DNNs tipically encoded simple interactions between very few input variables and complex interactions between almost all input variables, but were difficult to encode interactions between intermediate number of input variables Deng al. 2022).
C MORE EXPERIMENTAL RESULTS
C.1 VERIFYING THAT NEURAL NETWORK USUALLY LEARNED LOW-EREQUENT COMPONENTS HIRST.
In section, we provide more experimental results to verify that a neural network usually learned low- frequent components first, which had already been shown in Figure Ta) in the main paper. Here, we also constructed a cascaded convolutional auto-encoder by using the VGG-16 as the encoder network. The decoder network contained three upconvolutional layers for the CIFAR- 10 dataset, and contained three upconvolutional layers for the Broden dataset. Each convolutionallapcoovllutnnon layer in the auto-encoder applied zero-paddings and was followed by batch normalization layer and an ReLU layer. The auto-encoder was trained using the mean squared error (MSE) loss for image reconstruction. Results in F that the auto-encoder usually learned low-frequent components first and gradually learned higher frequecies. We also attached the generated image below its spectrum map Figure 7 in order to help people understand the learning process of the auto-encoder.
[a)
C.2 VERIFYING THAT THE UPSAMPLING OPERATION MADE A DECODER NETWORK REPEAT STRONG SIGNALS AT CERTAIN FREQUENCIES OF THE GENERATED IMAGE.
In section, we provide more experimental results to verify that the upsampling operation in the decoder repeats strong frequency components of the input to generate spectrums of upper layers.
First, we conducted experiments to verify Theoremp6in the main paper, which claims that the upsam- pling operation repeats the strong magnitude of the fundamental frequency Che of the lower layer to different frequency components Ve, (C) of the higher layer, where u 0, .M. 2.Mo. 3.M..... No. 2No. 3.N..... To verify this, given an image, let the image pass through four cascaded upsam- pling layers. We visualized the feature spectrum generated by each upsampling layer, in order to verify whether the upsampling operation repeated the strong magnitude of the fundamental fre- quency of the input image to different frequency components of the feature spectrum generated by upsampling layers. Results on the CIFAR-10 10 dataset and the Tiny-ImageNet dataset in Figure verified Theorem
Second, we provide more results on real neural networks, which have already been shown in Fig- ure (b) in the main paper. We also constructed cascaded convolutional auto-encoder by using the VGG- 16 as the encoder network. The decoder network contained four upconvolutional layers. Each convolutionaluupcoooottui layer in the auto-encoder applied zero-paddings and was followed by
27

---

Under review as a conference paper at ICLR 2023
Input, spectrum
Epochs
Input, spectrum
Epochs
(a) CIFAR-10 dataset
(b) Broden dataset
Figure 6: Magnitude maps feature spectrums of different epochs' network output. Each magni- tude map was averaged over all channels. For clarity, we moved low frequencies to the center of the spectrum map, and moved high frequencies to corners of the spectrum map. Note that we set the magnitude of the fundamental frequency to be the same with the frequency that had the second large magnitude. For resutls in (b), we only visualized components in the center of the spectrum map with the range of relatively low frequencies {u|0 < M/8) (u/7M/8 < M); {uj0 S N/8) U (u/7N/8 & o < N for clarity.
a batch normalization layer and an ReLU layer. The auto-encoder was trained on the Broden dataset using the mean squared error (MSE) loss for image reconstruction. Results in Figure verified Theorem
C.3 VERIFYING THAT THE ZERO-PADDING OPERATION STRENGTHENED THE ENCODING OF LOW-FREQUENCY COMPONENTS.
In section, we provide more experimental results to verify that the zero-padding operation strength- ened the encoding low-frequency components, which had already been shown in Figure in the main paper. Here, also constructed the following two baseline networks. The first bascline network contained convolutional layers, and each layer applied zero-paddings. Each convolutional layer contained 16 convolutional kernels (kernel size was 7x except for the last layer containing convolutional kernels. The second baseline network was constructed by replacing all zero-padding operations with circular padding operations. Results in Figure 100 verified that the rero-padding operation strengthened the encoding of low-frequency components.
C.4 VERIFYING THAT DEEP NETWORK STRENGTHENED LOW-FREQUENCY COMPONENTS.
In section, we provide more experimental results to verify that deep network strengthened low- frequency components, which had already been shown in Figure in the main paper. Here, we also constructed network with convolutional layers. Each convolutional layer applied zero- paddings avoid changing the size of feature maps, and was followed by an ReLU layer. We visualized feature spectrums different convolutional layers. Results on the CIFAR-10 dataset and the Tiny-Imagele dataset in Figure [1] show that magnitudes of low-frequency components increased along with the network layer number.
C.5 VERIFYING THAT A LARGER ABSOLUTE MEAN VALUE μr OF RACH 1-TH LAYER'S PARAMETERS STRENGTHENED LOW-FREQUENCY COMPONENTS.
In section, provide more experimental results to verify that larger absolute mean value p of each /-th layer's parameters strengthened low-frequency components, which had already been shown in Figure 5(b) in the main paper. Here, we also applied a network architecture with 5 con- volutional layers. Each layer contained 16 convolutional kernels (kernel size was 9x9), except for the last layer containing 3 convolutional kernels. Based this architecture, three networks, whose parameters were sampled from Gaussian distributions N(μ = 0, = 0.013), N(p 0.001, a² 0.013), and Mip 0.01, 0.01²). respectively. Results the CIFAR-10
constructed
28

---

Under review as a conference paper at ICLR 2023
Inpui, spodiam
Fpohi
Rpoak
Fyodkt
Fywcett
Fystiioo Fyeskbee Tpeckto
Input, spectnm
Epaad
Spudtuoe Epecatoo Excco boocko
(a) CIFAR-10 dataset
Impai
tredi
tavdo
toeddo
tpeczoo
tpucata
ttrectil
Inpuri, specturm
Fyucal
Fewtton Sleck
ryuckien
(b) Broden dataset
Figure 7: Magnitude maps of feature spectrums and the corresponding generated images of differ- ent epochs. Results show that the very few epochs the training, the network removed noisy signal caused by the upsampling, to some extent, which were in the grid pattern in the spectrum. After that, the network learned low-frequency components first, and then gradually learned higher frequencies. Each magnitude map in this figure averaged over all channels. For clarity, moved low frequencies to the center of the spectrum map, and moved high frequencies corners of the spectrum map. Note that with the frequency that had the second large magnitude.
the magnitude of the fundamental frequency to be the same
Input
32x32
64x64
128x 128 256x2565 512
Input 64x64 128x 128 256x256512x3 1024x 1024
(b) Tiny- ImageNet dataset
(a) CIFAR- 10 dataset
Output spectrums of upsampling layers
Output spectrums of upsampling layers
Figure 8: Magnitude maps of feature spectrums after onc/two/there/four upsampling layers. Each magnitude map was averaged over all channels. For clarity, we moved low frequencies to the center of the spectrum map, and moved high frequencies to corners the spectrum map.
29

---

Under review as a conference paper at ICLR 2023
Input
4x4
7x7
14x14
28x28
56x56
112x
12224X224
Output spectrums of upsampling layers
Figure 9: Magnitude maps of feature spectrums after one//wo/therefforr/iveesss upsampling layers. Each magnitude map was averaged over all channels. For clarity. we moved low frequencies to the center the spectrum map. and moved high frequencies corners of the spectrum map.
Input
circular
Input
circular
Input
circular
(b) Tiny- Tiny- ImageNer dataset (a) CIFAR-10 dataset
Figure 10: A network with zero-padding operations usually strengthened more low-frequency com- ponents than network with circular padding operations. Here, each magnitude map of the feature spectrum was averaged over all channels. For clarity, we move low frequencies to the center of the spectrum map, move high frequeneies corners of the spectrum map, and the magnitude of the fundamental frequency to be the same with the frequency that has the second large magnitude.
30

---

Under review as a conference paper at ICLR 2023
Input
Layer Layer 3 Layer 10
Input
Layer Layer
Layer 10
[a) CIFAR 10 dataoet
Input
Layer
Layer 3 Layer
Input
Layer
Layer Layer 10
(b) Tiny ImageNe dituset
Figure 11: Comparing feature spectrums different layers. Results show that higher layers a network usually generated features with more low- freeuuenenyy components. For clarity, move low frequencies to the center of the spectrum map, move high frequencies to corners of the spectrum map, and set the magnitude of the fundamental frequency to be the same with the frequency that has the second large magnitude. For resutls in (b), we only visualized components in the center of the spectrum map with the range of relatively low frequencies E {w|0 M h E {ujo < < N/6] U (u/5N/6 < < N for clarity.
M/63 {u/5M/6
Inpui
0.001 μ
Inpul
# 0.001
(+) CIEAR-IN 10 dataset
Input
0.001
0.01
0.001 0.01
(b) Tiny ImageNNe daiaset
Figure 12: A network whose convolutional weights had mean value significantly biased from 0 usually strengthened low- low-frequency components, but weakened high-frequency components. Here, each magnitude map of the feature spectrum averaged all channels. For clarity, we moved low frequencies to the center of the spectrum map, moved high frequencies to corners the spec- trum map, and the magnitude of the fundamental frequency be the same with the frequency that has the second large magnitude. For resutls in (b), we only visualized components in the center of the spectrum map with the range relatively low frequencies u (ul0 S a [u < < M);v (v < N/6) U [u|5N/6 & < N) for clarity.
dataset and the nents increased along with the absolute mean value of parameters.
dataset in Figure 12 show that magnitudes of low-frequency compo-
31


Reviews:
[{"review_id": "w8PgKxbl0Dx", "reviewer": "Reviewer_V1PA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the forward pass of a convolutional decoder in the frequency domain. It finds that the CNN layer forward propagates each frequency component in the\nspectrum map independently to other frequency components.\nIt also finds that the CNN operations make a convolutional decoder network more likely to weaken high-frequency components.\n", "summary_of_the_review": "Decent work with some issues that would warrant a new revision before acceptance.\n", "strength_and_weaknesses": "Strengths:\n\n—-------------------\n\nThe paper has many strengths and is tackling an important problem. For the sake of time  I am only writing about weaknesses in this review, since those are the ones that should be actioned upon.\n\n\nWeaknesses (W)\n\n—-------------------\n\nW: As far as I understand the analysis for the deep network is done on an idealized frequency domain network that does not correspond to a CNN with ReLUs and finite size filters. If this is true, this is the biggest drawback of the current paper. Furthermore, in real world networks, very small filters (e.g., 3x3 or even 2x2) usually provide the best results.\n\nW: Keeping in mind the previous comment, the paper’s language might be too strong. It talks about proving various aspects of the forward propagation of the CNN, but maybe the proofs only apply to the idealized network?\n\nW: Regarding zero-padding, there are previous works that try to compensate for the resulting border effects, e.g., by learning different filters to the borders or using circular or other types of padding and convolutions. Could the authors discuss how zero padding vs. these methods affect the end-results of the CNN decoder and whether it would be beneficial not to zero-pad.\n\nW: The paper makes the claim that decoder CNN is biased towards learning low-frequency content. What is missing is the analysis on when this is actually harmful and whether it could be beneficial in some settings. Also, in cases where this is harmful, it would be interesting if the authors could provide a suggestion for improvement.\n\nW: The analysis is done for randomly initialized networks. While this is an interesting point of analysis, it is lacking the effect of training altogether. I would suggest the authors to extend their work towards the training as well.\n\nW: Based on Figure 3, ReLU seems to have a big effect when comparing the freq domain version of the network with the original one. A big downside of the analysis is the lack of analysing the effect of ReLU, since, of course, the activation function is a key component in enabling nonlinearity in the neural networks.\n\nW: The authors should bring the related work section from the appendix to the main paper and also build a connection with related works on freq domain neural networks, for example. Pan, H., Chen, Y., Niu, X., Zhou, W., and Li, D., “Learning Convolutional Neural Networks in the Frequency Domain”, arXiv e-prints, 2022. https://arxiv.org/abs/2204.06718 , and many others.\n\nW: The authors note that their work is different from the “F-Principle” e.g., https://arxiv.org/abs/1807.01251 because the current paper focuses on a “fully different type of frequency”, i.e., the frequency w.r.t. the DFT on an input image or a feature map. To me this distinction is not so clear and both works seem to be related. I would suggest the authors to build a better bridge between the works, since even the authors seem to be doing similar experiments, e.g., in “C.1 VERIFYING THAT A NEURAL NETWORK USUALLY LEARNED LOW-FREQUENT COMPONENTS FIRST.”\n\n\n\nMinor issues and spelling mistakes:\n\nB, page 26: “DNN first qucikly” -> DNN first quickly\n\nPage 27: “DNNs tipically” -> DNNs typically\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LFi_3ay6Ax", "reviewer": "Reviewer_oTym", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of cascaded convolutional decoder network on the spectrum domain, and presents several findings summarized in the experiment section.", "summary_of_the_review": "While this paper presents various effects on the spectrum domain of different components in a cascaded convolutional decoder network, the paper lacks mathematical novelty in its theoretical results and discussion/comparison with closely related works. The presentation of this paper should also be improved significantly. ", "strength_and_weaknesses": "Strengths:\n\n1. The authors performed a detailed study on the effect of different components in a decoder network on the spectrum domain by discrete Fourier transform.\n2. Extensive experiments are conducted to support the claims of the authors.\n\nWeakness:\n\n1. The findings of the paper are not surprising, and using discrete Fourier transform to study the learning of neural networks on the frequency domain is not novel. Previous works have proved that neural networks tend to learn low-frequency components [1] with the help of discrete Fourier transforms [2,3]. More discussions need to be included on the significance of this paper compared to previous works. \n\n2. The findings are limited to decoder networks. The theoretical proof does not consider common activation layers in decoder networks such as ReLU.\n\n3. The paper's organization within Section 2 and Section 3 is messy.\n\nBesides the above weakness, the other issue is about the (mathematical) novelty in many theoretical results. For example, Theorem 1, Corollary 1, and Theorem 2 are just another format of the very famous convolution theorem: Fourier transform of the convolution of two inputs signals is the elementwise product of the Fourier transform of the two input signals. \n\n[1] Kiessling, Jonas, and Filip Thor. \"A Computable Definition of the Spectral Bias.\" AAAI 2022.\n\n[2] Rahaman, Nasim, et al. \"On the spectral bias of neural networks.\" ICML 2019.\n\n[3] Xu, Zhi-Qin John, et al. \"Frequency principle: Fourier analysis sheds light on deep neural networks.\" arXiv preprint arXiv:1901.06523 (2019).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HcLTDq35gzl", "reviewer": "Reviewer_fHYG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims to prove representation bottlenecks of ConvNets considering the capacity of representing different frequency components of an input sample. It claims to introduce the rule of the forward propagation of intermediate-layer spectrum maps, as equivalent to the forward propagation of feature maps through a convolutional layer. It focus on layers representing convolutions, and zero-padding operations.\n", "summary_of_the_review": "I dont see the novelty of this paper. \n\nIt presents as new, a theorem that conducting the convolution operation on an input feature F is essentially equivalent to\nconducting matrix multiplication on spectrums of F (Theo 2). That is a well known results for Fourier analysis.\n\nNext, it reformulates the cascade of convolutional layers, without activation functions. It is also well known that a stack of convolutional layers can be rewritten as a single convolutional layer if no non-linearities are used. But such a simplification is much less expressive than deep models adopting non-linearities. \n\nOn the experimental section, the paper claims to 'Verifying that a neural network usually learned low-frequent components first.' This result and analysis was previously presented in Rahaman et al.(2018) On the Spectral Bias of Neural Networks.\n\nThe paper also investigate the effect of zero padding, and upsampler layers, but for this last one, the formulation adopted seams to be the introduction of zeros on new subpixel positions (similar to those modeled by a dilated convolutions), other than investigating what happen with upsampling filters. ", "strength_and_weaknesses": "The formulation seams to claim as novel existent well known concepts, at the same time that oversimplify the analytical model of ConvNets.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2zOjK6NqYS", "reviewer": "Reviewer_wmw7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides the bottleneck of representations in linear convolutional networks. This is based on the frequency analysis of the input and how it propagates into the network. They characterize and analyze the network in detail. Here are some of their statements as contributions: Each frequency propagates independently of the other through network mapping. The convolution and other operations related to it (e.g., zero padding) damp high-frequency components. The last finding (in their own wording) is that upsampling results in strong signals repetitively appearing at certain frequencies.", "summary_of_the_review": "I very much enjoyed reading the paper. The paper gives lots of insights into linear convolutional neural networks and how frequency information passes through the network. However, all those insights are already known in the literature. I do not recommend the acceptance of this work for the following three reasons:\n\n1. Almost all the analysis and intuitions given by the paper are known for convolution operators (convolutions of convolutions is also a convolution). When the authors reduce their neural network into a convolution, then all results are immediate. Hence, results on the independence of frequency propagation, how down zero-padding works, and effect upsampling are already known for convolutions. \n2. Assumptions and simplifications are extreme.\n3. The paper does not acknowledge signal processing literature and its wording implies their work is new.", "strength_and_weaknesses": "The paper is written well and explained very clearly. The authors make it clear how their approach to frequency encoding differs from prior works. I very much enjoyed reading the paper. However, I have mixed feelings about this work. The paper gives lots of insights into linear convolutional neural networks. For someone with a signal processing background, however, I find the work to explain the known basics of discrete-time signal processing [1] to the deep learning community. Below I explain in detail.\n\nI found the work to miss acknowledging the digital signal processing literature and their terminology in this paper. The paper misses the literature on discrete-time signal processing. All the properties mentioned in this paper are known for convolution operators. Moreover, the authors lack certain terminologies known in the literature. For example, the \"repeats strong frequency components of the input\" in Figure 1 and (Conclusion 4) is known as \"aliasing\". Below are a few examples.\n- Their first theorem and the second are basic principles of convolutions and discrete Fourier transform. Could the authors explain what is new about this result?\n- The independence of DFT points in a convolution operator is known in the literature. What is new here?\n- Upsampling and aliasing are known in the signal processing literature. It is not clear what is new in their results on upsampling.\n- It is very clear from the perspective of signal processing that if the kernels are initialized with Gaussian distributed data, then the spectrum allows follows a Gaussian shape, and depending on the variance, etc. it acts as smoothing. Is this something new?\n\nThere are certain simplifications that are extreme.\n- On page 3, the authors mention that the diffraction process is small and can be ignored, but perhaps the main reason the cosine similarity in Figure 3 is not exactly 1 or very very close to 1 is due to the diffraction process term. Assuming the theorem is correct, Figure 3 indicates that the diffraction process should not be ignored.\n- Non-linear activations are building blocks of neural networks. This paper considers a linear network. This is extreme simplification and reduces the setting into a case that is known and studied already in the literature (Convolution operator or cascade of convolution operators are known for how they perform in the signal processing literature). Please state clearly in the abstract that you are considering a linear network.\n- They provide analysis of the network when it is NOT trained (or very early stage which is still assumed to behave like an untrained network) with independent variables and the assumption that the parameters are drawn from Gaussian distribution.\n\nMisleading statements:\n- \"zero-padding operation boosts magnitudes of low-frequency components of feature spectrums of the feature map\". Zero-padding in the spatial-image domain, from the perspective of DFT, increases the resolution in the frequency domain. It does not boost low-freq or filter high.\n- The author's statement on the weakening of high-freq encoding is not generally true. This depends on the frequencies that each convolutional kernel passes. In their framework, they assume that the kernels pass low frequencies implicitly by using Gaussian distributed data. Moreover, this depends on the data the network will be trained on. For example, as pointed out by the authors in Remark 5, natural images dominate low-frequencies, hence, CNNs trained on natural images tend to filter high-frequencies.\n\nSome word usage is not precise: \"Random noises\" in Section 3? I guess you mean drawn randomly from a particular (Gaussian) distribution. Please clarify \"irrelevant\" in Section 3? Do you mean \"independent\"? By \"upsampling\" do you mean \"expansion\"?\n\nPerhaps, if the work can be generalized to non-linear convolutional neural networks or networks that are trained and their dynamics, the findings would be new.\n\nMinor comments:\n\n1. I suggest not using a bold font in sentences.\n2. I found the first paragraph very disconnected from the main focus of the paper. It's the authors' choice, but I recommend making a better connection or starting with the second paragraph. For example, does the paper directly explain why DNN fails in any of the problems in the first paragraph? No, but you may say indirectly they can provide insights.\n3. Section 3, line 7, \"kenel\" --> \"kernel\".\n4. There is an extra \")\" in (11).\n\n[1] Discrete-time signal processing by Oppenheim and Schafer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject