Phase Collapse in Neural Networks
Authors: Florentin Guth
John Zarka
Stéphane Mallat
Venue: ICLR 2022 Poster (2022)
Keywords: phase collapse
neural collapse
concentration
classification
imagenet
deep networks
complex networks
sparsity in deep networks

Abstract: Deep convolutional classifiers linearly separate image classes and improve accuracy as depth increases. They progressively reduce the spatial dimension whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels. A fundamental challenge is to understand the role of non-linearities together with convolutional filters in this transformation. ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity. This paper demonstrates that it is a different mechanism called \emph{phase collapse} which eliminates spatial variability while linearly separating classes. We show that collapsing the phases of complex wavelet coefficients is sufficient to reach the classification accuracy of ResNets of similar depths. However, replacing the phase collapses with thresholding operators that enforce sparsity considerably degrades the performance. We explain these numerical results by showing that the iteration of phase collapses progressively improves separation of classes, as opposed to thresholding non-linearities.

Published as a conference paper at ICLR 2022
PHASE COLLAPSE IN NEURAL NETWORKS
Florentin Guth, John Zarka
Stéphane Mallat
DI, ENS, CNRS, PSL University, Paris, France (florentin.gut) john. zarka] @ens. fr
Collège de France, Paris, France Flatiron Institute, New York, USA
ABSTRACT
Deep convolutional classifiers linearly separate image classes and improve accuracy depth increases. They progressively reduce the spatial dimension whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels. A fundamental challenge is to understand the role of non- non-lincarities together with comvolutional filters in this transformation. ReLUs with biases are often interpreted as thresholding operators that improve discrimi- nation through sparsity. This paper demonstrates that it is different mechanism called phase collapse which eliminates spatial variability while linearly separating classes. We show that collapsing the phases of complex wavelet coefficients is sufficient to reach the classification accuracy ResNets of similar depths. How- ever. replacing the phase collapses with thresholding operators that enforce sparsity considerably degrades the performance. We explain these numerical results by showing that the iteration of phase collapses progressively improves separation of classes, as opposed to thresholding non-linearities.
INTRODUCTION
CNN image classifiers progressively eliminate spatial variables through iterated filterings and subsam- plings, while linear classification accuracy improves as depth increases (Oyallon 2017]. has also been numerically observed that CNNs concentrate training samples of each class small separated regions progressively lower-dimensional space. It can ultimately produce a neural collapse (Papyan al. 2020). where all training samples of each class are mapped single point. In this case, the elimination of spatial variables comes with a collapse within-class variability and perfect linear separability. This increase in linear classification accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases.
A difficulty in understanding the underlying mathematics comes from the flexibility of ReLUs. Indeed, a linear combination of biased ReLUS can approximate any non-linearity. Many papers interpret iterations on ReLUs and linear operators sparse code computations (Sun et al. 2018 Sulam et al.) 2018 2019 Mahdizadehaghdam et al. 2019: Zarka et al. 2020; 2021). We show that it is a different mechanism, called phase collapse, which underlies the increase in classification accuracy of these architectures. A phase collapse is the elimination of phases of complex-valus with modulus, which we show concentrate spatial variability. This demonstrated by introducing structured convolutional neural network with wavelet filters and no biases.
wavelet coefficients
Section introduces and explains phase collapses. Complex-valle -allued representations are used because they reveal the mathematics of spatial variability. Indeed, translations are diagonalized in the Fourier basis, where they become a complex phase shift. Invariants to translations are computed with a modulus, which collapses the phases of this complex representation. Section 2 explains how this can improve linear classification. Phase collapses can also be calculated with ReLUs and real filters. A CNN with complex-valuee filters is indeed just particular instance of real-valued CNN, whose channels are paired together to define complex numbers.
Section 3 demonstrates the role of phase collapse in deep classification architectures. It introduces Learned Scattering network with phase collapses. This network applies a learned 1 x convolutional complex operator P, on each layer z, followed by phase collapse, which is obtained with complex wavelet filtering operator W and a modulus:
Kft1
WWP IWPrayl

---

Published as a conference paper at ICLR 2022
It does not any bias. This network architecture is illustrated in Figure I With the addition of skip-connections, we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10.
Section compares phase collapses with other non- non-linearities such as thresholdings or more general amplitude reduction operators. Such non-linearities enforce sparsity but do not modify the phase. We show that the accuracy of Learned Scattering network considerably reduced when the phase collapse modulus replaced by soft-thresholdings with learned biases. This is also of more general phase-preserving non-linearities and architectures.
Section 5 exxlains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability. On the opposite, the improvements in classification accuracy of successive sparse code computations are shown quickly saturate.
The main contribution of this paper is a demonstration that the classification accuracy of deep neural networks mostly relies on phase collapses, which are sufficient linearly separate the different classes on natural image databases. This is captured by the Learned Scattering architecture which reaches ResNet- accuracy on ImageNet and CIFAR-10. We also show that phase collapses are necessary to reach this accuracy, by demonstrating numerically and theoretically that iterating phase-preserving non-linearities leads significantly worse performance.
id
id
id
Po 1 x
W
H
P1
W
P
W
|-|
PJ 1 x
FC class.
y
x
2
X
2
Figure I: Architecture of Learned Scattering network with phase collapses. has J + 1 layers with J= for ImageNet and for CIFAR-10. Each layer computed with 1 convolutional operator P, which linearly combines channels. It is followed by phase collapse, computed with spatial convolutional filtering with complex wavelet W and a complex modulus |-]. layer of depth j corresponds to a scale 20/2 and subsampling by 2 is applied every two layers, after W. A skip-comeectinn concatenates the outputs of W P3 and W P,|. A final P, reduces the dimension before linear classifier.
2 ELIMINATING SPATIAL VARIABILITY WITH PHASE COLLAPSES
Deep convolutional classifiers achieve linear separation of image classes. We show that linear classification on raw images has poor accuracy because image classes are invariant to local translations. This geometric within-class variability takes the form of random phase fluctuations, and as a result all classes have a zero mean. To improve classification accuracy, non-linear operators must separate class means, which therefore requires to collapse these phase fluctuations.
Translations and phase shifts Translations capture the spatial topology of the grid on which the image is defined. These translations are transformed into phase shifts by Fourier transform. We prove that this remains approximately valid for images convolved with appropriate complex filters.
Let æ be an image indexed by E 2 We write x, (u) = æ(u - T) the translation of x by r. It is diagonalized by the Fourier transform F(w) = En. (u)) e iunn which creates phase shift:
T., (w) T (WW).
This diagonalization explains the need introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities. Computations can however be carried with real numbers, will show.
Fourier transform computed by filtering E with complex exponentials e sw One may replace these by complex wavelet filters & that are localized in space and in the Fourier domain. The following theorem proves that small translations can still be approximated by phase shift this case. We denote by the convolution of images.
2

---

Published as a conference paper at ICLR 2022
Theorem 1. Let v: z² - C be filter with | 1, whose center frequency and bandwidth are defined by:
1 { Then, for any T E ZZ
w [[(W)) du and o² (2T)² f --mmpple - -E((E(W))
(2x)²
|-m,]]
||-, * w (x* S
The proof is in Appendix This theorem proves that if |r| < 1/0 then V * 15- w.. this case, a translation by produces phase shift by g.
Phase collapse and stationarity We define a phase collapse as the elimination of the phase created by a spatial filtering with a complex wavelet v. We now show that phase collapses improve linear classification of classes that are invariant to global or local translations.
The training images corresponding to the class label y may be represented as the realizations of a random vector Xy. To achieve linear separation, is sufficient that class means E Xx, are separated and within-class variances around these means are small enough (Hastie et al. 2009). The goal of classification is find a representation of the input images in which these properties hold.
To simplify the analysis, we consider the particular case where each class y is invariant to translations. More precisely, each random vector X, is stationary, which means that its probability distribution is invariant to translations. Equation (2) then implies that the phases of Fourier coefficients of X, are
for w + O. The class means E[X,] are thus constant images whose pixel values are all equal to E[X, E[X,(0)) (0)]. A linear classifier can then only rely on the average colors of the classes, which are often equal in practice. thus cannot discriminate such translation-invarian classes.
uniformly distributed in [0, 2.T]. leading to E[X, (w)]
Eliminating uniform phase fluctuations of non-zero frequencies thus necessary to create separated class means, which can be achieved with the modulus of the Fourier transform. It is translation- invariant representation: |ž, = [+] This improves linear discriminability of stationary classes, because -[IX,| | may be different for different y. However, J.x,, has a high variance, because the Fourier transform is unstable to small deformations (Bruna and Mallat 2013).
Fourier modulus descriptors can be improved by using filters v that have a localized support in space. Theorem I shows that the phase of X,, z also uniformly distributed in [0, 2T). It results that E[X,, y a 0, and .r * v still provides no information for linear classification. Applying a modulus similarly computes approximate invariants small translations: |r= + |ar * vb), with an error bounded by a |r| ||=r||2- More generally, these phase collapses compute approximate invariants to deformations which are well approximated by translations over the support of tb. This representation improves linear classification by creating different non-zero class means ||| b while achieving a lower variance than Fourier coefficients, as it is stable to deformations (Bruna and Mallat, 2013).
Image classes are usually not invariant global translations, because of e.g. centered subjects the sky located in the topmost part of the image. However, classes are often invariant to local translations, up to unknown maximum scale. This is captured by the notion of local stationarity, which means that the probability distribution of X,, is nearly invariant to translations smaller than some maximum scale (Priestley| 1965). The above discussion remains valid if X,, is only locally stationary over domain larger than the support of tb. The use of so-called "windowed absolute spectra" E B||x,, ,, + for locally stationary processes has previously been studied in Tygert et al. (2016).
Real or complex networks The use of complex numbers mathematical abstraction which allows diagonalizing translations, which are then represented by complex phases. It provides mathematical interpretation of filtering operations performed on real numbers. We show that a real network can still implement complex phase collapses.
In the first layer of a CNN, one can observe that filters are often oscillatory patterns with small supports, where some filters have nearly the same orientation and frequency but with a phase shifted by some a (Krizhevsky et al. 2012). We reproduce in Appendix [A]a figure from (Shang et al. (2016) which evidences this phenomenon. It shows that real filters may be arranged in groups (1a that
3

---

Published as a conference paper at ICLR 2022
can be written v,. Re(e (b) for single complex filter & and several phases a. A CNN with complex filters is thus a structured real-valued CNN, where several real filters (%a)a have been regrouped into single complex filter yb. This structure simplifies the mathematical interpretation of non-linearities by explicitly defining the phase, which is otherwise a hidden variable relating multiple filter outputs within each layer.
phase collapse is explicitly computed with a complex wavelet filter and a modulus. It can also be implicitly calculated by real-valued CNNs. Indeed, for any real-valued signal I, we have:
J * w = _
ReLU(( ( * v.) do. v.)da.
(4)
Furthermore, this integral well approximated by a sum over phases, allowing to compute complex moduli with real-valued filters and ReLUs without biases. See Appendix [poen a proof of eq. (4) and its approximation.
LEARNED SCATTERING NETWORK WITH PHASE COLLAPSES
This section introduces learned scattering transform, which a highly structured CNN architecture relying on phase collapses and reaching ResNet accuracy on the ImageNet (Russakovsky et al. 2015) and CIFAR-10 (Krizhessky 2009) datasets.
Scattering transform Theorem 1 proves that a modulus applied to the output of a complex wavelet filter produces a locally invariant descriptor. This descriptor can then subsampled, depending upon the filter's s bandwidth. We briefly review the scattering transform (Mallat) 2012 Bruna and Mallat 2013), which iterates phase collapses.
A scattering transform over scales is implemented with network of depth J, whose filters are specified by the choice of wavelet. Let Tn = x. For S j < J, the (j + 1) layer . is computed by applying phase collapse on the j-th layer zj. It is implemented by modulus which collapses the phases created by wavelet filtering operator W:
IJ+1
Wxyl
(5)
The operator W is defined with Morlet filters (Bruna and Mallat 2013). It has one low-pass filter 9o- and L zero-mean complex band-pass filters (ge)e. having angular direction L for 0 < / < L. thus transforms an input image zuu) into L + sub-band images which are subsampled by 2:
Wx(u, E)
geu).).
(6)
The cascade of j low-pass filters 9o with a final band-pass filter ge, each followed by subsampling. computes wavelet coefficients a scale 2². One can also modify the wavelet filtering W to compute intermediate scales 2 as explained in Appendix G The spatial subsampling then only computed every other layer, and the depth of the network becomes twice larger. Applying linear classifier on such a scattering transform gives good results on simple classification problems such as MNIST (LeCun et al. 2010). However, results are well below ResNet accuracy on CIFAR-10 and ImageNet, as shown in Table
Learned Scattering The prior work of zarka al. (202)) showed that a scattering transform can reach ResNet accuracy by incorporating learned 1 x convolutional operators and soft-thresholding non-linearities in-between wavelet filters. In contrast, we introduce a Learned Scattering architecture whose sole non-linearity is phase collapse. It shows that neither biases nor thresholdings are necessary to reach a high accuracy in image classification. A similar result had previously been obtained on image denoising (Mohan al. 2019).
The Learned Scattering (LScat) network inserts in eq. (5) a learned complex 1 x 1 convolutional operator Py which reduces the channel dimensionality of each layer z. before each phase collapse:
Tj+1 |WPjayl-
Similar architectures which separate space-mixing and channel-mixing operators had previously been studied in the context of basis expansion (Qiu al. 2018 Ulieny et al. 2019) to filter scattering
4

---

Published as a conference paper at ICLR 2022
Table 1: Error of linear classifiers applied scattering (Scat), learned scattering (L.Scat) and learned scattering with skip connections (+ skip), on CIFAR-10 and ImageNet. The last column gives the single-crop error ResNet-20 for CIFAR-10 and ResNet-18 for ImageNet, taken from
html
Scat
LScat
LScat skip
ResNet
CIFAR-10
Top-1 enror (%)
27.7
11.7
7.7
8.8
Top-5 error (%)
54.1
15.2
11.0
10.9
ImageNet
Top-1 error (%)
73.0
35.9
30.1
30.2
channels (Cotter and Kingsbury 2019). This separation is also a major feature of recent architectures such as Vision Transformers (Dosovitskiy et al. 2021) or MLP-Mixer (Tolstikhin et al., 2021).
Each Pj computes discriminative channels whose spatial variability is eliminated by the phase collapse operator. Their role is further discussed in Section []] Table T gives the accuracy of a linear classifier applied the last layer of this Learned Scattering. It provides an important improvement over scattering transform, but it does not yet reach the accuracy of ResNet-18.
Including the linear classifier, the architecture and J + 1 = 9 for CIFAR, by introducing intermediate scales. The number of channels of P, is the same as in a standard ResNet architecture (He al. 2016) and remains no larger than 512. More details are provided in Appendix G
total number of layers J + 12 for ImageNet
Skip-connections across moduli Equation imposes that phases are collapsed at each layer, after computing a wavelet transform. More flexibility is provided by adding a conpeconnettion which concatenates W/ Pj with modulus:
2++1 [WP,22/ WP
The skip-connection produces a cascade of convolutional filters W without non- con-linearities in in-between. The resulting convolutional operator W WW.... W is "wavelet packet" transform which generalizes the wavelet transform (Coifman and Wickerhauser 1992). Wavelet packets are obtained as the cascade of low-pass and band-past filters (ge)e- each followed by subsampling. Besides wavelets, wavelet packets include filters having larger spatial support and narrower Fourier bandwidth. A wavelet packet transform is then similar a local Fourier transform. Applying a modulus on such wavelet packet coefficients defines local spatial invariants over larger domains.
As discussed in Section 2 image classes are usually invariant to local rather than global translations. Section
2 explains that phase collapse improves discriminability for image classes that are locally translation-ines invarriantn over the filter's support. Indeed, phases of wavelet coefficients are then uniformly distributed over [0, 2Tr], yielding zero-mean coefficients for all classes. At scales where there is local translation- invariance, these phases are no longer uniformly distributed, and they encode information about the spatial localization of features. Introducing skip-connection provides the flexibility to choose whether to eliminate phases at different scales or to propagate them up to the last layer. Indeed, the next to use only one of these. This adds some localization information, which appears to be important.
vaassation-invarianne
operator Pj+1 linearly combines W P, and W Pjæ, and may learn
Table shows that the skip-connection indeed improves classification accuracy. A linear classifier on this Learned Scattering reaches ResNet- 18 accuracy on CIFAR- 10 and ImageNet. It demonstrates that collapsing appropriate phases is sufficient obtain a high accuracy on large-scale classification problems. Learning is reduced to 1 convolutions (Pş)₃ across channels.
4 PHASE COLLAPSES VERSUS AMPLITUDE REDUCTIONS
We now compare phase collapses with amplitude reductions, which are non-linearities which preserve the phase and act the amplitude. We show that the accuracy of Learned Scattering network is considerably reduced when the phase collapse modulus is replaced by soft thttsshhshho with learned biases. This result remains true for other amplitude reductions and architectures.
5

---

Published as a conference paper at ICLR 2022
Thresholding and sparsity A complex soft soft-thresholding reduces the amplitude of its input
|zle" by b while preserving the phase: pb(2) ReLU([=] - b) e Similarly to its real counterpart, is obtained as the proximal operator of the complex modulus (Yang et al. 2012):
1 _ 1Iu 21².
(b(2) = arg min bl/l
(9)
#EC
Soft-thresholdings and moduli have opposite properties, since soft-thresholdings preserve the phase while attenuating the amplitude, whereas moduli preserve the amplitude while eliminating the phase. In contrast, ReLUs with biases are more general non-linearities which can act both phase and amplitude. This is best illustrated over R where the phase is replaced by the sign, through the even-odd decomposition. If R and b 0, then the even part of ReLU( b) is ReLU([==] b), which is an absolute value with a dead-zone [-b, bj. When b = 0, it becomes an absolute value |z|- The odd part is soft-thresholding pb(z) sign( (2) ReLU[[2] 6). Over C, similar result can be obtained through the decomposition into phase harmonics (Mallat et al. 2019).
We have explained how phase collapses can improve the classification accuracy of locally stationary processes by separating class means E| Xy * x/ In contrast, since the phase of X,, z is uniformly distributed for such processes, then it is also true of Pb (X, * .²). This implies that IE [X +)] = for all b. Class means of locally stationary processes are thus not separated by a thresholding.
When class means E[X,, y bb] are separated, a soft-thresholding of X, L t/ may however improve classification accuracy. If X,,, * ip is sparse, then a soft thresholding Ps (X,, v) reduces the within- class variance (Donoho and Johnstone 1994; Zarka et al., (2021). Coefficients below the threshold may be assimilated to unnecessary "clutter" which is set to O. To improve classification, convolutional filters must then produce high-amplitude coefficients corresponding discriminative "features".
Phase collapses versus amplitude reductions A Learned Scattering with phase collapses pre- serves the amplitudes of wavelet coefficients and eliminates their phases. On the opposite, one may use a non-linearity which preserves the phases of wavelet coefficients but attenuates their amplitudes, such as a noft-thresholding. We show that such non-linearities considerably degrade the classification accuracy compared to phase collapses.
Several previous works made the hypothesis that sparsifying neural responses with thresholdings is major mechanism for improving classification accuracy (Sun et al., 2018 Sulam et al.) 2018) 2019: Mahdizadehaghdam et al. 2019 Zarka et al. 2020; (002T) The dimensionality of sparse representations then be reduced with random filters which implement a form of compressed sensing (Donhho| 2006 Candes et al., (2066). The interpretation of CNNs as compressed sensing machines with random filters has been studied (Giryes al. 2015), but it never led classification results close to e.g. ResNet accuracy.
To test this hypothesis, replace the modulus non-linearity in the Learned Scattering architecture with thresholdings, or more general phase-preservin preserving non-linearities. Learned Amplitude Reduction Scattering applies non- -on-eiiny p(z) which preserves the phases of wavelet coefficients v = I2 p(z) eeP p((=)). Without skip-connections, each layer T++1 is computed from 2, by:
xJ+1 = p(WP,3,))
(10)
and with skip- conneconnett:ons::
xj+1 [(WWP,x,), W WPJ.,
(11)
soft-thresholding is defined by p(|=1) ReLU([== b) for some threshold b. We also define an amplitude hyperbolic tangent p([2)) = - c + -11), an amplitude sigmoid as p(lz)) (1 + e log [=| -) and amplitude soft-sign p(l=)) |=l/(1 + |*]). The soft- thresholding and sigmoid parameters a and are learned for each layer and each channel.
We evaluate the classification performance of Learned Amplitude Reduction Scattering CIFAR- 10, by applying a linear classifier on the last layer. Classification results are given in Table 2 for different amplitude reductions, with without skip-connections. Learned Amplitude Reduction Scatterings yield much larger errors than a Learned Scattering with phase collapses. Without skip- connections, they are even above scattering transform, which also uses phase collapses but does not.

---

Published as a conference paper at ICLR 2022
Table 2:
Top-1 error (in %) CIFAR-10 with linear classifier applied to Scattering network (Scat) and several Learned Scattering networks (LScat) with several non-linearities. They include modulus (Mod), an amplitude soft-thresholding (Thresh), an amplitude hyperbolic tangent (ATanh), an amplitude sigmoid (ASigmoid), and an amplitude Soft-sign (ASign).
LScat
Scat
Mod
AThresh
ATanh
ASigmoid
ASign
Without skip
27.7
11.7
36.7
40.7
38.5
39.9
With skip
7.7
22.5
19.2
17.0
19.5
have learned x 1 convolutional projections (P)) It demonstrates that high accuracies result from phase collapses without biases, as opposed to amplitude reduction operators including thresholdings, which learn bias parameters. Similar experiments in the real domain with standard ResNet- architecture on the ImageNet dataset can be found in Appendix B
ReLUs with biases Most CNNs, including ResNets, use ReLUs with biases. ReLU with bias simultaneously affects the sign and the amplitude of its real input. Over complex numbers, it amounts to transforming the phase and the amplitude. These numerical experiments show that accuracy improvements result from acting the sign phase rather than the amplitude. Furthermore, this can be constrained to collapsing the phase of wavelet coefficients while preserving their amplitude.
Several CNN architectures have demonstrated good classification accuracy with iterated thresh- olding algorithms, which increase sparsity. However, all these architecture also modified the sign of coefficients by computing non-negative sparse codes (Sun al. 2018 Sulam al. [0188] Mahdizade haghdam et al. 2219)) or with additional ReLU or modulus layers (Zarka et al. 2020; 2021). It seems that it the sign or phase collapse of these non- con-linearities which responsible for good classification accuracies, as opposed to the calculation of sparse codes through iterated amplitude reductions.
ITERATING PHASE COLLAPSES AND AMPLITUDE REDUCTIONS
We now provide theoretical justification to the above numerical results in simplified mathematical frameworks. This section studies the behavior of phase collapses and amplitude reductions when they are iterated over several layers. It shows that phase collapses benefit from iterations over multiple layers, whereas there is no significant gain in performance when iterating amplitude reductions.
5.1 ITERATED PHASE COLLAPSES
We explain the role of iterated phase collapses with multiple filters at each layer. Classification accuracy is improved through the creation of additional dimensions to separate class means. The learned projectors (Py), are optimized for this separation.
We consider the classification of stationary processes X, E Rdd corresponding to different image classes indexed by y. Given realization of X and because of stationarity, the optimal linear clas- sifier is calculated from the empirical mean 1/d æ(u). computes an optimal linear estimation of E [x,,, [w) My. If all classes have the same mean Hby = μ. then all linear classifiers fail.
As explained in Section linear classification can be improved by computing (|x * v/s)) for some wavelet filters (z/)).- These phase collapses create additional directions with non-zero means which may separate the classes. If X, is stationary, then Xx, V remains stationary for any V'k- An optimal linear classifier applied to (Jx (1)) is thus obtained by linear combination of all empirical means E1/dE, |ar (ux(u)))s. They proportional to the e norm |æ 1/ull2- which is measure of sparsity x: * eha.
If linear classification on (|æ (u)))) fails, it reveals that the means E[|X, th (w)) My. are not sufficiently different. Separation can be improved by considering the spatial variations of X, vhg (u)l for different y. These variations be revealed by a phase collapse set of

---

Published as a conference paper at ICLR 2022
wavelet filters d'y', which computes (|] (|]x+ * Dx.'" This phase collapse iteration is the principle used by scattering transforms to discriminate textures (Bruna and Mallat 2013 Sifre and Mallat 2013): each successive phase collapse creates additional directions to separate class means.
However, this may still not be sufficient to separate class means. More discriminant statistical properties may be obtained by linearly combining (|æ * t/x 1)x across & before applying a new filter Vxz In a Learned Scattering with phase collapse, this is done with linear projector P1 across the channel indices k. before computing a convolution with the next filter V/. The x 1 operator Pi is optimized improve the linear classification accuracy. It amounts learning weights wa such that E. E "/. X whal [ is as different as possible for different y. Because these are proportional to the Es norms ||Ex wx |Ir * rh&l // I| means that the images Es Wg|2 1//l * the have different sparsity levels depending upon the class y of x. The weights (wx))e of P3 can thus be interpreted as features along channels providing different sparsifications for different classes. A Learned Scattering network learns such P3 each scale j.
5.2 ITERATED AMPLITUDE REDUCTIONS
Sparse representations and amplitude reduction algorithms may improve linear classification by reducing the variance of class mean estimations, which can be interpreted as clutter removal. Such approaches are studied in Zarka et al.] (202T) by modeling the clutter as an additive white noise. Although a single thresholding step may improve linear classification, we show that iterating more than one thresholding does not improve the classification accuracy, if no phase collapses are inserted.
To understand these properties, we consider the discrimination of classes X, for which class means ((xx)) My are all different. If there exists y such that I|14 - / is small, then the class y can still be discriminated from y' if we can estimate B(Xy) sufficiently accurately from single realization of X,, This is a mean estimation problem. Suppose that X,, My + + o²) is contaminated with Gaussian white noise, where the noise models some clutter. Suppose also that there exists línear orthogonal operator D such that Djr is sparse for every and hence has its energy concentrated in few non-zero coefficients. Such a D may be computed by minimizing the expected e norm Σ, | |X The estimation of Bty can be improved with a soft-thresholding estimator (Donoho and Johnstone 1994), which sets to zero all coefficients below a threshold b proportional to a. It amounts to computing P, where Ps is a soft- sot-thhrssooldng.
However, we explain below why this approach cannot be further iterated without inserting phase collapses. The reason that a sparse representation Pr (Dx) concentrates its entropy in the phases of the coefficients, rather than their amplitude. We then show that such processes cannot be further sparsified, which means that a second thresholding P. (D' "po (Dx)) will not reduce further the variance of class mean estimators. This entails that model of within-class variability relying on amplitude reductions cannot the sole mechanism behind the performance of deep networks.
Iterating amplitude reductions may however be useful if it is alternated with another iineaiiyy which partly or fully collapses phases. Reducing the entropy of the phases of pb (Dx) allows P3 D' to further sparsify the process and hence further reduce the within-class variability. As mentioned in Section this is the case for previous work which used iterated sparsification operators (Sun) et al. 2018 8 Sulam et al., 2018 Mahdizadehaghdann
al. 2019). Indeed, these networks compute non neg-negative sparse codes where sparsity is enforced with a ReLU, which acts both on phases and amplitudes. Our results shows that the benefit of iterating non-negative sparse coding comes from the sign collapse due to the non-negativity constraint.
We now qualitatively demonstrate these claims with two theorems. We first show that finding the sparsest representation of random process (i.e., minimizing its L norm) is the same as maximizing a lower bound on the entropy of its phases.
Theorem 2.
Let X denote random vector in Cd with a probability density p. Let H(X) be the entropy of X with respect to the Lebesgue measure:
H(X) / p(x) logp(z) dr.
8

---

Published as a conference paper at ICLR 2022
D Ud) is unitary operator then:
H H(((DX) (xXI)
2 H(X) - d 2d 2d (
where c E [0, 2T)" (resp. |xX] E R4) is the random process of the entry-wise phases (resp. moduli) of D.X
The proof is in Appendix This theorem gives a lower-bound on the conditional entropy of the phases of DX with decreasing function of the expected e² norm of DX. Minimizing over D this expected norm amounts to maximizing the lower bound on H (Dx() IDX)). An extreme situation arises when this entropy reaches its maximal value d (2T). In this case, the phase sp has maximum-entropy distribution and is therefore uniformly distributed in [0, 2TT] Moreover, in this extreme case (D.X) is independent from [X], since its conditional distribution does not depend on [DX] Such statistical properties have previously been observed on wavelet coefficients of natural images (Rao et al., 2001). where the wavelet transform seems to be a nearly optimal sparsifying unitary dictionary.
The second theorem considers the extreme case of random process whose phases are conditionally independent and uniform. It proves that such a process cannot be significantly sparsified with change of basis.
Theorem 3. Assume that (Px((DX) is uniformly distributed in [0, 2x] and independent from (Dx())) Then there exists a constant Ca > O which depends on the dimension d such that for any D' E Ud).
P[IlD'
C_E(II(((D))I1
The proof is Appendix F This theorem shows that random processes with conditionally in- dependent and uniform phases have an e norm which cannot be significantly decreased by any unitary transformation. Numerical evaluations suggest that the constant C₃ may be chosen to be / z 0.886, independently of the dimension d. This constant arises as the value of E[[Z]] when Z is complex normal random variable with =
These two theorems explain qualitatively that linear classification on Po (Dx) cannot be improved by another thresholding that would take advantage of another sparsification operator. Indeed, Theorem shows that if Ps (xr) sparse, then its phases have random fluctuations of high entropy. Theorem indicates that such random phases prevent further sparsification of Po( with some linear operator D'. Applying second thresholding Py (D' D'pp(( (Dz)) thus cannot significantly reduce the variance of class mean estimators.
CONCLUSION
This paper studies the improvement of linear separability for image classification in deep convolutional networks. We show that it mostly relies on a phase collapse phenomenon. Eliminating the phase of wavelet coefficients improves the separation of class means. We introduced a Learned Scattering network with wavelet phase collapses and learned 1 x 1 convolutional filters (Pz), which reaches ResNet accuracy. The learned x operators (P,) enhance discriminability by computing channels that have different levels of sparsity for different classes.
When class means are separated, thresholding non-linen an-linearities can improve classification by reducing the variance of class mean estimators. When used alone, the classification performance is poor over complex datasets such as ImageNet or CIFAR-10, because class means are not sufficiently separated. Furthermore, the iteration of thresholdings sparsification operators requires intermediary phase collapses.
These results show that linear separation of classes result from acting on the sign phase of network coefficients rather than their amplitude. Furthermore, this can be constrained to collapsing the phase of wavelet coefficients while preserving their amplitude. The elimination of spatial variability with phase collapses thus both necessary and sufficient to linearly separate classes on complex image datasets.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The code reproduce the experiments of the paper is available hatps::/giitub..om FlorentinGuthy wwwnnnthhhhaaeeehc Phase All experimental details and hyperparameters are also provided in Appendix G
ACKNOWLEDGMENTS
This work was supported by grant from the PRAIRIE 31A Institute of the French ANR- 19-P3IA- 0001 program. We would like to thank the Scientific Computing Core at the Flatiron Institute for the of their computing resources. We also thank Antoine Brochard, Brice Ménard and Rudy Morel for heplful comments.
REFERENCES
E. Oyallon. Analyzing and Introducing Structures in Deep Convolutional Neural Networks. Theses, Paris Sciences et Lettres October 2017.
V. Papyan, X. Y. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
X. Sun, N. M. Nasrabadi, and T. D. Tran. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference Image Processing (ICIP), pages 346-350, 2018.
J. Sulam, V. Papyan, Y. Romano, and M. Flad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing, 66115):400044104, 2018.
Sulam, A. Aberdam, A. Beck, and M. Elad. On multi-layer basis pursuit, efficient algorithms and convolutional neural networks. IEEE Transactions on Patlern Analysis and Muchine Intelligence, 2019.
Mahdizadehaghdam, A. Panahi, H. Krim, and L. Dai. Deep dictionary learning: A parametric network approach. IEEE Transactions on Image Processing,
4802, Oct 2019.
J. Zarka, L. Thiry, T. Angles, and S. Mallat. Deep network classification by scattering and homotopy dictionary learning. In International Conference on Learning Representations, ICLR, 2020.
Zarka, F. Guth, and S. Mallat. Separation and concentration in deep networks. In International Conference on Learning Representations, ICLR, 2021.
T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning: data mining, inference and prediction. Springer, edition, 2009.
J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 15(8):1872-1886 2013.
M. B. Priestley. Evolutionary spectra and non-stationary processes. Journal of the Royal Statis- fical Society: Series B (Methodological), 27(2):204-229, 1965. doi: http::/dhoi.or//0.11111
j,2517-616. 19t.:661...995.1.144999 abs has/10.1111/j.22
URL httph:tttt/sttt
wbbriiieeeirrrriiii.. www.rrywwwll.ccooooome doi/
6161. 1965 tb01488.x
M. Tygert, J. Bruna, S. Chintala, Y. LeCun, S. Piantino, and Szlam. A mathematical motivation for complex-valued convolutional networks. Neural computation, 28(5):815-877 2016.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, NeurlPS, pages 1097-1105. 2012.
W. Shang, K. Sohn, D. Almeida, and H. Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units, 2016.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bemsstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal Computer Vision (ICV), 115(3):211 1-252, 2015.
10

---

Published as a conference paper at ICLR 2022
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65(10): 1331-1398, 2012.
Y. LeCun, C. Cortes, and C.J. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. http://aan.lleuu..com wwt../////x///iistt.. 2, 2010.
S. Mohan, Z.
Kadkhodaie, E. P. Simoncelli, and C. Permandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. International Conference on Learning Representations, 2019.
Q. Qiu, X. Cheng, R. Calderbank, and G. Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. International Conference Machine Learning. 2018.
M. Ulieny, V. Krylov, and Dahyot. Harmonic networks for image classification. In Proceedings of the British Machine Vision Conference, Sep. 2019.
Cotter and N. G. Kingsbury. A learnable scatternet: Locally invariant convolutional layers. In 2019 IEEE International Conference Image Processing, ICIP, pages 350-354. IEEE, 2019.
A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, Yung. A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. arXiv preprint
2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
R.R. Coifman and M.V. Wickerhauser. Entropy-based algorithms for best basis selection. IEEE Transactions on Information Theory, 38(2):113-718, 1992. doi: 10. 109/18. 119732.
Z. Yang, C. Zhang, and L. Xie. On phase transition of compressed sensing in the complex domain. IEEE Signal Processing Letters, 19(1):47-50, Jan 2012. ISSN 1558-2361. 2177496. URL http:: //tp:://dxx.. doi . www.ooggg..o........ 1109/Is1 2011 2177496
doi: 10.1109/lsp.211.
S. Mallat, S. Zhang, and G. Rochette. Phase harmonic correlations and convolutional neural networks.
Information and Inference: A Journal of the IMA, 9(3):721-777, 11 2019.
D. L. Donoho and I. M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81(3): 425-455, 09 1994.
D. L. Donoho. Compressed sensing. IEEE Trunsactions information theory, 52(4)) 2006.
E. J. Candes, J. K. Romberg, and T. Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: Journal Issued by the Courant Institute of Mathematical Sciences, 59(8): 1203-1223, 2006.
R. Giryes, G. Sapiro, and A. M. Bronstein. Deep neural networks with random gaussian weights: 2015. URL http:t/t//aaxx/.ogg/
universal classification strategy? CoRR, abs/1504. 08291
L. Sifrr and S. Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference computer vision and patlern recognition, pages 1233- 1240, 2013.
R. Rao, B. Olshausen, M. Lewicki. M. Wainwright, O. Schwartz, and E. P. Simoncelli. Natural image statistics and divisive normalization: Modeling nonlincarities and adaptation in cortical neurons. Statistical Theories of the Brain, 01 2001.

---

Published as a conference paper at ICLR 2022
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, page 448-456, 2015.
M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzz, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. Andén, E. Belilovsky. J. Bruna, V. Lostanlen, M. J. Hirn, E. Oyallon, S. Zhang, C. E. Cella, and M. Eickenberg. Kymatio: Scattering transforms in python. Journal of Machine Learning Research, (1 2020.
12

---

Published as a conference paper at ICLR 2022
A PAIRED ALEXNET FILTERS
Section 2 explains that real networks can still implement phase collapses. This is done with several
real filters 1y Re(e Re(e-*00) (b) which correspond to several phases a of the same complex filter ab. Shang al. (2016) showed that the filters in e.g. the first layer of AlexNet (Krizhevsky et al. 2012)
can indeed be grouped in such way. For the sake of completeness, we reproduce Figure 2
figure from Shang et al. (2016). This suggests that real-valued networks may indeed implement phase collapses using eq.
Figure 2: First-layer filters from AlexNet (Krizhevsky et al. 2012). They have been paired so that they approximately correspond two different phases of the same complex filter vb. Figure reproduced from Shang et al. (2016).
B PHASE COLLAPSE VERSUS AMPLITUDE REDUCTION WITH RESNET
We now evaluate the classification error of phase collapses and amplitude reduction non-linearities in the real domain. We use a standard ResNet-18 architecture without biases. We replace the ReLU non-linearity by an absolute value or sign collapse |a|| and several sign- prenrrrreervingggg (i.e., odd) non- linearities. They include soft-thresholding Pb(x) = (ign(x) ReLU
b). an hyperbolic tangent p(x) = (e" - {)(((" + e ") and soft-sign p(x) = //(11 (æl)). We do not report results for an amplitude sigmoid p(x:) sign(x)) + 12| because of optimization instabilities when leamiing the parameters and b.
Classification results on the ImageNet dataset are given in Table The error of bias-free ReLUs and sign collapses are comparable to standard ResNet- 88, and contirm that sign collapses are sufficient reach such accuracies. preserve the sign of network coefficients, is significantly worse. The conclusions of Section 4 thus still hold in the real domain and when the spatial filters are not constrained to be wavelets.
contrast, the performance of amplitude reduction non-linearities, which
Table 3: Classification errors on ImageNet of bias-free ResNet- 18 (BFResNet) architectures with several non-linearities. They include a ReLUU, absolute value which performs sign collapses (Abs), a soft-thresholding (Thresh), hyperbolic tangent (Tanh), and soft-sign (Sign). They are compared to the original ResNet-18 architecture, which uses ReLU and learns biases.
BFResNet
ResNet
ReLUU
Abs
Thresh
Tanh
Sign
Top-5 error (%)
10.9
12.3
13.9
25.7
22.4
24.2
Top-1 error
30.2
32.6
35.3
50.0
44.6
49.3
13

---

Published as a conference paper at ICLR 2022
C PROOF OF THEOREM 1
We have:
|1, * 10 E -i ())0)) = 1 (v- -e 15 >
by covariance of convolution,
1 i6 7 1 111211>11
by Young's inequality,
and then:
11// -6
1 111 7
|| (w-(w) - (w)
by Plancherel,
(2xx)²
KoiT B(w) - (()) dw
since vu(x)) 10(u 7),
(2T)²
1-
le - 15 (
(2x)²
|-m
$ (2
1( () [
since x E IR + e" 1-Lipschitz,
|-
w ((w)
by Cauchy-Sehwarc
(2 2
|-),1]
= 1
which leads to the desired result of eq. 3
|| * 1 - T (x +) s So |
D PROOF OF EQUATION 4
We have:
ReLU(x vha) ReLU(x * Re(e- = RRLU(Re(E -ia xxxx x)),,)
since x is real. By writing: * ab = |æ w (o(()))) where fpx * vb) is the phase of x * v. this leads
to:
ReLU(Re(e "xxx ReLU whee (((())))))))
ReLU([ +0 cos(p(x * 1) - -a))
|z +b (esUUccs(ppp v) a)),
since ReLU activation is cesitie-hhomogeneous of degree 1. Thus:
1
ReLU(x va)da
I dbl ReLU(cos(({(r 1b) a))da
7 p(x+))
2 Ja 1/1
ReLU(cos(-a)dda with a change of variable,
p(zxv)
or 161
Rel.((co((a)dda
since 2n periodic and even,
7/2 cos(a)dax
Ja vbl
2
-/2
|oc wb]
For C, we have |2| / Re(2)) [m) [Re(=)] + [Im(=)] in the following sense:
1
Re( 2) + [m)))
< |a| < [Re(z)] + [lm(2)].
2
14

---

Published as a conference paper at ICLR 2022
We write:
[Re(z) ReLU(Re(z)) + ReLU( Re(=)),
Im [z) = ReLU((m(=)) ReLU(- Im(z)).
and then, using Im(x) = Re(e"*/22) and ed = =-1:
2
|a| = ReLU(R(()) RcLU(Ro(e -e,=,"")))) + ReL.U(Roee 18/2 2)) RcL.U((((e-1222)))
Finally,
[ ReLU(s da
x 1/
E
ReLU(Re(x * vh,))
aet oE(-n/22U./22.1)
which shows that the integral can be well approximated with sum of phases of the complex filter th.
E PROOF OF THEOREM 2
We first use the chain rule for the entropy:
H (v(Dx)) |xX) H(IDX),+(DX) H(IDX).
The first term is rewritten with change of variable:
H(IDX),,((X)) = H(DX)
Ellog (DX)
k=1
H()
Bllog (DX).ll
1 is unitary and hence (det(D)
HX) dI
log
|XXII
]
by concavity,
H(X) d log
[[I
by concavity.
The second term is bounded using the fact that the exponential distribution (()) is the maximum- entropy distribution on R2 with mean :=
H(DXI) < Σ H(((X))1)
k=1
log(eR (DX)a)
S d
[xII) by concavity.
d
Combining both incqualities and rearranging terms yields the stated bound:
H(((D)) |XXI) 2 H(X) - d -2dlog [[IDXX11)
F PROOF OF THEOREM 3
We begin with the following lemma:
Lemma 1. Let (f.....,6) ..)aa be i.i.d. uniform random variables in [0, 2n). Then there exists a constant C2 > such that for all (p Pa) E IRd then:
E
Phe s0x
2 Ca
Σ
Pk.
A=1
15

---

Published as a conference paper at ICLR 2022
This is proved by observing that the left-hand side is R One indeed verify that it is positive definite, homogeneous and satisfies the triangle inequality. Since all norms on R equivalent, there exists a constant Ca > 0 such that:
Pre 1
2 Ca
Σ 1=1
Ph.
for all ( Pa) E Ra.
Going back to the proof of Theorem
and letting X' = -Po(DX), we then have:
d
E 1'X''1 [x']]
=
E
m.
X
|X
m=11
k=1
Cd Σ
D mm F 2
by the above lemma,
Cd Σ
D'n |
by concavity, because
Σ
D'm,el 2 1,
Rm
km
x1ll
because
Σ
D kl =1.
Taking the expectation finishes the proof:
[II'''
2 (I
(12)
G EXPERIMENTAL DETAILS
Channel operators In all experiments we set Po = Id, and factorize the classifier with an additional complex convolutional operator Py, which reduces the dimension before all channels and positions are linearly combined. The architectures implemented are thus also written as 1 PypW. where p is the non-lincarity. Each operator (P)) is preceded by standardization. It sets the complex mean
[[=] of every channel to zero, and the real variance o² E| 2 of every channel to one. This similar to a complex 2D batch- natch-oommalizttinn layer (loffe and Szegedy) (2015), but without learned affine parameters. Each operator (P))ısss is additionally followed by spatial divisive normalization (Rao et al. 200T), similarly to the local response normalization of Krizhevsky et al. (2012). It sets the norm across channels of each spatial position one. The sizes of the (P)) are specified in Table
The total numbers of parameters for each architecture specified in Table Learned Scattering with phase collapse have large number of parameters compared to ResNet, despite the comparable width. This is because the predefined wavelet operator W expands the dimension by factor of L 1, which means that the input dimension of the learned (P₁), is higher than in ResNet. The coipeconnettion further increases this input dimension by factor of 2.
Table 4: Number cj of complex output channels of P, 15jj J. The total number of projectors is = for CIFAR and 11 for ImageNet.
j
2
3
5
6
8
9
10
11
CIFAR-10
C
64
128
256
512
512
512
512
512
ImageNet
C
32
64
64
128
256
512
512
512
512
512
256
16

---

Published as a conference paper at ICLR 2022
Table 5: Number of real parameters (in millions) of Learned Scattering network architectures. A complex parameter is counted as two real parameters.
PCScat
PCScat skip
ResNet
CIFAR-10
41.6
83.1
0.27
ImageNet
36.0
62.8
11.7
Spatial filters
We use elongated Morlet filters for the L complex band-pass filters (ge)e which are rotated versions of mother wavelet gi ge(u) = g(r-nt/L with the rotation by angle 6. The mother wavelet g defined as:
o²
00 (o o 2
g(u) = 2
-Ke -u.hu/2
with E =
(13)
2
Its parameters are its center frequency & = ((3T/44/22 /4//2" 0), its bandwidth o = 1.25 x 2 and its slant 0.5, where 2" designates the scale of the band hand-pass filter and is be adjusted.
is rotated along L = 8 angles for Imagenet and L = 4 angles for CIFAR: 6, = ( L) IsesL. The (ge)e are then discretized for numerical computations, and K is adjusted so that they have a zero mean.
Finally, we use for the low frequency gu Gaussian window:
20llu //2
(o(u)
2m
The filters are implemented with the Kymatio package (Andreux et 2020).
Intermediate scales 222 are obtained by applying a subsampling by 2 after each block of layers. This introduces intermediate scales and generates wavelet filterbank with 2 scales per octave: the filters are designed so that when j low-pass filters and one band-pass filter are cascaded, with a subsampling every 2 layers, the scale of the resulting wavelet is 2//2
Each block comprises in its first layer a low-frequency filter dù with 7 = -1/2 and band-pass filters with y 0. In the second layer, we use the same low-frequency filter gà gó with 7 -1/2. The band-pass filters o are obtained with parameters g ().
1.25 2/3, and s' v0.2.
For CIFAR experiments, the J = 8 layers are grouped in successive blocks of layers. For ImageNet experiments, the first layer consists of band-pass, elongated Morlet filters ge and low-pass Gaussian window with y 0, followed by a subsampling of 2. The 10 following layers are grouped in 5 blocks of 2 layers.
Optimization We use the optimizer SGD with an initial learning rate of 0.01, a momentum of 0.9, a weight decay of 0.0001, and a batch size of 128. The classifier preceded by a 2D batch- normalization layer. We traditional data augmentation: horizontal flips and random crops for CIFAR, random resized crops of size 224 and horizontal flips for ImageNet. Classification error on ImageNet validation set is computed on a single center crop of size 224. On CIFAR, training lasts for 300 epochs and the learning rate is divided by 10 every 70 epochs. On ImageNet, training lasts for 150 epochs and the learning rate is divided 10 every 45 epochs. All experiments ran during the preparation of this paper, including preliminary ones, required around 10k 32GB NVIDIA V100 GPU-hours.
17


Reviews:
[{"review_id": "UC9ohhfPV1b", "reviewer": "Reviewer_LHxR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength:\nThis paper is a nice extension to the related works [Zarka et al (2020, 2021), Ulicny et al.(2019)]. The main contribution (and difference to the literature) is that they hypothesize classification performance mostly results from iterated phase collapses. They supported their hypothesis  (necessary and sufficient) by first explaining the performance of iterated phase collapses by showing that it progressively improves linear discriminability. Secondly, they replaced the phase collapse with a soft-thresholding and showed that this considerably decreases the classification accuracy.\n\n\n### Correctness: \nThe core assumptions seem correct to me, e.g. the images classes are often locally invariant to translation. Small translations are used to be approximated by a phase shift. The empirical results are correct, the theoretical results seem reasonable, but I did not fully examine the proofs in the appendix or reproducibility of the code in the supplementary materials.\n\n### Weakness: \nGenerally, the writing quality is not good and hard to read, e.g. long sentences. Page 3, before the (Krizhevsky et al., 2012) reference, I am not sure if filters are usually localized oscillatory patterns or filters usually localize oscillatory patterns? Sometimes the text is not scientifically addressed, e.g. page 4: $\\mathbf{E}[X_y * \\psi] = 0$ then $\\mathbf{E}[\\rho_b(X_y * \\psi)] $ is \"usually\" close to zero. What does usually mean here? Formula4: \"One can verify...\"? It is now clear to me how. And \"This integral is already well approximated by a sum over 4 phases\"?  MNIST probably needs to be cited for \"The MNIST database of handwritten digits (1998)\". The operator \"$*$\" is not defined. It is also not clear to me why the authors used skip-connection-based neural networks? What about CNNs without skip connections? \n\n\n\n>update after rebuttal: I would like to thank the reviewers for addressing my concerns. However, I found it difficult to find the changes in the rebuttal version. I now see the Results with/without skip connections in Table1 and Table2. The text is still hard to read and follow (this is also addressed by other reviewers). As promised by the authors the text would be improved (e.g. shorter sentences). In conclusion, I would increase my score.", "summary_of_the_paper": "This paper studies within-class variability which reduces along the layers of deep neural networks. They mainly question the effect of sparsity and soft-thresholding introduced by ReLU. They show that these classification improvements by eliminating spatial within-class variabilities rather come from a phase collapse, which eliminates the phase of network coefficients. Eliminating the phase of zero-mean filters improves the separation of class means, hence increase in classification accuracy. They introduced a complex-valued neural network in which spatial filters are defined as complex multiscale wavelets and learning is reduced to $1 \\times 1$ complex filters across channels. Their results show that such a network is able to reach ResNet-18 performance on CIFAR10 and ImageNet.\n\n\n\n\n", "summary_of_the_review": "The paper mainly questions the effect of sparsity and soft-thresholding introduced by ReLU. The authors show that the classification improvements by eliminating spatial within-class variabilities come from a phase collapse. This counts as a contribution compared to related works. The hypothesis is clearly defined and supported using theory (and assumptions) and experiments. However, there are pitfalls that need to be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G4HNUTrRuMM", "reviewer": "Reviewer_GfRM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-The paper is interesting in that it presents theoretical results followed by solid experimental results related to the theoretical results demonstrating that the phase of the signal can be removed while improving the classification performance of DNNs\n-The paper is for the most part well written\n-The authors present good results on CIFAR10 and Imagenet. While the results do not seem state of the art I think they are sufficiently good in support of the authors' arguments\n\nWeaknesses: \n-Figure 1:The authors assume color printers are used (orange/blue). I suggest they use bold/solid, dashed lines to distinguish between the different steps\n-Page 2: clarify why both directions (necessary & sufficient) hold. I see how the sufficient condition holds but it is not clear how the necessary direction emerges. Does it make sense to phrase this as a conjecture?\n-Eq 4: It may help many readers to derive this in the appendix, it should be a quick derivation\n-The ResNet results while good are not state of the art. If SOA feasible using the PCS framework? If yes why wasn't a SOA presented in the paper. If not, what do the authors believe hinders achieving SOA?\n-Page 3: Clarify better when the random vector X_y represents. Does it represent the features used by the classifier? In that case I do not see how the classes are linearly separable if E[X_y] are all different. Do you mean this is a sufficient or necessary condition? For example if X_1, X_2 are 2D vectors, each representing two concentric circles. The two sets of features represent different classes but the means are the same (the circle center). Obviously we can have E[X_y] with the same mean but represent different classes. Limiting ourselves to linear separability makes the problem trivial. I suggest the author rewrite this section to clarify what they mean\nPage 3: The authors make a lot of assumptions that X_y are stationary. How important is this assumption in their results? Do they think perhaps that a significant set of problems, such as those often dealt with by RNNs, are not well modelled as stationary?", "summary_of_the_paper": "The authors present some theoretical results, followed by some experiments, in support of their argument that the linear separability for image classification in deep convolutional neural networks mostly relies on a phase collapse phenomenon. By eliminating the phase of zero mean filters they improve the separation of class means.They present a Phase Collapse Scattering network and demonstrate Resnet-like accuracy. The authors argue that phase collapse are both necessary and sufficient to discriminate class means on complex datasets.", "summary_of_the_review": "Overall I found this an interesting paper. It is well written, and has a fairly solid experimental section in support of some not so well known theoretical ideas as to why deep neural nets achieve good classification\nOverall here is my summarized assessment:\n\noriginality: The authors present some theoretical results to justify their experimental framework. The paper is not just blindly attempting different DNN architectures. From this point of view I consider this a fairly original piece of work\n\nquality: I believe the exposition and experimental results are of sufficient quality for ICLR. The paper is in general well written, though as I indicate elsewhere certain parts need to be improved. \n\nclarity: I mentioned above in the \"Weaknesses\" section some clarity issues that I noticed. The authors should address these issues. I was ambivalent as to whether the paper should be rejected because of these clarity issues, but since they can all be addressed I think I decided to be a bit lenient. But for the final ICLR submission they should be addressed.\n\nsignificance: Unless I missed it, are the authors providing source code? The paper's significance and clarity will be improved if source code is provided, to make the results reproducible", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5XiYyI-7Cm-", "reviewer": "Reviewer_oXQJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper addresses some interesting questions and it is somewhat interesting that a network with a fixed wavelet basis can achieve reasonable performance on ImageNet classification. I also think the paper provides quite a bit of theory to support some of the claims.\n\nHowever I feel the paper suffers from several weaknesses:\n\n* It is not clear to what is the purpose of the proposed model? if it is to show that such a model can indeed be comparable to current architectures then OK, but it does not go beyond that. I don't feel there is a lot \"actionable\" conclusions here nor that we learned anything fundamental about how these models work.\n\n* There is almost no discussion as to what the learnable operators P do - what happens if they are not used? how do they relate to other parts of the model and, more importantly, to what ResNets learn?\n\n* Why are the resulting networks so parameter heavy? they have many more parameters when compared to the corresponding ResNets and perform worse. What can be done?\n\n* How does the proposed model scale? with the number of layers? number of activations at each layer? etc.", "summary_of_the_paper": "This paper proposes that using phase modulus operators instead of thresholding non-linearites is beneficial to classification performance in the case of scattering like networks. The authors show that such a network with some learnable operators can come close to performance of small ResNets when they use the modulo operators and performance degrades when using other non-linearieis. This is demonstrated on ImageNet and CIFAR-10 and is accompanied with relevant analysis and theory. ", "summary_of_the_review": "I think this is a potentially interesting paper but I feel it leaves quite a bit to be desired. \n\npost author response update:\n\nI have read the other reviewers comments as well as the author responses - it would seem that I have been harsh in my scoring and this has been revised. I think this is a valuable contribution to our understanding of the field and a nice demonstration that fixed wavelet filters can, under the right circumstances, perform as well as learned networks. My confidence in the score still remains low as I feel was not the right person to review this.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1AZpZGVqhha", "reviewer": "Reviewer_3Vif", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are:\n- I find the novelty and the results, both theoretical and empirical, fascinating and thought provoking. Making a link between operators like ReLUs to complex numbers, their phases, and collapse is a great idea. In particular, I find the results in Table 2 very convincing.\n\n- The theoretical analysis is nice, and not to hard to follow, once one commits. It would be interesting to make more explicit the analysis for different types of nonlinearities. For instance, if sigmoid are soft-thresholding functions, I suppose the same is also for tanh. However, tanh tends to work better. Why is that the case, under this framework? What about other non-linear activations, like the swish function?\n\n- The results are generally strong and quite convincing, at least in ablation comparisons.\n\nThe weaknesses of the paper:\n- I found the writing involved, if not subpar for the quality of the paper. It is clear that the authors have a very good understanding of the area, however, they do not make it very easy for the average reader to understand their thought process, The sentences are not really immediately connected, that is there exist jumps, which sometimes can be inferred based on follow-up text, while other times they are simply hard to understand. It could be that for readers who are better versed in the literature, the text is easier. However, any paper should do a minimum effort to communicate the message clearly. Certainly, the introduction contains several forward references and that at least can be improved.\n\n- I found section 2 particularly hard to follow, as various concepts are being introduced at random points, without any clear structure. Almost ironically, I couldn’t find any direct definition of what comprises phase collapse, which made matters all that much harder. While it is sort of expected/intuitive what phase collapse could mean, given that the paper is on the theoretical/conceptual side of things, it makes it hard to follow the thought process. \n\nThe only ‘definition’ I found was that phase collapse eliminates the phase of a complex number with a modules just below eq (3). It is not clear if this a general definition, just a logical deduction based on fundamental concepts, or something that the paper introduces. In any case, it is also unclear why this holds. I assume it is because the modulus of the exponential is 1, so the phase that is introduced by the exponential is eliminated?\n\n- It is also unclear how equation (5) is derived. Again, is it a logical deduction based on the nature of the ReLU? Or is it derived based on the aforementioned definitions of filtering with wavelets and complex numbers? It should be immediately clear from the text what is expected to be background knowledge, what is derived knowledge, what is a simple deduction.\n\n- From a technical point of view, what I found unclear was what is the point of stacking ‘phase collapse’ operators, since one of them is supposed to remove all the phases from the feature, that is, it learns translation invariant features. From deep networks, I assume that deeper layers combined the wavelet activations from the previous layers. However, I find hard to reconcile this with ‘translation invariance’. For instance, a full connected layer is translation invariant, while a convolutional layer is translation equivariant. Does phase collapse then correspond to fully connected layers, and if yes, what is the point of stacking them? After all, with fully connected layers it is also of little added value to stack them, usually.\n\n- Complementing on the previous point, can it be that adding the skip connections is beneficial such that to make the operations also translation equivariant, thus not completely eliminate all phases and translation information? Something like that is analyzed in the bottom of page 5, but maybe this point can be furthered.\n\n- I find the motivation of the projection operators P a logical one in the context of deep networks, however, not really convincing in the context of phase collapse scattering networks. Do they contribute anything to the amplitude or the phase? What would be the benefit, considering that all phases are anyways immediately eliminated afterward? Do they help with having class means that are well separated perhaps? Maybe there can be a motivation that relates more to the main story of the paper.\n\n- Last, I would like to clarify that in my attempt to find out more about ‘phase collapse’ as a concept, I found out the arXiv version of the paper accidentally. I got a bit annoyed by that, as I believe it biases my opinion, and I do not find it fair against other papers.", "summary_of_the_paper": "This is a very interesting paper that is based on scattering networks, for which it shows that the so called ‘phase collapse’ leads to state-of-the-art results on par with modern architectures, like ResNet. To derive this, the paper shows that having neural networks on complex numbers is similar to a structure deep network (like with wavelet filters) on the reals. Phase collapse is then when there is an operation, like the modulus (absolute value function), which eliminates the phase from the complex number and maintains only the amplitude. All in all, the paper shows that maintaining the amplitude while eliminating the phase, is what brings the high accuracies, while the reverse (keeping the phase, eliminating the amplitude) yields terrible accuracies. As a corollary, I would say that the paper explains why non-linearities, like ReLUs, are so successful.", "summary_of_the_review": "Based on the novelty and the strong results, I vote for acceptance. I have one remark that I couldn’t completely understand and it would be nice if the authors could help me there. All in all, I find that I learned something from this paper, which I think is a great result for any publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes that the superior performance of modern convolutional networks is partly due to a phase collapse mechanism that eliminates spatial variability while ensuring linear class separation. To support their hypothesis, authors introduce a complex-valued convolutional network (called  Learned Scattering network) which includes a phase collapse on the output of its wavelet filters and show that such network has comparable performance to ResNets but its performance degrades if the phase collapse is replaced by a threshold operator.

Reviewers are all in agreement about the novelty and significance of the work. They also find the empirical results compelling. The main weakness of this work which was highlighted by all reviewers is clarity. The paper can be significantly improved in terms of the writing. While I am recommending acceptance, I strongly recommend authors to take reviewers' feedback into account and improve the writing significantly for the final version so that more people would benefit from this paper and build on it in the future.
Decision: Accept (Poster)