Year: 2021

Abstract:
Deep convolutional classifiers separate linearly image classes, with an improving accuracy as depth increases. The spatial dimension is progressively reduced whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels. A fundamental challenge is to understand the role of non-linearities together with convolutional filters in this transformation. ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity. This paper demonstrates that it is a different mechanism called phase collapse which eliminates spatial variability while linearly separating classes.This is explained and numerically demonstrated by defining a simplified complex-valued convolutional network architecture. It implements spatial convolutions with wavelet filters and uses a complex modulus to collapse phase variables. This phase collapse network reaches the classification accuracy of ResNets of similar depths. Its performance is considerably degraded when replacing the phase collapses with thresholding operators that enforce sparsity. This is justified by explaining how the iteration of phase collapses progressively improve separation of class means, as opposed to thresholding non-linearities.

heading: 1 INTRODUCTION
text: CNN image classifiers progressively eliminate spatial variables through iterated filterings and subsamplings, while linear classification accuracy improves as depth increases (Oyallon, 2017). It has also been observed numerically that CNNs concentrate training samples of each class in small separated regions of a progressively lower-dimensional space. It can ultimately produce a neural collapse (Papyan et al., 2020), where all training samples of each class are mapped to a single point. In this case, the elimination of spatial variables comes with a collapse of within-class variability and perfect linear separability. This increase in linear classification accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases.
A difficulty in understanding the underlying mathematics comes from the flexibility of ReLUs. Any non-linearity can indeed be approximated by a linear combination of biased ReLUs. Many papers interpret iterations on ReLUs and linear operators as sparse code computations (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020; 2021). We show that it is a different mechanism, called phase collapse, which underlies the increase in classification accuracy of these architectures. A phase collapse is the elimination of phases of complex-valued network coefficients with a modulus, which we show to concentrate spatial variability. This is demonstrated by introducing a structured convolutional neural network, with wavelet filters and no biases.
Section 2 introduces and explains phase collapses, and contrasts them with thresholding operators. Complex-valued representations are used because they reveal the mathematics of spatial variability. Indeed, translations are diagonalized in the Fourier basis, where they become a complex phase shift. Invariants to translations are computed with a modulus which collapses the phases of this complex representation. Section 2 explains how this can improve linear classification. Phase collapses can also be calculated with ReLUs and real filters. A CNN with complex-valued filters is indeed just a particular instance of a real-valued CNN, whose channels are paired together to define complex numbers.
Section 3 demonstrates the role of phase collapse in deep classification architectures, as opposed to thresholdings or more general amplitude reduction operators. It introduces a Learned Scattering network with phase collapses. This network applies a learned 1× 1 convolutional complex operator
Pj on each layer xj , followed by a complex wavelet filtering operator W and a phase collapse:
xj+1 = |WPjxj |. (1) It does not use biases, which are also not needed in deep networks for image denoising (Mohan et al., 2019). This network architecture is illustrated in Figure 1. With the addition of skip-connections, we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10. On the contrary, the accuracy is considerably reduced when replacing the phase collapse modulus by soft-thresholdings with learned biases, which can enforce sparsity but do not modify the phase. This is also true of more general phase-preserving non-linearities.
Section 4 explains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability. On the opposite, the improvements in classification accuracy of successive sparse code computations are shown to quickly saturate.
The main contribution of this paper is a demonstration that the classification accuracy of deep neural networks mostly relies on phase collapses, which are sufficient to linearly separate the different classes on complex image databases. This is captured by the Learned Scattering architecture which reaches ResNet-18 accuracy on ImageNet and CIFAR-10. We also show that phase collapses are necessary to reach this accuracy, by showing numerically and theoretically that iterating phase-preserving non-linearities leads to a significantly worse performance.

heading: 2 PHASE COLLAPSE VERSUS THRESHOLDINGS AND RELUS
text: Deep convolutional classifiers achieve linear separation of image classes. We show that linear classification on raw images has a poor accuracy because image classes are invariant to local translations. This geometric within-class variability takes the form of random phase fluctuations, which induce zero class means. To improve classification accuracy, non-linear operators must therefore eliminate these fluctuations with phase collapses, so as to separate class means. We also analyze the effect of other non-linearities such as soft-thresholdings and ReLUs.
Translations and phase shifts Translations capture the spatial topology of the grid on which the image is defined. They are transformed into phase shifts by a Fourier transform. We prove that this remains approximately valid for images convolved with appropriate complex filters.
Let x be an image indexed by u ∈ Z2. We write xτ (u) = x(u− τ) the translation of x by τ . It is diagonalized by the Fourier transform x̂(ω) = ∑ u x(u) e −iω·u, which creates a phase shift:
x̂τ (ω) = e −iω·τ x̂(ω). (2)
This diagonalization explains the need to introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities. Computations can however be carried with real numbers, as we will show.
A Fourier transform is computed by filtering x with complex exponentials eiω·u. One may replace these by complex filters ψ that are localized in space and in the Fourier domain. The following theorem proves that small translations can still be approximated by a phase shift in this case. A convolution is written ∗.
Theorem 1. Let ψ : Z2 → C be a filter with ‖ψ‖2 = 1, whose center frequency ξ and bandwidth σ are defined by:
ξ = 1
(2π)2 ∫ [−π,π]2 ω |ψ̂(ω)|2 dω and σ2 = 1 (2π)2 ∫ [−π,π]2 |ω − ξ|2|ψ̂(ω)|2 dω.
Then, for any τ ∈ Z2, ‖xτ ∗ ψ − e −iξ·τ (x ∗ ψ)‖∞ ≤ σ |τ | ‖x‖2. (3)
The proof is in Appendix B. This theorem proves that if |τ | 1/σ then xτ ∗ ψ ≈ e −iξ·τx ∗ ψ. In this case, a translation by τ produces a phase shift by ξ · τ .
Phase collapse and stationarity We define a phase collapse as the elimination of the phase of a complex number with a modulus. We now show that phase collapses improve linear classification of classes that are invariant to translations or to local translations.
Each image class indexed by y may be represented by a random vector Xy, whose realizations are the class samples. To achieve linear separation, it is sufficient that class means E [ Xy ]
are sufficiently separated, and that each class mean can be estimated from a single realization, with a sufficiently small variance E[ ∥∥Xy − E[Xy]∥∥2] (Hastie et al., 2009). The goal of classification is to find a representation of the input images in which these properties hold.
To simplify the analysis, we consider the particular case where each class y is invariant to translations. In this case, each random vector Xy is stationary, which means that its probability distribution is invariant to translations. Equation (2) then implies that the phases of Fourier coefficients of Xy are uniformly distributed in [0, 2π], leading to E[X̂y(ω)] = 0 for ω 6= 0. If class-average pixel values of images are the same, then E[X̂y(0)] does not depend on y either. A linear classifier thus cannot discriminate such translation-invariant classes, as class means are all equal.
A phase collapse is then necessary to create non-zero means from these Fourier coefficients. It defines translation-invariant coefficients: |x̂τ | = |x̂|. It improves linear discriminability of stationary classes, because E[|X̂y(ω)|] may be different for different y. However, one can not estimate accurately the value of E[|X̂y(ω)|] at all frequencies ω from a single image. Indeed, there are as many frequencies as the number of image pixels. Moreover, the Fourier transform is unstable to small deformations (Bruna and Mallat, 2013) at high frequencies. This explains the limited performances of a Fourier transform modulus for linear classification. Fourier modulus descriptors can be improved by using filters ψ that have a localized support in space. If ∑ u ψ(u) = 0, then E[Xy ∗ ψ] = 0 if Xy is stationary. Theorem 1 shows that the phase of Xy ∗ ψ is again uniformly distributed in [0, 2π]. A phase collapse similarly computes approximate invariants to small translations: |xτ ∗ ψ| ≈ |x ∗ ψ| with an error bounded by σ |τ | ‖x‖2. This representation improves linear classification by creating different non-zero class means E[|Xy ∗ψ|]. It defines better descriptors than Fourier coefficients because E[|Xy ∗ ψ|] can be accurately estimated from a single image with a spatial average of |Xy ∗ ψ(u)|, if the support of ψ is not too large. Moreover,these expected values are stable to deformations when the support of ψ is small (Bruna and Mallat, 2013).
Image classes are usually not invariant to global translations. For instance, subjects are generally centered or the sky is usually at the top. However, the classes are often invariant to local translations, up to an unknown maximum scale. This is captured by the notion of local stationarity, which means that the probability distribution of Xy is nearly invariant to translations smaller than some maximum scale (Priestley, 1965). The above discussion remains valid if Xy is only locally stationary over a domain larger than the support of ψ. The use of so-called “windowed absolute spectra” E
[∣∣Xy ∗ ψ∣∣] for locally stationary processes has previously been studied in Tygert et al. (2016).
Complex versus real in CNNs: phases and ReLUs The use of complex numbers is a mathematical abstraction to diagonalize translations, which are captured by complex phases. It provides a mathematical interpretation of filtering operations performed on real numbers. We show that a real network can still implement complex phase collapses.
In the first layer of a CNN, one can observe that filters are usually oscillatory patterns with small supports, where some filters have nearly the same orientations and frequency but have a phase shifted
by some α (Krizhevsky et al., 2012). We reproduce in Appendix F a figure from Shang et al. (2016) which evidences this phenomenon. This means that such filters can be written ψα = Re(e
−iαψ) for a single complex filter ψ and different α. Grouping such channels specifies the complex filter ψ. A CNN with complex filters is thus a structured real-valued CNN, whose channels correspond to phase shifts of these complex filters. This structure simplifies the mathematical interpretation of non-linearities by explicitly defining the phase, which is otherwise a hidden variable relating multiple filter outputs within each layer.
A phase collapse is computed with a modulus. It can be calculated by real-valued CNNs by applying ReLUs without biases over filters having different phases: ψα = Re(e
−iαψ). Indeed, for any real-valued signal x, we have:
|x ∗ ψ| = 1 2 ∫ π −π ReLU(x ∗ ψα) dα. (4)
Furthermore, this integral is well approximated by a sum over 4 phases, allowing to compute complex moduli with ReLUs without biases over real-valued CNN filters. See Appendix A for a proof of eq. (4) and its approximation.
Thresholding and sparsity versus phase collapse The action of a ReLU can be linearly decomposed into an elimination of sign and a soft-thresholding, which attenuates the amplitude. As both can improve linear classification in different cases, an important issue is to understand which property is important for classification.
If z ∈ R and b ≥ 0, then the even part of ReLU(z − b) is ReLU(z − b) + ReLU(−z − b), which is an absolute value with a dead-zone [−b, b]. When b = 0, it becomes a phase collapse |z|. The odd part is a soft-thresholding ρb(z) = ReLU(z − b) − ReLU(−z − b). A soft-thresholding operator over a complex z = |z| eiϕ reduces the amplitude by b while preserving the phase:
ρb(z) = max(|z| − b, 0) e iϕ. (5)
Similarly to its real counterpart, this amounts to defining the complex soft-thresholding as the proximal operator of the complex modulus (Yang et al., 2012):
ρb(z) = arg min w∈C b|w|+ 1 2 |w − z|2. (6)
Soft-thresholdings and phase collapses have opposite properties, since soft-thresholdings preserve the phase while attenuating the amplitude, whereas phase collapses preserve the amplitude while eliminating the phase.
We have explained how phase collapses can improve the classification accuracy of locally stationary processes by separating class means E [∣∣Xy ∗ ψ∣∣]. In contrast, if Xy ∗ ψ has a distribution which is symmetric around 0 then E [ ρb(Xy ∗ ψ) ] = 0 for all b because the thresholding preserves the phase
and hence this symmetry around 0. This symmetry is often observed when ∑ u ψ(u) = 0, in which case class means are not separated by a thresholding.
If E [ Xy ∗ ψ ] 6= 0 andXy ∗ψ is sparse, then a soft-thresholding ofXy ∗ψ may reduce the variance of class mean estimators (Donoho and Johnstone, 1994). In this case, it can improve linear classification. Coefficients below the threshold may be assimilated to unnecessary “clutter” which is set to 0. To improve classification, convolutional filters must produce high-amplitude coefficients corresponding to discriminative “features”, whereas small-amplitude clutter coefficients are eliminated (Zarka et al., 2021). The dimensionality of sparse representations can also be reduced with random filters which implement a form of compressed sensing (Donoho, 2006; Candes et al., 2006). The interpretation of CNNs as compressed sensing machines with random filters has been studied (Giryes et al., 2015), but it never led to classification results close to e.g. ResNet accuracy. On the contrary, we shall see that the use of phase collapse alone reaches this performance.

heading: 3 LEARNED SCATTERING NETWORK WITH PHASE COLLAPSE
text: This section introduces a learned scattering transform, which reaches ResNet accuracy on the ImageNet (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky, 2009) datasets. It is a highly structured CNN architecture computed with wavelet filters and phase collapses. We describe this architecture and then compare the performance of phase collapses and thresholding non-linearities.
Scattering transform Theorem 1 proves that a phase collapse applied to the output of a complex filter produces a locally invariant descriptor that can then be subsampled, depending upon the filter’s bandwidth. We briefly review the scattering transform (Mallat, 2012; Bruna and Mallat, 2013), which iterates such convolutions with wavelet filters and moduli.
A scattering transform over J scales is implemented with a network of depth J , whose filters are specified by the choice of wavelet. Let x0 = x. For 0 ≤ j < J , the j + 1-th layer xj+1 is computed by applying a wavelet filtering operator W and a modulus on the j-th layer xj :
xj+1 = ∣∣Wxj∣∣. (7)
The operator W is defined with Morlet filters (Bruna and Mallat, 2013). It has one low-pass filter g0, and L zero-mean complex band-pass filters g`, having an angular direction `π/L for 0 < ` ≤ L. It thus transforms an input image x(u) into L+ 1 sub-band images which are subsampled by 2:
Wx(u, `) = x ∗ g`(2u). (8)
The cascade of j low-pass filters g0 with a final band-pass filter g`, each followed by a subsampling, computes wavelet coefficients at each scale 2j . One can also modify the wavelet filtering W to compute intermediate scales 2j/2, as explained in Appendix E. The spatial subsampling is then only implemented every other layer, and the depth of the network becomes twice larger. One can restrict the channels of xJ to the ones obtained from less than 2 moduli (Bruna and Mallat, 2013). Applying a linear classifier on such a scattering transform classifier gives good results on simple classification problems such as MNIST (LeCun et al., 2010). However, results are well below ResNet performances on CIFAR-10 and ImageNet, as shown in Table 1.
Learned Scattering Zarka et al. (2021) showed that a scattering transform can reach ResNet accuracy by alternating wavelet transform filters with two learned 1 × 1 convolutional operators. The first is preceded by a ReLU with no bias, and the second is followed by a soft-thresholding non-linearity. However, this learned scattering architecture is difficult to interpret in the context of phase collapses versus thresholdings, because it mixes both. In line with several papers on the use of sparsity for classification (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020), Zarka et al. (2021) made the hypothesis that sparsifying neural responses with thresholdings is a major mechanism for improving classification accuracy. Our Learned Scattering architecture with phase collapse modulus shows that it is not necessary to use any biases nor thresholdings. We demonstrate numerically that high image classification performance mostly results from iterated phase collapses which separate class means.
The Learned Scattering (LScat) network inserts in eq. (7) a learned complex 1 × 1 convolutional operator Pj which reduces the channel dimensionality of each layer xj , before applying wavelet filters and a phase collapse:
xj+1 = ∣∣WPjxj∣∣. (9)
Similar architectures which separate space-mixing and channel-mixing operators had previously been studied in the context of basis expansion (Qiu et al., 2018; Ulicny et al., 2019) or to filter scattering channels (Cotter and Kingsbury, 2019). This separation is also a major feature of recent architectures such as Vision Transformers (Dosovitskiy et al., 2021) or MLP-Mixer (Tolstikhin et al., 2021). Each Pj computes discriminative channels whose spatial variability is eliminated by the phase collapse of complex wavelet coefficients. Their role is further discussed in Section 4. Table 1 gives the accuracy of a linear classifier applied to the last layer of this Learned Scattering. It provides an important improvement over a scattering transform, but it does not yet reach the accuracy of ResNet-18.
The architecture uses a total number of layers J = 11 for ImageNet and J = 8 for CIFAR, by introducing intermediate scales 2j/2. The number of channels of Pjxj are the same as in a standard ResNet architecture (He et al., 2016) and remain no larger than 512. Details of this Learned Scattering architecture are provided in Appendix E.
Skip-connections across moduli The architecture of eq. (9) imposes that all phases are collapsed at each layer, after computing a wavelet transform. More flexibility is provided by adding a skipconnection which concatenates WPjxj with its modulus:
xj+1 = [∣∣WPjxj∣∣ , WPjxj]. (10)
The skip-connection produces a cascade of convolutional filtersW without non-linearities in-between. The resulting convolutional operator WW · · ·W is a “wavelet packet” transform which generalizes the wavelet transform (Coifman and Wickerhauser, 1992; Mallat, 2008). Wavelet packets are obtained as the cascade of low-pass and band-pass filters (g`)`, each followed by a subsampling. Besides wavelets, wavelet packets include filters having a larger spatial support than wavelets and a narrower Fourier bandwidth. A wavelet packet transform is then similar to a local Fourier transform. Applying a modulus on such wavelet packet coefficients defines local spatial invariants over wider domains.
As discussed in Section 2, image classes are usually invariant to local rather than global translations. Section 2 explains that a phase collapse improves discriminability for image classes that are locally translation-invariant over the filter’s support. Indeed, phases of wavelet coefficients are then uniformly distributed over [0, 2π], yielding zero-mean coefficients for all classes. At a scale where there is no local translation-invariance, these phases are no longer uniformly distributed, and they encode information about the spatial localization of features. Introducing a skip-connection provides the flexibility to choose whether to eliminate phases at different scales or to propagate them up to the last layer. Indeed, the next 1× 1 operator Pj+1 linearly combines
∣∣WPjxj∣∣ and WPjxj and may learn to use only one of these. This adds some localization information, which appears to be important.
Table 1 shows that the skip-connection improves classification accuracy. Applying a linear classifier on this Learned Scattering reaches ResNet-18 accuracy on CIFAR-10 and ImageNet. It demonstrates that collapsing appropriate phases is sufficient to obtain 89% Top-5 accuracy on large-scale classification problems. Learning is reduced to 1× 1 filters (Pj)j across channels.
Phase collapse versus amplitude reduction A Learned Scattering with phase collapse preserves the amplitudes of wavelet coefficients and eliminates their phases. Section 2 explains that it improves separation of class means. On the opposite, one may use a non-linearity which preserves the phases of wavelet coefficients but attenuates their amplitudes. Among such non-linearities is the softthresholding, which can improve classification by reducing the variance of class mean estimators, within a sparse representation. These sparse representations are usually considered important for image classification (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019). The following results show that it is not the case. Several amplitude reduction non-linearities are used, including soft-thresholdings, sigmoids, and an amplitude collapse. We show that it considerably degrades the accuracy compared to a phase collapse, to the point that the classification error becomes larger than with a scattering without learning.
A Learned Amplitude Reduction Scattering applies a non-linearity ρ(z) on each wavelet coefficient z = |z|eiϕ and preserves the phase: ρ(z) = eiϕ ρ(|z|). Without skip-connections, each layer xj+1 is computed from xj by:
xj+1 = ρ(WPjxj), (11)
and with skip-connections:
xj+1 = [ ρ(WPjxj) , WPjxj ] . (12)
A soft-thresholding is defined by ρ(|z|) = ReLU(|z| − b) for some threshold b. For an amplitude sigmoid we have ρ(|z|) = (1 + e−a|z|−b)−1, and an amplitude collapse is ρ(|z|) = 1. The softthresholding and sigmoid parameters a and b are learned for each layer and each channel.
The classification performance of a Learned Amplitude Reduction Scattering is evaluated on CIFAR10, by applying a linear classifier on the last layer. Classification results are given in Table 2 for different amplitude reductions, with or without skip-connections. Learned Amplitude Reduction Scatterings yield much larger errors than a Learned Scattering with phase collapses. They are even above a scattering transform which also uses phase collapses but without learned 1× 1 convolutional projections Pj . It demonstrates that high accuracies result from phase collapses without biases, as opposed to amplitude reduction operators including thresholdings, which learn bias parameters.
ReLUs with biases Most CNNs, including ResNets, use ReLUs with biases. As explained in Section 2, a ReLU with bias simultaneously affects the sign and the amplitude of a real coefficient. When decomposing it into its even and odd part, we separate the effects on sign and on amplitude. Over complex numbers, it amounts to transforming the phase or the amplitude. These numerical experiments show that accuracy improvements result from phase collapses rather than amplitude reductions, which is a surprising new result.
Several CNN architectures have demonstrated a good classification accuracy with iterated thresholding algorithms, which increase sparsity. However, all these architecture also modified the sign of coefficients by computing non-negative sparse codes (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019) or with additional ReLU or modulus layers (Zarka et al., 2020; 2021). It seems that it is the sign or phase collapse of these non-linearities which is responsible for good classification accuracies, as opposed to the calculation of sparse codes through iterated amplitude reductions.

heading: 4 ITERATING PHASE COLLAPSES AND AMPLITUDE REDUCTIONS
text: We now provide a theoretical justification to the above numerical results in simplified mathematical frameworks. This section studies the behavior of phase collapses and amplitude reductions when they are iterated over several layers. It shows that phase collapses benefit from iterations over multiple layers, whereas there is no significant gain in performance when iterating amplitude reductions.

heading: 4.1 ITERATED PHASE COLLAPSES
text: We explain the importance of phase collapses to improve linear classification, as well as the role of iterated phase collapses with multiple filters at each layer, together with learned projectors Pj . For this purpose, we consider the classification of ergodic stationary processes Xy ∈ R
d corresponding to different image textures indexed by y. Given a realization x of Xy , and because of ergodicity, the optimal linear classifier is calculated from the empirical mean 1/d ∑ u x(u). It computes an optimal
linear estimation of E [ Xy(u) ] = µy. If all textures have the same mean µy = µ, then all linear estimators fail. More precisely, the textures can be linearly separated only if the distance between different means ‖µy − µy′‖ is greater than the average estimation error of the empirical means.
As explained in Section 2, linear classification can be improved by computing (|x ∗ ψk|)k for some filters (ψk)k such that ∑ u ψk(u) = 0, because phase collapses create non-zero means. If Xy is stationary, then |Xy ∗ ψk| remains stationary for any ψk. An optimal linear classifier applied to (|x ∗ψk(u)|)k is thus obtained by a linear combination of all empirical means (1/d ∑ u |x ∗ψk(u)|)k. They are proportional to the `1 norm ‖x ∗ ψk‖1, which is a measure of sparsity of x ∗ ψk.
If linear classification on (|x ∗ ψk(u)|)k fails, it reveals that the means E [ |Xy ∗ ψk(u)| ] = µy,k are not sufficiently different. Separation can be improved by considering the spatial variations of |Xy ∗ ψk(u)| for different y. These variations can be revealed by a new set of filters ψk′ having a zero mean, such as a new set of wavelet filters. Since E [ |Xy ∗ ψk| ∗ ψk′ ] = 0, we must again
cancel the phase and compute E [ ||Xy ∗ ψk| ∗ ψk′ | ] . This phase collapse iteration is the principle used by scattering transforms to discriminate textures (Bruna and Mallat, 2013; Sifre and Mallat, 2013). However, this may still not be sufficiently accurate. More discriminant statistical properties may be obtained by linearly combining |Xy ∗ ψk| across k before applying a new filter ψk′ . In a Learned Scattering with phase collapse, the x ∗ ψk are combined with a linear projector P1 across the channel indices k, before computing a convolution with the next filter ψk′ . The 1× 1 filter P1 is optimized to improve the linear classification accuracy. It amounts to computing weights wk such that E[ ∣∣∑ k wk
∣∣Xy ∗ ψk∣∣ ∗ ψk′ ∣∣] is as different as possible for different y. Because these are proportional to the `1 norms ∥∥∑ k wk|x ∗ ψk| ∗ ψk′ ∥∥ 1 , it means that the images ∑ k wk|x ∗ ψk| ∗ψk′ have different sparsity levels depending upon the class y of x. The filter weights of P1 can thus be interpreted as features along channels providing different sparsifications for different classes. A Learned Scattering computes such Pj at each scale j.

heading: 4.2 ITERATED AMPLITUDE REDUCTIONS
text: Sparse representations and amplitude reduction algorithms may improve linear classification by reducing the variance of class mean estimations, which can be interpreted as clutter removal. Such approaches are studied in Zarka et al. (2021) by modeling the clutter as an additive white noise. Although a single thresholding step may improve linear classification, we show that iterating more than one thresholding does not improve the classification accuracy, if no phase collapses are inserted.
To understand these properties, we consider the discrimination of classes represented by non-stationary random processes Xy(u). We suppose that class means are linearly separable and hence that all E(Xy) = µy are different for different y. If there exists y
′ such that ‖µy − µy′‖ is small, then the class y can still be discriminated, if we can estimate E(Xy) sufficiently accurately from a single realization x of Xy. This is a mean estimation problem. Suppose that Xy = µy + N (0, σ
2) is contaminated with Gaussian white noise, where the noise models some clutter. Suppose also that there exists a linear orthogonal operator D such that Dµy is sparse for every y, and hence has its energy concentrated in few non-zero coefficients. Such a D may be computed by minimizing the expected `1 norm ∑ y E [ ‖DXy‖1 ] . The estimation of µy can be improved with a soft-thresholding estimator (Donoho and Johnstone, 1994), which sets to zero all coefficients below a threshold b proportional to σ. It amounts to computing ρb(Dx), with ρb(x) = ReLU(x− b)− ReLU(−x− b). However, we explain why this approach cannot be further iterated without inserting phase collapses. The reason is that sparsity and entropy of phases are linked, in the sense that a sparse representation concentrates its entropy in the phases of the coefficients, rather than their amplitude. If the entropy is concentrated in the phases, then the process cannot be further sparsified. The entropy of the phases must then be reduced before the next sparsification. This entails that a model of intra-class variability relying on sparsity and amplitude reduction cannot be the sole mechanism behind the performance of deep networks. Iterating sparsity may however be useful if it is alternated with another non-linearity which partly or fully collapses phases. As mentioned in Section 3, this is the case for previous work which used iterated sparsification operators (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019). Indeed, these networks compute non-negative sparse codes where sparsity is enforced with a ReLU, which acts both on phases and amplitudes. Our results shows that the benefit of iterating non-negative sparse coding comes from the sign collapse due to the non-negativity constraint.
We now qualitatively demonstrate the two directions of this link with two theorems. We first show that finding the sparsest representation of a random process (i.e., minimizing its `1 norm) is the same as maximizing a lower bound on the entropy of its phases.
Theorem 2. Let X denote a random vector in Cd with a probability density p. Let H(X) be the entropy of X with respect to the Lebesgue measure:
H(X) = − ∫ p(x) log p(x) dx.
If D ∈ U(d) is a unitary operator then:
H ( ϕ(DX) ∣∣∣ |DX|) ≥ H(X)− d− 2d log(1 d E[‖DX‖1] ) ,
where ϕ(DX) ∈ [0, 2π]d (resp. |DX| ∈ Rd+) is the random process of the entry-wise phases (resp. moduli) of DX .
The proof is in Appendix C. This theorem gives a lower-bound of the conditional entropy of the phases of DX with a decreasing function of the expected `1 norm of DX . Minimizing over D this expected `1 norm amounts to maximizing the lower bound on H ( ϕ(DX) ∣∣∣ |DX|). An extreme situation arises when this entropy reaches its maximal value of d log(2π). In this case, the phase ϕ(DX) has a maximum-entropy distribution and is therefore uniformly distributed in [0, 2π]d. Moreover, in this extreme case ϕ(DX) is independent from |DX|, since its conditional distribution does not depend on |DX|. Such statistical properties have previously been observed on wavelet coefficients of natural images (Rao et al., 2001), where the wavelet transform seems to be a nearly optimal unitary dictionary to enforce sparsity.
The second theorem considers the extreme case of a random process whose phases are conditionally independent and uniform. It proves show that such a process cannot be significantly sparsified with a change of basis.
Theorem 3. Let X be a random vector of Cd such that ϕ(X) is uniformly distributed in [0, 2π]d and independent from |X|. Then there exists a constant Cd > 0 which depends on the dimension d, such that for any D ∈ U(d),
E[‖DX‖1] ≥ CdE[‖X‖1].
The proof is in Appendix D. This theorem shows that random processes with conditionally independent and uniform phases have an `1 norm which cannot be significantly decreased by any unitary transformation. Numerical evaluations suggest that the constant Cd may be chosen to be√ π/2 ≈ 0.886, independently of the dimension d. This constant arises as the value of E[|Z|] when Z is a complex normal random variable with E[|Z|2] = 1. These two theorems explain qualitatively that linear classification on ρb(Dx) cannot be improved by another thresholding that would take advantage of another sparsification operator. Indeed, Theorem 2 shows that if Dx is sparse, then its phases have random fluctuations of high entropy. Theorem 3 indicates that such random phases prevent a further sparsification of ρb(Dx) with some linear operator D′. Applying a second thresholding ρb′(D
′ρb(Dx)) thus cannot significantly reduce the variance of class mean estimators.

heading: 5 CONCLUSION
text: This paper studies the improvement of linear separability for image classification in deep convolutional networks. We show that it mostly relies on a phase collapse phenomenon. Eliminating the phase of wavelet coefficients improves the separation of class means. We introduced a Learned Scattering network with a phase collapse on the output of wavelet filters and learned 1× 1 convolutional filters Pj , which reaches ResNet accuracy. The learned 1 × 1 operators Pj enhance discriminability by computing channels that have different levels of sparsity for different classes.
When class means are different, thresholding non-linearities can improve classification by reducing the variance of class mean estimators. When used alone, the classification performance is poor over complex datasets such as ImageNet or CIFAR-10, because class means are not sufficiently separated. The iteration of thresholdings on sparsification operators indeed requires intermediary phase collapses.
These results show that the elimination of spatial variability with phase collapses is both necessary and sufficient to linearly separate classes on complex image datasets.
REPRODUCIBILITY STATEMENT
The code to reproduce the experiments of the paper is available in the supplementary material. All experimental details and hyperparameters are also provided in Appendix E.

heading: A PROOF OF EQUATION (4)
text: We have: ReLU(x ∗ ψα) = ReLU(x ∗ Re(e −iαψ)) = ReLU(Re(e−iαx ∗ ψ)),
since x is real. By writing: x ∗ ψ = |x ∗ ψ|eiϕ(x∗ψ) where ϕ(x ∗ ψ) is the phase of x ∗ ψ, this leads to:
ReLU(Re(e−iαx ∗ ψ)) = ReLU(Re(|x ∗ ψ|ei(ϕ(x∗ψ)−α))) = ReLU(|x ∗ ψ| cos(ϕ(x ∗ ψ)− α)) = |x ∗ ψ|ReLU(cos(ϕ(x ∗ ψ)− α)),
since ReLU activation is positive-homogeneous of degree 1. Thus:
1
2 ∫ π −π ReLU(x ∗ ψα)dα = 1 2 ∫ π −π |x ∗ ψ|ReLU(cos(ϕ(x ∗ ψ)− α))dα
= 1
2 |x ∗ ψ| ∫ π−ϕ(x∗ψ) −π−ϕ(x∗ψ) ReLU(cos(−α))dα with a change of variable,
= 1
2 |x ∗ ψ| ∫ π −π ReLU(cos(α))dα since cos is 2π periodic and even,
= 1
2 |x ∗ ψ| ∫ π/2 −π/2 cos(α)dα
= |x ∗ ψ|.
For z ∈ C, we have |z| = √ |Re(z)|2 + |Im(z)|2 ≈ |Re(z)|+ |Im(z)| in the following sense:
1√ 2 (|Re(z)|+ |Im(z)|) ≤ |z| ≤ |Re(z)|+ |Im(z)|.
We can write:
|Re(z)| = ReLU(Re(z)) + ReLU(−Re(z)), |Im(z)| = ReLU(Im(z)) + ReLU(− Im(z)).
and then, using Im(z) = Re(eiπ/2z) and eiπ = −1:
|z| ≈ ReLU(Re(z)) + ReLU(Re(e−iπz)) + ReLU(Re(e−iπ/2z)) + ReLU(Re(eiπ/2z)).
Finally,
|x ∗ ψ| = 1 2 ∫ π −π ReLU(x ∗ ψα)dα ≈ ∑
α∈{−π/2,0,π/2,π}
ReLU(Re(x ∗ ψα)),
which shows that the integral can be well approximated with a sum of 4 phases α of the complex filter ψ.

heading: B PROOF OF THEOREM 1
text: We have:
‖xτ ∗ ψ − e −iξ·τ (x ∗ ψ)‖∞ = ‖x ∗ (ψτ − e −iξ·τψ)‖∞ by covariance of convolution, ≤ ‖ψτ − e −iξ·τψ‖2‖x‖2 by Young’s inequality,
and then:
‖ψτ − e −iξ·τψ‖
2 2 = 1
(2π)2 ∫ [−π,π]2 |ψ̂τ (ω)− e −iξ·τ ψ̂(ω)|2dω by Plancherel,
= 1
(2π)2 ∫ [−π,π]2 |e−iω·τ ψ̂(ω)− e−iξ·τ ψ̂(ω)|2dω since ψτ (u) = ψ(u− τ),
= 1
(2π)2 ∫ [−π,π]2 |e−iω·τ − e−iξ·τ |2|ψ̂(ω)|2dω
≤ 1 (2π)2 ∫ [−π,π]2 |(ω − ξ) · τ |2|ψ̂(ω)|2dω since x ∈ R 7→ eix is 1-Lipschitz,
≤ 1 (2π)2 ∫ [−π,π]2 |ω − ξ|2|τ |2|ψ̂(ω)|2dω by Cauchy-Schwarz,
= σ2|τ |2,
which leads to the desired result of eq. (3):
‖xτ ∗ ψ − e −iξ·τ (x ∗ ψ)‖∞ ≤ σ |τ | ‖x‖2.

heading: C PROOF OF THEOREM 2
text: We first use the chain rule for the entropy: H ( ϕ(DX) ∣∣∣ |DX|) = H(|DX|, ϕ(DX))−H(|DX|). The first term is rewritten with a change of variable:
H(|DX|, ϕ(DX)) = H(DX)− d∑ k=1 E[log |(DX)k|]
= H(X)− d∑ k=1 E[log |(DX)k|] as D is unitary and hence |det(D)| = 1,
≥ H(X)− dE [ log ( 1
d ‖DX‖1
)] by concavity,
≥ H(X)− d log ( 1
d E[‖DX‖1]
) by concavity.
The second term is bounded using the fact that the exponential distribution E(λ) is the maximumentropy distribution on R+ with mean 1λ :
H(|DX|) ≤ d∑ k=1 H(|(DX)k|)
≤ d∑ k=1 log(eE[|(DX)k|])
≤ d log ( e d E[‖DX‖1] ) by concavity.
Combining both inequalities and rearranging terms yields the stated bound: H ( ϕ(DX) ∣∣∣ |DX|) ≥ H(X)− d− 2d log(1 d E[‖DX‖1] ) .

heading: D PROOF OF THEOREM 3
text: We begin with the following lemma:
Lemma 1. Let (θ1, . . . , θd) be i.i.d. uniform random variables in [0, 2π]. Then there exists a constant Cd > 0 such that for all (ρ1, . . . , ρd) ∈ R d, then:
E [ | d∑ k=1 ρke iθk | ] ≥ Cd √√√√ d∑ k=1 ρ2k.
This is proved by observing that the left-hand side is a norm on Rd. One can indeed verify that it is positive definite, homogeneous and satisfies the triangle inequality. Since all norms on Rd are equivalent, there exists a constant Cd > 0 such that:
E [ | d∑ k=1 ρke iθk | ] ≥ Cd √√√√ d∑ k=1 ρ2k.
for all (ρ1, . . . , ρd) ∈ R d.
Going back to the proof of Theorem 3, we then have: E [ ‖DX‖1 ∣∣∣ |X|] = d∑ m=1 E [ | d∑ k=1 Dm,kXk| ∣∣∣∣∣ |X| ]
≥ Cd d∑
m=1 √√√√ d∑ k=1
∣∣Dm,k∣∣2|Xk|2 by the above lemma, ≥ Cd
d∑ m=1 d∑ k=1 ∣∣Dm,k∣∣2|Xk| by concavity, because d∑ k=1 ∣∣Dm,k∣∣2 = 1, = Cd‖X‖1 because
d∑ m=1 ∣∣Dm,k∣∣2 = 1. Taking the expectation finishes the proof:
E[‖DX‖1] ≥ CdE[‖X‖1]. (13)

heading: E EXPERIMENTAL DETAILS
text: Channel operators In all experiments we set P0 = Id, and factorize the classifier with an additional complex 1 × 1 convolutional operator PJ , which reduces the dimension before all channels and positions are linearly combined. The architectures implemented are thus also written as ∏J j=1 PjρW , where ρ is the non-linearity. Each operator (Pj)1≤j≤J is preceded by a standardization. It sets the complex mean µ = E[z] of every channel to zero, and the real variance σ2 = E[|z|2] of every channel to one. This is similar to a complex 2D batch-normalization layer (Ioffe and Szegedy, 2015), but without learned affine parameters. Each operator (Pj)1≤j≤J is additionally followed by a spatial divisive normalization (Rao et al., 2001), similarly to the local response normalization of Krizhevsky et al. (2012). It sets the norm across channels of each spatial position to one. The sizes of the (Pj)j are specified in Table 3.
The total numbers of parameters for each architecture are specified in Table 4. Learned Scattering with phase collapse have a large number of parameters compared to ResNet, despite the comparable width. This is because the predefined wavelet operator W expands the dimension by a factor of L + 1, which means that the input dimension of the learned (Pj)j is higher than in ResNet. The skip-connection further increases this input dimension by a factor of 2.
Spatial filters We use elongated Morlet filters for the L complex band-pass filters (g`)` which are rotated versions of a mother wavelet g: g`(u) = g(r−π`/Lu), with rθ the rotation by angle θ. The mother wavelet g is defined as:
g(u) = σ2
2π/s2 (eiξ·u −K)e−u·Σu/2 with Σ =
( σ2 0
0 σ2s2
) , (14)
Its parameters are its center frequency ξ = ((3π/4)/2γ , 0), its bandwidth σ = 1.25× 2−γ , and its slant s = 0.5, where 2γ designates the scale of the band-pass filter and is to be adjusted.
g is rotated along L = 8 angles for Imagenet and L = 4 angles for CIFAR: θ` = (π`/L)1≤`≤L. The g` are then discretized for numerical computations, and K is adjusted so that they have a zero mean.
Finally, we use for the low frequency g0 a Gaussian window:
g0(u) = σ2
2π e−σ 2‖u‖22/2.
The filters are implemented with the Kymatio package (Andreux et al., 2020).
Intermediate scales 2j/2 are obtained by applying a subsampling by 2 after each block of 2 layers. This introduces intermediate scales and generates a wavelet filterbank with 2 scales per octave: the filters are designed so that when j low-pass filters and one band-pass filter are cascaded, with a subsampling every 2 layers, the scale of the resulting wavelet is 2j/2.
Each block comprises in its first layer a low-frequency filter g10 with γ = −1/2 and band-pass filters with γ = 0. In the second layer, we use the same low-frequency filter g20 = g 1 0 with γ = −1/2. The band-pass filters g2` are obtained with parameters ξ ′ = (π/ √ 2, 0), σ′ = 1.25 √ 2/3, and s′ = √ 0.2.
For CIFAR experiments, the J = 8 layers are grouped in 4 successive blocks of 2 layers. For ImageNet experiments, the first layer consists of band-pass elongated Morlet filters g` and a low-pass Gaussian window g0 with γ = 0, followed by a subsampling of 2. The 10 following layers are grouped in 5 blocks of 2 layers.
Optimization We use the optimizer SGD with an initial learning rate of 0.01, a momentum of 0.9, a weight decay of 0.0001, and a batch size of 128. The classifier is preceded by a 2D batchnormalization layer. We use traditional data augmentation: horizontal flips and random crops for CIFAR, random resized crops of size 224 and horizontal flips for ImageNet. Classification error on ImageNet validation set is computed on a single center crop of size 224. On CIFAR, training lasts for 300 epochs and the learning rate is divided by 10 every 70 epochs. On ImageNet, training lasts for 150 epochs and the learning rate is divided by 10 every 45 epochs. The experiments in the paper required around 10k 32GB NVIDIA V100 GPU-hours.

heading: F PAIRED ALEXNET FILTERS
text: Section 2 explains that real networks can still implement phase collapses. This is done with several filters ψα = Re(e
−iαψ) which correspond to several phases α of the same complex filter ψ. Shang et al. (2016) showed that the filters in e.g. the first layer of AlexNet (Krizhevsky et al., 2012) can
indeed be grouped in such a way. For the sake of completeness, we reproduce in Figure 2 a figure from Shang et al. (2016). This suggests that real-valued networks may indeed implement phase collapses using eq. (4).

Review summaries:
This paper studies within-class variability which reduces along the layers of deep neural networks. They mainly question the effect of sparsity and soft-thresholding introduced by ReLU. They show that these classification improvements by eliminating spatial within-class variabilities rather come from a phase collapse, which eliminates the phase of network coefficients. Eliminating the phase of zero-mean filters improves the separation of class means, hence increase in classification accuracy. They introduced a complex-valued neural network in which spatial filters are defined as complex multiscale wavelets and learning is reduced to $1 \times 1$ complex filters across channels. Their results show that such a network is able to reach ResNet-18 performance on CIFAR10 and ImageNet.     

This paper proposes that using phase modulus operators instead of thresholding non-linearites is beneficial to classification performance in the case of scattering like networks. The authors show that such a network with some learnable operators can come close to performance of small ResNets when they use the modulo operators and performance degrades when using other non-linearieis. This is demonstrated on ImageNet and CIFAR-10 and is accompanied with relevant analysis and theory. 

The authors present some theoretical results, followed by some experiments, in support of their argument that the linear separability for image classification in deep convolutional neural networks mostly relies on a phase collapse phenomenon. By eliminating the phase of zero mean filters they improve the separation of class means.They present a Phase Collapse Scattering network and demonstrate Resnet-like accuracy. The authors argue that phase collapse are both necessary and sufficient to discriminate class means on complex datasets.

This is a very interesting paper that is based on scattering networks, for which it shows that the so called ‘phase collapse’ leads to state-of-the-art results on par with modern architectures, like ResNet. To derive this, the paper shows that having neural networks on complex numbers is similar to a structure deep network (like with wavelet filters) on the reals. Phase collapse is then when there is an operation, like the modulus (absolute value function), which eliminates the phase from the complex number and maintains only the amplitude. All in all, the paper shows that maintaining the amplitude while eliminating the phase, is what brings the high accuracies, while the reverse (keeping the phase, eliminating the amplitude) yields terrible accuracies. As a corollary, I would say that the paper explains why non-linearities, like ReLUs, are so successful.