Phase Collapse in Neural Networks

1 INTRODUCTION . CNN image classifiers progressively eliminate spatial variables through iterated filterings and subsamplings , while linear classification accuracy improves as depth increases ( Oyallon , 2017 ) . It has also been observed numerically that CNNs concentrate training samples of each class in small separated regions of a progressively lower-dimensional space . It can ultimately produce a neural collapse ( Papyan et al. , 2020 ) , where all training samples of each class are mapped to a single point . In this case , the elimination of spatial variables comes with a collapse of within-class variability and perfect linear separability . This increase in linear classification accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases . A difficulty in understanding the underlying mathematics comes from the flexibility of ReLUs . Any non-linearity can indeed be approximated by a linear combination of biased ReLUs . Many papers interpret iterations on ReLUs and linear operators as sparse code computations ( Sun et al. , 2018 ; Sulam et al. , 2018 ; 2019 ; Mahdizadehaghdam et al. , 2019 ; Zarka et al. , 2020 ; 2021 ) . We show that it is a different mechanism , called phase collapse , which underlies the increase in classification accuracy of these architectures . A phase collapse is the elimination of phases of complex-valued network coefficients with a modulus , which we show to concentrate spatial variability . This is demonstrated by introducing a structured convolutional neural network , with wavelet filters and no biases . Section 2 introduces and explains phase collapses , and contrasts them with thresholding operators . Complex-valued representations are used because they reveal the mathematics of spatial variability . Indeed , translations are diagonalized in the Fourier basis , where they become a complex phase shift . Invariants to translations are computed with a modulus which collapses the phases of this complex representation . Section 2 explains how this can improve linear classification . Phase collapses can also be calculated with ReLUs and real filters . A CNN with complex-valued filters is indeed just a particular instance of a real-valued CNN , whose channels are paired together to define complex numbers . Section 3 demonstrates the role of phase collapse in deep classification architectures , as opposed to thresholdings or more general amplitude reduction operators . It introduces a Learned Scattering network with phase collapses . This network applies a learned 1× 1 convolutional complex operator Pj on each layer xj , followed by a complex wavelet filtering operator W and a phase collapse : xj+1 = |WPjxj | . ( 1 ) It does not use biases , which are also not needed in deep networks for image denoising ( Mohan et al. , 2019 ) . This network architecture is illustrated in Figure 1 . With the addition of skip-connections , we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10 . On the contrary , the accuracy is considerably reduced when replacing the phase collapse modulus by soft-thresholdings with learned biases , which can enforce sparsity but do not modify the phase . This is also true of more general phase-preserving non-linearities . Section 4 explains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability . On the opposite , the improvements in classification accuracy of successive sparse code computations are shown to quickly saturate . The main contribution of this paper is a demonstration that the classification accuracy of deep neural networks mostly relies on phase collapses , which are sufficient to linearly separate the different classes on complex image databases . This is captured by the Learned Scattering architecture which reaches ResNet-18 accuracy on ImageNet and CIFAR-10 . We also show that phase collapses are necessary to reach this accuracy , by showing numerically and theoretically that iterating phase-preserving non-linearities leads to a significantly worse performance . 2 PHASE COLLAPSE VERSUS THRESHOLDINGS AND RELUS . Deep convolutional classifiers achieve linear separation of image classes . We show that linear classification on raw images has a poor accuracy because image classes are invariant to local translations . This geometric within-class variability takes the form of random phase fluctuations , which induce zero class means . To improve classification accuracy , non-linear operators must therefore eliminate these fluctuations with phase collapses , so as to separate class means . We also analyze the effect of other non-linearities such as soft-thresholdings and ReLUs . Translations and phase shifts Translations capture the spatial topology of the grid on which the image is defined . They are transformed into phase shifts by a Fourier transform . We prove that this remains approximately valid for images convolved with appropriate complex filters . Let x be an image indexed by u ∈ Z2 . We write xτ ( u ) = x ( u− τ ) the translation of x by τ . It is diagonalized by the Fourier transform x̂ ( ω ) = ∑ u x ( u ) e −iω·u , which creates a phase shift : x̂τ ( ω ) = e −iω·τ x̂ ( ω ) . ( 2 ) This diagonalization explains the need to introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities . Computations can however be carried with real numbers , as we will show . A Fourier transform is computed by filtering x with complex exponentials eiω·u . One may replace these by complex filters ψ that are localized in space and in the Fourier domain . The following theorem proves that small translations can still be approximated by a phase shift in this case . A convolution is written ∗ . Theorem 1 . Let ψ : Z2 → C be a filter with ‖ψ‖2 = 1 , whose center frequency ξ and bandwidth σ are defined by : ξ = 1 ( 2π ) 2 ∫ [ −π , π ] 2 ω |ψ̂ ( ω ) |2 dω and σ2 = 1 ( 2π ) 2 ∫ [ −π , π ] 2 |ω − ξ|2|ψ̂ ( ω ) |2 dω . Then , for any τ ∈ Z2 , ‖xτ ∗ ψ − e −iξ·τ ( x ∗ ψ ) ‖∞ ≤ σ |τ | ‖x‖2 . ( 3 ) The proof is in Appendix B . This theorem proves that if |τ | 1/σ then xτ ∗ ψ ≈ e −iξ·τx ∗ ψ . In this case , a translation by τ produces a phase shift by ξ · τ . Phase collapse and stationarity We define a phase collapse as the elimination of the phase of a complex number with a modulus . We now show that phase collapses improve linear classification of classes that are invariant to translations or to local translations . Each image class indexed by y may be represented by a random vector Xy , whose realizations are the class samples . To achieve linear separation , it is sufficient that class means E [ Xy ] are sufficiently separated , and that each class mean can be estimated from a single realization , with a sufficiently small variance E [ ∥∥Xy − E [ Xy ] ∥∥2 ] ( Hastie et al. , 2009 ) . The goal of classification is to find a representation of the input images in which these properties hold . To simplify the analysis , we consider the particular case where each class y is invariant to translations . In this case , each random vector Xy is stationary , which means that its probability distribution is invariant to translations . Equation ( 2 ) then implies that the phases of Fourier coefficients of Xy are uniformly distributed in [ 0 , 2π ] , leading to E [ X̂y ( ω ) ] = 0 for ω 6= 0 . If class-average pixel values of images are the same , then E [ X̂y ( 0 ) ] does not depend on y either . A linear classifier thus can not discriminate such translation-invariant classes , as class means are all equal . A phase collapse is then necessary to create non-zero means from these Fourier coefficients . It defines translation-invariant coefficients : |x̂τ | = |x̂| . It improves linear discriminability of stationary classes , because E [ |X̂y ( ω ) | ] may be different for different y . However , one can not estimate accurately the value of E [ |X̂y ( ω ) | ] at all frequencies ω from a single image . Indeed , there are as many frequencies as the number of image pixels . Moreover , the Fourier transform is unstable to small deformations ( Bruna and Mallat , 2013 ) at high frequencies . This explains the limited performances of a Fourier transform modulus for linear classification . Fourier modulus descriptors can be improved by using filters ψ that have a localized support in space . If ∑ u ψ ( u ) = 0 , then E [ Xy ∗ ψ ] = 0 if Xy is stationary . Theorem 1 shows that the phase of Xy ∗ ψ is again uniformly distributed in [ 0 , 2π ] . A phase collapse similarly computes approximate invariants to small translations : |xτ ∗ ψ| ≈ |x ∗ ψ| with an error bounded by σ |τ | ‖x‖2 . This representation improves linear classification by creating different non-zero class means E [ |Xy ∗ψ| ] . It defines better descriptors than Fourier coefficients because E [ |Xy ∗ ψ| ] can be accurately estimated from a single image with a spatial average of |Xy ∗ ψ ( u ) | , if the support of ψ is not too large . Moreover , these expected values are stable to deformations when the support of ψ is small ( Bruna and Mallat , 2013 ) . Image classes are usually not invariant to global translations . For instance , subjects are generally centered or the sky is usually at the top . However , the classes are often invariant to local translations , up to an unknown maximum scale . This is captured by the notion of local stationarity , which means that the probability distribution of Xy is nearly invariant to translations smaller than some maximum scale ( Priestley , 1965 ) . The above discussion remains valid if Xy is only locally stationary over a domain larger than the support of ψ . The use of so-called “ windowed absolute spectra ” E [ ∣∣Xy ∗ ψ∣∣ ] for locally stationary processes has previously been studied in Tygert et al . ( 2016 ) . Complex versus real in CNNs : phases and ReLUs The use of complex numbers is a mathematical abstraction to diagonalize translations , which are captured by complex phases . It provides a mathematical interpretation of filtering operations performed on real numbers . We show that a real network can still implement complex phase collapses . In the first layer of a CNN , one can observe that filters are usually oscillatory patterns with small supports , where some filters have nearly the same orientations and frequency but have a phase shifted by some α ( Krizhevsky et al. , 2012 ) . We reproduce in Appendix F a figure from Shang et al . ( 2016 ) which evidences this phenomenon . This means that such filters can be written ψα = Re ( e −iαψ ) for a single complex filter ψ and different α. Grouping such channels specifies the complex filter ψ . A CNN with complex filters is thus a structured real-valued CNN , whose channels correspond to phase shifts of these complex filters . This structure simplifies the mathematical interpretation of non-linearities by explicitly defining the phase , which is otherwise a hidden variable relating multiple filter outputs within each layer . A phase collapse is computed with a modulus . It can be calculated by real-valued CNNs by applying ReLUs without biases over filters having different phases : ψα = Re ( e −iαψ ) . Indeed , for any real-valued signal x , we have : |x ∗ ψ| = 1 2 ∫ π −π ReLU ( x ∗ ψα ) dα . ( 4 ) Furthermore , this integral is well approximated by a sum over 4 phases , allowing to compute complex moduli with ReLUs without biases over real-valued CNN filters . See Appendix A for a proof of eq . ( 4 ) and its approximation . Thresholding and sparsity versus phase collapse The action of a ReLU can be linearly decomposed into an elimination of sign and a soft-thresholding , which attenuates the amplitude . As both can improve linear classification in different cases , an important issue is to understand which property is important for classification . If z ∈ R and b ≥ 0 , then the even part of ReLU ( z − b ) is ReLU ( z − b ) + ReLU ( −z − b ) , which is an absolute value with a dead-zone [ −b , b ] . When b = 0 , it becomes a phase collapse |z| . The odd part is a soft-thresholding ρb ( z ) = ReLU ( z − b ) − ReLU ( −z − b ) . A soft-thresholding operator over a complex z = |z| eiϕ reduces the amplitude by b while preserving the phase : ρb ( z ) = max ( |z| − b , 0 ) e iϕ . ( 5 ) Similarly to its real counterpart , this amounts to defining the complex soft-thresholding as the proximal operator of the complex modulus ( Yang et al. , 2012 ) : ρb ( z ) = arg min w∈C b|w|+ 1 2 |w − z|2 . ( 6 ) Soft-thresholdings and phase collapses have opposite properties , since soft-thresholdings preserve the phase while attenuating the amplitude , whereas phase collapses preserve the amplitude while eliminating the phase . We have explained how phase collapses can improve the classification accuracy of locally stationary processes by separating class means E [ ∣∣Xy ∗ ψ∣∣ ] . In contrast , if Xy ∗ ψ has a distribution which is symmetric around 0 then E [ ρb ( Xy ∗ ψ ) ] = 0 for all b because the thresholding preserves the phase and hence this symmetry around 0 . This symmetry is often observed when ∑ u ψ ( u ) = 0 , in which case class means are not separated by a thresholding . If E [ Xy ∗ ψ ] 6= 0 andXy ∗ψ is sparse , then a soft-thresholding ofXy ∗ψ may reduce the variance of class mean estimators ( Donoho and Johnstone , 1994 ) . In this case , it can improve linear classification . Coefficients below the threshold may be assimilated to unnecessary “ clutter ” which is set to 0 . To improve classification , convolutional filters must produce high-amplitude coefficients corresponding to discriminative “ features ” , whereas small-amplitude clutter coefficients are eliminated ( Zarka et al. , 2021 ) . The dimensionality of sparse representations can also be reduced with random filters which implement a form of compressed sensing ( Donoho , 2006 ; Candes et al. , 2006 ) . The interpretation of CNNs as compressed sensing machines with random filters has been studied ( Giryes et al. , 2015 ) , but it never led to classification results close to e.g . ResNet accuracy . On the contrary , we shall see that the use of phase collapse alone reaches this performance .

Summary: This is a very interesting paper that is based on scattering networks, for which it shows that the so called ‘phase collapse’ leads to state-of-the-art results on par with modern architectures, like ResNet. To derive this, the paper shows that having neural networks on complex numbers is similar to a structure deep network (like with wavelet filters) on the reals. Phase collapse is then when there is an operation, like the modulus (absolute value function), which eliminates the phase from the complex number and maintains only the amplitude. All in all, the paper shows that maintaining the amplitude while eliminating the phase, is what brings the high accuracies, while the reverse (keeping the phase, eliminating the amplitude) yields terrible accuracies. As a corollary, I would say that the paper explains why non-linearities, like ReLUs, are so successful.