Phase Collapse in Neural Networks

1 INTRODUCTION . CNN image classifiers progressively eliminate spatial variables through iterated filterings and subsamplings , while linear classification accuracy improves as depth increases ( Oyallon , 2017 ) . It has also been observed numerically that CNNs concentrate training samples of each class in small separated regions of a progressively lower-dimensional space . It can ultimately produce a neural collapse ( Papyan et al. , 2020 ) , where all training samples of each class are mapped to a single point . In this case , the elimination of spatial variables comes with a collapse of within-class variability and perfect linear separability . This increase in linear classification accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases . A difficulty in understanding the underlying mathematics comes from the flexibility of ReLUs . Any non-linearity can indeed be approximated by a linear combination of biased ReLUs . Many papers interpret iterations on ReLUs and linear operators as sparse code computations ( Sun et al. , 2018 ; Sulam et al. , 2018 ; 2019 ; Mahdizadehaghdam et al. , 2019 ; Zarka et al. , 2020 ; 2021 ) . We show that it is a different mechanism , called phase collapse , which underlies the increase in classification accuracy of these architectures . A phase collapse is the elimination of phases of complex-valued network coefficients with a modulus , which we show to concentrate spatial variability . This is demonstrated by introducing a structured convolutional neural network , with wavelet filters and no biases . Section 2 introduces and explains phase collapses , and contrasts them with thresholding operators . Complex-valued representations are used because they reveal the mathematics of spatial variability . Indeed , translations are diagonalized in the Fourier basis , where they become a complex phase shift . Invariants to translations are computed with a modulus which collapses the phases of this complex representation . Section 2 explains how this can improve linear classification . Phase collapses can also be calculated with ReLUs and real filters . A CNN with complex-valued filters is indeed just a particular instance of a real-valued CNN , whose channels are paired together to define complex numbers . Section 3 demonstrates the role of phase collapse in deep classification architectures , as opposed to thresholdings or more general amplitude reduction operators . It introduces a Learned Scattering network with phase collapses . This network applies a learned 1× 1 convolutional complex operator Pj on each layer xj , followed by a complex wavelet filtering operator W and a phase collapse : xj+1 = |WPjxj | . ( 1 ) It does not use biases , which are also not needed in deep networks for image denoising ( Mohan et al. , 2019 ) . This network architecture is illustrated in Figure 1 . With the addition of skip-connections , we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10 . On the contrary , the accuracy is considerably reduced when replacing the phase collapse modulus by soft-thresholdings with learned biases , which can enforce sparsity but do not modify the phase . This is also true of more general phase-preserving non-linearities . Section 4 explains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability . On the opposite , the improvements in classification accuracy of successive sparse code computations are shown to quickly saturate . The main contribution of this paper is a demonstration that the classification accuracy of deep neural networks mostly relies on phase collapses , which are sufficient to linearly separate the different classes on complex image databases . This is captured by the Learned Scattering architecture which reaches ResNet-18 accuracy on ImageNet and CIFAR-10 . We also show that phase collapses are necessary to reach this accuracy , by showing numerically and theoretically that iterating phase-preserving non-linearities leads to a significantly worse performance . 2 PHASE COLLAPSE VERSUS THRESHOLDINGS AND RELUS . Deep convolutional classifiers achieve linear separation of image classes . We show that linear classification on raw images has a poor accuracy because image classes are invariant to local translations . This geometric within-class variability takes the form of random phase fluctuations , which induce zero class means . To improve classification accuracy , non-linear operators must therefore eliminate these fluctuations with phase collapses , so as to separate class means . We also analyze the effect of other non-linearities such as soft-thresholdings and ReLUs . Translations and phase shifts Translations capture the spatial topology of the grid on which the image is defined . They are transformed into phase shifts by a Fourier transform . We prove that this remains approximately valid for images convolved with appropriate complex filters . Let x be an image indexed by u ∈ Z2 . We write xτ ( u ) = x ( u− τ ) the translation of x by τ . It is diagonalized by the Fourier transform x̂ ( ω ) = ∑ u x ( u ) e −iω·u , which creates a phase shift : x̂τ ( ω ) = e −iω·τ x̂ ( ω ) . ( 2 ) This diagonalization explains the need to introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities . Computations can however be carried with real numbers , as we will show . A Fourier transform is computed by filtering x with complex exponentials eiω·u . One may replace these by complex filters ψ that are localized in space and in the Fourier domain . The following theorem proves that small translations can still be approximated by a phase shift in this case . A convolution is written ∗ . Theorem 1 . Let ψ : Z2 → C be a filter with ‖ψ‖2 = 1 , whose center frequency ξ and bandwidth σ are defined by : ξ = 1 ( 2π ) 2 ∫ [ −π , π ] 2 ω |ψ̂ ( ω ) |2 dω and σ2 = 1 ( 2π ) 2 ∫ [ −π , π ] 2 |ω − ξ|2|ψ̂ ( ω ) |2 dω . Then , for any τ ∈ Z2 , ‖xτ ∗ ψ − e −iξ·τ ( x ∗ ψ ) ‖∞ ≤ σ |τ | ‖x‖2 . ( 3 ) The proof is in Appendix B . This theorem proves that if |τ | 1/σ then xτ ∗ ψ ≈ e −iξ·τx ∗ ψ . In this case , a translation by τ produces a phase shift by ξ · τ . Phase collapse and stationarity We define a phase collapse as the elimination of the phase of a complex number with a modulus . We now show that phase collapses improve linear classification of classes that are invariant to translations or to local translations . Each image class indexed by y may be represented by a random vector Xy , whose realizations are the class samples . To achieve linear separation , it is sufficient that class means E [ Xy ] are sufficiently separated , and that each class mean can be estimated from a single realization , with a sufficiently small variance E [ ∥∥Xy − E [ Xy ] ∥∥2 ] ( Hastie et al. , 2009 ) . The goal of classification is to find a representation of the input images in which these properties hold . To simplify the analysis , we consider the particular case where each class y is invariant to translations . In this case , each random vector Xy is stationary , which means that its probability distribution is invariant to translations . Equation ( 2 ) then implies that the phases of Fourier coefficients of Xy are uniformly distributed in [ 0 , 2π ] , leading to E [ X̂y ( ω ) ] = 0 for ω 6= 0 . If class-average pixel values of images are the same , then E [ X̂y ( 0 ) ] does not depend on y either . A linear classifier thus can not discriminate such translation-invariant classes , as class means are all equal . A phase collapse is then necessary to create non-zero means from these Fourier coefficients . It defines translation-invariant coefficients : |x̂τ | = |x̂| . It improves linear discriminability of stationary classes , because E [ |X̂y ( ω ) | ] may be different for different y . However , one can not estimate accurately the value of E [ |X̂y ( ω ) | ] at all frequencies ω from a single image . Indeed , there are as many frequencies as the number of image pixels . Moreover , the Fourier transform is unstable to small deformations ( Bruna and Mallat , 2013 ) at high frequencies . This explains the limited performances of a Fourier transform modulus for linear classification . Fourier modulus descriptors can be improved by using filters ψ that have a localized support in space . If ∑ u ψ ( u ) = 0 , then E [ Xy ∗ ψ ] = 0 if Xy is stationary . Theorem 1 shows that the phase of Xy ∗ ψ is again uniformly distributed in [ 0 , 2π ] . A phase collapse similarly computes approximate invariants to small translations : |xτ ∗ ψ| ≈ |x ∗ ψ| with an error bounded by σ |τ | ‖x‖2 . This representation improves linear classification by creating different non-zero class means E [ |Xy ∗ψ| ] . It defines better descriptors than Fourier coefficients because E [ |Xy ∗ ψ| ] can be accurately estimated from a single image with a spatial average of |Xy ∗ ψ ( u ) | , if the support of ψ is not too large . Moreover , these expected values are stable to deformations when the support of ψ is small ( Bruna and Mallat , 2013 ) . Image classes are usually not invariant to global translations . For instance , subjects are generally centered or the sky is usually at the top . However , the classes are often invariant to local translations , up to an unknown maximum scale . This is captured by the notion of local stationarity , which means that the probability distribution of Xy is nearly invariant to translations smaller than some maximum scale ( Priestley , 1965 ) . The above discussion remains valid if Xy is only locally stationary over a domain larger than the support of ψ . The use of so-called “ windowed absolute spectra ” E [ ∣∣Xy ∗ ψ∣∣ ] for locally stationary processes has previously been studied in Tygert et al . ( 2016 ) . Complex versus real in CNNs : phases and ReLUs The use of complex numbers is a mathematical abstraction to diagonalize translations , which are captured by complex phases . It provides a mathematical interpretation of filtering operations performed on real numbers . We show that a real network can still implement complex phase collapses . In the first layer of a CNN , one can observe that filters are usually oscillatory patterns with small supports , where some filters have nearly the same orientations and frequency but have a phase shifted by some α ( Krizhevsky et al. , 2012 ) . We reproduce in Appendix F a figure from Shang et al . ( 2016 ) which evidences this phenomenon . This means that such filters can be written ψα = Re ( e −iαψ ) for a single complex filter ψ and different α. Grouping such channels specifies the complex filter ψ . A CNN with complex filters is thus a structured real-valued CNN , whose channels correspond to phase shifts of these complex filters . This structure simplifies the mathematical interpretation of non-linearities by explicitly defining the phase , which is otherwise a hidden variable relating multiple filter outputs within each layer . A phase collapse is computed with a modulus . It can be calculated by real-valued CNNs by applying ReLUs without biases over filters having different phases : ψα = Re ( e −iαψ ) . Indeed , for any real-valued signal x , we have : |x ∗ ψ| = 1 2 ∫ π −π ReLU ( x ∗ ψα ) dα . ( 4 ) Furthermore , this integral is well approximated by a sum over 4 phases , allowing to compute complex moduli with ReLUs without biases over real-valued CNN filters . See Appendix A for a proof of eq . ( 4 ) and its approximation . Thresholding and sparsity versus phase collapse The action of a ReLU can be linearly decomposed into an elimination of sign and a soft-thresholding , which attenuates the amplitude . As both can improve linear classification in different cases , an important issue is to understand which property is important for classification . If z ∈ R and b ≥ 0 , then the even part of ReLU ( z − b ) is ReLU ( z − b ) + ReLU ( −z − b ) , which is an absolute value with a dead-zone [ −b , b ] . When b = 0 , it becomes a phase collapse |z| . The odd part is a soft-thresholding ρb ( z ) = ReLU ( z − b ) − ReLU ( −z − b ) . A soft-thresholding operator over a complex z = |z| eiϕ reduces the amplitude by b while preserving the phase : ρb ( z ) = max ( |z| − b , 0 ) e iϕ . ( 5 ) Similarly to its real counterpart , this amounts to defining the complex soft-thresholding as the proximal operator of the complex modulus ( Yang et al. , 2012 ) : ρb ( z ) = arg min w∈C b|w|+ 1 2 |w − z|2 . ( 6 ) Soft-thresholdings and phase collapses have opposite properties , since soft-thresholdings preserve the phase while attenuating the amplitude , whereas phase collapses preserve the amplitude while eliminating the phase . We have explained how phase collapses can improve the classification accuracy of locally stationary processes by separating class means E [ ∣∣Xy ∗ ψ∣∣ ] . In contrast , if Xy ∗ ψ has a distribution which is symmetric around 0 then E [ ρb ( Xy ∗ ψ ) ] = 0 for all b because the thresholding preserves the phase and hence this symmetry around 0 . This symmetry is often observed when ∑ u ψ ( u ) = 0 , in which case class means are not separated by a thresholding . If E [ Xy ∗ ψ ] 6= 0 andXy ∗ψ is sparse , then a soft-thresholding ofXy ∗ψ may reduce the variance of class mean estimators ( Donoho and Johnstone , 1994 ) . In this case , it can improve linear classification . Coefficients below the threshold may be assimilated to unnecessary “ clutter ” which is set to 0 . To improve classification , convolutional filters must produce high-amplitude coefficients corresponding to discriminative “ features ” , whereas small-amplitude clutter coefficients are eliminated ( Zarka et al. , 2021 ) . The dimensionality of sparse representations can also be reduced with random filters which implement a form of compressed sensing ( Donoho , 2006 ; Candes et al. , 2006 ) . The interpretation of CNNs as compressed sensing machines with random filters has been studied ( Giryes et al. , 2015 ) , but it never led to classification results close to e.g . ResNet accuracy . On the contrary , we shall see that the use of phase collapse alone reaches this performance .

Summary: This paper studies within-class variability which reduces along the layers of deep neural networks. They mainly question the effect of sparsity and soft-thresholding introduced by ReLU. They show that these classification improvements by eliminating spatial within-class variabilities rather come from a phase collapse, which eliminates the phase of network coefficients. Eliminating the phase of zero-mean filters improves the separation of class means, hence increase in classification accuracy. They introduced a complex-valued neural network in which spatial filters are defined as complex multiscale wavelets and learning is reduced to $1 \times 1$ complex filters across channels. Their results show that such a network is able to reach ResNet-18 performance on CIFAR10 and ImageNet.