Frequency Pooling: Shift-Equivalent and Anti-Aliasing Down Sampling

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have achieved great success on image processing Goodfellow et al . ( 2016 ) , natural language processing Yin et al . ( 2017 ) , game playing Mnih et al . ( 2013 ) and so on . One of the reasons is that convolutions utilize the shift-equivalent prior of signals . Modern CNNs include not only convolutional layers but also down sampling or pooling layers . As an important part of CNNs , pooling layers are used to reduce spatial resolution of feature maps , aggregate spatial information , and reduce computational and memory cost . Based on classical Nyquist sampling theorem Nyquist ( 1928 ) , the sampling rate must be at least as twice as the highest frequency of a signal . Otherwise , frequency aliasing will appear , i.e . highfrequencies of the signal alias into low-frequencies . This leads sub-optimal when reconstruction the signals and misleads the following processing because orthogonal components are mixed again . To anti-alias , a traditional solution is that applying low-pass filter to the signal before down sampling it . Following the spirit of blurred down sampling , early CNNs use average pooling to achieve down sampling Lecun et al . ( 1989 ) . Later , empirical evidence suggests max-pooling Scherer et al . ( 2010 ) and stride convolutions Long et al . ( 2015 ) provide better performance . They are widely used in CNNs but they don ’ t consider about anti-aliasing . Shift-equivalent is another expected property of pooling . Otherwise it will destroy the shiftequivalent of CNNs and thus the shift-equivalent prior of signals is not fully utilized . Unfortunately , most commonly used poolings are not shift-equivalent . Worse , small shifts in the input can drastically change the output when stacking multiple max-pooling or stride convolutions Engstrom et al . ( 2017 ) ; Azulay & Weiss ( 2018 ) ; Zhang ( 2019 ) . Shift-equivalent is believed to be a fundamental property of CNNs . However , the fact that CNNs with poolings are not shift-equivalent has been ignored by the community . Until recently , Zhang ( 2019 ) propose anti-aliasing pooling ( AA-pooling ) by low-pass filtering before down sampling . They observe increased accuracy and better generalization on image classification when low-pass filtering is integrated correctly . Specifically , they decompose a pooling operator with down sampling factor s into two parts : a pooling operator with factor 1 and a blur filter with factor s. Although AApooling reduces the aliasing effects and makes the outputs more stable w.r.t input shifts , it is neither strict shift-equivalent nor anti-aliasing in theory . In this paper , we propose a strict shift equivalent and anti-aliasing pooling in theory . We first transform a signal or image into frequency domain via Discrete Fourier Transform ( DFT ) . Then we only retain its low frequencies , i.e . the frequencies which are smaller than Nyquist sampling rate . Finally , we transform the low frequencies back into time domain via inverse DFT . We call our method frequency pooling ( F-pooling ) . Note that a similar method is proposed in Ryu et al . ( 2018 ) . However , they only focus on classification accuracy . Shift equivalent of their method is not evaluated in both theory and practice . Compared with previous works , the novelties and contributions of Fpooling are summarized as follows : • We formally define shift-equivalent of functions which contain down sampling operations . A suitable up sampling operation U must be involved in the definition . Without it , the definition for discrete signal is ill-posed . We believe a formal mathematical treatment has great value for further research . • We prove that F-pooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction . We also prove that F-pooling is shift equivalent . The up sampling operation U plays an important rule in our proofs . To best of our knowledge , F-pooling is the first pooling method which has those properties . • Experiments on CIFAR-100 and a subset of ImageNet demonstrate that F-pooling remarkably increases accuracy and robustness w.r.t shifts of commonly-used network architectures . Moreover , the shift consistency of F-pooling is improved more when we replace zero padding of convolutions with circular padding . 2 METHOD . In this section , we first define shift-equivalent for CNNs formally . Then we describe F-pooling in detail and prove its properties . Finally we discuss our implementation and some practical issues . 2.1 DEFINITION OF SHIFT-EQUIVALENT . Denote X as an h0 × w0 × c tensor where h0 , w0 , and c are the height , width , and number of channels of X respectively . Denote Y as an h1×w1× c tensor . We suppose h0 > h1 and w0 > w1 . M : X → Y is a pooling operation and U : Y → X is a up sampling operation . We say M is shift-equivalent if and only if Shift4h,4w ( UMX ) =UMShift4h,4w ( X ) ( 1 ) ∀ ( 4h,4w ) , ∃U That is , if there is a suitable up sampling operation U which makes UM and Shift4h,4w commutable , thenM is shift-equivalent . The shift operation is required to be circular or periodic . When a shifted element hits the edge , it rolls to other side . Shift4h,4w ( Xa , b ) = X ( a+4h ) % h0 , ( b+4w ) % w0 ( 2 ) where % is the modulus function . Our definition of shift-equivalent is similar with the one in Zhang ( 2019 ) . The difference is that we introduce an up sampling U to make this definition strict . Without introducing U , the definition of shift-equivalent for pooling is ill-posed . SupposeM down samples signals by a factor s. Without introducing U , one may define shift-equivalent as follows : Shift4h/s,4w/s · M ( X ) =M · Shift4h,4w ( X ) , ∀ ( 4h,4w ) ( 3 ) However , Shift4h/s,4w/s is not operable for discrete signals when 4h % s 6= 0 or 4h % s 6= 0 . To make the shifted element which is not on lattice operable , one must interpolate or up sample the down sampled signals . This leads to the definition in equation 1 . As in equation 1 , to makeM shift-equivalent , one must find the corresponding up sampling operation U . For a givenM , the first line in equation 1 may hold for some up sampling operations . But it may not hold for other up sampling operations . 2.2 F-POOLING . The basic idea of F-pooling is removing the high frequencies of signals and reconstructing the signals only using the low frequencies . In this paper , high frequencies mean the frequencies which are beyond Nyquist frequency , i.e . half of signal resolution . And low frequencies mean the frequencies which are not higher than Nyquist frequency . To remove high frequencies , we first transform signals into frequency domain via DFT . Then only the low frequencies are retained . Finally , we transform the low frequencies back to time domain via inverse DFT ( IDFT ) . Figure 2 gives an illustration of the forward process of F-pooling . Since 2D ( inverse ) DFT can be decomposed into two 1D ( inverse ) DFT , we provide the formal representation of F-pooling in 1D case . Denote x ∈ RN as time domain signal and y ∈ CM as frequency domain signal . Without loss of generality , we suppose M is even . FN ∈ CN×N is the so-called DFT-matrix : FN = ω0·0N ω 0·1 N . . . ω 0· ( N−1 ) N ω1·0N ω 1·1 N . . . ω 1· ( N−1 ) N ... ... . . . ... ω ( N−1 ) ·0 N ω ( N−1 ) ·1 N . . . ω ( N−1 ) · ( N−1 ) N ( 4 ) where ωN = e−2πi/N . F∗N is the inverse DFT-matrix . By definition , 1 NFNF ∗ N = I where I is identity matrix . We denote Tµ as a operation which selects the first µ rows and the last µ rows of a matrix . Tµ ( x ) = [ x1 : µ ; xN−µ+1 : N ] ( 5 ) That is we select the basis of frequencies ranged in [ −µ , µ ) , due to periodicity of DFT . When applying Tµ to a signal , we obtain its lowest µ frequencies . Then the function of F-pooling for 1D signals is represented as : y = 1 N F∗MTM 2 FNx def = Px ( 6 ) where P ∈ CM×N is the transform matrix of F-pooling . Now we formalize F-pooling for CNNs . Recall that X is an h0×w0×c tensor and Y is an h1×w1×c tensor . We apply F-pooling to each channel of X. Y : , : ,i = PhX : , : ,iP ∗ w , i ∈ [ 1 , c ] ( 7 ) where Ph ∈ Ch1×h0 and Pw ∈ Cw1×w0 are the transform matrices of F-pooling along vertical direction and horizontal direction respectively . Since F-pooling can be represented as two times matrix-matrix multiplications , its back propagation rule is easily derived . As mentioned in section 2.1 , the choice of up sampling operation U is important . In this paper , U is set to the inverse process of F-pooling . Specifically , we transform a signal into frequency domain . Then we zero pad the transformed signal to match the resolution of output . Finally , we transform it back to time domain . This process can also be represented by matrix multiplications . For 1D signals , we have : x = 1 M F∗NZM 2 FMy ( 8 ) where is Z is the zero padding operation . Zµ = [ x1 : µ ; 0 ; xM−µ+1 : M ] ( 9 ) The formulations of F-pooling and U are easily extended to 2D case . 2.3 OPTIMAL ANTI-ALIASING DOWN SAMPLING . In this section , we prove that F-pooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction given U . We focus on 1D case here . Given U , the optimal antialiasing down sampling is obtained by solving the following problem : min M ||UMx− x||22 , s.t . M∈ A ( 10 ) where A is a set of all possible anti-aliasing down sampling operations . That is , we hope to find an anti-aliasing down sampling which minimizes the reconstruction error . Based on Nyquist sampling theory , M must remove high frequencies of signals and this holds for F-pooling . We focus on the optimality of F-pooling . We decompose x into low frequencies part xl and high frequencies part xh , i.e . xl = 1NF ∗ NDM2 FNx and xh = x − xl . Dµ is a diagonal matrix whose the first µ and the last µ diagonal elements equal to 1 while others equal to 0 . ||UMx− x||22 = ||UMx− xl − xh||22 ( 11 ) = ||UMx− xl||22 + ||xh||22 + 〈UMx , xh〉 − 〈xl , xh〉 ( 12 ) = ||UMx− xl||22 + ||xh||22 ( 13 ) equation 13 holds because the third term and the forth term of equation 12 equal to 0 . M removes high frequencies and U doesn ’ t introduce new frequencies . Thus UMx only contains low frequencies . Due to the orthogonality of frequency spectrum , 〈UMx , xh〉 = 0 . Similarly , 〈xl , xh〉 = 0 . WhenM is F-pooling , we have UMx = 1 MN F∗NZM 2 FMF ∗ MTM 2 FNx ( 14 ) = 1 N F∗N ( ZM 2 TM 2 ) FNx ( 15 ) = 1 N F∗NDM 2 FNx def = xl ( 16 ) equation 16 holds due to the definition of operations T , Z , and D. See appendix A for proof . Since the fist term of equation 13 equals to 0 , equation 10 is minimized . Thus , we have proved that Fpooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction . We choose reconstruction optimality because : 1 ) we believe reconstruction optimality relates to the final performance , e.g . classification optimality for image classification . If we accept that the feature extracted by previous convolution layers is useful , then it is best to keep it as much as possible for the current pooling layer . Prior works have shown that using self reconstruction loss as an auxiliary is helpful for classification Rasmus et al . ( 2015 ) ; 2 ) It is difficult to directly define classification optimality for an intermediate layer .

Summary: This paper proposed "F pooling" for Frequency Pooling, which is a pooling operation satisfying shift equivalence and anti-aliasing properties. The method is very simple: first, transform the input 1D/2D signal into the spectrum domain based on discrete Fourier transform (DFT), then cut the high-frequencies, then transform back to the time domain using the inverse DFT. The method can be implemented using FFT and auto differentiation frameworks. The method is tested on Resnet/Desnet on CIFAR-100 and subsets of ImageNet, showing better performance than the original models.