Frequency Pooling: Shift-Equivalent and Anti-Aliasing Down Sampling
Authors: Zhendong Zhang
Cheolkon Jung
Venue:  (2020)
Keywords: pooling
anti-aliasing
shift-equivalent
frequency

Abstract: Convolutional layer utilizes the shift-equivalent prior of images which makes it a great success for image processing. However, commonly used down sampling methods in convolutional neural networks (CNNs), such as max-pooling, average-pooling, and strided-convolution, are not shift-equivalent. This destroys the shift-equivalent property of CNNs and degrades their performance. In this paper, we propose a novel pooling method which is \emph{strict shift equivalent and anti-aliasing} in theory. This is achieved by (inverse) Discrete Fourier Transform and we call our method frequency pooling. Experiments on image classifications show that frequency pooling improves accuracy and robustness w.r.t shifts of CNNs. 

Under review as a conference paper at ICLR 2020
FREQUENCY POOLING: SHIFT-BQUIVALENT AND
DOWN SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional layer utilizes the shift-equivalent prior of images which makes it a great success for image processing. However, commonly used down sampling methods in convolutional neural networks (CNNs), such as max max-pooling, average- pooling, and stride convolution, are not shift-equivalent. This destroys the shift- equivalent property of CNNs and degrades their performance. In this paper, we provide a formal definition of shift-equivalent when down sampling involved. We propose strict shift equivalent and anti-aliasing pooling method. This is achieved by (inverse) Discrete Fourier Transform and we call our method fre- queney pooling. Experiments image classifications show that frequency pool- ing improves accuracy and robustness w.r.t shifts of CNNs.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved great success on image processing Goodfel- low et al. (2016), natural language processing Yin et al. (2017), game playing Mnih et al. (2013) and so on. One of the reasons is that convolutions utilize the shift-equivalett prior of signals. Modern CNNs include not only convolutional layers but also down sampling or pooling layers. As an im- portant part CNNs, pooling layers are used reduce spatial resolution of feature maps, aggregate spatial information, and reduce computational and memory cost.
Based on classical Nyquist sampling theorem Nyquist (1928), the sampling rate must be least as twice as the highest frequency of a signal. Otherwise, frequency aliasing will appear, i.e. high- frequencies the signal alias into low-frequencies. This leads sub-optimal when reconstruction the signals and misleads the following processing because orthogonal components anti-alias, a traditional solution is that applying low-pass filter to the signal before down sampling it. Following the spirit of blurred down sampling, early CNNs use average pooling to achieve down sampling Lecun et al. (1989). Later, empirical evidence suggests max- max-pooling [ccheeer et al. (2010) and stride convolutions Long al. (2015) provide better performance. They are widely used in CNNs but they don't t consider about anti-aliasing.
mixed again. To
CemplingLLecun
Shift-equivalent is another expected property of pooling. Otherwise it will destroy the shift- equivalent of CNNs and thus the shift-equivalent prior of signals is not fully utilized. Unfortunately, most commonly used poolings are not shift-equivalent. Worse, small shifts in the input can drasti- cally change the output when stacking multiple max-pooling or stride convolutions Engstrom et al. 2017 Azulay & Weiss (2018;; Zhang (2019).
Shift-equivalent is believed to be fundamental property of CNNs. However, the fact that CNNs with poolings are not shift--qquvalent has been ignored by the community. Until recently, Zhang 2019) propose anti-aliasing pooling (AA-pooling) by low-pass filtering before down sampling. They observe increased accuracy and better generalization on image classification when low-pass filtering is integrated correctly. Specifically, they decompose pooling operator with down sampling factor into two parts: a pooling operator with factor 1 and a blur filter with factor s. Although AA- pooling reduces the aliasing effects and makes the outputs more stable w.r.t input shifts, it is neither strict hhuit-equivleent nor anti-aliasing in theory.
In this paper, propose strict shift equivalent and anti-aliasing pooling in theory. We first transform a signal or image into frequency domain via Discrete Fourier Transform (DFT). Then only retain its low frequencies, i.e. the frequencies which are smaller than Nyquist sampling

---

Under review as a conference paper at ICLR 2020
PW
116
14
(a) Max-pooling
(b) Avg-pooling
(c) F-pooling
Figure 1: Simple tests of shift-equivalent. Blue lines are obtained by pooling, up sampling, and shift order. Orange lines are obtained by shift, pooling, and up sampling in order. All Pooling operations down sample original signals by a factor 4. The shift operation shifts original signals by Best viewed on screen.
pixels. The up sampling operation is to equation g
rate. Finally, we transform the low frequencies back into time domain via inverse DFT. We call our method frequency pooling (F-pooling). Note that a similar method is proposed in [yy et al. (2018). However, they only focus on classification accuracy. Shift equivalent of their method not evaluated in both theory and practice. Compared with previous works, the novelties and contributions of F- pooling are summarized as follows:
We formally define shift-cquivalent of functions which contain down sampling operations. A suitable up sampling operation must be involved in the definition. Without it, the definition for discrete signal is ill-posed. We believe formal mathematical treatment has great value for further research.
We prove that F-pooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction. We also prove that F-pooling is shift equivalent. The up sampling operation U plays an important rule in our proofs. To best of our knowledge, F-pooling is the first pooling method which has those properties.
Experiments on CIFAR- 100 and subset of ImageNet demonstrate that F-pooling remark- ably increases accuracy and robustness w.r.t shifts of commonly-used network architec- tures. Moreover, the shift consistency of F-pooling is improved more when we replace zero padding of convolutions with circular padding.
METHOD
In this section, we first define shift-equivalent for CNNs formally. Then we describe F-pooling in detail and prove its properties. Finally we discuss our implementation and some practical issues.
2.1 DEFINITION OF SHIFT-EQUVVAEET
Denote X as an ho x wp X c tensor where hgs wo. and c are the height, width, and number of channels of X respectively. Denote Y an hı w1 tensor. We suppose ho hy and wo wi- - X is up sampling operation. We say M is
M: : X + Y is a pooling operation and shift-equivalent if and only if
Shiftar. (MAftAAAMMM (UMX) UMShiftaaaaa(X
(I)
v(Dh, Aw), 3V
That is, there is a suitable up sampling operation 24 which makes UM and Shi [hifl'A Alr com- mutable, then M is shift-cquivalent. The shift operation is required to be circular periodic. When a shifted element hits the edge, it rolls to other side.
Shift (X,b)) = X (00
where % the modulus function. Our definition of chift-equivalent is similar with the one in Zhang 2019). The difference is that we introduce an up sampling 24 make this definition strict. Without
2

---

Under review as a conference paper at ICLR 2020
Image
Frequency
Low Frequency
Image
DFT
Retain frequency
IDFT
Figure 2: An illustration of the forward process of F-pooling. We assume the lowest frequencies are at center.
introducing the definition of shift-equivalent for pooling ill-posed. Suppose M down samples signals by a factor s. Without introducing U1 one may define shift-equivalent as follows:
Shiflah/sbb/s M(X) = M Shiflan,Aw (X), V(Ah, Av)
(3)
However, Shiftshhs is not operable for discrete signals when 44%s * 0 or 4h%s + 0. To make the shifted element which down sampled signals. This leads to the definition in equation]
on lattice operable, one must interpolate or up sample the
As in equation I make M shift-equivalent, one must find the corresponding up sampling opera- tion U. For given M, the first line in equation may hold for some up sampling operations. But may hold for other up sampling operations.
2.2 F-POOLING
The basic idea of F-pooling is removing the high frequencies of signals and reconstructing the sig- nals only using the low frequencies. In this paper, high frequencies mean the frequencies which are beyond Nyquist frequency, i.e. half of signal resolution. And low frequencies mean the frequencies which are not higher than Nyquist frequency. To remove high frequencies, we first transform signals into frequency domain via DFT. Then only the low frequencies retained. Finally, transform the low frequencies back to time domain via inverse DFT (IDFT). Figure gives an illustration of the forward process of F-pooling.
Since 2D (inverse) DFT can be decomposed into two 1D (inverse) DFT, we provide the formal representation of F-pooling in ID case. Denote N time domain signal and y E CM as frequency domain signal. Without loss of generality, we suppose M is even. FN E CNNN is the so-called DFT-matrix:
0-0
0-1
O(N-1) N
w
1.
why
1(N-1)
FN-
N
(N 1
(N
1)-1)
(N WN
()-
N
WN
where WN = 2xiiNN F is the inverse DFT-matrix. By definition, F = I where is identity matrix. We denote Tμ operation which selects the first rows and the last of matrix.
(x)
[X1:2 XN-
That is we select the basis of frequencies ranged in [-p, 4). due to periodicity of DFT. When ap- plying Tn to signal, we obtain its lowest frequencies. Then the function of F-pooling for 1D signals represented as:
FNX def Px
F
(6)
N
where M is the transform matrix F-pooling.
Now we formalize F-pooling for CNNs. Recall that X as ho x we xe tensor and Y is an by x x tensor. We apply F-pooling to each channel of X.
Y ...
P,X. P
i [1,c

---

Under review as a conference paper at ICLR 2020
where Ps E Cha and Pu C x are the transform matrices of F-pooling along vertical direction and horizontal direction respectively. Since F-pooling can be represented as two times matrix-matrix multiplications, its back propagation rule is easily derived.
As mentioned section 2.1 the choice of up sampling operation 24 is important. In this paper. 24 is set to the inverse process of F-pooling. Specifically, we transform a signal into frequency domain. Then we zero pad the transformed signal to match the resolution of output. Finally, we transform back to time domain. This process can also be represented by matrix multiplications. For ID signals, we have:
1 M
Z FMy
F
(8)
where is Z is the zero padding operation.
Zμ [1 0; XM-1+1:M
(9)
The formulations of F-pooling and U are easily extended to 2D case.
2.3 OPTIMAL ANTI-ALIASING DOWN SAMPLING
In this section, prove that F-pooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction given U. We focus on ID case here. Given U. the optimal anti- aliasing down sampling is obtained by solving the following problem:
min |UMx - 8.l. MEA M
(10)
where A is a set of all possible anti-aliasing down sampling operations. That is, we hope to find an anti-aliasing down sampling which minimizes the reconstruction error. Based on Nyquist sampling theory. M must remove high frequencies of signals and this holds for F-pooling. We focus on the optimality of F-pooling. We decompose x into low frequencies part x and high frequencies part Xh, i.e. = n F ae last a diagonal elements equal to while others equal to O.
F x and X
X1- p. is diagonal matrix whose the first μ and the
|uMx - x|1 = |uMMx - X; Xx
(11)
||MMx x:113 + 1|x 11 + (UMx, xn) - (x1,
(12)
1 x:11ē + ||x
(13)
equation 13 holds because the third term and the forth term of equation 12 equal to 0. M removes
high frequencies and  doesn't 't introduce new frequencies. Thus UMx only contains low frequen-
cies. Due to the orthogonality of frequency spectrum, (UMx, Xx) = 0. Similarly, (xi, xn) 0. When M is F-pooling, we have
1 UMx
F
FM
Z
(14)
MN 1
F
M T
FNX
(15)
N
1
FNX del
F N
F
(16)
equation 16 holds due to the definition of operations T, Z, and D. See appendix A for proof. Since the fist term of equation 13 equals to O, equation 10 is minimized. Thus, we have proved that F- pooling is the optimal anti-aliasing down sampling operation from the perspective of reconstruction.
We choose reconstruction optimality because: 1) we believe reconstruction optimality relates to the final performance, e.g. classification optimality for image classification. If we accept that the feature extracted by previous convolution layers is useful, then it is best to keep it as much as possible for the current pooling layer. Prior works have shown that using self reconstruction loss auxiliary is helpful for classification Rasmus et al. (2013); 2) difficult to directly define classification optimality for intermediate layer.
4

---

Under review as a conference paper at ICLR 2020
MaxPooling Stride
Conv Strida
AugPooling Stride
Fopoling
MaxPooling Stride
F-pooling
Canv
F-pooling
F-pooling
Pooling
Average Poolinn
Figure 3: An illustration of how replace max- max-pooling, average pooling, and stride convolution with F-pooling, We follow the settings in /Zhang (2019).
2.4 SHIFT-EQUIVALENT
Based on shift theorem of Fourier transform, we have
FxShifttaxx) Fxx @ Sah
(17)
Shiftan (F = x O SAa
(18)
where S AA E CN whose kth element is e 2 Ah and @ is element- multiplication. Combining
with equation 16 have
UMShiftan())
FiND F FxShiftAnx))
(19)
N F (D FNX
o Sah
(20)
N
= SShfttsh
F
NX
(21)
N
=Shiftan(MMx
(22)
We have proved that F-pooling is shift-equivalent. Note that equation 22 is hold if other up sampling operations are used, such as linear interpolation.
The proof's section 2.3 and 2.4 can be easily extended for 2D signals.
2.5 TRANSITIVITY OF SHIFT-EQUIVALENT
We study the transitivity of shift-equivalent for F-pooling. Denote f as a shift-equivalent function without down sampling or up sampling. straightforward to shown
U.ShiffAs (MMS)))Ls(((x)) Shi (LiL())(UM((())
(23)
That is shift-equivalent of F-pooling is transitive for any shift-equivalen function f without down sampling or up sampling. And it is transitive for F-pooling itself. For example, a stack of two F-pooling layers is still shift-equivalent function.
Unfortunately, this transitivity F-pooling is not hold for shift-equivalent function with down sampling. For example, function of Fpooling - cone relu - Fpooling is not shift- equivalent. This prevents us to obtain strict shift-equivalent deep CNNs.
2.6 PRACTICAL ISSUES
Dealing with imaginary part: generally, the output of F-pooling, i.e. Mx contains both real part and imaginary part. However, for commonly used CNNs, the feature maps must be real. Thus one must ignore the imaginary part of F-pooling. On the other hand. Mx treated as complex in the proofs in section 2.3 and 2.4 If we ignore the imaginary part, F-pooling is no longer the optimal anti-aliasing down sampling (but still anti-aliasing) and F-pooling is no longer strict shiia-equiv
Fortunately, this issue is easy to overcome. When the resolution of down sampled signals is odd, the imaginary part Mx zero. Suppose the resolution is 2 +
& 1, Xn, 81
(24)

---

Under review as a conference paper at ICLR 2020
Table 1: Accuracy and consistency sub-ImageNet
accuracy/consistency
densenet-121
resnet-50
mobilenet-v2
Origin
77.47/82.74
74.44/80.17
73.01/78.08
AA-pon Aooolonn F-pooling
77.14/83.09
76.12/82.26
74.03/79.12
77.56/84.10
76.05/82.63
74.72/80.34
Due to its symmetry, it only contains real part when transforms back into time domain. If the resolution is +
R....... Xo; ...,, Xp Xx+1
(25)
In this case, it contains imaginary part. But we can recover the symmetry and eliminate the imag- inary part by setting Xu+1 to zero. We call this trick odd padding. We show the effects of odd padding in appendix
drawback of odd padding is that more information is lost during down sampling which may reduces accuracy. We don't t use it in this paper.
Compared with equation 24 and equation 25 the imaginary part is introduced by & Thus, the error of ignoring the imaginary part is smaller than ||8,+1112. In practice, we find this effeet is small.
Dealing with zero padding: F-pooling designed to be circular shift equivalence. However, zero padding is commonly used in convolution layers. Convolutional layers with zero padding destroy circular shift-equivalence. Thus, one should expect that using circular padding in convolutional layers will greatly increase the shift consistency of F- F-pooling. In this paper, evaluate the per- formance of F-pooling with both zero-padding and circular padding. We suggest one replaces zero padding with circular padding for better shift-equivalence.
2.7 IMPLEMENTATION
We implement F-pooling in PyTorch Paszke et al. (2017). Theoretically, it is best to implement F-pooling with fast Fourier transform (FFT) and inverse FFT. Suppose F-pooling down samples an hg x We X tensor to a h1 x w1 x tensor. Then its time complexity is cho log( (wyhu)). Unfortunately, we find such implementation in PyTorch is comparatively slow.
Instead, we implement F-pooling via 1 X 1 convolutions. As in equation 7 2D F-pooling can be represented as two times matrix-matrix multiplications along vertical direction and horizontal direction respectively. This is equivalent to 1 x convolutions along vertical direction and direction horizontal. Its time complexity is (h, w()) which is higher than the optimal complexity.
F-pooling requires much more computational costs than average pooling max-pooling, F-pooling faster than a convolutional layer when the resolution of feature maps is smaller than the number of channels, as the situations of image classifications. Moreover, the number of pooling layers is limited compared with the number of convolutional layers. Thus, introducing F-pooling will not introduce too many computational costs into CNNs.
Zhang (019)) claims that it is important to integrate anti-aliasing pooling into CNNs in a correct way. In this paper, follow their settings. Specifically, max-pooling with stride is replaced with a max- max-pooling with stride and an F-pooling with stride s. A convolution with stride is replaced with convolution with stride I and F-pooling with stride s. An average pooling with stride s is replaced with an F-pooling with stride s. See the illustration in figure
EXPERIMENTS
3.1 ID SIGNALS
We test the shift-equivalent of F-pooling on ID signals. In figure I the original signal is a randomly selected row of 512 x 512 image. We apply max-pooling, average pooling, and F-pooling with stride 4 to those signals. As shown in figure F-pooling is perfectly eqiit-equivllent. And average pooling is better than max-pooling from shift-equivalent perspective. odd padding is used here.

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy and consistency CIFAR-100
accuracy/consistency shift argument
densenet-41
resnet- 18
resnet-34
Origin
with 74.90/71.55
75.52/70.21
76.56/72.21
AA-pooling
with 75.55/71.71
77.43/73.08
76.95/73.38
F-pooling
with 75.45/71.91
77.36/73.43
77.68/73.54
Origin
without
71.81/57.27
67.60/45.00
68.11/46.56
AA- AA-pooling
without 73.81/60.29
74.49/58.24
74.00/57.72
F-pooling
without
73.19/60.51
73.93/57.48
73.51/56.54
3.2
IMAGE CLASSIFICATION
we test classification of fow-resolution 32 32 color images. This dataset contains 50k images for training and 10k images for test. Images are classified into one of 100 categories.
CIFAR-100
sub-ImageNet: we then test classification of high-resolution color images. Original ImageNet dataset Russakovsky et al. (2013) contains 1.28M training and 50k validation images, classified into one of 1,000 categories. To reduce the computational resources for training, we use subset of ImageNet (sub-Iman sub-ImmaeNee) in this work. We randomly select 200 categories from 1,000 categories. For each category, we randomly select 500 images. Thus, we collect 100k images for training. And we select corresponding 10k validation images. All models are trained on a single GPU with batchsize 64 and 100 epochs. We decrease the initial learning rate by a factor 10 every 40 epochs. For other hyper pere--paamterss we follow PyTorch's s official training script
We train CIFAR- 100 using resnet He et al. (0016) and densenet Huang et al. (017)) We train sub- ImageNet using resnet, densenet, and mobilenet-v2 [andler al. (2018). Those models are widely used as benchmarks. Max-pooling, average pooling, and stride convolution are covered in those models. We also compare F F-pooling with AA-pooling Zhang (2019). For model trained sub- ImageNet, its first down sampling operation is kept to reduce computational costs. This setting is also used in /hang 2019). The results CIFAR-100 100 are averaged by 3 runs.
We study not only accuracy but also consistency. We follow the metric of consistency used in Zhang (2019): we check how often the network outputs the same classification, given the same image with two different shifts.
Eno.ky 1 {arg max ((Shiftan Sha wo) = arg max
(26)
We only evaluate diagonal shifts in this paper. For CIFAR-100, the number of shifted pixels ranges from -7 For sub- aut-lmageNet, ranges from -63 to 63.
Zero padding: we keep the padding method of convolutions as zero padding. We measure accuracy and consistency of different pooling methods. Results on sub-ImageNNt is shown in table [ and results on CIFAR-100 are shown in table respectively.
Circular padding: as mentioned earlier, circular padding may improve the consistency of F- pooling. Thus we replace the zero padding of convolutions with circular padding. To further evaluate the robustness w.r.t shifts, we use standard deviation (std) of probabilities of the correct label. Std and consistency on sub-ImageNet an CIFAR-100 are shown in table 3 and 4able/4resper respectivel..
Based on those results, we conclude that
F-pooling is significantly and consistently better than original pooling methods in terms of accuracy and shift consistency.
F-pooling is comparable with AA-pooling in term of accuracy and shift consistency with zero padding. With circular padding, F-pooling is significantly better than AA AA-pooling.
Shift consistency of F-pooling improved more when replacing zero padding with eircular padding, especially on CIFAR-100.
https://www..sstrrotto....u/ kriz/cifar.hmml
7

---

Under review as a conference paper at ICLR 2020
Table 3: Std and consisteney on sub- sub-lmageNet with circular padding.
std/consistency densenet- 121
resnet-50
mobilenet-v2
Origin
0.043/90.44
0.051/87.97
0.051/87.71
AA-pooling
0.055/88.21
0.056/87.77
0.059/86.19
F-pooling 0.035/91.88
0.037/91.01
0.041/90.32
Table 4: Std and consistency on CIFAR-100 with circular padding without shift argument.
std/consistency
densenet-41
resnet-
resnet-34
Origin AA-pooling 0.166/66.97
0.111/77.62
0.176/59.93
0.183/58. 94
0.164/65.67
0. 0.180/60.53 0.088/80.90
F-pooling
0.088/82.99
0.073/84.34
RELATED WORKS
Pooling which reduces the resolution of feature maps is an important part of CNNs. Early CNNs Lecun al.) (1989) use average pooling which good for anti-aliasing. Later empirical evidence suggests max- max-pooling Scherer et al. (2100) and strided-convolution Long et al. (2015) provide bet- ter performance. However, small shifts the input can drastically change the output when stacking multiple max-pooling
Engstrom al. ( Azulay & Weiss ( Other variants such as Gsahramm (2014); [e et al. (2015); Lee et al. (2016) (we just list a few of them), focus on extending the functionality of pooling Lee et al. 2016) or making pooling adjusted to variable
strided-convolutee
input size Graham (2014); He et al. (2015).
Recently, Zhang (2019) shows that CNNs will have better shift-equivalent and anti-aliass anti-aliasing prop- erties when low-pass filtering is integrated correctly. However, their method is not strict shift- equivalent and anti-aliasing in theory. Williims & Li (2018) propose Wavelet-pooling, They decom- pose a signal via wavelet transform and retain the lowest sub-band. This process is repeated until the designed down sampling factor met. The spirit of Wavelet-pooling is similar F-pooling. However, they claim that Wavelet-pooling is better than others because it is global transform in- stead of a local transform. They neither focus on shift-equivalent nor prove Wavelet-pooling is shift-equivalent or not.
F-pooling is a complex transformation because DFT and IDFT are involved in it. Many works in- tegrate complex transformations or complex values into neural networks. AAmin & Murase (2009) study single-layered complex-vale -valued neural networks for veal-valued classification problems. Com- plex numbers represented in polar coordinates are more suitable to deal with rotations. Based on nohen et al. (2018) propose spherical CNNs which are rotation-equivalent to deal with signals projected from spherical surface. 'Trabelsi et al. (2018) propose general deep complex CNNs. They adjust batch normalization and non-linear activations for complex CNNs. F-pooling can used in their method without worry about the imaginary part. F-pooling utilizes shift theorem of DFT and achieves shift-equivalent. This is a new success for the combination of complex transformations and neural networks.
5 CONCLUSIONS
In this paper, we have proposed frequency pooling (F-pooling) for CNNs. F-pooling reduces the dimension of signals in frequency domain. We have defined shift-equivalent of functions which contain down sampling operations by introducing an up sampling operation. Under this definition, we have proved that F-pooling is the optimal anti-aliasing down sampling operation and F-pooling chitt-equivaeent. We have integrated F-pooling into modern CNNs. We have verified that F- pooling remarkably increases accuracy and robustness w.r.t shifts of modern CNNs. We believe that F-pooling plays a more important role in applications where equivvetui is more serious, such as object detection and semantic segmentation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Md Faijul Amin and Kazuyuki Murase. Single-layered complex-valued neural network for real- valued classification problems. Neurocomputing, 72(4-6):945 955, 2009.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: 12177, 2018.
Taco S. Cohen, Mario Geiger, Jonas Khler, and Max Welling. Spherical CNNs. In International Conference Learning Representations, 2018. htthttts://ppeereviiii net/ forum?
Add-Hkbd5228
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Alcksander Madry. A rotation and a translation suffice: Fooling cnns with simple transformations. arXiv preprint 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Benjamin Graham. Fractional max-pooling. arXiv preprint arXiv: 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep con- volutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994)1999-1191 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778. 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Yann Lecun, Berhard E Boser, John S Denker, D Henderson, R E Howard, W Hubbard, and I. D network. pp. 396-404, 1989.
Jackel. Handwritten digit recognition with a
Chenyu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolu- tional neural networks: Mixed, gated, and tree. pp. 464-472, 2016.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. pp. 3431-3440, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. Computer Science, 2013.
Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of The American Institute of Electrical Engineers, 47(2):617-644, 1928.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- supervised learning with ladder networks. pp. 3546-3554, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2211-55)) 2015. doi: 10.1007/s1 10007//11233-01--001111
Jongbin Ryu, Minghsuan Yang, and Jongwoo Lim. Dft-based transformation invariant pooling layer for visual classification. pp. 89-104, 2018.

---

Under review as a conference paper at ICLR 2020
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Dominik Scherer, Andreas Muller, and Sven Behnke. Evaluation of pooling operations in convolu- tional architectures for object recognition. pp. 92-101, 2010.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Fe- lipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Pal. Deep complex networks. In International Conference on Learning Representations, 2018. URL
htttps://ppenrevdlt ftt.:n//ooooummmdd111110mmmmmm
net
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International Conference Learning Representations, 2018. URL//ttpss:/opennevieeww net/
id=rkhlb8 IC2
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schutze. Comparative study of cnn and rnn for natural language processing. arXiv: Computation and Language, 2017.
Richard Zhang. Making convolutional networks shift-invariant again.
arXiv preprint
1.XY:1904.114866 2019.
A PROOF OF OPERATION
We prove Dy v = Zu y. By definition, T can be represented as an M x N matrix and Z can be represented as a N x M matrix.
T
I M
0
0
N M
AM
az
(27)
M
x
M
a
I
where IM is M M identity matrix and 0. is an M x M zero matrix. Z ag is equal to the transpose of T y by definition. Then
Z
M T
ON-M M
(28)
MM
which is the same D by definition.
B EFFECT OF odd padding
C LOSs CURVES ON SUB-IMAGENET
10

---

Under review as a conference paper at ICLR 2020
6.7-
0.7
MM
0.6
mw
0.4
0.4
6.3
posting
acstrpahil
(a) with odd padding
(b) without odd padding
Figure 4: With odd padding, F-pooling strict shift-equivalent. Without it, F-pooling is not strict shift-equivalent. But the error acceptable. Best viewed on screen.

---

Under review as a conference paper at ICLR 2020
Orgn
Orinn popoodi
18-
3.8
1.6
2.2
2.4
3.0
2.8-
3.4
5.8
100
160
(a) densenet-tree train
(b) densenet-test
on
3,4
1.2
2.
1.00
1.5
3.0
Sporhh
Spach
(c) resnet-train
(d) resnet-test
Fpowling ind
Fene in
2.6
Traniig
2.0
8.8-
Epoch
Epoch
(e) mobile-train
(1) mobile-test
Figure 5: Loss curves sub-ImageNet. Best viewed on screen.
12


Reviews:
[{"review_id": "r1lTmBYHtB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed \"F pooling\" for Frequency Pooling, which is a pooling operation satisfying shift equivalence and anti-aliasing properties. The method is very simple: first, transform the input 1D/2D signal into the spectrum domain based on discrete Fourier transform (DFT), then cut the high-frequencies, then transform back to the time domain using the inverse DFT. The method can be implemented using FFT and auto differentiation frameworks. The method is tested on Resnet/Desnet on CIFAR-100 and subsets of ImageNet, showing better performance than the original models.\n\nThe reviewer votes for rejection as the method has limited novelty. Spectrum pooling has been used in the community of computer vision and machine learning. Taking a random example (there are others by simple searching), in the ECCV paper \"DFT-based Transformation Invariant Pooling Layer for Visual Classification, Ryu et al., 2018\" The DFT magnitude pooling is almost the same as the authors' propositions, where the \"Fourier coefficients are cropped by cutting off high-frequency components\". \n\nThe reviewer encourages the authors to make further new developments and have a more comprehensive literature review. But in the current form, the paper has less value to be published in ICLR.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skx-jEBr5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new pooling method (Frequency pooling) which is strict shift equivalent and anti-aliasing in theory. The authors first derived the theory of F-pooling to be optimal anti-aliasing down sampling and is shift-equivalent in sec 2, and then demonstrated the experimental results of 1D signals and image classification tasks. \n\nThe experimental results are actually less impressive than what are claimed in contribution and conclusion. The authors stated that \"F-pooling remarkably increases accuracy and robustness w.r.t. shifts of moderns CNNs\"; however, in Table 1-3, the winning margin of accuracy is actually quite small (<2%), and the consistency (<3.5% compared to the second best baseline except resnet-18 on CIFAR 100 has large improvement ~7-8%). \n\nQuestions:\n1. For the experiment of 1D signal on sine wave, the AA-pooling and F-pooling give the same result? \n2. Compared to AA-pooling, it seems that F-pooling has a better theoretical guarantee (i.e. the optimal anti-aliasing down sampling operation given U). But other than this, the empirical performance seem not showing particular advantage over AA-pooling. Are there any other advantages for F-pooling s.t. people might want to use it as opposed to AA-pooling?\n3. What are the limitations of the F-pooling? It is good to me that the authors discuss one limitation on the imaginary part of output and I would like to hear more on other potential limitations for this method. \n- also, if the authors can explain more on sec 2.5 it will be helpful. If we simply ignore the imaginary part, although the theory is not applicable, but what would the empirical performance be? \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyl73mFK9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper researches the pooling operation, which is an important component in convolutional neural networks (CNN) for image classification. Taking the perspective from signal processing, this paper proposes a pooling operation called frequency pooling (F-pooling). The key motivation is to make the pooling operation shift-equivalent and anti-aliasing. This paper gives an improved definition on shift-equivalent functions and shows that the proposed F-pooling is optimal in the sense of reconstructing the orignal signal. The F-pooling is then implemented with matrix multiplications and tested with recent convolutional neural networks for image classifiation on CIFAR-100 and a subset of ImageNet dataset. \n\nIt is interesting to take the perspective from signal processing to give pooling operation in CNN a formal treatment. As indicated in the recent literature, enforcing shift-invariance does help to improve the performance of a CNN on classification accuracy and the robustness with respect to image shift. At the same time, this work can be further enhanced at the following aspects:\n1. This work can make it clearer in principle how anti-aliasing contributes to improving the classification performance and robustness. This will help to make this paper more self-contained. \n2. When showing the optimality of F-pooling in Section 2.3, the criterion is to reconstruct the original signal x. Considering that the ultimate goal is classification, the information to be maximally preserved through each operation through the layers shall be the information that relates to the class label y. In light of this, some justification and explanation shall be provided for using this criterion for optimality.\n3. The experimental study is weak. Experiments could be conducted on more benchmark datasets with more CNN architectures to convincingly show the effectiveness of the proposed F-pooling. Also, from the three Tables in the experimental part, the improvement of F-pooling over AA-pooling (developed by the main reference of this work) does not seem to be significant or consistent. For example, in Table 2, the F-pooling only wins at either accuracy (marginally) or consistency, but not both. In Table 3, the F-pooling consistently shows inferior classification performance, although obtaining slightly higher consistency. This makes the advantage of F-pooling over the existing AA-pooling unclear.      ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission has been assessed by three reviewers and scored 3/6/1. The reviewers also have not increased their scores after the rebuttal. Two reviewers pointed to poor experimental results that do not fully support what is claimed in contributions and conclusions. Theoretical support for the reconstruction criterion was considered weak. Finally, the paer is pointed to be a special case of (Zhang 2019). While the paper has some merits, all reviewers had a large number of unresolved criticism. Thus, this paper cannot be accepted by ICLR2020.

Decision: Reject