LiftPool: Bidirectional ConvNet Pooling
Authors: Jiaojiao Zhao
Cees G. M. Snoek
Venue: ICLR 2021 Poster (2021)
Keywords: bidirectional
pooling

Abstract: Pooling is a critical operation in convolutional neural networks for increasing receptive fields and improving robustness to input variations. Most existing pooling operations downsample the feature maps,  which is a lossy process.   Moreover, they are not invertible: upsampling a downscaled feature map can not recover the lost information in the downsampling.  By adopting the philosophy of the classical Lifting Scheme from signal processing, we propose LiftPool for bidirectional pooling layers, including LiftDownPool and LiftUpPool.  LiftDownPool decomposes a feature map into various downsized sub-bands,  each of which contains information with different frequencies. As the pooling function in LiftDownPool is perfectly invertible, by performing LiftDownPool backward, a corresponding up-pooling layer LiftUpPool is able to generate a refined upsampled feature map using the detail subbands, which is useful for image-to-image translation challenges.  Experiments show the proposed methods achieve better results on image classification and semantic segmentation,  using various backbones. Moreover, LiftDownPool offers better robustness to input corruptions and perturbations.

Published as a conference paper at ICLR 2021
LIFTPOOL: BIDIRECTIONAL CONVNET POOLING
Jiaojiao Zhao & Cees G. M. Snoek Video & Image Sense Lab University of Amsterdam {izhao3, semanoek)@uva..... n1
ABSTRACT
Pooling is a critical operation in convolutional neural networks for increasing re- ceptive fields and improving robustness to input variations. Most existing pooling operations downsample the feature maps, which is a lossy process. Moreover, they not invertible: upsampling downscaled feature map can not recover the lost information in the downsampling. By adopting the philosophy of the classi- cal Lifting Scheme from signal processing, we propose LiftPool for bidirectional pooling layers, including LifiDownPool and LiflUpPool. LiftDownPool decom- poses a feature map into various downsized sub-bands, each of which contains information with different frequencies. As the pooling function in LiftDownPool is perfectly invertible, by performing LiftDownPool backward, corresponding up-pooling layer LiftUpPool is able to generate refined upsampled feature map using the detail sub-bands, which is useful for image-to-image translation chal- lenges. Experiments show the proposed methods achieve better results image classification and semantic segmentation, using various backbones. Moreover, LiftDDownPool offers better robustness input corruptions and perturbations.
INTRODUCTION
Spatial pooling has been a critical ConvNet operation since its inception (Fukushima, 1979; LeCun al., 1990; Krizhevsky al., 2012; He al., 2016; Chen al., 2018). It crucial that a pooling layer maintains the most important activations for the network's discriminability (Saeedan et al., 2018; Boureau et al., 2010). Several simple operations, such as average pooling or max pooling, have been explored for aggregating features in local area. Springenberg et al. (2015) employ convolutional layer with an increased stride replace a pooling layer, which is equivalent to down- sampling. While effective and efficient, simply using the average or maximum activation may ignore local structures. In addition, as these functions are not invertible, upsampling the downscaled feature maps can not recover the lost information. Different from existing pooling operations, propose in this paper a bidirectional pooling called LiftPool, including LiftDownPool which preserves details when downsizing the feature maps, and LiftUpPool for generating finer upsampled feature maps.
LiftPool is inspired by the classical Lifting Scheme (Sweldens, 1998) from signal processing, which commonly used for information compression (Pesquet-Popeseu & Bottreau, 2001), reconstruc- tion (Dogiwal et al., 2014), and denoising (Wu et al., 2004). The perfect invertibility of the Lifting Scheme stimulates some works on invertible networks (Dinh al., 2017: Jacobsen al., 2018: Atanov et al., 2019; Izmailov et al., 2020). The Lifting Scheme decomposes an input signal into various sub-bands with downscaled size and this process is perfectly invertible. Applying the idea of Lifting Scheme, LiftDoonPool factorizes an input feature map into several downsized spatial sub-bands with different correlation structures. the I.L sub-band is an approximation removing several details. The L.H, HL and H represent details along horizontal, vertical and diagonal directions. LiftDownPool respects preserving any sub-band(s) as the pooled result. Moreover, due to the invertibility of the pooling function, Lift- UpPool is introduced for upsampling feature maps. Upsampling a feature map is more challenging seen for the MaxUpPool (Badrinarayanan et al., 2017), which generates output with many 'holes' (shown in Figure 1). LiftUpPool utilizes the recorded details to recover a refined output by performing LiftDownPPol backwards.
shown in Figure 1, for an image feature map,

---

Published as a conference paper at ICLR 2021
Up-pool
Down-pool
MaxUpPool
MaxPool
Max Indices
Original
LiftDownPool
LL
LH
LiftUpPool
HL HH
Figure I: Illustration of the proposed LiftDownPool and LiftUpPool vs. MaxPool and MaxUp- Pool on image from CIFAR- 100. Where MaxPool takes the maximum activations from the input, LiftDownPool decomposes the input into four sub-bands: LI, LH, HL and HH. LL contains low-pass coefficients. It better reduces aliasing compared to MaxPool. LH, H L and H repre- sent details along horizontal, vertical and diagonal directions. For simplicity, we just upsample the down-powe down-pooled results for illustrating the up pooling. MaxUpPool generates sparse map with lost de- tails. LiftUpPool produces refined output from the recorded details by performing LiftDownPPol backwards.
We analyze the proposed LiftPool from several viewpoints. LiftDownPool allows a flexible choice for any sub-band(s) as the pooled result. It outperforms baselines image classification with var- ious ConvNet backbones. LiftDownPool also presents better stability corruptions and pertur- bations of inputs. By performing LiftDownPool backwards, LiftUpPool generates refined up- sampling feature map for semantic segmentation.
METHODS
The down-pooling operator is formulated as minimizing the information loss caused by downsizing feature maps, in image downscaling by Saeedan al. (2018); Kobayashi (2019a). The Lifting Scheme (Sweldens, 1998) naturally matches the problem. The Lifting Scheme was originally de- signed to exploit the correlated structures present in signals build downsized approximation and several detail form realizable and always provides a perfect reconstruction the input. LiftPool is derived from the Lifting Scheme for bidirectional pooling layers.
in the spatial domain (Daubechies & Sweldens, 1998). The inverse trans-
2.1 LIFTDOWNPOOL.
Taking one-dimension (ID) signal as an example. LiftDownPool decomposes a given signal x= x=[x1, T2 Ø3. ... Fn). on into a downscaled approximation signal and difference signal d by,
s, = Fx).
(I)
where F(-)=fupdate o Sprediet (splic()), consisting of three functions: split (downsample), predict and update. Here o indicates the function composition operator. The LiftDownPool-ID is illustrated in Figure 2(a). Specifically,
Split fapli + (x",x"). The given signal x is split into two disjoint sets x x==[x2 [2= TAy F21 with even indices and ="=[0,, Ta. 224+1] with odd indices. The two sets are typically closely correlated.
2

---

LiftDDownPoo and Lift- UpPool implementations. LiftDownPool-ID. x is split into x and x". The Predictor and Updater generate details and an approximation Published as a conference paper at ICLR 2021
Figure 2:
S
P
M
(a)
Spiit
Predict
Update
Merge
5
JP
u
u
M
(b) LiftUpPool-ID. By running LiftDownPool backwards, x and x" are generated from and d, and then merged into.x.
x
d
d
(a) LiftDonPPool
(b) LiftUpPool
Predict Spredict (xx"x*) + d. Given one set e.g. x², another set x" is able to be predicted by a predictor P(.). (-). The predictor is not required to be precise, so the difference with the high-pass coefficients d is defined as:
= x - -P(x))
Update fupdate (x", + s. Taking x" as an approximation of x causes a serious aliasing because x" is simply downsampled from x. Particularly, the running average of x" is not the same as that of To correct it, smoothed version is generated by adding (d) to x":
x (dd)).
The update procedure equivalent to applying a low-pass filter to x. Thus, s with low-pass coeffi- cients is taken approximation of the original signal.
The classific Lifting Scheme method applies pre-defined low-pass filters and high-pass filters to decompose image into four sub-bands However, pre-designing filters in P(-) and u((-) is diffi- cult (Zheng et al., 2010). Earlier, Zheng et al. (2010) proposed to optimize these filters by a back- propagation network. All functions in LiftDownPPol are differentiable. P(--) and are able be simply implemented by convolution operators followed by non-linear activation functions (Glorot al., 2011). Specifically, we design P()) and (1)) as:
P(-) Tanh() o Conv(k=1,=11,-- ReLU() Conv(--K,=Ig-GGI),
(4)
u(-) = Tanh() Conv(k=1,==1.==G.) ReLU() Convk==
(5)
Here K is the kernel size and G1 and G2 the number of groups. We prefer learn the filters in P(.) and (d() with deep neural networks in an end-to-end fashion. To that end, two constraints need to be added the final loss function. Recall, s the downsized approximation of As is updated from x according to Eq 3, is essentially close to x Thus, is naturally required to be close to x as well. Therefore, one constraint term C2 is for minimising the L2-norm distance between and x With Eq 3.
Cu = - xx2 ([((d) + +x" x²112.
(6)
The other constraint term cp is for minimising the detail d, with Eq 2.
Cp = -|10 - P(xx))l2.
(7)
The total loss is:
C Lussk + Aucu + Appp:
(8)
where Lusk is the loss for a specific task, like a classification or semantic segmentation loss. We set =,=0001 and 00 Our experiments show the two terms bring good regularization to the model.
LifiDownPool-2D is easily decomposed into several ID LiftDoonPPol operators. Following the standard Lifting Scheme, we first perform LiftDownPool-I1 along the horizontal direction to obtain an approximation part s (low frequency in the horizontal direction) and a difference part (high frequency in the horizontal direction). Then, for each of the two parts, apply the same LiftDownnPoll along the vertical direction. By doing so, $ is further decomposed into LL (low frequency in vertical and horizontal directions) and LH (low frequency in the vertical direction and high frequency in the horizontal direction). is further decomposed into H L (high frequency in the vertical direction and low frequency in the horizontal direction) and H H (high frequency in vertical
3

---

Published as a conference paper at ICLR 2021
LL
LH
HL
HH
Figure 3: LiftDownPool visualization. Selected feature maps of an image CIFAR- 100, from the first LiftDownPool layer in VGGI3. LL represents smoothed feature maps with less details. LH, HL and HH represent detailed features along horizontal, vertical and diagonal directions. Each sub-bands contains different correlation structures.
and horizontal directions). We can flexibly choose sub-band(s) for down-pooling and keep the other sub-band(s) for reversing the operation. Naturally, LiftDownPool- ID can be generalized further for any n-dimensional signal. In Figure 3, we show several feature maps from the first LiftDownPool layer based on VGGI3. L.I. has smoothed features with less details. LHH, H I. and H capture the details along horizontal, vertical and diagonal directions.
Discussion MaxPool usually formulated as first performing Max and then down- sampling: MaxPools. :ddwwssppl................ Max (Zhang, 2019). By contrast, LiftDownPool is: LiftDownPooll,
predict, downsample, First downsampling and then performing two lifting steps (prediction and updating) helps anti-alian aliisnaiasing. A simple analysis is provided in the Ap- pendix. As shown Figure I, LiftDownPool keeps more structured information and better reduces aliasing then MaxPool.
2.2 LIFTUPPOOL
LiftPool inherits the invertibility of the Lifting Scheme. Taking the 1D signal as an example, Lift- UpPool generates upsampled signal from s. by:
x ((s.d).
(9)
where G(-)=fmerge fpedict fapdane (-), including the functions: update, predict and merge. Specifi- cally, s, + xx ,d + x xx + x are realized by:
x" -d(d)))
(10)
x d P(xx).
(11)
fmerge xx)).
(12)
We simply get the even part x" and odd part x" from and d, and then merge x" and x" into x. In this way, we generate upsampled feature maps with rich information.
Discussion Up-pooling has been used in image-to-image translation tasks such as semantic seg- mentation (Chen et al., 2017), super-resolution (Shi et al., 2016), and image colorization (Zhao et al., 2020). It is generally used in encoder-decoder networks such as SegNet (Badrinarayanan al., 2017) and UNet (Ronneberger et al., 2015). However, most existing pooling functions are not invertible. Taking MaxPool as the baseline, it is required to record the maximum indices during max pooling. For simplicity, we use the down-pooled results as inputs to the up-pooling in Figure I. When performing Maxol, the values of the input feature maps responding maximum indices of the output and other indices will be given zeros. By doing so, the output looks sparse and loses much of the structured information, which is harmful for generating good-resolle rosoluusol outputs. LiftUpPool performing an inverse transformation of LiftDownPool.. is able produce finer outputs by using the multiphase sub-bands.
directly filled the cor-
4

---

Published as a conference paper at ICLR 2021
3 RELATED WORK
Taking the average over a feature map region was the pooling mechanism of choice in the Neocog- nitron (Fukushima, 1979; 1980) and LeNet (LeCun et al., 1990). Average pooling is equivalent to Max pooling later proved even more effective (Scherer et al., 2010) and became popular in deep ConvNets. Yet, averaging activations or picking the maximum activations causes loss of details. Zeiler & Fergus (2013) and Zhai et al. (2017) introduced stochastic pro- cess to pooling and downsampling, respectively, for a better regularization. Lee et al. (2016) mixed AveragePool and MaxPool by a gated mask to adapt complex and variable input patterns. Saeedan et al. (2018) introduced detail-preserving pooling (DPP) for maintaining structured details. By con- trast, Zhang (2019) proposed a BlurPool by applying low-pass filter, which removes details. In- terestingly, both methods improved image classification, indicating that (empirically) determining the best pooling strategy is beneficial (Saeedan et al., 2018). Williams & Li (2018) introduced the wavelet transform into pooling for reducing jagged edges and other artifacts. Rippel et al. (2015) suggested pooling in the frequency domain, which enabled flexibility in the choice of the pooling output dimensionality. Pooling based on a probabilistic model was proposed in (Kobayashi, 2019a) and (Kobayashi, 2019b). Kobayashi (2019a) first used a Gaussian distribution to model the local activations and then aggregates the activations into the two statistics of mean and standard devia- tion. Kobayashi (2019b) estimated parameters from global statistics in the input feature map, to flexibly represent various pooling types. Our proposed LiftDownPool decomposes the input feature map into a downsized approximation and several details. It is flexible to choose any sub-band(s) (s) as pooled result.
While existing pooling functions are not invertible, our proposed LiftPool is able to perform both down-pooling and up-pooling. Previously, MaxUpPool (Badrinarayanan et al., 2017) was intro- duced for semantic segmentation. As the max pooling function is not invertible, the lost details can not be recovered during up-pooling. Hence, the output suffers from aliasing, Although adding a BlurPool to MaxUpPool may help to reduce the aliasing (Zhang, 2019), several details are still lost. LiftUpPool, performing the LiftDDownPol functions backwards, capable of producing a refined high-resolution output with the help of the details sub-bands
Earlier, Zheng et al. (2010) introduce back pack-propagation for the Lifting Scheme to perform nonlinear wavelet decomposition. They propose an update-first Lifting Scheme and use back propapaapaaatinnn to replace the Updater and Predictor in the Lifting Scheme. this way, they realize back- mack-propagation neural network in lifting steps for signal processing. There is no pooling layer used. We develop down-pooling and up-pooling layers by leveraging the idea of the Lifting Scheme for image process- ing. We utilize convolution layers and ReLU layers to implement the Updater and Predictor, which are optimized end-to-end with the deep neural network. Our pooling layers are easily plugged into various backbones. Recently, Rodriguez et al. (2020) introduce the Lifting Scheme for multires- olution analysis in a network. Specifically, they develop an adaptive wavelet network by stacking several convolution layers and Lifting Scheme layers. They focus on an interpretable network by integrating multiresolution analysis. rather than pooling. Our paper aims at developing a pooling layer by utilizing the lifting steps. We develop a down-pooling that constructs various sub-bands with different information, and an up-pooling which generates refined upsampled feature maps.
EXPERIMENTS
4.1 CONVNET TESTBEDS
Image Classification We first verify the proposed LiftDownPPol for image classification on CIFAR-100 (Krizhevsky & Hinton, 2009) with 32x32 low-resolution images. CIFAR- 100 has 100 classes with 600 images each. There are 500 training images and 100 testing images per class. A VGG13 (Simonyan & Zisserman, 2015) network is trained on this dataset. For experiments con- ducted on CIFAR-100, 100, we repeat each experiment three times with different initial random seeds during training and report the averaged error rate with the standard deviation. We also report results on ImageNet (Deng et al., 2009) with 1.2M training and 5000 validation images for 1000 classes. We plug the LiftDownPool into several popular ConvNet backbones to verify its generalizability for image classification. We replace the local pooling layers by LiftDownPool in all the networks. Error rate is utilized as the evaluation metric. All training settings are provided in the Appendix.
5

---

Published as a conference paper at ICLR 2021
Top-1
Table I: Flexibility. Top-1 image classifi- cation enror rate with varying sub-bands on CIFAR-100. Mixing low-pass and high-pass obtains the best result. Adding cu and helps improve the result.
LL
25.64 0.04
LH HI.
25.71 1: 0.04
24.88 1: 0.05
HH
25.18 = 0.08
LLALH+HL*HH (wlo cu and cp)
26,43 = 0.07
LLALH+HL+HH (wl cu and cp)
24.35 +: 0.11
Kernel
Top-1
Top-1
2
25.53 0.13
Skip
27.09 + 0.11
25.06 1 0.22
MaxPool
25.71 + 0.13
4
24.89 0.07
AveragePool
25.87 1: 0.03
24.35 = 0.11
7
24.40 0.08
LiftDownPool 24.35 0.11
Table 2: Effectiveness. Top-1 image clas- sification error rate with varying kernel size on CIFAR-100. Kernel size achieves better result.
Table 3: Effectiveness. Top-1 image classi- fication error rate with various pooling meth- ods on CIFAR-100. LiftDoonPool outper- forms baselines.
Semantic Segmentation We also test the LiftDownPool and LiftUpPool for semantic segmenta- tion on PASCAL-VOCI2 (Everingham et al., 2010), which contains 20 foreground object classes and one background class. An augmented version with 10582 training images and 1449 val- idation images is used. We consider SegNet (Badrinarayanan et al., 2017) with VGG13 and V3Pplab (Chen et 2018) with ResNet50 as ConvNets for semantic segmentation. The per- formance is measured in terms of pixel meaa-interrection--ver-em (mloU) across the 21 classes. Code is available
4.2 ABLATION STUDY
Flexibility We first test VGGI3 on CIFAR-100. Different from previous pooling methods, Lift- DownPool generates four sub-bands, each of which contains a different type of information. Lift- DownPool allows to flexibly choose which sub- (aab-ban(s) keep the final pooled results. In Table 1, we show the Top-/ eтror rate for the classification based on different sub-bands. Interestingly. is observed that vertical details contribute more for image classification. Low-pass coefficients and high-pass coefficients along horizontal direction get similar error rate. Whether the two spatial di- mensions should be treated equally leave for our future work. To realize less lossy pooling. we combine all the sub-bands by summing them up with almost no additional compute cost. Such pooling significantly improves the results. In addition, the constrains C, and ep help decrease the error rate. Moreover, seen from Table and Table 3, we further conclude LiftDDwnPool outperforms other baselines even based on any single sub-band. We believe the learned LiftDownPool provides an effective regularization to the model.
Effectiveness Table 2 ablates the performance when varying kernel sizes for the filters in P(-) and u(.). A larger kernel size, covering more local information, performs slightly better. When kernel size equals 7, it brings more computations but no performance gain. Unless specified otherwise, use for all experiments from now on a kernel size of and we sum up all the sub-bands. We also compare our LiftDownPool with the commonly-used MaxPool, AveragePool, as well as the convolutional layer with stride (Springenberg et al., 2015), which is called Skip by Kobayashi (2019a). Seen from Table 3, LiftDownPool outperforms other pooling methods on CIFAR-100.
Generalizability We apply LiftDownPool to several backbones including ResNet18, ResNe150 (He et al., 2016) and MobileNet-V2 (Sandler al., 2018) on ImageNet. In Ta- ble 4, LiftDownPool has 2% lower Top-1 error rate than MaxPool and AveragePool. While combining MaxPool and AveragePool in Gated (Lee al., 2016) or Mixed (Lee et al., 2016) fashion, still has a 1% gap with LiftDownPPol. Gauss (Kobayashi, 2019a) and GFGP (Kobayashi, 2019b) comparable Pift with ResNet50, but not with lighter networks. Compared

---

Published as a conference paper at ICLR 2021
ResNet18
ResNet50
MobileNer V2
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Skip (Springenberg et al., 2015)
30.22
10.23
24.31
7.34
28.66
9.70
MaxPool
28.60
9.77
24.26
7.22
28.65
9.82
AveragePool
28.03
9.55
24.40
7.35
28.32
9.72
Pool (Zhai al., 2017)
33.91
13.09
27.98
9.34
40.56
17.91
WaveletPool (Williams Li, 2018)
30.33
10.82
24.43
7.36
29.27
10.26
BlurPool* (Zhang, 2019)
29.88
10.58
24.60
7.73
30.58
11.26
DPP* (Saeedan et al., 2018)
29.12
10.21
24.62
7.49
29.85
10.53
SpectralPool (Rippel al., 2015)
28.69
9.87
24.81
7.57
33.38
12.56
GatedPool (Lee et al., 2016)
27.78
9.44
23.79
7.06
28.94
9.90
MixedPool (Lee al., 2016)
27.76
9.50
24.08
7.32
29.00
9.97
GFGP* (Kobayashi, 2019b)
26.88
8.66
22.76
6.34
28.42
9.59
GaussPool* (Kobayashi, 2019a)
26.58
8.86
22.95
6.30
27.13
8.92
LiftDownPool
25.80
8.14
22.36
6.11
26.09
8.22
Generalizability of LiftDoowPPol ImageNet. LiftDownPool outperforms alternative pooling methods, no matter what ConvNet backbone is used. means the numbers are based on running the code provided by authors. Others are based on our re-implementation.
Table 4:
Normalized
Unnormalized
ImageNet-C
ImageNet-P
ImageNet-C
ImageNet-P
mCE
mPR
mCE
mPR
Skip
72.71
61.75
57.05
7.56
MaxPool
73.09
62.64
57.40
7.57
AvcragePool
72.09
56.23
56.56
6.90
BlurPool (Zhang, 2019)
72.14
56.54
56.58
6.90
DPP (Saeedan et al., 2018)
72.12
62.30
56.67
7.62
GatedPool (Lee et al., 2016)
72.58
58.05
57.00
7.23
GaussPool (Kobayashi, 2019a)
69.27
54.83
54.40
6.76
LiftDownPool
68.45
52.91
53.80
6.55
Table 5: Out-of-distributiio robustness of LiftDownPool on ImageNet-C and ImageNet-P, Lift- DownPool is more robust to corruptions and perturbations compared to baselines.
Spectral pooling (Rippel et al., 2015) and Wavelet pooling (Williams Li, 2018), which are applied in the frequency space-frequene, domain, LiftDownPool offers an advantage by learning correlated structures and details the spatial domain. Compared to DPP (Saeedan et al., 2018), which preserves details, and BlurPool (Zhang, 2019), smoothing feature maps by low-pass filter, our LiftDownPool retains all sub-bands which proves to more powerful for image classification. Stochastic approaches like Pool obtain poor results on the large-scale dataset because randomness in pooling hampers network training, as earlier observed by Kobayashi (2019b). To conclude, LiftDownPPol performs better matter what backbone is used.
Parameter Efficiency. For all pooling layers in one network, we use the same kernel size in LiftPool. For the trainable parameters, recall P or U has a ID convolution, so each has e/GxxCxK+H parameters. C' the number of the input channels and G2 equals the num- ber of internal channels. A 2D LiftDownPool shares these parameters three times without extra parameters. We compare our LiftDownPool (with 25.58 M) to two other parameterized pooling methods using ResNet50 ImageNet: GFGP (31.08 M) and GaussPool (33.85 M). We achieve lower error rate compared to GFGP and GaussPool with less parameters. Our performance boost is due to the LiftPool scheme, not the added capacity.

---

Published as a conference paper at ICLR 2021
98
MaxPool
Figure 4: Shift Robustness com- parisons between various pool- ing methods including MaxPool, Skip, AveragePool and the pro- posed LiftDownnPol. LiftDDwn- Pool improves classification con- sistency and meanwhile boosts the accuracy, independent of the back- bone used.
96
Skip
AveragePool
94
LittDownPoal
92
90
Resivet5o
88
86
Mobilenet-v2
29
70
Accuracy
Image
MaxUpPool
MaxUpPool+
LiftUpPool
Ground-truth
Figure 5: LiftUpPool for Semantic Segmentation. Visualization of semantic segmentation maps on PASCAL-VOC12 based on SegNet with varying up-pooling methods. LiftUpPool presents more completed, precise segmentation maps with smooth edges.
4.3 STABILITY ANALYSIS
intttf-distribution Robustness A good down-pooling method is expected be stable to per- turbations and noise. Following Zhang (2019), we test the robustness of LiftDownPool to corrup- tions on ImageNet-C and stability to perturbations on ImageNet-P using ResNet50. Both datasets come from Hendrycks & Dietterich (2019). We report the mean Corruption Error (mCE) and mean Flip Rate (mFR) for the two tasks, with both unnormalized raw values and normalized values by AlexNet's mCE and mFR, following Hendrycks & Dietterich (2019).
From Table 5, LifiDownPool effectively reduces raw mCE compared to the baselines. We show CE for each corruption type for further analysis in Figure 9 in the Appendix. LiftDDonPPol enables robustness to both "high-frequency" corouuttc"ccoruuptoonns corruptions, such as noise or spatter, and "low-frequency" cor- ruptions, like blur and jpeg compression. We believe LiftDownPool benefits from the mechanism that all sub-bands are used. similar conclusion is obtained for robustness to perturbations on ImageNet-P from Table 5 and Figure 9 in the Appendix. ImageNet-P contains short video clips of a single image with small perturbations added. Such perturbations are generated by several types of noise, blur, geometric changes, and simulated weather conditions. The metric FR measures how often the Top-1 classification changes in consecutive frames. It is designed for testing a model's stability under small deformations. Again, LiftDownPool achieves lower FR for most perturbations.
Shift Robustness We then test the shift-invariance of our model. Following Zhang (2019), we use classification consistency to measure the shift-invariance. It represents how often the network outputs the same classification, given the same image with two different shifts. We test the models with varying backbones trained on ImageNet. In Figure 4, LiftDownPPol boosts classification ac- curacy as well as consistency matter which backbone is used. Besides, we have other interesting findings. The deeper ResNet50 network has more stable invarinceei..... Various pooling methods including MaxPool, Skip, AveragePool, do not make significant difference consistency. How-
8

---

Published as a conference paper at ICLR 2021
mloU
mloU
Skip
76.1
MaxUpPool
62.7
MaxPool
76.2
MaxUpPool + BlurPool
64.0
AveragePool
76.4
Gauss (Kobayashi, 2019a)
77.4
LiftUpPool
68.9
LiftDownnPol
78.7
Table 6: LiftUpPool for Semantic Segmentation on PASCAL-VOC12 based on SegNet with varying up- pooling methods.
Table Semantic Segmentation with DeepLab on PASCAL-VOCI2 with various pooling methods. Lift- DownPool performs best.
lighter ResNet18 network is influenced much by the pooling method. LiftDownPool brings more than 10% improvement on consistency using ResNet18. We leave for future work how the depth of the network affects the shift-invariance of the network itself.
4.4 RESULTS FOR SEMANTIC SEGMENTATION
LiftUpPool for Semantic Segmentation LiftDownPool functions are invertible as described in Eq 10 and Eq 11. It naturally benefits corresponding up-pooling operation, which is popularly used in Encoder-Decode networks for sauug-to-immge translation tasks. Usually, the Encoder downsizes feature maps layer by layer to generate high-level embedding for understanding the image. Then the Decoder needs to translate the embedding with a tiny spatial size to a required map with the same spatial size as the original input image. Interpreting details is pivotal for producing high-resolution outputs. We replace all down-pooling and up- up-pooling layers with LiftDownPool and LiftUpPool in SegNet for semantic segmentation
PASCAL-VOCIZ. For LiftDownPool LL sub-band. For LiftUpPool, the detail-preserving sub-bands LH, HL and HH are used for generating upsampled feature maps. MaxUpPool is taken as the baseline. We also test MaxUpPool followed by a BlurPool (Zhang. 2019), which is expected to help anti-aliasing, Table reveals LiftUpPool improves over the baselines with a considerable margin. As illustrated in Figure 1, MaxUpPool is unable to compensate for the lost details. Although BlurPool helps smoothing local areas, can only provide a small improvement. by fusing it with details, it is beneficial for per-pixel prediction tasks like semantic segmentation. We show several examples for semantic segmentation in Figure 5. LiftUpPool more precise on details and edges. We also show the feature maps per predicted class in the Appendix.
only keep the
LiftUpPool is capable of refining the feature map
Semantic Segmentation with D As discussed, LiftDownPPol helps to lift Conv- Nets accuracy and stability for image classification. ImageNet-traineee ConvNets often serve as the backbones for downstream tuning. It is expected to transfer the nature of LiftDownPool to other tasks. We still consider semantic segmentation BaaabbV/PlluRRee//eee
our example. We leverage the state-of-the-an art (Chen et al., 2018). The input image has size 512x512, The output feature map of the encoder 32 x 32. The decoder upsamples the feature map 128 128 and con- catenates them with the low-level feature map for the final pixel-level classification. As before, all local pooling layers are replaced by LiftDoonPool. We use the pre-trained weights for image clas- sification to initialize the corresponding model. As shown in Table 7, LiftDownPoo outperforms all the baselines with considerable gaps.
CONCLUSION
Applying classical signal processing theory to modern deep neural networks, we propose novel pooling method: LiftPool. LiftPool is able to perform both down-pooling and up-pooling. Lift- DownPool improves both accuracy and robustness for image classification. LiftUpPool, generating refined upsampling feature maps, outperforms MaxUpPool by considerable margin semantic segmentation. Future work may focus on applying LiftPool to fine- fine-grained image classification, super-resolution challenges other tasks with high demands for detail preservation.

---

Published as a conference paper at ICLR 2021
REFERENCES
Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semi- conditional normalizing flows for semi-supervised learning. INNFw, 2019. 1
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convolutional encoder- decoder architecture for image segmentation. TPAMI, 2017. 1, 4,4,5 5,
Y-Lan Boureau, Jean Ponce, and Yann LeCun. theoretical analysis of feature pooling in visual recognition. In ICML, pp. 111-118, 2010. 1
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Dee Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 1, 6,9
Ingrid Daubechies and Wim Sweldens. Factoring wavelet transforms into lifting steps. Journal of Fourier Analysis and Applications, 4(3), 1998.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Sanwta Ram Dogiwal, YS Shishodia, and Abhay Upadhyaya. Efficient lifting scheme based super resolution image reconstruction using low resolution images. In Advanced Computing, Network- ing and Informatics, pp. 259-266. Springer, 2014.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-338, 2010.
Kunihiko Fukushima. Neural network model for a mechanism of pattern recognition unaffected by
shift in position-neocconiitee IEICE Technical Report, A, 62(10))658-66,, 1979. 1,5
Kunihiko Fukushima. A self-organizing neural network model for mechanism of pattern recogni-
tion unaffected by shift position. Biol. Cybern.. 36:193-200 202, 1980. 5
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2.11.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016. 6,
Dan Hendryeks and Thomas G Dietterich. Benchmarking neural network robustness common corruptions and surface variations. In ICLR, 2019. 8, 13
Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. Semi-supervised learn- ing with normalizing flows. In ICML, 2020.
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-RevNet: Deep invertible net-
works. In ICLR, 2018. 1
Takumi Kobayashi. Gaussian-based pooling for convolutional neural networks. In NeurlPS, 2019a. 2, 6, 7,9
Takumi Kobayashi. Global feature guided local pooling. In ICCV, 2019b. 6,
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012. 1
10

---

Published as a conference paper at ICLR 2021
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back- propaaattnn net- work. In NIPS, 1990. 5,5
Chen-Yu Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolu- tional neural networks: Mixed, gated, and tree. In AISTATS, pp. 464-472, 2016. 5, 6,
Adam Paszke et al. Automatic differentiation in PyTorch. 2017. 12
Béatrice Pesquet-Popsee and Vincent Bottreau. Three-dimensional lifting schemes for motion
compensated video compression. In ICASSP, 2001.
Oren Rippel, Jasper Snock, and Ryan Adams. Spectral representations for convolutional neural networks. In NeurIPS, 2015. 5,
Maria Ximena Bastidas Rodriguez, Adrien Gruson, Luisa Polania, Shin Fujieda, Flavio Prieto, Ko- hei Takayama, and Toshiya Hachisuka. Deep adaptive wavelet network. In WACV, 2020. 5
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In MICCAI, 2015.
Faraz. Saeedan, Nicolas Weber, Michael Goesele, and Stefan Roth. deep networks. CVPR, 2018. 1, 5,7
Detail-preserving pooling in
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. MobileNet-Ve V2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Dominik Scherer, Andreas Müller, and Sven Behnke. Evaluation of pooling operations in convolu- tional architectures for object recognition. In ICANN, pp. 92-101, 2010. 5
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In CVPR, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. 5, 12
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR, 2015. 1, 6,
Wim Sweldens. The lifting scheme: A construction of second generation wavelets. SIAM Journal on Mathematical Analysis, 29(2):511 1998. 1, 2, 12, 13
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In ICLR, 2018. 5.7
Yonghong Wu, Quan Pan, Hongcai Zhang, and Shaowu Zhang. Adaptive denoising based on lifting scheme. In ICSP, 2004.
Matthew D Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional neural networks. arXiv preprint arXiv:1301.3557 2013. 5
Shuangfei Zhai, Hui Wu, Abhishek Kumar, Yu Cheng, Yongxi Lu. Zhongfei Zhang, and Rogerio Feris. S3Pool: Pooling with stochastic spatial sampling. In CVPR, 2017. 5,
Richard Zhang. Making convolutional networks shift-invariant again. In JCML, 2019. 4, 5, 7,8
Jiaojiao Zhao, Jungong Han, Ling Shao, and Cees GM Snoek. Pixelated semantic colorization. IJCV, 128(4):814-834, 2020.
Yi Zheng, Ruijin Wang, and Jianping Li. Nonlinear wavelets and BP neural networks adaptive lifting scheme. In ICACIA, 2010. 3.5
11

---

Published as a conference paper at ICLR 2021
a
b
d
MaxPool
LiftDowPPool
h
LL
Approximation
h'
LL"
y'
a'
0
0
0
X
a
6
d'
0
0
0
h
MaxUJpPool
e
h
LiteUpPool
0
0
V
0
X
X
HL
HH
0
0
0
Max-pooling Indices
Details
Figure 6: Comparisons between MaxPool and LiftDownPool, MaxUpPool and LiftUpPool. MaxPool looses details. With the recorded maximum indices, MaxUpPool generates very sparse output. LiftDownPool decomposes the input into approximation and several details sub-bands. It realizes pooling by summing up sub-bands. LiftUpPool produces refined output by perform- ing LiftDownPPol backwards.
A
APPENDIX
We show additional analysis and results for robustness and semantic segmentation in this Appendix.
LiftDownPPol vs. MaxPool We provide a schematic diagram in Figure to further illustrate the difference between MaxPool and LiftDownnPol. Max MaxUpPool and LiftUpPool. Taking kernel size 2, stride 2 as an example, MaxPool selects the maximum activations in local neighbourhood. Hence, it looses 75% information. The lost details could be important for image recognition. LiftDownPool decomposes a feature map into LL, LH, HL and HH. LL containing low-pass coefficients is an approximation of the input. bun-bands contain detail coefficients along different directions. The pooling is implemented by summing up all the sub-bands. The final pooled result containing both the approximation and details expected to be more effective for image classification.
designed for capturing correlated structures of the input. Other
LiftUpPool vs. MaxUpPool The pooling function in MaxPool not invertible. MaxPool records the maximum indices for performing the corresponding MaxUpPool. MaxUpPool takes the acti- vations at the corresponding positions for the recorded maximum indices on the output. For other indices, there will be zeros. The final upsampled output has many 'holes'. By contrast, the pooling functions LiftDownPool are invertible. Leveraging the property by performing a LiftDownPon backwards, LiftUpPool is able to generate refined output from input, including the recorded details.
Experiment Settings The VGG13 (Simonyan & Zisserman, 2015) network trained CIFAR- 100 optimized by SGD with batch size of 100, weight decay of 0.0005, momentum of 0.9. The learning rate starts from 0.1 and is reduced by multiplying 0.1 after 80 and 120 epochs for total of 160 epochs. We train ResNets for 100 epochs and MobileNet for 150 epochs on ImageNet, following the standard training recipe from the public PyTorch (Paszke et al., 2017) repository.
High-resolution Feature Maps Visualization By using ResNet50 with input size 224x 224, we extract the feature maps of an image from the first pooling layer. We show the high-resolution feature maps in Figure 7. We only show the LL sub-band from LiftDownPool. Compared to MaxPool, LiftDownPPol better maintains the local structure.
LiftDownPool effectively reduces aliasing following the Lifting Scheme (Sweldens, 1998) compared to naive downsizing. Figure 8(b) provides simple illustration of LiftoowPPol.. The dashed line is an original signal According to Eq 1, the predictor P(.) for the odd part x could easily take the average of two even neighbors:
Anti-aliasing
d x2
(x2x
x2x+2)/2
(13)
12

---

Published as a conference paper at ICLR 2021
Input image
MaxPool
LiftDownPool
Figure 7: High-resolution feature maps visualization. LiftDownPPol better maintains local struc- ture.
32
3.-
24-
22
2k 2
2k
2k
2k
2k + 2k
+ 2k +4 2k+
(a) Downsizing
32
dx
24
2k
2k
2k Zk 2k
2k +
2k
(b) LiftDownPool
Figure 8: Illustration how LiftDownPool reduces aliasing compared to downsizing (Sweldens, 1998). Dashed line means original signal. (a) solid line is after downsizing. (b) solid line is after LiftDownPool. The solid and dashed lines cover the same area in (b).
Thus, ifr is linear in local area, the detail dg is zero. The prediction step takes of some of the spatial correlation. If an approximation of the original signal is simply taken from the even part x". it is really downsizing the signal shaped in the red line. There is serious aliasing. The running average of x" is not the same as that of the original signal x. The updater l1(-) in Eq 3 corrects this by replacing x with smoothed values s. Specifically, 2(-)) restores the correct running average and thus reduces aliasing:
S X2x (d + dx)/4
(14)
As shown in Figure 8, da is the difference between the odd sample x24+1 122ple and the average of even samples. This causes a loss ds/2 2 in the area with the red shade. To preserve the running average, this area is redistributed piecewise linear signal in the solid line. The signal after LiftDownPool, drawn as solid line, covers the same area with the original signal dashed line. LiftDownPool reduces aliasing compared the downsizing drawn in the solid line in (a).
the two neighbouring even samples x22 and x24-2, which shapes coarser
-ittrf-distribution Robustness We show the robustness of pooling methods for each corrup- tion and perturbation type in Figure 9. Comuption Error (CE) is the metric of the robustness to corruptions on ImageNet-C. And Flip Rate (FR) is reported for the robustness to perturbation on ImageNet-P. Following (Hendrycks & Dietterich, 2019), report both unnormalized raw values
13

---

Published as a conference paper at ICLR 2021
Comparisons
Imagehat-C
Unnormitzed
glvss Mur
MatoovaMar
brightnsse
Hugtaa blur
trossage
20
20
8- Btuffod
Wivedst
central
U1
Spatter
Cated
ebeli Trr
SCAR
piecide
specklt pose
(a)
(b)
Comparisons of Normiized Fill on ImageNer-P
mation Nit
Normlizdd
zoom bkrr
brightrea
know
Apatter
spocak_noss
guusia
(c)
(d)
Figure 9: Comparisons between the robustness various pooling methods per kind of cor- ruption on Ietge and perturbation on ImageNet-PP. LiftDownPool presents stronger ro- bustness to almost the corruptions and perturbations.
and normalized values by AlexNet's CE and FR. Lower values better. As seen in Figure 9a) and (c), LiftDownPood gets the lowest CE for most of the "high frequency" corruptions including gaus- sian noise and spatter, well the "low frequency" corruptions such motion blur, zoom blur. In Figure 9() and (d), it clearly shows LiftDownPoo has less sensitivity to most of the perturbations such speckle noise and gaussian blur.
Visualization of Up-pooling In Figure 10, we show the feature map for each predicted category from the last layer of SegNet using varying up-pooling methods. Using MaxUpPool, the feature maps look noisy and less continuous due to the fact that MaxUpPool generates the output with many 'zeros', where there is no information. By applying a BlurPool following the Max the feature maps turn more smooth, while still with less details. LiftUpPool, benefiting from the recorded details during LiftDownPool, produces finer feature maps for each category. It has smooth edges, continuous segmentation maps and less aliasing.
14

---

Published as a conference paper at ICLR 2021
COW
person
--2
motobike
car
Ground-truth
MaxUpPool
MaxUppool+BlurPooll
LiftUpPool
Figure 10: Visualization of feature maps per-predicted-cateeory from the last layer of SegNet.
Lift-UpPool generates more precise predictions for each category.
15


Reviews:
[{"review_id": "wAjtrBPbcBF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a learnable pooling strategy to preserve details when down-sampling feature maps. The proposed LiftPool method is derived from the classical lifting scheme of signal processing, which contains a LiftDownPool to decompose a feature map into various down-sampled sub-bands in the down-sampling process, and a LiftUpPool to merge these sub-bands in the up-sampling process. The proposed pooling strategy is incorporated into existing methods to achieve  better results on image classification and semantic segmentation tasks.\n\n\nI have the following main concerns.\n\n1. The motivation of this work is not well-explained. First, it is not explained clearly why the pooling operation is desired to be inversible. Second, the main purpose of pooling is to obtain a larger receptive field. While it does lose spatial information, one can seek to other methods to either avoid using pooling (e.g., using dilation convolutions) or enhancing spatial informaiton (e.g., skip connection or feature fusion). Hence, it would be better to highlight the necessity of preserving details and being inversible in pooling operations.\n\n2. Section 2 only describes the proposed method. However, it is important to highlight the novelties in the proposed method. I am not sure whether the method described in section 2 is an easy adaption of the existing Lifting Scheme method. I also do not understand why the proposed LiftDownPool is formulated as the multiplication of f_{split}, f_{predict} and f_{update}.\n\n3. In the second paragraph of section 3, it is said that most existing pooling methods are not inversible. However, this means that there are some methods that are inversible. However, the paper do not explain it.\n\n4. It is not clear which sub-bands should be used as pooled results.\n\n5. The Lift scheme has already been studied in neural networks in (Zheng et al., 2010). The difference is not discussed in depth.\n\n6. The proposed method is only evaluated on the PASCAL-VOC12 (Everingham et al., 2010) with some methods. Evaluations on more datasets and \n\n7. Experiments in Table 6 seems not fair, as the proposed method introduces more learnable parameters. \n\n\nMinor issues.\n\n1. The title is called \"bidirectional\". However, after reading the paper, I still do not understand how the propsoed method relates to it.\n\n2. In figure 5, I am not sure how we can evaluate the \"Invertibility\" by comparing the segmentation results. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "tf_dNCuUbC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pooling method for CNNs using a Lifting scheme. The proposed LiftPool consists of LiftDownPool and LiftUpPool. The LiftDownPool decomposes a feature map into four sub-bands (LL/LH/HL/HH). The ListUpPool is reversible to the original features maps from the decomposed components. The decomposition is realized by convolutional layers and relu operation and trained with CNNs. Experimental results show LiftPool achieves better results on image classification and segmentation. \nOverall, the paper is well written and easy to follow. The method is well motivated and technically sound. Experimental results are compelling. \n\nPros\n+ LiftPool is well motivated by the difficulty of conducting upsampling (Badrinaryanan et al. 2017) and the aliasing problem (Zhang 2019) of max pooling. \n+ LiftPool shows better performance than state-of-the-arts pooling methods in image classification evaluated on the ImageNet dataset with three backbone networks.   \n+ The robustness of the LiftPool is widely evaluated in terms of out-of-distribution and shift-robustness. \n+ The invertible property of LiftPool seems to be very suitable to Encoder-Decoder architecture in semantic segmentation. It shows better scores than MaxUpPool + BlurPool(Zhang, 2019), and the semantic segmentation results are more smooth. \n\nCons \n- Details of P() and U() are not clear. According to p.3, they are convolutional operators followed by non-linear relu operators. How many convolutional operators are used? \n- To learn P() and U(), the loss functions of Eq.(4) and Eq.(5) are used. Is it necessary to introduce regularization parameters to balance with the original loss function?  \n- LiftPool is applied to each of the local pooling of each layer of CNNs. Are different filters learned for each position and layers? How many parameters (trainable and hyper-parameters) are added by LiftPool?\n- How to perform 1D-operations along the two dimensions of the image to obtain LL,LH,HL,HL should be clarified, eg., orders. \n- Table1 shows that larger kernel sizes perform better. Why did the authors not test larger kernel sizes than 5?\n- There is a work that uses Lifting-wavelet for CNNs. The difference between LiftPool to this work should be discussed. \\\nM.X.B.Rodriguez, A.Gruson, L.F.Polania, S.Fujieda, F.P.Oritz, K.Takayama, T.Hachisuka, Deep Adaptive Wavelet Network, In Proc. WACV2020\n\nMinor problem\n- In Sec.2.1, a clear explanation of the correspondence of s,d, and L,H would help readers to understand the LiftPool. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "C6aX6eJwQkp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple downscaling / upscaling method LiftPool, inspired by Lifting Scheme from signal processing. Compared with traditional max / avg pooling used in CNNs, LiftPool decomposes input signal into a downscaled approximation and difference component, which can be used for classification (by summing both components) or segmentation (by skip-layer connections). Experiments on both tasks show proposed method yields better performance, and additional analysis on stability indicates LiftPool is more robust than other pooling methods.\n\nPros:\n- A simple but effective method by transferring knowledge from signal processing to vision/ML. \n- Pooling is a fundamental block in CNN architecture and this work would benefit a wide range of research and applications\n- Consistently better experiment results compared with other commonly used pooling methods, by a significant margin\n- Paper is relatively clear written and easy to follow\n\nCons:\n- My main concern with this paper is lacking measurements over # of params, FLOPs and latency. Unlike traditional pooling methods which are usually parameter-free and (relatively) fast to compute, LiftPool does need (from authors) \"convolution operators followed by non-linear relu operators\" to simulate the filters in  $\\mathcal{P}$ and $\\mathcal{U}$. The implementation details of these conv operators are missing (except the kernel size is 5, from ablation study). It is unclear whether the performance boost of proposed method is from the effectiveness of LiftPool, or from added capacity of network with more parameters and computations.\n\n\nMinor comments:\n- Abstract: \"upsampling a down-scaled feature map loses much information\": this is not necessarily true.  Downscaling could lose information while upscaling alone usually don't bring more information.  Moreover, CNNs could still \"memorize\" spatial information in its depth channel (consider a space-to-depth that reduces spatial resolution but still preserves all information).\n\n- Both $\\mathcal{P}$ and $\\mathcal{U}$ should be real-valued and using conv + relu might limit filter responses to non-negative values? If multiple conv layers are used and the last one do not have any activation, please specify.\n\n- Eq 4/5 proposes additional loss to help training LiftPool but its effect is not backed up by experiments.\n\n- Figure 3: it might be beneficial to show the original high res feature map together with baseline pooling results (e.g. max / avg pooled) to demonstrate information preserved by LiftPool.\n\n- Experiments conducted are more towards lightweight backbones (ResNet18/50 and MobilenetV2). Would LiftPool also work well with larger architectures? This is specially important given the extra params and computes LiftPool brings.\n\n- On segmentation experiments: It is well-known that bringing low level features in decoders of modern segmentation networks could boost model performance. How would LiftPool work with architectures that have decoder with skip-connections? The results on DeeplabV3plus looks good, but it is more from transferability (pretrained on image classification using liftpool). \n\n- On deeplabv3plus setup: it would be great to clarify more details on this model, e.g. decoder and output stride used in training and validation.\n\nFinal review:\n\nThe authors addressed most of my concerns. Just a nit that it could be helpful to add the total FLOPs into the table (together with # of params) just for completeness. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7sfcPvzqSoo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new pooling layer that is based on the Lifting Scheme from signal processing. It motivates this approach with the desire for reversible pooling functions for certain tasks. The benefits of this reversibility are demonstrated on a semantic segmentation task. As a drop-in replacement for the pooling lawyer in various neural-network backbones, it also outperforms many other pooling layers on classification tasks (ImageNet).\n\nI thought this paper had great collection of analyses: flexibility (choice of pooling band), effectiveness across kernal sizes, generalizability across various backbones, and robustness to corruptions and perturbations.\n\n*I recommend to accept*. While this may be just another pooling layer, it seems a quite well motivated pooling layer coming from a particular need for reversibility.\n\nSome highlights:\n- very clear writing\n- very well situated in historical and contemporary literature\n- exactly the experiments that I would want to see\n- the observation that the sub-band that represents vertical details constributes more to classification accuracy than other sub-bands; curious to know whether this holds outside of VGG13 on CIFAR-100.\n\nSome points for improvement:\n- the presentation of the lifting scheme and the use of the LL/HL/HH/HH notation is perhaps a little non-intuitive for anyone without previous exposure; you could make it clearer that not all bands would be necessarily used during pooling, but that the information would be retained for reversing the operation.\n- on p.7 you say \"sift-invariance\"; I think you mean \"shift-invariance\"\n- Instead of saying how you believe your findings will stimulate people to think about problems (\"These findings may stimulate researchers to rethink\", \"We believe such findings will stimulate one to think\"), it would be better to perhaps make a claim that needs to be evaluated in the future, or to point out exactly what is left unknown or surprising by your findings.\n- Figure 7 is not very clear. Think about people with colour blindness. Also, consider two separate graphs side-by-side or one above the other. It isn't clear what the shaded red area is meant to indicate and how it relates to what is \"redistributed.\" I see what you're trying to show, and I actually just think it is a quite difficult thing to visualize, so maybe Figure 7 is the best you can get to, but I would brainstorm some more on this.\n- p 12. \"Visualiztion\"\n- p 12. the closing quotation marks around \"high frequency\" and \"low frequency\" go the wrong direction\n- Figure 9 caption: the hyphenation in LiftUpPool should be customized; it breaks the word at a weird spot\n- Figure 8 consider using various line types instead of colors in order to better accomodate people with color blindness\n- Throughout: inconsistent hyphenation \"downsizing\" vs \"down-sizing\", \"up-sampling\" vs \"upsampling\"\n- Bibliography: I think you need to force capitalization in some of the titles. See e.g. \"pytorch\" and \"Mobilenetv2\"\n\n----------------\n\nQuestion: When you \"combine all the sub-bands by summing them up\", do you literally just add up the values from the corresponding indices across each sub-band so that you're still reducing the dimensionality? I am a little surprised that this doesn't *reduce* performance. Can you say more about this? Why does this work?\n\nQuestion: Why did you only compare against three baseline pooling methods for the corruptions and perturbations instead of the full gamut as in Table 4?\n\nQuestion: do you have error bars for your experiments? Or did you run them only once each? For which results did you run your own experiments vs reporting numbers from previous literature (particularily in Table 4)?\n\n----------------\nMy main uncertainty (why I am not giving this a 5 for confidence) is that I cannot be sure this hasn't been proposed in the past, but it is hard to prove a negative. I can say is that this does appear novel to me.\n\nI am also uncertain about the evaluation on the semantic segmentation task. I am familiar with this problem and the evaluations seem reasonable, but I cannot be sure whether the choices of comparator methods are the strongest alternatives.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a bidirectional pooling layer inspired by the classical Lifting scheme from signal processing. LiftDownPool is able to preserve structure and details in different sub-bands, whereas LiftUpPool is able to generate a refined up sampled feature map using the detail sub-bands. This is very useful for image to image translation tasks and all tasks that involve up scaling.
This is a solid contribution with extensive and thorough experiments and direct practical usage, clear accept.

Decision: Accept (Poster)