Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a pooling method for CNNs using a Lifting scheme. The proposed LiftPool consists of LiftDownPool and LiftUpPool. The LiftDownPool decomposes a feature map into four sub-bands (LL/LH/HL/HH). The ListUpPool is reversible to the original features maps from the decomposed components. The decomposition is realized by convolutional layers and relu operation and trained with CNNs. Experimental results show LiftPool achieves better results on image classification and segmentation. 
Overall, the paper is well written and easy to follow. The method is well motivated and technically sound. Experimental results are compelling. 

Pros
+ LiftPool is well motivated by the difficulty of conducting upsampling (Badrinaryanan et al. 2017) and the aliasing problem (Zhang 2019) of max pooling. 
+ LiftPool shows better performance than state-of-the-arts pooling methods in image classification evaluated on the ImageNet dataset with three backbone networks.   
+ The robustness of the LiftPool is widely evaluated in terms of out-of-distribution and shift-robustness. 
+ The invertible property of LiftPool seems to be very suitable to Encoder-Decoder architecture in semantic segmentation. It shows better scores than MaxUpPool + BlurPool(Zhang, 2019), and the semantic segmentation results are more smooth. 

Cons 
- Details of P() and U() are not clear. According to p.3, they are convolutional operators followed by non-linear relu operators. How many convolutional operators are used? 
- To learn P() and U(), the loss functions of Eq.(4) and Eq.(5) are used. Is it necessary to introduce regularization parameters to balance with the original loss function?  
- LiftPool is applied to each of the local pooling of each layer of CNNs. Are different filters learned for each position and layers? How many parameters (trainable and hyper-parameters) are added by LiftPool?
- How to perform 1D-operations along the two dimensions of the image to obtain LL,LH,HL,HL should be clarified, eg., orders. 
- Table1 shows that larger kernel sizes perform better. Why did the authors not test larger kernel sizes than 5?
- There is a work that uses Lifting-wavelet for CNNs. The difference between LiftPool to this work should be discussed. \
M.X.B.Rodriguez, A.Gruson, L.F.Polania, S.Fujieda, F.P.Oritz, K.Takayama, T.Hachisuka, Deep Adaptive Wavelet Network, In Proc. WACV2020

Minor problem
- In Sec.2.1, a clear explanation of the correspondence of s,d, and L,H would help readers to understand the LiftPool. 
review_title: Good paper about invertible pooling

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a learnable pooling strategy to preserve details when down-sampling feature maps. The proposed LiftPool method is derived from the classical lifting scheme of signal processing, which contains a LiftDownPool to decompose a feature map into various down-sampled sub-bands in the down-sampling process, and a LiftUpPool to merge these sub-bands in the up-sampling process. The proposed pooling strategy is incorporated into existing methods to achieve  better results on image classification and semantic segmentation tasks.


I have the following main concerns.

1. The motivation of this work is not well-explained. First, it is not explained clearly why the pooling operation is desired to be inversible. Second, the main purpose of pooling is to obtain a larger receptive field. While it does lose spatial information, one can seek to other methods to either avoid using pooling (e.g., using dilation convolutions) or enhancing spatial informaiton (e.g., skip connection or feature fusion). Hence, it would be better to highlight the necessity of preserving details and being inversible in pooling operations.

2. Section 2 only describes the proposed method. However, it is important to highlight the novelties in the proposed method. I am not sure whether the method described in section 2 is an easy adaption of the existing Lifting Scheme method. I also do not understand why the proposed LiftDownPool is formulated as the multiplication of f_{split}, f_{predict} and f_{update}.

3. In the second paragraph of section 3, it is said that most existing pooling methods are not inversible. However, this means that there are some methods that are inversible. However, the paper do not explain it.

4. It is not clear which sub-bands should be used as pooled results.

5. The Lift scheme has already been studied in neural networks in (Zheng et al., 2010). The difference is not discussed in depth.

6. The proposed method is only evaluated on the PASCAL-VOC12 (Everingham et al., 2010) with some methods. Evaluations on more datasets and 

7. Experiments in Table 6 seems not fair, as the proposed method introduces more learnable parameters. 


Minor issues.

1. The title is called "bidirectional". However, after reading the paper, I still do not understand how the propsoed method relates to it.

2. In figure 5, I am not sure how we can evaluate the "Invertibility" by comparing the segmentation results. 
review_title: review 3017

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a new pooling layer that is based on the Lifting Scheme from signal processing. It motivates this approach with the desire for reversible pooling functions for certain tasks. The benefits of this reversibility are demonstrated on a semantic segmentation task. As a drop-in replacement for the pooling lawyer in various neural-network backbones, it also outperforms many other pooling layers on classification tasks (ImageNet).

I thought this paper had great collection of analyses: flexibility (choice of pooling band), effectiveness across kernal sizes, generalizability across various backbones, and robustness to corruptions and perturbations.

*I recommend to accept*. While this may be just another pooling layer, it seems a quite well motivated pooling layer coming from a particular need for reversibility.

Some highlights:
- very clear writing
- very well situated in historical and contemporary literature
- exactly the experiments that I would want to see
- the observation that the sub-band that represents vertical details constributes more to classification accuracy than other sub-bands; curious to know whether this holds outside of VGG13 on CIFAR-100.

Some points for improvement:
- the presentation of the lifting scheme and the use of the LL/HL/HH/HH notation is perhaps a little non-intuitive for anyone without previous exposure; you could make it clearer that not all bands would be necessarily used during pooling, but that the information would be retained for reversing the operation.
- on p.7 you say "sift-invariance"; I think you mean "shift-invariance"
- Instead of saying how you believe your findings will stimulate people to think about problems ("These findings may stimulate researchers to rethink", "We believe such findings will stimulate one to think"), it would be better to perhaps make a claim that needs to be evaluated in the future, or to point out exactly what is left unknown or surprising by your findings.
- Figure 7 is not very clear. Think about people with colour blindness. Also, consider two separate graphs side-by-side or one above the other. It isn't clear what the shaded red area is meant to indicate and how it relates to what is "redistributed." I see what you're trying to show, and I actually just think it is a quite difficult thing to visualize, so maybe Figure 7 is the best you can get to, but I would brainstorm some more on this.
- p 12. "Visualiztion"
- p 12. the closing quotation marks around "high frequency" and "low frequency" go the wrong direction
- Figure 9 caption: the hyphenation in LiftUpPool should be customized; it breaks the word at a weird spot
- Figure 8 consider using various line types instead of colors in order to better accomodate people with color blindness
- Throughout: inconsistent hyphenation "downsizing" vs "down-sizing", "up-sampling" vs "upsampling"
- Bibliography: I think you need to force capitalization in some of the titles. See e.g. "pytorch" and "Mobilenetv2"

----------------

Question: When you "combine all the sub-bands by summing them up", do you literally just add up the values from the corresponding indices across each sub-band so that you're still reducing the dimensionality? I am a little surprised that this doesn't *reduce* performance. Can you say more about this? Why does this work?

Question: Why did you only compare against three baseline pooling methods for the corruptions and perturbations instead of the full gamut as in Table 4?

Question: do you have error bars for your experiments? Or did you run them only once each? For which results did you run your own experiments vs reporting numbers from previous literature (particularily in Table 4)?

----------------
My main uncertainty (why I am not giving this a 5 for confidence) is that I cannot be sure this hasn't been proposed in the past, but it is hard to prove a negative. I can say is that this does appear novel to me.

I am also uncertain about the evaluation on the semantic segmentation task. I am familiar with this problem and the evaluations seem reasonable, but I cannot be sure whether the choices of comparator methods are the strongest alternatives.
review_title: Review of "LiftPool: Bidirectional ConvNet Pooling"

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a simple downscaling / upscaling method LiftPool, inspired by Lifting Scheme from signal processing. Compared with traditional max / avg pooling used in CNNs, LiftPool decomposes input signal into a downscaled approximation and difference component, which can be used for classification (by summing both components) or segmentation (by skip-layer connections). Experiments on both tasks show proposed method yields better performance, and additional analysis on stability indicates LiftPool is more robust than other pooling methods.

Pros:
- A simple but effective method by transferring knowledge from signal processing to vision/ML. 
- Pooling is a fundamental block in CNN architecture and this work would benefit a wide range of research and applications
- Consistently better experiment results compared with other commonly used pooling methods, by a significant margin
- Paper is relatively clear written and easy to follow

Cons:
- My main concern with this paper is lacking measurements over # of params, FLOPs and latency. Unlike traditional pooling methods which are usually parameter-free and (relatively) fast to compute, LiftPool does need (from authors) "convolution operators followed by non-linear relu operators" to simulate the filters in  $\mathcal{P}$ and $\mathcal{U}$. The implementation details of these conv operators are missing (except the kernel size is 5, from ablation study). It is unclear whether the performance boost of proposed method is from the effectiveness of LiftPool, or from added capacity of network with more parameters and computations.


Minor comments:
- Abstract: "upsampling a down-scaled feature map loses much information": this is not necessarily true.  Downscaling could lose information while upscaling alone usually don't bring more information.  Moreover, CNNs could still "memorize" spatial information in its depth channel (consider a space-to-depth that reduces spatial resolution but still preserves all information).

- Both $\mathcal{P}$ and $\mathcal{U}$ should be real-valued and using conv + relu might limit filter responses to non-negative values? If multiple conv layers are used and the last one do not have any activation, please specify.

- Eq 4/5 proposes additional loss to help training LiftPool but its effect is not backed up by experiments.

- Figure 3: it might be beneficial to show the original high res feature map together with baseline pooling results (e.g. max / avg pooled) to demonstrate information preserved by LiftPool.

- Experiments conducted are more towards lightweight backbones (ResNet18/50 and MobilenetV2). Would LiftPool also work well with larger architectures? This is specially important given the extra params and computes LiftPool brings.

- On segmentation experiments: It is well-known that bringing low level features in decoders of modern segmentation networks could boost model performance. How would LiftPool work with architectures that have decoder with skip-connections? The results on DeeplabV3plus looks good, but it is more from transferability (pretrained on image classification using liftpool). 

- On deeplabv3plus setup: it would be great to clarify more details on this model, e.g. decoder and output stride used in training and validation.

Final review:

The authors addressed most of my concerns. Just a nit that it could be helpful to add the total FLOPs into the table (together with # of params) just for completeness. 
review_title: Simple yet effective method, but need more analysis / evidence

Meta-review: The paper presents a bidirectional pooling layer inspired by the classical Lifting scheme from signal processing. LiftDownPool is able to preserve structure and details in different sub-bands, whereas LiftUpPool is able to generate a refined up sampled feature map using the detail sub-bands. This is very useful for image to image translation tasks and all tasks that involve up scaling.
This is a solid contribution with extensive and thorough experiments and direct practical usage, clear accept.

Decision: Accept (Poster)