Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper generalizes polyphase resampling and applies learnable selection of the polyphase components, instead of based on signal magnitudes. It achieved better accuracy and shift consistency on classification and segmentation.
strengths_and_weaknesses: Strength: The paper writes with professionism in notations and arangement. The authors did both experiment in high-level classification and low-level segmentation.

Weakness: My personal feeling here is that: the idea of polyphase resampling is being derived to maybe too complicated, to lose its original meaning in signal processing, and now it becomes a fancy way of 2x2 pooling. Even APS can be used for backpropagation actually, it is just another non-smooth operator adding to some existing non-smooth operators, Relu, maxpooling, so no big trouble. So I am conservative about the motivation to be the polyphase resampling parameterized. The computation added by this layer cannot be neglet, considering the shiting doubles of workload of whatever layer follows it.
questions: Maybe I missed something, as APS is a specicial case of LPU, and why is LPU more shift-invariant than APS theoretically, and on segmentation empirically? As LPU is data dependent, so it could be approximate APS on some bad test case?

In figure 4 caption "This indicates LPS did not learn to reproduce APS" I can infer this from the algorithm, but is this good from visual meaning perspective? Difficult to see.

The running time, computation complexity comparison with APS, and other baseline should be included.
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a  new framework to enable the design of shift invariant and equivalence convolutional neural networks. The key idea in this paper is introducing a  learnable polyphase sampling scheme which can be used in both down/up sampling layers.  Compared to existing methods, the proposed LPS is trained from data instead of manually.  The LPS is introduced with detailed  analysis and the implementation is also shown in both down/up sampling layers.  Experiments on image classification and semantic segmentation valid its effectiveness in practices. 
 
strengths_and_weaknesses: Pros:
The main contribution in this paper is extending the manually designed adaptive polyphase sampling layers to learnable polyphase sampling layers with theoretically prove.  Especially, the introduced conditional probability directly learn from the data while the LPS keep the property of APS.   Besides the LPS, both downsampling and upsampling LPD and LPU have been showed with implementation and detailed analysis. 

 Cons:
The main idea of this paper is generalized the existing work APS in a learnable manner.  Compared to the significance of APS the contribution is limited.  Also it is not clear how LPU and LPD work in real cases. For example, in Semantic Segmentation cases, LPD  only seems works better than LPD and LPU while it is not clear how LPU works and what the reason behind that.  


questions: Q1.In general LPS has similar performance to APS in Resnet 18 setting. Is the result consistent in Resnet 50 or 101? 


Q2.  Table 4 shows that  LPD only works better than LPS, while there is no ablation shows that the importance of LPD and LPU in the neural network when keep replacing LPD LPU in downsampling and upsampling layers.   Will is possible LPD + a few LPU layers  can have even better performance?


limitations: The limitation in this paper is the marginal performance improvement over APS, which leads the bias for justifying the importance of the motivation in this paper.   This is because APS is a special case of  LPS, the contribution of this paper can only be valid through theoretical analysis.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the issue of shift invariance/equivariance in CNNs. Prior work (APS) suggested, when downsampling, to select the index for whose value has the largest norm. This paper suggests a generalization which consists of learning the index-selection rule instead. Experiments on classification (invariance: CIFAR, ImageNet) and segmentation (equivariance ; PascalVOC) show convincing results.

Learning this selection is implemented via a carefully crafted small NN and the gumbel-softmax trick.
strengths_and_weaknesses: ## Strengths

- The paper clearly motivates the problem.
- The paper gives just the right amount of background knowledge and prior work.
- The evaluations are to the point.

## Weaknesses

The experiments show convincing results: shift invariance matching theory, and small but consistent improvement in "regular" accuracy measures. However, I wonder whether the small but consistent improvements may not be simply due to the fact of introducing more learnable parameters. This is almost addressed in Appendix A7, but what I am missing is the results of either (a) resnet + LPS but with slightly reduced ResNet size to match plain resnet params/speed, or (b) plain resnet with slightly increased size to match resnet + LPS params/speed.

Also, this should be mentioned in the main paper, with reference to the appendix. Actually, most of the appendix should be mentioned in the main paper.
questions: See above.
limitations: None.
review_title: null

Meta-review: metareview: The paper proposes an end-to-end learnable polyphase sampling and shows competitive performance and sufficient novelty. Major concerns of the reviewers seem to be addressed during the rebuttal and therefore it can be accepted. 
confidence: Less certain
Decision: Accept