Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Authors propose a Spatially-Adaptive Squeeze-Excitation drop-in module for GAN-base image synthesis and recognition, with improved accuracy and reduced computational cost.
strengths_and_weaknesses: Strengths: the proposed Spatially-Adaptive Squeeze-Excitation seems rational and meaningful

Weaknesses: limited novelty, weak presentation (typos, verbose statements, too small diagrams, ...)


questions: Generalization ability is unconvincing.

Visual effect of the results should be compared too.
limitations: Not really. The module could potentially help synthesize or fake very realistic facial images and videos, hence may have negative social impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity, while retaining the efficiency of SE and the inductive basis of convolution. It presents two designs of spatially-adaptive squeeze-excitation (SASE) modules for image synthesis and image recognition respectively. For image synthesis tasks, the proposed SASE is tested in both low-shot and one-shot learning tasks and shows better performance than prior arts. For image recognition tasks, the proposed SASE is used as a drop-in replacement for convolution layers in ResNets and achieves better accuracy than the vanilla ResNets.
strengths_and_weaknesses: Strengths:
It presents a Spatially-Adaptive Squeeze-Excitation module with two realizations for better learning of generative models from low-shot / one-shot images.
It shows better performance for high-resolution image synthesis at the resolution of 1024×1024 when deployed in the FastGANs and better performance in image classification and object detection with smaller models. 
It enables a simplified workflow for SinGANs, and shows a stronger capability of preserving image structures than prior arts.
The experiments are comprehensive.



Weaknesses: 
1. The formulation introduction part is problematic. The current introduction includes too much detailed designs network structures. The introduction should more focus on the motivation and deliver the contribution in a high-level, instead of giving the detailed network design which should be appear in the method section.
2. Why put the related work section after the method section?
3. line 251, interms --> in terms
questions: See above.
limitations: Limitations and Potential Negative Impacts of the Proposed Work have been discussed in the paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper propose a self-attention module(SASE) for image synthesis and image recognition.
The self-attention is based on the **S**queeze-**E**xcitation(SE) mechanism, which is also 
free from matrix multiplication. 
By inplacing the matrix multiplication by Hadamard product and using the SE mechanism, 
the computational efficiency improves while preserving decent performance.
Sufficient comparisons to traditional attention (not self-attention) mechanism demonstrate 
the effectiveness of SASE in image synthesis. 
While in image recognition, the SASE has slightly disadvantage compared with 
Swin-tiny([32] in the main paper), but with only 67% parameters and 75% FLOPs.
strengths_and_weaknesses: **Strengths**
1. The proposed SASE module is parameter-less and computationally inexpensive, 
   while with respectable performance 
2. The proposed SASE module is robust and easy to use. With simply replacing modules 
   in existing network structures, performance can be improved.
3. The the operation of weighting $Q$ with $K$ in the SASE for image systhesis is novel, 
   which is an effective way for aggregating the spatial attention and channel attention.

**Weakness**
1. This authors need to explain that why their module is effective in the few-shot and one-shot image systhesis.
   For example, in FastGAN([6] in the main paper), they proposed a self-supervised Discriminator in Auto-Encoder manner
   to fully utilize the information for the encoder in the discriminator to extract a more comprehensive representation.
   That's an effective strategy they designed individually for few-shot image systhesis.
2. There is no ablation studies in this paper, so that the design of the module is not guaranteed to be optimal.
   Such as where to adopt the SE mechanism is effective? In the $Q,K,V$ procudure or after the self-attention?
3. This paper, which does not explain some of the original intentions of the module design, can not convince me.
   Such as why the authors choose $X$ to compute the $Q, K$ and $Y$ for the $V$ instead of $X$ for the $Q$ and $Y$ for $K, V$?
   (in the SLE SASE module)
4. Using the self-attention mechanism, the authors need to compare their modules with currently *state-of-the-arts* 
   self-attention modules in Table 3. Such as Swin-Transformer block([32] in the main paper) and Focal transformer Block[1] (at least one of them).

[1] Yang, Jianwei, et al. "Focal attention for long-range interactions in vision transformers." NeruIPS 2021.
questions: 1. Why your module is effective in the few-shot and one-shot image systhesis? Is there any special design for this task?
   If so, what is it? (Q for weakness 1)
2. Why the SE mechanism is adopted in the $Q, K, V$ calculating procedure? Please explain your intention. (Q for weakness 3)
3. Why the authors choose $K$ (a.k.a spatial attention) for normalizing $A$ in figure 2, 
   instead of choosing $Q$ (a.k.a channel attention) for normalizing $A$? (Q for weakness 3)
4. The traditional cross attention in transformer requires that the $K, V$ comes form the same feature[2][3] for better refining.
   However, the authors chooses $Q, K$ from the same feature. Please explain your intention. (Q for weakness 3)

[2] Chen, Chun-Fu Richard, Quanfu Fan, and Rameswar Panda. "Crossvit: Cross-attention multi-scale vision transformer for image classification." ICCV 2021.

[3] Vaswani, Ashish, et al. "Attention is all you need." NeruIPS 2017.
limitations: Yes, the authors adequately the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work presents to extend light-weight SE module to be spatially-adaptive to improve model's efficiency and effectiveness. The proposed module is applied to both image synthesis tasks and image recognition tasks with a little different design. Its effectiveness is validated by  experiments on low-shot image synthesis and image classification to some extent.
strengths_and_weaknesses: Strengths:  
A lightweight SE module is applied to image synthesis tasks and image recognition tasks, whose effectiveness is validated by the corresponding experiments to some extent.

Weaknesses:  
1. Novelty is quite limited, modules similar to the proposed spatially adaptive SE has been proposed before, such the ones in CBAM, RCAN and SFTGAN, SPADE. Please explain in detail what the main difference from these modules is.
2. The organization of this work look more like a technical report rather than an academic paper.

==========================post rebuttal==========================  
Although authors addressed my comments to some extent, I still have a little concern about the novelty of this work. Therefore, I only raise the score from reject to borderline accept.
questions: Please check that in weaknesses.
limitations: Authors mention the limitations and potential engative societal impact, however, they do not give detailed anaylsis and constructive suggestions.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a Spatially-Adaptive Squeeze-Excitation (SASE) module which aims to learn a full attention mechanism as a 3D attention matrix. Two variants are developed to address the image classification and image synthesis tasks. Experimental results on public datasets show its performance.
strengths_and_weaknesses: + The paper is easy to follow. Its representation is clear.
+ The idea of learning a full attention matrix is interesting.
+ Extensive experimental studies are conducted.

- The experimental results are not strong.
- The idea is essentially similar to the dual attention mechanism.
questions: There are some issues which are listed as follows.

1, The idea of SASE is modified based on SE to include the spatial attention mechanism, thus it can learn the so-called full attention as the form of 3D matrix. While this idea is similar to the dual attention mechanism [a], which learns attention matrixes to consider spatial attention and channel attention at the same time. 

[a] Dual Attention Network for Scene Segmentation.

2. The experimental results are not sufficiently strong. For example, in table 8, the improvement over the existing methods is only tiny. At the same time, in table 6, the proposed method achieves the worst diversity performance, is there any reason for that?
limitations: Please see the above sections.
review_title: null

Meta-review: metareview: This work proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity while retaining the efficiency of SE and the inductive basis of convolution. All reviewers have recognized the merit of the work, but raise critical concerns on presentation, insufficient justification of methods, and ethical issues. Specifically, the paper does not well explain why the proposed modules would necessarily perform well. This, however,  is important for a NeurIPS submission. Moreover, the risks of the method are not properly discussed while the paper has the statement like "The proposed SASE module does not show any potential negative impacts with its current form", which somehow is misleading. Last, the experiments may need to be carefully re-orgnized, and more visualization results would be helpful for understanding the proposed methods.  
confidence: Certain
Decision: Reject