Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors introduce a novel Transformer (or Network) architecture, termed as Focal Modulation Network. FocalNet deals with the problem of efficient long-range feature modeling. Different from window-wise self-attention (Swin) and focal attention, FocalNet adaptively aggregates surrounding tokens from different levels of granularity. This is novel and interesting. Extensive experiments on MS COCO, ImageNet, and ADE20k demonstrate the effectiveness of the proposed method. In summary, the idea is interesting, and the performance is promising; I would like to recommend acceptance.
strengths_and_weaknesses: **Strengths:**
1. I enjoy reading this paper, the writing, and the presentation. 
2. The idea is interesting and novel. Considering different levels of granularity makes sense in image processing.
3. The experimental results are promising, demonstrating state-of-the-art performance.

**Weaknesses:**
1. From L124-L132, it is easy to understand the Translation invariance and Decoupled feature granularity, but why input-dependency is considered as an advantage? Also,  Spatial- and channel-specific is achieved by depth-wise operations and cannot be considered as an advantage.
2. I like the idea of different levels of granularity, but this can simply be considered as a multi-scale depth-wise convolution in the implementation (Fig. 2c), which limits the novelty. Multi-scale/ multi-branch always works and introduces no novelty.
3. Some important papers are missing (not compared), like ConvNeXt, that only uses depth-wise convolution as well, and MixFormer, which can be considered as a strong baseline. Both are published in CVPR'22 and released much earlier than the submission dealine. The authors should cite and compare these baselines.
[1] Liu, Zhuang, et al. "A convnet for the 2020s." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[2] Chen, Qiang, et al. "MixFormer: Mixing Features across Windows and Dimensions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: See above weaknesses.

Also, a question is why the authors consider the contextualizations hierarchically rather than parallelly? I thought the contextualizations would be independent when I read the introduction for the first time. 
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Focal Modulation, which uses context at multiple spatial scales from a stack of convolutions combined with gated aggregation to produce a modulation.  This modulates each query point through elementwise multiplication.  Focal Modulation is evaluate as a drop-in replacement for Self-Attention.  The method is tested in several experiments against strong baselines, and has a thorough ablation analysis.

## Update after author response

Thank you for the responses and further experiments.  To be clear, I was mostly concerned with the kernel size experiment as a way to glean the variance of other results.  Since you have provided direct experiments to address that concern, I am more confident that this is a paper worthy of acceptance.
strengths_and_weaknesses: The paper proposes a novel method for improving computer vision architectures based on visual transformers.  Focal Modulation is a clever way to include expanded context at each layer while simultaneously removing expensive pairwise interaction terms present in self-attention.  The method is described clearly, the diagrams are illustrative, and the experimental evaluations and ablations are thorough.  This paper is a solid contribution to an important field of computer vision. However, I would like to see some better estimate of variance of results.  For example, in Figure 5 in the appendix, the mAP results vary considerably depending on the setting of the kernel size.  There seems to be little relationship between the kernel size and the mAP, but the mAP varies considerably (41.2 to 41.6).

questions: I would like to see some notion of noise in the results based on random seed.  It is tempting to draw conclusions from the results as presented, but some of the effect sizes are small.  
limitations: The limitations and societal impact are well addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper propose a focal modulation module that is more effective and efficient for modeling
token interactions. Their main contributions are proposing an efficient way to input-dependent long-range contextual interactions. The authors conduct experiments on tasks of image classification, detection, and segmentation. The experiments' results show the SoTA performance.
strengths_and_weaknesses: + The performance achieves SOTA on almost tasks.
+ The paper is well-written.
+ The authors conduct abundant ablation studies to validate the effectiveness of each design in the proposed method.


questions: ### questions
(1) About training consuming. Will Context Aggregation cause low latency?
(2) About Table 10. It seems that low receptive field impacts the performance little. What if bottom-only with small kernels? Is it means the context is not important? It's confusing. 
(3) About Network Inspection. A Global Pooling Aggregator will lead to a significant drop. May because of low parameters? If we replace local aggregations with several simple downsample convolutions, does it will work?

limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a focal modulation module to replace the attention module in Transformers. Specifically, this module contains Hierarchical Contextualization (several layers of depthwise convolutions), Gated Aggregation, and element-wise modulation to fuse information from the token itself and the context. Actually, the element-wise modulation can also be regarded as a gate mechanism. Thus, this module sounds like a convolution + gate mechanism. This paper does extensive experiments on various vision tasks to show the advantages of the proposed modules. 
strengths_and_weaknesses: Strengths:

1) The writing of this paper is clear and integrated. 

2) The experiments in this paper are extensive, including several vision tasks and ablation studies. It is convinced that Focal Net is better than Swin and Focal Transformers. 

Neutrality: Focal Modulation can be regarded as convolutions plus gate mechanisms. The idea is OK, not incremental but not good enough. Thus, I write the idea aspect into neutrality.

Weaknesses:

In ablation table 9, about the fusion between token itself and context, it only shows the experiment that replaces Multiplication with Addition. What about totally removing the query branch and moving the parameters and computation to other components?

questions: See weaknesses. 
limitations: Yes
review_title: null

Meta-review: metareview: All the reviewers acknowledge that the paper is well-written, novel, and shows strong performance gain. Besides, all the reviewers are satisfied with the authors' response to the raised concerns. AC double-checks the paper, reviews, and response, and finds that the paper is well-shaped and generally flawless. AC recommends acceptance.
confidence: Certain
Decision: Accept