Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a novel iFormer architecture to capture both the high and low frequencies in visual data. Motivated by the observation that ViT tends to capture few high-frequency signals (i.e., showing the characteristics of low-pass filters), the authors propose an Inception token mixer, comprising high- and low-frequency mixers to extract the corresponding frequency information on the feature channels. Also, derived from the finding that lower and higher layers generally need more local and global information, respectively, the authors devise a frequency ramp structure to trade off different-frequency components across all the layers. Experiments across various tasks including classification, detection, and segmentation, convincingly validate the effectiveness of the proposed iFormer with detailed explanations and discussions。
strengths_and_weaknesses: [Strengths]
- The proposed iFormer is very well-motivated. Deriving from the results of the Fourier spectrum and the relative log Fourier amplitudes, the authors accordingly develop the iFormer backbone to enhance the perception capability of the Transformer in the frequency spectrum.  
- The experiments are comprehensive and convincing. Results across three challenging tasks show that the proposed iFormer outperforms existing Transformer architecture. Extensive ablation studies and visualizations also demonstrate the effectiveness of the devised two key modules of the Inception token mixer and the frequency ramp structure.
- The paper is well-written and easy to follow. I enjoy the writing style that starts from the detailed discussions and analysis on the derived observations and then accordingly elaborates the proposed method.

[Weaknesses]
- The authors are suggested to denote the symbols like X_{h1}, X_{h2}, Y_{h1}, and Y_{h2} in Fig. 3, which will make it much easier to relate Fig. 3 back to the text.
- The design of a frequency ramp structure that uses a channel ratio to balance the high- and low-frequency components seems straightforward. I wonder if the authors have tried other designs for frequency information balancing and how they perform.
- It will be great if the authors could share some insights on the future work that can be derived from this work.
- The punctuations for Eq. 1 and Eq. 5 are missing.

questions: - Have you tried other balancing strategies other than the manner of setting the sum of the ratios to 1 in the proposed frequency ramp structure?
- Can you share some insights on the future work that can be derived from this work?
- Do you plan to make your code publicly available for follow-up research?


limitations: Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper designs a new family of multi-branched visual transformer models, called Inception Transformer. Motivated by the spectrum properties of local operations (convolution & pool) and long-range operations (self-attention), a multi-branched Inception Mixer block is proposed to capture both high/low frequency information from the images. A channel splitting mechanism is applied to adopt hybrid operation at each stage. 
In addition, the author assumes that low layers of network favor local/high-frequency pattern and high layers prefer global/low-frequency pattern. Therefore, a frequency ramp structure is utilized to increase the attention dimensions and shrink the convolution dimensions as the network goes deeper.
Comprehensive evaluation on image classification, object detection, instance segmentation and semantic show strong performance.
strengths_and_weaknesses: Overall, I enjoy reading this paper, with straightforward motivation and clear writing.
Strengths:
1.	Strong motivation. As the self-attention operation captures the long-range dependency (shown in Figure 1(a)(b)), a local branch with convolution and pooling operations is well-motivated to compensate the frequency preference of attention operation. 
2.	Simple and efficient structure. Instead of applying different operations on the same feature maps, the high/low frequency mixer is applied on different channel slices in a parallel manner. Also, the structure that gradually increases the global operation channel makes sense to me
3.	Promising empirical results. On ImageNet, iFormer-S/iFormer-B surpass the current best performed architecture by 0.5 %/0.7% in accuracy on 224 resolution. The iFormer-B achieves 48.3 mAP on COCO2017 val and 48.6 mIOU on ADE20K. All results are surprisingly well.
	
Weakness:
1.	Lack of discussion with previous multi-branched network structure. Since the current paper focus on the design of a multi-branched block structure, a detailed comparison and discussion should be included, like ResNeXt[1], GoogleNet Family and Inception Family. Although they did not include self-attention in their branches, some of the networks do have the mixed kernel size (1x1 Vs 5x5), which is also another form of low/high frequency mixer. Also, the multi-branched network seems to have better generalization and optimization properties [2] compared with the single-branched counter parts. A simple citation is insufficient to highlight your improvement over them. In addition, as the paper is largely driven by the low/high pass filter design in signal processing, I would encourage the author incudes some reference on this topic.
2.	Problems on ablation study. To be scientific, ablation study means quantifying the influence of each modular design by removal/changing of one component from the entire system. But in table 5 row 1-4, the authors are accurately adding one component at each time. I know that a lot of paper take this style of ablation study, but scientifically, this is not a valid ablation study. Better fix this. 

[1] Aggregated Residual Transformations for Deep Neural Networks (CVPR 2017)
[2] Deep Neural Networks with Multi-Branch Architectures Are Intrinsically Less Non-Convex (ICML 2019)
questions: 1.	I hope the author should provide more discussions on the core difference between iFormer and previous multi-branched network structure.
2.	Frequency ramp strategy. In the “Frequency ramp structure” section, no description on the how to increase the channel ratio C_l/C. Is it linear scaling according to the depth or exponential increasing? I see the channel number in the appendix, but it seems to be a “magic number” that I am not sure how did you decide on the current number. Moreover, an ablation on how to select the channel number should be included in the experimental part. 
3.	Down-sample and Up-sample Self-attention. To reduce the computational cost, the author proposes in Equation 7 to (1) down-sample the feature map with average pooling (2) Do self-attention (3) Up-sample to the full size. This operation is not mentioned in previous studies. I am curious about its computational complexity and performance, with the vanilla ViT structure.
limitations: Please put more discussions on societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Inception Transformer (iFormer) to enhance the low-pass filters in standard transformers. Specifically, the authors introduce an Inception mixer that has parallel convolution/max-pooling path and self-attention path, capturing high- and low-frequency information respectively. The Inception mixer differs from existing parallel CNN/attention hybrids in two aspects: 1) Unlike [23, 26, 47] that process all channels in each branch, the Inception mixer adopts a channel splitting mechanism to reduce information redundancy. 2) The Inception mixer also contains a fusion module to merge the outputs from low- and high-frequency branches, instead of simple feature concatenation.

Another novelty of the proposed iFormer is the frequency ramp structure. It trade-offs high- and low-frequency info across different layers by gradually decreasing the channel dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer.

The full iFormer achieves SOTA performance on a series of vision tasks, including image classification, object detection and segmentation. It shows better performance-cost trade-off than representative ViTs, CNNs and their hybrid variants, showing the potential to serve as a general transformer backbone.
strengths_and_weaknesses: Strengths:
+ It's new to consider enhancing the capability of transformers to capture high-frequency information.
+ The two key novelties--Inception mixer and frequency ramp structure--are intuitive and make sense. There are ablations provided to support their positive roles.
+ Although the Inception mixer has a parallel structure that is similarly adopted by existing papers, there are still technical differences (ie. channel splitting and fusion module) that prove useful.
+ Strong performance is obtained on various vision tasks, and that comes without sacrificing model efficiency.

Weaknesses:
- The paper should benefit from a detailed quantification of the high- and low-frequency information (e.g. via Fourier analysis) captured for each layer. Is there any interesting pattern across layers? This is the core contribution of the whole paper, but the analysis about how Inception mixer fulfills the task in its low- and high-frequency branches is missing (example qualitative visualization in Fig. 4 may not be enough).
- As mentioned in the limitations, the channel splitting scheme is lacking analysis and/or ablations.
questions: - In the low-frequency mixer, multi-head self-attention is made more efficient by using average pooling first and then upsampling. How much saving is obtained in compute? Will this hurt the attention performance too much (although I get the intention is to focus on embedding global information in the attention branch)?
- What's the exact policy to change the channel ratio across layers? It's linear? How do different policies perform (or do they perform differently at all)?
- How does the feature fusion module compares to direct concatenation [47] in performance?
limitations: Both the limitations and potential negative societal impact are given. It makes sense to bring up the limitation that iFormer requires manually defined channel ratio in the frequency ramp structure.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an iFormer architecture to capture high-frequency and low-frequency visual information. The proposed inception mixer module (i) splits the feature maps along the channel dimension and (ii) applies parallel convolution/max-pooling/self-attention paths to gather high-frequency and low-frequency information.
Besides, the authors also introduce a simple frequency ramp structure that adjusts the channel partitions across different layers. The experimental results look encouraging.
strengths_and_weaknesses: > Strengths

👍 The motivation is clear and the presented idea is very simple and easy to follow.

👍 The proposed iFormer shows slightly better performance while enjoying smaller GFLOPs.

> Weaknesses

👎 The proposed method is TOO NAIVE & lacks NOVELTY. According to Figure 3 and Figure 4, we can see that these three different branches capture information of different frequencies. However, the default **multi-head self-attention is capable to learn such information given enough training epochs** and previous efforts ([ConVit, ICML 2021](https://arxiv.org/abs/2103.10697), [EarlyConvolutions, NIPS 2021](https://proceedings.neurips.cc/paper/2021/file/ff1418e8cc993fe8abcfe3ce2003e5c5-Paper.pdf), [MetaFormer, CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf))have shown that **injecting the inductive biases with either convolution or max-pooling show better training efficiency**.

👎 The proposed method does not provide any results under LARGE configurations. The authors should justify the position of the proposed iFormer on either a TINY configuration that pursues faster running speed & better efficiency or a LARGE configuration that pursues higher accuracy. However, the authors fail to show the proposed method is promising in both aspects. Therefore, it is necessary to present the comparisons to either [Mobile-Former, CVPR 2022](https://arxiv.org/pdf/2108.05895.pdf)/[MobileViT, ICLR 2022](https://openreview.net/pdf?id=vh-0sUt8HlG) or Swin-L/Focal-L/CSwin-L under the fair settings instead of only on an intermediate setting that the community does not care.

👎 The authors should include the results with the combination of ``Attention and DWConv''. According to our previous experience, we can apply large kernel DWConv to achieve better performance than the naive MaxPool operation.
questions: Please carefully address the above-listed weaknesses. I will increase the ratings if the proposed method shows advantages under either the TINY or  LARGE configuration.

> Update:

After reading the comments of other reviewers and the authors' responses carefully, I tend to increase my ratings due to the potential high impact on the community.
limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes a novel multi-branch style architecture for vision tasks, motivated by a frequency perspective of deep network behaviors. All reviewers are very positive about the motivation, presentation and experimental results. The AC believes this should be a good contribution to the neural architecture design community.
confidence: Certain
Decision: Accept