Inception Transformer
Authors: Chenyang Si
Weihao Yu
Pan Zhou
Yichen Zhou
Xinchao Wang
Shuicheng YAN
Venue: NeurIPS 2022 Accept (2022)
Keywords: Convolution
Vision Transformer

Abstract: Recent studies show that transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information. To tackle this issue, we present a novel and general-purpose $\textit{Inception Transformer}$, or $\textit{iFormer}$ for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data. Specifically,  we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high- and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range. Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e., gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and low-frequency components across different layers. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on  image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models are released at https://github.com/sail-sg/iFormer.

Inception Transformer
Chenyang Sil Weihao Yu
Yu 1,2 Pan Zhou'
Yichen Zhou 1.2 Xinchao Wang Shuicheng Yan'
Sea Aab
National University of Singapore
{sicy, ywwelhao, zhoupan, zhouye, ysasc) com, xinchao@nus.com edu.sg
Abstract
Recent studies show that Transformer has strong capability building long-range dependencies, yet is incompetent in capturing high frequencies that predomi- nantly convey local information. To tackle this issue, we present a novel and general-purpose Inception Transformer, iFormer for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data. Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information Transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-poolll ppooling path and self-attentic path as high- and low- frequency mixers, while having the flexibility to model discriminative information scattered within wide frequency range. Considering that bottom layers play more roles capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e., grad- ually decreasing the dimensions fed to the high- frequeeyy mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and low- frequency components across different layers. We benchmark the iFormer a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-IK, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models are released at https:: /ttttp://github com/ cww/sail-ss sawlssalllgg//sss /oog//Foreer...
Introduction
Transformer [1] has taken the natural language processing (NLP) domain by storm, achieving surpris- ingly high performance many NI.P tasks, e.g., machine translation [2] and question-answering [3]. This is largely attributed to its strong capability of modeling long-range dependencies in the data with self-attention mechanism. Its success has led researchers to investigate its adaptation to the computer vision field, and Vision Transformer (ViT) [4] is a pioneer. This architecture is directly inherited from NLP [1]. but applied to image classification with raw image patches as input. Later, many ViT variants [S-13] have been developed to boost performance scale to wider range of vision tasks, e.g., object detection [10, 11] and segmentation [12, 13].
ViT and its variants are highly capable of capturing low- wwwffreuuencies in the visual data [14], mainly including global shapes and structures of a scene or object, but are not very powerful for learning figh-frequencces, mainly including local edges and textures. This can be intuitively explained: self- attention, the main operation used VITs to exchange information among non-overlap patch tokens, is a global operation and much more capable of capturing global information (low frequencies) in the
Equal contribution. Weihao Yu did this work during an internship at Sea Al Lab.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0.0
85
iFormer
a Log amplitude
-0.5
VIT
Top-1 Accuracy
84
83
-1.0
82
Former(Curs
ViTAEv2 ConvNeXt
-1.5
81
Swin
-2.0
80
PVT
DelT
79
0.0m
0.2m 0.5n Frequency
0.8mt
1.0m
10
20
30
40
50
60
70
80 90
iFormer
# Param (M)
(a)
(b)
(c)
Figure I: (a) Fourier spectrum of ViT [18] and iFormer. (b) Relative log amplitudes of Fourier transformed feature maps. (c) Performance of models on ImageNet-1K validation set. (a) and (b) show that iFormer captures more high-frequency signals.
data than local information (high frequencies). As shown in Fig. I(a) and ((b), the Fourier spectrum and relative log amplitudes of the Fourier show that ViT tends well capture cew-freqenccy signals but few high-frequency signals. This observation also accords with the empirical results in [14], which shows VIT presents the characteristics of low-pass filters. This low-frequency preferability impairs the performance of ViT's, as 1) low-frequency information filling in all the layers may deteriorate figh-frreuency components, e.g., local textures, and weakens modeling capability of ViTs; 2) high-frequency information is also discriminative and can benefit many tasks, e.g., (fine- grained) classification. Actually, human visual system extracts visual elementary features different frequencies [15-17]: low frequency provides global information about a visual stimulus, and high frequency conveys local spatial changes in the image (e.g., local edges/textres)) Hence, it necessary to develop new ViT architecture for capturing both high and low frequencies in the visual data.
CNNs are the most fundamental backbone for general vision tasks. Unlike ViTs, they cover more local information through local convolution within the receptive fields, thus effectively extracting high-ffreuency representations [19, 20]. Recent studies [21-25] have integrated CNNs and ViTs considering their complementary advantages. Some methods [21, 22, 24, 25] stack convolution and attention layers in a serial manner to inject the local information into global context. Unfortunately. this serial manner only models one type of dependency, either global or local, in one layer, and discards the global information during locality modeling, or vice versa. Other works [23, 26] adopt parallel attention and convolution to learn global and local dependencies of the input at the same time. However, it is found [27] that part the channels are for processing local information and the other for global modeling, meaning current parallel structures have information redundancy if processing all channels in each branch.
To address this issue, we propose a simple and efficient Inception Transformer (iFormer), as shown in Fig. 2, which grafts the merit of CNNs for capturing high-frequencies ViTs. The key component in iFormer is an Inception token mixer shown in Fig. 3. This Inception mixer aims to augment the perception capability of ViTs in the frequency spectrum by capturing both high and low frequencies in the data. To this end, the Inception mixer first splits the input feature along the channel dimension, and then feeds the split components into gigh-freueecc mixer and low-frequency mixer respectively. Here the high-frequency mixer consists of max-pooling operation and a parallel convolution operation, while the low-frequency mixer is implemented by a vanilla self-attention this way, our iFormer can effectively capture particular frequency information on the corresponding channel, and thus learn more comprehensive features within a wide frequency range compared with vanilla ViTs, which can be clearly observed in Fig. ((a) and 1(b).
ViTs. In
Moreover, we find that lower layers often need more local information, while higher layers desire more global information, which also accords with the observations in [27]. This is because, like in human visual system, the details in high frequency components help lower layers to capture visual elementary features and also to gradually gather local information for having a global understanding of the input. Inspired by this, we design frequency ramp structure. particular, from lower to higher layers, gradually feed more channel dimensions to low- freeuenccyy mixer and fewer channel

---

dimensions to sesh-freuuency mixer. This structure can trade-ofi off high-freeun frequency and low-free ww-frequeney components across all layers. Its effectiveness has been verified by experimental results in Sec. 4.
Experimental results show that iFormer surpasses state-of-the-art ViT' and CNNs on several vision tasks, including image classification, object detection and segmentation. For example, shown in Fig. I(c), with different model sizes, iFormer makes consistent improvements over popular frameworks on ImageNet-1K [28], e.g.. DeiT [29], Swin [5] and ConvNeXt [30]. Meanwhile, iFormer outperforms recent frameworks on COCO [31] detection and ADE20K [32] segmentation.
2 Related work
Transformers [1] are firstly proposed for machine translation tasks and then become popular in other tasks like natural language understanding [33-35] and generation [36, 37J in NLP domain, as well as image classification [18, 29, 38], object detection [6, 39, 40] and semantic segmentation [41, 42] in computer vision. The attention module in Transformers has an outstanding ability to capture global dependency. but it makes the models produce similar representations across layers [27]. Moreover, self-attention mainly captures low-frequency information and tends to neglect high-frequency components related to the detailed information [14].
CNNs [43-47] are the de-facto model for vision tasks due to their outstanding ability to model local dependency [47-49] as well as extract high-frequency [19, 50J. With these advantages, CNNs rapidly introduced into Transformers in a serial or parallel manner [23-26, 51-53J. For serial methods, convolutions are applied at different positions of the Transformer. CvT [25] and PVT-v2 [54] replace the hard patch embedding with a layer of overlapping convolution. VVT [51]. ViT [55] and V [21] further stack several layers of convolutions as the stem for models, which is found helpful in training and achieving better performance. Besides the stem, ViT-hybrid [18], CoANNet [24]. Hybrid-MS [56] and UniFormer [22] design early stages with convolution layers. However, the combination of convolution and attention in a serial order means each layer can only process either high or low frequency and neglects the other part. To enable each layer to process different frequencies, we adopt the parallel manner to combine convolution and attention in a token
mixer.
Compared with serial methods, there are not many works combining attention and convolution in parallel manner in literature. CoaT [26] and ViTAE [23] introduce convolution as a branch parallel to attention and utilize elementwise sum to merge the output of the two branches. However, Raghu er al. find that some channels tend to extract local dependency while others are for modeling global information [27], indicating redundancy for the current parallel mechanism to process all channels in different branches. In contrast, we split channels into branches of high and low frequencies. GLiT [53] also adopt parallel manner but it directly concatenate the features from convolution and attention branches as the mixer output, lacking the fusion of features in different frequencies. Instead, we design a explicit fusion module to merge the outputs from low- and high-frequency branches.
3 Method
3.1 Revisit Vision Transformer
We first revisit the Vision Transformer. For vision tasks, Transformers first split the input image into a sequence of tokens, and each patch token is projected into hidden representation vector with a leaner layer, denoted as {æ1, 22,,.., ..., FNN) or X E R XC, where N is the number of patch tokens and C' indicates the dimension of features. Then, all of the tokens are combined with a positional embedding and fed into the Transformer layers that contain multi-head self-attention (MSA) and feed-forward network (FFN).
In MSA, the attention-basse mixer exchanges information between all patch tokens so that it strongly focuses on aggregating the global dependency across all layers. However, excessive propagation of global information would strengthen the low-frequency representation. It can be seen from the visualization of Fourier spectrum in Fig. ((a) that low-frequency information dominates the representations of VIT [18]. This actually impairs the performance of ViTs, as it may deteriorate the frequncyy components, e.g., local textures, and weakens the modeling capability of ViTs [14]. In the visual data, high- tighffrequency information is also discriminative and can benefit many tasks
3

---

Input 3xit xW
Stage1
Stage2
Stage3
Stage4
H W G 4
H W C2x 8
H
N
x
B
16
4
32
FFN
Patch eme mmbeding
Bujpaquee yaRd
spacd
Patch aune iFormer Blocks
Norm
iFormer Blocks
iFormer Blocks
iFormer Blocks
Supaa
Inception
Norm
The overall architecture of iFormer and details of iFormer block For each block, yellow and green indicate low- and high-frequency information, respectively. Best viewed color.
Figure 2:
[19, 20]. Hence, to address the issue, propose simple and efficient Inception Transformer, as shown in Fig. 2, with two key novelties, i.e., Inception mixer and frequency ramp structure.
3.2 Inception token mixer
We propose an Inception mixer graft the powerful capability of CNNs for extracting high-frequency representation to Transformers. Its detailed architecture depicted in Fig. 3. We use the name "Inception" since the token mixer is highly inspired by the Inception module [46, 57-59] with multiple branches. Instead of directly feeding image tokens into the MSA mixer, the Inception mixer first splits the input feature along the channel dimension, and then respectively feeds the split components into high-frequency mixer and low-frequence mixer. Here the high-frequency mixer consists max-pooling operation and parallel convolution operation, while the fow-frequency mixer is implemented by a self-attention.
Fusion
Upsample
Yx
Linew
RWFonn
Attention
MasPuul
Linsar
AvePool
Aat
Technically, given the input feature map X E RNxC, it is factor- ized into X,, R NCC, and X; I NCr along the channel dimension, where Cy + C = C. Then, X and X are assigned to high-frequency mixer and low-frequency mixer respectively.
Split
Figure 3: The details of In- ception mixer.
High-frequency mixer. Considering the sharp sensitiveness of the maximum filter and the detail perception of convolution operation, we propose a parallel structure to learn the high-frequency components. We divide the input X into
X hi N.Nx e, and X h2 RNNx Ct. along the channel. As shown in Fig. 3, X is embedded with a max-pooling and a linear layer [46], and X fed into a linear and a depthwise convolution layer [60-62]:
Y n FC (MaxPool (Xm)))
(1)
Y h2 = DwConv (FC (Xxz)),
(2)
where in and 12 denote the outputs of high-frequency mixers.
Finally, the outputs of low- and sigh-frequency mixers are concatenated along the channel dimension:
Ye Concat (Y,, Y Y22).
(3)
The upsample operation in Eq. (7) selects the value of the nearest point for each position to be interpolated regardless of any other points, which results in excessive smoothness between adjacent tokens. We design a fusion module to elegantly overcome this issue, i.e. depthwise convolution exchanging information between patches, while keeping cross-channel linear layer that works per location like in previous Transformers. The final output can be expressed as
Y = FC + DwConv (Y.)).
(4)

---

Like the vanilla Transformer, our iFormer is equipped with feed-forward network (FFN), and differently it also incorporates the above Inception token mixer (ITM); LayerNorm (LN) is applied before ITM and FFN. Hence the Inception Transformer block is formally defined as
Y = + ITM (LN (X)),
(5)
H ==Y + FFN (LN (Y)).
(6)
Low-frequency mixer. We use the vanilla multi-head self-attention communicate information among all tokens for the low-frequency mixer. Despite the strong capability of the attention for learning global representation, the large resolution of feature maps would bring large computation cost in lower layers. We therefore simply utilize an average pooling layer to reduce the spatial scale of X, before the attention operation and an upsample layer to recover the original spatial dimension after the attention. This design largely reduces the computational overhead and makes the attention operation focus on embedding global information. This branch can be defined as
Yi Upsample (MSA (AvePooling (X:)))
where Y, is the output of low- www-frequency mixer. Note that the kernel size and stride for the pooling and upsample layers set to 2 only at the first two stages.
3.3 Frequency ramp structure
In the general visual frameworks, bottom layers play more roles capturing high-frequency details while top layers more in modeling low-frequency global information, i.e., the hierarchical repre- sentations of ResNet [47]. Like humans, by capturing the details in high frequency components, lower layers can capture visual elementary features, and also gradually gather local information to achieve a global understanding of the input. We are inspired to design a frequency ramp structure which gradually splits more channel dimensions from lower to higher layers to low-frequency mixer and thus leave fewer channel dimensions to high- figh-frequeccy mixer. Specifically, as shown in Fig. 2. backbone has four stages with different channel and spatial dimensions. For each blocks, define channel ratio to better balance the high-frequency and low frequency components, i.e., Sh and G. where C G 1. In the proposed frequency ramp structure, C. gradually decreases from shallow to deep layers, while C gradually increases. Hence, with the flexible frequency ramp structure, iFormer can effectively trade-off high- and low-frequency components across all layers. The configuration of different iFormer models will be described in the appendix.
Experiments
We evaluate our iFormer on several vision benchmark tasks, i.e., image classification, object detection and semantic segmentation, by comparing it with representative ViTs, CNNs and their hybrid variants. Ablation analysis is also conducted to show the contribution of each novelty in our method. More results will be reported in the appendix.
4.1 Results on image classification
Setup. For image classification, we evaluate iFormer the ImageNet dataset [28]. We train the iFormer model with the standard procedure in [6, 22, 29]. Specifically, we use AdamW optimizer with initial learning
x 10-3 via cosine decay [70], momentum of 0.9, and weight decay of 0.05. We set the training epoch number as 300 and the input size as 224 x 224. We adopt the same data augmentations and regularization methods in DeiT [29] for fair comparison.
We also use LayerScale [7] to train deep models. Like previous studies [5, 67], we further fine tune iFormer on the input size of 384 x 384, with the weight decay of x 10 learning rate of 1 10-5, batch size of 512. For fairness, we adopt Timm [72] to implement and train iFormer.
Results. Table summarizes the image classification accuracy of all compared methods on ImageNet. For the small model size (~200M,, our iFormer surpasses both the SoTA ViTs and hybrid ViTs, although some ViTs, e.g., Swin [5], Focal [64] and CSwin [65], actually already introduce convolution- like inductive bias into their architectures, and hybrid ViTs directly integrate convolution into ViTs. Specifically, our iFormer-S respectively gains 0.7% and 0.5% top-1 accuracy ad advantage over SoTA
5

---

Table 1: Comparison of different types of models on ImageNet- KK [28].
#Param. (M)
FLOPs
Input Size
ImageNet
Model Size
Arch.
Method
Train
Test
Top-1
Top-5
RSB- 33B-ResNet-50 [47, 63]
26
4.1
224
224
80.4
CNN
ConvNeXt-T [30]
224
Deit-S [29]
22
224
224
79.8
95.0
PVT-S [6]
224
79.8
small model size
T2T-14 [38]
22
5.2
224
224
80.7
VIT
(~-20M)
Swin-T [5]
29
4.5
224
224
81.3
95.5
Focal-T [64]
29
4.9
224
224
82.2
95.9
CSwin-T [65]
224
82.7
CvT-13 [25]
20
4.5
224
224
81.6
CoAtNet-O [24]
224
81.6
Container [66]
22
8.1
224
224
82.7
Hybrid
VITAE-S [23]
24
5.6
224
224
82.0
95.9
ViTAEv2-S [67]
19
5.7
224
224
82,6
96.2
UniFormer-S [22]
224
224
82.9
iFormer-S
20
4.8
224
224
83.4
96.6
RSB-ResNNet-001 [47, 63]
45
7.9
224
224
81.5
CNN
RSB-ReeNNe-15211 [47, 63]
60
11.6
224
224
82.0
ConvNeXt-S [30]
224
medium model size (~550M
PVT-L [6]
61
9.8
224
224
81.7
T2T-24 [38]
64
13.2
224
82.2
VIT
Swin-S [5]
50
8.7
224
224
83.0
96.2
Focal-S [64]
51
9.1
224
224
83.5
96.2
CSwin-S [65]
35
6.9
224
224
83.6
CvT-21 [25]
32
7.1
224
224
82.5
CoAtNet-1 [24]
42
8.4
224
224
83.3
Hybrid
ViTAEv2-48M [67]
224
83.8
UniFormer-B [22]
50
8.3
224
224
83.9
iFormer-B
48
9.4
224
224
84.6
97.0
RegNetY-16GF [29, 68]
84
16.0
224
224
82.9
CNN
ConvNeXt-B [30]
15.4
224
83.8
large model size (~00)))
DeiT-B [29]
86
17.5
224
224
81.8
95.6
Swin-B [5]
88
15.4
224
224
83.3
96.5
VIT
Focal-B [64]
90
16.0
224
224
83.8
96.5
CSwin-B [65]
78
15.0
224
224
84.2
BoTNet-T7 [69]
79
19.3
256
256
84.2
CoAtNet-3 [24]
168
34.7
224
224
84.5
Hybrid
ViTAEv2-B [67]
90
24.3
224
224
84.6
96.9
iFormer-I.
14.0
224
84.8
ViT's ( (ie, CSwin-T) and hybrid ViT's ( (ee, UniFormer-S), while enjoying the same or smaller model size.
For the medium model size (~50M), iFormer-B achieves 84.6% top-1 accuracy, and improves over the SoTA ViTs and hybrid ViTs with similar model sizes by significant margins 1.0% and 0.7% respectively. For CNNs, similar to comparison results on medium model size, our iFormer-B B outperforms ConvNeXt-S by 1.5%. As for the large mode (~100M), on small and medium model sizes.
observe similar results
Table 2 reports the fine-tuning accuracy the larger resolution, iee. 384 x 384. One can observe that iFormer consistently outperforms the counterparts by a significant margin across different computation settings. These results clearly demonstrate the advantages of iFormer image classifications.

---

Table 2: Fine-tuning Results with larger resolution (384 x 384) on ImageNet- [28]. The models gray color are trained with larger input size.
#Param. (M)
FLOPs (G)
Input Size
ImageNet Top-1
Method
Train
Test
EfficientNet-55 [73]
30
9.9
456
456
83.6
EfficientNer12-SS [74]
22
8.5
384
384
83.9
CSwin-T1384 [65]
23
14.0
224
384
84.3
CvT-131384 [25]
20
16.3
224
384
83.0
CoA1Net-01384 [24]
20
13.4
224
384
83,9
ViTAEv2-S1384 [67]
17.8
224
384
83.8
iFormer-S1384
20
16.1
224
384
84.6
EfficientNet-B7 [73]
66
39.2
600
600
84.3
EfficientNetV2-M [74]
54
25.0
480
480
85.1
ViTAEv2-48M 1384 [67]
49
41.1
224
384
84.7
CSwin-S7384 [65]
35
22.0
224
384
85,0
CoAtNet-11388 [24]
42
27.4
224
384
85.1
FoormerBB388
48
30.5
224
384
85.7
EfficientNerV2-I. [74]
121
480
480
85.7
Swin-B1384 [5]
88
47.0
224
384
84.2
#Swin-B138k [65]
78
47.0
224
384
85.4
ViTAEv2-B1384 [67]
90
74.4
224
384
85.3
CoA [24]
75
224
384
85.7
iFormer-L
87
45.3
224
384
85.8
Table Performance of object detection and instance segmentation on COCO val2017 [31]. AP and Apm represent bounding box AP and mask AP, respectively. All models are based on Mask R-CNN [75] and trained by training schedule. The FLOPs are measured at resolution 800 1280.
#Param. (M)
FLOPs
Mask R-CNN AP
Method
(G)
AP
AP
Apm
AP
AP%
ResNet50 [47]
260
38.0
58.6
41.4
34,4
55.1
36.7
PVT-S [6]
44
245
40.4
62.9
43.8
37.8
60.1
40.3
TwinsP-S [76]
44
245
42.9
65.8
47.1
40.0
62.7
42.9
Twins-S [76]
44
228
43.4
66.0
47.3
40.3
63.2
43.4
[win-T
48
264
42.2
64.6
46.2
39.1
61.6
42.0
ViL.S [77]
45
218
44.9
67.1
49.3
41.0
64.2
44.1
Focal-T [64]
49
291
44.8
67.7
49.2
41.0
64.7
44.2
UniFormer-S..4 [22]
41
269
45.6
68.1
49.7
41.6
64.8
45.0
iFormer-S
40
263
46.2
68.5
41.9
65.3
45.0
ResNet101 [47]
63
336
40.4
61.1
44.2
36.4
57.7
38.8
X101-32
63
340
41.9
62.5
45.9
37.5
59.4
40.2
PVT-M [6]
64
302
42.0
64.4
45.6
39.0
61.6
42.1
TwinsP-B [76]
64
302
44.6
66.7
48.9
40.9
63.8
44.2
Twins-B [76]
76
340
45.2
67.6
49.3
41.5
64.5
44.8
Swin-S [5]
69
354
44.8
66.6
48.9
40.9
63.4
44.2
Focal-S [64]
71
401
47.4
69.8
51.9
42,8
66.6
46.1
CSWin-S [65]
54
342
47.9
70.1
52.6
43.2
67.1
46.2
UniFormer-B [22]
69
399
47.4
69.7
52.1
43.1
66.0
46.5
iFormer-B
67
351
48.3
70.3
53.2
43.4
67.2
46.7
4.2 Results object detection and instance segmentation
Setup. We evaluate iFormer on the COCO object detection and instance segmentation tasks [31], where the models are trained on 118K images and evaluated on validation set with 5K images. Here, we use iFormer the backbone in Mask R-CNN [75]. In the training phase, we use iFormer pretrained on ImageNet to initialize the detector, and adopt AdamW train with an initial learning rate of x 10-4, a batch size of 16, and 1x training schedule with 12 epochs. For training, the input

---

images are resized to be 800 pixels on the shorter side an no more than 1,333 pixels on the longer side. For the test image, its shorter side is fixed 800 pixels. All experiments are implemented on mmdetection [78] codebase.
Results. Table 3 reports the box mAP (APt) and mask mAP (APm) of the compared models. Under similar computation configurations, iFormers outperforms all previous backbones. Specifically, compared with popular ResNet [47] backbones, our iFormer-S brings 8.2 points of AP and 7.5 points APM improvements over ResNer50. Compared with various Transformer backbones, our iFormers still maintain the performance superiority over their results. For example, our iFormer-B surpasses UniFormer-B [22], Swin-S [5] by 0.9 points of AP" and 3.5 points of AP" respectively.
4.3 Results on semantic segmentation
Setup. We further evaluate the generality of iFormer through challenging scene parsing benchmark on semantic segmentation, i.e.,
Table 4: Semantic segmentation with semantic
FPN [79] on ADE20K [32]. The FLOPs are mea-
sured at resolution 512x2048.
ADE20K [32]. The dataset contains 20K train- ing images and 2K validation images. We adopt iFormer pretrained on ImageNet as the back- bone of the Semantic FPN [79] framework. Fol- lowing PVT [6] and UniFormer [22], we use AdamW W with initial learning rate of 2 x 10-4 with cosine learning rate schedule to train 80k iterations. All experiments are implemented mmsegmentation [80] codebase.
#Param. (M)
FLOPs
mloU
Method
(G)
(%)
ResNe150 [47]
29
183
36.7
PVT-S [6]
28
161
39.8
TwinsP-S [76]
28
162
44.3
Twins-S [76]
28
144
43.2
Swin-T [5]
32
182
41.5
UniFormer-S [22]
25
199
46.2
UniFormer-S [22]
25
247
46.6
Results. In Table 4, we report the mloU re- sults of different backbones. On the Semantic FPN [79] framework, our iFormer consistently outperforms previous backbones on this task, in- cluding CNNs and (hybrid) ViTs. For instance, iFormer-S achieves 48.6 mloU, surpassing UniFormer-
UniFormer-B B [22]
54
48.0
iFormer-S
24
181
48.6
[22] by 2.0 mloU, while using less computation complexity. Moreover, compared with UniFormer-
[22], our iFormer-S still achieves 0.6 mloU improvement with only 1/2 parameters and nearly 1/3 FLOPs.
4.4 Ablation study and visualization
In this section, we conduct experiments to better understand iFormer. All the models are trained for 100 epochs on ImageNet, with the same training setting as described Sec. 4.1.
Inception token mixer. The Inception mixer is proposed to augment the perception capability of ViTs in the frequency spectrum. To evaluate the effects of the components the Inception mixer, remove the max-pooling or convolution from the full model and then report the results in Table 5, where /f and denote whether or not the corresponding branch is enabled. Observably, combining attention with convolution and max-pos max-pooling can the highest classification accuracy. To further explore this scheme, Fig. visualizes the Fourier spectrum of the Attention, MaxPool and DwConv branches
Table Ablation study Inception mixer and frequency ramp structure on ImageNet- - All the models trained for 100 epochs.
Attention
MaxPool
DwConv
#Param. (M)
FLOPss (G)
Top- (%)
Mixer
x
4,9
81.2
X
81.4
/
4.8
81.5
L. C//C= C./C C/01,00/01
19
4.7
80.5
Structure
4.7
80.7
20
4.8
81.2

---

MaxPool
DwConv
Attention
MaxPool
DwConv
Attention
(a) 4-th layer
8-th layer
Figure 4: (a) (b) Fourier spectrum of iFormer-S for the MaxPool, DwConv and Attention branches the Inception mixer. We can observe that attention mixer tends to reduce high- frequencies, while MaxPool and DwConv enhance them.
(a) Input
(b) Swin-T
(c) iFormer-S
Figure 5: Grad-CAM [81] activation maps of Swin-T [5] and iFormer-S trained on ImageNet.
in Inception mixer. We can see the attention mixer has higher concentrations on low frequencies; with the high-frequency mixer, i.e. convolution and max-pooling, the model is encouraged to learn high frequency information. Overall, these results prove the effectiveness of the Inception mixer for expanding the perception capability of the Transformer the frequency spectrum.
Frequency ramp structure. Previous investigations [27] show requirement of more local information at lower layers of the Transformer and more global information at higher layers. We accordingly assume that a frequency ramp structure, i.e., decreasing dimensions at high-frequency components and increasing dimensions low- oowffrequeccy components from lower to higher layers, has better trade-off between high-frequency and low-frequency components across all layers. order to justify this hypothesis, we investigate the effects of the channel ratio ( and & Table 5. can clearly seen that the model with Ci/C 1, Ca/C + outperforms the other two models, which is consistent with the previous investigations. Hence, this indicates the rationality of the frequency ramp structure and its potential for leaning discriminating vision representations.
Visualization. We visualize the Grad-CAM [81] activation maps of iFormer-S well as Swin-T [S] models trained on ImageNet-IK IK in Fig. 5. It can be seen that compared with Swin, iFormer can more accurately and completely locate the objects. For example, in the hummingbird image, iFormer skips the branch and accurately attends to the whole bird including the tail.
5 Conclusion
In this paper, we present an Inception Transformer (iFormer), novel and general Transformer back- bone. iFormer adopts channel splitting mechanism to simply and efficiently couple convolution/max- pooling and self-attention, giving more concentrations on high frequencies and expanding the per- ception capability of the Transformer in the frequency spectrum. Based on the flexible Inception token mixer, we further design a frequency ramp structure, enabling effective trade-off between freguerry and wwwffreuency components across all layers. Extensive experiments show that iFormer outperforms representative vision Transformers on image classification, object detection and caneral-purrose
semantic segmentation, demonstrating the great potential of iFormer to backbone for computer vision. We hope this study will provide valuable insights for the community design efficient and effective Transformer architectures.
Limitation. One obvious limitation of the proposed iFormer is that requires manually defined channel ratio the frequency ramp structure i.e., Cp and C for each iFormer block, which needs rich experience to define better on different tasks. is not trained on large scale datasets, e.g.

---

ImageNet-21K [48], due to computational constraint, which will be explored in further. Also, iFormer requires manually defined channel ratio in the frequency ramp structure i.e., & and & for each iFormer block, which needs rich experience define better on different tasks. A straightforward solution would be to use neural architecture search.
Acknowledgement
Weihao Yu would like to thank TRC program and GCP research credits for the support of partial computational resources. This project is in part supported by the National Research Foundation Singapore under its Al Singapore Programme (Award Number: AISG2-RP-2021-023)
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Schastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02. 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[6] Wenhai Wang, Enze Xie, Xiang Li, Deng- ling-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 568-578, 2021.
[7] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, 2022.
[8] Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In International Conference on Machine Learning, pages 4487-444 4499. PMLR, 2021.
[9] Anurag Arnah, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 6836-6846, 2021.
[10] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer- based object detection. arXiv preprint arXiv:2012.09958 2020.
[11] Yuxin Fang, Bencheng Liao, Xinggang Wang. Jiemin Fang. Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34, 2021.
[12] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfong Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-tee sequence-to-scquente to-sqquence perspective with transformers. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pages 6881 16890, 2021.
[13] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, Advances in Neural Information Processing Systems, 34, 2021.
10

---

[14] Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Reprexentations, 2021.
[15] Jean Bullier. Integrated model of visual processing. Brain research reviews, 36(2-3)96-107,,, 2001.
[16] Moshe Bar. A cortical mechanism for triggering top-down facilitation in visual object recognition. Journal of cogmitive neuroscience, 15(4):600-60, 2003.
[17] Louise Kauffmann, Stephen Ramanoël, and Carole Peyrin. The neural bases of spatial frequency processing during scene perception. Frontiers in integrative neuroxcience, 8:37, 2014.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[19] Haohan Wang. Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the gencralization of convolutional neural networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684-8694, 2020.
[20] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness computer vision. Advances in Neural Information Processing Systems, 32, 2019.
[21] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, Ross Girshick. Early convolutions help transformers see better. Advances in Newral Information Processing Systems, 34:30392-30400 2021.
[22] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.0460 2022.
[23] Yufei Xu, Qiming Zhang. Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34, 2021.
[24] Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965 3977, 2021.
[25] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu. Xiyang Dai. Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.
[26] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9981-9990, 2021.
[27] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
[28] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 -25.. Icce, 2009.
[29] Hugo Touvron. Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[30] Zhuang Liu, Hanzi Mao. Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. arXiv preprint arXiv:2201.035:: 2022.
[31] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitncck. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[32] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k datasct. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[33] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: traitiain of deep bidirec- tional transformers for language understanding, arXiv preprint arX0v:1800004805, 2018.
11

---

[34] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances neural information processing systems, 32, 2019.
[35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer, and Veselin Stnyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[36] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[37] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[38] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-tor toeens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[39] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[40] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[41] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-see -o-suueeee perspective with transforerrs In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881 6890, 2021.
[42] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan I. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102 04306, 2021.
[43] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(12) 2324,, 1998.
[44] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[45] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1556, 2014.
[46] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[48] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1400. 1556, 2014.
[50] Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3435 3444, 2019.
[51] Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojic Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information P Processing Systems, 34, 2021.
[52] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang. Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01697 2022.
12

---

[53] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun. Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-21, 2021.
[54] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lui, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, pages 1-10, 2022.
[55] Benjamin Graham, Alaackdin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22599112669 2021.
[56] Yucheng Zhao, Guangting Wang, Chuanxin Tang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. A battle of network structures: An empirical study of cnn, transformer, and mlp. arXiv preprint arXiv:2 13002, 2021.
[57] Sergey loffc and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 4566. PMLR, 2015.
[58] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 2888-2826, 2016.
[59] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception- resnet and the impact of residual connections on lcarning, In Thirty-first AAAI conference on artificial intelligence, 2017.
[60] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258 2017.
[61] Franck Mamalet and Christophe Garcia. Simplifying convnets for fast learning. In International Conference Artificial Neural Networks, pages 58-65. Springer. 2012.
[62] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4510-4520, 2018.
[63] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. Xixiv preprint arXi:22100.00476 2021.
[64] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attentio for local-globe interactions in vision transformers. arXiv preprint (eXXv:2107.00611 2021.
[65] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan. Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2 2021.
[66] Jiasen Lu, Roozbeh Mottaghi, Aniruddha Kembhavi, et al. Container: Context aggregation networks. Advances in Neural Information Processing Systems, 34, 2021.
[67] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. arXiv preprint arXiv:2202.10108, 2022.
[68] Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10428-10436, 2020.
[69] Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IFFF/CVF conference computer vision and pattern recognition, pages 16519-16529, 2021.
[70] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arxiv: 03388, 2016.
[71] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32 42, 2021.
13

---

[72] Ross Wightman. Pytorch image models. https:///gtthne http:///hhtaa......aaaaaaaaaaaaa 2019.
[73] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105 6114. PMLR, 2019.
[74] Mingxing Tan and Quoe Le. Efficientmetv2: Smaller models and faster training. In International Conference on Machine Learning, pages 10096-10106. PMLR, 2021.
[75] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[76] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
[77] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: new vision transformer for high-resolution image encoding. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2998-3008, 2021.
[78] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benehmark. arXiv preprint arXiv: Drkiv:/90... 190..0755,.0500000 2019.
[79] Alexander Kirillov, Ross Girshick, Kaiming Hc, and Piotr Dollár. Panoptic feature pyramid networks. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399-6408, 2019.
[80] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. /ttpp://github.. wwt.ccoo/ooooe.cc coom/open-oee mml ab/ //m////memtttiinnnn.mmmmmm//omot 2020.
[81] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[82] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418 434, 2018.
[83] Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint 10Xv::2006.06650 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The code will be released in future.
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "x-IhQInmNkp", "reviewer": "Reviewer_X9JV", "summary": "In this paper, the authors propose a novel iFormer architecture to capture both the high and low frequencies in visual data. Motivated by the observation that ViT tends to capture few high-frequency signals (i.e., showing the characteristics of low-pass filters), the authors propose an Inception token mixer, comprising high- and low-frequency mixers to extract the corresponding frequency information on the feature channels. Also, derived from the finding that lower and higher layers generally need more local and global information, respectively, the authors devise a frequency ramp structure to trade off different-frequency components across all the layers. Experiments across various tasks including classification, detection, and segmentation, convincingly validate the effectiveness of the proposed iFormer with detailed explanations and discussions。", "questions": "- Have you tried other balancing strategies other than the manner of setting the sum of the ratios to 1 in the proposed frequency ramp structure?\n- Can you share some insights on the future work that can be derived from this work?\n- Do you plan to make your code publicly available for follow-up research?\n\n", "limitations": "Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The proposed iFormer is very well-motivated. Deriving from the results of the Fourier spectrum and the relative log Fourier amplitudes, the authors accordingly develop the iFormer backbone to enhance the perception capability of the Transformer in the frequency spectrum.  \n- The experiments are comprehensive and convincing. Results across three challenging tasks show that the proposed iFormer outperforms existing Transformer architecture. Extensive ablation studies and visualizations also demonstrate the effectiveness of the devised two key modules of the Inception token mixer and the frequency ramp structure.\n- The paper is well-written and easy to follow. I enjoy the writing style that starts from the detailed discussions and analysis on the derived observations and then accordingly elaborates the proposed method.\n\n[Weaknesses]\n- The authors are suggested to denote the symbols like X_{h1}, X_{h2}, Y_{h1}, and Y_{h2} in Fig. 3, which will make it much easier to relate Fig. 3 back to the text.\n- The design of a frequency ramp structure that uses a channel ratio to balance the high- and low-frequency components seems straightforward. I wonder if the authors have tried other designs for frequency information balancing and how they perform.\n- It will be great if the authors could share some insights on the future work that can be derived from this work.\n- The punctuations for Eq. 1 and Eq. 5 are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mTWPy6FAs_h", "reviewer": "Reviewer_yYdE", "summary": "This paper presents an iFormer architecture to capture high-frequency and low-frequency visual information. The proposed inception mixer module (i) splits the feature maps along the channel dimension and (ii) applies parallel convolution/max-pooling/self-attention paths to gather high-frequency and low-frequency information.\nBesides, the authors also introduce a simple frequency ramp structure that adjusts the channel partitions across different layers. The experimental results look encouraging.", "questions": "Please carefully address the above-listed weaknesses. I will increase the ratings if the proposed method shows advantages under either the TINY or  LARGE configuration.\n\n> Update:\n\nAfter reading the comments of other reviewers and the authors' responses carefully, I tend to increase my ratings due to the potential high impact on the community.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "> Strengths\n\n👍 The motivation is clear and the presented idea is very simple and easy to follow.\n\n👍 The proposed iFormer shows slightly better performance while enjoying smaller GFLOPs.\n\n> Weaknesses\n\n👎 The proposed method is TOO NAIVE & lacks NOVELTY. According to Figure 3 and Figure 4, we can see that these three different branches capture information of different frequencies. However, the default **multi-head self-attention is capable to learn such information given enough training epochs** and previous efforts ([ConVit, ICML 2021](https://arxiv.org/abs/2103.10697), [EarlyConvolutions, NIPS 2021](https://proceedings.neurips.cc/paper/2021/file/ff1418e8cc993fe8abcfe3ce2003e5c5-Paper.pdf), [MetaFormer, CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf))have shown that **injecting the inductive biases with either convolution or max-pooling show better training efficiency**.\n\n👎 The proposed method does not provide any results under LARGE configurations. The authors should justify the position of the proposed iFormer on either a TINY configuration that pursues faster running speed & better efficiency or a LARGE configuration that pursues higher accuracy. However, the authors fail to show the proposed method is promising in both aspects. Therefore, it is necessary to present the comparisons to either [Mobile-Former, CVPR 2022](https://arxiv.org/pdf/2108.05895.pdf)/[MobileViT, ICLR 2022](https://openreview.net/pdf?id=vh-0sUt8HlG) or Swin-L/Focal-L/CSwin-L under the fair settings instead of only on an intermediate setting that the community does not care.\n\n👎 The authors should include the results with the combination of ``Attention and DWConv''. According to our previous experience, we can apply large kernel DWConv to achieve better performance than the naive MaxPool operation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "jj9oE2rG2kC", "reviewer": "Reviewer_Bzyx", "summary": "This paper designs a new family of multi-branched visual transformer models, called Inception Transformer. Motivated by the spectrum properties of local operations (convolution & pool) and long-range operations (self-attention), a multi-branched Inception Mixer block is proposed to capture both high/low frequency information from the images. A channel splitting mechanism is applied to adopt hybrid operation at each stage. \nIn addition, the author assumes that low layers of network favor local/high-frequency pattern and high layers prefer global/low-frequency pattern. Therefore, a frequency ramp structure is utilized to increase the attention dimensions and shrink the convolution dimensions as the network goes deeper.\nComprehensive evaluation on image classification, object detection, instance segmentation and semantic show strong performance.", "questions": "1.\tI hope the author should provide more discussions on the core difference between iFormer and previous multi-branched network structure.\n2.\tFrequency ramp strategy. In the “Frequency ramp structure” section, no description on the how to increase the channel ratio C_l/C. Is it linear scaling according to the depth or exponential increasing? I see the channel number in the appendix, but it seems to be a “magic number” that I am not sure how did you decide on the current number. Moreover, an ablation on how to select the channel number should be included in the experimental part. \n3.\tDown-sample and Up-sample Self-attention. To reduce the computational cost, the author proposes in Equation 7 to (1) down-sample the feature map with average pooling (2) Do self-attention (3) Up-sample to the full size. This operation is not mentioned in previous studies. I am curious about its computational complexity and performance, with the vanilla ViT structure.", "limitations": "Please put more discussions on societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I enjoy reading this paper, with straightforward motivation and clear writing.\nStrengths:\n1.\tStrong motivation. As the self-attention operation captures the long-range dependency (shown in Figure 1(a)(b)), a local branch with convolution and pooling operations is well-motivated to compensate the frequency preference of attention operation. \n2.\tSimple and efficient structure. Instead of applying different operations on the same feature maps, the high/low frequency mixer is applied on different channel slices in a parallel manner. Also, the structure that gradually increases the global operation channel makes sense to me\n3.\tPromising empirical results. On ImageNet, iFormer-S/iFormer-B surpass the current best performed architecture by 0.5 %/0.7% in accuracy on 224 resolution. The iFormer-B achieves 48.3 mAP on COCO2017 val and 48.6 mIOU on ADE20K. All results are surprisingly well.\n\t\nWeakness:\n1.\tLack of discussion with previous multi-branched network structure. Since the current paper focus on the design of a multi-branched block structure, a detailed comparison and discussion should be included, like ResNeXt[1], GoogleNet Family and Inception Family. Although they did not include self-attention in their branches, some of the networks do have the mixed kernel size (1x1 Vs 5x5), which is also another form of low/high frequency mixer. Also, the multi-branched network seems to have better generalization and optimization properties [2] compared with the single-branched counter parts. A simple citation is insufficient to highlight your improvement over them. In addition, as the paper is largely driven by the low/high pass filter design in signal processing, I would encourage the author incudes some reference on this topic.\n2.\tProblems on ablation study. To be scientific, ablation study means quantifying the influence of each modular design by removal/changing of one component from the entire system. But in table 5 row 1-4, the authors are accurately adding one component at each time. I know that a lot of paper take this style of ablation study, but scientifically, this is not a valid ablation study. Better fix this. \n\n[1] Aggregated Residual Transformations for Deep Neural Networks (CVPR 2017)\n[2] Deep Neural Networks with Multi-Branch Architectures Are Intrinsically Less Non-Convex (ICML 2019)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fKTBFqNONB", "reviewer": "Reviewer_S6Nf", "summary": "This paper proposes Inception Transformer (iFormer) to enhance the low-pass filters in standard transformers. Specifically, the authors introduce an Inception mixer that has parallel convolution/max-pooling path and self-attention path, capturing high- and low-frequency information respectively. The Inception mixer differs from existing parallel CNN/attention hybrids in two aspects: 1) Unlike [23, 26, 47] that process all channels in each branch, the Inception mixer adopts a channel splitting mechanism to reduce information redundancy. 2) The Inception mixer also contains a fusion module to merge the outputs from low- and high-frequency branches, instead of simple feature concatenation.\n\nAnother novelty of the proposed iFormer is the frequency ramp structure. It trade-offs high- and low-frequency info across different layers by gradually decreasing the channel dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer.\n\nThe full iFormer achieves SOTA performance on a series of vision tasks, including image classification, object detection and segmentation. It shows better performance-cost trade-off than representative ViTs, CNNs and their hybrid variants, showing the potential to serve as a general transformer backbone.", "questions": "- In the low-frequency mixer, multi-head self-attention is made more efficient by using average pooling first and then upsampling. How much saving is obtained in compute? Will this hurt the attention performance too much (although I get the intention is to focus on embedding global information in the attention branch)?\n- What's the exact policy to change the channel ratio across layers? It's linear? How do different policies perform (or do they perform differently at all)?\n- How does the feature fusion module compares to direct concatenation [47] in performance?", "limitations": "Both the limitations and potential negative societal impact are given. It makes sense to bring up the limitation that iFormer requires manually defined channel ratio in the frequency ramp structure.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ It's new to consider enhancing the capability of transformers to capture high-frequency information.\n+ The two key novelties--Inception mixer and frequency ramp structure--are intuitive and make sense. There are ablations provided to support their positive roles.\n+ Although the Inception mixer has a parallel structure that is similarly adopted by existing papers, there are still technical differences (ie. channel splitting and fusion module) that prove useful.\n+ Strong performance is obtained on various vision tasks, and that comes without sacrificing model efficiency.\n\nWeaknesses:\n- The paper should benefit from a detailed quantification of the high- and low-frequency information (e.g. via Fourier analysis) captured for each layer. Is there any interesting pattern across layers? This is the core contribution of the whole paper, but the analysis about how Inception mixer fulfills the task in its low- and high-frequency branches is missing (example qualitative visualization in Fig. 4 may not be enough).\n- As mentioned in the limitations, the channel splitting scheme is lacking analysis and/or ablations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept