CycleMLP: A MLP-like Architecture for Dense Prediction
Authors: Shoufa Chen
Enze Xie
Chongjian GE
Runjian Chen
Ding Liang
Ping Luo
Venue: ICLR 2022 Oral (2022)
Keywords: MLP
Dense Prediction

Abstract: This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions. As compared to modern MLP architectures, e.g. , MLP-Mixer, ResMLP, and gMLP, whose architectures are correlated to image size and thus are infeasible in object detection and segmentation, CycleMLP has two advantages compared to modern approaches. (1) It can cope
with various image sizes. (2) It achieves linear computational complexity to image size by using local windows. In contrast, previous MLPs have $O(N^2)$ computations due to fully spatial connections. We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g. Swin Transformer, while using fewer parameters and FLOPs. We expand the MLP-like models’ applicability, making them a versatile backbone for dense prediction tasks. CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation. In particular, CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs. Moreover, CycleMLP also shows excellent zero-shot robustness on ImageNet-C dataset.

Published as a conference paper at ICLR 2022
CYCLEMLP: A MLP-LIKE ARCHITECTURE FOR DENSE PREDICTION
Shoufa Chen' Enze Xie
Chongjian Ge
Runjian Chen'
Ding Liang"
Ping Luo
The University of Hong Kong Seme Research
3 Shanghai AI Laboratory, Shanghai, China
(shoufach, xieenze, rhettgee, richen] rjchen)econneee cooneect. .hku.b
1 bragcinn@eeeneimme..oo pluo8cs. hhu.hk
ABSTRACT
This paper presents a simple MLP-like architecture, CycleMLP, which is versa- tile backbone for visual recognition and dense predictions. As compared to mod- ern MLP architectures, e.g. MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Tou- vron al., 2021a), and gMLP (Liu al.. 2021a), whose architectures related to image size and thus are infeasible in object detection and segmen- tation, CycleMLP has two advantages compared to modern approaches. (I) It can cope with various image sizes. (2) It achieves linear computational com- plexity to image size by using local windows. contrast, previous ML.Ps have O(N2) computations due to fully spatial connections. We build family of models which surpass existing MLPs and even state-of-the-art Transformer- based models, e.g. Swin Transformer (Liu et al., 2021b), while using fewer parame- ters and FLOPs. We expand the MLP-like models' applicability, making them versatile backbone for dense prediction tasks. CycleMLP achieves competi- tive results on object detection, instance segmentation, and semantic segmenta- tion. In particular, CycleMLP- P-Tnny outperforms Swin-Tiny by 1.3% mloU ADE20K dataset with fewer FL.OPs. Moreover, CycleMLP also shows excel- lent zero-shot robustness on ImageNet-C dataset. Code is available at https: / //github. com/ Shwufahhen/Cycleeeeeeeee
cor-
INTRODUCTION
Vision models in computer vision have been long dominated by convolutional neural net- works (CNNs) (Krizhevsky et al., 2012; He et al., 2016). Recently, inspired by the successes in Natural Language Processing (NLP) field, Transformers (Vaswani et al., 2017) are adopted into the computer vision community. Built with self-attention layers, multi-layer perceptrons (MLPs), and skip connections, Transformers make numerous breakthroughs 2020; Liu et al., 2021b). More recently, (Tolstikhin et al., 2021; Liu et al., 2021a) have validated that building models solely MLPs and skip connections without the self-attention layers achieve surprisingly promising results on ImageNet (Deng et al., 2009) classification.
visual tasks (Dosovitskiy al.,
Despite promising results on visual recognition tasks, these MLP-like models can not used in dense pre- diction tasks (e.g., object detection and semantic segmentation) due to the three challenges: (1) Current models are composed of blocks with non-hierarchical architectures, which make the model infcasible to provide pyramid and high-resolution feature representations. (2) Current models cannot deal with flexible input scales due to the Spatial FC shown Figure 1b. The spatial FC is configured by an image-size related weight'. Thus, this
Scale Stepsize (HH) Variable
ImgNet COCO Top-1
ADE20K
FC
mloU
Channel
HW
/
79.4
35.0
36.3
Spatial
H²W2
x
80.9
X
Cycle
HW
81.6
41.7
42.4
Table Comparison of three types of FC operators.
We omit bias here for discussion convenience.

---

Published as a conference paper at ICLR 2022
Skpalc
6,0)=1
RxW
(a) Channel FC
4.7mm
RxW
(b) Spatial PC
6(00)=0
6:0))m
(d) Sq 3,5
(e) Sp H.S
(1) Sy
1,5
ixW
(c) Cycle PC
Figure I: (a)-(c): motivation of Cycle Fully-Connected Layer (Cycle FC) compared Channel FC and Spatial FC. (a) Channel FC aggregates features in the channel dimension with spatial size '1' can handle various input scales but cannot learn spatial context. (b) Spatial FC (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) has a global receptive field in the spatial dimension. However, its parameter size is fixed and it has quadratic computational complexity to image scale. (c) Our proposed Cycle Fully- Colly-Connected Layer (Cycle FC) has linear complexity the same as channel FC and a larger receptive field than Channel FC. (d)-(f): Three examples of different stepsizes. Orange blocks denote the sampled positions.
denotes the output position.
For simplicity, we omit batch dimension and set the feature's 's width to 1 here for example. Several more general cases can be found in Figure (Appendix G). Best viewed in color.
structure typically requires the input image with a fixed scale during both the training and infer- ence procedure. It contradicts the requirements of dense prediction tasks, which usually adopt multi-scale training strategy (Carion al., 2020) and different input resolutions in training and ference stages (Lin et al., 2014; Cordts et al., 2016). (3) The computational and memory costs of the current MLP models are quadratic to input image sizes for dense prediction tasks (e.g., COCO benchmark (Lin et al., 2014)).
To address the first challenge, we construct hierarchical architecture to generate pyramid features. For the second and third issues, propose novel variant of fully connected layer, named as Cycle Fully-Connecter Layer (Cycle FC), as illustrated in Figure Ic. The Cycle FC is capable of dealing with various image scales and has linear computational complexity to image size.
Our Cycle FC is inspired by Channel FC layer illustrated in Figure 1a, which is designed for channel information communication (Lin et al., 2013; Szegedy al., 2015; He et al., 2016; Howard et al., 2017). The main merit of Channel FC lies in that it can deal with flexible image sizes since it is configured by image-size agnostic weight of Cn and Coue- However, the Channel FC is infeasible to aggregate spatial context information due to its limited receptive field.
Our Cycle FC designed to enjoy Channel FC's merit of taking input with arbitrary resolution and linear computational complexity while enlarging its receptive field for context aggregation. Specif- ically, Cycle FC samples points cyclical style along the channel dimension (Figure Ic). this way, Cycle FC has the same complexity (both the number of parameters and FLOPs) as channel FC while increasing the receptive field simultaneously. To this end, we adopt Cycle FC to replace the Spatial FC for spatial context aggregation (i.e., token mixing) and build a family of ML.P-ikke models for both recognition and dense prediction tasks.
The contributions of this paper are as follows: (1) We propose new MLP-like operator, Cycle FC, which is computational friendly to cope with flexible input resolutions. (2) We take the first attempt to build family of hierarchical MLP-like architectures (CycleMLP) based on Cycle FC operator for dense prediction tasks. (3) Extensive experiments on various tasks (e.g.. ImageNet classification, COCO object instance detection, and segmentation, and ADE20K semantic segmentation) demon- strate that CycleMLP outperforms existing MLP-like models and comparable to and sometimes better than CNNs and Transformers dense predictions.
2

---

Figure 2: ImageNet accuracy model capacity. trained ImageNet- (Deng et al., 2009) without extra data. CycleMLP surpasses existing MLP-like models such as MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron al., 2021a), Published as a conference paper at ICLR 2022
models are
02
ImageNet Top-1 Acc
78-
Dell
Mixer
ResMLP
gMLP (Liu et al., 2021a), S²-MLP (Yu al., 2021) and ViP (Hou al., 2021).
74-
CycleMLP (ours)
capacity (MAdds:
Related Work. Convolution Neural Networks (CNNs) has dominated the visual backbones for several years (Krizhevsky al., 2012; Simonyan & Zisserman, 2014; He et al., 2016). (Dosovitskiy et al., 2020) introduced the first pure Transformer-based (Vaswani et al., 2017) model into computer vision and achieved promising performance, especially pre-trained on the large scale JFT dataset. Recently, some works (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) removed the attention in Transformer and proposed pure MLP-based models. Please see Appendix A for comprehensive review of the literature on the visual backbones.
2
METHOD
In this section, we introduce CycleMLP models for vision tasks including recognition and dense predictions. To begin with, Sec. 2.1 we formulate our proposed novel operator, Cycle FC, which serves basic component for building CycleMLP models. Then compare Cycle FC with Channel FC and multi-head attention adopted in recent Transformer-based models (Dosovitskiy al., 2020; Touvron al., 2020; Liu et al., 2021b) in Sec. 2.2. Finally, we present the detailed configurations of CycleMLP models in Sec. 2.3.
2.1 CYCLE FULLY-CONNECTIO LAYER
Notation. We denote an input feature map as X E HR where H, W denote the height and width of the image and Cn is the number of feature channels. We use subseripts to index the feature map. For example, X.j.e is the value of :h channel at the spatial position (i,j) and X are values of all channels at the spatial (i.
The motivation behind Cycle FC is to enlarge receptive field of MLP-like models to cope with downstream dense prediction tasks while maintaining the computational efficiency. As illustrated in Figure la, Channel FC applies weighting matrix on X along the channel dimension on fixed position ( However, Cycle FC introduces a receptive field of Sw, where Sy and Sw are stepsize along with the height and width dimension respectively (illustrated in Figure I (d)). The basic Cycle FC operator can be formulated as below:
C₁.
Wmlp + CycleFC( j.j. = Σ X (4
where Wmip E CC and b E C are parameters of Cycle FC. 6,(c) and dz(c) are the spatial offset of the two axis on the th channel, which are defined as below:
(c) (c mod Sn) - 1,
5,(c) = (1 SH
mod Sw) -1
Examples. We provide several examples (Figure (d)-(f) to illustrate the stepsize. For the sake of visualization convenience, we set the tensor's W = Thus, these three examples naturally all have Sw 1. Figure (d) illustrates the offsets along two axis when Sg 3, that is 6y(c) 0 and (c(c) = 11 0, 1, 11, 0, 1.... when e = 0, 1, ... ,8. Figure (c) shows that when S, = H,
Cycle FC has global receptive field. Figure 1 (f) shows that when S 1, there will be offset
3

---

Published as a conference paper at ICLR 2022
along either axis and thus Cycle FC degrades Channel FC (Figure 1 (a)). We also provide more general case where W * and Sh = 3, Sw = 3 in Figure 7 (Appendix).
The offsets (c(c) and (c) enlarge the receptive field of Cycle FC compared to Channel FC (Figure la), which applies weights solely on the same spatial position for all channels. The larger receptive field in return brings improvements
dense prediction tasks like semantic segmentation and object detection as shown in Table 1. Meanwhile, Cycle FC still maintains computational effi ciency and flexibility on input resolution. Both the FLOPs and the number of parameters are linear to the spatial scale which are exactly the same as those of Channel FC. In contrast, although Spatial FC has global receptive field over the whole spatial space, its computational cost is quadratic to the image scale. Besides, it fails to handle inputs with different resolutions.
2.2 COMPARISON BETWEEN MULTI-HEAD SELF-ATTENTIO (MHSA) AND CYCLE FC
Inspired by Cordonnier et al. (2020), when re- ce-paammetred properly, a multi-head self-attention layer with N₁ heads can be formulated as below, which is similar to a convolution with kernel size v /N, x VNn. (Please refer Appendix C for detailed derivation)
MHSA(XL (.)
Σ
X (I)+ W mhssa, b
2E(..2....NN.)
where Withsu,h E RC is the parameter matrix for th head in MHSA. b E Rost is the bias vector. {h,(h), (5) (h)) = {(0. 0), (-,,), contains all possible positional shift in convolution with kernel size VN,, VNn. Further, we stack all W together and reshape into Wmhssa E IR K K x CCunt, Then a relationship between Wmmp and Wimlisa can be formulated as follow.
W
Wimhsa
(4)
6,
equation 4 shows that only the weights of Wmmhs on spatial shift (L) are taken into account in Wmmlp, This indicates that Cycle FC introduce an inductive bias that the weighting matrix in MHSA should be sparse. Thus Cycle FC inherits the large receptive field introduced in MHSA. The receptive field in Cycle FC is enlarged to (SH, . Sw ). which enables Cycle FC to tackle with downstream dense prediction tasks better. Meanwhile, with the sparsity inductive bias, Cycle FC maintains computational efficiency in MLP-based methods as compared to convolution and multi- head self-attention. The parameter size in Cycle FC is Cin, x Cout while W/ilssa R x XCoue
2.3 OVERALL ARCHITECTURE
Patch Embedding. Given the raw input image with the size of H x W x 3, our model first splits into patches by patch embedding module (Dosovitskiy al., 2020). Each patch is then treated as "token". Specifically, we follow (Fan et al., 2021; Wang et al., 2021a) to adopt an overlapping patch embedding module with the window size and stride These raw patches are further projected to a higher dimension (denoted as C) by a linear embedding layer. Therefore, the overall patch embedding module generates the features with the shape of 4
X C.
CLc Block. Then, we sequentially apply several Cycle FC Bloc blocks. Comparing with the previous MLP blocks (Tolstikhin et al., 2021; Touvron al., 2021a; Liu et al., 2021a) visualized in Figure 5 (Appendix), the key difference of Cycle FC block is that it utilizes our proposed Cycle Fully-Connected Layer (Cycle FC) for spatial projection and advances the models in context aggre- gation and information communication. Specifically, the Cycle FC block consists of three parallel Cycle FCs, which have stepsizes SH x Sw the factorization of convolution (Szegedy et al., 2016) and criss-cross attention (Huang et al., 2019). Then, there is a channel-MLP with two linear layers and a GELU (Hendrycks & Gimpel, 2016) nin-liny in between. A LayerNorm (LN) (Ba et al., 2016) layer is applied before both parallel Cycle FC layers and channel-MLL modules. A residual connection (He et al., 2016) is applied after each module.
7, 1, and X 1. This design is inspired by
4

---

Published as a conference paper at ICLR 2022
Model
Param
FLOPs
Top-1
Model
Family
Scale
Param
FLOPs
Top-
EAMLP-14
30M
78.9
ResNer18
CNN
224
12M
1.8G
69.8
EAMLP-19
79.4
EI[Net-B3
CNN
300
12M
1.8G
81.6
Mixer-B/16
59M
12.7G
76.4
GFNet-H-Ti
224
15M
2.0G
80.1
Mixcr-B/16
59M
12.7G
77.3
CycleMLP-BI
MLP
224
15M
2.1G
78.9
ResMLP-S12
15M
3.0G
76.6
ResNet50
CNN
224
26M
4.1G
78.5
ResMLP-S24
30M
6.0G
79.4
DeiT-S
Trans
224
22M
4.6G
79.8
ResMLP-B24
116M
23.0G
81.0
BoT-S1-50
Hybrid
224
21M
4.3G
79.1
gMLP-Ti
6M
1.4G
72.3
PVT-S
Trans
224
25M
3.8G
79.8
gMLP-
20M
4.5G
79.6
Swin-T
Trans
224
29M
4.5G
81.3
gMLP-B
73M
15.8G
81.6
GENet-HI-
FFT
224
32M
4.5G
81.5
4--MIP-widd
71M
14.0G
80.0
CycleMLP-B2
MLP
224
27M
3.9G
81.6
$ -M-Peddppp
51M
10.5G
80.7
ResNet101
CNN
224
45M
7.9G
79.8
ViP-Small/7
25M
6.9G
81.5
RegNerY-8G
CNN
224
39M
8.0G
81.7
ViP-Medium/7
55M
16.3G
82.7
BoT-S1-59
Hybrid
224
34M
7.3G
81.7
ViP-Large/7
88M
24.4G
83.2
PVT-M
Trans
224
44M
6.7G
81.2
AS-MLP-T
28M
4.4G
81.3
CycleMLP-B3
ML.P
224
38M
6.9G
82.4
AS-MLPPS
50M
8.5G
83.1
AS-MLP-B
88M
15.2G
83.3
G-N
FFT
224
54M
8.4G
82,9
CycleMLP-B
15M
2.1G
79.1
Swin-S
Trans
224
50M
8.7G
83.0
CycleMLP-B B2
27M
3.9G
81.6
PVT-L
Trans
224
61M
9.8G
81.7
CycleMLPPB3
38M
6.9G
82.6
CycleMLP
MLP
224
50M
8.5G
82.9
CycleMI.--B
52M
10.1G
83.0
ViT-B/16
Trans
3842
86M
55.4G
77.9
CycleMLP-BI B5
76M
12.3G
83.1
DeiT-B
Trans
224
86M
17.5G
81.8
CycleMLP-T
28M
4,4G
81.3
DeiT-B
Trans
3842
86M
55.4G
83.1
CycleMLP-S
8.5G
82.9
Swin-B
Trans
224
88M
15.4G
83.3
CycleMLP-B
88M
15.2G
83.4
CycleMLP-B
MLP
224
88M
15.2G
83.4
Table 2: ImageNet- IK classification for ML.P-like models.
Table 3: Comparison with SOTA models on
ImageNet-1K without extra data.
Stage. The blocks with the same architecture are stacked to form one Stage (He et al., 2016). The number of tokens (feature scale) is maintained within each stage. At each stage transition, the channel capacity of the processed tokens is expanded while the number of tokens is reduced. This strategy effectively reduces the spatial resolution complexity. Overall, each of our model variants has four stages, and the output feature at the last stage has shape of H W 32 32 x C4- These stage settings are widely utilized in both CNN (Simonyan & Zisserman, 2014; He et al., 2016) and Trans- former (Wang al., 202 bb; Liu et al., 2021b) models. Therefore, CycleMLP can conveniently serve as gerrroe visual backbone and generic replacement for existing backbones.
Model Variants. The design principle of the model's macro structure is mainly inspired by the phi- losophy of hierarchical Transformer (Wang et al., 2021b; Liu et al., 2021b) models, which reduce the number of tokens the transition layers the network goes deeper and increase the channel dimension. In this way, we can build a hierarchical architecture that is critical for dense predic- tion tasks (Lin et al., 2014; Zhou et al., 2017). Specifically, build two model zoos following two widely used Transformer architectures, PVT (Wang et al., 2021b) and Swin (Liu et al., 2021b). Models in PVT-style are named from CycleMLP-BI as CycleMLP-T, -S, and -B, which represent models in tiny, small, and base sizes. These models are built by adapting several architecture-relatt represent the stride of the transition, the token channel dimension, the number of blocks, and the expansion ratio respectively at Stage i. Detailed configurations of these models Table 11 (Ap- pendix).
CycleMLP-BS and in Swin-Style are named
aperrparameters, including S, C₁, E, and L; which
EXPERIMENTS
In this section, we first examine CycleMLP by conducting experiments on ImageNet-IK (Deng et al.. 2009) image classification. Then, present bunch of bascline models achieved by Cy- cleMLP in dense prediction tasks, i.e., COCO (Lin et al., 2014) object detection, instance segmen- tation, and ADE20K (Zhou al., 2017) semantic segmentation.
5

---

Published as a conference paper at ICLR 2022
1x7 7 7 1x
Params FLOPs
Top-1 Acc
ImgNet Top-1
ADE20K
Stepsize
80.5
mloU
24.5M
3.6G
80.4
3
81.6
42.4
81.3
5
81.6 (+0.0)
43.2 (+0.8)
80.6
7
81.6 (40.0)
43,9 (+1.5)
26.8M
3.9G
11
80.5
9
81.5 0.1)
43.2 (4-0.)
26.8M
3,9G
81.6
Table 5: Stepsize ablation: CycleMLP achieves the highest mloU on ADE20K when stepsize is 7. However, the stepsize has negligible influence on the ImageNet classification.
Table 4: Ablation on three parallel branches. We adopt CycleMLP-B3 variant for this abla- tion study. Double check marks (VV) denote two same branches.
3.1 IMAGENET-IE IK CLASSIFICATION
The experimental settings for ImageNet classification are mostly from DeiT (Touvron et al., 2020), Swin (Liu et al., 2021b). The detailed experimental settings for ImageNet classification can be found in Appendix E.
Comparison with MLP-like Models. We first compare CycleMLLP with existing MLP-like models and the results are summarized in Table and Figure 2. The accuracy-FLOPs tradcoff of CycleMLP consistently outperforms existing MLP-like models (Tolstikhin al., 2021; Touvron al., 202 Liu et al., 2021 Guo et al., 2021; ;; Yu et al., 2021; Hou et al., 2021) under wide range of FLOPs, which we attribute the effectiveness of our Cycle FC. Specifically, compared with one of the pioneering MLP work, i.e,, gMLP (Liu et al., 2021a), CycleMLP-BB achieves the same top-1 accu- racy (81.6%) as gMLP-B while reducing more than FLOPs (3.9G for CycleMLP-B2 and 15.8G for gMLP-B). Furthermore, compared with existing SOTA MLP-like model, i.e. ViP (Hou et al., 2021), I), our model CycleMLP-B utilizes less FLOPs (15.2G) than ViP-Large/7 (24.4G, the largest one of ViP family) while achiving higher top- accuracy.
is noted that all previous MLP-like models listed in Table 2 do not conduct experiments on dense prediction tasks due to the incapability of dealing with variable input scales, which is discussed in Sec. 1. However, CycleMLP solved this issue by adopting Cycle FC. The experimental results on dense prediction tasks are presented in Sec. 3.3 and Sec. 3.4.
Comparison with SOTA Models. Table further compares CycleMLP with previous state-of- the-art CNN, Transformer and Hybrid architectures. It is interesting to that CycleMLP models achieve comparable performance to Swin Transformer (Liu et al., 2021b), which is the state-of- the-art Transformer-based model. Specifically, CycleMLP-B achieves slightly better top-1 accu- racy (83,4%) than Swin-B (83.3%) with similar parameters and FLOPs. GFNet (Rao et al., 2021) utilizes the fast Fourier transform (FFT) (Cooley Tukey, 1965) learn spatial information and achieves similar performance CycleMLP on ImageNet- IK classification. However, the architec- ture of GFNet correlated with the input resolution, and extra operation (parameter interpolation) is required when input scale changes, which may hurt the performance of dense predictions. We will thoroughly compare CyclcMLP with GFNet in Sec. 3.4 on ADE20K.
3.2 ABLATION STUDY
In this subsection, we conduct extensive ablation studies analyze each component of our design. Unless otherwise stated, We adopt CycleML P-B2 instantiation in this subsection.
demonstrate the advantage of the Cycle FC, we compare CycleMLP-BB with two other baseline models equipped with channel FC and Spatial FC as spatial context aggregation operators, respectively. The differences of these operators are visualized in Fig- ure 1, and the comparison results are shown in Table 1. CycleMLP-B2 outperforms the counterparts built on both Spatial and Channel FC for ImageNet classification, COCO object detection, instance segmentation, and ADE20K semantic segmentation. The results validate that Cycle FC is capable of serving as general-purpose, purpose, plug-and-ple operator for spatial information communication and context aggregation.
Cycle Fully-Connected Layer.

---

Published as a conference paper at ICLR 2022
ImageNet Top-1 Acc (%)
82.5
& Top-1 Acc ImageNet
0.0
Figure Resolution adaptability, All models are trained on 2 and evaluated on various resolutions with- out fine-tuning. Left: Absolute top-1 accuracy; Right: Accuracy difference relative to that tested on 224x224. The superiority of CycleMLP's robustness becomes more significant when scale varies to greater extent.
80.0
77.5
-2.5
75.0
-5.0
72.5
-7.5
70.0
DellS
110.0
Dells Gffict-5
67.5
CydiMMF--22
-2255
Cyd
416
128
224
320
416
Testing Resolution
Testing Resolution
Table 4 further details the ablation study on the structure of CycleMLP block. It is observed that the top-1 accuracy drops significantly after removing one of the three parallel branches, especially when discarding the 1x7 or 7x branch. To eliminate the probability that the fewer parameters and FLOPs cause the performance drop, we further use two same branches (denoted as "VV" in Table 4) and one I 1x1 branch to align the parameters and FLOPs. The accuracy still drops relative to CycleMLP, which further demonstrates the necessity of these three unique branches.
Resolution adaptability. One remarkable advantage of CycleMLP is that it can take arbitrary- resolution images as input without any modification. On the contrary, GFNet (Rao et al., 2021) needs to interpolate the learnable parameters the fly when the input scale is different from the one for training. We compare the resolution adaptability by directly evaluating models at a broad spec- trum of resolutions using the weight pre-trained on 224 x 224, without fine-tuning. Figure 3 (left) shows that the absolute Top-1 accuracy on ImagNet and Figure 3 (right) shows the accuracy dif- ferences between one specific resolution and the resolution of 224x224. Compared with DeiT and GFNet, CycleMLP is more robust when resolution varies. In particular, at the 128 x 128, CycleMLP saves more than points drop compared to GFNet. Furthermore, at higher resolution, the perfor- mance drop of CycleMLP is less than GFNet. Note that the superiority of CycleML becomes more significant when the resolution changes to a greater extent.
3.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
Settings. We conduct object detection and instance segmentation experiments on COCO (Lin al., 2014) dataset. We first follow the experimental settings of PVT (Wang et al., 2021b), which are introduced Appendix. E.2. The corresponding results are presented in Table Then, in order to compare fairly with Swin Transformer, which adopts a different experimental recipe with PVT, we further follow the experimental settings of Swin with our CycleMLP-S model and the results are presented in Table 7.
RetinaNet 1x
Mask R-CNN 1x
Backbone
Param AP
AP50 AP7s
APs
aP
AP,
Param
AP
APE
AP
AP
AP
APm
75
ResNet18
21.3M
31.8 49.6
33.6
16.3
34.3
43.2
31.2M
34.0
54.0 36.7
31.2
51.0
32.7
PVT-Tiny
23.0M
36.7 56.9
38.9
22.6
38.8
50.0
32.9M
36.7
59.2
39.3
35.1
56.7
37.3
CycleMLP-BI
24.9M
38.1 58.7
40.1
21.9
41.9
50.4
34.8M
39.8
61.7
43.3
37.0
58.8
39.7
ResNet50
37.7M
36.3 55.3
38.6
19.3
40.0
48.8
44.2M
38.0
58.6
41.4
34.4
55.1
36.7
PVT-Small
34.2M
40.4 61.3
43.0
25.0
42.9
55.7
44.1M
40.4
62.9
43.8
37.8
60.1
40.3
CycleMLP-B2
36.5M
40.6 61.4
43.2
22.9
44.4
54.5
46.5M
42.1
64.0
45.7
38.9
61.2
41.8
ResNet101
56.7M 38.5 57.8
41.2
21.4
42.6
51.1
63.2M
40.4
61.1
44.2
36.4
57.7
38.8
RcsNeXt101-32xk4
56.4M 39.9 59.6
42.7
22.3
44.2
52.5
62.8M
41.9 62.5
45.9
37.5
59.4
40.2
PVT-Medium
53.9M
41.9 63.1
44.3
25.0
44.9
57.6
63.9M
42.0 64.4
45.6
39.0
61.6
42.1
CycleMLP-B3
48.1M
42.5 63.2
45.3
25.2
45.5
56.2
58.0M
43.4 65.0
47.7
39.5
62.0
42.4
PVT-Large
71.1M
42.6 63.7
45.4
25.8
46.0
58.4
81.0M
42.9
65.0
46.6
39.5
61.9
42.5
CycleML--B4
61.5M
43.2 63.9
46.2
26.6
46.5
$7.4
71.5M
44.1
65.7
48.1
40.2
62.7
43.5
ResNext101-64x4d
95.5M
41.0 60.9
44.0
23.9
45.2
54.0
101.9
19.
42.8
63.8
47.3
38.4
60.6
41.3
CycleMLP-B5
85.9M
42.7 63.3
45.3
24.1
46.3
57.4
95.3M
44.1
65.5
48.4
40.1
62.8
43.0
Table 6: Object detection and instance segmentation COCO va12017 (Lin et al., 2014).
We compare CycleMLP with various backbones including ResNet (He et al., 2016), ResNeXt (Xie al., 2017) and PVT (Wang al., 2021b).

---

Published as a conference paper at ICLR 2022
Backbone
AP"
AP
AP/s
APM
AP50
AP75
Params
FLOPs
ResNet50 (He et al., 2016)
41.0
61.7
44.9
37.1
58.4
40.1
44M
260G
PVT-Small (Wang et al., 2021b)
43.0
65.3
46.9
39.9
62.5
42.8
44M
245G
Swin-T (Liu et al., 2021b)
46.0
68.2
50.2
41.6
65.1
44.8
48M
264G
CycleMLP-T (ours)
46.4
68.1
51.1
41.8
64.9
45.1
48M
260G
Table 7:
The instance segmentation results of different backbones on the COCO val2017 dataset. Mask R-CNN frameworks are employed.
Results. Firstly, shown in Table 6, CycleMLLP-asse RetinaNet consistently surpasses the CNN- based ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Transformer-based PVT (Wang al., 2021b) under similar parameter constraints, indicating that CycleMLP serve cellent general-purpose backbone. Furthermore, using Mask R-CNN (He et al., 2017) for instance segmentation also demonstrates similar comparison results. Furthermore, from Table 7, the Cy- cleMLP can achieve a slightly better performance than Swin Transformer.
3.4 SEMANTIC SEGMENTATION
Settings. We conduct semantic segmentation experiments on ADE20K (Zhou et al., 2017) dataset and present the detailed settings in Appendix. E.3, Table and Table 9 show the experimental results using training recipes from PVT and Swin respectively.
Semantic FPN
Backbone
Param
mloU (%)
ResNet18 (He al., 2016)
15.5M
32.9
PVT-Timy (Wang et al., 2021h)
17.0M
35.7
CycleMLP-BI (ours)
18.9M
40.8
0.8
ResNer50 (He et al., 2016)
28.5M
36.7
PVT-Small (Wang al., 2021h)
28.2M
39.8
Swin-T' (Liu et al., 202
31.9M
41.5
0.6
GFNet-Tiny (Rao et al., 2021)
26.6M
41.0
CycleMI.PBB2 (ours)
30.6M
43.4
(a) Swin
ResNerTOT (He al., 2016)
47.5M
38.8
-0.4
ResNeXt101-32x4d (Xie et al., 2017)
47.1M
39.7
PVT-Medium (Wang al., 2021b)
48.0M
41.6
GFNet-Small (Rao et al., 2021)
47.5M
42.5
CycleMLP-B3 (ours)
42.1M
44.3
0.2
PVT-Large (Wang ct al., 2021b)
65.1M
42.1
Swin-St (Liu et al., 2021b)
53.2M
45,2
CycleMLP-BB (ours)
55.6M
45.1
0.0
GFNet-Base (Rao et al., 2021)
74.7M
44,8
Res.Next101-6xx4d (Xic al., 2017)
86.4M
40.2
(b) CycleMLP
CycleMLP-B5 (ours)
79.4M
45.5
Figure 4: Effective Receptive Field
Table 8: Semantic segmentation ADE20K (Zhou et al., 2017) val. All models are equipped with Se- mantic FPN (Kirillov et al., 2019). Results are from GFNet (Rao et al., 2021).
(ERF). We visualize the ERFs of the last stage for both Swin (Liu al., 202 and CycleMLP. Best viewed with zoom in.
Results. As shown in Table 8, CycleMLP outperforms ResNet (He al., 2016) and PVT (Wang al., 2021b) significantly with similar parameters. Moreover, compared the state-of-the-ar Transformer-ban based backbone, Swin Transformer (Liu et al., 2021b), CycleMLP can obtain com- parable or even better performance. Specifically, CycleMLP-B2 surpasses Swin-T by 0.9 mloU with slightly less parameters (30.6M 31.9M).
Although GFNet (Rao et al., 2021) achieves similar performance as CycleMLP on ImageNet clas- sification, CycleMLP notably outperforms GFNet ADE20K semantie segmentation where input scale varies. We attribute the superiority of CycleMLP under a scale-variable scenario to the capa- bility of dealing with arbitrary scales. On the contrary, GFNet (Rao al., 2021) requires additional
8

---

Published as a conference paper at ICLR 2022
val MS mloU
Method
Backbone
Params
FLOPs
Swin-T (Liu al., 2021b)
45.8
60M
945G
UperNet (Xiao et al., 2018)
AS-MLP-T (Lian et al., 2021)
46.5
60M
937G
CycleMLP-T (ours)
47.1
60M
937G
Swin-S (Liu al., 2021b)
49.5
81M
1038G
UperNet (Xiao et al., 2018)
AS-MLP-S (Lian al., 2021)
49.2
81M
1024G
CycleMLP-S (ours)
49.6
81M
1024G
Swin-B (Liu et al., 202
49.7
121M
1188G
UperNet (Xiao al., 2018)
AS-MLP-B(Lian al., 2021) 49.5
121M
166G
CycleMLP-B (ours)
49.7
121M
1166G
Table 9: The semantic segmentation results of different backbones on the ADE20K validation set.
heuristic operation (weight interpolation) when the input scale varies, which may hurt the perfor- mance.
Moreover, we also visualized the receptive field following (Xie et al., 2021), and the results are visualized in Figure 4, which demonstrate that our CycleMLP has a larger effective receptive field than Swin.
3.5
ROBUSTNESS
Noise
Blur
Weather
Digital
Network
mCEt
Gauss
Shot
Impulse
Defocus
Glass
Motion Zoom Snow Frost
Fog
Bright
Contrst
Flastic
Pixel JPEG
Net-50
76.7
79.8
81.6
82.6
74.7
88.6
78.0
79.9 77.8 74.8 66.1 56.6
71.4
84.7 76.9
DeiT-S
54.6
46.3 47.7
46.4
61.6
71.9
57.9
71.9 49.9 46.2 46.0
44.9
42.3
66.6 59.1 60.4
Swin-S
62.0
52.2
53.7
53.6
67.9
78.6
64.1
75.3
55.8 52.8 51.3
48.1
45.1
75.7
76.3 79.1
MLP-Mixer
78.8
80.9
82.6
84.2
86.9
79.1
78.3 67.4 64.6
59.5
57.1
90.5
72.7 92.2
ResMLP-12
66.0
57.6
58.2
57.8
72.6
83.2
67.9
76.5
61.4 57.8 63.8
53.9
52.1
78.3
72.9 75.3
gMIP-S
64.0
52.1
53.2
52.5
73.1
77.6
64.6
79.9
77.7 78.8
54.3 55.3
43.6
70.6
58.6 67.5
CycleMLP-S
53.7
42.1
43.4
43.2
61.5
76.7
56.0 66.4 51.5 47.2 50.8 41.2
39.5
72.3
57.5 56.1
Table 10: Robustness on ImageNet-C (Hendryeks & Dietterich, 2019). The mean corruption error (mCE) normalized by AlexNet (Krizhevsky et al., 2012) errors is used as the robustness metric. The lower, the better.
We further conduct experiments on ImageNet-C (Hendrycks Gimpel, 2016) analyze the bustness ability of the CycleMLP, following (Mao et al., 2021) and results are presented Table 10. Compared with both Transformers (e.g. DeiT and Swin) and existing MLP models (e.g. MLP-Mixer, ResMLP, gMLP), CycleMLP achieves stronger robustness ability.
4 CONCLUSION
We present versatile MI.P-ike architecture, CycleMLP, in this work. CycleMLP is built upon the Cycle Fully-Connected Layer (Cyele FC), which is capable of dealing with variable input scales and can serve generic, plug-and-play replacement of vanilla FC layers. Experimental results demonstrate that CycleMLP outperforms existing MLP-like models on ImageNet classification and achieves promising performance dense prediction tasks, iee. object detection, instance segmen- tation and semantic segmentation. This work indicates that an attention free architecture can also serve general vision backbone.
Acknowledgment. Ping Luo is supported by the General Research Fund HK No.27208720, No.1 nt.1122220.. and the HKU-TCL Joint Research Center for Artificial Intelligence.

---

Published as a conference paper at ICLR 2022
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint trXiv:2103.1569/ 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? arXiv preprint arXiv:2 102.00220, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 0ttd::/900000//11 2019.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benehmark. https: / com/ open- mm1 ab/ www.mb////aa...aa...a....aaaaaaa seanseemmttttionn 2020.
James W Cooley and John W Tukey, An algorithm for the machine calculation complex fourier series. Mathematics of computation, 29990))277-31,, 1965.
Cean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In International Conference on Learning Representations, 2020. URL httph:ttp//oom....t.. http::///peereveeeeeeeeee net/ (irrmmmmi==HInnCC rPBB.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2022
Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. Mdmmt: Mul- tidomain multimodal transformer for video retrieval. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3354-3363 2021.
Haoqi Fan, Bo Xiong. Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104.11227 2021.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp- 3146-3154, 2019a.
Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive context network for scene parsing. In Proceedings of the IEEF/CV International Conference on Computer Vision, pp.
2019b.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, 214-229. Springer, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint arXiv:2105.0m 02358, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019.
Dan Hendryeks and Kevin Gimpel. arXiv:1606.0844 2016.
Gaussian error
linear units (gelus). arXiv preprint
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vi- sion permutator: A permutable mlp-like architecture for visual recognition. arXiv preprint arXiv:2106. 12366, 2021.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 017.77444444.... 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 612, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
11

---

Published as a conference paper at ICLR 2022
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 15:1097-1 2012.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, (4)):411555) 1989.
Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang- Bossnas: Exploring hybrid con-transformers with block-wisely self-supervised neural architecture search. arXiv preprint arXiv:2103. /2424, 2021.
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mip: An axial shifted mip architecture for vision. arXiv preprint arXiv:2107.0839) 2021.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint 1xXiv:2105.08000 2021a.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2 /4030, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:171..05011, 2017.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye, Yuan He, and Hui Xue. Rethinking the design principles of robust vision transformer. arXiv preprint arXiv:2105.07926 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perfopperfommncc deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bus -Bce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprini 2xXivv2103.000 103.00020. 2021.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. arXiv preprint arXiv:2107.00645. 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16519-16529, 2021.
12

---

Published as a conference paper at ICLR 2022
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang. et al. Sparse r-cmn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14454-14463, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2818-2826, 2016.
lya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mip architecture for vision. arXiv preprint arXiv:2105.0160) 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar 201:220/2.128777 2020.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03444 034044 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:/804.0746). 2018
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint arXiv:2106.133 13797, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 12122, 2021b.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 8741-8750, 2021c
Ross Wightman. Pytorch image models.
com/ www.c//wowwwwiiee.
pytorch- immii--aaoeoooeeee 1 s, 2019.
Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2 158888888 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pp. 4344 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. arXiv preprint arXiv:2105. /05.:03,5,/203, 2021.
13

---

Published as a conference paper at ICLR 2022
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. 2019.
Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disen- tangled non-local neural networks. In European Conference on Computer Vision, pp. 191-207. Springer, 2020.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S²-mlp: Spatial-shift mip architecture for vision. arXiv preprint arXiv:2106.07471 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 173-190. Springer, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEF/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXh:: 2017.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2021.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through 2de20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633-641, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010004159, 2020.
14

---

Published as a conference paper at ICLR 2022
A LITERATURE ON VISION MODEL
CNN-based Models. Originally introduced over twenty years ago (LeCun et al., 1989), convolutional neural networks (CNN) have been widely adopted since the success of the AlexNet (Krizhevsky et al., 2012) which outperformed prevailing approaches based on hand-crafted image features. There have been several attempts made to improve the design of CNN-based mod- els. VGG (Simonyan & Zisserman, 2014) demonstrated state-of-the-art performance on ImageNet via deploying small (3 x 3) convolution kernels to all layers. He et al.introduced skip-connections in ResNets (He et al., 2016), enabling model variant with more than 1000 layers. DenseNet (Huang ct al., 2017) connected each layer to every other layer in a feed-forward fashion, strengthening fea- ture propagation and reducing the number of parameters. In parallel with these architecture design works, some other works also made significant contributions to the popularity of CNNs, including normalization (Ioffe& & Szegedy, 2015; Ba et al., 2016), data augmentation (Cubuk et al., 2020; Yun et al., 2019; Zhang et al., 2017), etc.
Transformer- based Models. Transformers were first proposed by Vaswani el al. machine trans- lation and have since become the dominant choice in many NLP tasks (Devlin et al., 2018; Wang al., 2018; Yang et al., 2019; Brown et al., 2020). Recently, transformer have also led a series of breakthroughs in computer vision community since the invention of ViT (Dosovitskiy et al., 2020), and have been working as a de facto standard for various tasks, e.g., image classification (Dosovit- skiy et al., 2020; Touvron et al., 2020; Yuan et al., 2021), detection and segmentation (Wang et al., 202 lb; Liu et al., 202 Zheng et al., 2021; Xie et al., 2021), video recognition (Wang et al., 2021c; Bertasius et al., 2021; Arnab et al., 2021; Fan et al., 2021) and so on. Moreover, there has also been lots of interest in adopting transformer to cross aggregate multiple modality information (Radford et al., 2021: Gabeur et al., 2020; Dzabraev et al., 2021). 1). Furthermore, combining CNNs and trans- formers is also explored in (Srinivas et al., 2021; Li et al., 2021; Wu et al., 2021; Touvron et al., 2021b).
MLP-based Models. MLP-based models (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) differ from the above discussed CNN- and Transformer- based models because they resort to neither convolution nor self-attention layers. Instead, they use MLP layers over feature patches on spatial dimensions to aggregate the spatial context. These MLP-bbasdd models share similar macro structures but differ from each other in the detailed design of the micro block. In addition, MLP-based models provide more efficient computation than transformer-basee models since they do not need to calculate affinity matrix using key-query multiplication. Concurrent to our work, S2-MLP (Yu et al., 2021) utilizes a spatial-shift operation for spatial information communication. The similar aspect between our work and S2-MLP lies in that we all conduct MLP operations along the channel dimension. However, our Cycle FC is different from S in: (I) S2MMLP achieves communications between patches by splitting feature maps along channel dimension into several groups and shifting different groups in different directions. introduces extra splitting and shifting operations the feature map. On the contrary, Layer-for spatial context aggregation. It does not modify the feature map and is formulated as a generic, plug-and-play MIP unit that can be used as direct replacement of vanilla without any adjustments. (2) We design a pyramid structure for and conduct extensive experiments on classi- fication, object detection, instance segmentation, and semantic segmentation. However, the output feature map of S2-MLP has only one single scale in low resolution, which is unsuitable for dense prediction tasks. Only ImageNet classification is evaluated on M2LL We compared Cycle FC with S2MMLP in details in the Section 3.
propose novel operator- Cycle Fully-Connected
B COMPARISON OF MLP BLOCKS
We summary MLP blocks proposed by recent MLP-related works in Figure 5. We notice that ex- isting MLP blocks, i.e., MixeMixer, ResMLP and gMLP share similar method of Spatial Proj: Transpose - Fully-Connected over spatial dimension -> Transpose back. These models can not cope with variable image scales the FC layers in Spatial Proj are configured by the len.
The blocks used for building CycleMLP consist of our proposed novel Cycle FC, whose configura- tion has nothing to do with image scales and can naturally deal with dynamic image scales.
15

---

Published as a conference paper at ICLR 2022
Fc(4"din, din) GeLU
FC(4rdin, dim)
Affine
Channel Pro
Channel Proj
GeLU
LayerNorm
FC(din, d+dim)
Affine (Bias)
FC(din, d+dim)
Transpose
FC(e.stseeglen len, seg]len)
Affine
Transpose
Spatial Pro
Spatial Proj
FC(seq Len, seq_len)
GeLU
LayerNorm
Affine (Bias)
Transpose(1,
FC(seq Jon, 0.5m)
Transpose 2)
Input Embeddings
Input Embeddings
(a) MLP-Mixer (Tolstikhin al., 2021)
(b) ResMLP (Touvron al., 2021a)
Channel Proj
FC(6*dim, din)
FC(r*din, din)
Transpose (1, 2)
Channel Proj
Spatial Proj
FC(seq_ len, seq seq_len)
LayerNorm
FC(dim, r*dim)
LayerNorm
Transpose 2)
Linear
Split
GeLU
Sun
Channel Proj
Spatial Proj
FC(dim, 6'din)
LayerNorm
LayerNorm
Input Embeddings
Input Embeddings
(c) gMLP (Liu et al., 2021a)
(d) CycleMLP (Ours)
Figure 5: Comparison of MI.P blocks in details.
C FROM MULTI-HEAD SELF-ATTENTION TO CONVOLUTION
In this section, we provide details in how MHSA can be transferred into convolution-like operator in equation 3, To start with, the MHSA layer be formulated below:
MHSA(X)
concat [SAA + he(l....N,)
(5)
where Wout (NN,Coo)) and R... are parameters for the final linear projection. SA, is the ttth self-attention module. Then reshape X into X HW and let T H x W. which indicates that there are T tokens X. SA, can be defined as follow:
SA(X), k. softmax (A,.m + A = (Q + P) PPT
(6)
where va Q XW K WWky are respectively the value, query and key matrix
with learnable matrices W E RCC. Wa E C.nXCe W BCnx P E TxC is the
16

---

Published as a conference paper at ICLR 2022
positional embedding matrix containing positional information for every input token, which can be replaced by the output of any function fp that encodes the position of tokens. And A E RT* is the attention matrix where each element A,, is the attention score between the 7th and jth token in X. With absolute positional encoding, the second line in equation 6 can be expanded as (Cordonnier et al., 2020):
g. = (X,, + Pap (X Pa.))
=x,, W WhyyT X,, W (Px, WhytT + Ph (X,, + P... W (Pr, WheyT
When we apply relative positional encoding scheme in (Dai et al., 2019), is re-parametried into:
A Xg.: (X Wker) T + X..: W Wkeyy + ((x, W +0(66 WkerjT (8)
where a is positional encoding for relative distance 6q.4 = (61, 02) between token y and in X. Whey is introduced to only pertain the positional encoding 4..A: and learnable parameter vectors that replace the original p.:: W4y term, which implies that the attention bias remains the same regardless of the absolution positions of the query. If we set Wory = Whey = 0 and Wkey = I, the first three terms in equation 8 vanish and A, = or q,k- We set {A.(h), A, {(0, 0), (1,0), (-1,0). ..... contains all possible positional shift in convolution with kernel size VN,, x /Nu. For each head h, let rg = (116, 11,,,11,,,,, 5,,, and e -a"(1, 22) -2,, (h)), each softmax attention matrix becomes:
[1 if (A,)
softmax (A (g.).
(9)
otherwise
Substitute softmax(A (A) into equation and get
MHSA (A)A(X)La i..
Σ
X (H))
Wimlsa,h
(10)
(E(L......NN)
D ARCHITECTURE VARIANTS
In order to conduct fair and convenient comparison, we build two model zoos: the one is in PVT- Style (named CycleMLP-BI to -B5) and the other in Swin-Style (named as CycleMLP-T, -S and B). These models are scaled up by adapting several architecture-related hyper-parameters, including St. C. E, and L, which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i. Detailed configurations of these models in Table 11.
E EXPERIMENTAL SETUPS
E. IMAGENET CLASSIHICATION
Settings, We train our models on the ImageNet-IK dataset (Deng et al., 2009), which contains 1.2M training images and 50K validation images evenly spreading 1,000 categories. We follow the standard practice in the community by reporting the top-1 accuracy on the validation set. Our code implemented based on PyTerch (Paszke al., 2019) framework and heavily relies on the t timm (Wightman, 2019) repository. For apple-to-apple comparison, our training strategy is mostly adopted from DeiT (Touvron et al., 2020), which includes RandAugment (Cubuk et al., 2020), Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019) random erasing (Zhong et al., 2020) and stochastic depth (Huang et al., 2016). The optimizer Adam (Loshchilov & Hutter, 2017) with the momentum of 0.9 and weight decay of 5 10 by default. The cosine learning rate schedule is adopted with the initial value of x 10 3 All models are trained for 300 epochs on Tesla V100 GPUs with total batch size of 1024.
17

---

Published as a conference paper at ICLR 2022
PVT-Style (Wang 202/bb
Swin-Style (Liu et al,, 2021b)
Output Size
Layer Name
B1
B2
B44
135
Tiny
Small
Base
Overlapping
S1
S1
Stage
X
Patch Embeedding
C₁
C1
C1
CycleMLP
F
4
B1 = 4
Bj =8
B. =
E1 4
E1 = 4
E1 =
E1 =
Block
L
==
L1 =
L1 =
L1 =
=
L =
Ln =2
L1 =
Overlapping
Sq =
Sq =2=2
Stage
H
X
W
Patch Embedding
Cs 128
C2 192
Ca 192
C2 E1
256
CycleMLP
E2
Eg 8
8
B1
Block
L ===
===3 3
Lg
Lg 8
L2 =
41 2
L 2
4
Overlapping
Sq 2
Stage
Patch Emhedding
C3 320
Cg
C3
C3
16
CycleMLP
Eg = 4
Es =
Eg 4
Eg 4
Ea =
B1 = 4
E1 = 4
E1 4
Block
Lg
La 10
Lg 18
La 27
La 24
L1 =
41 18
L1 18
Overlapping
S4 =2
9, =
Stuge
32
H
32
W
Patch Embeeding
C4 512
C4 768
C4 768
C4 1024
CyclMML
E,
E1
Ej
Block
I. mm 2
Ia 3
L4 m
La 3
L4 3
1.1 2
I. 2
J,1
Parameters (M)
15.2
26.8
38.4
51.8
75.7
28.3
49.6
87.7
FL.OPs (G)
2.1
3.9
6.9
10.1
12.3
4.4
8.6
15.2
Table 11: Instantiations of the CycleMLP with varying complexity. The E, and L; denote the expand ratio and number of repeated layers. Our design principle is inspired by the philosophy of ResNet (He al., 2016), where the channel dimension increases while the spatial resolution shrinks with the layer going deeper.
Further kernel optimization for Cycle FC may bring faster speed but is beyond the scope of this work.
E.2 COCO INSTANCE SEGMENTATION
We conduct object detection and instance segmentation experiments COCO (Lin al., 2014) dataset, which contains 118K and 5K images for train and validation splits. We adopt the mindttectin in (Chen et al., 2019) toolbox for all experiments in this subsection. To evalu- ate the our CycleMLP backbones, we adopt two widely used detectors, i.e., RetinaNet (Lin et al., 2017) and Mask R-CNN (He et al., 2017). All backbones are initialized with ImageNet pre-trained weights and other newly added layers are initialized via Xavier (Glorot & Bengio, 2010). We use the AdamW (Loshchilov & Hutter, 2017) optimizer with the initial learning rate of 1 x x1000 All models are trained on 8 Tesla V100 GPUs with total batch size of 16 for 12 epochs (i.e., 1x train- ing scheduler). The input images are resized the shorted side of 800 pixels and the longer side does not exceed 1333 pixels during training. We do not use the multi-scale (Carion et al., 2020; Zhu al., 2020; Sun al., 2021) training strategy. In the testing stage, the shorter side of input images is resized 800 pixels while constraint on the longer side.
E.3 ADE20K SEMANTIC SEGMENTATION
We conduct semantic segmentation experiments on ADE20K (Zhou et al., 2017) dataset, which cov- ers a broad range of 150 semantic categories. ADE20K contains 20K training, 2K validation and 3K testing images. We adopt the mmse ion (Contributors, 2020) toolbox as our codebase in this subsection. The experimental settings mostly follow PVT (Wang et al., 2021b), which trains models for 40K iterations on 8 Tesla V100 GPUs with 4 samples per GPU. The backbone is initial- ized with the pre-trained weights on ImageNet. All models are optimized by AdamW (Loshchilov & Hutter, 2017). The initial learning rate is configured as 2 x 10 with the polynomial decay parame- ter of 0.9. Input images are randomly resized and cropped to 512x512 at the training phase. During testing, we scale the images to the shorted side of 512. We adopt the simple approach Semantic FPN (Kirillov et al., 2019) as the semantic segmentation method following (Wang et al., 2021b) for fair comparison.
18

---

Published as a conference paper at ICLR 2022
val MS mloU
Method
Backbone
Params
FLOPs
DANet (Fu et al., 2019a)
ResNet-101
45.2
69M
1119G
DeepLabv3+ (Chen et al., 2018)
ResNet-100
44.1
63M
1021G
ACNet (Fu et al., 2019b)
ResNet-100
45.9
DNL (Yin et al., 2020)
ResNet- 101
46.0
69M
1249G
OCRNet (Yuan et al., 2020)
ResNet-101
45.3
56M
923G
UperNet (Xiao al,, 2018)
ResNet-100
44.9
86M
1029G
OCRNet (Yuan et al., 2020)
HRNet-w48
45.7
71M
664G
Deepl.abv3+ (Chen et al., 2018)
ResNeSt-101
46.9
66M
1051G
DeepLabb33+ (Chen et al., 2018)
ResNeSt-200
48.4
88M
1381G
Swin-T (Liu et al., 2021b)
45.8
60M
945G
UperNet (Xiao al., 2018)
AS-MLP-T (Lian al., 2021)
46.5
60M
937G
CycleMLPPT (ours)
47.1
60M
937G
Swin-S (Liu et al.. 2021b)
49.5
81M
1038G
UperNet (Xiao al., 2018)
AS-MLP-S (Lian al., 2021)
49.2
81M
1024G
CycleML.PSS (ours)
49.6
81M
1024G
Swin-B (Liu al., 202 1b)
49.7
121M
1188G
UperNet (Xiao et al., 2018)
AS-MLP-B(Lin) et al., 2021)
49.5
121M
1166G
CycleMLP-B (ours)
49.7
121M
1166G
Table 12: The semantic segmentation results of different backbones on the ADE20K validation set.
F SAMPLING STRATEGIES
We explore more sampling strategies in this subsection, including random sampling and dilated sam- pling inspired by dilated convolution (Yu & Koltun, 2016; Chen et al., 2018) (as shown in Figure 6). We also compare the dense sampling method with ours.
Random sampling. As shown in Table 13, conduct experiments with random sampling for three independent trials and observe that the averaged Top-1 accuracy on ImageNet-I 1K drops by 1.3%. We hypothesize that the decreased performance is caused by the fact that random sampling will totally disturb the semantic information of objects, which essential to image recognition. Compared with the random sampling strategy, cyclical sampling is able to aggregate the adjacent pixels, which benefits in capturing the semantic information.
Dilated Stepsize (Figure 6). As shown in Table 13, observe the result of dilated sampling is better than the random one (+1.0% acc) but lower than ours (-0.5% acc). In fact, compared with the random sampling, dilated solutions take their advantages local information aggregation. However, compared with the cyclical sampling strategy, dilated solutions lose the fine fine-grained information for recognition. It may hurt the accuracy performance some extent.
Dense sampling. we conduct ablation studies by using dense sampling strategies (i.c., vanilla con- volution with kernel size 1 xx3 and 3x Since dense sampling strategies incredibly increase the models' parameters and FLOPS, we do not have enough time to thoroughly optimize the model for 300 epochs. Therefore, for fair comparisons, we conducted extra ablation studies on training mod- els for 100 epochs with the strictly same learning configurations. The results shown in Table 14 demonstrate that the sparse sampling strategy (ours) outperforms the dense one. The comparison in- dicates that the dense sampling strategies introduce redundant parameters, which makes the model hard to optimize. Our sparse sampling strategy with fewer parameters is proven to be efficient and optimization-friendle
19

---

Published as a conference paper at ICLR 2022
Sampling
Params
FLOPs
Top-1 Acc
dilation-2
26.8M
3.9G
81.1
Random,
80.4
Random, S-2
26.8M
3.9G
80.2
Random, S=3
80.4
CycleMLP
26.8M
3.9G
81.6
Table 13: Comparison with dilated and random sampling. random sampling, we conduct experiments for three independent trials with three seeds (S-1, 2, 3).
Operators
HLOPs
Top-1
Cony: 1x3 3x1
34.3M
5.1G
75.0
CycleMLP: 1x3 3x
x
26.8M
3.9G
76.1
Table 14: Comparison with dense sampling: On the consideration of training time, only train both models 100 epochs fair comparison.
ImgNet Top-1
ADE20K
branch1
branch2
mloU
7x1
1x 7
81.6
43,9
Figure 6: An example of dilated CycleMLP where di ion=2 and stepsize=3.
7x2
2x7
81.5
43.4
7x3
3x7
81.4
42.7
4x4
4x4
81.5
43.1
Table 15:
Comparison on different stepsizes (e.g.. even stepsize and odd stepsize), including 7x2,
4x4.
G VISUALIZATION EXAMPLES
For easier understanding of proposed CycleMLP, visualize several instances of CycleMLP in Figure 7, including general case with stepsize 3x3 (7a), even stepsize (7b), and examples where stepsize along height or width equals to 1 (7c, 7d).
We note that given specific number of input and output channels, no matter how the stepsize changes, the number of parameters of the CycleMLP does not change. Therefore, there is trade-ofh off of representation abilities between spatial and channel dimensions, which will be discussed in details following experimental analysis.
Experiments: We further conduct experiments CycleMLPs with stepsize of 2x7, 7x2, 7 7x3, 3x7, and 4x4, respectively. The results are summarized Table 15. For fair comparisons, all the models in the above table have the same parameters and FLOPs. We observe that the model with stepsize of x and 7x achieves the best performance, especially for semantic segmentation on ADE20K. To analyze the impact of stepsize
the performance, we take Figure 7 for better illus- tration. One can see that enlarging the stepsize can expand the spatial receptive field. However, at cost, it will reduce the number of periods (groups) running along the channel dimension, which may hurt the channel-wise representation abilities. Taking a feature map with = 18 for example, the CycleMLP with stepsize 3x3 (Figure 7(a)) runs through only 2 channel groups (curly brackets in the figure). However, the CycleMLP with stepsize 3x (Figure 7(c)) will run through 6 groups in total, making better of the representation the channel dimension. That's say, there is tra between spatial and channel representation. We empirically found that CyCleMLP with stepsize of 7 and achieves the best performance.
20

---

Published as a conference paper at ICLR 2022
E2
D1
[0,
[2.
[00.
[1.
[3, 8, 53
[0, 1, 12
[1. 1, 131
[2, 1, 14
(0, 2,
11. 2, 14
(2
25
Hb
(b) Se x Sw: 3 x
(a) Sy Sw: 3xx 3
[N.
Di.
(0.
[00
Do.
(b) 33.
8 323
P.
[6, 333
18 2, 340
W
96
H: W.
(c) Su Sw:
(d) Su Sw:
x
Figure 7: Examples of Stepsize cases: Here we separate the feature map along the width dimension for convenient visualization. * denotes the output position. We place the absolute coordinates (h, w, c) the sampled points the left of the feature. Sampled points within curly bracket (() belong to the same period (group). Dash lines link two cyclical periods.
21


Reviews:
[{"review_id": "wy-kmTDcSqk", "reviewer": "Reviewer_4cNH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n(1)  This paper designs a new MLP-like architecture that can cope with various image sizes and achieves linear computational complexity. \n(2)  The authors contribute amounts of experiments to show the comparable performance even higher performance in semantic segmentation. \n\nCons:\n(1)  It is a little difficult to understand Figure 1. In Figure 1(c), you implement cycle operation in the HW dimension, but in Figure 1(d), it seems that you only implement cycle operation in the H dimension. It needs to be clarified.\n(2)  In Table 4 and Table 5, the authors conduct ablation studies about stepsize. It is implemented when stepsize is 1x7 and 7x1, what if the stepsize is 2x7 or 7x2? Does this situation exist? Another, can the stepsize be an even number? \n(3)  Besides, ViP uses the 3-branch structure and AS-MLP seems to be the concurrent work. What are the specific differences between CycleMLP and these architectures?\n(4)  About Figure 4, what means the effective receptive field? And what does the higher color value mean? From this figure, Swin has the sharp ERF and CycleMLP has smoother ERF. Can you explain it?\n", "summary_of_the_paper": "This paper presents a simple MLP-like architecture, CycleMLP for image classification, object detection and segmentation. It can cope with various image sizes and achieves linear computational complexity to image size. CycleMLP achieves competitive results on object detection, instance segmentation and semantic segmentation. Further, it outperforms Swin-Tiny on the ADE20K dataset with fewer FLOPs. ", "summary_of_the_review": "This paper shows competitive results of MLP-like architecture on image classification, object detection and segmentation. However, there exist some unclear descriptions. Currently I rate it as a borderline paper and the authors need to consider the above questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RJKfiDbE-S0", "reviewer": "Reviewer_34rq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: 1. The Cycle FC is capable of dealing with various image scales, and has linear computational complexity the same as channel FC and a larger receptive field than Channel FC. 2. CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation. \n\nWeaknesses: 1. The Cycle FC align features at different spatial locations to the same channel, but analysis is slightly insufficient. There could be many different designs of it. For example, the experiments or analysis with different sampling intervals and sample size. 2. CycleMLP is slightly insufficient in discussion of the design and ablation studies.", "summary_of_the_paper": "This paper presents a  MLP-like architecture, CycleMLP. The proposed CycleMLP can be used for dense prediction, which is more suitable for object detection or image segmentation tasks.  The experiments show the effectiveness of the proposed model. The proposed Cycle FC can reduce the amount of network parameters and calculation, and is insensitive to image resolution.", "summary_of_the_review": "The proposed method is simple and interesting. Compared to modern MLP architectures, the CycleMLP can cope with various image sizes and achieves linear computational complexity. The benefits of each component and the performance difference are shown. All experiments of this paper is detailed and the overall content is sufficient. The proposed method is applicable to many vision tasks in the future.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q11by1Dmi8m", "reviewer": "Reviewer_QA1s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method modifies the existing 1x1 convolution (channel FC in paper), to sample points in a cyclical manner inside neighbourhood window along the channel dimension. It has the same weight shape as 1x1 convolution, ie: C_in x C_out. Stepsize is used to control the receptive field and can be thought of as analogous to kernel size in convolution. Overall Cycle MLP mimics the effect of sparse convolution kernel where each element along channel dimension is sampled at a different spatial position in a cyclical manner.\n\nStrengths: \n- Simple idea with promising results.\n- Readily adaptable to existing architectures with vanilla MLP.\n- Increases the receptive field without increasing the number of  parameters\n\nWeakness/suggested improvements:\n- There are no ablations or explanations on how/why the induced sparsity is helping in performance.\n- It is not evident that why cyclical sampling is better than random sampling or what is the significance of cyclical sampling. Some discussion and experiments around this point would be useful. \n- A minor typo: MHSA is written as MSHA below eqn3\n\n", "summary_of_the_paper": "The paper proposes Cycle MLP architecture, the idea is to bring spatial context into Channel FC and increase its receptive field.  The main objective of the paper is to address the challenges faced by the current MLP-Architectures. Cycle MLP allows flexible image resolution and avoids quadratic computational complexity in dense architectures. The paper presents results in a variety of tasks and shows demonstrates promising performance. \n", "summary_of_the_review": "Overall a simple idea and seems to work well. However, some more intuition around cyclic sampling would be worthwhile and might improve the paper. Exploring alternate sampling strategies would be interesting (random sampling or some other pattern can be explored). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IwLKiPCV4RX", "reviewer": "Reviewer_dMw3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is strong in the sense that it proposes a simple yet effective idea to extend previous MLP work for image tasks.  In a sense the basic idea of a sparse sampling across spatial and channel space is a natural variation to the other extremes in previous work such as channel and spatial FC.  Thus the methodology follows the basic pattern as previous work so doesn't require much explanation in terms of details.  In any case, the primary contribution here is the extensive sense of experimental results.  Figure 2 and Table 2 already provide decent support for CycleMLP, and the results showing FLOPs and parameter counts further strengthens the results.  Also as CycleMLP should be generic in terms of tasks the additional results on detection and segmentation make the idea more convincing.  Overall the basic contribution of improving MLP models to have linear complexity and applicability to varying image sizes is solid.\n\nAlthough the main work here is experimental in proving out the idea, it would be useful to have a deeper analysis of why the method works well.  In particular, the strided sampling of CycleMLP seems a bit motivated by ease and computational complexity, but seems it doesn't quite have the basic property of CNNs in being adapted to local correlations in natural images.  Presumably the patch embedding part is critical here as it already explicitly enforces spatial locality before CycleMLP is applied?  Do the previous MLP methods use patch embedding, and if they do, would they also have similar performance?  In other words, how much of the desired properties of linear computational complexity and applicability to varying image sizes comes from the patch embedding itself?", "summary_of_the_paper": "This paper proposes an improvement to existing MLP style models for image tasks that extends the general MLP competitive work compared to CNN models while attempting to capture some desirable CNN properties such as applicability to varying image sizes.  In addition the method has linear computational complexity as compared to previous MLP work which have quadratic complexity.  The basic idea is a variant of previous channel and spatial FC ideas by having a node's receptive field sample across spatial and channel domains rather than being fixed to either just channel or spatial aggregation.  The overall architecture uses the previous work on patch embeddings and idea of stages with stacked blocks.  Comprehensive experimental results are show improvement over previous MLP models and competitive results to other methods such as CNNs and transformers on multiple tasks including classification and segmentation.", "summary_of_the_review": "Overall the comprehensive and varied experimental results give the paper its strength.  The results are either competitive, such as on segmentation, or better than the previous MLP methods.  The idea is simple and a natural extension of previous ideas for fully connected layers.  There is a question on the relative contribution of the patch embedding when comparing to previous MLP methods, but generally no major weaknesses in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new MLP-Mixer-like architecture called Cycle MLP which has two main advantages with respect to MLP-Mixer: (i) it’s applicable to varying input image sizes, and (ii) linear computational complexity. The authors present competitive results on image classification, object detection and segmentation.

The reviewers felt that both (i) and (ii) are key issues in the current MLP-Mixer-based models. The reviewers also appreciated the simplicity of the idea and the execution of the empirical evaluation. During the rebuttal and discussion phase the authors provided compelling evidence for the issues pointed out in the initial review. 

Given that MLP-Mixer-based architectures are becoming increasingly popular, I believe that these contributions will be of great interest to the ICLR community and I will recommend acceptance.
Decision: Accept (Oral)