Year: 2021

Abstract:
This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions. As compared to modern MLP architectures, e.g. , MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron et al., 2021a), and gMLP (Liu et al., 2021a), whose architectures are correlated to image size and thus are infeasible in object detection and segmentation, CycleMLP has two advantages compared to modern approaches. (1) It can cope with various image sizes. (2) It achieves linear computational complexity to image size by using local windows. In contrast, previous MLPs have O(N) computations due to fully spatial connections. We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g. Swin Transformer (Liu et al., 2021b), while using fewer parameters and FLOPs. We expand the MLP-like models’ applicability, making them a versatile backbone for dense prediction tasks. CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation. In particular, CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs. Moreover, CycleMLP also shows excellent zero-shot robustness on ImageNet-C dataset. Code is available in the supplementary material.

heading: 1 INTRODUCTION
text: Vision models in computer vision have been long dominated by convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016). Recently, inspired by the successes in Natural Language Processing (NLP) field, Transformers (Vaswani et al., 2017) are adopted into the computer vision community. Built with multi-head self-attention layers, multi-layer perceptrons (MLPs), and skip connections (He et al., 2016), Transformers make numerous breakthroughs on visual tasks (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b). More recently, (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) have validated that building models solely on MLPs and skip connections without the self-attention layers can achieve surprisingly promising results on ImageNet (Deng et al., 2009) classification. Compared with CNNs, these MLP-based models even achieve comparable recognition performance when trained on large scale datasets (e.g., ImageNet-21K and JFT-300M).
Despite promising results on visual recognition tasks, these MLP-like models can not be used in dense prediction tasks (e.g., object detection and semantic segmentation) due to the three challenges: (1) Current models are composed of blocks with non-hierarchical architectures, which make the model infeasible to provide pyramid and high-resolution feature
representations. (2) Current models cannot deal with flexible input scales due to the Spatial FC as shown in Figure 1b. The spatial FC is configured by an image-size related weight1, which requires input image to have a fixed scale. Thus, this structure typically requires the input image with
1We omit bias here for discussion convenience.
a fixed scale during both the training and inference procedure. It contradicts the requirements of dense prediction tasks, which usually adopt a multi-scale training strategy (Carion et al., 2020) and different input resolutions in training and inference stages (Lin et al., 2014; Cordts et al., 2016). (3) The computational and memory costs of the current MLP models are quadratic to input image sizes for dense prediction tasks (e.g., COCO benchmark (Lin et al., 2014)).
To address the first challenge, we construct a hierarchical architecture to generate pyramid features. For the second and third issues, we propose a novel variant of fully connected layer, named as Cycle Fully-Connected Layer (Cycle FC), as illustrated in Figure 1c. The Cycle FC is capable of dealing with various image scales and has linear computational complexity to image size.
Our Cycle FC is inspired by Channel FC layer illustrated in Figure 1a, which is designed for channel information communication (Lin et al., 2013; Szegedy et al., 2015; He et al., 2016; Howard et al., 2017). The main spirit of Channel FC lies in that it can deal with flexible image sizes since it is configured by image-size agnostic weight of Cin and Cout. However, the Channel FC is infeasible to aggregate spatial context information due to its limited receptive field.
Our Cycle FC is designed to enjoy Channel FC’s merit of taking input with arbitrary resolution and linear computational complexity while enlarging its receptive field for context aggregation. Specifically, Cycle FC samples points in a cyclical style along the channel dimension (Figure 1c). In this way, Cycle FC has the same complexity (both the number of parameters and FLOPs) as channel FC while increasing the receptive field simultaneously. To this end, we adopt Cycle FC to replace the Spatial FC for spatial context aggregation (i.e., token mixing) and build a family of MLP-like models for both recognition and dense prediction tasks.
The contributions of this paper are as follows: (1) We propose a new MLP-like operator, Cycle FC, which is computational friendly to cope with flexible input resolutions. (2) We take the first attempt to build a family of hierarchical MLP-like architectures (CycleMLP) based on Cycle FC operator for dense prediction tasks. (3) Extensive experiments on various tasks (e.g., ImageNet classification, COCO object instance detection, and segmentation, and ADE20K semantic segmentation) demonstrate that CycleMLP outperforms existing MLP-like models and is comparable to and sometimes better than CNNs and Transformers on dense predictions.
Related Work. Convolution Neural Networks (CNNs) has dominated the visual backbones for several years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016). (Dosovitskiy
5 10 15 20 25 Model capacity (MAdds: G)
72
74
76
78
80
82
Im ag
eN et
To p-
1 Ac
c (%
)
DeiT Mixer S2MLP ResMLP gMLP ViP CycleMLP (ours)
et al., 2020) introduced the first pure Transformer-based (Vaswani et al., 2017) model into computer vision and achieved promising performance, especially pre-trained on the large scale JFT dataset. Recently, some works (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) removed the attention in Transformer and proposed pure MLP-based models. Please see Appendix A for a comprehensive review of the literature on the visual backbones.

heading: 2 METHOD
text: In this section, we introduce CycleMLP models for vision tasks including recognition and dense predictions. To begin with, in Sec. 2.1 we formulate our proposed novel operator, Cycle FC, which serves as a basic component for building CycleMLP models. Then we compare Cycle FC with Channel FC and multi-head attention adopted in recent Transformer-based models (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b) in Sec. 2.2. Finally, we present the detailed configurations of CycleMLP models in Sec. 2.3.

heading: 2.1 CYCLE FULLY-CONNECTED LAYER
text: Notation. We denote an input feature map as X ∈ RH×W×Cin , where H,W denote the height and width of the image and Cin is the number of feature channels. We use subscripts to index the feature map. For example, Xi,j,c is the value of cth channel at the spatial position (i, j) and Xi,j,: are values of all channels at the spatial (i, j).
The motivation behind Cycle FC is to enlarge receptive field of MLP-like models to cope with downstream dense prediction tasks while maintaining the computational efficiency. As illustrated in Figure 1a, Channel FC applies weighting matrix on X along the channel dimension on fixed position (i, j). However, Cycle FC introduces a receptive field of (SH , SW ), where SH and SW are stepsize along with the height and width dimension respectively (illustrated in Figure 1 (d)). The basic Cycle FC operator can be formulated as below:
CycleFC(X)i,j,: = Cin∑ c=0 Xi+δi(c),j+δj(c),c ·W mlp c,: + b (1)
where Wmlp ∈ RCin×Cout and b ∈ RCout are parameters of Cycle FC. δi(c) and δj(c) are the spatial offset of the two axis on the cth channel, which are defined as below:
δi(c) = (c mod SH)− 1, δj(c) = (b c
SH c mod SW )− 1 (2)
Examples. We provide several examples (Figure 1 (d)-(f)) to illustrate the stepsize. For the sake of visualization convenience, we set the tensor’s W = 1. Thus, these three examples naturally all have SW = 1. Figure 1 (d) illustrates the offsets along two axis when SH = 3, that is δj(c) ≡ 0 and δi(c) = {−1, 0, 1,−1, 0, 1, · · · } when c = 0, 1, 2, · · · , 8. Figure 1a (e) shows that when SH = H , Cycle FC has a global receptive field. Figure 1a (f) shows that when SH = 1, there will be no offset along either axis and thus Cycle FC degrades to Channel FC (Figure 1a). We also provide a more general case where W 6= 1 and SH = 3, SW = 3 in Figure 7 (Appendix).
The offsets δi(c) and δj(c) enlarge the receptive field of Cycle FC as compared to Channel FC (Figure 1a), which applies weights solely on the same spatial position for all channels. The larger receptive field in return brings improvements on dense prediction tasks like semantic segmentation and object detection as shown in Table 1. Meanwhile, Cycle FC still maintains computational efficiency and flexibility on input resolution. Both the FLOPs and the number of parameters are linear to the spatial scale which are exactly the same as those of Channel FC. In contrast, although Spatial FC has a global receptive field over the whole spatial space, its computational cost is quadratic to the image scale. Besides, it fails to handle inputs with different resolutions.

heading: 2.2 COMPARISON BETWEEN MULTI-HEAD SELF-ATTENTION (MHSA) AND CYCLE FC
text: Inspired by Cordonnier et al. (2020), when re-parametried properly, a multi-head self-attention layer withNh heads can be formulated as below, which is similar to a convolution with kernel size √ Nh×√
Nh. (Please refer to Appendix C for detailed derivation)
MHSA(X)i,j,: = ∑
h∈{1,2,...,Nh}
Xi+∆i(h),j+∆j(h),:W mhsa,h + b (3)
where Wmhsa,h ∈ RCin×Cout is the parameter matrix for hth head in MHSA. b ∈ RCout is the bias vector. {∆i(h),∆j(h)} = {(0, 0), (1, 0), (−1, 0), · · · } contains all possible positional shift in convolution with kernel size √ Nh × √ Nh. Further, we stack all Wmhsa,h together and reshape it into Wmhsa ∈ RK×K×Cin×Cout . Then a relationship between Wmlp and Wmhsa can be formulated as follow.
Wmlpc,: = W mhsa δi(c)+1,δj(c)+1,c,:
(4)
equation 4 shows that only the weights of Wmhsa on spatial shift (δi(c)+1, δj(c)+1) are taken into account in Wmlp. This indicates that Cycle FC introduce an inductive bias that the weighting matrix in MHSA should be sparse. Thus Cycle FC inherits the large receptive field introduced in MHSA. The receptive field in Cycle FC is enlarged to (SH , SW ), which enables Cycle FC to tackle with downstream dense prediction tasks better. Meanwhile, with the sparsity inductive bias, Cycle FC maintains computational efficiency in MLP-based methods as compared to convolution and multihead self-attention. The parameter size in Cycle FC isCin×Cout while Wmhsa ∈ RK×K×Cin×Cout .

heading: 2.3 OVERALL ARCHITECTURE
text: Patch Embedding. Given the raw input image with the size of H ×W × 3, our model first splits it into patches by a patch embedding module (Dosovitskiy et al., 2020). Each patch is then treated as a “token”. Specifically, we follow (Fan et al., 2021; Wang et al., 2021a) to adopt an overlapping patch embedding module with the window size 7 and stride 4. These raw patches are further projected to a higher dimension (denoted as C) by a linear embedding layer. Therefore, the overall patch embedding module generates the features with the shape of H4 × W 4 × C.
CycleMLP Block. Then, we sequentially apply several Cycle FC Bloc blocks. Comparing with the previous MLP blocks (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) visualized in Figure 5 (Appendix), the key difference of Cycle FC block is that it utilizes our proposed Cycle Fully-Connected Layer (Cycle FC) for spatial projection and advances the models in context aggregation and information communication. Specifically, the Cycle FC block consists of three parallel Cycle FCs, which have stepsizes SH × SW of 1 × 7, 7 × 1, and 1 × 1. This design is inspired by the factorization of convolution (Szegedy et al., 2016) and criss-cross attention (Huang et al., 2019). Then, there is a channel-MLP with two linear layers and a GELU (Hendrycks & Gimpel, 2016) non-linearity in between. A LayerNorm (LN) (Ba et al., 2016) layer is applied before both parallel Cycle FC layers and channel-MLP modules. A residual connection (He et al., 2016) is applied after each module.
Stage. The blocks with the same architecture are stacked to form one Stage (He et al., 2016). The number of tokens (feature scale) is maintained within each stage. At each stage transition, the channel capacity of the processed tokens is expanded while the number of tokens is reduced. This
strategy effectively reduces the spatial resolution complexity. Overall, each of our model variants has four stages, and the output feature at the last stage has a shape of H32 × W 32 × C4. These stage settings are widely utilized in both CNN (Simonyan & Zisserman, 2014; He et al., 2016) and Transformer (Wang et al., 2021b; Liu et al., 2021b) models. Therefore, CycleMLP can conveniently serve as a general-purpose visual backbone and a generic replacement for existing backbones.
Model Variants. The design principle of the model’s macro structure is mainly inspired by the philosophy of hierarchical Transformer (Wang et al., 2021b; Liu et al., 2021b) models, which reduce the number of tokens at the transition layers as the network going deeper and increase the channel dimension. In this way, we can build a hierarchical architecture that is critical for dense prediction tasks (Lin et al., 2014; Zhou et al., 2017). Specifically, we build two model zoos following two widely used Transformer architectures, PVT (Wang et al., 2021b) and Swin (Liu et al., 2021b). Models in PVT-style are named from CycleMLP-B1 to CycleMLP-B5 and in Swin-Style are named as CycleMLP-T, -S, and -B, which represent models in tiny, small and base sizes. These models are built by adapting several architecture-related hyper-parameters, including Si, Ci, Ei and Li which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i. Detailed configurations of these models are in Table 11 (Appendix).

heading: 3 EXPERIMENTS
text: In this section, we first examine CycleMLP by conducting experiments on ImageNet-1K (Deng et al., 2009) image classification. Then, we present a bunch of baseline models achieved by CycleMLP in dense prediction tasks, i.e., COCO (Lin et al., 2014) object detection, instance segmentation, and ADE20K (Zhou et al., 2017) semantic segmentation.

heading: 3.1 IMAGENET-1K CLASSIFICATION
text: The experimental settings for ImageNet classification are mostly from DeiT (Touvron et al., 2020), Swin (Liu et al., 2021b). The detailed experimental settings for ImageNet classification can be found in Appendix E.1.
Comparison with MLP-like Models. We first compare CycleMLP with existing MLP-like models and the results are summarized in Table 2 and Figure 2. The accuracy-FLOPs tradeoff of CycleMLP consistently outperforms existing MLP-like models (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a; Guo et al., 2021; Yu et al., 2021; Hou et al., 2021) under a wide range of FLOPs, which we attribute to the effectiveness of our Cycle FC. Specifically, compared with one of the pioneering MLP work, i.e., gMLP (Liu et al., 2021a), CycleMLP-B2 achieves the same top-1 accuracy (81.6%) as gMLP-B while reducing more than 3× FLOPs (3.9G for CycleMLP-B2 and 15.8G for gMLP-B). Furthermore, compared with existing SOTA MLP-like model, i.e., ViP (Hou et al., 2021), our model CycleMLP-B utilizes less FLOPs (15.2G) than ViP-Large/7 (24.4G, the largest one of ViP family) while achiving higher top-1 accuracy.
It is noted that all previous MLP-like models listed in Table 2 do not conduct experiments on dense prediction tasks due to the incapability of dealing with variable input scales, which is discussed in Sec. 1. However, CycleMLP solved this issue by adopting Cycle FC. The experimental results on dense prediction tasks are presented in Sec. 3.3 and Sec. 3.4.
Comparison with SOTA Models. Table 3 further compares CycleMLP with previous state-ofthe-art CNN, Transformer and Hybrid architectures. It is interesting to see that CycleMLP models achieve comparable performance to Swin Transformer (Liu et al., 2021b), which is the state-ofthe-art Transformer-based model. Specifically, CycleMLP-B achieves slightly better top-1 accuracy (83.4%) than Swin-B (83.3%) with similar parameters and FLOPs. GFNet (Rao et al., 2021) utilizes the fast Fourier transform (FFT) (Cooley & Tukey, 1965) to learn spatial information and achieves similar performance as CycleMLP on ImageNet-1K classification. However, the architecture of GFNet is correlated with the input resolution, and extra operation (parameter interpolation) is required when input scale changes, which may hurt the performance of dense predictions. We will thoroughly compare CycleMLP with GFNet in Sec. 3.4 on ADE20K.

heading: 3.2 ABLATION STUDY
text: In this subsection, we conduct extensive ablation studies to analyze each component of our design. Unless otherwise stated, We adopt CycleMLP-B2 instantiation in this subsection.
Cycle Fully-Connected Layer. To demonstrate the advantage of the Cycle FC, we compare CycleMLP-B2 with two other baseline models equipped with channel FC and Spatial FC as spatial context aggregation operators, respectively. The differences of these operators are visualized in Figure 1, and the comparison results are shown in Table 1. CycleMLP-B2 outperforms the counterparts built on both Spatial and Channel FC for ImageNet classification, COCO object detection, instance segmentation, and ADE20K semantic segmentation. The results validate that Cycle FC is capable of serving as a general-purpose, plug-and-play operator for spatial information communication and context aggregation.
128 224 320 416 Testing Resolution
67.5
70.0
72.5
75.0
77.5
80.0
82.5
Im ag
eN et
To p-
1 Ac
c (% ) DeiT-S GFNet-S CycleMLP-B2
128 224 320 416 Testing Resolution
12.5
10.0
7.5
5.0
2.5
0.0
T op
-1 A
cc o
n Im
ag eN
et
DeiT-S GFNet-S CycleMLP-B2
Figure 3: Resolution adaptability. All models are trained on 224×224 and evaluated on various resolutions without fine-tuning. Left: Absolute top-1 accuracy; Right: Accuracy difference relative to that tested on 224×224. The superiority of CycleMLP’s robustness becomes more significant when scale varies to a greater extent.
Table 4 further details the ablation study on the structure of CycleMLP block. It is observed that the top-1 accuracy drops significantly after removing one of the three parallel branches, especially when discarding the 1×7 or 7×1 branch. To eliminate the probability that the fewer parameters and FLOPs cause the performance drop, we further use two same branches (denoted as “33” in Table 4) and one 1×1 branch to align the parameters and FLOPs. The accuracy still drops relative to CycleMLP, which further demonstrates the necessity of these three unique branches.
Resolution adaptability. One remarkable advantage of CycleMLP is that it can take arbitraryresolution images as input without any modification. On the contrary, GFNet (Rao et al., 2021) needs to interpolate the learnable parameters on the fly when the input scale is different from the one for training. We compare the resolution adaptability by directly evaluating models at a broad spectrum of resolutions using the weight pre-trained on 224×224, without fine-tuning. Figure 3 (left) shows that the absolute Top-1 accuracy on ImagNet and Figure 3 (right) shows the accuracy differences between one specific resolution and the resolution of 224×224. Compared with DeiT and GFNet, CycleMLP is more robust when resolution varies. In particular, at the 128×128, CycleMLP saves more than 2 points drop compared to GFNet. Furthermore, at higher resolution, the performance drop of CycleMLP is less than GFNet. Note that the superiority of CycleMLP becomes more significant when the resolution changes to a greater extent.

heading: 3.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
text: Settings. We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset. We first follow the experimental settings of PVT (Wang et al., 2021b), which are introduced in Appendix. E.2. The corresponding results are presented in Table 6. Then, in order to compare fairly with Swin Transformer, which adopts a different experimental recipe with PVT, we further follow the experimental settings of Swin with our CycleMLP-S model and the results are presented in Table 7.
Results. Firstly, as shown in Table 6, CycleMLP-based RetinaNet consistently surpasses the CNNbased ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Transformer-based PVT (Wang et al., 2021b) under similar parameter constraints, indicating that CycleMLP can server as an excellent general-purpose backbone. Furthermore, using Mask R-CNN (He et al., 2017) for instance segmentation also demonstrates similar comparison results. Furthermore, from Table 7, the CycleMLP can achieve a slightly better performance than Swin Transformer.

heading: 3.4 SEMANTIC SEGMENTATION
text: Settings. We conduct semantic segmentation experiments on ADE20K (Zhou et al., 2017) dataset and present the detailed settings in Appendix. E.3. Table 8 and Table 9 show the experimental results using training recipes from PVT and Swin respectively.
Results. As shown in Table 8, CycleMLP outperforms ResNet (He et al., 2016) and PVT (Wang et al., 2021b) significantly with similar parameters. Moreover, compared to the state-of-the-art Transformer-based backbone, Swin Transformer (Liu et al., 2021b), CycleMLP can obtain comparable or even better performance. Specifically, CycleMLP-B2 surpasses Swin-T by 0.9 mIoU with slightly less parameters (30.6M v.s. 31.9M).
Although GFNet (Rao et al., 2021) achieves similar performance as CycleMLP on ImageNet classification, CycleMLP notably outperforms GFNet on ADE20K semantic segmentation where input scale varies. We attribute the superiority of CycleMLP under a scale-variable scenario to the capability of dealing with arbitrary scales. On the contrary, GFNet (Rao et al., 2021) requires additional
heuristic operation (weight interpolation) when the input scale varies, which may hurt the performance.
Moreover, we also visualized the receptive field following (Xie et al., 2021), and the results are visualized in Figure 4, which demonstrate that our CycleMLP has a larger effective receptive field than Swin.

heading: 3.5 ROBUSTNESS
text: We further conduct experiments on ImageNet-C (Hendrycks & Gimpel, 2016) to analyze the robustness ability of the CycleMLP, following (Mao et al., 2021) and results are presented in Table 10. Compared with both Transformers (e.g. DeiT and Swin) and existing MLP models (e.g. MLP-Mixer, ResMLP, gMLP), CycleMLP achieves a stronger robustness ability.

heading: 4 CONCLUSION
text: We present a versatile MLP-like architecture, CycleMLP, in this work. CycleMLP is built upon the Cycle Fully-Connected Layer (Cycle FC), which is capable of dealing with variable input scales and can serve as a generic, plug-and-play replacement of vanilla FC layers. Experimental results demonstrate that CycleMLP outperforms existing MLP-like models on ImageNet classification and achieves promising performance on dense prediction tasks, i.e., object detection, instance segmentation and semantic segmentation. This work indicates that an attention-free architecture can also serve as a general vision backbone.

heading: A LITERATURE ON VISION MODEL
text: CNN-based Models. Originally introduced over twenty years ago (LeCun et al., 1989), convolutional neural networks (CNN) have been widely adopted since the success of the AlexNet (Krizhevsky et al., 2012) which outperformed prevailing approaches based on hand-crafted image features. There have been several attempts made to improve the design of CNN-based models. VGG (Simonyan & Zisserman, 2014) demonstrated a state-of-the-art performance on ImageNet via deploying small (3×3) convolution kernels to all layers. He et al.introduced skip-connections in ResNets (He et al., 2016), enabling a model variant with more than 1000 layers. DenseNet (Huang et al., 2017) connected each layer to every other layer in a feed-forward fashion, strengthening feature propagation and reducing the number of parameters. In parallel with these architecture design works, some other works also made significant contributions to the popularity of CNNs, including normalization (Ioffe & Szegedy, 2015; Ba et al., 2016), data augmentation (Cubuk et al., 2020; Yun et al., 2019; Zhang et al., 2017), etc.
Transformer-based Models. Transformers were first proposed by Vaswani et al.for machine translation and have since become the dominant choice in many NLP tasks (Devlin et al., 2018; Wang et al., 2018; Yang et al., 2019; Brown et al., 2020). Recently, transformer have also led to a series of breakthroughs in computer vision community since the invention of ViT (Dosovitskiy et al., 2020), and have been working as a de facto standard for various tasks, e.g., image classification (Dosovitskiy et al., 2020; Touvron et al., 2020; Yuan et al., 2021), detection and segmentation (Wang et al., 2021b; Liu et al., 2021b; Zheng et al., 2021; Xie et al., 2021), video recognition (Wang et al., 2021c; Bertasius et al., 2021; Arnab et al., 2021; Fan et al., 2021) and so on. Moreover, there has also been lots of interest in adopting transformer to cross aggregate multiple modality information (Radford et al., 2021; Gabeur et al., 2020; Dzabraev et al., 2021). Furthermore, combining CNNs and transformers is also explored in (Srinivas et al., 2021; Li et al., 2021; Wu et al., 2021; Touvron et al., 2021b).
MLP-based Models. MLP-based models (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) differ from the above discussed CNN- and Transformer-based models because they resort to neither convolution nor self-attention layers. Instead, they use MLP layers over feature patches on spatial dimensions to aggregate the spatial context. These MLP-based models share similar macro structures but differ from each other in the detailed design of the micro block. In addition, MLP-based models provide more efficient computation than transformer-based models since they do not need to calculate affinity matrix using key-query multiplication. Concurrent to our work, S2-MLP (Yu et al., 2021) utilizes a spatial-shift operation for spatial information communication. The similar aspect between our work and S2-MLP lies in that we all conduct MLP operations along the channel dimension. However, our Cycle FC is different from S2-MLP in: (1) S2-MLP achieves communications between patches by splitting feature maps along channel dimension into several groups and shifting different groups in different directions. It introduces extra splitting and shifting operations on the feature map. On the contrary, we propose a novel operator-Cycle Fully-Connected Layer-for spatial context aggregation. It does not modify the feature map and is formulated as a generic, plug-and-play MLP unit that can be used as a direct replacement of vanilla without any adjustments. (2) We design a pyramid structure for and conduct extensive experiments on classification, object detection, instance segmentation, and semantic segmentation. However, the output feature map of S2-MLP has only one single scale in low resolution, which is unsuitable for dense prediction tasks. Only ImageNet classification is evaluated on S2-MLP. We compared Cycle FC with S2-MLP in details in the Section 3.

heading: B COMPARISON OF MLP BLOCKS
text: We summary MLP blocks proposed by recent MLP-related works in Figure 5. We notice that existing MLP blocks, i.e., MLP-Mixer, ResMLP and gMLP share similar method of Spatial Proj: Transpose → Fully-Connected over spatial dimension → Transpose back. These models can not cope with variable image scales as the FC layers in Spatial Proj are configured by the seq len.
The blocks used for building CycleMLP consist of our proposed novel Cycle FC, whose configuration has nothing to do with image scales and can naturally deal with dynamic image scales.

heading: C FROM MULTI-HEAD SELF-ATTENTION TO CONVOLUTION
text: In this section, we provide details in how MHSA can be transferred into a convolution-like operator in equation 3. To start with, the a MHSA layer can be formulated as below:
MHSA(X) = concat h∈{1,...,Nh}
[SAh(X)]W out + b (5)
where W out ∈ R(NhCout)×C′out and b ∈ RC′out are parameters for the final linear projection. SAh is the hth self-attention module. Then we reshape X into X ∈ RHW×Cin and let T = H ×W , which indicates that there are T tokens in X . SAh can be defined as follow:
SA(X)t,: = softmax(At,:)V + b
A = (Q + P )(K + P )ᵀ (6)
where V = XW val, Q = XW qry, K = XW key are respectively the value, query and key matrix with learnable matrices W v ∈ RCin×Cout , W q ∈ RCin×Ck , W k ∈ RCin×Ck . P ∈ RT×Cin is the
positional embedding matrix containing positional information for every input token, which can be replaced by the output of any function fP that encodes the position of tokens. And A ∈ RT×T is the attention matrix where each element Ai,j is the attention score between the ith and jth token in X . With absolute positional encoding, the second line in equation 6 can be expanded as (Cordonnier et al., 2020):
Aq,k =(Xq,: + Pq,:)W qry((Xk,: + Pk,:)W key)ᵀ
=Xq,:W qry(Xk,:W key)ᵀ +Xq,:W qry(Pk,:W key)ᵀ + Pq,:W qry(Xk,:W key)ᵀ + Pq,:W qry(Pk,:W key)ᵀ
(7)
When we apply relative positional encoding scheme in (Dai et al., 2019), A is re-parametried into:
Aq,k = Xq,:W qry(Xk,:W key)ᵀ+Xq,:W qry(rδq,kŴ key)ᵀ+u(Xk,:W key)ᵀ+v(rδq,kŴ key)ᵀ (8)
where rδq,k is a positional encoding for relative distance δq,k = (δ1, δ2) between token q and k in X . Ŵ key is introduced to only pertain to the positional encoding rq,k. u and v are learnable parameter vectors that replace the original Pq,:W qry term, which implies that the attention bias remains the same regardless of the absolution positions of the query. If we set W qry = W key = 0 and Ŵ key = I , the first three terms in equation 8 vanish and Aq,k = vr ᵀ q,k. We set {∆i(h),∆j(h)} = {(0, 0), (1, 0), (−1, 0), · · · } contains all possible positional shift in convolution with kernel size√ Nh× √ Nh. For each head h, let rq,k = (‖δq,k‖2, δ1, δ2) and vh = −αh(1,−2∆i(h),−2∆j(h)), each softmax attention matrix becomes:
softmax(Ah)q,k =
{ 1 if δq,k = (∆i(h),∆j(h))
0 otherwise (9)
Substitute softmax(Ah) into equation 5 and we get
MHSA(X)i,j,: = ∑
h∈{1,2,...,Nh}
Xi+∆i(h),j+∆j(h),:W mhsa,h + b (10)

heading: D ARCHITECTURE VARIANTS
text: In order to conduct fair and convenient comparison, we build two model zoos: the one is in PVTStyle (named as CycleMLP-B1 to -B5) and the other in Swin-Style (named as CycleMLP-T, -S and - B). These models are scaled up by adapting several architecture-related hyper-parameters, including Si, Ci, Ei and Li which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i. Detailed configurations of these models are in Table 11.

heading: E EXPERIMENTAL SETUPS
text: E.1 IMAGENET CLASSIFICATION
Settings. We train our models on the ImageNet-1K dataset (Deng et al., 2009), which contains 1.2M training images and 50K validation images evenly spreading 1,000 categories. We follow the standard practice in the community by reporting the top-1 accuracy on the validation set. Our code is implemented based on PyTorch (Paszke et al., 2019) framework and heavily relies on the timm (Wightman, 2019) repository. For apple-to-apple comparison, our training strategy is mostly adopted from DeiT (Touvron et al., 2020), which includes RandAugment (Cubuk et al., 2020), Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019) random erasing (Zhong et al., 2020) and stochastic depth (Huang et al., 2016). The optimizer is AdamW (Loshchilov & Hutter, 2017) with the momentum of 0.9 and weight decay of 5×10−2 by default. The cosine learning rate schedule is adopted with the initial value of 1×10−3. All models are trained for 300 epochs on 8 Tesla V100 GPUs with a total batch size of 1024.
Further kernel optimization for Cycle FC may bring a faster speed but is beyond the scope of this work.
E.2 COCO INSTANCE SEGMENTATION
We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset, which contains 118K and 5K images for train and validation splits. We adopt the mmdetection (Chen et al., 2019) toolbox for all experiments in this subsection. To evaluate the our CycleMLP backbones, we adopt two widely used detectors, i.e., RetinaNet (Lin et al., 2017) and Mask R-CNN (He et al., 2017). All backbones are initialized with ImageNet pre-trained weights and other newly added layers are initialized via Xavier (Glorot & Bengio, 2010). We use the AdamW (Loshchilov & Hutter, 2017) optimizer with the initial learning rate of 1×10−4. All models are trained on 8 Tesla V100 GPUs with a total batch size of 16 for 12 epochs (i.e., 1× training scheduler). The input images are resized to the shorted side of 800 pixels and the longer side does not exceed 1333 pixels during training. We do not use the multi-scale (Carion et al., 2020; Zhu et al., 2020; Sun et al., 2021) training strategy. In the testing stage, the shorter side of input images is resized to 800 pixels while no constraint on the longer side.
E.3 ADE20K SEMANTIC SEGMENTATION
We conduct semantic segmentation experiments on ADE20K (Zhou et al., 2017) dataset, which covers a broad range of 150 semantic categories. ADE20K contains 20K training, 2K validation and 3K testing images. We adopt the mmsegmenation (Contributors, 2020) toolbox as our codebase in this subsection. The experimental settings mostly follow PVT (Wang et al., 2021b), which trains models for 40K iterations on 8 Tesla V100 GPUs with 4 samples per GPU. The backbone is initialized with the pre-trained weights on ImageNet. All models are optimized by AdamW (Loshchilov & Hutter, 2017). The initial learning rate is configured as 2×10−4 with the polynomial decay parameter of 0.9. Input images are randomly resized and cropped to 512×512 at the training phase. During testing, we scale the images to the shorted side of 512. We adopt the simple approach Semantic FPN (Kirillov et al., 2019) as the semantic segmentation method following (Wang et al., 2021b) for fair comparison.

heading: F SAMPLING STRATEGIES
text: We explore more sampling strategies in this subsection, including random sampling and dilated sampling inspired by dilated convolution (Yu & Koltun, 2016; Chen et al., 2018) (as shown in Figure 6). We also compare the dense sampling method with ours.
Random sampling. As shown in Table 13, we conduct experiments with random sampling for three independent trials and observe that the averaged Top-1 accuracy on ImageNet-1K drops by 1.3%. We hypothesize that the decreased performance is caused by the fact that random sampling will totally disturb the semantic information of objects, which is essential to image recognition. Compared with the random sampling strategy, our cyclical sampling is able to aggregate the adjacent pixels, which benefits in capturing the semantic information.
Dilated Stepsize (Figure 6). As shown in Table 13, we observe the result of dilated sampling is better than the random one (+1.0% acc) but lower than ours (−0.5% acc). In fact, compared with the random sampling, dilated solutions take their advantages in local information aggregation. However, compared with the cyclical sampling strategy, dilated solutions lose the fine-grained information for recognition. It may hurt the accuracy performance to some extent.
Dense sampling. we conduct ablation studies by using dense sampling strategies (i.e., vanilla convolution with kernel size 1×3 and 3×1). Since dense sampling strategies incredibly increase the models’ parameters and FLOPs, we do not have enough time to thoroughly optimize the model for 300 epochs. Therefore, for fair comparisons, we conducted extra ablation studies on training models for 100 epochs with the strictly same learning configurations. The results shown in Table 14 demonstrate that the sparse sampling strategy (ours) outperforms the dense one. The comparison indicates that the dense sampling strategies introduce redundant parameters, which makes the model hard to optimize. Our sparse sampling strategy with fewer parameters is proven to be efficient and optimization-friendly.
G VISUALIZATION EXAMPLES
For easier understanding of our proposed CycleMLP, we visualize several instances of CycleMLP in Figure 7, including general case with stepsize 3×3 (7a), even stepsize (7b), and examples where stepsize along height or width equals to 1 (7c, 7d).
We note that given specific number of input and output channels, no matter how the stepsize changes, the number of parameters of the CycleMLP does not change. Therefore, there is a trade-off of representation abilities between spatial and channel dimensions, which will be discussed in details in following experimental analysis.
Experiments: We further conduct experiments on CycleMLPs with stepsize of 2×7, 7×2, 7×3, 3×7, and 4×4, respectively. The results are summarized Table 15. For fair comparisons, all the models in the above table have the same parameters and FLOPs. We observe that the model with stepsize of 1×7 and 7×1 achieves the best performance, especially for semantic segmentation on ADE20K. To analyze the impact of stepsize on the performance, we take Figure 7 for better illustration. One can see that enlarging the stepsize can expand the spatial receptive field. However, at a cost, it will reduce the number of periods (groups) running along the channel dimension, which may hurt the channel-wise representation abilities. Taking a feature map with C = 18 for example, the CycleMLP with stepsize 3×3 (Figure 7(a)) runs through only 2 channel groups (curly brackets in the figure). However, the CycleMLP with stepsize 3×1 (Figure 7(c)) will run through 6 groups in total, making better use of the representation in the channel dimension. That’s to say, there is a trade-off between spatial and channel representation. We empirically found that CyCleMLP with stepsize of 1×7 and 7×1 achieves the best performance.

Review summaries:
The paper proposes Cycle MLP architecture, the idea is to bring spatial context into Channel FC and increase its receptive field.  The main objective of the paper is to address the challenges faced by the current MLP-Architectures. Cycle MLP allows flexible image resolution and avoids quadratic computational complexity in dense architectures. The paper presents results in a variety of tasks and shows demonstrates promising performance.  

This paper presents a simple MLP-like architecture, CycleMLP for image classification, object detection and segmentation. It can cope with various image sizes and achieves linear computational complexity to image size. CycleMLP achieves competitive results on object detection, instance segmentation and semantic segmentation. Further, it outperforms Swin-Tiny on the ADE20K dataset with fewer FLOPs. 

This paper proposes an improvement to existing MLP style models for image tasks that extends the general MLP competitive work compared to CNN models while attempting to capture some desirable CNN properties such as applicability to varying image sizes.  In addition the method has linear computational complexity as compared to previous MLP work which have quadratic complexity.  The basic idea is a variant of previous channel and spatial FC ideas by having a node's receptive field sample across spatial and channel domains rather than being fixed to either just channel or spatial aggregation.  The overall architecture uses the previous work on patch embeddings and idea of stages with stacked blocks.  Comprehensive experimental results are show improvement over previous MLP models and competitive results to other methods such as CNNs and transformers on multiple tasks including classification and segmentation.

This paper presents a  MLP-like architecture, CycleMLP. The proposed CycleMLP can be used for dense prediction, which is more suitable for object detection or image segmentation tasks.  The experiments show the effectiveness of the proposed model. The proposed Cycle FC can reduce the amount of network parameters and calculation, and is insensitive to image resolution.