AS-MLP: An Axial Shifted MLP Architecture for Vision
Authors: Dongze Lian
Zehao Yu
Xing Sun
Shenghua Gao
Venue: ICLR 2022 Poster (2022)
Keywords: Architecture Design
MLP
Classification
Detection
Segmentation

Abstract: An Axial Shifted MLP architecture (AS-MLP) is proposed in this paper. Different from MLP-Mixer, where the global spatial feature is encoded for information flow through matrix transposition and one token-mixing MLP, we pay more attention to the local features interaction. By axially shifting channels of the feature map, AS-MLP is able to obtain the information flow from different axial directions, which captures the local dependencies. Such an operation enables us to utilize a pure MLP architecture to achieve the same local receptive field as CNN-like architecture. We can also design the receptive field size and dilation of blocks of AS-MLP, \emph{etc}, in the same spirit of  convolutional neural networks. With the proposed AS-MLP architecture, our model obtains 83.3\% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset. Such a simple yet effective architecture outperforms all MLP-based architectures and achieves competitive performance compared to the transformer-based architectures (\emph{e.g.}, Swin Transformer) even with slightly lower FLOPs. In addition, AS-MLP is also the first MLP-based architecture to be applied to the downstream tasks (\emph{e.g.}, object detection and semantic segmentation). The experimental results are also impressive. Our proposed AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mIoU on the ADE20K dataset, which is competitive compared to the transformer-based architectures. Our AS-MLP establishes a strong baseline of MLP-based architecture. Code is available at \url{https://github.com/svip-lab/AS-MLP}.

Published as a conference paper at ICLR 2022
AS-MLP: AN AXIAL SHIFTED MLP ARCHITECTURE FOR VISION
Dongze Lian", Zehao Yu"
Xing Sun
ShanghaiTech University
Youtu Lab, Tencent
(1iandz, yuzh y Jesaaggaitthhhhhhh hwwahaaittccce.uu......o en
(winfreedsun/etencent..
Shenghua Gao
ShanghaiTecc University &
Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Engineering Research Center of Energy Efficient and Custom Al IC {gaoshh] beooshh)@shanghhae. tech. edu. cn
ABSTRACT
An Axial Shifted MLP architecture (AS-MLP) proposed in this paper. Different from MLP-Mixer, where the global spatial feature is encoded for information flow through matrix transposition and one token token-mixing MLP, we pay more attention the local features interaction. By axially shifting channels of the feature map, AS- MI.P able to obtain the information flow from different axial directions, which captures the local dependencies. Such an operation enables us to utilize pure MLP architecture to achieve the same local receptive field as CNN-like architec- ture. We can also design the receptive field size and dilation of blocks of AS-MLP, etc, in the same spirit of convolutional neural networks. With the proposed AS- model obtains 83.3% Top-1 accuracy with 88M parameters
MLP architecture, and 15.2 GFLOPs on the ImageNet- K dataset. Such a simple yet effective archi- tecture outperforms all MLP-based architectures and achieves competitive perfor- mance compared to the transformer-baseee architectures (e.g.. Swin Transformer) even with slightly lower FLOPs, In addition, AS-MLP also the first MLP-basee architecture to be applied to the downstream tasks (e.g., object detection and se- mantic segmentation). The experimental results are also impressive. Our proposed AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mloU on the ADE20K dataset, which competitive compared to the transformer-bast tectures. Our AS-MLP establishes strong baseline of MLP-based architecture. Code is available at
archi-
com/
MLP
INTRODUCTION
In the past decade, Convolutional Neural Networks (CNNs) (Krizhessky et al., 2012 He et al, 2016) have received widespread attention and have become the de-facto standard for computer sion. Furthermore, with the in-depth exploration and research on self-attention, transformer-bas architectures have also gradually emerged, and have surpassed CNN-based architectures in natural language processing (e.g., Bert (Devlin et al. 2018)) and vision understanding (e.g.. ViT (Dosovit- skiy et al. 2021), DeiT (Touvron et al., (02Tb)) with amounts of training data. Recently, (Tolstikiin et al. 2021 first propose MLP--baed architecture, where almost all network parameters are learned from MLP (linear layer). achieves amazing results, which comparable with CNN-like models.
Such promising results drive our exploration of MLP-based architecture. In the MI.P-Mixer (Tol- stikhin et al. 202I), the model obtains the global receptive field through matrix transposition and token-mixing projection such that the long-range dependencies are covered. However, this rarely makes full use of the local information, which is very important in CNN-like architecture (Simonyan & Zisserman 2015) He al. 2016) because not all pixels need long-range dependencies, and the
"qqual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2022
local information focuses more extracting the low-level features. In the transformer- based chitectures, Swin Transformer (Liu et al. 2021b) computes the self-attention in a window (7 x 7) instead of the global receptive field, which 18 similar directly using convolution layer with large kernel size (7 x 7) to cover the local receptive field. Some other papers have also already emphasized the advantages of local receptive fields, and introduced local information in the transformer, such as Localvit (Li et al., 2021), NesT (Zhang et al., (202]), etc. Driven by these ideas, we mainly explore the influence of locality on MLP-based architectures.
In order to introduce locality into the MLP-based architecture, one of the simplest and most intuitive ideas is to add a window to the MLP-Mixer, and then perform a token- mixiing projection of the local features within the window, just as done in Swin Transformer (Liu et al. (021b) and LINMAPPER (Fang et af.] 202T). However, if we divide the window (e.g., 7 x 7) and perform the token-mixing projection in the window, then the linear layer has the 49 x 49 parameters shared between windows, which greatly limits the model capacity and thus affects the learning of parameters and final results. Conversely, if the linear layer is not shared between windows, the model weights trained with fixed image size cannot be adapted to downstream tasks with various input sizes because unfixed input sizes will cause a mismatch in the number of windows.
Therefore, a more ideal way to introduce locality is to directly model the relationship between a feature point and its surrounding feature points at any position, without the need to set a fixed window (and window size) in advance. To aggregate the features of different spatial positions in the same position and model their relationships, inspired by (Wu et al.] 2018]] Lin et al. 2019] [Wang] iet al. 2020; Ho et al. 2019), we propose an axial shift strategy for MLP-based architecture, where spatially shift features in both horizontal and vertical directions. Such an approach not only aggregates features from different locations, but also makes the feature channel only need to be divided into l groups instead of k22 groups to obtain a receptive field of size k l with the help of axial operation. After that, a channel channel-mixing MLP combines these features, enabling the model to obtain local dependencies. It also allows us to design MI.P structure as the same as the convolutional kernel, for instance, to design the kernel size and dilation rate.
Based on the axial shift strategy, we design Axial Shifted MLP architecture, named AS-MLP. Our AS-MLP obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs in the ImageNet- IK dataset without any extra training data. Such a simple yet effective method outperforms all MLP-based architectures and achieves competitive performance compared to the rransformer-base architectures. It is also worth noting that the model weights in MLP-Mixer trained with fixed image size cannot be adapted downstream tasks with various input sizes because the token-mixing MLP has a fixed dimension. On the contrary, the AS-MLP architecture can be transferred to downstream tasks (e.g., object detection) due to the design of axial shift. As far we know, it is also the first work to apply MLP-based architecture to the downstream task. With the pre-trained model in the ImageNet- IK dataset, AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mloU on the ADE20K dataset, which is competitive compared to the transformer-base architectures.
2 RELATED WORK
CNN-based Architectures. Since AlexNet (Krizhevsky et al. 2012) won the ImageNet competition in 2012, the CNN-based architectures have gradually been utilized to automatically extract image features instead of hand-crafted features. Subsequently, the VGG network (Simonyan & Zisserman] 2015) is proposed, which purely uses series of 3 x 3 convolution and fully connected layers. ResNet (He et al., 2016) utilizes the residual connection to transfer features in different layers, which alleviates the gradient vanishing and obtains superior performance. Some papers make further improvements to the convolution operation in CNN-based architecture, such as dilated convolution (Yu & Koltun, (016) and deformable convolution (Dai et al.) (017).. EfficientNet (Tan & Le| 2019 2021) introduces neural architecture search into CNN to search for suitable network structure. These architectures build the CNN family and are used extensively in computer vision tasks.
mransformer-bssed Architectures. Transformer is first proposed in ( (Vaswani al. 2017). where the attention mechanism is utilized to model the relationship between features from the different spatial positions. Subsequently, the popularity of BERT (Devlin et al., 2018) in NLP also promotes the research on transformer in the field of vision. ViT (Dosovitskiy et al. (02T)) uses a transformer framework to extract visual features, where image is divided into 16 x 16 patches and the convo-
2

---

Published as a conference paper at ICLR 2022
W
#
W
H W
H W
H 32
W 32
2Cx
4Cx
T6
Stage
Stage
Stage
Stage
Linear Emteedding
Patch Per Porritionn
Patch Merging
Patcht Merging
Patch Merging
Axi al Shift MLP Block
Axial Shift MLP Block
Axial Shift MLP
Axial Shift MLP
Images
Block
Block
3xH XW
X2
X6
X2
Figure 1: A tiny version of the overall Axial Shifted MLP (AS-MLP) architecture.
lution layer is completely abandoned. It shows that the wansformer-based architecture can perform well in large-scale datasets (e.g., JFT-300M). After that, DeiT (Touvron et al. 2021b) carefully de- signs training strategies and data augmentation to further improve performance on the small datasets (e.g., ImageNet- - IK) Die (Zhou al. 202T) and CaiT (Tourron al. 2021c) consider the op- timization problem when the network deepens, and train a deeper transformer network. Cross ViT (Chen al. 202 la) combines the local patch and global patch by using two vision transformers. CPVT (Chu et al. 2021b) uses conditional position encoding to effectively encode the spatial po- sitions of patches. LeVIT (Graham et al.) 202T) improves ViT from many aspects, including the convolution embedding, extra non- non-lineat projection and batch normalization, etc. Transformer-LS (Zhu et al. 2021) proposes a long-range attention and a short-term attention to model long sequences for both language and vision tasks. Some papers also design hierarchical backbone extract spatial features at different scales, such as PVT (Wang et al. 2021). 1). Swin Transformer (Liu et al.. 20216), Twins (Chu al. 2021a) and NesT (Zhang et al. 2021). which can be applied downstream tasks.
MLP-based Architectures. MLP-Mixer (Tolstikhin et al. 2021) designs a very concise framework that utilizes matrix transposition and MLP transmit information between spatial features, and obtains promising performance. The concurrent work FF (Melas-Kyriazi 2021) also applies a sim- ilar network architecture and reaches similar conclusions. Subsequently, Res-MLP (Touvron al., 2021 is proposed, which also obtains impressive performance with residual MLP only trained on ImageNet-1K 1K gMLP (Liu et al., 2021a) and EA (Guo et al., 202T) introduce Spatial Gating Unit (SGU) and the external attention to improve the performance of the pure MLP-based architecture, respectively. Recently, Container (Gao et al. 2021) proposes a general network that unifies convo- lution, transformer, and MLP-Mixes S²-MLP (Yu et al. 2021) uses spatial-shift MLP for feature exchange. ViP (Hou et al. 202I). proposes a Permute-Min
layer for spatial information encoding to capture long- long-range dependencies. Different from these work, focus on capturing the local dependencies with axially shifting features in the spatial dimension, which obtains better perfor- mance and be applied the downstream tasks. Besides, the closest concurrent work with us, CycleMLP (Chen et al. 2021b) and 5V-MLPv2 (Yu et al., 2021) are also proposed. 52-MLPy2 im- proves S2-MLP and CycleMLP designs Cycle Fully-Connected Layer (Cycle FC) to obtain larger receptive field than Channel FC.
3
THE AS-MLP ARCHITECTURE
3.1 OVERALL ARCHITECTURE
Figure shows our Axial Shifted MIP (AS-MLP) architecture, which refers to the style of Swin Transformer (Liu et al. 2021b). Given an RGB image 1 g3xHXx where H and W are the height and width of the image, respectively, AS-MLP performs the patch partition operation, which splits the original image into multiple patch tokens with the patch size of x 4, thus the combination of all tokens has the size of 48 x x 4 numbers of AS-MLP blocks in different stages. Figure Only shows the tiny versxion of AS-MLP, and other variants will be discussed in Sec. 3.4 All the tokens in the previous step will go through these four stages, and the final output feature will be used for image classification. In Stage linear embedding and the AS-MLP blocks are adopted for each token. The output has the dimension of C x W. where C is the number of channels. Stage first performs patch merging on the features outputted from the previous stage, which groups the neighbor 2 x patches to obtain feature with the size of 4C g 8 20'x x H x W.
W AS-MLP has four stages in total and there are different
W and then linear layer is adopted to warp the feature size to 8 , followed by the cascaded AS AS-MLP blocks. Stage and Stage 4 have similar structures to Stage 2, and the hierarchical representations will be generated in these stages.
3

---

Published as a conference paper at ICLR 2022
W
Zaro
3
2
P
Ouannel Projection
MIPP
4
Zaro podling
Nermh
Channel Projection
Chanmel Projection
W
Vertral Shift
Flriionnall Shift
P
Norm
5
Projection
Horizontal shift
(b)
Figure 2: (a) shows the structure of the AS-MLP block; (b) shows the horizontal shift, where the arrows indicate the steps, and the number in each box is the index of the feature.
3.2 AS-MLP BLOCK
The core operation of AS-MLP architecture is the AS-MLP block, which is illustrated in Figure 2a It mainly consists of the Norm layer, Axial Shift operation, MLP, and residual connection. In the Axial Shift operation, we utilize the channel projection, vertical shift, and horizontal shift to extract features, where the channel projection maps the feature with a linear layer. Vertical shift and horizontal shift are responsible for the feature translation along the spatial directions.
As shown in Figure 2b we take the horizontal shift as an example. The input has the dimension of C x h W. For convenience,
omit á and assume C = 3, w 5 in this figure. When the shift size is 3, the input feature is split into three parts and they are shifted by {-1, O. }} units along the horizontal direction, respectively. In this operation, zero padding is performed (indicated by gray blocks), and we also discuss the experimental results of using other padding methods in Sec. 4 After that, the features in the dashed box will be taken out and used for the next channel projection. The same operation is also performed the vertical shift. In the process of both shifts, since the feature performs different shift units, the information from different spatial positions be combined together. In the next channel projection operation, information from different spatial locations can fully flow and interact. The code of AS-MLP block is listed in Alg. !
Complexity. In the transformer-basee architecture, the multi-head self-attention (MSA) is usually adopted, where the attention between tokens computed. Swin Transformer (Liu et al. 202T6 introduces a window to partition the image and propose window multi-head self-attention (W-MSA), which only considers the computation within this window. It significantly reduces the computation complexity. However, in the AS-MPP block, without the concept of the window, we only Axially Shift (AS) the feature from the previous layer, which does not require any multiplication and addition operations. Further, the time cost of Axial Shift is very low and almost irrelevant to the shift size. Given a feature map (is usually named patches each Axial shift operation in Figure 2a only has four channel projection operations, which has the computation complexity 4hwC. If the window size in Swin Transformer (Liu et al. 2021b is M, the complexities of MSA, W-MSA and AS
transformer) with the dimension of C. x h x w,
follows:
!(MSA) 4hwC² + 2C, 2 2(W-MSA) = 4hwC2 + 2M2hwC ((AS) 4hw:C
Therefore, the AS-MLP architecture has slightly less complexity than Swin Transformer. The spe- cific complexity calculation of each layer is shown in Appendix A.2
3.3 COMPARISONS BETWEEN 1S-MLP, CONVOLUTION, TRANSFORMER AND
In this section, we compare AS-MLP with the recent distinet building blocks used in computer vision, e.g., the standard convolution, Swin Transformer, and MLP-Mixer. Although these modules are explored in completely different routes, from the perspective of calculation, they are all based on given output location point, and the output depends on the weighted sum of different sampling location features (multiplication and addition operation). These sampling location features include local dependencies (e.g., convolution) and long-range dependencies (e.g., MLP-Mixer). Figure shows the main differences of these modules in the sampling location. Given an input feature map X TR Hxxx XC the outputs Vi, with different operations in position (i.j) j) are follows:

---

Published as a conference paper at ICLR 2022
Algorithm 1 Code of AS-MLP Block in P PyTorch-like style.
norm: sormalization layer preji channel projection activation layer
import torch
Convolation
Transformer
MI.P-Mixer
import perrch.n..funstiin
det shift Ix, dim):
F.pad (x, "constant", 0)
torch. (x, shift size,_ 1)
ix_,, ahift, dim) for x_ shirt in zip (Kr range pad, pádíl11]
torch.
torch.cat (x,
return x|;, i, padi-pad, pad:-pad)
AS-MLP (s=3,d=1)
AS-MLP 5,=,8=11
AS-MLP (s= =)
as_alp_block block
shortcut
norm (x)
Figure 3: The different sampling locations of convolution, Swin Transformer, MLP-Mixer, and AS-MLP. e.g., AS-MLP (s 3, d 1) shows the sampling locations when the shift size is and dilation is 1.
sctn (noom(proo]]] (x)
1_1r acta (pro] (shift (x, 311)
x_td acta (proj lshift (x, 2)))
x_1r x_td
proj (normixi)
return x shortout
Convolution. For convolution operation, a sliding kernel with the shape of k: x (receptive field region R) is performed on X to obtain the output Yeym: Xitmj+nn
Yreunv
Σ (m,n)ER
Wreany m,n.31
where W/comy E [ is the learnable weight. / and w are the height and width of X, respectively. As shown in Figure the convolution operation has a local receptive field, thus it is more suitable at extracting features with the local dependencies.
Swin Transformer. Swin Transformer introduces a window into the transformer-based architecture to cover the local attention. The input X from a window is embedded to obtain Q, K, V matrix, and the output Y.wi is the attention combination of features within the window:
yssin Softmax Q ((x,,))))))))))))) (X))) /Va))(//V/V))))))) f.j
The introduction of locality further improves the performance of the transformer-based architecture and reduces the computational complexity.
MLP-Mixer. MLP-Mixer abandons the attention operation. It first transposes the input X, and then token-mixing MLP is appended to obtain the output Ymixer.
ymixer L
(X"
Wrmixer
(4)
where Wrmixer ho is the learnable weight in token-mixing mixing MLP. MLP-Mixer perceives the global information only with matrix transposition and MLP.
AS-MLP. AS-MLP axially shifts the feature map as shown in Figure 2b Given the input X, shift size & and dilation rate d, is first divided into s splits in the horizontal and vertical direction. After the axial shift in Figure 2b the output Y is:
1.
X 1 Warh
Wash + Σ 1- Wassvv Was
(5)
where Waash Was-v RC are the learnable weights of channel projection in the horizontal and vertical directions (here we omit activation function and bias). Unlike MLP-Mixer, we pay more attention to the local dependencies through axial shift of features and channel projection. Such an operation is closely related to Shift (Wu et al., (018) and TSM (Lin et al. 2019). However, our method has the following characteristics: i) we use axial shift in the horizontal and vertical directions, which focuses more on the information exchange in two directions; ii) the proposed network is built upon Swin transformer and pure MLP-based architecture, where only linear layer is used and BN is replaced by LN; iii) as will be shown in Sec. our network achieves superior performance, which shows the effectiveness of our method. Although such an operation can be implemented in the original shift (Wu et al. 2018), the feature channel needs to split into A² groups to achieve a receptive field of size x k. However, the axial operation makes the feature channel only need to be divided into l groups instead of 122 groups, which reduces the complexity.
5

---

Published as a conference paper at ICLR 2022
Input Resolution
Throughput (image /
Network
Top-1 (%)
Params
FLOPs
CNN-based
RegNetY-8GII Radosavooiic et al. 2020
x 224
81.7
8.0G
591.6
RegNerY [Ralosavovic EfficientNet-BS Tan 2019
224
82,9
15.9G
334.7
456
83.6
30M
9.9G
169.1
Transformerbusee
VIT-B/16 Dosovitskiy al. 2021
384 x 384
77.9
86M
55.5G
85.9
DeTT-B/16 Touvron al.
202
224 x 224
81.8
86M
17.6G
292.3
LVT-Large
Wang et al.
212
224 x 224
82.3
61M
9.8G
Swin-T
Liu ct al.
2021b
224 x 224
81.3
29M
4.5G
755.2
Swin-S
Lau eu al.
1b2
224 x 224
83.0
SOM
8.7G
436.9
Swin-B
Lau et
x 224
278.1
Swin-B
384
84.2
88M
47.0G
84.7
MLP-based
gMIP-S (Liu et al.
202
224 x 224
79.4
20M
4.5G
ViP-Small/ 14 Hou et al.
202
224 x 224
80.5
30M
789.0
ViP-Small/7 HHou et al
2021
224 x 224
81.5
25M
719.0
AS-MLP-T (ours)
224 x 224
81.3
28M
4.4G
1047.7
Mixer- B/16 (Tolstikhin et al. 2
224 x 224
76.4
59M
11.7G
FF (Melas Kyriazi 2021
224 x 224
74.9
62M
11.4G
ResMLF Touvron al 202
224 x 224
79.7
45M
8.9G
478.7
S S'-MLP-wide Yu al.
2021
224 x 224
80.0
68M
13.0G
S2-ML.Pdeeep Yu al.
202
224 x 224
80.7
5IM
9.7G
ViP-Medium/7 Hou
2021
224 x 224
82.7
55M
418.0
AS-MLP:S (ours)
224 x 224
83.1
50M
8.5G
619.5
gMLP-B (Liu et 202
224
81.6
73M
15.8G
ViP-Largef (Hou 2021
224 224
83.2
88M
298.0
AS AS-MLF-B (ours)
224 x 224
83,3
88M
15.2G
455.2
AS-MLP-B (ours)
384 x 384
84.3
88M
44.6G
179.2
Table 1: The experimental results of different networks ImageNet- IK. Throughput is measured with the batch size 64 on a single V100 GPU (32GB). The more complete accuracy and through- put comparisons listed in Appendix B.2
3.4 VARIANTS AS-MLP ARCHITECTURE
Figure [ only shows the tiny version of our AS-MLP architecture. Following DeiT (Tourron al., 2021b) and Swin Transformer (Liu et al. (021b), we also stack different number AS-MLP blocks (the number of blocks in four stages) and expand the channel dimension (C in Figure I to obtain variants of the AS-MLP architecture of different model sizes, which are AS-MLP-Tiny (AS-MLP-
T). AS-MLP-Small (AS-MLP& and AS-MLP-BB MLP-Base (AS-MLP-B). respectively. The specific figuration is as follows:
AS-MLP-T: C 96, the number of blocks in four stages (2, 6, 2):
AS-MLP-S: 96, the number of blocks four stages 22, 18, 2}:
AS-MLP-B: C' 128, the number of blocks in four stages 2, 18.
The detailed configurations can be found in Appendix A.1 Table in Sec. 4shows Top-1 accuracy, model size (Params), computation complexity (FLOPs) and throughput of different variants.
4
EXPERIMENTS
4.1 IMAGE CLASSIHCATION ON THE IMAGENET-IK IK DATASET
Settings. To evaluate the effectiveness of our AS-MLP, we conduct experiments of the image classi- fication on the ImageNet- 1K benchmark, which is collected in (Deng et al. It contains 1. 1.28M training images and 20K validation images from a total 1000 classes. We report the experimental results with single-crop Top-1 accuracy. We use initial learning rate of 0.001 with cosine decay and 20 epochs of linear warm-up. The AdamW (Loshchilov Hutter 2019) optimizer is employed to train the whole model for 300 epochs with batch size of 1024. Following the training strategy of Swin Transformer (Liu et al. (021b), we also use label smoothing (Szegedy et al. 2016) with smooth ratio of 0.1 and DropPath (Huang et al. 2016) strategy.

---

Published as a conference paper at ICLR 2022
Results. All image classification results shown in Table I We divide all network architec- tures into ENN-based, Transformer-based and MLP-based architectures. The imput resolution is 224 x 224. Our proposed AS-MLP outperforms other MLP-based architectures when keeping sim- ilar parameters and FLOPs. e-g. AS-MLP-S obtains higher top-1 accuracy (83. 1%) with fewer parameters than Mixer-B/16 (Tolstikhin al. (02T)) (76.4%) and ViP-Medium/7 (Hou et al.) 202T) (82.7%). Furthermore, it achieves competitive performance compared with transformer- chitectures, e.g., AS-MLP-B (83.3%) vs. Swin-B (Liu et al. 2021b (83.3%), which shows the effectiveness of our AS-MLP architecture.
ar-
Results of Mobile Setting, In addition to standard experiments, we also compare the results of AS-MLP in the mobile setting, which is shown in Table 2 We build the Swin (mobile) model and AS-MLP (mo- bile) model with similar parameters (about 10M). The specific network details can be found in Appendix A.3
Method
Top-1 (%)
Top-5 (%)
Params
Swin (mobile)
75.11
92.50
11.2M
AS-MLP (mobile)
76.05
92.81
9.6M
Table 2: The result comparisons of the mobile setting.
The experimental results show
that model significantly exceeds Swin Transformer (Liu et al. 2021b in the mobile setting (76.05% vs. 15).
4.2 THE CHOICE AND IMPACT OF AS-MLP BLOCK
The core component in the AS-MLP block is the axial shift. We perform experiments to analyze the choices of different configurations of the AS-MLP block, its connection types and the impact of AS-MLP block. All ablations are conducted based on the AS-MLP-T, as shown in the setting of
Sec. 3.4
Shift size
Padding method
dr.
Top-1
Top-5
Connection type
Structure
Top- I (%)
Top-5 (%)
(l. I)
NIA
1
74.17
91.13
(l. I) + (l. I)
74.32
91.46
(3. 3)
7ero padding
Circular padding
81.26
95.48
Serial
(5.5) (5.5)
3) + (3,3)
81.28
81.21
95.58
95.42
(3.3)
Reflect pukding
81.14
95.37
(7. 7) - (7.7)
81.17
95.54
(3.3)
Replicate padding
81.14
95.42
(L.
74.17
91.13
(3.3)
Zero padding
2
80.50
95.12
(3, () + (3, 3)
81.26
95.48
(5.5)
Zero padding
2
80.57
95.12
Parallel
81.34
95.56
(5. 5)
Zero pudding
1
81.34
95.56
(7. 7) (7.
81.32
95.55
(7. 7)
Zero padding
1
81.52
95.55
(9.9)
Zero pudding
81.16
95.45
(a) The impacts of the different configurations of the AS-MLP architecture. d.r. means dilation rate.
(b) The impacts of the different connection types.
'_y' means serial and '+' means parallel.
Table 3: Choices of different configurations and connection types.
Different Configurations of AS-MLP Block. In order to encourage the information flow from different channels in the spatial dimension, the features from the horizontal shift and the vertical shift are aggregated together in Figure 2 We evaluate the influence of different configurations of AS-MLP block, including shift size, padding method, and dilation rate, which are similar to the configuration of convolution kernel. All experiments of different configurations are shown in Table 3a We have three findings as follows: i) 'Zero padding' is more suitable for the design AS-MLP block than other padding method ii) increasing the dilation rate slightly reduces the performance of AS-MIP,, which is consistent with CNN-hased architecture. Dilation is usually used for semantic segmentation rather than image classification; iii) when expanding the shift size, the accuracy will increase first and then decrease. A possible reason is that the receptive field is enlarged (shift size 5 or 7) such that AS-MLP pays attention to the global dependencies, but when shift size is 9, the network pays too much attention to the global dependencies, thus neglecting the extraction of local features, which leads lower accuracy. Therefore, we use the configuration (shift size zero padding, dilation rate 1) in all experiments, including object detection and semantic segmentation.
We also compare the different connection types of AS-MLP block, such as se- rial connection and paraliel connection, and the results are shown in Table 3b Parallel connection consistently outperforms serial connection terms of different shift sizes, which shows the effec-
Connection Type.
Since use circular shift, thus 'No padding' and 'Cireular padding' equivalent.
7

---

Published as a conference paper at ICLR 2022
tiveness of the parallel connection. When the shift size is 1, the serial connection is better but it is not representative because only fhannel-mixing MLP is used.
The Impact of AS-MLP Block. We also evalu-
ate the impact of AS-MLP block in Table 4 Here design five baselines: i) Global-MLP; ii) Axial- MLP; ii Window-MLP; iv) shift size (3, 1); v) shift size (1, 5). The first three baselines signed from the perspective of how to use MLP for feature fusion different positions, and the latter two are designed from the perspective of the axial shift in single direction. The specific settings are listed in Appendix A.5
Method
Top-1 (%)
Method Top-1 (%)
Global-MLP
79.81
(5, 1)
78.37
Axial-MLP
79.69
(1.5)
78.45
de-
Vindow-MLP
78.40
(5,
81.34
Table 4: The impact of AS-MLP block.
The results Table
show that our AS-MLP block with shift size (5, 5) outperforms other baselines.
Backbone
AP"
AP
APSs
AP"
AP
AP
Params
FLOPs
Mask R-CNN (3x)
ResNer50 (He et al. 2016
41.0
61.7
44.9
37.1
58.4
40.1
44M
260G
PVT-Small Wang et al. 2021
43.0
65.3
46.9
39,9
62.5
42,8
44M
245G
Swin-T [iu al. 20216
46.0
68.2
50.2
41.6
65.1
44.8
48M
267G
A- (ours)
46.0
67.5
50.7
41.5
64.6
44.5
48M
260G
ResNer101 (He al. 2016
42.8
63.2
47.1
38.5
60.1
41.3
63M
336G
PVT-Medium (Wane et al. 2021
44.2
66.0
48.2
63.1
43.5
64M
302G
Swin-S (Liu al. 20216
48.5
70.2
53.5
43.3
67.3
69M
359G
AS-MLP-S (ours)
47.8
68.9
52.5
42.9
66.4
46.3
69M
346G
Cascade Mask R-CNN (3x)
DeiT-S (Touvron et al. 2021b
48.0
67.2
51.7
41.4
64.2
44.3
80M
889G
ResNeL50 (He ct al.
2016
46.3
64.3
50.5
40.1
61.7
43.4
82M
739G
Swin-T [Liu ct al.
2021b
50.5
69.3
54.9
43.7
66.6
47.1
86M
745G
AS-MLP-T (ours)
50.1
68.8
54.3
43.5
66.3
46.9
86M
739G
ResNext101-30 (Xie 2017)
48.1
66.5
52.4
41.6
63.9
819G
Swin-S (Liu (0221
51.8
70.4
56.3
44.7
67.9
838G
AS-MLP-S (ours)
51.1
69.8
55.6
44.2
67.3
48.1
107M
824G
ResNext101 (Xie et al. 2017)
48.3
66.4
52.3
41.7
64.0
45.1
140M
972G
Swin-B [Liu at 20216
51.9
70.9
56.5
45.0
68.4
48.7
145M
982G
AS-MI.P-B (ours)
51.5
70.0
56.0
44.7
67.8
48.4
145M
961G
Table 5:
The object detection and instance segmentation results different backbones with 3x schedule on the COCO val2017 dataset. The results with 1x schedule listed in Appendix B.1 B.1
4.3 ОВJЕCT DETECTION ON COCO
The experimental setting is listed in Appendix A.4 and the results are shown in Table 5 It is worth noting that we not compare our method with ML.P-Mixer (Tolstikhin et al., 2021 because uses a fixed spatial dimension for token-mixing MIP, which cannot be transferred to the object detection. As far as we know, we are the first work to apply MLP-based architecture to object detection. Our AS-MLP achieves comparable performance with Swin Transformer in the similar resource limitation. To be specific, Cascade Mask R-CNN + Swin-B achieves 51.9 AP with 145M parameters and 982 GFLOPs, and Cascade Mask R-CNN AS-MLP-B obtains 51.5 AP' with 145M parameters and 961 GFLOPs. The visualizations object detection are shown in Appendix
4.4 SEMANTIC SEGMENTATION
ADE20K
The experimental setting is listed in Appendix A.4 and Table 6 shows the performance of our AS- MLP on the ADE20K dataset. Note that we are also the first to apply the MLP-based architecture to semantic segmentation. With slightly lower FLOPs, AS-MLP-T achieves better result than Swin-T (46.5 v.s. 45.8 MS mloU). )). For the large model, UperNet + Swin-B B has 49.7 MS mloU with 12 pa- rameters and 1188 GFLOPs, and UperNet AS-MLP-B has 49.5 5,5 MS mloU with 121M parameters and I 1166 GFLOPs, which also shows the effectiveness our AS-MLP architecture processing the downstream task. The visualizations of semantic segmentation shown in Appendix C
8

---

Published as a conference paper at ICLR 2022
val MS mloU
Method
Backbone
Params
FLOPs
DANet Fu ct al.
2019a
ResNet-101
45.2
69M
1119G
DeepLabv
Chen ct al. 2018
ResNet-101
44.1
63M
1021G
ACNet (uu ci al.
2019b
ResNet-101
45.9
DNL Ym et al
20200
ResNet-101
46.0
69M
1249G
OCRNet
Yuan ei al 2020
ResNet-101
45,3
56M
923G
UperNet
X180 el at.
2018
ResNet-101
44.9
86M
1029G
OCRNet
Yuan al. 2020
HRNet-w48
45.7
71M
664G
DeepLabv3+
Chen et al.
2018
ResNeSt-
46.9
66M
1051G
DeepLabv3+
Chen et a1.
2018
ResNeSt-200
48.4
88M
1381G
(Liu Swin-T
al. 202
45.8
60M
945G
UperNet Xiao et al.
2018
AS-MLP-T (ours)
46.5
60M
937G
Swin-S (Lu et al. AS-MLP-S (ours)
2021b
49.5
81M
1038G
UperNet Xiao et
2018
49.2
81M
1024G
Swin-B (L.iu et al. AS-MLP-B (ours)
2021b
49.7
121M
1188G
UperNet Xiao al.
2018
49.5
121M
1166G
Table The semantic segmentation results different backbones on the ADE20K validation set.
4.5 VISUALIZATION
We visualize the heatmap of learned features from Swin Transformer and AS-MLP in Fig- ure where the first column shows the image from ImageNet, and the second column shows the activation heatmap of the last layer of Swin transformer (Swin-B). BB). The third, fourth, and fifth columns respectively indicate the response after the horizontal shift MML (h)), the ver- tical shift (AS-MLP (v)) and the combination of both in the last layer of AS-MLP (AS-MLP- B). From Figure 4 one can see that AS-MLP can better focus object regions compared to Swin transformer; ii) AS-MLP (h) can better focus on the vertical part of objects (as shown in the second row) because shifts feature in the horizontal direction. It is more reasonable because the shift in the horizontal direction can cover the edge of the vertical part, which is more helpful for recognizing the object. Sim-
Image
Swin
AS-MLP (h)
AS-MLP (V)
AS-MIP
Figure 4: The visualization features from Swin Transformer and our AS-Mi AS-MLP.
ilarly, AS-MLP (v) can better focus the horizontal part of objects (as shown in the fourth row).
CONCLUSION AND FUTURE WORK
In this paper, we propose an axial shifted MLP architecture, named AS-MLP, for vision. Compared with MLP-Mixer, pay more attention the local features extraction and make full of the channel interaction between different spatial positions through simple feature axial shift. With the proposed AS-MLP, we further improve the performance of MLP-basee architecture and the experimental results are impressive. Our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset. Such a simple yet effective method outperforms all MLP-based architectures and achieves competitive performance compared to the transformer- based architectures even with slightly lower FLOPs. We are also the first work to apply AS-MLP to the downstream tasks (e.g., object detection and semantic segmentation). The results are also competitive or even better compared to transformer based architectures, which shows the ability of MaPedased architectures in handling downstream tasks.
For future work, we will investigate the effectiveness of AS-MLP in natural language processing, and further explore the performance of AS-MLP on downstream tasks.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We would like to thank Ke Li, Hao Cheng and Weixin Luo for their valuable discussions and feed- back. This work was supported by National Key R&D Program of China (2018AAA0100704), NSFC #61932020, #62172279, Science and Technology Commission of Shanghai Municipality and "Shuguang Program" supported by Shanghai Education Devel- opment Foundation and Shanghai Municipal Education Commission.
(Grant No.
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4154-61622 2018
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), pp. 213-229 Springer, 2020.
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. CrossViT:
multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 357-366, October 2021a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprini arXiv: 01000007155, 2019.
Liang-Chieh Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818, 2018.
Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: mlp-like architec- ture for dense prediction. arXiv preprint arXiv: 210.v:2107.1000 10224, 2021b.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Ad- vances Neural Information Processing Systems (NeurIPS). 2021a.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint 2rXi::2102.1088.. 2021b.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https::/ www open mml hww.laammmmmetttiionn. 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE International Conference Computer Vi-
sion (ICCV), pp. 764-773, 2017..
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-2888 288-255,22 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04480 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Yuxin Fang. Xinggang Wang, Rui Wu, Jianwei Niu, and Wenyu Liu. What makes for hierarchical vision transformer? arXiv preprint arXiv:2107.02174 2021.
10

---

Published as a conference paper at ICLR 2022
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang. and Hanqing Lu. Dual attention network for scene segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3146-3154 2019a.
Jun Fu, Jing Liu, Yuhang Wang. Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive context network for scene parsing. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 6748-6757, 2019b.
Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herve Jegou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12259- 12269, 2021.
Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint arXiv:2105.0m 02358, 2021.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- 2019.
mensional transformers. arXiv preprint
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vi- sion permutator: A permutable mip-like architecture for visual recognition. arXiv preprint arXiv:2/06. 12366. 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pp. 646-661 Springer, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint urXiv:2104.05700 2021.
Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7083-7093, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint 21X.::2005.08000 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2021b.
11

---

Published as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Luke Melas-Kyriaze Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. arXiv preprint arXiv:2105.02723, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations (ICLR), 2015.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse R-CNN: End-to-end object detection with learnable proposals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14454-14463, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning (ICML), pp. 6105-6114, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning (ICML), 2021.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. MLP-Mixer: all-mlp architecture for vision. arXiv preprint arXiv:2105.01600 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03404, 202
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), 2021b.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021c.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In European Conference Computer Vision (ECCV), 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative spatial con- volutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9127-9135, 2018.
12

---

Published as a conference paper at ICLR 2022
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 418-434 4344, 2018.
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disen- tangled non-local neural networks. In European Conference Computer Vision (ECCV), pp. 191-207. Springer, 2020.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- fional Conference on Learning Representations (ICLR), 2016.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S²-MLP: Spatial-shift mip architecture for vision. arXiv preprint arXiv:2106.0777 07477, 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 173-190. Springer, 2020.
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister. Aggregating nested trans- formers. arXiv preprint arXiv:2 12723, 2021.
Daquan Zhou, Bingyi Kang. Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Die: Towards deeper vision transformer. arXiv preprint 2rXiv:2103.11888 2021.
Chen Zhu Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. arXiv preprint arXi:22107.02192 2021.
13

---

Published as a conference paper at ICLR 2022
A THE ARCHITECTURE DETAILS
A.1 THE DETAILED CONFIGURATIONS OF DIFFERENT ARCHITECTURES
We show the detailed configurations of different architectures in Table where assume the size of the input image 224 x 224. The second column shows the output size of the image after each stage. Following Swin Transformer (Liu al. 202Tb), concatenation of n x neighboring features in a patch. "shift size (5, 5)" means that the shift size in the horizontal and vertical directions is 5.
"Concat x n" indicate
downsp. (output size)
AS-MLP-T
AS-MLP-S
AS-MLP-B
concat 4x4, 96-d, LN
concat 44, 96-d, LN
concat 4x4, 128 2, LN shift size (5, 5),
4x
stage 1
shift size (5, 5),
shift size (5, 5),
56)
dim 96
dim 96
x
dim 128
2
concat 2x2, 192-d. LN
concat 2x2, 192-d LN
concat 2x2, 256-d LN shift size (5, 5), 2
8x (28x28)
stage 2
shift size (5, 5),
shift size (5, 5),
dim 192
x 2
dim 192
x
dim 256
concat 2x2, 384-d. LN
concat 2 384-d LN shift size (5, 5), dim 384
concat 2x2, 512-d. LN shift size (5, 5), dim 512
16x (14x 14)
stage 3
shift size (5, 5),
x 6
18
x 18
dim 384
concat 2x2, 768-d. LN
concat 2x2, 768-d LN
concat 2x2, 1024-d LN shift size (5, 5), dim 1024
32x (7x7)
stage
shift size (5, 5).
shift size (5, 5).
dim 768
x 2
dim 768
x
2
Params
28M
50M
88M
FLOPs
4.4G
8.5G
15.2G
Table The detailed configurations of different architectures.
A.2 THE COMPUTATIONAL COMPLEXITY OF AS-MLP ARCHITECTURE
In this section, we show the specific computational complexity each layer of AS-MLP architec- ture. The symbol definition is first given follows. An input image: I E R3xH*W patch size (p, p): the number of blocks in four stages: {N1. n2, n,, n4); Channel dimension C: MLP ratio: The specific computational complexity is shown in Table where only convolution operation is computed.
Stage
Stage
Linear embedding
AS-MLP block
Patch merging
AS-MLP block
Params FLOPPs
3Cp²
(4 (4+2r)C²mp
8C2
(4 + 2r)4C ny
3Cp² H
(r) H
8C2 H
(4+24 4r 2p 2p
W
2p 2p
Stage
Stage
Patch merging
AS-MLP block
Patch merging
AS-MLP block
Params FLOPs
32C² 2 H W. 3p 4p
(4 + 2r) (4 26) H. W.
128C2 128C H. W 8p 8p
(4+ 2r 2r) (4 24+2r)64C² W. 8p 8p n4
4p 4p mg
Table 8: The computational complexity of the AS-MLP Architecture.
14

---

Published as a conference paper at ICLR 2022
A.3 THE NETWORK DETAILS IN THE MOBILE SETTING
In addition to AS- AS-MLP-S, and AS-MLP-B, we also design AS-MLP in the mobile setting. For a fair comparison, we modify the Swin Transformer correspondingly adopt to the mobile setting. The configurations are as follow:
Swin (mobile); C' = 64, the number of blocks in four stages = {2, 2, 2}, the number of heads {2. 4, 8, 16]:
AS-MLP (mobile): C = 64, the number of blocks in four stages 2):
A.4 THE SETTINGS OF OBJECT DETECTION AND SEMANTIC SEGMENTATION
Object Detection on COCO. For the object detection and instance segmentation, we employ mmde- tection (Chen et al. 2019) the framework and COCO (Lin et al.) 2014) the evaluation dataset, which consists of 18K training data and 5K validation data. We compare the performance of our AS-MLP with other backbones COCO. Following Swin Transformer (Liu et al. 2021b), we sider two typical object detection frameworks: Mask R-CNN (He et al. 2017) and Cascade R-CNN as follows:
optimizer (AdamW), learning rate (0.0001), weight decay (0.05), and batch size (2 imgs/per GPUx8 GPUs). We utilize the typi- cal multi-scale training strategy (Carion et al. 2020 Sun et al. 202T) (the shorter side between 480 and 800 and the longer side is at most 1333). All backbones are initialized with weights pre- pre-traine on ImageNet- IK and all models are trained with 3x schedule (36 epochs).
(Cai & Vasconcelos 2018). The training strategies
2021b). we conduct experiments of AS-MLP on the challenging semantic segmentation dataset, ADE20K, which con- tains 20,210 training images and 2,000 validation images. We utilize UperNet (Xiao et al., 2018) and AS-MIPP backbone as our main experimental results. The framework based on mmsegmenta- tion (Contributors) 2020). The training strategies are as follows: optimizer (A (AdamW), learning rate (6 x 10 1). weight decay (0.01), and batch size (2 imgs/per GPUx8 GPUs). We utilize random horizontal flipping. random re-scaling within ratio range [0.5, 2.0] and random photometric distor- tion as data augmentation. The input image resolution is 512 x 512, the stochastic depth ratio is set as 0.3 and all models are initialized with weights pre-trained ImageNet- K and are trained 160K iterations.
Semantic Segmentation on ADE20K. Following Swin Transformer (Liu et al.
A.5 BASELINES
We list the specific configurations of baselines in Sec. 4.2 as follows.
Global-MLP: following MLP-Mixer Tolstikhhin et al. 2021 , we use global MLP (token- mixing MLP) along with full spatial size instead of AS-MLP block in our architecture configurations. For Global-MI.P, the model weights trained with fixed image size cannot be adapted to downstream tasks with various input sizes.
Axial-MLP: built upon Global-MLP, Axial-MLP employs two axial MLPs along with hor- izontal and vertical directions instead of global MLP. Similar to Global-MLP, the model weights trained with fixed image size cannot be adapted downstream tasks with various input sizes.
Window-MLP: stated in Sec I we set fixed window (7 x 7) in our architecture configu- rations and perform MLP operations within the window.
Shift size (5, 1): horizontal shift is 5 and vertical shift is AS-MLP block.
Shift size 5): horizontal shift is I and vertical shift is 5 in AS-MLP block.
A.6 DIFFERENCES BETWEEN AS-MLP AND TSM
We elaborate the differences between AS-MLP and TSM as follows.
TSM performs a shift in the temporal dimension and, as stated in (Lin et al. 2019), they target the temporal dimension for efficient video understanding. However, explore the
15

---

Published as a conference paper at ICLR 2022
Backhone
AP"
AP
AP
APm
AP
AP76
Params
FLOPs
Mask R-CNN (1x)
ResNet50 He el al. 2016
38.0
58.6
41.4
34.4
53.1
36.7
44M
260G
PVT-Small Wang el 2021
40.4
62.9
43.8
37.8
60.1
40.3
44M
245G
Swin-T (Lu
2021b)
43.7
66.6
47.7
39.8
63.3
42.7
48M
267G
AS-MLP-1 (ours)
44.0
66.0
48.5
40.0
62.8
43.1
48M
260G
ResNet101 He et al.
2016
40.4
61.1
44.2
36.4
57.7
38.8
63M
336G
PVT-Medium Wang 2021
42.0
64.4
45.6
39.0
61.6
42.1
64M
302G
AS-MLP-S (ours)
46.7
68.8
51.4
42.0
65.6
45.2
69M
346G
Cascade Mask R-CNN (Ix)
ResNet50 (He al
2016
46.3
64.3
50.5
40.1
61.7
43.4
82M
739G
Swin-T (Liu al.
2021b
67.1
52.2
41.7
64.4
45.0
745G
A (ours)
48.4
67.1
52.6
42.0
64.5
45.3
86M
739G
AS-MLP-S (ours)
50.5
69.4
54.7
43.7
66.9
47.3
107M
824G
AS-MLP-B (ours)
51.1
70.0
55.6
44.2
67.4
47.8
145M
961G
Table 9: The object detection and instance segmentation results of different backbones with 1x
schedule on the COCO val2017 dataset. Mask R-CNN and Cascade Mask R-CNN frameworks are employed.
shift from a spatial perspective, for the more general tasks, such as image classification, object detection, and segmentation.
TSM shows that shifting too many channels in a network will significantly hurt the spatial modeling ability and result in performance degradation. However, Table reftable: abla- tion(a) shows that as the shift size increases, the channel needs to be divided into more parts, but the performance does decrease significantly. This suggests that the argument of TSM is not obvious in the pure MLP architecture.
Our motivation is quite different. TSM is designed be more effective and efficient on video. Our motivation is derived from Swin Transformer's 's exploration of the local receptive field of the transformer. We use shift to explore the local receptive field of MLP. Also, AS- MLP is the first ML.P-based architecture for object detection and semantic segmentation with the help of such a method. Furthermore, we use axial shift reduce the complexity of the shift split.
B MORE EXPERIMENTS
B.1 MORE EXPERIMENTAL RESULTS ON COCO
Table 5 lists the object detection and instance segmentation results of different backbones with 3x schedule (36 epochs). For complete comparison, (12 epochs). The results are shown in Table 9 Our AS-MLP-T outperforms Swin-T (Liu et al. 2021b) under the Mask R-CNN (44.0 vs. 43.7 AP²) and Cascade Mask R-CNN (48.4 48.1 AP") frameworks.
also conduct experiments with 1x schedule
B.2 THE COMPLETE CLASSIFICATION ACCURACY AND THROUGHPUT COMPARISON
Table 10 shows the complete accuracy comparison with other state-of-the-art architectures on the ImageNet- 1K dataset. In addition, Table shows the throughput results of the AS-MLP architecture measured with the batch size 64 on single V100 GPU (32GB). In order to make fair comparison with other papers, we also conduct a thorough evaluation of throughput. The results are shown in Figure 5 where list the throughputs when the batch size is 1, 4, 8. 16, 32, 64, 128, respectively.
16

---

Published as a conference paper at ICLR 2022
Input Resolution
Throughput (image s)
Network
Top-1 (%)
Params
FLOPs
CNN-based
RegNetY-8GF Radosavovic al.
2020
224
224
81.7
39M
8.0G
591.6
RegNerY-16GF Radosavovic al.
2020
82.9
84M
15.9G
334.7
EfficientNet Tan & Le 2019
300
300
81.6
12M
1.8G
732.1
EfficientNet-BB Tan & Le
2019
456 456
83.6
30M
9.9G
169.1
Transformer-basee
ViT-B/16 Dosovitskiy al.
2021
384
77.9
86M
55.5G
85.9
DeiT-B/16 Touvron ot al.
b021bb
224 x 224
81.8
86M
17.6G
292.3
PVT-Large Wang et al. 2021
224 224
82.3
61M
9.8G
CPVT-B CChu at. 20216
224 224
82.3
88M
17.6G
285.5
TNT-B Han al. 2021)
82.8
66M
14.1G
T2T-ViT;
Yuan et al. 2021
224 x 224
82.6
65M
15.0G
CaiT-S36 (ouvrro ct al.
202
224 224
83.3
68M
13.9G
Swin-T Liu al.
202
224 224
81.3
29M
4.5G
755.2
Swin-S Liu
202161
224
83.0
50M
8.7G
436.9
Swin-B
Lau
2021b
83.3
88M
15.4G
278.1
Nest-B Zhang et al.
2022
224 221
83.8
68M
17.9G
235.8
Container (Gao et al.
2021
224 224
82.7
22M
8.1G
347.8
Swin-B B
Liu
2021b)
384 384
84.2
88M
47.0G
84.7
MI.P-hased
gMLP-S (Liu et al. 2021aa)
224 x 224
79.4
20M
4.5G
ViP-Small/ 14 (Hou al.
2021
224 224
80.5
30M
789.0
ViP-Small/7 (Hou
2021
224 224
81.5
25M
719.0
AS-MLP-1 (ours)
81.3
28M
4.4G
1047.7
Mixer- B/16 Tolstikhin
2021
224
76.4
59M
11.7G
FF Melas Kymazı 2021
224 224
74.9
62M
11.4G
ResMLP-36 Louvron et al
1
224 x 224
79.7
45M
8.9G
478.7
S--MLP-wide Yu al. 202
224 224
80.0
68M
13.0G
SH-MLP-deep el al. 202
224 224
80.7
51M
9.7G
ViP-Medium/7 (Hou et al
2021
224 224
82.7
55M
418.0
AS-MLP-S (ours)
224 224
83.1
50M
8.5G
619.5
gMLP-B B (Liu et al.
2021a)
224 224
81.6
73M
15.8G
LirgLar HHu
2021
224 224
83.2
88M
298.0
A (ours)
224
83.3
88M
15.2G 44.6G
455.2
AS-MI.P-B (ours)
384
84.3
88M
179.2
Table 10: The complete experimental results of different networks ImageNet-IK. Throughput is measured with the batch size of on a single V100 GPU (32GB).
B.3 EVALUATION ACCURACY
In Figure visualize the evaluation accuracy of AS-MLP and Swin transformer the ImageNet, COCO and ADE20K datasets during training. For image classification on ImageNet, AS-MLP-T keeps pace with Swin-T in each epoch and they finally converge the similar accuracy (81.3 81.3). For object detection and semantic segmentation on COCO and ADE20K, we can see that AS-MLP-T achieves better performance than Swin-T T in the early stage, and keeps winning during the training process.
B.4 COMPARISONS SHIFTRESNET
In this section, compare the performance with ShiftResNet (Wu et al., 2018) in Table 11 The results of ShittResNet50 with different configurations are from Table of ShiftResNet paper (Wu al. 2018). Our AS-MLP (mobile) achieves better accuracy with fewer parameters.
17

---

Published as a conference paper at ICLR 2022
1000
AS-MLP-T
AS-MLP-S
throughout (img/s)
800
AS-MLP-B (224)
AS-MLP-B (384)
600
400
200
0
20
40
60
80
100
120
batch size
Figure The throughput curve when the batch size is 8, 16. 32, 64, 128, respectively.
0.65
8.400
50
3 6350
0.82
033
SwinT
SwinTT
Swin T
ASMUPT
6235
MEPTT
0.30
ASMULP MUP-T
50
300
150 opoch
200
250
300
2
30
120
140
opoch
30]
(a) Accuracy AS-MLP.
(b) AP of AS-MLP.
(c) mloU of AS-MLP.
Figure 6: The evaluation accuracy of AS-MLP and Swin transformer the ImageNet, COCO and ADE20K datasets during training.
C THE VISUALIZATION OF RESULTS ON COCO AND ADE20K
We visualize the object detection and instance segmentation results the COCO dataset in Figure 7, where the Cascade Mask R-CNN model with the AS-MLP-T backbone used. We also visualize the semantic segmentation results the ADE20K dataset in Figure where utilize the UperNet model with the AS-MLP-T backbone. The object can be detected and segmented correctly.
Method
Top-1 (%)
Top-5 (%)
Params
Shi50Ressee55000
70.6
89.9
6M
ShittResNet50-11
73.7
91.8
11M
Shi5tResNet50-2
75.6
92.8
22M
AS-MLP (mobile)
76.05
92.81
9.6M
AS-MLP-T
81.34
95.56
28M
Table 11: The comparisons with ShiftResNet.
18

---

Published as a conference paper at ICLR 2022
Figure 7: The object detection and instance segmentation results on the COCO dataset.
Figure 8: The semantic segmentation results on the ADE20K dataset.
19


Reviews:
[{"review_id": "w1swBMAoUkd", "reviewer": "Reviewer_uLzi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The approach is an interesting combination of existing ideas.\n* The presented experimental results are competitive and cover three tasks.\n\nWeaknesses\n* The novelty of the approach seems somewhat limited. Taking Swin, MLP-Mixer (and colleagues), and Shift/TSM/ViP/S2-MLP, the added delta seems not very large. The main differences are highlighted at the end of Section 2 (\"we focus on capturing the local dependencies with axially shifting features in the spatial dimension, which obtains better performance and can be applied to the downstream tasks.\") and at the end of Section 3.3. (as (i) through (iii)). None of these differences is very convincing in my opinion, specifically the claimed \"better performance\" is not obvious after looking at all the result tables in detail.\n* The relation to both plain MLP-Mixer and to convolution is not clearly shown experimentally. It seems like both convolution and MLP-Mixer could appear as a limiting case of (some form of) AS-MLP. It would be great to understand this better both in principle and through experimental comparisons.\n* The main idea of the \"axial shift\" is not introduced with sufficient clarity in my opinion. Even after carefully paying attention to Sections 3.2 and 3.3, Figures 2 and 3, Algorithm 1, (and briefly looking at the code), I still had some doubts as to whether I have understood all details correctly. (Of course that could be my fault, I'm curious what the other reviewers think.) In my opinion, the notation used in Sec 3.3. is also slightly incomplete.\n\nMinor points:\n\nIn the introduction, the \"axial shift strategy\" is mentioned, but the explanation remains very vague and the reader has to wait until Section 3.2 to get a more detailed description. I think it would be better to introduce this main idea earlier and with more clarity. \n\n\"enables the model to obtain more local dependencies, thus improve the performance\" - The \"thus\" here is not a priori given in my opinion. At least it would require more elaboration.\n\n\"the first work to apply MLP-based architecture to the downstream task\" - There seems to be a strong relation to the recent CycleMLP, which may be concurrent work, but should nevertheless be cited and contrasted, I think.\nChen  et al.: CycleMLP: A MLP-like Architecture for Dense Prediction, arXiv:2107.10224, 2021\n\nFootnote 2: This is a very strong statement (\"cannot\"). It is probably clear that this does not work \"out of the box\", but e.g. the MLP-Mixer paper describes how an increase in image resolution can be handled (Appendix C). At least this statement could be discussed and explained in more detail.\n\n4.1 Settings: among the myriad of regularization strategies that exist, why are specifically label smoothing and DropPath chosen? How many other regularization strategies were tried? Or maybe is this based on Swin, in which case it should be noted here.\n\n4.2 This Section seems more like a parameter choice discussion than an ablation study to me.\n\n4.5 The statements derived from the four examples in Figure 4 (\"From Figure 4, one can see that\") seem like overclaiming, at least in parts: \"These phenomena further show the superiority of AS-MLP.\"\n\nIn the context of \"axial attention\" (even though the AS-MLP does not use attention) it seems there are at least two papers that are quite well-known and it might be useful to add them to the related work:\nWang et al.: Axial-DeepLab [...], ECCV, 2020\nHo et al.: Axial Attention in multidimensional transformers, arXiv:1912.12180, 2019\n\nMinor details (not influencing the recommendation):\n* Grammar: There are several errors in English grammar, the most prominent category seems the use of articles (definite \"the\" vs. indefinite \"a\", vs. no article).\n* \"firstly\" - in most cases where this word is used in the paper I think it should just be \"first\"\n* Footnote 1: I did not understand this footnote.\n* Algorithm 1: x_c -> x_s\n* \"uses a conditional position encoding to effectively encodes\" -> encode\n* Fig.1 seems to be heavily based on Swin Transformer's Fig. 3a - maybe cite that paper here? Similarly for the specific configurations in Sec 3.4.\n* \"The introduction of locality further improves the performance of the transformer-based architecture\nand reduce the computational complexity.\" -> reduces\n* References:\n** Some arXiv citations have appeared in conferences, e.g. the Transformer paper was published in NeurIPS or ResNet in CVPR, I think. It would be nice to cite the conference version for such papers (I did not check all).\n** Some words (mostly abbreviation) in reference titles should not be lowercase, e.g. \"r-cnn\", \"mlp\", \"Resmlp\" \n** Consistency: MMDetection has >10 authors listed, but other papers stop after the first N authors with \"et al.\"\n* Figure 7 and 8, while looking ok when viewed on screen, turned out very garbled in the color space when printed on paper, it would be useful to check this.\n\n--------------------------\n\nUpdate after reading other reviewers' comments, the authors' comments, and considering the updated paper:\n\nAll three reviewers seem to have a similar view of the submitted work and agree in their rating. I think the updated paper and the authors' comments have addressed some of the concerns that the reviewers have raised. I think that the updated paper has improved in quality. On the other hand I think that some of the weaknesses still remain. Overall, I believe that the paper is close to the acceptance threshold. Seeing the improvements to the paper I am willing to raise my score to \"marginally above the acceptance threshold\".", "summary_of_the_paper": "The paper proposes a new architecture for computer vision that is inspired by (a) the Swin Transformer, (b) MLP-Mixer (and colleagues) and (c) CNN-like local context via shifts (like Shift, TSM, ViP, S2-MLP). The architecture is based on the Swin Transformer, removes the windowed-attention, and then adds \"local shifts\" of channels to introduce local context via MLP. The architecture is applied to ImageNet-1k classification, COCO detection, and ADE20k segmentation with good results relative to model size and performance.\n", "summary_of_the_review": "An interesting combination of existing ideas evaluated on three computer vision tasks. Overall novelty seems limited, the paper does not explain its approach very well, and the gained understanding is limited. Empirical results are interesting but not super-convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rR0moUMhJQZ", "reviewer": "Reviewer_HTjG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe work shows good results on different vision tasks. \n\n\nWeaknesses:\n\n1. The novelty of the work is quite limited. The idea of shifting operation in vision has been already explored in numerous previous works, such as TSM (Lin et al., 2019) and S2-MLP (Yu et al., 2021). Especially, the proposed work is very similar to S2-MLP and, furthermore, the authors proposed even an improved version, S2-MLPV2, which reports even better results for 224x224 image resolution. \n\n2. The presentation of the results is not fair. For instance, in the ablation study (Table 3a) the authors present the proposed approach of shifting operation as superior comparing it without shifting operation. However, when the shift size is (1,1)  there is no spatial interaction in the architecture (all the information exchange is between the channels at the same spatial location). The authors claimed that introducing locality into MLP-based architecture “enables the model to obtain more local dependencies, thus improve the performance.” But the authors did not actually prove this very clearly. I think in this ablation study, it is necessary to include also the case when using a MLP along spatial dimension. For instance, there can be two cases, first, using  a global MLP along full spatial size, and second, using two MLPs (one MLP along horizontal spatial direction and another MLP along vertical spatial direction). Otherwise, it can be the case that big part of the improvement comes actually from just applying a standard MLP on the horizontal spatial direction and another MLP on the vertical direction. ", "summary_of_the_paper": "The work proposed an approach to introduce locality into the MLP-based architecture by using a shifting operation along vertical and horizontal directions. ", "summary_of_the_review": "The novelty of the work is limited and the authors did not show the results when using a standard MLP approach on the spatial direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q41HAZNW_nx", "reviewer": "Reviewer_Wh4q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The paper is well written and presented clearly.\n2. The proposed axial shift module is simple and elegant.\n3. Experiments are done extensively beyond ImageNet.\n\nWeaknesses\n1. Limited technical novelty given the fact that the shift operation (CVPR 2018) was already presented as an alternative (though similar) operation to convolution. The shift-ResNet is already spatial convolution free except the first stem layer. Now it is not surprising that using a convolution alternative in MLP-mixer can boost its performance to convolution level (i.e. better than previous MLP-mixers).\n2. The modification of using “axial” on top of “shift” lacks justification or ablation. What if the original shift operation is applied to the nearby 8 points in the 3x3 window? The channels can be kept the same by replacing “horizontal shift” with 4 points and “vertical shift” with another 4 points. This should be very similar to the “5x5” configure used in the paper.\n3. Another intuitive baseline is simply to use depth wise convolution with a depth multiplier of 2, instead of the 2 parallel axial shift module. Would this simple convolutional baseline affect performance/params/FLOPs/throughput by a large margin? Possibly not. If this simple convolutional baseline works, then what is the benefit of using AS-MLP-mixers?\n4. Efficientnetv2 (ICML 2021), a representative convolution-based method is not cited/discussed/compared. For example, Table 11 of Efficientnetv2 presented V2-S with 8.8G FLOPs, 901 images/s, 83.6% top-1. This is better than AS-MLP-B, 15.2G FLOPs, 455.2 images/s. Although the throughput comparison is not 100% fair (Efficientnetn2 uses a larger batch size, FP16 inference, and SE module), I think AS-MLP-B can hardly perform better than Efficientnetv2 in a fair throughput comparison.", "summary_of_the_paper": "This paper proposes to use the shift operation (Wu et al. CVPR 2018) in an axial manner for MLP-mixer architectures. The proposed method performs much better than previous MLP-based methods on ImageNet-1K and on par with Swin-transformer.", "summary_of_the_review": "I think this paper is around the acceptance threshold, mainly due to the limited technical novelty besides applying Shift, an alternative to convolution, to MLP-mixers, and achieving on par performance as previous transformer/convolution methods. I will raise my score if the weaknesses are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a MLP-based architecture that makes extensive use of the shift operation on the feature maps. The model performs well on several vision tasks and datasets.

The reviews are mixed even after the authors' response. Main pros are that the proposed architecture is elegant and reasonable, and the experimental evaluation is thorough and strong. The main con is that the novelty is somewhat limited to some prior papers.

Overall, I recommend acceptance. The reviewers point out that the architecture is good and the results are strong. Similarities to prior works do not seem serious enough to warrant rejection - even an author of arguably the most related (concurrent) works - S2-MLP and S2-MLPv2 - confirms that there is sufficient difference. Moreover, this is one of the first papers to show very strong results on detection and segmentation.
Decision: Accept (Poster)