Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work proposed an approach to introduce locality into the MLP-based architecture by using a shifting operation along vertical and horizontal directions. 
main_review: Strengths:
The work shows good results on different vision tasks. 


Weaknesses:

1. The novelty of the work is quite limited. The idea of shifting operation in vision has been already explored in numerous previous works, such as TSM (Lin et al., 2019) and S2-MLP (Yu et al., 2021). Especially, the proposed work is very similar to S2-MLP and, furthermore, the authors proposed even an improved version, S2-MLPV2, which reports even better results for 224x224 image resolution. 

2. The presentation of the results is not fair. For instance, in the ablation study (Table 3a) the authors present the proposed approach of shifting operation as superior comparing it without shifting operation. However, when the shift size is (1,1)  there is no spatial interaction in the architecture (all the information exchange is between the channels at the same spatial location). The authors claimed that introducing locality into MLP-based architecture “enables the model to obtain more local dependencies, thus improve the performance.” But the authors did not actually prove this very clearly. I think in this ablation study, it is necessary to include also the case when using a MLP along spatial dimension. For instance, there can be two cases, first, using  a global MLP along full spatial size, and second, using two MLPs (one MLP along horizontal spatial direction and another MLP along vertical spatial direction). Otherwise, it can be the case that big part of the improvement comes actually from just applying a standard MLP on the horizontal spatial direction and another MLP on the vertical direction. 
summary_of_the_review: The novelty of the work is limited and the authors did not show the results when using a standard MLP approach on the spatial direction.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new architecture for computer vision that is inspired by (a) the Swin Transformer, (b) MLP-Mixer (and colleagues) and (c) CNN-like local context via shifts (like Shift, TSM, ViP, S2-MLP). The architecture is based on the Swin Transformer, removes the windowed-attention, and then adds "local shifts" of channels to introduce local context via MLP. The architecture is applied to ImageNet-1k classification, COCO detection, and ADE20k segmentation with good results relative to model size and performance.

main_review: Strengths:
* The approach is an interesting combination of existing ideas.
* The presented experimental results are competitive and cover three tasks.

Weaknesses
* The novelty of the approach seems somewhat limited. Taking Swin, MLP-Mixer (and colleagues), and Shift/TSM/ViP/S2-MLP, the added delta seems not very large. The main differences are highlighted at the end of Section 2 ("we focus on capturing the local dependencies with axially shifting features in the spatial dimension, which obtains better performance and can be applied to the downstream tasks.") and at the end of Section 3.3. (as (i) through (iii)). None of these differences is very convincing in my opinion, specifically the claimed "better performance" is not obvious after looking at all the result tables in detail.
* The relation to both plain MLP-Mixer and to convolution is not clearly shown experimentally. It seems like both convolution and MLP-Mixer could appear as a limiting case of (some form of) AS-MLP. It would be great to understand this better both in principle and through experimental comparisons.
* The main idea of the "axial shift" is not introduced with sufficient clarity in my opinion. Even after carefully paying attention to Sections 3.2 and 3.3, Figures 2 and 3, Algorithm 1, (and briefly looking at the code), I still had some doubts as to whether I have understood all details correctly. (Of course that could be my fault, I'm curious what the other reviewers think.) In my opinion, the notation used in Sec 3.3. is also slightly incomplete.

Minor points:

In the introduction, the "axial shift strategy" is mentioned, but the explanation remains very vague and the reader has to wait until Section 3.2 to get a more detailed description. I think it would be better to introduce this main idea earlier and with more clarity. 

"enables the model to obtain more local dependencies, thus improve the performance" - The "thus" here is not a priori given in my opinion. At least it would require more elaboration.

"the first work to apply MLP-based architecture to the downstream task" - There seems to be a strong relation to the recent CycleMLP, which may be concurrent work, but should nevertheless be cited and contrasted, I think.
Chen  et al.: CycleMLP: A MLP-like Architecture for Dense Prediction, arXiv:2107.10224, 2021

Footnote 2: This is a very strong statement ("cannot"). It is probably clear that this does not work "out of the box", but e.g. the MLP-Mixer paper describes how an increase in image resolution can be handled (Appendix C). At least this statement could be discussed and explained in more detail.

4.1 Settings: among the myriad of regularization strategies that exist, why are specifically label smoothing and DropPath chosen? How many other regularization strategies were tried? Or maybe is this based on Swin, in which case it should be noted here.

4.2 This Section seems more like a parameter choice discussion than an ablation study to me.

4.5 The statements derived from the four examples in Figure 4 ("From Figure 4, one can see that") seem like overclaiming, at least in parts: "These phenomena further show the superiority of AS-MLP."

In the context of "axial attention" (even though the AS-MLP does not use attention) it seems there are at least two papers that are quite well-known and it might be useful to add them to the related work:
Wang et al.: Axial-DeepLab [...], ECCV, 2020
Ho et al.: Axial Attention in multidimensional transformers, arXiv:1912.12180, 2019

Minor details (not influencing the recommendation):
* Grammar: There are several errors in English grammar, the most prominent category seems the use of articles (definite "the" vs. indefinite "a", vs. no article).
* "firstly" - in most cases where this word is used in the paper I think it should just be "first"
* Footnote 1: I did not understand this footnote.
* Algorithm 1: x_c -> x_s
* "uses a conditional position encoding to effectively encodes" -> encode
* Fig.1 seems to be heavily based on Swin Transformer's Fig. 3a - maybe cite that paper here? Similarly for the specific configurations in Sec 3.4.
* "The introduction of locality further improves the performance of the transformer-based architecture
and reduce the computational complexity." -> reduces
* References:
** Some arXiv citations have appeared in conferences, e.g. the Transformer paper was published in NeurIPS or ResNet in CVPR, I think. It would be nice to cite the conference version for such papers (I did not check all).
** Some words (mostly abbreviation) in reference titles should not be lowercase, e.g. "r-cnn", "mlp", "Resmlp" 
** Consistency: MMDetection has >10 authors listed, but other papers stop after the first N authors with "et al."
* Figure 7 and 8, while looking ok when viewed on screen, turned out very garbled in the color space when printed on paper, it would be useful to check this.

--------------------------

Update after reading other reviewers' comments, the authors' comments, and considering the updated paper:

All three reviewers seem to have a similar view of the submitted work and agree in their rating. I think the updated paper and the authors' comments have addressed some of the concerns that the reviewers have raised. I think that the updated paper has improved in quality. On the other hand I think that some of the weaknesses still remain. Overall, I believe that the paper is close to the acceptance threshold. Seeing the improvements to the paper I am willing to raise my score to "marginally above the acceptance threshold".
summary_of_the_review: An interesting combination of existing ideas evaluated on three computer vision tasks. Overall novelty seems limited, the paper does not explain its approach very well, and the gained understanding is limited. Empirical results are interesting but not super-convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use the shift operation (Wu et al. CVPR 2018) in an axial manner for MLP-mixer architectures. The proposed method performs much better than previous MLP-based methods on ImageNet-1K and on par with Swin-transformer.
main_review: Strengths
1. The paper is well written and presented clearly.
2. The proposed axial shift module is simple and elegant.
3. Experiments are done extensively beyond ImageNet.

Weaknesses
1. Limited technical novelty given the fact that the shift operation (CVPR 2018) was already presented as an alternative (though similar) operation to convolution. The shift-ResNet is already spatial convolution free except the first stem layer. Now it is not surprising that using a convolution alternative in MLP-mixer can boost its performance to convolution level (i.e. better than previous MLP-mixers).
2. The modification of using “axial” on top of “shift” lacks justification or ablation. What if the original shift operation is applied to the nearby 8 points in the 3x3 window? The channels can be kept the same by replacing “horizontal shift” with 4 points and “vertical shift” with another 4 points. This should be very similar to the “5x5” configure used in the paper.
3. Another intuitive baseline is simply to use depth wise convolution with a depth multiplier of 2, instead of the 2 parallel axial shift module. Would this simple convolutional baseline affect performance/params/FLOPs/throughput by a large margin? Possibly not. If this simple convolutional baseline works, then what is the benefit of using AS-MLP-mixers?
4. Efficientnetv2 (ICML 2021), a representative convolution-based method is not cited/discussed/compared. For example, Table 11 of Efficientnetv2 presented V2-S with 8.8G FLOPs, 901 images/s, 83.6% top-1. This is better than AS-MLP-B, 15.2G FLOPs, 455.2 images/s. Although the throughput comparison is not 100% fair (Efficientnetn2 uses a larger batch size, FP16 inference, and SE module), I think AS-MLP-B can hardly perform better than Efficientnetv2 in a fair throughput comparison.
summary_of_the_review: I think this paper is around the acceptance threshold, mainly due to the limited technical novelty besides applying Shift, an alternative to convolution, to MLP-mixers, and achieving on par performance as previous transformer/convolution methods. I will raise my score if the weaknesses are addressed.
review_title: null

Meta-review: The paper proposes a MLP-based architecture that makes extensive use of the shift operation on the feature maps. The model performs well on several vision tasks and datasets.

The reviews are mixed even after the authors' response. Main pros are that the proposed architecture is elegant and reasonable, and the experimental evaluation is thorough and strong. The main con is that the novelty is somewhat limited to some prior papers.

Overall, I recommend acceptance. The reviewers point out that the architecture is good and the results are strong. Similarities to prior works do not seem serious enough to warrant rejection - even an author of arguably the most related (concurrent) works - S2-MLP and S2-MLPv2 - confirms that there is sufficient difference. Moreover, this is one of the first papers to show very strong results on detection and segmentation.
Decision: Accept (Poster)