Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new model architecture using LSTM for image classification. By adapting 2-dimensional LSTM (Bi-LSTM for vertical and horizontal directions) into the Transformer-like architecture, the model outperforms ViT-based and SOTA CNN-based architectures with less number of parameters. 


strengths_and_weaknesses: ## Strengths

1. The paper is clearly written. 

2. The paper proposes a simple yet effective framework using LSTM. The model outperforms transformer and CNN-based models for image classification. This work provides a great alternative to Transformer and CNNs for image classification. 

3. The proposed model is especially efficient for higher resolutions. 

## Weaknesses

1. Lack of related work

- There are a number of studies using multi-directional LSTM/RNN for vision tasks that are very relevant to this work e.g., [1-4]. The authors should cite and discuss the similarities and differences. 

- ReNet [69] is very relevant to this work. The authors pointed out that the major difference is to use a transformer-like block structure. However, the benefit of this structure and what it provides to the model compared to ReNet or other related works [1-4] are missing. 

2. Due to LSTM's sequence nature, LSTM-based models are not easily parallelizable, especially compared to transformer and CNN-based models. I see that throughput is much worse than other models. I assume training time would be especially slow. It is unclear to me how throughput improves with higher resolutions.

[1] "Multi-dimensional recurrent neural networks." ICANN 2007.

[2] "Pixel recurrent neural networks." ICML 2016.

[3] "Scene labeling with lstm recurrent neural networks." CVPR 2015.

[4] "Semantic Object Parsing with Local-Global Long Short-Term Memory" CVPR 2016




questions: 1. The authors mentioned that 'The higher the input resolution, the more memory-efficient and throughput-economical are on Sequencers' (Line 251-252 and Figure 4). I believe LSTM, in particular, 2D LSTM is not memory efficient and cannot be throughput-economical as it requires computing and saving the activation for all directions. Especially it should be worse for the higher resolutions. Could authors explain how sequencer becomes memory-efficient and throughput-economical? 

2. It would be great to discuss and address the weaknesses mentioned above during the rebuttal. 
limitations: The authors explained the limitations and potential negative societal impact of their work in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an architecture for image classification named Sequencer, which utilizes the BiLSTM module to replace the self-attention module in the vision transformer model. The BiLSTM module is further improved by processing the vertical and horizontal axes in parallel from top/botton and left/right directions. Experiments on image classification tasks demonstrate that the proposed method can acheive similar performance with existing classification models with similar number of paramters.
strengths_and_weaknesses: Strengths:

+ This paper is well-written. The idea is easy to understand. 

+ The proposed method is the first work to empirically show the effectiveness of LSTM modules in large scale image classification tasks, which would have a board impact in investigating the potential of LSTM-like architectures in the computer vision field.

+ Ablations and visualization results are rich, which present the validity of the proposed method in terms of the importance of each component.

Weaknesses:

- The novelty of the proposed method is limited. The proposed Sequencer replaces the self-attention module in ViT with the existing BiLSTM module. Besides, [r1] shows that the self-attention module in ViT can be replaced with a simple spatial pooling operator, which suggests that such replacement is incremental. 

- Although the proposed model can achieve similar performance with existing SOTA architecures, it requires much higher FLOPs and throughput as shown in Table 1. 

- Evaluation is only conducted on image classification. It would be better to evaluate the proposed architecture on more vision tasks such as detection and segmetation to show its generalization ability.

[r1] MetaFormer Is Actually What You Need for Vision. CVPR 2022.


questions: Please refer to the weaknesses part
limitations: The limitations are mainly about the limited novelty of the proposed method and the poor experimental results (much higher FLOPs, lack of experiments on other vision tasks).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new Sequencer architecture that replaces self-attention in ViT with BiLSTM(2D) for the image classification task. On ImageNet-1K dataset, Sequencer achieves better performance than current other similar scale models. The authors also show Sequencer is more robust to resolution variation and suffers from less severe accuracy degradation when the input resolution is increased.
strengths_and_weaknesses: Pros:
1. This paper makes an attempt to use LSTM, an unexplored inductive bias, to replace self-attention in ViT for image classification and shows its effectiveness. This line of research helps the community understand what is indeed essential for vision tasks.

2. Strong results and extensive experiments. It compares with a series of related works based on various inductive biases and shows that it has superior performance and transferability under a similar scale of parameters. Besides, ablation studies are conducted.

Cons:
1. The computational cost is too high. As shown in Table 1, under a similar scale of model parameters, Sequencer usually needs 2x FLOPs and is 2x~10x lower throughput compared with other methods. Although this is not surprising due to the recursion in LSTM, I am still concerned about the practicality of this model with such a high computational cost.

2. Lack of reasoning on how using LSTM captures the spatial information and why it is so effective. In BiLSTM2D, it uses LSTM to capture dependencies from horizontal and vertical patches respectively. From my point of view, this design should not be as effective as global dependencies in self-attention since you may need to involve patches that are not necessarily in the same horizontal and vertical line to understand the objects in the images. Besides, I am also curious about what role the memory in LSTM plays in processing spatial information. The above analysis is critical for readers to understand the model but is missing in the paper.

questions: 1. What is the running time of Sequencer and other baselines on ImageNet-1K? Are there any potential ways to reduce the computational cost, such as reducing the length of the sequence by downsampling? If so, can the authors report the model performance compared with the baselines under a similar scale of FLOPs and throughput?

2. See (2) of Cons. Can the authors provide more discussions and potential visualization on why Sequencer block is effective?

3. What if replace LSTM with other RNN cells, such as GRU? Will the model still work well? If not, which part is essential to the model performance?
limitations: The authors have discussed the limitations in the conclusion. Actually, it would be better if the authors can test the model's effectiveness on tasks that require sequence modeling such as video action recognition in the main paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed Sequencer by using deep LSTMs instead of self-attention for image classification. And many related works were compared in experiments to validate the performance of sequencer.
strengths_and_weaknesses: Strength: This paper proposed Sequencer, which uses LSTM instead of the self-attention for sequence modeling. This paper also proposed a two-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs to enhance performance. Experiments showed the advantages of Sequencers compared to the self-attention mechanism in transferability and resolution adaptability. The work is clearly stated, and the manuscript is well written.
Weakness: Some experimental results are not clearly explained.

questions: 1.	From Table 1, compared with ConvNetXt, Sequencer achieved worse performance at FLOPS and throughput with 0.2~0.3 top-1 accuracy increase. Meanwhile, the fine-tuned sequencer is worse than the fine-tuned ConvNetXt.
2.	In Figure 3, the performance between Sequencer and ConvNetXt seems to conflict with Table 1？
limitations: Some experimental results are not clearly explained.
review_title: null

Meta-review: metareview: Four reviewers provided detailed feedback on this paper. The authors responded to the reviews and I appreciate the authors' comments and clarifications, specifically that each question/comment is addressed in detail. The authors also uploaded a revised version of the paper.

After the two discussion periods, all four reviewers suggest to accept the paper (although the scores do not exceed a "weak accept"). After considering the reviewers' and authors' comments, I believe that the paper should be accepted to NeurIPS.

Weaknesses include:
* Some concerns about experimental results, e.g. highlighting accuracy vs. number of parameters but not also highlighting limitations when looking throughput (comparing only parameters (or FLOPS) can sometimes be misleading, see also [The efficiency misnomer, ICLR22](https://arxiv.org/abs/2110.12894)). But it's good that throughput numbers are presented in the paper and the paper acknowledges this limitation. Related: concerns about computational cost.
* Some concerns regarding relevant related literature (addressed in comments and revision) and novelty of the approach.
* Limitation to image classification only in the experiments (partially addressed in comments and revision).
* More interpretation of the effect of using LSTMs could be helpful to the reader (partially addressed in comments).

Strengths include:
* Interesting, conceptually simple approach that revisits LSTMs for images, which could be specifically useful for high resolution images.
* Reviewers agree that the paper is well-written.
* Experimental results and ablations are strong with respect to the claims made.

Minor points (not affecting this decision, but potentially useful to authors when preparing the final revision):
* MLP-based methods "cannot cope with flexible input sizes during inference" - I think this is only partially true, even the original MLP-Mixer paper shows how this can be solved e.g. in fine-tuning by "modifying the shape of Mixer’s token-mixing MLP blocks"
* minor typo I randomly encountered: Table 3, row 3, column "Flowers" 89.5 -> 98.5
* "It is demonstrated that modeling long-range dependencies by self-attention is not necessarily essential in computer vision" - To some degree similar "demonstrations" are visible in CNNs and MLP-Mixers, so this claim seems a bit strong, maybe?
confidence: Less certain
Decision: Accept