Sequencer: Deep LSTM for Image Classification
Authors: Yuki Tatsunami
Masato Taki
Venue: NeurIPS 2022 Accept (2022)
Keywords: computer vision
image classification
network architecture
long short-term memory

Abstract: In recent computer vision research, the advent of the Vision Transformer (ViT) has rapidly revolutionized various architectural design efforts: ViT achieved state-of-the-art image classification performance using self-attention found in natural language processing, and MLP-Mixer achieved competitive performance using simple multi-layer perceptrons. In contrast, several studies have also suggested that carefully redesigned convolutional neural networks (CNNs) can achieve advanced performance comparable to ViT without resorting to these new ideas. Against this background, there is growing interest in what inductive bias is suitable for computer vision. Here we propose Sequencer, a novel and competitive architecture alternative to ViT that provides a new perspective on these issues. Unlike ViTs, Sequencer models long-range dependencies using LSTMs rather than self-attention layers. We also propose a two-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs to enhance performance. Despite its simplicity, several experiments demonstrate that Sequencer performs impressively well: Sequencer2D-L, with 54M parameters, realizes 84.6% top-1 accuracy on only ImageNet-1K. Not only that, we show that it has good transferability and the robust resolution adaptability on double resolution-band. solution-band. Our source code is available at https://github.com/okojoalg/sequencer.

Sequencer: Deep LSTM for Image Classification
Yuki Tatsunami 1.2 Masato Taki
Rikkyo University, Tokyo, Japan 2 "ny AncTech Co., Ltd., Tokyo. Japan {y tatsunami, taki (iciimmm@mm mrikkyorkkkkk.....mmmm ac jp
Abstract
In recent computer vision research, the advent of the Vision Transformer (ViT) has rapidly revolutionized various architectural design efforts: ViT achieved state- of-the-art image classification performance using self-attention found in natural language processing, and MLP-Mixer achieved competitive performance using simple multi-layer perceptrons, In contrast, several studies have also suggested that carefully redesigned convolutional neural networks (CNNs) can achieve advanced performance comparable to VIT without resorting to these new ideas. Against this background, there is growing interest what inductive bias is suitable for computer vision. Here we propose Sequencer, novel and competitive architecture alternative to ViT that provides a new perspective on these issues. Unlike ViTs, Sequencer models long-range dependencies using LSTMs rather than self-attention layers. We also propose a woo-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs enhance performance. Despite its simplicity. several experiments demonstrate that Sequencer performs impressively well: Sequencer2D-L. with 54M parameters, realizes 84.6% top-1 accuracy on only ImageNet- K. Not only that, we show that it has good transfer- ability and the robust resolution adaptability double resolution-bans Our source code is available at pattps://giie //github. com/ calojaaggeueueeeee
Introduction
The de-facto standard for computer vision has been convolutional neural networks (CNNs)
22
65 66 | 29 677. However, inspired by the many breakthroughs in natural language processing (NLP)
ImageNet-1k Top-1 Acc. (9)
84
Bequunnce/2D--g
achieved by Transformers 75 35 57 applications of T'Tansffommrss for computer vision are now being actively studied. In particular, Vision Transformer (ViT) [I is a pure Transformer applied to image recognition and achieves performance competitive with CNNs. Various studies triggered by ViT have shown that the state-of-the-art (SOTA) performance can be achieved for wide range of vision tasks us-
82
Cycettu
79
Secuensartpurs
40
100
Number Parameters(M)
ing self-and attention alone 77 convolution.
15), without
Figure 1: IN-1K top-1 accuracy v.s. model parameters. All models are trained on IN-IK at resolution 224 from scratch.
The reason for this success is thought be due the ability of self-attention to model long-range depen- dencies. However, it is still unclear how essential the self-attention is to the effectiveness of Transformers for vision tasks. Indeed, the MLP-Mixer 77 based only on multi-layer perceptrons (MLPs) proposed as appealing alternative to Vision Trans-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

formers (ViTs). In addition, some studies 49 14 have shown that carefully designed CNNs are still competitive enough with Transformers in computer vision. Therefore, identifying which architectural designs are inherently effective for computer vision tasks is of great interest for current research [3 This paper provides a new perspective on this issue by proposing a novel and competitive alternative to these vision architectures.
We propose the Sequencer architecture, that uses the long short-term memory (LSTM) 27 rather than the self-attention for sequence modeling. The macro-architecture design of Sequencer follows ViTs, which iteratively applies token mixing and channel mixing, but the self-attention layer is replaced by one based on LSTMs. In particular, Sequencer uses bidirectional LSTM (BiLSTM) [3] as a building block. While simple BiLSTM shows a certain level of performance, Sequencer can be further improved by using ideas similar to Vision Permutator (ViP) [28]. The key idea in ViP is to process the vertical and horizontal axes in parallel. We also introduce two BiLS for top/bottom and left/right directions in parallel. This modification improves the efficiency and accuracy of Sequencer because this structure reduces the length of the sequence and yields a spatially meaningful receptive field.
When pre-trained on ImageNet-IK IK (IN- IK) dataset, our new attention-free architecture outperforms advanced architectures such as Swin 4 and ConvNeXt 49] of comparable size, see Figure [ It also outperforms other attention-free and CNN-free architectures such as MLP-Mixer [70 and GFNet [6],, making Sequencer an attractive new alternative to the self-attention mechanism in vision tasks.
This study also aims to propose novel architecture with practicality by employing LSTM for spatial pattern processing. Notably, Sequencer exhibits robust resolution adaptability, which strongly prevents accuracy degradation even when the input's resolution is increased double during inference. Moreover, fine-tuning Sequencer on high-resolution data can achieve higher accuracy than Swin-B 4 and Sequencer is also useful for semantic segmentation. On peak memory, Sequencer tends be more economical than ViTs and recent CNNs for high-resolution input. Although Sequencer requires more FLOPs than other models due to recursion, the higher resolution improves the relative efficiency of peak memory, enhancing the accuracy/cost trade-off at a high- nigh-resolution regime. Therefore, Sequencer also has attractive properties as a practical image recognition model.
2 Related works
Inspired by the success of Transformers in NLP 75 B3 57 58 3 60 various applications of self-attention have been studied in computer vision. For example, in iGPT 6 an attempt was made to apply autoregressive pre-training with causal self-attention (57] to image classification. However, due the computational cost of pixel-wise attention, it could only be applied to low-resolution images, and its ImageNet classification performance was significantly inferior to the SOTA. ViT [6), on the other hand, quickly brought Transformer's image classification performance closer SOTA with its idea of applying bidirectional self-attention architectural and training improvements
35 to image patches rather than pixels. Various
5 have been attempted for ViT [T In this paper, we do not improve self-attention itself but propose completely new module for image classification to replace it.
The extent to which attention-basse cross-token communication inherently contributes to ViT's success is not yet well understood, starting with MLP-Mixer (70), which completely replaced VIT's
self-attention with MLP, various MLP-based architectures [71 46 competitive performance
28
69
[3] have achieved 68
the ImageNet dataset. We refer to these architectures as global MI.s (GMLPs)) because they have global receptive fields. This series of studies cast doubt on the need for self-attention. From a practical standpoint, however, these MLP-based models have a drawback: they need to be finetuned to cope with flexible input sizes during inference by modifying the shape of their token-mixing MLP blocks. This resolution adaptability problem has been improved in CycleMLP 7, for example, by the idea of realizing local kernel with a cyclic MLP. There are similar ideas such as 22 Besides the MLP-based idea, several other interesting self-attention alternatives have been found. GFNet 61 uses Fourier transformation of the tokens and mixes the tokens by global filtering in the frequency domain. PoolFormer [3]. on the other hand, achieved competitive performance with only local pooling of tokens, demonstrating that simple local operations are also a suitable alternative. Our proposed Sequencer is new alternative to self-attention that differs from both of the above, and
21 which are collectively referred as local MLPs (LMLPs).

---

Sequencer is an attempt to realize token mixing in vision architectures using only LSTM. It achieved competitive performance with SOTA on the IN-1K benchmark, especially with an architecture that can flexibly adapt to higher resolution.
The idea of spatial axis decomposition has heen used several times in neural architecture in computer vision. For example, SqueezeNeXi 17 decomposes 3x3 convolution layer into Ix3 and 3x1 convo- lution layers, resulting in a lightweight model. Criss-cross attention [3] reduces memory usage and computational complexity by restricting the attention to only vertical and horizontal portions. Current architectures such as CSwin [5], Couplformer
40 ViP 28 RaftMLP 69), SparseMLP
68), and
MorphMLP [6] have included similar ideas to improve efficiency and performance.
In the early days of deep learning, there were attempts use RNNs for image recognition. The earliest study that applied RNNs to image recognition is 19]. The primary difference between our study and [99 is that we utilize usual RNN in place of a --multi-dimensional RNN(2MDRNN), The 2MDRNN requires H + W sequential operations; The LSTM requires H sequential operations, where H. and W are height and width, respectively. For subsequent work on image recognition using 2MDRNNs, see 120
4 proposed an architecture in which information is collected from four directions (upper left, lower left, upper right, and lower right) by RNNs for understanding natural scene images. 4 proposed a novel 2MDRNN for semantic object parsing that integrates global and local context information, called LG-LSTM. The overall architecture design structured input deep ConvNet features into the LG-LSTM, unlike Sequencer which stacks LSTMs. ReNet [77]] is most relevant to our work; ReNet [77 uses a 4-way LSTM and non-overlapping patches as input. In this respect, it is similar to Sequencer. Meanwhile, there are three differences. First, Sequencer is the first MetaFormer S realized by adopting LSTTM as the token mixing block. Sequencer also adopts a larger patch size than ReNet 777. The benefit of adopting these designs is that we can modernize LSTM-based vision architectures and fairly compare LSTM-based models with ViT. As a result, our results provide further evidence for the extremely interesting hypothesis MetaFormer [8] Second, the way vertical BiLSTMs and horizontal BiLSTMs are connected is different. Our work connects them in parallel, allowing us gather vertical and horizontal information simultaneously. On the other hand, in ReNet [77], the output of the horizontal BiLSTM is used as input to the vertical BiLSTM. Finally, limited to small datasets as MNIST 41. CIFAR- 10 B8. and SVHN [54]. and has not shown the effectiveness of LSTM for larger datasets. ReSeg 77 applied ReNet semantic segmentation. RNNs have been applied not only to image recognition, but also to generative models: PixceIRNN (74] is pixel-channel autoregressive generative model of images using Row RNN, which consists of ID-convolution and a usual RNN, and Diagonal BiLSTM, which is computationally expensive.
32
E 3
trained Sequencer large datasets such as ImageNet, whereas ReNet [27 is
In NLP, attempts have been made to avoid the computational cost of attention by approximating causal self-attention with recurrent neural network (RNN) KE or replacing it with RNN after training 3 In particular, in [34], autoregressive pixel-wise image generation task is experimented with an architecture where the attentions in iGPT are approximated by RNNs. These studies are specific to unidirectional Transformers, in contrast to our baseed Sequencer which is the bidirectional analog of them.
3 Method
In this section, we briefly recap the preliminary background on LSTM and further describe the details of the proposed architectures.
3.1 Preliminaries: Long short-term memory
LSTM 2 is specialized recurrent neural network (RNN) for modeling long-term dependencies of
sequences. Plain LSTM has an input gate i, that controls the storage of inputs, forget gate that controls the forgetting of the former cell state G-1 and an output gate o, that controls the cell output hy from the current cell state et. Plain LSTM is formulated follows:
i ( We + Wither + b,), fc (W,, 22 + hi.hh-111
(1)
G = ft 0--1- + is @ tanh (Wao + Whehe-1 + be), or = a (Wao + Whohe-1 + bo), (2)
h o tanh(c:),
where a is the logistic sigmoid function and @ is Hadamard product.
3

---

Dedrect orai LETT
Aidincdirall LSIN
Recovard
Rewardd LETM
Pach Vtegin
Oequrter Musa
Dequeter Stckk
MV Lanoa
Pequtierr Mick
Cass
(a) Sequencer
(b) BiLSTM2D layer
Marm
Korm
Anaentonn
BLSTH
Morm
Nom
korm
Inpst
Inout
Irputt
(c) Transformer block
(d) Vanilla
(e) Sequencer2D black
Sequencer block
Figure 2: a The architecture Sequencers: bb) The figure outlines the BiLSTM2D layer, which
the main component of Sequence22D. (c) Transformer block consist of multi-head attention. In contrast, ( Vanilla Sequencer block and (
Sequencer2D block, utilized on our archtecture, composed of BiLSTM or BiLSTM2D instead of multi-head attention.
BiLSTM [3 is profitable for sequences where mutual dependencies are expected. A BiLSTM consists two plain LSTMs. Let 7 be the input series and Fr the rearvangement of 7' in reverse order. Mae and Aback are the outputs obtained by processing Z and ' with the corresponding LSTMs, respectively. Let bback be the output bback rearranged in the original order, and the output of BiLSTM is obtained as follows:
hfor Rback = LSTM (z). LSTMMack (F) h concatenate(hior hback)
(4)
Assume that both hfor and bback have the same hidden dimension D, which is hyperparameter of BiLSTM. Accordingly, vector h has dimension 2D.
3.2
Sequencer architecture
Overall architecture In the last few years, ViT and its many variants based on self-attention
16
72
48
have attracted much attention computer vision. Following these, several works 700
46
28
have been proposed to replace self-attention with MLP. There have also been studies of
replacing self-attention with a hard local induced bias module
[ and with global filter [6]
using the fast Fourier transform algorithm (FFT) [1O]. This paper continues this trend and attempts to replace the self-attention layer with L.TTM 27 we propose a new architecture aiming memory saving by mixing spatial information with LSTM, which is memory-economical compared to ViT, parameter-saving. and has the ability to learn long-range dependencies.
Figure 2a shows the overall structure of Sequencer architecture. Sequencer architecture takes non- overlapping patches as input and projects them onto the feature map. Sequencer block, which is core component of Sequencer, consists of the following sub-components: (1) BiLSTM layer can mix spatial information more ceanorr-eccnomicaal and more globally than CNN. (2) MLP for Mtannel-mixing called Vanilla Sequencer block when plain BiLSTM layers are used as BiLSTM layers as Figure 2d and Sequencer2D block when Bil STM2D layers are used Figure 2c We define BiLSTM2D layer later. The output of the last block sent the linear classifier via global average pooling layer, in most other architectures.
for high-resolution images than Transformer layer
well as [16] 77 Sequencer block is

---

BiLSTM2D layer We propose the BiLSTM2D layer as technique to mix 2D spatial information efficaciously. It has two plain BiLSTMs: vertical BiLSTM and a horizontal one. For an input X E R XC (X.. E H w=1 is viewed as set of sequences, where H is the number of tokens in the vertical direction, W is the number of sequences in the horizontal direction, and C' the channel dimension. All sequences X w,; are input into the vertical BiLSTM with shared weights and hidden dimension D:
H ver M,
BiLSTM((X, ......)
(5)
In very similar manner. (X E IR W Xa,g are imput into the horizontal BiLSTM with shared weights and hidden dimension D as well:
M=l is viewed as set of sequences, and all sequences
H Hhor BiLSTM((X..))
We combine (H E I " ===1 into Her E IR 2xxxx2 and (H E R #
D==l into
Hhor R They are then concatenated and processed point-wisely in fully-connection layer. These processes are formulated as follows:
H concatenate(H .ooo), X = FCCHH,,
where FC(.) denotes the cully-connected layer with weight W E gCx4D, The PyTorch-like pseudocode shown in Appendix B.I
BiLSTM2D is more nemory-eeconomen and throughput-effciienyy than multi-head-ene -eeddatttnntoon of VIT for high-resolution imput. BiLSTM2D involves (WC'+ + HC)/2 dimensional cell states, while multi-head-attention involves & (HW)² dimensional attention map where h is a number of heads. Thus, H and W increase, the memory cost of an attention map increases more rapid than the cost of a cell state. On throughput, the computational complexity of self-attention is (WAC)). whereas the computational complexity of BiLSTM is (() where we assume W И for simplicity. There are (W)) sequential operations for BiLSTM2D. Therefore, assuming we use a sufficiently efficient LSTM cell implementation, such as official PyTorch LSTMs are using, the increase of the complexity of self-attention is much more rapid than BiLSTM2D, It implies a lower throughput of attention compared to BiLSTM2D. See an experiment in Section
Architecture variants For comparison between models of different depths consisting of Se- quencer2D blocks, we have prepared three models with different depths: 18, 24, and 36. The names of the models are Sequencer2D-S, Sequencer2D-M, and Sequencer2D-L, respectively. The hidden dimension is set to C/4. Details of these models are provided in Appendix B.2
As shown in subsection 4.1 these architectures outperform typical models. Interestingly, however, subsection 3 shows that replacing Sequencer2D block with the simpler Vanilla Sequencer block maintains moderate accuracy. We denote such model as Vanilla Sequencer. Note that some of the explicit positional information lost in the Vanilla Sequencer because the model treats patches as 1D) sequence.
4 Experiments
In this section, we compare Sequencers with previous studies on the IN-1K benchmark 33 carry out ablation studies, transfer learning studies, and analysis the results demonstrate the effectiveness of Sequencers. We adopt PyTorch |56. and timm [8 library to implement models in the conduct all experiments. See Appendix for more setup details.
We also
4.1 Scratch training on IN-1K
We utilize IN-IK 39 which has 1000 classes and contains 1,281,167 training images and 50,000 validation images. We adopt Adam V optimizer [50]. Following the previous study 72 we adopt the base learning rate hatch 512
x 10-4 The batch sizes for Sequencer2--S, Sequencer2--M, and Sequencer2D-L. are 2048, 1536, and 1024, respectively. As a regularization method, stochastic depth [30] and label smoothing [66] are employed. As data augmentation methods, mixup [8]]
cutout 1p
cutmix [55] random erasing [ and randaugment [[] are applied.
5

---

Table 1: The table shows the top-1 accuracy when trained on IN-IK, comparing our model with other similar scale representative models. Training and inference throughput and their peak memory were measured with images per batch on a single V100 GPU. The left sides of the slashes are values during training, and the right sides of the slashes are values during inference. Fine-tuned models marked with "1" Note Sequeneer2D-I.f are compared Swin-B and ConvNeXt-Br with more parameters since Swin and ConvNeXt have not fine-tuned models of similar parameters with Sequencer2D-L1 the original papers.
Throughput (image/s)
Peak Mem.
Top-1 Acc.(%)
Pre FT Top-1 Ac).(%)
Model
Family Res. #Param. FLOPS
(MB)
Training from scratch
RegNetY-44F 59
CNN 224
21M
4.0G
228/823
1136/225
80.0
non-fine- tune
ConvNeXt-T
CNN 224
29M
4.5G
337/1124
1418/248
82.1
as above
DeiT-S 12
Trans. 224
22M
4.6G
480/1569
1195/180
79.9
as above
Swin-T
Trans. 224
4.5G
268/894
1613/308
81.2
as above
ViP-S/7 2X
GMLP 224
25M
6.9G
214/702
1587/195
81.5
as above
CyclcMLP-B2
LMLP 224
27M
3,9G
158/586
1357/234
81.6
as above
PoolFormer-S24
83
LMLP 224
21M
3.6G
313/988
1461/183
80.3
as above
Sequencer2D-S
Seq. 224
28M
8.4G
110/347
1799/196
82.3
above as above
RegNetY-8GF 59
CNN 224
39M
8.0G
211/751
1776/333
81.7
12T-ViT;-19 X
Trans. 224
39M
9.8G
197/654
3520/1 140
82.2
as above
CyclcMI.P-B3
LMLP 224
38M
6.9G
100/367
2326/287
82.6
above
PoolFormer-S36
[3
LMLP 224
31M
5.2G
213/673
2187/220
81.4
as above
GFNet-H-S [1
FFT 224
32M
4.5G
227/755
1740/282
81.5
above
Sequenccr21D-M
Seq. 224
38M
11.1G
83/270
2311/244
82.8
as above
RegNetY-12GF
59
CNN 224
46M
12.0G
199/695
2181/440
82.4
as above
ConvNeXt-S 44
CNN 224
50M
8.7G
212/717
2265/341
83.1
above as above
Swin-S +
Trans. 224
50M
8.7G
165/566
2635/390
83.2
Mixer-B/16
70
GMLP 224
59M
12.7G
338/1011
1864/407
76.4
as above
ViP-M/7 AX
GMLP 224
16.3G
130/395
3095/396
82.7
as above
CyclcMLP-BH
7
LMLP 224
52M
10.1G
70/259
3272/338
83.0
as above
PoolFormer-M31
33
LMLP 224
56M
9.1G
171/496
3191/368
82.1
as above
GFNet-HH-B 61
FFT 224
54M
8.4G
144/482
2776/367
82.9
as above
Sequencer2D-L
Seq.
224
54M
16.6G
54/173
3516/322
83.4
above
Fine-tuning
ConvNeXI-Bt
49
CNN 3842
89M
45.1G
78/234
7329/870
85.1(+1.3)
83.8
Swin-B 43
Trans.
384
88M
47.1G
54/156
12933/1532 84.5(+1.0)
83.5
GFNet-B t
61
FFT 384
47M
23.2G
137/390
3710/416
82.1(+0.8)
82.9
Sequencer2D-Lt
Seq. 3922
54M
50.7G
26/84
9062/481
84.6(+1.2)
83,4
T the results that are comparing the proposed models to others with a comparable number of parameters models, including models with local and global receptive fields such CNNs, ViTs, and MLP-based and FFF- models. Sequencers has the disadvantage that its throughput slower than other models because it uses RNNs. In the scratch training on IN-1K, however, they outperform these recent comparative models in accuracy across their parameter bands. In particular, Seqeuncer2D-L. is competitive with recently discussed models with comparable parameters such as ConvNeXt-S 20 and Swin-S [48]. with accuracy outperformance of 0.3% and 0.2%, respectively.
Table demonstrates that Sequencer's throughput is not good. The training throughput is about three times the inference throughput for all these models. Compared to other models, both measured inference and training time are not good.
4.2 Fine-tuning on IN-1K
In this fine-tuning study, Sequencer2D-LL pre-trained on IN-1K at 224 resolution is fine-tuned on IN- IK 3 resolution. We compare it with the other models fine-tuned IN-IK at 384

---

Table 2: Sequencer ablation experiments. We adopt Sequencer2D-S variant for these ablation studies. C1 denotes vertical BiLSTM, C2 denotes horizontal BiLSTM, and C3 denotes channel fusion component. When vertical BiL.TMM only, horizontal BiL.TMM only or unidirectional BiL.STM2D, its hidden dimension needs to be doubled from the original setting because it compensates the output dimension for the excluded LSTM and matches the dimensions.
(a) Components
(b) LSTM Direction
(c) Vanilla Sequencer
C1
C2
C3
Acc.
Bidirectional
Acc. 79.7
Model
#Params,
FLOPs
Acc.
75.6
VSequencer-S
33M
8.4G
78.0
82.3
VSequencer(HD-S
28M
8.4G
81.6
VScquencen(PE)-S
33M
8.4G
78.1
82.3
Sequencer2D-S
28M
8,4G
82.3
(d) Hidden dimension
(e) Various RNNs
Hidden dim. ratio #Params.
FLOPs
Ace.
Model
#Parrms.
FLOPs
Ace.
1x
28M
8.4G
82.3
RNN- SeN-eequencr)]
19M
5.8G
80.6
2x
4SM
13.9G
82.6
GRU-Sequencer2D
25M
7.5G
82.3
Seqeucer2D-S
28M
8.4G
82.3
resolution. Since 392 is divisible by 14, the input at this resolution can split into patches without padding. However, note that this is not the case with resolution of 384².
As Table I indicates, even when higher-resolution Sequencer fine-tuned, competitive with the latest models such as ConvNeXt 49 Swin [48] and GFNet [1].
4.3 Ablation studies
This subsection presents ablation studies based Sequencer2D-S for further understanding of Sequencer. We seek to clarify the effectiveness and validity of the Sequencers architecture in terms of the importance of each component, bidirectional necessaries, setting of the hidden dimension, and the comparison with simple BiLSTM.
We show where and how relevant the components of BiLSTM2D are: The BiLSTM2D is composed of vertical BiLSTM, horizontal BiLSTM, and channel fusion elements. We want to see the validity of vertical BiLSTM, horizontal BiLSTM, and channel fusion. For this purpose, examine the removal of channel fusion and vertical or horizontal BiLSTM. Table 2a shows the results. Removing channel fusion shows that the performance degrades from 82.3%
81.6%. Furthermore, the additional removal of vertical or horizontal BiLSTM exposes a 6.0% or 6.6% performance drop, respectively. Hence, each component discussed here is necessary for Sequencer2D.
We show that the bidirectionality for BiLSTM2D is important for Sequencer. We compare Sequencer2D-S with a version that replaces the vertical and horizontal BiLSTMs with vertical and horizontal unidirectional LSTMs. Table 26 shows that the unidirectional model 2.6% less accurate than the bidirectional model. This result attests to the significance of using not unidirectional LSTM but BiLSTM.
It is important set the hidden dimension of LSTM to reasonable size. As described in subsection 3.2 Sequencer2D sets the hidden dimension of BiL.STM to C/4, but this not necessary the model has channel fusions. Table 2d compares Sequencer2D-S with the model with increased D. Although accuracy is 0.3% improved, FLOPs increase by 65%, and the number of parameters increases 60%. Namely, the accuracy has not improved for the increase in FLOPs. Moreover, the increase in dimension causes overfitting, which is discussed in Appendix
Vanilla Sequencer can also achieve accuracy that outperforms MLP- 70 but is not as accu- rate as Sequencer2D. Following experimental result supports the claim. We experiment with the Sequencer2D-$ variants, where Vanilla Sequencer blocks replace the Sequencer2D blocks, called VSequencer-Ste S(H), with incomplete positional information. addition, we experiment with a variant of VSequencer--S(H) without the hierarchical structure, which we call VSequencer-S. VSequencer- S(PE) is VSequencer-S using ViTs-style learned positional emhedding (PE) [16]. Table 2c indicates effectiveness for combination of LSTM and ViTs-like architecture. Surprisingly, even with Vanilla
7

---

Table 3: Left. Results on transfer learning. We transfer models trained on IN-IK datasets from
different domains. Sequencers use 224 resolution images, while ViT-B/16 and EfficientNet -B work on higher resolution, see Res. column. Right. Semantic segmentation results on ADE20K
models are Semantic FPN (36] based. We show mloU for the ADE20K validation set.
Model
Res. #Pr.
FI.OPs CF1o
CF10o
Flowers
Cars
Model
#Pr.
mloU
ResNet50 [22
224
26M
4.1G
96.2
90.0
PVT-Small [79]
28M
39.8
EN-B7 77
600 26M
37.0G
98.9
91.7
98.8
94.7
PoolFormer-S22 S24
83
23M
40.3
ViT-B/16 [6]
3843
86M
55.4G
98.1
87.1
89.5
Sequencer2D-S
32M
46.1
DeiT-B 12]
224
86M
17.5G
99.1
90.8
98.4
92.1
PVT-Medium
48M
41.6
CañT-S-36 7X
224
68M
13.9G
99.2
92.2
98.8
93.5
PoolFormer-S36 33
35M
42.0
ResMLP-22
71
224
30M
6.0G
98.7
89.5
97.9
89.5
Sequencer2D-M
42M
47.3
GFNet-H-B
61]
224
54M
8.6G
99.0
90.3
98.8
93.2
PVT-Large [79
63M
42.1
PoolFormer-M36
83
60M
42.4
Sequencer2D-S
224
28M
8.4G
99.0
90.6
98.2
93.1
Sequencer2D-L.
58M
48.6
Scqueneer2D-M
224
38M
11.1G
99.1
90.8
98.2
93.3
Soquencer2D-L
224 54M
16.6G
99.1
91.2
98.6
93.1
Sequencer and Vanilla Sequencer(H) without PE, the performance reduction from Sequencer2D-S is only 4.3% and 3.5%, respectively. According to these results, there is no doubt that Vanilla Sequencer using BiLSTMs is significant enough, although not accurate Sequencer2D.
All LSTMs in the BiLSTM2D layer can be replaced with other recurrent networks such as gated recurrent units (GRUs) & tanh-RNNs define BiGRU2D layer or BiRNN2I layer. We also trained these models on IN-1K, so see Table 2e for the results. The table suggests that all these variants, including RNN-cell, work well. Also, tanh-RNN performs slightly worse than others, probably due to its lower ability to model long-range dependence.
4.4 Transfer learning and semantic segmentation
Sequencers perform well on IN-IK, and they have good transferability. In other words, they have satisfactory generalization performance for a new domain, which shown below. We utilize the commonly used CIFAR-10 38], CIFAR- 1-0 this experiment. See the references and Appendix B.4 for details the datasets. The results of the
38 Flowers-102 [55], and Stanford Cars (37] for
proposed model and the results in previous studies of models with comparable capacity are presented Table particular, Sequencer2D-L achieves results that are competitive with CaiT-S-36 and EfficientNet-B7 67].
We experiment for semantic segmentation ADE20K 8 dataset. See Appendix
C.4 for details
with similar parameters, compared the setup. Sequencer outperforms PVT [79] and PoolFormer 3 to PoolFormer, mloU is about pts higher.
We have investigated commonly object detection model with Sequencer as the backbone. Its performance is not much different from the case of ResNet 2 future work. See Appendix
backbone. Its improvement is the
4.5 Analysis and visualization
In this subsection, we investigate the properties of Sequencer in terms of resolution adaptability and efficiency. Furthermore, effective receptive field (ERF) [51] and visualization of the hidden states provides insight into the question how Sequencer recognizes images.
One of the attractive properties of Sequencer is its flexible adaptability to the resolution, with minimal impact on accuracy even when the resolution of the input image varied from one-half to twice. In comparison, architectures like MLP-Mixer [70] have fixed input resolution, and GFNet [61] requires interpolation of weights in the Fourier domain when inputting images with resolution different from training images. We evaluate the resolution adaptability of models comparatively by inputting different resolution images each model, without fine-tuning, with pre-trained weights on IN-1K at the resolution of 2242. Figure 3a compares absolute top-1 accuracy on IN-1K, and Figure 3b compares relative one to the input image with the resolution of 224 By increasing the resolution by 28 for Sequencer2D-S and by 32 for other models, we avoid padding and prevent

---

the effect of padding on accuracy. Compared to DeiT-S 722,, GFNet-S [61, CycleMLP-BI 7 and ConvNeXt-T 49 Sequencer-''s performance is more sustainable. The relative accuracy is consistently better than ConvNeXt [9 which is influential in the lower-resolution band, and, at 448² resolution, 0.6% higher than CycleML. 7 which influential in the double-resolution band. It is noteworthy that Sequencer continues to maintain high accuracy on double resolution.
The higher the input resolution, the higher memory-efficiency and throughput of Sequencers when compared to DeiT
Figure E shows the efficiency of Sequencer2D-S when compared to DeiT-S Memory consumption increases rapidly in DeiT-S and ConvNeXt-T with increasing input resolution, but more gradual increase in Sequencer2D-S The result strongly implies that it has more practical potential as the resolution increases than the ViTs. At a resolution of 224 it is behind DeiT in throughput, but it stands ahead of DeiT when images with resolution of 8962 are input.
722
and -t--
49
ImageNet-1k Top-1 Acc.0%)
95
AlmageNet-1k Top-1 Acc.
80
75
70
Grvend
Connuttol
Eewurenrdo-ooo Sound
85
410
128
224
220
416
Resalution
Resalution
(a) Absolute top-1 acc.
(b) Relative top-1 acc. to 224 res.
Pesk Memoyying
Convlestc
Cop wwp.mredt
2016
2010 2040
Resolucion
Rosolution
(c) GPU peak memory
(d) Throughput
Figure 3: Top. Resolution adaptability. Every model is trained 224 resolution and evaluated at various resolutions with no fine- tuning. Bottom. Comparisons among Sequencer2ID-S, DeiT-S (72), and ConvNeXI-T 49 in ( GPU peak memory and ( throughput for different input image resolutions. Measured for each increment of 224 resolution, points not plotted are when GPU memory is ex- hausted. The measurements founded batch size of 16 and single V100.
Figure 4: Part of states the last BiLSTM2D layer in the Sequencer block stage 1. From top to bottom: out- puts of ver-LSTM, hor- LSTM, and ch-fusions and original images.
In general, CNNs have localized, layer-by-layer expanding receptive fields, and ViTs without shifted windows capture global dependencies, working the self-attention mechanism. In contrast, in the case of Sequencer,
not clear how information is processed in Sequencer block. We calculated ERF 51 for ResNet-50 [22). DeïT-S [72), and Sequencer2D-S as shown in Figure 5 ERFs of Sequencer2D-$ form a cruciform shape in all layers. The trend distinguishes from well-known models such as DeiT-S and ResNet-50. More remarkably, in shallow layers, Sequencer2D-S has wider ERF than ResNet-50, although not as wide as DeiT. This observation confirms that LSTMs in Sequencer model long-term dependencies as expected and that Sequencer recognizes sufficiently long vertical or horizontal regions. Thus, can be argued that Sequencer recognizes an image in very different way than CNNs ViTs. For more details on ERF and additional visualization. see Appendix D
Moreover we also visualized hidden state of vertical and horizontal BiLSTM, and feature map after channel fusion, and the results are visualized in Figure It demonstrates that our Sequencer has the hidden states interact with each other over the vertical and horizontal directions. The closer tokens are position, the stronger their interaction tends to be; the farther tokens are position, the less their interaction tends to be.

---

(a) R.550ffrstt (b) DeiTS/first
(c) SeqS/first
(d) RN50/last
(c) DeiTS//ast
(f) SeqS/last
Figure The visualizations are the ERFs of Sequencer2D-S and comparative models such ResNet- 50 and DeiT-S. The left of the slash denotes the model name, and the right of the slash denotes the location of the block output used to generate the ERFs. The ERFs are rescaled from 0 to 1. The brighter and more influential the region is, the closer to and the darker, the closer to 0.
5 Conclusions
We propose novel and simple architecture that leverages LSTM for computer vision. It demon- strated that new modeling with LSTM instead of the self-attention layer can achieve competitive performance with current state-of-the-ard models. Our experiments show that Sequencer has good memooryresource/cccurag and parameter/accuracy tradeoffs, comparable to the main existing meth- ods. Despite the impact of recursion on throughput, we have demonstrated benefits over it. We believe that these results raise number of interesting issues. Improving Sequencer's poor throughput one example. Moreover, we expect that investigating the internal mechanisms our model using methods other than ERF will further would be important to analyze in more detail the features learned by Sequencer in comparison to other architectures. We hope this will lead to a better understanding of the role of various inductive biases in computer vision. Furthermore, we expect that our results trigger further study beyond the domain research area. Especially, it would be very interesting open question to see if such a design works with time-series data in vision such as video or in a combined with another modality such video with audio.
understanding of how this architecture works. In addition,
problem setting
Acknowledgments and Disclosure of Funding
Our colleagues at AnyTech Co., Ltd. provided valuable comments on the early versions and encour- agement. We thank them for their cooperation. In particular, We thank Atsushi Fukuda for organizing discussion opportunities. We also thank people who support us, belonging to Graduate School of Artificial Intelligence and Science, Rikkyo University.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In NeurlPS, 2016.
[2] Lucas Beyer. Olivier Hénaff. Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint wrXiv:2006.07155 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. In NeurlPS, volume 33, pages 1877-1901, 2020.
[4] Wonmin Byeon, Thomas M Breuel, Federico Raue, and Marcus Liwicki. Scene labeling with Istm recurrent neural networks. In CVPR, pages 3547-3555, 2015.
[5] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In /CCV, pages 357-366, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, pages 1691-1703, PMLR, 2020.
[7] Shoufa Chen, Faze Xie, Chongjian GF, Runjian Chen, Ding Liang, and Ping Luo. CycleMLP: A MI.P-ike architecture for dense prediction. In ICLR, 2022.
[8] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder -ncoder-decoder appreaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, 2014.
10

---

[9] François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pages 1251- 2017.
[10] James W Cooley and John W Tukey, An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19(90):297-300 1965.
[11] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. RandAugment: Practical automated data augmentation with a reduced search space, In CVPRW, pages 702-703, 2020.
[12] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1rXiv::708.0444 2017.
[13] Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmipnet: Hierarchi- cal vision mlp with re- re-parameterized locality. In CVPR, 2022.
[14] Xiaohan Ding, Xiangyu Zhang. Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, 2022.
[15] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICIR, 2021.
[17] Amir Gholami, Kiscok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, and Kurt Keutzer. Squeezenext: Hardware-aware neural network design. In CVPRW, pages 1638-1647, 2018.
[18] Ian J Gondfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamnessing adversarial examples. In ICLR, 2015.
[19] Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural networks, pages 549-558. Springer, 2007.
[20] Alex Graves and Jürgen Schmidhuber. Offfine handwriting recognition with multidimensional recurrent neural networks. In NeurlPS, volume 21, 2008
[21] Jianyuan Guo, Ychui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mlp via hicrarchical rearrangement. arXiv preprint arXiv: 108. 13341, 2021.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[23] Dan Hendryckss Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340-8349, 2021.
[24] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2018.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), arXiv preprint arXiv: 1606.0845.. 2016.
[26] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262-15271, 2021.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 18):17355 1997.
[28] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator:
A permutable mip-like architecture for visual recognition. IEEÉ TPAMI, 2022.
[29] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[30] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, pages 646-66) 661, 2016.

---

[31] Zilong Huang. Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Conet: Criss-cross attention for semantic segmentation. In ICCV, pages 603-612, 2019.
[32] Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. In In Proceedings of the IEEE Workshop on Spoken Language Technology. 2015.
[33] Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into mns. In EMNLP, pages 10630-10643, 2021.
[34] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleurct. Transformers mns: Fast autoregressive transformers with linear attention. In ICML, pages 5156-5165. PMLR, 2020.
[35] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova, Bert: Pre-training of deep hidirectional transformers for language understanding. In NAACI-HI.T, pages 4171-4186, 2019.
[36] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, pages 6399-6408, 2019.
[37] Jonathan Krause, Michael Stark. Jia Deng. and Li Fei-Fei. 3D object representations for fine- graigga categorization. In ICCVW, pages 554-561, 2013.
[38] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[39] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, Imagenet classification with deep convolutional neural networks. In NeurIPS, volume 25, pages 1097-1105, 2012.
[40] Hai Lan, Xihao Wang, and Xian Wei. Couplformer: Rethinking vision transformer with coupling attention map. arXiv preprint arXiv:2112 05455 2021.
[41] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2788-2324, 1998.
[42] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. In ICLR, 2022.
[43] Xiaodan I. Liang, Xiaohui Shen, Donglai Xiang, Jiashi Feng, Liang Lin, and Shuicheng Yan. Semantic object parsing with local-global long short-term memory. In CVPR, pages 3185-3193, 2016.
[44] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object 2017.
detection. In ICCV, pages
[45] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[46] Hanxiao Liu, Zihang Dai, David So, and Quoc Le. Pay attention to mlps. In NewrlPS, volume 34, 2021.
[47] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xic, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
[48] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[49] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022.
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[51] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In NeurlPS, volume 29, 2016.
[52] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
[53] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li. Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. arXiv preprint arXiv:2105.0790 2021.
[54] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS Workshop, 2011.
12

---

[55] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pages 722-729, 2008
[56] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Newr/PS, volume 32, 2019.
[57] Alcc Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018.
[58] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAI blog, 1(8)9, 2019.
[59] lija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pages 10428-10436, 2020.
[60]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 21:1-65 66, 2020.
[61] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou, Global filter networks for image classification. In NeurlPS, volume 34, 2021.
[62] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenet? In ICML, pages 5389-5400. PMLR, 2019.
[63] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE TSP, 45(1 2681, 1997.
[64] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
[65] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.
[66] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, pages 2818-2826, 2016.
[67] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114, 14, 2019.
[68] Chuanxin Tang. Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, and Wenjun Zeng. Sparse mlp for image recognition: Is self-attention really necessary? arXiv preprint axXiv:2109005422 2021.
[69] Yuki Tatsunami and Masato Taki. Raftmlp: How much can be done without attention and with less spatial locality? arXiv preprint 108::2108.043844 2021.
[70] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. In NeurIPS, volume 34, 2021.
[71] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnacve, Jakob Verbeek, and Hervé Jégou. Resmlp: Foedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.0344 2021.
[72] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In JCML, 2021.
[73] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pages 32 42.42.22 2021.
[74] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, volume 30, 2017.
13

---

[76] Francesco Visin, Marco Ciccone, Adriana Romero, Kyle Kastner, Kyunghyun Cho, Yoshua Bengio, Matteo Matteucci, and Aaron Courville. Reseg: A recurrent neural network-based model for semantic segmentation. In CVPRW, pages 41-48, 2016.
[77] Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, and Yoshua Ben- gio. Renet: recurrent neural network based alternative convolutional networks. arXiv preprint arXiv:1505.00394 2015.
[78] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurlPS, volume 32, 2019.
[79] Wenhai Wang, Enze Xie, Xiang Li, Deng- Leng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[80] Ross Wightman. Pytorch image models. 2019.
[81] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S--mlpv: Improved spatial-shift mip architecture for vision. arXiv preprint arXiv:2108.01072 2021.
[82] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mip: Spatial-shift mlp architecture for vision. In WACV, pages 297-306, 2022.
[83]
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[84] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, pages 558 567, 2021.
[85] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, pages 6023-6032, 2019.
[86] David Junhao Zhang, Kunchang Li. Yunpeng Chen, Yali Wang. Shashwat Chandra, Yu Qiao, Luoqi Liu, and Mike Zheng Shou. Morphmlp: A self-attention free, mlp-like backbone for image and video. arXiv preprint ar 111i::2111.12227, 2021.
[87] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
[88]
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmentation. In AAAI, volume 34, pages 13001-13008, 2020.
[89] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633 641, 2017.
[90] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 2021.
[91] Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self attention for vision transformers. arXiv preprint arXiv:2106.037/1 2021.
14


Reviews:
[{"review_id": "h_kpfnBLW_X", "reviewer": "Reviewer_GTX2", "summary": "This paper proposes a new Sequencer architecture that replaces self-attention in ViT with BiLSTM(2D) for the image classification task. On ImageNet-1K dataset, Sequencer achieves better performance than current other similar scale models. The authors also show Sequencer is more robust to resolution variation and suffers from less severe accuracy degradation when the input resolution is increased.", "questions": "1. What is the running time of Sequencer and other baselines on ImageNet-1K? Are there any potential ways to reduce the computational cost, such as reducing the length of the sequence by downsampling? If so, can the authors report the model performance compared with the baselines under a similar scale of FLOPs and throughput?\n\n2. See (2) of Cons. Can the authors provide more discussions and potential visualization on why Sequencer block is effective?\n\n3. What if replace LSTM with other RNN cells, such as GRU? Will the model still work well? If not, which part is essential to the model performance?", "limitations": "The authors have discussed the limitations in the conclusion. Actually, it would be better if the authors can test the model's effectiveness on tasks that require sequence modeling such as video action recognition in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. This paper makes an attempt to use LSTM, an unexplored inductive bias, to replace self-attention in ViT for image classification and shows its effectiveness. This line of research helps the community understand what is indeed essential for vision tasks.\n\n2. Strong results and extensive experiments. It compares with a series of related works based on various inductive biases and shows that it has superior performance and transferability under a similar scale of parameters. Besides, ablation studies are conducted.\n\nCons:\n1. The computational cost is too high. As shown in Table 1, under a similar scale of model parameters, Sequencer usually needs 2x FLOPs and is 2x~10x lower throughput compared with other methods. Although this is not surprising due to the recursion in LSTM, I am still concerned about the practicality of this model with such a high computational cost.\n\n2. Lack of reasoning on how using LSTM captures the spatial information and why it is so effective. In BiLSTM2D, it uses LSTM to capture dependencies from horizontal and vertical patches respectively. From my point of view, this design should not be as effective as global dependencies in self-attention since you may need to involve patches that are not necessarily in the same horizontal and vertical line to understand the objects in the images. Besides, I am also curious about what role the memory in LSTM plays in processing spatial information. The above analysis is critical for readers to understand the model but is missing in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c6ypb13dnO4", "reviewer": "Reviewer_ivfp", "summary": "This paper proposed Sequencer by using deep LSTMs instead of self-attention for image classification. And many related works were compared in experiments to validate the performance of sequencer.", "questions": "1.\tFrom Table 1, compared with ConvNetXt, Sequencer achieved worse performance at FLOPS and throughput with 0.2~0.3 top-1 accuracy increase. Meanwhile, the fine-tuned sequencer is worse than the fine-tuned ConvNetXt.\n2.\tIn Figure 3, the performance between Sequencer and ConvNetXt seems to conflict with Table 1？", "limitations": "Some experimental results are not clearly explained.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This paper proposed Sequencer, which uses LSTM instead of the self-attention for sequence modeling. This paper also proposed a two-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs to enhance performance. Experiments showed the advantages of Sequencers compared to the self-attention mechanism in transferability and resolution adaptability. The work is clearly stated, and the manuscript is well written.\nWeakness: Some experimental results are not clearly explained.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EsmSLbo41JW", "reviewer": "Reviewer_gJFV", "summary": "This paper introduces a new model architecture using LSTM for image classification. By adapting 2-dimensional LSTM (Bi-LSTM for vertical and horizontal directions) into the Transformer-like architecture, the model outperforms ViT-based and SOTA CNN-based architectures with less number of parameters. \n\n", "questions": "1. The authors mentioned that 'The higher the input resolution, the more memory-efficient and throughput-economical are on Sequencers' (Line 251-252 and Figure 4). I believe LSTM, in particular, 2D LSTM is not memory efficient and cannot be throughput-economical as it requires computing and saving the activation for all directions. Especially it should be worse for the higher resolutions. Could authors explain how sequencer becomes memory-efficient and throughput-economical? \n\n2. It would be great to discuss and address the weaknesses mentioned above during the rebuttal. ", "limitations": "The authors explained the limitations and potential negative societal impact of their work in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is clearly written. \n\n2. The paper proposes a simple yet effective framework using LSTM. The model outperforms transformer and CNN-based models for image classification. This work provides a great alternative to Transformer and CNNs for image classification. \n\n3. The proposed model is especially efficient for higher resolutions. \n\n## Weaknesses\n\n1. Lack of related work\n\n- There are a number of studies using multi-directional LSTM/RNN for vision tasks that are very relevant to this work e.g., [1-4]. The authors should cite and discuss the similarities and differences. \n\n- ReNet [69] is very relevant to this work. The authors pointed out that the major difference is to use a transformer-like block structure. However, the benefit of this structure and what it provides to the model compared to ReNet or other related works [1-4] are missing. \n\n2. Due to LSTM's sequence nature, LSTM-based models are not easily parallelizable, especially compared to transformer and CNN-based models. I see that throughput is much worse than other models. I assume training time would be especially slow. It is unclear to me how throughput improves with higher resolutions.\n\n[1] \"Multi-dimensional recurrent neural networks.\" ICANN 2007.\n\n[2] \"Pixel recurrent neural networks.\" ICML 2016.\n\n[3] \"Scene labeling with lstm recurrent neural networks.\" CVPR 2015.\n\n[4] \"Semantic Object Parsing with Local-Global Long Short-Term Memory\" CVPR 2016\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BCUgzjlVL7", "reviewer": "Reviewer_ouBL", "summary": "This paper proposes an architecture for image classification named Sequencer, which utilizes the BiLSTM module to replace the self-attention module in the vision transformer model. The BiLSTM module is further improved by processing the vertical and horizontal axes in parallel from top/botton and left/right directions. Experiments on image classification tasks demonstrate that the proposed method can acheive similar performance with existing classification models with similar number of paramters.", "questions": "Please refer to the weaknesses part", "limitations": "The limitations are mainly about the limited novelty of the proposed method and the poor experimental results (much higher FLOPs, lack of experiments on other vision tasks).", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This paper is well-written. The idea is easy to understand. \n\n+ The proposed method is the first work to empirically show the effectiveness of LSTM modules in large scale image classification tasks, which would have a board impact in investigating the potential of LSTM-like architectures in the computer vision field.\n\n+ Ablations and visualization results are rich, which present the validity of the proposed method in terms of the importance of each component.\n\nWeaknesses:\n\n- The novelty of the proposed method is limited. The proposed Sequencer replaces the self-attention module in ViT with the existing BiLSTM module. Besides, [r1] shows that the self-attention module in ViT can be replaced with a simple spatial pooling operator, which suggests that such replacement is incremental. \n\n- Although the proposed model can achieve similar performance with existing SOTA architecures, it requires much higher FLOPs and throughput as shown in Table 1. \n\n- Evaluation is only conducted on image classification. It would be better to evaluate the proposed architecture on more vision tasks such as detection and segmetation to show its generalization ability.\n\n[r1] MetaFormer Is Actually What You Need for Vision. CVPR 2022.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept