SeaFormer: Squeeze-enhanced Axial Transformer for Mobile Semantic Segmentation
Authors: Qiang Wan
Zilong Huang
Jiachen Lu
Gang YU
Li Zhang
Venue: ICLR 2023 poster (2023)

Abstract: Since the introduction of Vision Transformers, the landscape of many computer vision tasks (e.g., semantic segmentation), which has been overwhelmingly dominated by CNNs, recently has significantly revolutionized. However, the computational cost and memory requirement render these methods unsuitable on the mobile device, especially for the high resolution per-pixel semantic segmentation task. In this paper, we introduce a new method squeeze-enhanced Axial Transformer (SeaFormer) for mobile semantic segmentation. Specifically, we design a generic attention block characterized by the formulation of squeeze Axial and spatial enhancement. It can be further used to create a family of backbone architectures with superior cost-effectiveness. Coupled with a light segmentation head, we demonstrate state-of-the-art results on the ADE20K, Pascal Context and COCO-stuff datasets. Critically, we beat both the mobile-friendly rivals and Transformer-based counterparts with better performance and lower latency without bells and whistles. Beyond semantic segmentation, we further apply the proposed SeaFormer architecture to image classification problem, demonstrating the potentials of serving as a versatile mobile-friendly backbone.

Published as a conference paper at ICLR 2023
SEAFORMER: SQUEEZE-ENHANCED AXIAL TRANS- FORMER FOR MOBILE SEMANTIC SEGMENTATION
Qiang Wan': Zilong Huang", Jiachen Lu', Gang Yu², Li Zhang't School of Data Science, Fudan University
Tencent PCG
ABSTRACT
Since the introduction of Vision Transformers, the landscape of many computer vision tasks (e.g., semantic segmentation), which has been overwhelmingly dom- inated by CNNs, recently has significantly revolutionized. However, the com- putational cost and memory requirement render these methods unsuitable on the mobile device, especially for the high- high-resolution per-pixel semantic segmentation task. In this paper, we introduce new method squeeze-enhunced Axial Trans- former (SeaFormer) for mobile semantic segmentation. Specifically, we design generic attention block characterized by the formulation of squeeze Axial and detail enhancement. can be further used to create a family of backbone archi- tectures with superior cost-effectiveeness. Coupled with a light segmentation head, we achieve the best trade-off between segmentation accuracy and latency on the ARM-based mobile devices the ADE20K and Cityscapes datasets. Critically, we beat both the mobile-friendly rivals and Transformer-base counterparts with better performance and lower latency without bells and whistles. Beyond seman- tic segmentation, we further apply the proposed ScaFormer architecture to im- age classification problem, demonstrating the potentials of serving as versatile mobile-friendly backbone. Our code and models are made publicly available at Sordann-vg//ae SeaFormer
btttps:/giie /ttpbtgggttt.../////
INTRODUCTION
As a fundamental problem in computer vision, semantic segmentation aims to assign a semantic class label to each pixel in an image. Conventional methods rely on stacking local convolution kernel Long et al. (2015) to perceive the long-range structure information of the image.
Since the introduction of Vision Transformers Dosovitskiy al. (202T), ). the landscape of semantic segmentation has significantly revolutionized. Transformer-basee approaches [Zheng et al. (2021); Xie et al. (02T)) have remarkably demonstrated the capability of global context modeling. However, the computational cost and memory requirement of Transformer render these methods unsuitable mobile devices, especially for high-resolution imagery inputs.
Following conventional wisdom of efficient operation, Docal/window-basee attention Luong et al.
2015): ; Liu et al. (202T): Huang et al. (2021a); Yuan et al. (2021). Axial attention Huang et al.
2019b g Ho et al. (2019: Wang et al. (2020a), dynamic graph message passing Zhang et al. (2020)
2022b) and some lightweight attention mechanisms Hou et al. (2020); et al. (2021b 2020)
al. 2018); Shen al. (2021): Xu et al. (2021) Cao al. (2019); Woo (2018): 8); Wang et al.
(2020b) Choromanski et al. (2021): Chen et al. (2017); Mehta & Rastegari (2022a) 22) are introduced.
However, these advances are still insufficient to satisfy the design requirements and constraints for mobile devices due to the high latency on the high-resolution inputs (see Figure []. Recently there surge of interest building a Transformer-bas based semantic segmentation. In order to reduce the computation cost at high resolution, TopFormer Saarmerrhannggg et al. (2022c) dedicates to applying the global attention a 1/64 scale of the original input, which definitely harms the segmentation performance.
To solve the dilemma of high-resolution computation for pixel-wise segmentation task and low latency requirement on the mobile device performance harmless way, we propose a family
WWrk done while Qiang Wan was an intern at Tencent GY-Lab.
1 "ooresponniig author: lizhangfd@ wudeefff@@uud...cmm www.ffudd...c

---

Published as a conference paper at ICLR 2023
SeaFormettarrs)
Seyformer
6000
Transtormer
42
TopFormer
MixFormer
Seaformer(ours)
5000
ACmix
Axial attention
40
Local attention
4000
Latency(ms))
38
3000
LiPP-A
10.0
7.5
36
2000
5.0
2.5
1000
0.0
34
ASPPPMVL3
32
LR-ASPP(MV3-LL)
LR-ASPP(MV22)
128a
256
512=
55
90
150
245
400
665
1095
Resolution
Latency(ms)
Figure Left: Latency comparison with Transformer Vaswani et al. (017)), MixFormer Chen et al. (002aa). ACmix Pan et al. (2022b), Axial attention Ho et al. (219)) and local attention Luong measured with a single module of channel dimension 64 on a Qualcomm Snap- dragon 865 processor. Right: The mloU versus latency on the ADE20K val set. MV2 means Mo- (Sandler et al. (2018), MV3-L means MobileNerV3-Large Howard et al. (2019). MV3-Lr denotes Sandler
al. (015).
bileNetV2
Howard et al. (2019). The latency is measured on single Qualcomm Snapdragon 865, and only an ARM CPU core used for speed testing. No other means of acceleration, e.g., GPU quantification, is used. For figure Right, the input size is 512x512. SeaFormer achieves superior trade-off between mloU and latency.
of mobile-friendly Transformer-based semantic segmentation model, dubbed squeeze-enhanced Axial Transformer (SeaFormer), which reduces the computational complexity axial attention from O(H W)) to O(H)). to achieve superior accuracy-efficieme trade-off mobile devices and fill the vacancy froiilefrieenly efficient Transformer.
The core building block squeeze-entn eeneeeeee Axial attention (SEA attention) seeks to squeeze (pool) the input feature maps along the ceiizontal/verticee axis into a compact column/row and computes self-attention. We concatenate query, keys and values to compensate the detail information sacrificed during squeeze and then feed it into a depth-wise convolution layer to enhance local details.
Coupled with a light segmentation head, design (see Figure with the proposed SeaFormer layer in the small-scale feature is capable of conducting high-resolution image semantic segmenta- tion with low latency on the mobile device. As shown in Figure [ the proposed SeaFormer out- performs other efficient neural networks on the ADE20K dataset with lower latency. In particular, superior the lightweight CNN counterpart MobileNetV3 (41.0 33.33. mIoU) with lower latency (106ms vs. 126ms) on an ARM-based mobile device.
SeaFormer-Bate
We make the following contributions: (i) We introduce novel enneeze-enhanced Axial Trans- former (SeaFormer) framework for mobile semantic segmentation; (ii) Critically, we design generic attention block characterized by the formulation of squeeze Axial and detail enhancement; It can be used to create family of backbone architectures with superior cost-effectiveness: (iii) We show top performance on the ADE20K and Cityscapes datasets, beating both the mobile-friendly rival and Transformer-basee segmentation model with clear margins; (iv) Beyond semantic segmen- tation, we further apply the proposed SeaFormer architecture to the image classification problem, demonstrating the potential of serving versatile mobile- mobile-friendly backbone.
RELATED WORK
Combination of Transformers and convolution Convolution is relatively efficient but not suit- able to capture long-range dependencies and vision Transformer has the powerful capability with a global receptive field but lacks efficiency due to the computation of self-attention. In order make full use of both of their advantages, Mobile ViTViMMMiii & Rastegari (202aa), TopFormer Zhang et al. (202c)). LVT Yang al. (2022), Mobile-Formee Chen et al. (2022b). bb. Edge ViTs Pan al. (2022a), Mobile VITv2 Mehta & Rastegari (2022b). EdgeFormer Zhang et al. (2022a) and EfficicetFormer al. (2022) constructed as efficient ViT's by combining convolution with Transformers. Mobile-
2

---

Published as a conference paper at ICLR 2023
VIT. Mobile-Formet TopFormer and EfficientFormer restricted by Transformer blocks and have to trade off between efficiency and performance in model design. LVT, Mobile V and EdgeVit ViTs keep the model size small the cost of relatively high computation, which also means high latency.
Axial attention and variants Axial attention Huang et al. (2019b); Ho et al. (0019): Wang et al. (2020a) is designed reduce the computational complexity original global self-attention Vaswani et al. (2017). It computes self-attention over a single axis at time and stacks a horizontal and a
vertical axial attention module obtain the global receptive field. Strip pooling Hoo al. (2020)
and Coordinate attention Hou et al. (2021) uses a band shape pooling window to pool along either gather long-range context. Kronecker Attention Net- works Gao et al. (2020) uses the juxtaposition of horizontal and lateral average matrices to average the input matrices and performs attention operation. These methods and other similar implementa- tions provide performance gains partly at considerably low computational cost compared with Axial attention. However, they ignore the lack of local details brought by the pooling/average operation.
the horizontal or the vertical dimension
Mobile semantic segmentation The mainstream of efficient segmentation methods are based on lightweight CNNs. DFANet [i et al. (2019) adopts a lightweight backbone to reduce computation cost and adds feature aggregation module refine high-level and low-level features. ICNet Zhao jet al. 2018) designs an image cascade network to speed up the algorithm, while BiSeNet Yu et al. (2018 2021) proposes two-stream paths for low-Ievel details and high-level context information, separately. Fast-SCNN Poudel et al. (2019) shares the computational cost of the multi-branch net- work yield run-time fast segmentation CNN. TopFormer Zhang al. (022c) presents new architecture with a combination of CNNs and ViT and achieves good trade-ofi off between accuracy and computational cost for mobile semantic segmentation. However, it is still restricted by the heavy computation load of global self-attention.
3 METHOD
3.1 OVERALL ARCHITECTURE
Inspired by the two-branch architectures Yu al. (022): Poudel et al. (2019): ; Hong et al. (2021) )
Huang et al. (202 Chen et al. (2022b), we design a gqueeze-enhancee Axial Transformer (SeaFormer) framework. As shown Figure SeaFormer consists of these parts: shared STEM, context branch, spatial branch, fusion block and light segmentation head. For fair comparison, we follow TopFormer Zhang et al. (2022c) design the STEM. It consists of one regular convolution with stride of 2 followed by four MobileNet blocks where stride of the first and third block is 2. The context branch and the spatial branch share the produced feature map, which allows us to build a fast semantic segmentation model.
Context branch The context branch is designed to capture context-rich information from the fea- ture map X,- As illustrated in the red branch of Figure the context branch is divided into three stages. To obtain larger receptive field, we stack SeaFormer layers after applying a MobileNet block down-sampling and expanding feature dimension. Compared with the standard convolution the down-sampling module, MobileNet block increases the representation capacity of the model while maintaining a lower amount of computation and latency. For variants except LeaFormer-Large SeaFormer layers are applied in the last two stages for superior trade-off between accuracy and ef- ficiency. For LeaFor we insert ScaFormer layers in each stage of context branch. To achieve good trade-off between segmentation accuracy and inference speed, we design squeeze- enhanced Axial attention block (SEA attention) illustrated in the next subsection.
Spatial branch The spatial branch is designed to obtain spatial information in high resolution. Identical to the context branch, the spatial branch reuses feature maps x,- However, the feature from the early convolution layers contains rich spatial details but lacks high-level semantic information. Consequently, we design a fusion block to fuse the features in the context branch into the spatial branch, bringing high-level semantic information into the low-level spatial information.
As depicted in Figure 2 high resolution feature maps in the spatial branch are fol- lowed by 1 convolution and batch normalization layer to produce feature to fuse. Low
Fusion block
3

---

Published as a conference paper at ICLR 2023
MVZ
12
MA22
SeaFormer Layers
MV2
SeaFormer Layers
NIV2 12
Seaformer
Layers
Context branch
Spatial branch
15
Comarbl
Enrowlll
Light segmentatiin head
up
Conendi
shared STEM
Fusion block
Fusion block
Fusion block
The overall architecture of SeaFormer. It contains shared STEM, context branch (red), spatial branch (blue), fusion block and light segmentation head. MV2 block means MobileNetV2 block and MV2 12 means MobileNetV2 block with downsampling. SeaFormer layers and fusion block with dash box only exist in SeaFormer-L.. The symbol ( denotes element-wiss multiplication.
Figure 2:
resolution feature maps in the context branch are fed into a x convolution layer, a batch normal- ization layer, sigmoid layer and up-sampled high resolution to produce semantics weights by bilinear interpolation. Then, the semantics weights from context branch are semnent-wisely multi- plied to the high resolution feature from spatial branch. The fusion block enables low-level spatial features to obtain high-level semantic information.
Light segmentation head The feature after the last fusion block is fed into the proposed segmen- tation head directly, as demonstrated Figure For fast inference purpose, our light segmentation head consists of two convolution layers, which are followed by batch normalization layer sepa- rately and the feature from the first batch normalization layer is fed into an activation layer.
3.2
AXIAL ATTENTION
The global attention can be expressed as
Yo Σ
softmax, kp) Vp
PEG(o)
where x E R#xWxC k. v are linear projection of x, i.e.q Wgx,k W W,x where Wg. WA CC XC W. E RCxxC' are learnable weights. (o) means all positions on the feature map of location o (i,j). When traditional attention module is applied on a feature map of H x W C, the time complexity can be O(H2W?(C, + C.))). leading to low efficiency and high latency.
Yo
Σ
softmax, (g kp) Vp
(2)
peNmxxm(o)) (o)
Yo = (x)
Σ
softmaxp (q kp) Vp +
Σ
softmaxp (q. kp) Vp
(3)
pENixi(t)
To improve the efficiency, there are some works Liu et al. (202)); Huang al. (201bb); Ho et al. 2019 computing self-attention within the local region. We show two most representative efficient Transformer in Equation 2, 3. Equation 2 is represented by window-based attention Luong et al.
2015 successfully reducing the time complexity to O(m² Omm2 (C + C.)) O(HW), where mm (o) means the neighbour m x m positions of o, but loosing global receptiveness. The Equa- tion 3 is represented by Axial attention Ho et al. (2019), which only reduces the time complexity to O(H - WWHWWHC (C,z + C.)) O(HW)) where N Ni means all the positions of the column of o; Ni (o) means all the positions of the row of
According to their drawbacks, we propose the mobile-friendly squeeze-enhanced Axial attention, with succinct squeeze Axial attention for global semantics extraction and efficient convolution- based detail enhancement kernel for local details supplement.
9(k) W (a ( (.W)
(HH,Cok)
9(x) = (
H (Ca.W.H.))
(W.C.()
(4)
4

---

Published as a conference paper at ICLR 2023
Aul0 T
Concat
Squeeze-enhanced
Detail enhancement karnel
K
Hx
Mui
V
Squeeze Axial
Horizontal squeeze
Multi-head
Broadcast
attention
FAIDO EYE
Vertical
Multi-head
squeaze
attention
Broadcast
the schematic illustration of the proposed squeeze-enhanced Axial Transformer layer including a squeeze-enhanced Axial attention and a squeeze-enhanced Axial Transformer layer, including detail enhancement kernel and squeeze Axial attention. The symbol + indicates an glement-wise addition operation. Mul means multiplication.
Figure 3: Right:
Network (FFN). Left the
efficient computation and aggregate global informa- tion at the same time, we resort to a more radical strategy. In the same way, q. k, v are first get from with w w(s) E RCaxxC, w(() RC. According Equation 4 we first imple- ment horizontal squeeze by taking average of query feature map on the horizontal direction. In the same way, the right shows the vertical squeeze on the vertical direction. x-) means permuting the dimension of tensor
Squeeze Axial attention To achieve
given, and 1'm RT is vector with all the elements equal to I. The squeeze operation on q also repeats on k and v, so we finally get 9(h), k()), V(N) E #Hx
((v)) k(n). V(w) E RWxC,, The squeeze operation reserves the global information to single axis, thus greatly alleviating the following global semantic extraction showing by Equation
H
W
((i))
softmax, (qT))ck()pp))
V(K)p
softmax, ((o))>((o))) ((u
(5)
Each position of feature map propagates information only on two squeezed axial features. Although shows no distinct computation reduction comparing Equation 3 repeat of Equation 5 can be simply implemented by the most efficient broadcast operation. The detail is shown in Figure
Time complexity for squeezing q. k, O((H + W2C2 + C.)) and the attention operation takes O((H² + (C) C.)) time. Thus, our squeeze Axial attention successfully reduces time complexity to O(HW).
Squeeze Axial position embedding Equation 4 are, however, not positional-aware, including no positional information of feature map. Hence, we propose squeeze Axial position embedding to squeeze Axial attention. For squeeze Axial attention, we render both q(h) and k(A) to be aware of their position in squeezed axial feature by introducing positional embedding r (h)) (()) R#fxC,,,
which are linearly interpolated from learnable parameters B()). B E RLxxC, L is a constant. In the same way, r/m); (v) R WW are applied to ((v), (())- Thus, the positional-aware squeeze Axial attention can be expressed Equation
softmax, (Qu()) - rl ( (kibpp + +((o))) v(n)
Y(d)
(6)
softmax, (q + +r(og)) (k())p + r(osp)) v(e)p
The squeeze operation, though extracting global semantic informa- tion efficiently, sacrifices the local details. Hence an auxiliary convolution-based kernel is applied enhance the spatial details. As is shown in the upper path of Figure q. k, v are first get from with another Wle) w(() e XC w(") e RCxXC and are concatenated on the channel dimen- sion and then passed to block made up of 3x3 depth-wise convolution and batch normalization.
Detail enhancement kernel
A
5

---

Published as conference paper at ICLR 2023
Backbone
Decoder
Params
FLOPS
mloU
Latency
MobileNetV2
LR-ASPP
2.2M
2.8G
32.0
177ms
MohieeNeeVE-Lare--vddueee
LR-ASPP
1.6M
1.3G
32.3
81
MobileNetV3-Large
LR-ASPP
3.2M
2.0G
33.1
126ms
HRRNet-W1-SSS
HRNet-W Seie-Wi-Smma
4.0M
10.2G
33,4
633mm
TopFormer-T
Simple Head
1.4M
0.6G
33.6
43ms
TopFormer-T
Simple Head
1.4M
0.6G
34.6
43ms
SeaFormer-T
Light Head
1.7M
0.6G
35.0
40ms
SeaFormer-T*
Light Head
1.7M
0.6G
35.8++0.35
40ms
ConvMLP-S
Semantic FPN
12.8M
33.8G
35.8
777ms
EfficientNet
DecpLabb33+
17.1M
26.9G
36.2
970ms
MobileNetV2
Lite-ASPI
2.9M
4.4G
36.6
235ms
TopFormer-S
Simple Head
3.1M
1.2G
36.5
74ms
TopFormer-S*
Simple Head
3.IM
1.2G
37.0
74ms
ScaFormer-S
Light Head
4.0M
1.IG
38.1
67ms
ScaFormer-Se
Light Head
4.0M
1.IG
39.4
67ms
MiT-BO
SegFormer
3.8M
8.4G
37.4
770ms
ResNet18
Lite-ASPP
12.5M
19.2G
37.5
648ms
ShuffleNetV2-1.5x
DeepL.abV3+
16.9M
15.3G
37.6
960ms
MobileNetV2
DeepLabV3+
15.4M
25.8G
38.1
1035ms
TopFormer-B
Simple Head
5.IM
1.8G
38.3
110ms
TopFormer-B
Simple Head
5.1M
1.8G
39.2
110ms
SeaFormer-B
Light Head
8.6M
1.8G
40.2
106ms
SeaFormer-Ba
Light Head
8.6M
1.8G
41.0
106ms
MiT-BI
SegFormer
13.7M
15.9G
41.6
1300ms
SeaFormer-L
Light Head
14.0M
6.5G
42.7
367ms
SeaFormer-L
Light Hcad
14.0M
6.5G
43.7-+0.36
367ms
Table I: Results of semantic segmentation on ADE20K val set, indicates training batch size is
32. The latency is measured on a single Qualcomm Snapdragon 865 with input size 512x5 12, and only an ARM CPU core is used for speed testing. References: MobileNetV2[Sanndeee MobileNetV3 Howard et al. (2019), HRNet Yuan et al.] (2020), TopFormer /Zhang et al. (2022c),
et al. (2018),
ConvMLP Li et al. (2021a), Semantic FPN Kirillov et al. (2019), EfficientNet Tan & Le (2019).
DeepLab and ASPP Chen et al. (2018a), SegFormer Xie al. (2021). ResNet He et al. (016), ShuffleNetV2-LL Ma et al. (2018).
By using a 3x3 convolution, auxiliary local details can be aggregated from q, q, And then linear projection with activation function and batch normalization are used to squeeze (2Cgt + C.) dimension C and generate detail enhancement weights. Finally, the enhancement feature will be fused with the feature given by squeeze Axial attention. Different enhancement mode including element-wise addition and multiplication will be compared in experiment section. Time complexity for the x depth-wise convolution is (33 HW (2Cgk- + C.)) and the time complexity for the x convolution (OHWCC2C, + +.)). Time for the other operations like activation can omitted.
Architecture and Variants We introduce four variants, SeaFormer-Tiny, Small, Base and Large (T. S, B and L). More configuration details are listed in the supplementary material.
EXPERIMENTS
We evaluate our method on semantic segmentation and image classification tasks. First, we de- scribe implementation details and compare results with state of the art. We then conduct series of ablation studies to validate the design of SeaFormer. Each proposed component and important syper-parameters examined thoroughly.
4.1
EXPERIMENTAL SETUP
4.1.1 DATASET
We perform segmentation experiments over ADE20K Zhou et al. (017)). SitySs Cordts et al. (2016). The mean of intersection union (mloU) set the evaluation metric. We convert full-precision models to TNN Contributors (2019) and measure latency on an ARM-based device with single Qualcomm Snapdragon 865 processor.

---

Published as conference paper at ICLR 2023
Method
Backbone
FLOPs
mvoU(val)
(teS(
Latency
FCN
MobileNetV2
317G
61.5
24190ms
PSPNet
MobileNetV2
423G
70.2
31440ms
SegFormer(h)
MîT-BO
17.7G
71.9
1586ms
SegFormer(f)
MiT-BO
125.5G
76.2
11000ms
L-ASPP
MobileNetV2
12.6G
72.7
887ms
LR-ASPP
MobileNetV3-L
9.7G
72.4
72.6
660ms
LR-ASPP
MobileNetV3-S
2.9G
68.4
69.4
21 Ims
Simple Head(h)
TopFormer-B
2.7G
70.7
173ms
Simple Head(f)
TopFormer-B
11.2G
75.0
75.0
749ms
Light Hh)
SeaFormer-S
2.0G
70.7
71.0
129ms
Light Head(f)
SeaFormer-S
8.0G
76.1
75.9
518ms
Light Head(h)
Sea FormerB
3.4G
72.2
72.5
205ms
Light Head(f)
SeaFormer-B
13.7G
77.7
77.5
821ms
Table 2: Results Cityscapes val set. The results test set of some methods are not presented due to the fact that they are not reported in their original papers.
ADE20K dataset covers 150 categories, containing 25K images that are split into 20K/2K/3K for Train, val and test. CityScapes driving dataset for semantic segmentation. It consists of 5000 fine annotated high-resolution images with 19 categories.
4.1.2 IMPLEMENTATION DETAILS
We set ImageNet- IK Deng et al. (2009) pretrained network as the backbone, and training details of ImageNet- IK are illustrated in the last subsection. For semantic segmentation, the standard Batch- Norm loffe & Szegedy (013) layer is replaced by synchronized BatchNorm.
Training Our implementation is based public codebase mmsegmentatinn Contributors (2220). We follow the batch size, training iteration scheduler and data augmentation strategy of TopFormer Zhang et al. (2022c) for a fair comparison. The initial learning rate is 0.0005 and the weight decay is 0.01. A "poly" learning rate scheduled with factor 1.0 is adopted. During inference, we set the same resize and crop rules as TopFormer to ensure fairness. The comparison of Cityscapes contains full-resolution and half-resolution. For the full-resolution version, the training images are randomly scaled and then cropped to the fixed size of 1024 1024. For half-resolution version, the training images are resized to 1024 x 512 and randomly scaling, the crop size is 1024 x 512.
4.2 COMPARISON WITH STATE OF THE ART
ADE20K Table I shows the results of SeaFormer and previous efficient backbones on ADE20K val set. The comparison covers Params, FLOPs, Latency and mloU. As shown Table SeaFormer outperforms these approaches with comparable or less FLOPs and lower latency. Compared with specially designed mobile backbone, TopFormer, which sets global attention as its semantics extrac- tor, SeaFormer achieves higher segmentation accuracy with lower latency. And the performance of SeaFormer-B surpasses MobileNetV3 by large margin of +7.9% mloU with lower latency (-16%). The results demonstrate our SeaFormer layers improve the representation capability significantly.
Cityscapes From the t can be seen that SeaFormer-S achieves comparable or better results than TopFormer-B with less computation cost and latency, which proves that SeaFormer could also achieve a good trade-off between performance and latency in high-resolution scenario.
4.3 ABLATION STUDIES
In this section, we ablate different self-attention implementations and some important design ele- ments in the proposed model, including our squeeze-enhanced Axial attention module (SEA atten- tion) and fusion block on ADE20K dataset.
The influence of components in SEA attention We conduct experiments with several configura- tions, including detail enhancement kernel only. squeeze Axial attention only, and the fusion of both. As is shown in Table 3 only detail enhancement squeeze Axial attention achieves relatively poor

---

Published as a conference paper at ICLR 2023
Enhance Attn branch
Enhance input
Enhance mode
Params
FLOPs
Latency
Top1
mloU
kernel
1.3M
0.58G
38ms
65.9
32.5
1.4M
0.57G
38ms
66.3
33.5
<<<r
conv(x)
Mul
1.6M
0.60G
40ms
67.2
34.9
upconv(x)
Mul
1.8M
0.62G
mm
68.1
35,9
concat[qkv]
Mul
1.7M
0.60G
40ms
67.9
35.8
(oncat[qkv]
Add
1.7M
0.60G
40ms
67.3
35.4
Table 3: Ablation studies on components in SEA attention on ImageNet-1K and ADE20K datasets. Enhancement input means the input of detail enhancement kernel. conv(x) means followed by point-wise conv. upconv(x) is the same as conv(x) except different channels as upconv(x) is from C.n C2 + Ck + C, and cxnv(x) is from Can to Cin- concat[qkv] indicates concat of Q.K, V.
performance, and enhancing squeeze Axial attention with detail enhancement kernel brings per- formance boost with gain of 2.3% mloU on ADE20K. The results indicate that enhancing global semantic features from squeeze Axial attention with local details from convolution optimizes the feature extraction capability of Transformer block. For enhancement input, there an apparent per- formance gap between upconv(x) and conv(x). And we conclude that increasing the channels will boost performance significantly. Comparing concat[qkv] and upconv(x), which also correspond to w/ or w/o convolution weight sharing between detail enhancement kernel and squeeze Axial atten- tion, we can find that sharing weights makes our model improve inference efficiency with minimal performance loss (35.8 vs.35.9). As for enhancement modes, multiplying features from squeeze Axial attention and detail enhancement kernel outperforms add enhancement by +0.4% mloU.
Comparison with different self- order to
Model
Params(B)
FLOPs(()
mloU
Latency
attention modules eliminate the impact of our architecture and demonstrate the effectiveness and generalization ability of SEA attention,
Swin
27.5M
25.6G
44.5
3182ms
CCNet
41.6M
37.4G
43.1
3460ms
ISSA
31.8M
33.3G
37.4
2991 ms
A2-Nets
37.2M
31.1G
28.9
2502ms
we ran experiments on Swin Trans- former Liu al. (2021) by replacing window attention in Swin Transformer with different attention blocks. We set the same training protocol, hyper- parameters, and model architecture configurations as Swin for a fair comparison. When replacing window attention with CCAttention (CCNet) or DoubleAttention (A2-Nets), they have much lower FLOPs than SeaFormer and other attention blocks. Considering that we may not be able to draw conclusions rigorously, doubled the number of their Transformer blocks (including MLP). As ACmix has the same architecture configuration as Swin, we borrow the results from the original paper. From Table 4 be seen that SeaFormer outperforms other attention mechanisms with lower FLOPs and latency.
Axial
36.2M
32.5G
45.3
312 Im
Local
27.5M
25.1G
34.2
3059ms
MixFormer
27.5M
24.9G
45.5
2817ms
ACmix
27.9M
26.6G
45.3
3712ms
Global
27.5M
0.144T
OOM
14642ms
ScaFormer
34.0M
24.9G
46.5
2278ms
Table 4: Results on ADE20K val set based on Swin Trans-
former architecture. (B) denotes backbone. OOM means
CUDA out of memory. References: ISSA Huang et al. (019a). A2-Nets ChenneestCheeneee et (2018b)
The influence of the width in fusion block To study the influence of the width in fusion block, we perform experiments with different embedding dimensions in fusion blocks on SeaFormer-BBar Base, M denotes the channels that spatial branch and context branch features mapping in two fusion blocks. Results are shown in Table
4.4 IMAGE CLASSIFICATION
We conduct experiments on ImageNet- IK Deng et al. (2009), which contains 1.28M training
images and 50K validation images from 1,000 classes. We employ Adam Kingma & Ba (2014)
optimizer for 600 epochs using a cosine decay learning rate scheduler. A batch size of 1024, an mitial learning rate of 0,064, and weight decay of 2e-5 are used. The results illustrated in Table 6 Compared with other efficient approaches, SeaFormer achieves a relatively better trade-off between latency and accuracy.
8

---

Published as a conference paper at ICLR 2023
Method
P(M)
F(G)
Top1
I.
Params
FLOPS
Latency
mloU
MobileNerV3-SSalll
2.9
0.1
67.4
5ms
64,96
8.5M
1.7G
102ms
40.3
SeaFormer-T
1.8
67.9
7ms
128,160
8.6M
1.8G
106ms
41.0
MobilcViT-XXX
1.3
0.4
69.0
24ms
192,256
8.7M
2.0G
m2
41.2
MobileViTv2-0050
1.4
0.5
70.2
32ms
Posbias x
Params
FLOPs
Latency
mloU
MobileOne-SO0
2.1
0.3
71.4
14ms
1.65M
0.60G
40ms
35.6
MobileNetV2
3.4
72.0
17ms
1.67M
0.60G
40ms
35.8
Mobile-FormerSo
4.8
0.1
72.8
ms
SeaFormer-S
4.1
0.2
73.3
12ms
Table 5: Ablation studies on embedding dimen-
ViT-XXX
4.1
0.6
74.4
71ms
sions and position bias.M = [128, 160] is an op-
LVT
5.5
74.8
97ms
timal embedding dimension in fusion blocks.
Mobile ViT-XS
2.3
0.9
74.8
54ms
MobileNetV3-Lange
5.4
0.2
75.2
16ms
Mobile-Formerl51
7.7
0.2
75.2
42ms
MobileViTv2-0.75
2.9
1.0
75.6
68ms
MobileOne-S1
4.8
75.9
40ms
3.6ms
SeaFormer-B
8.7
0.3
76.0
20ms
7.5%
9.0%
MobileOne-$2-
7.8
1.3
77.4
6.3ms
EdgeViT-XS
6.8
77.5
124ms
MobileViTv2-1.00
4.9
78.1
115ms
49.1%
MobileOne-S3*
10.1
1.9
78.1
91ms
MobileViT-S
5.6
1.8
78.4
88ms
34.3%
EfficientNNe-BII
7.8
0.7
79.1
6lms
EfficientFormer-L1
12.3
79.2
94ms
13.7ms
14.8
0.5
79.3
102ms
MobilcOne-S4*
14.8
3.0
79.4
143ms
ScaFormer-I.
14.0
1.2
79.9
61ms
Shared
Context branch Spatial branch
Light segmentation head
Table 6: Image classification results on ImageNet- IK val The FLOPs and latency mea- sured with imput size 224x224, except for Mo- ViT and MobileViTva that are measured with 256x256 according to their original implementa- tions. P, and mean Pa Parameters, FLOPs s and latency. indicates re-parameterized vari- ants Vasu et al. (2022). The latency is measured on single Qualcomm Snapdragon 865, and only an ARM CPU core used for speed testing. No other means of acceleration, e.g., GPU quan- tification, used.
Figure 4: The inference latency of components.
4.5 LATENCY STATISTICS
We make the statistics of the latency of the pro- posed SeaFormer- Tiny, as shown in Figure 4 the shared STEM takes up about half of the la- tency of the whole network (49%). The latency
of the context branch is about third of the to- tal latency (34%), whilst the actual latency of the spatial branch is relatively low (8%) due to sharing early convolution layers with the context branch. Our light segmentation head (8%) also contributes to the success of building light model.
CONCLUSION
In this paper, we have proposed squeeze-enhanced Axial Transformer (SeaFormer) for mobile se- mantic segmentation, filling the vacancy of mobile-friendly efficient Transformer. Moreover, create family of backbone architectures of SeaFormer and achieve cost cost-effectivvenes.. The superior performance on the ADE20K and Cityscapes, and the lowest latency demonstrate its effectiveness on the IRM-based mobile device. Beyond semantic segmentation, we further apply the proposed SeaFormer architecture image classification problem, demonstrating the potential of serving as versatile mobile-friendly backbone.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (Grant No. 62106050), Lingang Laboratory (Grant No. (F--S-202220-077) Natural Science Foundation of Shanghai (Grant No. 22ZR1407500) and CCF-Tencent Open Research Fund (No. CCF-Tencent RAGR20210111).

---

Published as a conference paper at ICLR 2023
REFERENCES
Yue Cao, Jiarui Xu. Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze- excitation networks and beyond. ICCV workshops, 2019.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint, 2017.
Liang-Chiee Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018a.
Qiang Chen, Qiman Wu, Jian Wang. Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang- Mixformer: Mixing features across windows and dimensions. CVPR, 2022a.
Yinpeng Chen, Xiyang Dai, Dongdong Chen. Mengehen Liu. Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile- former: Bridging mobilenet and transformer. In CVPR, 2022b.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A 2-nets: Double attention networks. In NeurlPS, 2018b.
Krzysztof Choromanski, Valeri Likhosherstov, David Dohan, Xingyou Song. Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, al. Rethinking attention with performers. In ICLR, 2021.
MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and bench- mark, 2020.
TNN Contributors. Tnn: A high-performance, lightweight neural network inference framework., 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR. 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In ICLR, 2021.
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Kronecker attention networks. In ACM SIGKDD, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers, arXiv preprint, 2019.
Yuanduo Hong, Huihui Pan, Weichao Sun, and Yisong Jia. Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes. arXiv preprint, 2021.
Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng. Strip pooling: Rethinking spatial pooling for scene parsing. In CVPR, 2020.
Qibin Hou, Daquan Zhou, and Jiashi Feng. Coordinate attention for efficient mobile network design. In CVPR, 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Searching for mobilenetv3. In ICCV, 2019.
Lang Huang, Yuhui Yuan, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Interlaced sparse self-attention for semantic segmentation. arXiv preprint, 2019a
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss- cross attention for semantic segmentation. In ICCV, 2019b.
Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu, Shuffle transformer: Rethink- ing spatial shuffle for vision transformer. arXiv preprint, 2021a.
10

---

Published as a conference paper at ICLR 2023
Zilong Huang, Yunchao Wei, Xinggang Wang. Wenyu Liu, Thomas S Huang, and Humphrey Shi. Alignseg: Feature-aligned segmentation networks. TPAMI, 2021b.
Sergey loffe and Christian Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint, 2014.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
Hanchao Li, Pengfei Xiong, Haoqiang Fan, and Jian Sun. Dfanet: Deep feature aggregation for real-time semantic segmentation. In CVPR, 2019.
Jiachen Li, Ali Hassani, Steven Walton, and Humphrey Shi. Convmlp: Hierarchical convolutional mips for
vision. arXiv preprint, 2021a.
Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong. Improving semantic segmentation decoupled body and edge supervision. ECCV, 2020.
Xiangtai Li, Li, Ansheng You, Li Zhang, Guangliang Cheng, Kuiyuan Yang. Yunhai Tong, and Zhouchen Lin. Towards efficient scene understanding via squeeze reasoning. 77P, 2021b.
Xiangtai Li, Li Zhang, Guangliang Cheng, Kuiyuan Yang. Yunhai Tong. Xiatian Zhu, and Tao Xiang. Global aggregation then local distribution for scene parsing. TIP, 2021c.
Yanyu Li. Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers mobilenet speed. arXiv preprint, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Peter Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. ICLR, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Fffective approaches to attention-hase -hased neural machine translation. arXiv preprint, 2015.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. ECCV, 2018.
Sachin Mehta and Mohammad Rastegari. Mohilevit: light, general-purpose, and frbill- vision transformer. In ICLR, 2022a.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. preprint, 2022b.
arXiv
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak. Hongsheng Li. Georgios Tzimiropoulos, mobile devices with vision transformers. In
and Brais Martinez. Edgevits: Competing light- ECCV, 2022a.
Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In CVPR, 2022b.
Rudra PK Poudel, Stephan Liwicki, and Roberto Cipolla. Fast-scan: Fast semantic segmentation network. In BMVC, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV, 2021.
11

---

Published as a conference paper at ICLR 2023
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. An improved millisecond mobile backbone. arXiv preprint, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. In NeurlPS, 2017.
Huiyu Wang. Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chies Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020a.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprins, 2020b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Sanghyun Woo, Jongchan Park, Joon- Joon-Young Lee, and In Kweon. Cbam: Convolutional block attention module. In ECCV, 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, 2021.
Haotian Yan, Zhe Li, Weijian Li, Changhu Wang, Ming Wu, and Chuang Zhang. Contnet: Why not use convolution and transformer at the same time? arXiv preprint, 2021.
Chenglin Yang. Yilin Wang. Jianming Zhang, Zhang, Zijun Wei, Zhe Lin, Alan Yuille, Lite vision transformer with enhanced self-attention, In CVPR, 2022.
Changqian Yu, Jingho Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmen- tation network for real-time semantic segmentation. In ECCV, 2018.
Changqian Yu, Changxin Gao, Jingbo Wang. Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. IJCV, 2021.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020.
Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint, 2021.
convnets by learning from Li Zhang, Dan Xu, Anurag Amab, and Philip HS Torr. Dynamic graph message passing networks. CVPR, 2020.
Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Edgeformer: Improving vision transformers. arXiv preprint, 2022a.
Li Zhang, Mohan Chen, Anurag Arnab, Xiangyang Xue, and Philip HS Torr. Dynamic graph message passing networks for visual recognition. TPAMI, 2022b.
Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chun- hua. Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In CVPR, 2022c.
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Jenet for real-time semantic segmentation on high-resolution images. In ECCV, 2018.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequee perspective with transformers. In CVPR, 2021.
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba, Scone parsing through ade20k dataset. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2023
Appendix
A ARCHITECTURE DETAILS AND VARIANTS
SeaFormer backbone contains stages, corresponding to the shared STEM and context branch in Figure in the main paper. When conducting the image classification experiments, pooling layer and linear layer are added at the end of the context branch.
Table details the family of our SeaFormer configurations with varying capacities. We construct SeaFormer-Tiin Tiny, SeaForrer-Smmll, SeaFormer-Base and SeaFormer-Large models with different scales via varying the number of SeaFormer layers and the feature dimensions. We use input image size of 512 x 512 by default. For variants except SeaFormer-Large SeaFormer layers are applied in the last two stages for superior trade-off between accuracy and efficiency. For SeaFormer-Large we apply the proposed SeaFormer layers in each stage of the context branch.
B COMPLEXITY ANALYSIS
analyze the complexity of our proposed SEA attention in subsection 3.2 to demonstrate its ef- ficiency theoretically. In our application, we set Cgk = 0.5C, to further reduce computation cost. The total time complexity of squeeze-enhancee Axial attention is
O( (C C.) O(H + W) C.)) (WWCCHHWCC+/WW/ + C.)) =0((1, 1.5W 2HWC + 18HW +2H + 2W)C,)= O(HW).
we assume W and take channel as constant. SEA attention is linear the feature map size theoretically. Moreover, SEA attention only includes mobile-friendly operation like convolution, pooling, matrix multiplication and so on.
C PASCAL CONTEXT PERFORMANCE
We evaluate performance on Pascal Context val set over 59 categories and 60 categories. PAS- CAL Context dataset has 4998/5105 images for train and test, covering 59 semantic labels and background.
Following TopFormer Zhang et al. (2022c), train the models for 80,000 iterations PASCAL Context dataset. The same data augmentation strategy and batch size are adopted for a fair compari- son. The initial learning rate is 0.0002 and the weight decay 0.01. A poly learning rare scheduled with factor 1.0 is used.
demontteesssssattes that ScaFormer-S is +1.4% mloU higher (45.08% vs.43.68% than TopFormer- S with lower latency.
D COCO-STUFF PERFORMANCE
We compare SeaFormer with the previous approaches COCO-Stuff val set. COCO-Stufl dataset augments COCO dataset with pixel-level stuff annotations. 10K complex images are selected from COCO. The train and test contain 9K/IK images.
Following TopFormer Zhang et al. 2022c). we train the models for 80,000 iterations on COCO- Stuff dataset. The same data augmentation strategy and batch size adopted for fair comparison. The initial learning rate 0.0002 and the weight decay is 0.01. poly learning rare scheduled with factor 1.0 is used.
Table 9 reveals that SeaFormer-S +2.0% mloU higher (32.82% vs.30.83% than TopFormer-S with less computation cost and lower latency.
13

---

Published as a conference paper at ICLR 2023
Resolution
SeaFormer Tiny
SeaFormer-Soe
SeaForrer--Bss
SeaFormer-Lang
[Conv, 3, 2]
[Cony, 3, 2]
[Conv, 3, 16, 2]
[Conv, 3, 2]
1tage1
H/2 W/2
[MB, 3, 16, 1]
[MB, 3, 16, 1]
[MB, 3, 1, 16, 1]
[MB, 3,3, 1]
[MB, 3.4..4. 16,
[MB, 3.4, 21
[MB, 3.4, 12.
[MB. 3,4, 64,
Stage2
H/4 W/4
JMB,
[MB, 1,3,3,3,2 24,
3,
[MB, 3,4, 64,
5,3,3, 21
5,3, 48,
[MB,, 5, 64, 21
[MB, 5, 128,
Stage3
H/8 W/8
5.3.5.
(M3.5.3.48.
[MB.5.3.0 1]
[MB. 5, 128.
3,3, 2]
[MB, 3,3, 96, 2]
[MB, 3,
[MB. 3, 4, 192.
Stage4
H/16 x W/16
IMB, 3,3, 11
[MB, 3,3, 96,
[MB, 3, 128, 11
[MB, 3, 4, 192, 11
[Sea, 3, 8]
[MB, 5, 128, 2]
[MB. 5, 160, 2]
[MB, 5, 192, 2]
[MB, 5, 4, 256, 2]
Stage5
H/32 W/32
[Sca, 2, 4]
[Sea, 3. 6]
[Sea, 4, 8]
[Sea, 3, 8]
[MB, 3, 6, 160,
[MB, 3, 6, 192,
[MB 3, 6, 256,
[MB, 3, 6, 320,
Stage6
H/64 W/64
[Sea, 2, 4]
[Sea, 3, 6]
[Sea, 4. 8]
[Sea. 3, 8]
Table Architectures for semantic segmentation. [Conv, ,,16,2 2] denotes regular convolution layer
with kernel of 3, output channel of 16 and stride of [MB, 3,4, 16, 2] means 2obileNet (Sandlerr al. (2018) block with kernel of 3, expansion ratio of 4, output channel of 16 and stride of [Sea,
2, 4] refers to SeaFormer layers with number of layers of 2 and heads of 4.
Backbone
Decoder
F(G)
mloU(60/55)
MBV2-s16
D3epLabV3+
22.24
38.59/42.34
ENetts16
Deepl L333+
23.00
39.19/43.07
MBV3-s16
LR-ASPP
2.04
35.05/38.05
TopFormer-T
Simple Head
0.53
36.41/40,39
ScaFormer-T
Light Head
0.51
37.27/41.49
TopFormer-S
Simple Head
0.98
39.06743.68
SeaFormer-S
Light Head
0.98
40.20/45.08
TopFormer-B
Simple Head
1.54
41.01/45.28
ScaFormer-B
Light Head
1.57
41.77/45.92
Table 8: Results on Pascal Context val set. F means FLOPs.
We omit the latency as the input
resolution is almost the as that in table
Backbone
Decoder
F(G)
mloU
MBV2-s8
PSPNet
52.94
30.14
ENet-s16
D3epLabV3+
27.10
31.45
MBV3-s16
LR-ASPP
2.37
25.16
TopFormer-T
Simple Hcad
0.64
28.34
SeaFormer-T
Light Head
0.62
29.24
TopFormer-S
Simple Head
1.18
30.83
SeaFormer-S
Light Head
1.15
32.82
TopFormer-B
Simple Head
1.83
33.43
SeaFormer-B
Light Head
1.81
34.07
Table 9: Results on COCO-Sture Stuff test set. means FLOPs. We omit the latency in this table as the input resolution the same as that in table
14

---

Published as a conference paper at ICLR 2023
Backbone
AP
FLOPs
Params
ShufflcNetv2 Ma et al. (2018)
25.9
161G
10,4M
SeaForrer-1
31.5
160G
10.9M
MF151
34.2
161G
14.4M
MV3
27.2
162G
12.3M
SeaForrer-S
34.6
161G
13.3M
MF214
35.8
162G
15.2M
MF294
36.6
164G
16.1M
SeaFormer-B
36.7
164G
18.1M
ResNet50 He et al. (0016)
36.5
239G
37.7M
PVT-Tiny Wang et al (2021)
36.7
221G
23.0M
ConT-M Yan et al. (202T
37.9
217G
27.0M
SeaForrer-L.
39.8
185G
24.0M
Table 10: Results on COCO object detecion. MF denotes Mobile-Former Chen et al. (022b)) MV3 denotes MobileNetV3 Howard et al. (2019).
Fusion method
mIoU
Add directly
35.2
Multiply directly
35.2
Sigmoid add
34.8
Sigmoid multiply
35.8
Table 11: Ablation study fusion method on ADE20K val set.
E OBJECT DETECTION PERFORMANCE
To further demonstrate the generalization ability of our proposed SeaFormer backbone on other downstream tasks, we conduct object detection task on COCO dataset.
Setup We use RetinaNet Lin al. (2017) (one-stage) as the detection framework and follow the standard settings to use SeaFormer as backbone to generate feature pyramid at multiple scales. All models are trained on train2017 split 12 epochs (1x) from ImageNet pretrained weights.
Results From the table 10 We can observe that our SeaFormer achieves superior results on detec- tion task which further demonstrates the strong generalization ability of our method.
F ADDITIONAL ABLATION STUDY
In addition the ablation study in the submission paper, we investigate the effect of fusion method in fusion block in Figure
F.1 THE INFLUENCE OF FUSION BLOCK DESIGN
We set four fusion methods, including "dd directly". "Multiply directly". "Sigmoid add" and "Sig- moid multiply". directly means features from context branch and spatial branch X directly. Sig- moid X means feature from context branch goes through sigmoid layer and X feature from spatial branch.
From the Table II performance. Sigmoid multiply is our optimal fusion block choice.
can that replacing sigmoid multiply with other fusion methods hurts
F.2 EFFECTIVE AND EFFICIENCY OF SEA ATTENTION
To verify the effectiveness and efficiency of SEA attention based on our designed pipeline, we ex- periment with convolution, Global attention, Local attention, Axial attention and three convolution
15

---

Published as a conference paper at ICLR 2023
Method
Params
FLOPs
Latency
Top1
mloU
Cony
1.6M
0.59G
38ms
66.3
32.8
Local
1.3M
0.60G
48ms
65.9
32.8
Axial
166
0.63G
44ms
66.9
33.7
Global
1.3M
0.61G
43ms
66.7
34.2
ACmix
1.3M
0.60G
54ms
66.0
33.1
MixFormer
1.3M
0.60G
50ms
66,8
33.8
SeaFormer
1.7M
0.60G
40ms
67.9
35.8
Table 12: Performance of different self self-attention modules our designed pipeline ImageNet- 1K and ADE20K datasets.
Model
mloU
FP32
FPI6
TopFormer-T T
34.6
43ms
23ms
SeaFormer-T
35.8
40ms
22ms
TopFormer-S
37.0
74ms
41
SeaFormer-S
39.4
67ms
36ms
TopFormer-B
39.2
110ms
60ms
SeaFormer-B
41.0
106ms
56ms
SeaFormer-L
43.7
367ms
186ms
Table 13: Performance comparison on ADE20K val set under different precision.
enhanced attention methods including our SEA attention, ACmix and MixFormer. The ablation experiments are organized in seven groups. Since the resolution of computing attention is rela- tively small, the window size in Local attention, ACmix, and MixFormer is set to 4. We adjust the channels when applying different attention modules to keep the FLOPs aligned and compare their performance and latency. The results are illustrated Table
As demonstrated in the table, SEA attention outperforms the counterpart built on other efficient attentions. Compared with global attention, SEA attention outperforms by +1.2% Topl accuracy on ImageNet- IK and +1.6 mloU on ADE20K with less FLOPs and lower latency. Compared with similar convolution enhanced attention works, ACmix and MixFormer, our SEA attention obtains better results on ImageNet-1K and ADE20K with similar FLOPs but lower latency. The results indicate the effectiveness and efficiency of SEA attention module.
G PERFORMANCE UNDER DIFFERENT PRECISION OF THE MODELS
Following TopFormer, measure the latency in the submission paper on single Qualcomm Snap- dragon 865, and only an ARM CPU core is used for speed testing. No other means of acceleration, e.g., GPU quantification, is used. We provide more comprehensive comparison demonstrate the necessity of our proposed method. We test the latency under different precision of the models. From the table 13 it can be seen that whether it is full precision or half precision the performance of SeaFormer is better than that of TopFormer.
H VISUALIZATION
H.1 ATTENTION HEATMAP
To demonstrate the effectivvness of detail enhancement in our squeeze-enhanced Axial attention (SEA attention), we ablate our model by removing the detail enhancement. We visualize the atten- tion heatmaps of the two models in Figure Without detail enhancement, attention heatmaps from solely SA attention appears to be axial strips while our proposed SEA attention able to activate the semantic local region accurately, which is particularly significant in the dense prediction task.
16

---

Published as a conference paper at ICLR 2023
(a) Squeeze Axial attention heatmaps
(b) Squecze enhaneed Axial attention heatmaps
Figure 5: The visualization attention heatmaps from the model consisting of squeeze Axial atten- tion without detail enhancement (first row) and SeaFormer (second row). Heatmaps are produced by averaging channels the features from last attention block, normalizing to [0, 255] and up- sampling to the image size.
H.2 PREDICTION RESULTS
We show the qualitative results and compare with the alternatives on the ADE20K validation from two different perspectives. First we compare with mobile-frienndl rival TopFormer /Zhang et al. (2022c) with similar FLOPs and latency Figure 6 Besides, we compare with the Transformer- based counterpart SegFormer-B1 Xie et al. (202 in Figure 7 In particular, our SeaFormer-L. has lower computation cost than SegFormer-Bi As shown in both figures, we demonstrate better segmentation results than both the mobile counterpart and Transformer-basse approach.
I LIMITATIONS AND SOCIETAL IMPACT
The mobile-friendll friendly segmentation is deeply related the industrial application edge computation platforms, while few academic attempts are made to meet the requirement of the industry. We test method results to the alternatives. We believe our work can lead to expected and unexpected innovations in both academia and industry.
Qualcomm Snapdragon 865 processor (Fig.1 main paper) and shows superior
However, our system is not perfect yet and hence not fully trustworthy in real-world deployment. Also, the current system on mobile semantic segmentation and image classification tasks. New mobile-friendly method for more downstream tasks and extended GPU systems will be studied in the future.
exhaustively evaluated and tested due limited resources. We focus
17

---

Published as a conference paper at ICLR 2023
(a) Ground Truth
(b) TopFormer-B B
Zhang al.
2022c
(c) SeaFormer-B (Ours)
Figure 6: Visualization of prediction results on ADE20K val set.
18

---

Published as a conference paper at ICLR 2023
(a) Ground Truth
SegFormer
Xie et al. 2021
(c) SeaFormer-L (Ours)
Figure 7: Visualization of prediction results on ADE20K val set.
19


Reviews:
[{"review_id": "eWHVeqJzl6d", "reviewer": "Reviewer_xU22", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper present an incremental improvement of axial attention. Instead of performing attention across the two axes for all pixels, feature maps are collapsed along the horizontal and vertical axes into two vectors. Self-attention is performed on these and the updated vectors can then be distributed across the original feature map at the input resolution. An additional detail enhancement block provides the original context of the feature map yielding the final output of a squeeze-enhance axial attention block. It is combined with a backbone (called the STEM) relying on MobileNet V2 blocks and some fusion modules which provide higher resolution information. The resulting SeaFormer architecture is evaluated for semantic segmentation and image classification, where there is a strong emphasis on the inference speed of the overall network, including when measured on mobile devices.", "summary_of_the_review": "There are some compelling experiments show that some interesting aspects of the proposed attention mechanism exist, however, I would like to actually get a better understanding why it apparently works so much better and what the limitations are. Overall I'm thus a bit torn when it comes to accepting the paper or rejecting it. However, depending on the other reviews and rebuttal, I'm willing to improve my rating.", "strength_and_weaknesses": "Strengths:\n- The results show that the overall architecture is indeed somewhat better and faster in the evaluated settings, even when actually running on a mobile device.\n- The proposed architecture, specifically the new attention block does seem to have some novelty, however, it is fairly small incremental when comparing it to the axial attention.\n\nWeaknesses:\n- I'm mainly a bit surprised by some of the results. When looking at table 4, the clear take-away message is that the newly proposed attention is better than all other attentions. I'm assuming that the other attention mechanisms have been adjusted in such a way to keep the FLOP count roughly consistent, which could be a reasonable explanation why they don't outperform this fairly strong approximation of the vanilla global attention. But I find the investigation of this somewhat lacking. Does this now mean that we should start to use the new attention mechanism everywhere, by simply scaling it up to match the FLOP counts of the original global attention? Or is there some limit to this? Given that the main contribution of this paper is the squeeze-enhance axial attention, I would like to gain some more insights into why it seemingly works better than the global attention. However, I can't really gain these insights from the paper.\n- Most of the actually interesting results are in the supplementary and I don't understand why the authors chose to write the paper like this. The supplementary actually has results on cityscapes, the only dataset used for experiments with a proper private test set. Furthermore it contains a more detailed analysis of the runtime, which is one of the main motivations for the whole idea of the paper. And finally, it also contains results of experiments with the proposed attention block in a SWIN former. I think this experiment is crucial to show that the proposed novelty actually has some form of generality. However, none of these results are even mentioned in the main paper, I don't really understand why it would be written in this way.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dXa7X8xGvik", "reviewer": "Reviewer_sRfJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presented a new mobile ViT architecture (SeaFormer) for semantic segmentation. The authors improved the computational cost of ViT architecture by proposing an efficient attention block, and light segmentation head. They showed solid results on ADE20K, Pascal Context, and COCO-stuff segmentation datasets.\n\nPost rebuttal:\n\nThanks for clarifying the cityscapes results, and ImageNet. My rating will stay the same. ", "summary_of_the_review": "The paper presented SeaFormer: a new mobile-ViT model for semantic segmentation. The authors proposed an efficient architecture that model spatial and global information utilizing convnets, and an efficient axial attention block. Empirically, they showed solid results on ADE20K, Pascal Context, and COCO-stuff. This work benefit the effort to push ViT based models to be more mobile friendly.", "strength_and_weaknesses": "Strength:\n\n* The paper presents a new architecture for mobile semantic segmentation using ViT. Such dense prediction task is very challenging, and usually requires global attention blocks in ViTs to model long range dependencies. The paper presented an efficient attention block utilizing axial attention to tackle segmentation task.\n\n* The paper presented solid numbers on 3 different segmentation datasets: ADE20K, Pascal Context, and COCO-Stuff. They also showed promising results on ImageNet for the image classification task.\n\nWeaknesses:\n\n* The paper proposed an efficient model for segmentation, but they didn't show any results on CityScapes to show the efficiency on dataset with large input size.\n* The paper forgot to compare to some of recent work (e.g. EfficientFormer [1]).\n* The paper listed a whole subsection (Shared STEM) in the method section which can imply it is a new work, instead it is based on previous work.\n\n[1] Li, Y., Yuan, G., Wen, Y., Hu, E., Evangelidis, G., Tulyakov, S., Wang, Y. and Ren, J., 2022. EfficientFormer: Vision Transformers at MobileNet Speed. arXiv preprint arXiv:2206.01191.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b5LldCIbjuA", "reviewer": "Reviewer_6wEd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper presents SeaFormer: a new transformer architecture designed to reduce the well-known complexity problem with the vanilla structure. The resulting architecture, O(HW), is applied in a semantic segmentation task for mobile devices. Experiments shows SOTA results in a latency/mIoU. An ablation study shows the impact of each proposed contribution An additional short experiment is proposed to show that this new transformer architecture is generic and can be used in a classification task.   ", "summary_of_the_review": "The main contribution of the paper is a new attention block with O(WH) complexity, which combines an axial attention branch with a high resolution convolutional block. This block is used into an semantic segmentation task designed for mobile devices. Experiments are convincing. As I pointed before, it should be interesting to extend experiments in order to show the genericity of the approach regarding other tasks and GPU systems. ", "strength_and_weaknesses": "Strength:\n- The proposed transformer is O(WH), which breaks the original drawback of transformers. Even if many existing models already proposed modification in order to do similar optimisation, this paper combines an Axial attention branch (context branch) with a high resolution convolution branch (spatial branch). Both experiments and ablation study show that this is a relevant strategy. \n- Both experiments and ablation study are convincing for semantic segmentation. \n\nWeaknesses: \n- The main contribution seems to be the new transformer architecture combining context and spatial branches, resulting to a generic bloc that breaks the complexity. This contribution is mainly presented in the context of mobile segmentation application. I wonder if considering in a more generic way this contribution would implies more impact: 1) As you point with the small experiment on classification, this is very generic and can be used for classification, detection, segmentation (semantic, instances, ...); and 2) blocks are designed for mobile applications (MobileNet blocks) and could be extended to GPU systems with more classical operations without changing the O(WH) complexity. Extending experiments in this way would be interesting with may be more impact for the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SfvLKl7FTh", "reviewer": "Reviewer_wqtJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a model called squeeze-enhanced Axial Transformer (SeaFormer) for mobile semantic segmentation. The model is based on the previous TopFormer with two-branch architecture and integrates axial attention and enhancement into an attention block. The proposed attention block reduced memory and time complexity a lot. The proposed model achieved encouraging performance on several semantic segmentation datasets, i.e., ADE20K, Pascal Context and COCO-stuff.", "summary_of_the_review": "This paper presents a model based on previous work with incremental achievement. The proposed axial attention and enhancement is not novel. The evaluation of the important datasets, cityscape, is not clear. Thus, I consider this paper may not be ready to be published in ICLR.", "strength_and_weaknesses": "Strength:\n+ The squeeze axial attention saves a lot of cost, enabling real-time performance.\n+ To overcome the loss of details in the global context extraction, there is a design for detail enhancement using depth-wise convolution to aggregate local spatial details.\n+ The experiments show gains from TopFormer.\n\nWeakness\n- The pipeline of the proposed method is most built on TopFormer, which also have dual branches of spatial feature pyramid and global semantics, and fusion blocks. The only difference is the axial attention and detail enhancement kernel. However, the axial attention is not novel, and was utilized in previous segmentation works, e.g., Axial-DeepLab, Max-DeepLab. What’s the difference between the proposed axial attention and previous works? For the detail enhancement kernel, is it in the SeaFormer layer in context branch or spatial branch? The Fusion block in Figure 3 is the same as the one in Figure 2?\n- Although the proposed model have about 1% mIoU gain over TopFormer, it is not the state-of-the-art. It is not suitable to claim this method achieves the state-of-the-art performance.\n- A lot of citations are from arXiv, but some are already published.\n- For the results on Cityscape dataset in supplemental material, the latency seems higher than TopFormer with same backbone. Why not present results of the half resolution or more backbones for more comparison?\n- In table 3, it may be better to present the latency of components (although found in Supplemental material).\n- Better to separate Table 2, Table 4. No (a, b, c) annotation for Table 4. Table 4 is referenced before Table 3.\n- The number of parameters of the SeaFormer is about 1.2 ~1.6 times of TopFormer’s. What are the additional cost?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster