MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models
Authors: Chenglin Yang
Siyuan Qiao
Qihang Yu
Xiaoding Yuan
Yukun Zhu
Alan Yuille
Hartwig Adam
Liang-Chieh Chen
Venue: ICLR 2023 poster (2023)

Abstract: This paper presents MOAT, a family of neural networks that build on top of MObile convolution (i.e., inverted residual blocks) and ATtention. Unlike the current works that stack separate mobile convolution and transformer blocks, we effectively merge them into a MOAT block. Starting with a standard Transformer block, we replace its multi-layer perceptron with a mobile convolution block, and further reorder it before the self-attention operation. The mobile convolution block not only enhances the network representation capacity, but also produces better downsampled features. Our conceptually simple MOAT networks are surprisingly effective, achieving 89.1% / 81.5% top-1 accuracy on ImageNet-1K / ImageNet-1K-V2 with ImageNet-22K pretraining. Additionally, MOAT can be seamlessly applied to downstream tasks that require large resolution inputs by simply converting the global attention to window attention. Thanks to the mobile convolution that effectively exchanges local information between pixels (and thus cross-windows), MOAT does not need the extra window-shifting mechanism. As a result, on COCO object detection, MOAT achieves 59.2% AP$^{\text{box}}$ with 227M model parameters (single-scale inference, and hard NMS), and on ADE20K semantic segmentation, MOAT attains 57.6% mIoU with 496M model parameters (single-scale inference). Finally, the tiny-MOAT family, obtained by simply reducing the channel sizes, also surprisingly outperforms several mobile-specific transformer-based models on ImageNet. The tiny-MOAT family is also benchmarked on downstream tasks, serving as a baseline for the community. We hope our simple yet effective MOAT will inspire more seamless integration of convolution and self-attention. Code is publicly available.

Published as a conference paper at ICLR 2023
MOAT: ALTERNATING MOBILE CONVOLUTION AND ATTENTION BRINGS STRONG VISION MODELS
Chenglin Yang', Siyuan Qiao², Qihang Yu', Xiaoding Yuan', Yukun Zhu?,
Alan Yuille', Hartwig Adam², Blang-Chieh Chen'
'Te Johns Hopkins University Google Research
ABSTRACT
This paper presents MOAT, family of neural networks that build top of MObile convolution (i.e., inverted residual blocks) and ATtention. Unlike the current works that stack separate mobile convolution and transformer blocks, effectively merge them into a MOAT block. Starting with a standard Transformer block, we replace its multi-layer perceptron with a mobile convolution block, and further reorder it before the self self-attention operation. The mobile convolution block not only enhances the network representation capacity, but also produces better downsampled features. Our conceptually simple MOAT networks are surprisingly effective, achieving with ImageNet-
89.1% 81.5% top-1 accuracy on ImageNet-IK 1K ImageNet- 22K pretraining. Additionally, MOAT can be seamlessly applied to downstream tasks that require large resolution inputs by simply converting the global attention window attention. Thanks to the mobile convolution that effectively exchanges local information between pixels (and thus cross windows, MOAT does not need the extra window-shifting mechanism. As a result, COCO object detection, MOAT achieves 59.2% Apbox with 227M model parameters (single-scale inference, and hard NMS), and on ADE20K semantic segmentation, MOAT attains 57.6% mIoU with 496M model parameters (single-scale inference). Finally, the tiny- MOAT family, obtained by simply reducing the channel sizes, also surprisingly outperforms several mobile-specific transformer-based models on ImageNet. The tiny-MOAT family also benchmarked on downstream tasks, serving as a baseline for the community. We hope our simple yet effective MOAT will inspire more seamless integration of convolution and self-attention. Code publicly available.'
INTRODUCTION
The vision community has witnessed the prevalence of self-attention (Bahdanau et al., 2015) and Transformers (Vaswani et al., 2017). The success of Transformers natural language processing motivates the creation of their variants for vision recognition. The Vision Transformer (ViT) (Doso- vitskiy et al., 2021) has great representation capacity with global receptive field. However, requires pretraining large-scale proprietary dataset (Sun et al., 2017). Its unsatisfying performance, when trained with a small number of images, calls for the need of better training recipes (Touvron et al., 2021a; Steiner al., 2021) or architectural designs (Liu et al., 2021; Graham al., 2021). On the other hand, ConvNet has been the dominant network choice since the advent of AlexNet (Krizhevsky al., 2012) 2012. Vision researchers have condensed the years of network design experience into multiple principles, and have started to incorporate them to vision transformers. For example, there are some works adopting the ConyNet's hierarchical structure to extract multi-scale features for vision transformers (Liu et al., 2021; Fan et al., 2021; Wang et al., 2022), and others proposing to integrate the translation equivariance of convolution into transformers (Graham et al., 2021; d' A'soolii et al., 2021; Xiao et al., 2021).
Along the same direction of combining the best from Transformers and ConvNets, CoAtNet (Dai et al., 2021) and Mobile (Mehta & Rastegari, 2022a) demonstrate outstanding performance by
WWok done while an intern at Google. Official code in TensorFlow: https: //gi tuub....cc www.bb.ccommooggeeecee htaa://googleereaerr/////eeeeee seneeerrccddeeppapb/

---

Published as a conference paper at ICLR 2023
stacking Mobile Convolution (MBConv) blocks (i.e., inverted residual blocks (Sandler et al., 2018)) and Transformer blocks (i.e., a self-attention layer and a Multi-Layer Perceptron (MLP)). However, both works focus on the macro-level network design. They consider MBConv and Transformer blocks as individual separate ones, and systematically study the effect of stacking them to strike a better balance between the remarkable efficiency of MBConv and strong capacity of Transformer.
In this work, on the contrary, we study the micro-level building block design by taking a deeper look at the combination of MBConv and Transformer blocks. We make two key observations after a careful examination of those blocks. First, the MLP module in Transformer block is similar to MBConv, as both adopt the inverted bottleneck design. However, MBConv is a more powerful operation by employing one extra 3 x 3 depthwise convolution (to encode local interaction between pixels), and more activation (Hendrycks & Gimpel, 2016) and normalization (loffe & Szegedy, 2015) are employed between convolutions. Second, to extract multi-scale features using Transformer blocks, one may apply the wverage-poolingg (with stride 2) to input features before the self-attention layer. However, the pooling operation reduces the representation capacity of self-attention. Our observations motivate us to propose a novel MObile convolution with ATtention (MOAT) block, which efficiently combines MBConv and Transformer blocks. The proposed MOAT block modifies the Transformer block by first replacing its MLP with a MBConv block, and then reversing the order of attention and MBConv. The replacement of MLP with MBConv brings more representation capacity the network, and reversing the order (MBConv comes before self-attention) delegates the downsampling duty to the strided depthwise convolution within the MBConv, learning better downsampling kernel.
We further develop a family of MOAT models by stacking and increasing the channels of network blocks. Surprisingly, our extremely simple design results in remarkable impact. On the challenging ImageNet-IK K classification benchmark (Russakovsky et al., 2015), our model (190M parameters) achieves 86.7% top-1 accuracy without extra data. When further pretraining on ImageNet-22K,, our best model (483M parameters) attains 89.1% / 81.5% top-1 accuracy on ImageNet-IK (Tab. 2) ImageNet- 1 K-V2 (Tab. 9), setting a new state-of-the-arr.
Additionally, MOAT can be seamlessly deployed to downstream tasks that require large resolution inputs by simply converting the global attention to non-overlapping local window attention. Thanks to the MBConv that effectively exchanges local information between pixels (enabling cross-window propagation), MOAT does not need the extra window-shifting mechanism (Liu et al., 2021). As a result, on COCO object detection (Lin et al., 2014) and ADE20K semantic segmentation (Zhou et al., 2019), MOAT shows superior performances. Specifically, on COCO object detection (Tab. 3). our best model (227M parameters), achieves 59.2% A with single-scale inference and hard NMS, setting a new state-of-thh-ard Vasconcelos, 2018; He et al., 2017). On ADE20K semantic segmentation (Tab. 4), our best model (496M parameters), adopting DeepLabv3+ (Chen et al., 2018), attains 57.6% mIoU with single-scale inference, also setting a new state-of-the-art in the regime of models using input size 641 x 641.
the regime of model size 200M with Cascade Mask R-CNN (Cai &
Finally, to explore the scalability of MOAT models, we simply scale down the models by reducing the channel sizes (without any other change), resulting in the tiny-MOAT family, which also surprisingly models, such as Mobile-Former (Chen et al., 2022c)
outperforms mobile-specific transformer-base and MobileVIT's (Mehta & Rastegari, 2022a;b). Specifically, in the regime of model parameters 5M, 10M, and 20M, our tiny MOAT outperforms the concurrent MobileViTv2 (Mehta & Raste- gari, 2022b) by 1.1%, 1.3%, and 2.0% top-1 accuracy on ImageNet- classification benchmark (Tab. 5). Furthermore, we benchmark tiny-MOAT on COCO object detection and ADE20K semantic segmentation.
In summary, our method advocates the design principle of simplicity. Without inventing extra complicated operations, the proposed MOAT block effectively merges the strengths of both mobile convolution and self-attention into one block by a careful redesign. Despite its conceptual simplicity, impressive results have been obtained on multiple core vision recognition tasks. We hope our study will inspire future research seamless integration of convolution and self-attention.
2

---

Published as a conference paper at ICLR 2023
batch norm
layer norm
batch norm
cony 1x1,
self selfattienion
conv 1x1, 4c
betch norm, gelu
betth norm, gelu
depttwwks corv
depthiiis conv
betch norm, gelu
layer norm
betch norm, gelu
conv
giobal pool
ge
howish
conv 1xl,
conv 14),
ayer norm
32
solf- colf-attattonn
lxl.
(a) MBConv block
(b) Transformer block
(c) MOAT block
Figure 1: Block comparison. (a) The MBComv block (Sandler et al., 2018) employs the inverted bottleneck design with depthwise convolution and squeeze-and-excitation (Hu al., 2018) applied the expanded features. (b) The Transformer block (Vaswani al., 2017) consists of self attention module and a MLP module. (c) The proposed MOAT block effectively combines them. The illustration assumes the input tensor has channels c.
METHOD
Herein, review the Mobile Convolution (MBConv) (Sandler et al., 2018) and Transformer (Vaswani et al., 2017) blocks before introducing the proposed MOAT block. We then present MOAT, family of neural networks, targeting different trade-off's between accuracy and model complexity.
2.1 MOBILE CONVOLUTION AND TRANSFORMER BLOCKS
MBCony block. Also known as the inverted residual block, the Mobile Convolution (MBConv) (San- dler al., 2018) block (Fig. 1 (a)) is an effective building block that has been widely used in mobile models (Howard et al., 2019: Mehta & Rastegari, 2022a) or efficient models (Tan & Le, 2019; Dai al., 2021). Unlike the bottleneck block ResNet (He et al., 2016a), the MBComv block employs the design of an "inverted bottleneck", together with the efficient depthwise convolution (Howard et al., 2017). Specifically, a 1 convolution first applied to expand the input channels by factor of 4. Then, x 3 depthwise convolution is used to effectively capture the local spatial interactions between pixels. Finally, the features are projected back to the original channel size via a 1 x 1 convolution, enabling residual connection (He et al., 2016a). An optional Squeeze-ande -et---ndd----iiii (SE) (Hu et al., 2018) module (which uses the global information to re- reigh the channel activation) may also be used after the depthwise convolution, following MobileNetV3 (Howard al., 2019). Note that one could tune the channel expansion ratio and depthwise convolution kernel size for better performance. We fix them throughout the experiments for simplicity.
Formally, given an input tensor E #x xc (H, W,C C. are its height, width, and channels), the MBConv block is represented as follows:
MBConv(x) (N2 S AG)
(1xx) GeL.U(BN(Conr(z))
(2)
D(x) GeLUUBBNDeptthConv (x))),
(3)
S(x) (MLP)G))()))
(4)
N2(x:) (z))
(5)
where BN, GeLU, GAP, and MLP stand for Batch Normalization (loffe Szegedy, 2015), Gaussian
error Linear Unit (Hendrycks & Gimpel, 2016), Global Average Pooling, and Multi-Layer Perceptron (with reduction ratio 4 and hard-swish (Ramachandran et al., 2017)), respectively. The MBCony block consists of four main functions: N₁. D. S, and A2, which correspond to the 1 x 1 convolution for channel expansion (by 4x), x 3 depthwise convolution, squeeze-and- andeeezeaandeexiattiooon (Hu et al., 2018) (a is the sigmoid function), and convolution for channel projection (by 4x), respectively.
3

---

Published as a conference paper at ICLR 2023
Transformer block. The Transformer (Vaswani al., 2017) block (Fig. 1 (b)) is powerful building block that effectively captures the global information via the data-dependent self-attention operation. It consists of two main operations: self-attention and MLP. The self-attention operation computes the attention map based on the pairwise similarity between every pair of pixels the input tensor, thus enabling the model's receptive field encompass the entire spatial domain. Additionally, the attention map dynamically depends on the input, enlarging the model's representation capacity (unlike the convolution kernels, which are data-independent), The MLP operation contains two x 1 convolutions, where the first one expands the channels (by 4x), the second one shrinks back the channels, and GeLU non-linearity is used in-between.
Formally, given an input tensor z E R"xxxxx xc, the Transformer block is represented as follows:
Transformer(s)) x (M2 o M1 o Attn)(LN((a)) Mi Gel.U((on
M2(2)) = Conv(x);
where LN and Attn denote the Layer Normalization (Ba et al., 2016), and self-attention (Vaswani et al., 2017). The self-attention operation also includes a residual connection (He et al., 2016a), which not shown in the equations for simplicity, while the MLP operation is represented by two functions M, and M2, which correspond to the 1 x convolution for channel expansion (by 4x) and x convolution for channel projection, respectively.
2.2 MOBILE CONVOLUTION WITH ATTENTION (MOAT) BLOCK
Comparing MBConv and Transformer blocks. Before getting into the architecture of our MOAT block, is worthwhile to compare the MBConv (Sandler et al., 2018) and Transformer (Vaswani et al., 2017) blocks, which helps to understand our design motivations. Specifically, we make the following key observations.
First, both MBConv and Transformer blocks advocate the "inverted bottleneck" design, where the channels of input tensors are expanded and then projected by 1 x convolutions. However, MBCony additionally employs 3 3 depthwise convolution between those two 1 x 1 convolutions, and there are both batch normalization and GeLU activation between the convolutions.
Second, capture the global information, the MBConv block may employ gqueeze-and-Bee and-Exxitation (SE) module, while the Transformer block adopts the self-attention operation. Note that the SE module squeezes the spatial information via global average pooling, while the self-attention module maintains the tensor's 's spatial resolution.
Third, the downsampling operation is performed at different places within the block. To down- sample the features, the standard MBCоnv block uses the strided depthwise convolution, while the Transformer block, deployed in the modern hybrid model CoAtNet (Dai et al., 2021), adopts an average-pooling operation before the self-attention.
MOAT block. Given the above observations, we now attempt to design new block that effectively merges the best from both MBComv and Transformer blocks. We begin with the powerful Transformer block, and gradually refine over
Based on the first observation, both MBConv and Transformer blocks employ the "inverted bottleneck" design. Since depthwise convolution could effectively encode local interaction between pixels, which is crucial for modeling the translation equivariance in ConvNets, we thus start to add the depthwise convolution to Transformer's MLP module. However, we did not observe any performance improvement until we also added the extra normalization and activations between convolutions.
For the second observation, we simply do not add the SE module to the MBConv block. The self-attention operation is kept to capture the global information.
We found the third observation critical. The downsampling operation (average-pooling) right before the self-attention operation in Transformer block slightly reduces its representation capacity. On the other hand, the MBCony block is well-designed for the downsampling operation with the strided depthwise convolution, which effectively learns the downsampling convolution kernel for each input channel. Therefore, we further reorder the "inverted bottleneck" (containing depthwise convolution) before the self-attention operation, delegating the downsampling operation depthwise convolution.
4

---

Published as a conference paper at ICLR 2023
In this way, need extra downsampling layer like average-pooling in CoAtNet (Dai et al., 2021), 1), or patch-embedding layers in Swin (Liu et al., 2021) and ConvNeXt (Liu et al., 2022b). Finally, results in our MObile convolution with ATtention (MOAT) block, illustrated in Fig. (c).
Formally, given an imput tensor a E #B#xxxx xC the MOAT block is represented as follows:
MOAT ( x + (Attn o N2 o D o (BN(())))
where MBConv (w/o SE) contains functions Mi (Eq. 2), D (Eq. 3), and N2 (Bq. 5), and Attn denotes the self-attention operation. The MOAT block then simply consists of MBConv (w/o SE) and the self-attention operation, successfully combining the best from the MBConv block and Transformer block into one (which we will show empirically).
2.3 META ARCHITECTURE
Macro-level network design. After developing the MOAT block, then study how to effectively stack them to form our base model. We adopt the same strategy as the existing works (Liu et al., 2021; Wang al., 2021b; Graham et al., 2021; Xiao al., 2021; Dai al., 2021; Mehta & Rastegari, 2022a). Specifically, we summarize several key findings from those works, and use them as design principles of meta architecture.
Employing convolutions in the early stages improves the performance and training conver- gence of Transformer models (Wu et al., 2021; Graham et al., 2021; Xiao et al., 2021).
The Mobile Convolution (MBConv) (Sandler al., 2018) blocks are also effective building blocks in the hybrid Trnn-Transformer models (Dai et al., 2021; Mehta & Rastegari, 2022a).
Extracting multi-scale backbone features benefits the downstream tasks, such as detection and segmentation (Liu al., 2021; Wang et al., 2021b; Fan et al., 2021; Heo al., 2021). )).
As result, our meta architecture consists of the convolutional stem, MBComv blocks, and MOAT blocks. Additionally, through the ablation study in the appendix, we found the layer layout proposed by CoAtNet-1 (Dai et al., 2021) effective. We thus follow their layer layout, resulting in our base model MOAT-1. To form the MOAT model family, we then scale down or up MOAT-1 in the dimensions of number of blocks and number of channels, number of blocks in the third and fourth stages (out of five stages). The downsampling operation is performed in the first block of each stage. Note that our base model MOAT-1 and CoAtNet- I share the same layer layout and channel sizes. However, we take different scaling strategy: our MOAT is scaled up (or down) by alternatively increasing the depth and expanding the width between variants.
shown in Tab. 1. We only scale the
Table 1: MOAT variants differ in the number blocks and number of channels in each stage.
MOAT-O
MOAT-1
MOAT-2
MOAT-3
MOAT-4
NNN-MOAT-((.112.33
block
stride
B
B
C
C
C
c
B
conv
6%
MBComv
2
96
2
96
128
2
160
256
2
32
40
56
80
MBConv
MOAT
16
7
384
14
384
14
512
28
640
28
1024
7
128
163
224
320
MOAT
2
768
768
1024
1280
2048
2
256
320
448
640
EXPERIMENTAL RESULTS
In this section, we show that MOAT variants are effective on the ImageNet- (Russakovsky et al., 2015) image classification. We then deploy them to other recognition tasks, including COCO object detection (Lin et al., 2014), instance segmentation (Hariharan et al., 2014). and ADE20K (Zhou et al., 2019) semantic segmentation. MOAT can be seamlessly applied to downstream tasks. For small resolution inputs, directly fine-tune the global attention, while for large resolution inputs, simply convert the global attention to non-overlapping local window attention without using extra window-shifting mechanism. The detailed experiment setup could be found in the appendix.
5

---

Published as a conference paper at ICLR 2023
Table 2: Performance ImageNet-IK. 1K only: Using ImageNet- IK only. 22K 1K: ImageNet-222h pretraining and ImageNet- IK fine-tuning. Tab. shows comparisions with more SOTA methods and Tab. reports the performances on inageNet-IK-Ve
model
oval size
params
FLOPs
ImageNet-18 IK top-1 accuracy
IK only
22K+1K
FfficiemNNe122... (Tan & 2021)
480"
120M
53B
85.7
ElliceenNetV2-X3 (Tan & Le. 2021)
480"
208M
94B
87.3
ConvNeXt: T (1iu et al,, 20226)
224
29M
4.5B
82.1
82.9
ConvNets
ConvNeXt-1. (1in al,, 2022h)
384"
85.5
87.5
CunvNeXi- XL (Liu et al., 2022b)
384"
350M
179.0B
87.8
argeLarge (Wang al., 2021b)
224"
61.4M
9.8R
81.7
VIIs
Swin-T (Liu et al., 2021)
224"
28M
4,5B
81.3
Swin- (Liu et al., 2021)
3842
197M
103,98
87.3
SwinV2-L 2021)) 202.)
384"
87.7
MViTv2-HI (Li al., 2022)
512"
667M
763.5B
88.8
PVTv2-B5 (Wang al., 2022)
224"
82M
11.88
83.8
Hybrid
MaxViT-XL (Tu et 2022)
2022)
512"
475M
535,2B
88,7
CoANNet- (Dai al., 2021)
224
25M
4.2B
81.6
CoANNer-3 (Dai 2021)
384"
85.8
87.6
CoAtNet-4 (Dai 2021)
512"
275M
360.9B
88.6
MOAT-0 0
224
27.8M
5.7B
83.3
83.6
MOAT-1
224
41.6M
9.1B
84.2
84.9
MOAT-2
224"
86.0
MOAT-3
224"
190.0M
44.9B
85.3
86.8
Hybrid
MOAT-0
384"
27.8M
18.2B
84.6
85.7
(ours)
MOAT 1
384"
41.6M
29.6B
85.9
87.0
MOAT-2
384
73.4M
54.3B
86.2
87.5
MOAT-3
384"
86.5
88.2
MOAT-1
512"
87.2
MOAT-2
512"
73.4M
104.6B
86.5
87.7
MOAT 3
512"
190.0M
271.0B
86.7
88.4
MOAT-4
512"
483.2M
648.5B
89.1
86
86
MOAT-3
MDAT-3
85
CoAtNNet-3
CoAtNer-3 ConvNeXILL
Top sccurscy
8
ConvNeXt:L
Top-1 accuracy
84
1
Swin
89
MOAT
CoAtNet
ConwNekt Swin
ConwNeX:
0
Swin
150
200
10
20
30
40
50
Parameters (M)
FLOPs (B)
Figure 2: Parameters accuracy using ImageNet- IK only with input size 224.
Figure 3: FLOPs vs. accuracy using ImageNet- KK only with input size 224.
89
89
MOAT-3
88
CoAINei
CoAtNer-3
87
2x
accuacy
Top accuracy
86
0
Swin B
0
Swin
MOAT ConvNekL Swin
MOAT
CoAINet.
ConNNeXL-T
ConvNeXET
ConWNet
Swin
83
50
100
150
200
Parameters (M]
FLOPs (B)
Figure 4: Parameters vs. accuracy using ImageNet- 22K and ImageNet-I1 IK with input size 384.
Figure 5: FLOPs vs. accuracy using ImageNet-22X and ImageNet- with input size 384.

---

Published as a conference paper at ICLR 2023
ImageNet Image Classification. In Tab. 2, we include the current state-of-art methods in the categories of ConwNets, ViTs and Hybrid models. At similar model costs (parameters or FLOPs), our MOAT models consistently outperform all of them. Specifically, with the ImageNet-IK IK data only and input size 224, for light-weight models, our MOAT-O significantly outperforms ConvNe (Liu al., 2022b), Swin-T (Liu et al., 2022b), and CoAtNet-O (Dai et al., 2021) by 1.2%, 2.0%, and 1.7%, respectively. For large-scale models using input size 384, MOAT-3 is able to surpass ConvNeXI-L CoAtNet-3 by 1.0% and 0.7%, respectively. With the ImageNet-22K pretraining and input size 384, the prior arts ConvNeXt-L., Swin-L, and CoAtNet-3 already show strong performances (87.5%, 87.3% and 87.6%), while our MOAT-3 achieves the score of 68.2%, outperforming them by 0.7%, 0.9%. and 0.6%. respectively. For ImageNet-1K and input size 224, we plot the performances us. parameters and FLOPs in Fig. 2 and Fig. 3, respectively. For ImageNet-22K, pretraining and input size 384, we plot the performances vy. parameters and FLOPs in Fig. 4 and Fig. 5, respectively. In the figures, MOAT clearly demonstrates the best performance in all computation regimes. Finally, our largest model MOAT-4, with 222 and input size 512, further attains 89.1% accuracy.
COCO Detection. Tab. 3 summarizes the COCO object detection (box) and instance segmentation (mask) results. Our MOAT backbones significantly outperform the baseline methods, including Swin (Liu et al., 2021) and ConwNeXt (Liu et al., 2022b) across different model sizes. Specifically, our MOAT-0 outperforms Swin-T and ConyNeXt-T by 5.4% and 5.5% Apbox (3.7% and 3.7% A Our MOAT-1 surpasses Swin-S and ConvNeXI-S by 5.9% and 5.8% Apbox (4.3% and 4.0% Apmask). ). Our MOAT-2, with 110M parameters, is still 5.5% and 4.5% Aphux (3.5% and 2.4% Apmask) better than Swin-B and ConvNeXt-B. Finally, our MOAT-3, using 227M parameters, achieves 59.2% Apbax (50.3% APmask, ), setting new state-of-the-art in the regime of model size 200M that is built on top of Cascade Mask R-CNN (Cai & Vasconcelos, 2018; He et al., 2017). More comparisons with smaller input size can be found in Tab. 12. For tiny-MOAT, Any-MOAT-O// achieve the same performance as Swin-T/S and ConyNeXt-TTS but only use less than half of the parameters. Furthermore, tiny-MOAT-3 is pretrained with ImageNet- IK and attains 55.2 Apbox with 57M parameters, surpassing the ImageNet-225 pretrained Swin-L (53.9 Aphox with 254M parameters) and ConvNeXt-L (54.8 Aphox with 255M parameters).
Table 3: Object detection and instance segmentation on the COCO 2017 val set. We employ Cascade Mask-RCNN and single-scale inference (hard NMS). f: use ImageNet- 22K pretrained weights. When using tiny-MOAT series as backbones, most of the model parameters come from the decoder. More comparisons at input size 896 is reported in Tab. 12.
backbone
input size
parums
FLOPs
Ap
APhos
AP
Apmaa
Apmak
Appik
Swin-T
1280 x 800
86M
745B
50.5
69.3
54.9
43.7
66.6
47.1
Swin-S
1280 x 800
107M
838B
51.8
70.4
56.3
44.7
67.9
48.5
Swin-Bt
1280 800
145M
9828
53.0
71.8
57.5
45.8
69.4
49.7
Swin-L#
1280 x 800
254M
1382B
53.9
72.4
58.8
46.7
70.1
50.8
ConyNeXt-T
1280 x 800
86M
741B
50.4
69.1
54.8
43.7
66.5
47,3
ConvNeXI-S
1280 x 800
108M
827B
51.9
70.8
56.5
45.0
68.4
49.1
ConyNeXt-HI
1280 800
146M
9648
73.1
58.8
46.9
70.6
51.3
Con
1280 x 800
255M
1354B
54.8
73.8
59.8
47.6
71.3
51.7
Miny-MOAT-0
1344 x 1344
41M
612B
50.5
69.3
56.0
43.3
66.6
47.3
Miny-MOAT-I
1344 1314
42M
628B
51.9
71.6
56.1
44.6
68.4
48.3
-
1344 x 1344
47M
669B
53.0
72.2
58.0
45.0
69,4
48.8
-iny-MOAT-3
1344 x 1344
57M
754B
55.2
74.8
60.6
47.0
71.8
51.2
MOAT-0
1344 x 1344
65M
799B
55.9
73.9
60.9
47.4
70,9
52.1
MOAT-1
1344 1314
79M
92.1B
57.7
76.0
63.4
49.0
73.4
53.2
MOAT-21
1344 x 1344
110M
1217B
58.5
76.6
64.3
49.3
73,9
53.9
MOAT-3?
1344 x 1344
227M
2216B
59.2
77.8
64.9
50.3
74.8
55.5
ADE20K Semantic Segmentation. In Tab. 4, when using input size 5132, MOAT consistently outperforms the ConvNeXt counterparts. MOAT-0 surpasses ConvNeXt-T by 3.0% mloU. Moreover, MOAT-2, with ImageNet-22k pretraining, surpasses ConvNeXt-B by 3.1 %. The larger MOAT-3 and MOAT-4 further outperform ConvNeXt-L and ConvNeXI-XL by 4.9% and 5.4%, respectively. Finally, when using input size 6412, our MOAT-4 achieves the performance of 57.6% mloU, setting a new state-of of-the-arr in the regime of models using input size 6412. For tOny-MOAT, tiny-MOAT-3 achieves comparable performance with NonvNeXt-S with less than half of the parameters.
MonY on ImageNet. We simply scale down the channels of MOAT-O to obtain the tiny-MOAT family without any specific adaptions. In the left of Tab. 5, with the similar model parameters,
7

---

Published as a conference paper at ICLR 2023
Table 4: Semantic segmentation ADE20K val set. We employ DeepLabv3+ (single-scale inference). Results ConvNeXt and MOAT are obtained using the official code-base (Weber et al., 2021) with the same training recipe. t: use ImageNet-22K pretrained weights.
backbome
ingut
purames
ILOPss
mloU (S:)
tiny MOAT-O
513²
5.6M
11.8B
41.2
tiny-MOAT-1
513
7.8M
15.28
43.1
Liny-MOAT-2
backbone
input
FLOPs
mioU (%)
tiny-MOAT-3
513
24.2M
41.2B
47.5
ConyNeXi-S
Conx-NeXi-T
5133
313*
55,8M
34,2M
47.6B
70.88
47.8
45.8
MOAT-0
518"
33.3M
61.30
48.8
Con-NeXt-XLF
NoN-NeXt-B?
No
513
513"
513
364.0M
95.8M
446.2B
119.5B
51.8
50.5
MOAT-3#
MOAT-29
MOAT-1
MOAT-47
513'
515"
5132
496.3M
198.4M
80.5M
47.0M
331.58
144.3B
85.4B
53.6
$5.9
MOAT-2#
6412
80.5M
242.08
54.7
MOAT-3+
6412
198.4M
554.7B
56.5
MOAT-44
496.3M
1273.5B
tiny-MOAT-0/112 surpass the Mobile-Former counterparts by 6.8%, 5.5%, and 4.3%, respectively. In the right of Tab. 5, our tiny-MOAT also shows stronger performances than Mobile (Mehta & Rastegari, 2022a). Even compared with the concurrent work Mobile ViTv2 (Mehta & Rastegari, 2022b), tiny-MOAT-1/213 surpass their counterparts by 1.1%, %, 1.3%, and 2. %, respectively.
Table Performances tiny-MOAT family on ImageNet-IK.
input size 224"
params
FLOPs
top-1 uce.
input size 256
params
FLOPss
top-1 I acc.
Mobile-Former-S75 52M
3.5M
0.05B
68.7
MobileViT-XX.
2.3M
0.7B
74.8
Mobile-Former-966 96M
4.6M
0.18
72.8
MobileViT-S
5.6M
2.0B
78.4
Mobile-Fomer-2144
9.4M
0,2B
76.7
MobileViTV2--1.
4.9M
1.8B
78.1
Mobile-Fomerr 500mer-50MM
14.0M
0.5B
79.3
MobileViTv2-1.3
10.6M
4.0B
80.4
tiny Muy-MOAT-0
3.4M
0.8B
75.5
Mobile ViTv2-2.0
18.5M
7.5B
81.2
únn-MOATTI
5.1M
1.2B
78.3
tiny-MOAT-I
5.1M
L6B
79.2
tiny-MOAT-2
9.8M
2.3B
81.0
tiny- -OAT-2
9.8M
3.08
81.7
tiny-MOAT-3
19.5M
4.5B
82.7
tiny-MOAT-3
19.5M
6.0B
83.3
4 ABLATION STUDIES ON IMAGENET
At micro level, we perform ablation studies on the MOAT block design and downsampling layer in the following and the order of MBConv and Attention in MOAT block in section 1. At macro level, perform ablation studies the MOAT-basee model and MOAT meta architecture in section A.6.2.
MOAT block design. In Tab. 6, we ablate the MOAT block design, which only affects the last Two stages of MOAT, and keep everything else the same (e.g., training recipes). We start from the Transformer block, consisting of Attn (self-attemtion) and MIP, which already attains a strong top-1 accuracy (82.6%). Directly inserting a 3 x 3 depthwise convolution in the MLP degrades the performance by 0.9%. If we additionally insert batch normalization and GeLU between convolutions (i.e., replace MLP with MBConv, but no gquueze-and-Excitation 82.9%. Finally, placing MBConv before Attn reaches the performance of 83.3%. Additionally, our MOAT block brings more improvements (from 1.2% up 2.6% gains) in the tiny model regime.
the performance is improved to
Downsampling layer. For the MOAT block design, we do not need the extra downsampling layer like (1) everage-pooling in CoAtNet (Dai et al., 2021), (2) patch-embedding layer (i.e., x 2 convolution with stride 2) in Swin (Liu et al., 2021) and ConvNeXt (Liu et al., 2022b), or (3) strided depthwise convolution PiT (Heo et al., 2021) and Region ViT (Chen et al., 2022a). As shown in Tab. 7, using patch-embedding layer indeed improves over the average-pooling scheme by 0.2% accuracy, but takes more cost of model parameters. Additionally, using the strided depthwise convolution for downsampling leads to 0.2% worse performance than the patch-embedding layer. By contrast, MOAT design (i.e., delegating the downsampling to the MBCony block) shows the best performance with the least cost of parameters and comparable FLOPS.
8

---

Published as a conference paper at ICLR 2023
Table 6: Ablation studies of MOAT block design on ImageNet-Il IK with input size 224.
model
block composition
FLOPs
top-1 occ.
model
block composition
params
FLOPss
top-1 acc.
Maay-MOKT-2
MLP
9.8M
2.28
79.8
MBCom Alln
9.8M
2,38
81.0
Alin+ + MLP
28.0M
5.4B
82.6
MOAT-0
Atm MBConv
Atto- MLP P depth.. com))
28 2M
28 28.2M
5.4B
5.4B
82.9
81.7
MoyT-MAT-II
Attn MLP
MBCom Attn
5.1M
5.1M
1.28
1.1B
78.3
76.2
MBCome + Allin
27.8M
3.7B
83.3
tiay-MOAT-
MBComm Ame
Attn
3.4M
0.8
0.88
75.5
72.9
Table Ablation studies of the downsampling layer design on ImageNet- using MOAT-0 and input size 224, We compare our MOAT design (in grey) with (1) CoAtNet (using average-pooling for downsampling), (2) Swin/ConvNeXt designs (using strided 2 x 2 convolution for downsampling), and (3) PiT/RegionVi ViT designs
(using strided 3 x 3 depthwise convolution for downsampling).
block composition
downsampling type
purams (M)
FLOPs (B)
top-I acc.
AvergePoollng + Atto + MLP
CoAtNet
28.0
5.4
82.6
PatchEmbeskling + MI.P
Swin, Con~NeXt
30.2
StridedDepthCony Atto + MLP
PVT. Region
28.8
5.5
82,6
MBConr Attn
MOAT
27.8
5.7
83.3
5 RELATED WORK
Transformers ( (Vaswani et al., 2017) were recently introduced to the vision community (Wang et al., 2018; Ramachandran et al., 2019; Hu et al., 2019) and demonstrated remarkable performance on vision recognition tasks (Carion al., 2020; Zhu et al., 2021; Wang et al., 2021a; Amab al., 2021: Liu et al., 2021: Cheng et al., 2021; Yu et al., 2022a; Kim et al., 2022; Cheng et al., 2022; Yu al., 2022b), thanks their ability to efficiently encode long-range interaction via the attention mechanism (Bahdanau et al., 2015). Particularly, ViT (Dosovitskiy et al., 2021) obtains impressive results on ImageNet (Russakovsky al., 2015) by applying the vanilla Transformer with the novel large stride patch embedding, after pretraining on the proprietary large-scale JFT dataset (Sun et al., 2017). There have been several works aiming to improve the vision transformers, either with better training strategies (Touvron et al., 2021 a;b; Steiner et al., 2021; Zhai et al., 2022; Touvron et al., 2022) or with efficient local-attention modules (Huang et al., 2019; Ho al., 2019; Wang et al., 2020; Liu et al., 2021; Chu et al., 2021; Yang et al., 2021; Yu et al., 2021; Dong al., 2022; Tu et al., 2022).
Since the debut of AlexNet (Krizhevsky et al., 2012), the vision community has witnessed a rapid improvement the ImageNet benchmark using different types of ConvNets, including (but not limited to) VGGNet (Simonyan & Zisserman, 2015). Inceptions (Szegedy et al., 2015; Ioffe & Szegedy, 2015; Szegedy et al., 2016; 2017), ResNets (He et al., 2016a;b), ResNeXt (Xie et al., 2017), DenseNet (Huang et al., 2017), SENet (Hu et al., 2018), MobileNets (Howard et al., 2017: Sandler al., 2018; Howard et al., 2019), EfficientNets (Tan & Le, 2019; 2021), ), and ConvNeXt (Liu et al., 2022b) each focusing on different aspects of accuracy and efficiency. The ubiquity of ConvNets in computer vision could be attributed to their built-in inductive biases.
Given the success of Transformers and ConvNets, another line of research is to explore how to effectively combine them. Swin (Liu al., 2021; 2022a), PVT (Wang et al., 202 2022), MVIT (Fan et al., 2021: Li et al., 2022), and PiT (Heo et al., 2021) adopt the ConvNet hierarchical structure to extract multi- features for Transformers. SASA (Ramachandran et al., 2019), AA-ResNer (Bello et al., 2019), Nxial-ResNer (Wang et al., 2020) and BoTNee (Srinivas et al., 2021) incorporate the attention modules to ResNets. CvT (Wu al., 2021). )), LeViT (Graham al., 2021), Visformer (Chen al., 2021b), and ViTc (Xiao et al., 2021) replace VIT's patch embedding with strided convolutions. CeiT (Yuan et al., 202 la) and CMT (Guo et al., 2022) incorporate depthwise convolution to the transformer block's MLP. VITAE (Xu et al., 2021) adopts parallel attention modules and convolutional layers. LVT (Yang et al., 2022) introduces local self-attention into the convolution. Recently, CoAtNet (Dai et al., 2021) and MibileViT (Mehta& & Rastegari, 2022a) propose hybrid models that build on top of the efficient Mobile Convolution (Sandler et al., 2018 8) and Transformer block.
Acknowledgements We thank Wen-Sheng Chu for the support and discussion. We gratefully acknowledge supports from the Office of Naval Research. 200014-21-1-2812..

---

Published as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, 2016.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.064500 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. In ICLR, 2021.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In ICCV, 2019.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In ICML, 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In ICLR, 2022a.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Liang-Chieh Chen, Huiyu Wang, and Siyuan Qiao. Scaling wide residual networks for panoptic segmentation. 1.
I./ 1675, 2020.
Wuyang Chen, Xianzii Du, Fan Yang, Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, et al. A simple single-scale vision transformer for object localization and instance segmentation. arXiv preprint arXiv:2112.09747, 2021a.
Wuyang Chen, Xianzii Du, Fan Yang. Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, and Denny Zhou. A simple single-scale vision trans- former for object localization and instance segmentation. ECCV, 2022b.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengehen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In CVPR, 2022c.
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In ICCV, 2021b.
Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chich Chen. Panoptic-DeepLe A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In CVPR, 2020.
Bowen Cheng, Alexander Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NewrlPS, 2021.
10

---

Published as a conference paper at ICLR 2023
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, Masked- attention mask transformer for universal image segmentation. CVPR, 2022.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurIPS, 2021.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe data augmentation with reduced search space. In CVPR, 2020.
Randaugment: Practical automated
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. attention for all data sizes. In NeurIPS, 2021.
Coatnet: Marrying convolution and
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition scale. In ICLR, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In ICCV, 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang. Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, 2022.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv:1606.0844 08415, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
Jonathan Ho, Nal Kalchbrenner. Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. 191. 2019.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. ICCV, 2019.
11

---

Published as a conference paper at ICLR 2023
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04860 2017.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeezz-annd-exciaatin networks. In CVPR, 2018.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In ICML., 2015.
Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, and Llang-Chicl Chen. SobeFoormer-eeplLab Video Mask Transformer. In CVPR, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurlPS, 2012.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022a.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Sachin Mehta and Mohammad Rastegari. Mobilevit: liggt, general-purpose, and mobile- friendly vision transformer. In ICLR, 2022a.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.0266 2022b.
Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. 1rX1v:1710.05941, 2017.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019.
12

---

Published as a conference paper at ICLR 2023
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In JCML, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJC,, 115:211-255 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Bang-Chheh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Aravind Srinivas, Tsung-Yr Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv:2106. 10270, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning In ICCV, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception -4, inception- resnet and the impact of residual connections on learning. In AAAI, 2017.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In ICMI,, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the VIT. arX1v:2204.07118. 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, 2021.
Huiyu Wang, Yukun Zhu Bradley Green, Hartwig Adam, Alan Yuille. and Liang-Chich Chen. Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation. In ECCV, 2020.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-deeplab:
End-to-end panoptic segmentation with mask transformers. In CVPR, 2021a.
13

---

Published as a conference paper at ICLR 2023
Wenhai Wang. Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media, pp. 1-10, 2022.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Mark Weber, Huiyu Wang, Siyuan Qiao, Jun Xie, Maxwell D. Collins, Yukun Zhu, Liangzhe Yuan, Dahun Kim, Qihang Yu. Daniel Cremers, Laura Leal-Taixe, Alan L. Yuille, Florian Schroff, Hartwig Adam, and Liang- Chich Chen, DeepLab2: A TensorFlow Library for Deep Labeling. arXiv: 197409974, 2021.
Ross Wightman. Pytorch image models. pytorch- Lmaae-modees 2019.
fittp:://githn cwm/co////////.. cttp:///t..o.//..ttt..t....
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, 2021.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. In NeurlPS, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. In NeurlPS, 2021.
Chenglin Yang. Yilin Wang, Jianming Zhang, He Zhang, Zijun Wei, Zhe Lin, and Alan Yuille. Lite vision transformer with enhanced self-attention. In CVPR, 2022.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. NeurIPS, 2021.
Hongkun Yu, Chen Chen, Xianzhi Du, Yeqing Li, Abdullah Rashwan, Le Hou, Pengchong Jin, Fan Yang, Frederick Liu, Jaeyoun Kim, and Jing Li. TensorFlow Model Garden. https: wwweensoooooo/ooneeeeeeeeoe /ooees. 2020.
//github. coom/tensorfii low
Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu, Alan L Yuille, and Wei Shen. Glance-and-gaze vision transformer. In NeurlPS, 2021.
Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022a.
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022b.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In ICCV, 202
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021b.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 1rXiv:1710.09411 2017.
14

---

Published as a conference paper at ICLR 2023
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJC,, 127 321, 2019.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv:2103. 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
15

---

Published as a conference paper at ICLR 2023
A APPENDIX
In the appendix, we provide more details for both our model and experiments.
In section A. we provide MOAT implementation details.
In section A.2.1, we provide ImageNet experimental details.
In section A.2.2, we provide ImageNet-V2 experimental results.
In section A.3.1, we provide COCO detection experimental details.
In section A.3.2, we provide more COCO object detection experimental results.
In section A.4, we provide ADE20K semantic segmentation experimental detaills.
section A.5, we provide COCO panoptic segmentation experiments.
In section A.6.1, we provide ablation studies the MOAT micro-level design.
In section A.6.2, we provide ablation studies on the MOAT macro-level design.
In section A.7, we provide the ImageNet trainng time, peak training memory and throughput measurement of MOAT models.
In section A.8, we discuss limitations of our model.
A.1 MOAT IMPLEMENTATION DETAILS
In the MOTA networks, we employ kernel size for both convolutions and depthwise convolutions. We use the multi-head self attention (Vaswani et al., 2017), where each attention head has channels 32. For the MBConv and MOAT blocks, we use expansion ratio The SE module (Hu et al., 2018) in the MBConv blocks (i.e., 2nd and 3rd stages) adopt reduction ratio 4 (relative to the input channels).
Our MOAT block includes the relative positional embedding (Shaw et al., 2018; Dai et al., 2021) for ImageNet. However, the downstream tasks usually take a larger input resolution than ImageNet, demanding for a special adaptation (e.g., bilinear interpolation of pretrained positional embedding). For simplicity, remove the positional embedding, when running MOAT on downstream tasks.
A.2 IMAGENET IMAGE CLASSIFICATION
A.2.1 IMAGENET EXPERIMENTS
The ImageNet-IK dataset (Russakovsky et al., 2015) contains 1.2M training images with 1000 classes. We report top-1 accuracy on the ImageNet- validation set, using the last checkpoint. We also experiment with pretraining on the larger ImageNet 222 dataset, and then fine-tuming on the ImageNet-1K, We closely follow the prior works (Dai et al., 2021; Liu et al., 2022b) and provide more details below. In Tab. 8, we compare our MOAT with more state-of-the-ard models.
Experimental setup. We train MOAT models on ImageNet- K with resolution 224 for 300 epochs. If pretraining the larger ImageNet-22K,,
resolution 224 and 90 epochs. Afterwards, the models are fine-tuned on ImageNet-1K for 30 epochs. During fine-tuning, we also experiment with larger resolutions (e.g., 384 and 512). We employ the typical regularization methods during training, such as label smoothing (Szegedy et al., 2016), RandAugment (Cubuk et al., 2020), MixUp (Zhang al., 2017), stochastic depth (Huang et al., 2016), and Adam (Kingma & Ba, 2015) with decoupled weight decay (ie.,, Adam (Loshchilov & Hutter, 2019)). See Tab. 10 and Tab. 11 for detailed hyper-parameters.
A.2.2 IMAGENET- K-V2 V2 EVALUATION
To further demonstrate the transferability and generalizability of our MOAT models, we perform additional evaluations on the -K-eNNet-IK-V2 (Recht et al., 2019), using our ImageNet (Russakovsky et al., 2015) pretrained checkpoints. We report extensive evaluation, using MOAT and several input resolutions, on ImageNet- K-V2, aiming to establish another solid baseline for the community, notice that most of the existing models do not report results on ImageNet- -K- V2. As shown in the Tab. 9, MOAT does not overfit to ImageNet- IK-V1 dataset and generalizes well to ImageNet- 1 K-V2 dataset, observe continuous performance improvement from small to large models.
16

---

Published as a conference paper at ICLR 2023
Table 8: Performance ImageNet-IK with state-of-the- models included. 1K only: Using ImageNet- only. 22K 1K: ImageNet-22K pretraining and ImageNet- IK fine-tuning.
model
eval size
params
FLOPs
ImageNet IK top-1 accuracy
IK only
22K+1K
RegNetY-16G (Radosavovic ul., 2020)
224
84M
16.0B
NFNet-F5 (Brock al., 2021)
544
377M
86.0
EificiemiNeeV2-S (Tan Le, 2021)
480
22M
8.8B
83.9
84,9
EfficiemtNer22-MM (Tan Le, 2021)
480
54M
24B
86.2
ConvNets
(Tan (Tan IsficiemtNe/V2-X..ccoo (Tan Le, 2021)
EfficiemNeeV2-L
Le. 2021)
480
120M
53B
85.7
480²
ConvNeXI-T (Liu et al., 2022b)
224
29M
4,5B
82.1
82.9
Con-NeXt-S (Liiu et al., 2022h)
50M
8.7B
83.1
84.6
ConwNeXi-B (Liu et al., 20226)
89M
15.4B
83.8
ConyNeett-L el al., 20226)
198M
ConvNeXt-XL. (Liu al., 2022b)
384
350M
179.08
85.5
87.8
DeiT-B (Touvron et al., 2021a)
86M
55.4H
83.1
CaiT-S-36 (Touvron et al., 2021b)
68M
48.0B
DeepViT-L (Zhou et al., 2021)
221
55M
12.5B
LVTg-argee (Wang 2021h)
224
61.4M
HaloNet-H4 (Vaswani al., 2021)
38422
85M
85.6
VITS
HaloNet-HH (Vaswani al., 2021)
512²
85M
85.8
Swin-T (Liu al., 2021)
28M
4.5B
(1.i0 2021)
224
Swin-B (Liu al., 2021)
224
88M
15.4B
83.5
85.2
Swin-L (Liu 2021)
384
197M
103.9B
87.3
SwinV2-L (Liu 2022a)
3812
197M
115.4B
87.7
Focal-B (Yang al., 2021)
224
CSwin-B (Dong
2022)
384
78M
47.0B
85.4
87.0
CSwin-L (Dong al., 2022)
3812
173M
96.8B
87.5
MVITv2-H (1.i et al., 2022)
5122
667M
763.58
88.8
BotNet-77 (Srinivas al., 2021)
384
75.1M
84.7
Lamb (Bello, 2021)
320
T2T-VI-224 (Yaa 2021b)
2212
64.1M
15.0B
82.6
CMT- (Guo al., 2022)
224
25.1M
4,01
83.5
CeiT-S (Yuun al., 2021a)
384
24.2M
12.9B
83.3
Hybrid
CvT-21 (Wu al., 2021)
3842
32M
24.9B
83.3
PVTv2-B5 (Wang et al., 2022)
22/22
82M
11.8B
83.8
Max ViX-XXX (Tu al., 2022)
512"
475M
535.2B
88.7
CoAdNet-O (Dai al., 2021)
224
25M
4.2B
81.6
1oANNet-1 (Dai al., 2021)
224
42M
8.4B
83.3
CoAdNet-2 (Dai 2021)
224a
75M
15.7B
84.1
CoAdNet-3 (Dai et al., 2021)
384
168M
107 107.4B
85.8
87.6
CoAdNet-4 (Dai ct al., 2021)
512"
275M
360.9B
88.6
MOAT-0
224
27.8M
5.7B
83.3
83.6
MOAT-1
2212
41.6M
9.1B
84.2
84.9
MOAT-2
224
73.4M
17.2H
84.7
86.0
MOAT-3
224
190.0M
44.9B
85.3
86.8
Hybrid
MOAT-O
3842
27.8M
18.28
84.6
85.7
(ours)
MOAT-1
3842
41.6M
29.6B
85.9
87.0
MOAT-2
384
73.4M
54,3B
86.2
87.5
MOAT-3
190.0M
141.2B
86.5
88.2
MOAT-1
5122
41.6M
58.7H
86.2
87.2
MOAT-2
512a
73.4M
104.6B
86.5
87.7
MOAT-3
512²
190.0M
271.0B
86.7
88.4
MOAT-4
5122
483.2M
648.5H
89.1
Under the fair comparison, with ImageNet-22K pretrainng and input size 384, MOAT-2/3 surpass the current state-of-fe of-the-art model Swin by 0.6/1.7%, respectively. Additionally, our MOAT-4, with input size 512, achieves a new statc-of-the performance 81 81,5%, without extra proprietary training data.
17

---

Published as a conference paper at ICLR 2023
Table 9: Performance ImageNet-IK-Va
model
params
imput size
FLOPs
ImageNet- IK- top-1 accuracy (9)
1K only
22K
LeViT-256 (Graham 2021)
256
18.9M
1.BB
70.0
DeiT-B (Touvron al., 2021a)
224
86M
17.58
71.5
CarT-S36 (Touvron al., 2021b)
224
68M
13,9B
72.5
SwinV2-B (Liu et al., 2021)
384
88M
54.7B
78.1
SwinV2-L. (Liu et al., 2021)
384
197M
115.4B
78.3
tiny-MOAT-0
224
3.4M
0.8B
64.3
tiny-MOAT-1
224
5.1M
1.2B
67.3
tiny-MOAT-2
224
9.8M
2.3B
70.1
-iny-MOAT-3
224
19.5M
4.5B
72.1
tiny-MOAT-I
256
5.IM
1.6B
68.2
tiny-MOAT-2
256
9.8M
3.0B
70.9
tiny-MOAT-3
256
19.5M
6.0B
72.9
MOAT-0
224
27.8M
5.7B
72.8
74.1
MOAT-1
41.6M
9.1B
74.2
75.8
MOAT-2
224
73.4M
76.7
MOAT-3
224
190.0M
44.9B
75.5
78.4
MOAT-O
384
27.88 PM
18.2B
74.5
76.4
MOAT-1
384
41.6M
29.6B
76.2
78.1
MOAT-2
384
73.4M
54.3B
76.5
78.7
MOAT-3
384
190.0M
141.2B
77.5
80.0
MOAT-1
512
41.6M
58.7B
76.8
78.4
MOAT-2
512
73.4M
104.6B
77.1
79.3
MOAT-3
512
190.0M
271.0B
77.8
80.6
MOAT-4
512
483.2M
648.5B
81.5
Table 10: MOAT ImageNet hyper-parameter settings.
ImageNet-Il Net-1K
Image Net-22K
IK
IK + 1K
22K
22K - IK
hyper-parameter
pre-training
fine-tuning (MOAT -O/1//00///233
pre-training
fine-tuning (MOAT-001/2/3)
0.2
0,2/0.3
0.1
0.1 /..2
stochastic depth rate
0.7
/..5/099
/0.3
/0.3/0.6
center crop
true
false
true
false
randaugment
2. 15
15/15/15/20
2.
2.
mixup alpha
0.8
0.8
none
loss type
softmax
softmax
sigmoid
softmax
label smoothing
0.1
0.0001
0.1
train epochs
300
90)
30
train batch size
4096
512
4096
1024
optimizer type
AdamW
Adam
Adam
AdamW
peak learning
3c-3
Se-5
lc-3
5e-5
min learning
lc-5
5e-5
lc-5
5e-5
warm-up
10K steps
epochs
none
decay schedule
cosine
none
linear
none
weight decay rate
0.05
0.01
le-8
gradient clip
1.0
EMA decay
0.9999
0.9999
None
0.9999
A.3 COCO OBJECT DETECTION AND INSTANCE SEGMENTATION
A.3.1 COCO OBJECT DETECTION EXPERIMENTAL DETAILS
Experimental setup. We train Cascade Mask R-CNN (Cai & Vasconcelos, 2018; He et al., 2017) on the COCO 2017 dataset (Lin et al., 2014) with our MOAT architectures. The dataset contains 1 118K training and 5K validation samples. We use the official TensorFlow (Abadi et al., 2016) implementation of Cascade Mask R- R-CNN by TF- Vision Model Garden (Yu et al., 2020). Our training setting closely follows the prior works (Chen et al., 2022b; Tu et al., 2022), except that we use batch size 64 and initial learning rate 0.0001. To adapt the MOAT models to high-resolution inputs,
18

---

Published as a conference paper at ICLR 2023
Table 11: tiny-MOAT ImageNet hyper-parameter settings. *: use EMA decay rate 0.9999 for Liny-MAAT:
ImageNet-1K
IK
hyper-parameter
input size (iny-MOOTT-0/12555
input size 256
0.0
0.0
stochastic depth rate
0.0
10.0/0.1 0.1
center crop
true
true
randaugment
2. 15
2, 15
mixup alpha
0.8
0.8
loss type
softmax
softmax
label smoothing
0.1
0.1
train epochs
300
300
train butch size
4096
4096
optimizer type
AdamW
AdamW
peak learning rate
3e-3
3c-3
min learning rate
le-5
le-5
warm-up
10K steps cosine
10K steps
Ir decay schedule
cosine
weight decay rate
0.05
0.05
gradient clip
1.0
1.0
EMA decay rate
None
None
partition the features into non-overlapping windows for the self-attention computations with the window size set to 14 for the second last stage, and use global attention for the last stage. As a result of this window partition, the input size must be divisible by 14. The TF-Vision Model Garden codebase further requires the input size to be square (with padding) and divisible by 64. Hence, we choose 1344 as the input size, similar to the size used in the baseline methods (i.e., longest side is no more than 1333). We use Feature Pyramid Network (Lin et al., 2017) to integrate features from different levels.
A.3.2 MORE COCO OBJECT DETECTION EXPERIMENTAL RESULTS
In this section, we perform more COCO object detection experiments with 896 input size. All the backbone are pretrained on ImageNet- IK dataset. MOAT-0/1/2 surpass UVIT (Chen et al., 2021a) and MaxVIT (Tu et al., 2022) by 3.9/5.2/4 2/4.9% Apbox (3.1/4.1/3.99) Apmask), and 3.0/4.0/4.0% Apbox (2.4/3.2/3.0% Ammss)) respectively.
Table 12: Object detection and instance segmentation on the COCO 2017 val set. We employ Cascade and single-scale inference (hard NMS). All backbones pretrained ImageNet- IK.
backbone
imput size
params
FLOPs
Aphos
APh
APlus
Apmek
Aprask
Ap
UVIT-T (Chen al., 202
896 x 896
51M
720B
51.2
43.9
UVIT-S (Chen et ul., 2021a)
896 x 896
59M
882B
51.9
44.5
UVIT-B al., 2021a)
MaxVIT-T al., 2022)
86M
475B
$2.1
71.9
56.8
44.6
48.4
MaxVIT-S (Tu al., 2022)
896 x 896
108M
595B
53.1
72.5
58.1
45.4
69.8
49.5
MaxViT-B (Tu et al., 2022)
896 x 896
146M
856B
53.4
72.9
58.1
45.7
70.3
50.0
MOAT-0
896 896
65M
525B
55.1
73.6
59.9
47.0
70.5
51.1
MOAT-1 1
896 x 896
79M
5801
$7.1
75.7
62.6
48.6
72.9
52.7
MOAT-2
896 x 896
110M
710B
57.4
76.0
63.0
48.7
73.2
53.1
A.4 ADE20K SEMANTIC SEGMENTATION
Experimental setup. We experiment with the proposed MOAT models on ADE20K semantic segmentation dataset (Zhou et al., 2019) using DeepLabv3+ (Chen et al., 2018; 2017). We fine-tune the global attention for MOAT. The same training strategies are used for all backbone variants. train the model with 32 TPU cores for 180k iterations,
Specifically, for training hyper-paramene parameear,,,,rss with batch size 64, Adam (Kingma & Ba, 2015) optimizer, and a poly schedule learning rate starting 0.0001. For data augmentations, the inputs images resized and padded to either 513 x 513
19

---

Published as a conference paper at ICLR 2023
641 x 641, with random cropping, flipping, and color jittering (Cubuk et al., 2019). No test-time augmentation is used during inference.
A.5 COCO PANOPTIC SEGMENTATION
Experimental setup. We also evaluate the proposed MOAT architectures on the challenging COCO panoptic segmentation dataset (Lin et al., 2014) using Panoptic-DeepLab (Cheng et al., 2020) with the official codebase (Weber et al., 2021). We fine-tune the global attention on downstream segmentation tasks for MOAT. We adopt the same training strategies for MOAT and its counterparts. Specifically, for training hyper- syper-parameters, we train the model with 32 TPU cores for 200k iterations with the first 2k for warm-up stage. We use batch size 64, Adam (Kingma& & Ba, 2015) optimizer, and a poly schedule learning rate starting at 0.0005. For data augmentations, the inputs images are resized and padded to 641 x 641, with random cropping. flipping, and color jittering (Cubuk et al., 2019). No test-time augmentation is used during inference.
Main results. The results are summarized in Tab. 13, where MOAT consistently outperforms other backbones. Specifically, our MOAT-0 surpasses ConvNeXt-T significantly by 4.3% PQ. In the large model regime, MOAT-3 surpasses ConvNeXt-L by 3.5%. Our MOAT-4 achieves the performance of 46.7% PQ, outperforming the heavy backbone SWideRNet (Chen et al., 2020) by 2.3%.
Table 13: Panoptic segmentation on COCO val set. The results are obtained by applying different backbones with Sanoptic-DeepLane using single-scale inference (i.e., no test-time augmentation), Results for MobileNet, ResNet, and Xception are cited from (Cheng et al., 2020), and results for SWideRNet is cited from (Chen et al., 2020), while results for ConvNeXt and MOAT are obtained using the official code-base (Weber al., 2021) with the same training recipe. All models are trained and evaluated with input images resized 6-11 641, and thus FL.PPs are also measured w.r.t. size 641 x 641. f: use ImageNet-225 pretrained weights.
backbone
params
FLOPs
(Q))
PQ (%)
PQ (%)
MobileNet-V V3 (Howwrd et al., 2019)
12.2B
30,0
ResNet50 (He et al., 2016a)
77.8B
35.1
Xception-71 (Chollet, 2017)
109.2B
38.9
ConyNeXt-T (Liu et al., 2022b)
40.3M
51.3B
36.7
37.3
35.7
ConvNeXt-S (Liu et al., 20226)
61.9M
87.2B
40.0
41.4
37,9
Con-NeXt-Ba (Liu al., 2022b)
103.8M
146.2B
41.7
43.6
38.9
ConvNeXt-Li (Liu et al.. 20226)
220.1M
312.8B
41.9
43.6
39.4
Con f (Liu et al., 2022b)
379.6M
$44.1B
43.0
44.9
40.0
SWideRNet (Chen et al., 2020)
752.5M
2614.0B
44.4
MOAT-0
39.5M
76.8B
41.0
42.6
38.6
MOAT-1
53.1M
119.7B
43.0
44.7
40.4
M
88.5M
199.7B
43.9
45.9
40.8
MOAT-39
208.3M
493.3B
45.4
48.3
41.1
M0AT-49
512.0M
1134.7B
46.7
49.5
42.4
A.6 MORE ABLATION STUDIES
A. ABLATION STUDIES ON THE MOAT MICRO-LEVEL DESIGN
Order of MBCony and Attn in MOAT block. Our MOAT block design reverses the order of Attention (Attn) and Mobile Convolution (MBConv), delegating the downsampling duty to the strided depthwise convolution within the MBConv. However, the dowsampling can be still performed in the MBCony with the original order (i.e., Attn MBConv). Since the operations, Attn and MBConv, are interlaced, the key difference then comes from the first block in each stage, where the Attn is operated on the (1) spatially downsampled and/or (2) channel expanded features. To conduct the study, employ different blocks in the MOAT variants, using "Attn MLP", "Attn MBConv", or "MBCony + Attn", For the "Attn + MBConv" block, we further ablate the place (Attn vs. MBConv), where we apply the spatial downsampling and channel expansion operations.
In Tab. 14, we observe the following results. First, replacing the MLP with MBCony improves the performance by 0.3% and 0.7% for MOAT-0 and MOAT-2 Second, if perform both spatial downsampling and channel expansion at the MBConv block, the performance is further improved by 0.5% and 0.9% for MOAT-0 and -OnT-MOA,-2, showing that MBConv learns better downsampled
20

---

Published as a conference paper at ICLR 2023
Table 14: Ablation studies of the order of MBConv and Attention (Attn) ImageNet-IK with input 224.
We also ablate place, where we apply spatial downsampling and channel expansion.
comgosition
spotial downsampling
channel expansion
porams (M)
(B)
top-J
82.6
Atin MBConn
Alta
Atta
28.2
54
82,9
MOAT-0
Attn MBConv
MBCoav
MBComv
25.6
5.8
83.1
MBCom
MBComv
Atin MBCony
MBCoav
Attn
29.3
7.1
83.2
Attn MLP
Asm
Atn
9.8
2.2
79.8
tiny -OXT-2
Attn MBCony
MBComv
MBComv
9.0
2.3
80.7
MBCORY Attn
MBCoav
MBConv
9.8
2.3
$1.0
Attn MBConv
MBComm
Atm
10.3
2.8
81.0
features. However, this design is equivalent to shifting the first Attn layer to its previous stage, reducing the representation capacity of the current stage. More concretely, only the last stage will be affected, since one layer is shifted. Third, to enhance the representation capacity, reversing the order of Attn and MBConv allows us to keep the first Attn layer the same stage. This design further improves the performance by 0.7% and 1.2% for MOAT-0 and tiny-MOAT-2. Fourth, compensate for the shifting effect, we could also employ another x 1 convolution to expand the channels at the first Attn layer (then, MBCCnn only performs the spatial downsampling). However, this design performs similarly to our MOAT block design, but uses more parameters and FLOPs.
A.6.2 ABLATION STUDIES ON THE MOAT MACRO-LEVEL DESIGN
Ablation studies on MOAT-based model. In Tab. 15, we ablate the stage-wise design by using either MBConv MOAT block in stage 2 to stage 5. The first stage is the convolutional stem, containing two 3 X 3 convolutions. We use the layer layout of MOAT-0. As shown in the table, the pure MOAT-based model (i.e., using MOAT blocks for all four stages) achieves the best performance of 83.6%, which however uses the most FLOPs. Our MOAT model design (i.e., use MOAT block in the last two stages) attains the better trade-off between accuracy and model complexity.
Table 15: Ablation studies of MOAT-based model ImageNet-1K K, using MOAT-0 layer layout and input size 224. We change the block type (MBCony MOAT block) from stage 2 stage S. The first stage is fixed to the convolutional stem.
stage-2
stage-3
stage-4
stage-5
params (M)
FLOPs (B)
top-1 acc.
MOAT
MOAT
MOAT
MOAT
28.2
11.9
83.6
MBConv
MOAT
MOAT
MOAT
28.1
6.9
83.5
MBConv
MBConv
MOAT
MOAT
27.8
5.7
83.3
MBConv
MBConv
MBConv
MOAT
25.7
4.7
82.2
MBConv
MBConv
MBConv
MBConV
23.4
4.5
82.0
Ablation studies MOAT meta architecture. We perform ablation studies the meta-architecture by varying the number of blocks per stage. For simplicity, we only vary the block numbers in the third and fourth stages, while keeping the block numbers in the other stages unchanged. Note that the first stage corresponds to the convolutional stem. The studies with MOAT-1 meta architecture are shown in Tab. 16. In the end, we choose the layout {2, 14, 2) because has the best performance and lower parameter cost. Interestingly, our discovery echoes the layer layout proposed by CoAtNet (Dai et al., 2021). We visualize the architecture of MOAT-1 Fig.
Table 16: Ablation studies of MOAT meta-architecture design ImageNet- IK, using MOAT-1 and input size 224. We control the first, second and last stages have two blocks, and vary the block numbers of the third fourth stages.
number of blocks in five stages
params (M)
FLOPs (B)
top-1 acc.
(2, 2, 16.
43.7
8.9
84.1
4,
84.2
(2, 2,2,66 14,
41.6
9.1
84,2
(2, 2, 8, 2)
40.6
9.2
84.1
2, 10, 12,2)
39.5
9.3
84.1
21

---

Published as a conference paper at ICLR 2023
Hix WX
Il
2
2
W 96
o 8" 8
18 1B
384
W 32"
MBConv block
MBConw block
MOAT block
MOAT block
#2
x6
x14
x2
MilCoom
MoCoen
MOXT blak
Figure 6: Architecture of MOAT-1, including the convolutional stem, MBConv, and MOAT blocks.
A.7 IMAGENET TRAINING TIME, PEAK TRAINING MEMORY AND TROUGHPUT MEASUREMENTS
Table 17: ImageNet training time measured in hours. We TPUv4 for training M0A1-((01,2) and 32 TPUv4 cores MOAT-3. MOAT training efficient: for ImageNet-22K pretraining. MOAT takes no more than 2.05 days, while for ImageNet- pretraining, MOAT takes < day.
dataset
model
pre-training
fine-tuning
input size
224
224
224 x 224
384 x 384
MOAT-0
6.5h
2.8h
ImageNet-1K
MOAT-1
4.4h
MOAT-2
13.9h
6.
MOAT-3
16.0h
7.9h
input
224
224
224 224
384 384
MOAT-0
20.
0.9h
2.5h
ImageNet-22K
MOAT-1
30.0h
1.3h
3.9h
MOAT-2
42.6h
1.8h
5.4h
MOAT-3
49.2h
2.2h
7.0h
Table 18: ImageNet peak training memory of MOAT models. The input size is 224 x 224.
training statistics
model
total batch of TPUv4
batch size per core
peak memory per (MB)
MOAT-0
4096
16
256
19155
MOAT-1
4096
16
256
26170
MOAT-2
4096
16
256
26662
MOAT-3
26260
Table 19: ImageNet throughput measurement MOAT models. re-implement MOAT with the popular "timm" (Wightman, 2019) library in PyTorch, and measure the throughput on an Nvidia V100 GPU, following the same settings as DeiT (Touvron el al., 102 Swin (Liu et al., 2021), and ConvNeXi (Liu et al., 2022b).
input size
224
224
384
384
512 x 512
params (M)
FLOPs
throughput (images/sec)
FLOPs
throughput (images/sec)
FLOPS (B)
throughpul
(B)
(B)
(images/sec)
MOAT-0
27.8
5.7
536
18.2
155
MOAT-1
41.6
339
29.6
91
58.7
MOAT-2
73.4
17.2
209
54.3
104.6
27
MOAT-3
190.0
44.9
89
141.2
23
271.0
9
MOAT-4
483.2
648.5
4
22

---

Published as a conference paper at ICLR 2023
A.8 LIMITATIONS
Currently, the scaling rule of MOAT model variants are hand-designed. We, therefore, expect the architecture could be further improved by the breakthroughs in neural architecture search or network pruning (attaining faster inference speed while maintaining a similar accuracy).
23


Reviews:
[{"review_id": "yi5_uniTvK", "reviewer": "Reviewer_GsU5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a family of neural networks called MOAT, which combines Mobile ConvNet and Transformer. It studies how to build effective networks based on two observations from Mobile ConvNet and Transformer. The new proposed networks achieve good performance on different tasks.", "summary_of_the_review": "Refer to the above sections.", "strength_and_weaknesses": "Strength: Paper writing is good and easy to follow; Idea is simple and effective; Experiments are sufficient.\n\nWeaknesses: The paper is more like a technical report. The novelty is weak by combing MobileConvet and Transformer. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PXQQkS5I0bK", "reviewer": "Reviewer_W4v5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new transformer-based architecture for vision tasks, namely MOAT. \nThe key idea is to combine the convolutional priors, locality, and hierarchy into the pure transformer block design (self-attention, then MLP) and remove redundant operations. \nFirst, the locality is achieved by replacing the MLP block with a mobile convolution block (MBconv). In practice, the authors empirically verified that placing the MBconv before self-attention provides the best result. \nSecond, the hierarchy is achieved by introducing strided depth-wise convolution in the first block in each stage, and it is efficient compared to having additional patch embedding layers. \nFinally, the authors sweep the best macro-level block configurations and found that using MBconv blocks and MOAT blocks at the first and last two stages strikes the best efficiency-accuracy trade off.\nThe final model MOAT provides good basis for the model scaling (both up and downward) and achieves strong results in various vision tasks.\n\n\n", "summary_of_the_review": "My main concern with this paper is the limited technical contribution. Given the MaxViT, the claimed technical contributions are not new. The MOAT can be regarded as an incremental variant of MaxViT. It works slightly better with proper post-hoc engineering (e.g., introduce pure MBconv blocks in early stages, use global attention and convert it to the window attention if test-time input resolution is large, etc.).\n\nOverall, I appreciate extensive experiments and engineering efforts. The empirical results, code, and model checkpoints will help the practitioners. However, this cannot be the basis for a new ICLR conference paper. I am willing to hear the author's view on comparing MOAT with the MaxViT in various aspects, including experiment-level comparisons in a broad model-size spectrum (e.g., in which model regime the MOAT outperforms/underperforms the MaxViT?) and thorough descriptions of key differences and their resulting practical dis-/advantages (e.g., what is the core that makes MOAT practically strong compared to MaxViT?).", "strength_and_weaknesses": "**Strength**\n\n*+* The paper conducts extensive experiments. Especially having dense reference points on the broad regime of model sizes shows the strong scaling effect of the model and can allow the follow-up studies to compare it more thoroughly. I appreciate the authors for doing this.\n\n*+* The open-sourced code bases/checkpoints can provide broad visibility and high reproducibility to the community.\n\n*+* The MOAT achieves strong results in various vision benchmarks.\n\n**Weakness**\n\n*-* Given the recently presented MaxVIT, my main concern with this paper is ***limited architectural innovations***. The major observations and the resulting architectural changes are 1) MBconv + self-attention (to equip with the translation equivariance and locality) and 2) strided dwconv instead of extra patch-embedding layers (to equip with the hierarchy). However, both are already presented in the MaxViT. The remaining difference is the existence of SE in the MBConv block, which is a minor change (In fact, there is a recent trial of this in MaxViT by the 3rd group in https://github.com/rwightman/pytorch-image-models, i.e., MaxViT with ConvNeXt block, and works well).\n\n*-* Moreover, unlike MaxViT, which uses linear complexity self-attention, the MOAT directly adopts the quadratic complexity global self-attention. This makes the authors introduce inhomogeneous block configurations (i.e., using MBconv blocks at the first two stages) to reduce overeheads and perform post-hoc engineering (i.e., using window attention and setting the proper window size) to apply the model to high-resolution inputs. These reduce actual ***algorithmic simplicity*** overall.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7qpt1a41y1", "reviewer": "Reviewer_bFXs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new building block, which combined MBConv and Attention, for neural network design and the authors show its efficiency and effectiveness on many visual tasks.", "summary_of_the_review": "Please see my comments in Section 1, 2 and 3. \n\nAnd I have other questions:\n1. In the abstract, authors mentioned that the generalization of transformer is worse than ConvNet, but is there any experiment to support this argument?\n\n2. In Table 8, this ablation study focus on the effects of downsampling layer, why the block composition is also varied in this case?\n\n3. Under tiny-MOAT setting (Table 5), what is the performance of tiny-MOAT if we would like to match the FLOPs to 0.05B from MobileFormer? \n", "strength_and_weaknesses": "Strength\n1. The authors demonstrate the efficiency and effectiveness on the proposed MOAT module.\n2. The ablation studies are comprehensive, especially the order of MBConv and Attention block.\n\nWeaknesses\n1. The third observation seems to be found by other works already, e.g. PiT and RegionViT both use depthwise convolution for the donwsampling and they found it is effective. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0cWcAklPve2", "reviewer": "Reviewer_BAq9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way to incorporate MBconv in a transformers architecture. This in order to propose an efficient architecture for many tasks with good FLOPS- accuracy and parameters-accuracy trade-off.", "summary_of_the_review": "The paper is well written and the proposed idea is interesting; however, it lacks results in order to better evaluate the significance of the results and to know the impact of the proposed method compared to the existing one in the literature.\n\n====Post Rebuttal====\nThe authors address some of my concerns in particular on the experimental part. However, the proposed architecture lacks novelty and clear motivation in its design in comparison to the literature, but the experimental part is quite extensive. So I will keep my original score.", "strength_and_weaknesses": "Strenghts:\n\n- The paper is well written and easy to follow.\n- The proposed method is simple and seems to give good results on the different settings considered. \n\n\nWeakness:\n\n-Comparison with concurent approaches: The paper states in the abstract \"The research community has thus attempted to combine the strengths from both architectures. Unlike the current works that simply stack separate mobile convolution and transformer blocks, we effectively merge them into a MOAT block.\" However there is no analysis that compares the Moat block and other blocks proposed in the literature (CMT [1] , Yuan et al. [2], CoatNet [3])using the same architecture. In order to have a clear comparison of the Moat blocks and concurrent approaches.\n\n[1] Guo et al., CMT: Convolutional Neural Networks Meet Vision Transformers\n[2] Incorporating Convolution Designs into Visual Transformers\n[3] CoAtNet: Marrying Convolution and Attention for All Data Sizes\n\n-Comparison without EMEA:  According to Table 10 the paper use EMEA for models train on ImageNet-1k for a fairer comparison with the other approaches it would be interesting to also report the results without EMEA\n\n- Semantic segmentation: On ADE20k the UperNet evaluation setting is more popular than the one used in the paper. Currently the comparison is quite limited. It would be interesting to have a larger comparison with the upernet setting to see how the proposed architecture performs compared to competing approaches.\n\n- Missing metrics: There is only FLOPs and parameters that reported in the different tables. It is necessary to add the peak memory and the latency in order to better understand the different trade-offs. Indeed, MBconv are known to give good FLOPs accuracy/params-accuracy trade-offs but are not always very good for the other trade-offs.\n\n\n- Overfitting evaluations:There is no evaluation of overfitting but on ImageNet the evaluation is done on the validation set which can lead to overfitting. It is important to add evaluation on ImageNet-v2 [1] to evaluate the level of overfitting.\n\n[1] Recht et al., Do ImageNet Classifiers Generalize to ImageNet?\n\n\nOther comments:\n- FLOPs/ Parameters trade-off: Table 2 and Table 9 It is interesting to add all the EfficientNet-v2 models (S and M) because they have a good trade-off.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster