MogaNet: Multi-order Gated Aggregation Network
Authors: Siyuan Li
Zedong Wang
Zicheng Liu
Cheng Tan
Haitao Lin
Di Wu
Zhiyuan Chen
Jiangbin Zheng
Stan Z. Li
Venue: ICLR 2024 poster (2024)
Keywords: Network Architecture
Convolution Neural Network
Image Classification
Vision Transformer
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper designs MogaNet, which demonstrates excellent scalability and competitive performances among state-of-the-art models with more efficient use of parameters on ImageNet and various vision downstream tasks.

Abstract: By contextualizing the kernel as global as possible, Modern ConvNets have shown great potential in computer vision tasks. However, recent progress on \textit{multi-order game-theoretic interaction} within deep neural networks (DNNs) reveals the representation bottleneck of modern ConvNets, where the expressive interactions have not been effectively encoded with the increased kernel size. To tackle this challenge, we propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning in pure ConvNet-based models with favorable complexity-performance trade-offs. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks, including COCO object detection, ADE20K semantic segmentation, 2D\&3D human pose estimation, and video prediction. Notably, MogaNet hits 80.0\% and 87.8\% accuracy with 5.2M and 181M parameters on ImageNet-1K, outperforming ParC-Net and ConvNeXt-L, while saving 59\% FLOPs and 17M parameters, respectively. The source code is available at https://github.com/Westlake-AI/MogaNet.

Published as a conference paper at ICLR 2024
MOGANET: MULTI-ORDER GATED AGGREGATION NETWORK
Siyuan Li Zedong Wang Zicheng Liu
Cheng Tan Haitao Lin .2 Di Wu
Zhiyuan Chen Jiangbin Zheng! Stan Z. Li
1 AAl Lab, Research Center for Industries of the Future, Westlake University, Hangzhou, China *Zhejiang University, College of Computer Science and Technology. Hangzhou, China
ABSTRACT
By contextualizing the kernel global possible, Modern ConvNets have shown great potential in computer vision tasks. However, recent progress on multi-order game- thheretheo interaction within deep neural networks (DNNNs) reveals the repre- sentation bottleneck of modern ConvNets, where the expressive interactions have not been effectively encoded with the increased kernel size. To tackle this chal- lenge, we propose new family of modern ConvNets, dubbed MogaNet, for dis- criminative visual representation learning in pure ConvNeet-base models with fa- vorable camplexity-performmnt trade-offs. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adap- tively. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks, including COCO object detection, ADE20K semantic segmentation, 2D&3D human pose estimation, and video prediction. Notably, MogaNet hits 80.0% and 87.8% accuracy with 5.2M and 181M parameters ImageNet-IK IK, outperforming ParC-Net and ConvNeXt- L. while saving 59% FLOPs and 17M parameters, respectively. The source code is available https: / //gittiiie githhb..... com West Lake AI/MogaNet.
INTRODUCTION
Accuracy vs. Param. vs. GFLOPs
86.
9.90
15.9G
34.5G
By relaxing local inductive bias, Vision Trans- formers (ViTs) (Dosovitskiy al., 2021; Liu et al., 2021) have rapidly challenged the long dominance of Convolutional Neural Networks (ConvNets) (Ren et al., 2015; He et al., 2016; Kirillov et al., 2019) for visual recognition. It is commonly conjectured that such superiority of ViT stems from its self-attention operation (Bahdanau et al., 2015; Vaswani et al., 2017), which facilitates the global-range feature inter- action. From a practical standpoint, however, the quadratic complexity within self-attention prohibitively restricts its computational effi- ciency (Wang et al., 2021a; Hua et al., 2022) and applications to high-resolution fine-grained
5.0G
84.0
Top 1 Accuracy
$2.0
80.0
1.1G
RegNetY De'T (VIT)
78.0
76
Swin
CoAtNet
ConvNeXi
MogaNet (Ours)
72.0
300
# Parameters (M)
Figure Performance on ImageNet-1K val- idation set at 224 resolutions. MogaNet outperforms Transformers (DeiT(Touvron et al., 2021a) and Swin (Liu et al., 2021)), ConvNets (RegNetY (Radosavovic et al., 2020) and Con- vNeXt (Liu et al., 2022b)), and hybrid models (CoAtNet (Dai al., 2021)) across all scales.
scenarios (Zhu et al., 2021; Jiang et al., 202 Liu et al., 2022a). Additionally, the dearth of local bias induces the detriment of neighbor- hood correlations (Pinto al., 2022).
To resolve this problem, endeavors have been made by reintroducing locality priors (Wu et al., 202 Dai al., 2021; Han al., 2021a; a; Li al., 2022a: Chen al., 2022) and pyramid-like hier-
First authors contribute equally. Coorsponddng author (stan. @wwett lake. ed).

---

Published as a conference paper at ICLR 2024
archical layouts (Liu et al., 2021; Fan al., 2021; Wang et al., 2021b) ViTs, albeit the expense of model generalizability and expressivity. Meanwhile, further explorations toward ViTs (Tolstikhin et al., 2021; Raghu et al.. 2021; Yu et al., 2022) have triggered the resurgence of modern Con- vNets (Liu et al., 2022b; Ding et al., 2022b). With advanced training setup and ViT-style framework design, Conv can readily deliver competitive performance w.E.. well-tuned ViT's across wide range of vision benchmarks (Wightman et al., 2021; Pinto et al., 2022). Essentially, most of the mod- ern ConvNets aim perform feature extraction in local-global blended fashion by contextualizing the convolutional kernel or the perception module as global as possible.
Despite their superior performance, recent progress on multi-order game-theoretie interaction within DNNs (Ancona et al., 2019b; Zhang et al.. 2020; Cheng et al., 2021) unravels that the representation capacity of modern ConvNets has not been exploited well. Holistically, low-order interactions tend to model relatively simple and common local visual concepts, which are of poor expressivity and are incapable of capturing high-level semantic patterns. In comparison, the high-order ones represent the complex concepts of absolute global scope yet are vulnerable to attacks and with poor generalizabil- ity. Deng et al. (2022) first shows that modern networks are implicitly prone to encoding extremely low- high-order interactions rather than the empirically proved more discriminative middle ones. Attempts have been made to tackle this issue from the perspective of loss function (Deng et al., 2022) and modeling contextual relations (Wu et al., 2022a; Li et al., 2023a). This unveils the serious challenge but also the great potential for modern ConvNet architecture design.
To this end, we present new ConvNet architecture named Multi-order gated aggregation Network (MogaNet) to achieve adaptive context extraction and further pursue more discriminative and effi- cient visual representation learning first under the guidance of interaction within modern ConvNets. In MogaNet, we encapsulate both locality perception and gated context aggregation into a compact spatial aggregation block, where features encoded by the inherent overlooked interactions are forced to congregated and contextualized efficiently in parallel. From the channel perspective, as existing methods are prone to huge channel-wise information redundancy (Raghu et al., 2021; Hua et al., 2022), we design conceptually simple yet effective channel aggregation block to adaptively force the network to encode expressive interactions that would have originally been ignored. Intuitively, it performs channel-wise reallocation to the input, which outperforms prevalent counterparts (e.g., SE (Hu et al., 2018), RepMLP (Ding et al., 2022a)) with more favorable computational overhead.
Extensive experiments demonstrate the consistent efficiency of model parameters and competitive performance of MogaNet at different model scales on various vision tasks, including image classi- fication, object detection, semantic segmentation, instance segmentation, pose estimation, etc. As shown in Fig. 1, MogaNet achieves 83.4% and 87.8% top-1 accuracy with 25M and 181M parame- ters, which exhibits favorable computational overhead compared with existing lightweight models. MogaNet-T attains 80.0% accuracy on ImageNet- -K, outperforming the state-of-the-ar ParC-Net- S (Zhang et al., 2022b) by 1.0% with 2.04G lower FLOPs. MogaNet also shows great performance gain on various downstream tasks, e.g., surpassing Swin-L (Liu et al., 2021) by 2.3% AP on COCO detection with fewer parameters and computational budget. It is surprising that the parameter effi ciency of MogaNet exceeds our expectations. This is probably owing to the network encodes more discriminative middle-order interactions, which maximizes the usage of model parameters.
2
RELATED WORK
2.1 VISION TRANSFORMERS
Since the success of Transformer (Vaswani et al., 2017) in natural language processing (Devlin al.. 2018), ViT has been proposed (Dosovitskiy et al., 2021) and attained impressive results on ImageNet (Deng et al., 2009). Yet, compared to ConvNets, ViTs are over- over-parameterized and rely on large-scale pre-training (Bao et al., 2022; He et al., 2022; Li et al., 2023a). Targeting this problem, one branch of researchers presents lightweight ViTs (Xiao et al., 2021; Mehta & Rastegari, 2022; Li et al., 2022c; Chen et al., 2023) with efficient attentions (Wang et al., 2021a). Meanwhile, the hybrid backbone has been studied (Guo et al.,
incorporation of self-attention and convolution 2022: Wu et al., 2021a; Dai et al., 2021; d' Ascoli et al., 2021; Li et al., 2022a; Pan et al., 2022b; Si et al., 2022) for imparting locality priors to ViTs. By introducing local inductive bias (Zhu et al., 2021: Chen et al., 2021; Jiang et al., 2021a; Arnab et al., 2021), advanced training strategies (Tou- vron al., 2021a; a; Yuan et al., 2021a; Touvron al., 2022) extra knowledge (Jiang et al., 2021b;
2

---

Published as a conference paper at ICLR 2024
3 XXXI W
G x
G x x
N W
C3 x 16 16
H W
C, x 32
H
Input
8
Slage
Slage
Stage
Stage 4
Spaii Charnel Nopresatiion
Ston
Spar
Agp Agp Charrel
Agg Aagrrpation
Entreating
Spalial
fqpregatioon
Charnel
Aggresstion
Emnediing Ohmn
Spatiel
Agg Charnal
Aggi Aggitigation
Fc Laver
Moga Block
Block
Moga Block
Moga Block
223
Figure 2:
MogaNet architecture with four stages. Similar to (Liu et al., 2021; 2022b), MogaNet
uses hierarchical architecture of 4 stages. Each stage consists of an embedding stem and N, Moga Blocks, which contain spatial aggregation blocks and channel aggregation blocks.
Lin et al., 2022; Wu al., 2022c), ViTs can achieve superior performance and have been extended to various vision areas. MetaFormer (Yu et al., 2022) considerably influenced the roadmap of deep architecture design, where all ViTs (Trockman & Kolter, 2022; Wang et al., 2022a) can be classified by the token-m token-mixing strategy, such as relative position encoding (Wu et al., 2021b), local window shifting (Liu et al., 2021) and MLP layer (Tolstikhin et al., 2021), etc.
2.2 POST- VIT MODERN CONVNETS
Taking the merits of ViT-style framework design (Yu et al., 2022), modern ConvNets (Liu et al., 2022b; 2023; Rao et al., 2022: Yang et al., 2022) show superior performance with large kernel depth-wise convolutions (Han et al., 2021b) for global perception (view Appendix E for detail back- grounds). It primarily comprises three components: (i) embedding stem, (ii) spatial mixing block, and (iii) channel mixing block. Embedding stem downsamples the input to reduce redundancies and computational overload. We assume the input feature X is in the shape CCHHW
we have:
= Stem
(1)
where Z is downsampled features, e.g... Then, the feature flows to a stack of residual blocks. In each stage, the network modules can be decoupled into two separate functional components, SMixer(.) and CMixer(-) for spatial-wise and channel-wiss information propagation,
Y X + SMixer (Norm(X)), Z = Y + CMixer (ooxerrNorm(Y)))
(3)
where Norm(.) denotes a normalization layer, e.g., BatchNorm (loffe Szegedy, 2015a) (BN). SMixer(.) can be various spatial operations (e.g.. self-attention, convolution), while CMixer(-) is usually achieved by channel MIP with inverted bottleneck (Sandler et al., 2018) and expand ratio Notably, we abstract context aggregation in modern ConvNets as series of operations that can adaptively aggregate contextual information while suppressing trivial redundancies in spatial mixing block SMixer(.) between two embedded features:
0 ((Fa(X), Gpp (X)),
where (-) and (-) are the aggregation and context branches with parameters 6 and v.. Context aggregation models the importance of each position on X by the aggregation branch Fx(X) and reweights the embedded feature from the context branch Go (X) by operation S(..)).
MULTI-ORDER GAME-THEORETIC INTERACTION FOR DEEP ARCHITECTURE DESIGN
Representation Bottleneck of DNNs Recent studies toward the generalizability (Geirhos et al., 2019; Ancona et al., 2019a; Tuli et al., 2021; Geirhos et al., 2021) and robustness (Naseer et al., 2021: Zhou et al., 2022; Park & Kim, 2022) of DNNs delivers new perspective improve deep architectures. Apart from them, the investigation multi-order game-theoretic interaction unveils the representation bottleneck of DNNs. Methodologically, multi-order interactions between put variables represent marginal contribution brought by collaborations among these two and other involved contextual variables, where the order indicates the number of contextual variables within the collaboration. Formally, it can be explained by m-th order same-theoretic interaction (im) and m--rddrr interaction strength f(m), as defined in (Zhang et al., 2020; Deng et al., 2022). Con- sidering the image with n patches in total, (m) (i.j) measures the average interaction complexity between the patch pair i over all contexts consisting of m patches, where 0 S m S 2 and the
in-
3

---

Published as a conference paper at ICLR 2024
order reflects the scale of the context involved in the game- interactions between pixels i and j. Normalized by the average of interaction strength, the relative interaction strength J(m) with m (0, 1) measures the complexity of interactions encoded in DNNs. Notably, low-order interac- tions tend to encode common or widely- shhree- local texture, and the high-order ones are inclined to forcibly memorize the pattern of rare outliers (Deng et al., 2022; Cheng et al., 2021). As shown in Fig. 3, existing DNNs are implicitly prone to excessively low- or high-order interactions while suppressing the most expressive and versatile middle-order ones (Deng et al., 2022; Cheng et al., 2021). Refer to Appendix B.1 for definitions and more details.
Multi-order Interaction for Architecture Design. Existing deep architecture design is usually derived from intuitive insights, lacking hierarchical theoretic guidance. Multi-order interaction can serve as ref- erence that fits well with the already gained insights on computer vision and further guides the ongoing quest. For instance, the extremely high-order interactions en- coded in ViTs (e.g., DeiT in Fig. 3) may stem from its
Interaction Strength of Order
DeiT (Small) 22M
ConvNeXt (Tiny), 29M
Interaction Strength
SLaK (Tiny), 30M
3
MogaNet (Small), 25M
2
adaptive global-range self-attention mechanism. Its su-
perior robustness can be attributed to its excessive low-
order interactions, representing common and widely shared local patterns. However, the absence of local- ity priors still leaves ViTs lacking middle-order inter- actions, which cannot be replaced by the low-order ones. As for modern ConvNets (e.g., SLaK in Fig. 3). despite the 51 x 51 kernel size, it still fails to en- code enough expressive interactions (view more results in Appendix B. Likewise, we argue that such dilemma may be attributed to the inappropriate com- position of convolutional locality priors and global context injections (Treisman & Gelade, 1980; Tuli et al., 2021; Li et al., 2023a). A naive combination of self-attention or convolutions can be in- trinsically prone the strong bias of global shape (Geirhos et al., 2021: Ding et al., 2022b) or local texture (Hermann et al., 2020), infusing extreme-order interaction preference to models. In Mo- gaNet, aim to provide architecture that adaptively force the network to encode expressive interactions that would have otherwise been ignored inherently.
0.1 0. 0.5
0.7 0.9 1.0
Order / n
Figure 3: Distributions of the interac- tion strength (Tm) for Transformers and ConvNets ImageNet- with 224 res- olutions and n = 14 14.
4
METHODOLOGGY
4.1
OVERVIEW OF MOGANET
Built upon modern ConvNets, we design four-stage MogaNet architecture illustrated in Fig. 2. For stage i, the input image or feature is first fed into an embedding stem to regulate the resolutions and embed into Ca dimensions. Assuming the input image in H x W resolutions, features of the four W resolutions respectively. Then, the embedded 32
stages are in x I
W
x
W
8
16
H
x W
16
and x
32
Accuracy yz. MLP Ratio vz. Param.
Norm drec -
115
SLUSE
Norm
d
Gomy
24. AM
DVConn 3x3
GELU
Suberaac
Cone Telt
Coea Tal
Aggrogation
Ye
MI Pa Si moudia CAF)
Mcli-Crrter
Apprepation
SLU
Donvy 3xr1
CoepENt
Xout
Oubuu
Output
MLP Channel Expand Rarior
(a) Moga(.) Block
(b) CA(.) Block
(c)
Figure 4: (a) Structure of spatial aggregation block Moga(-). (b) Structure of channel aggre- gation block. (c) Analysis of channel MIP and the channel aggregation module. Based on
MogaNet-S, performances and model sizes of the channel MLP, MLP with SE block, and the channel aggregation is compared with the MLP ratio of {2, 4, 6, 8} on ImageNet-IK.
4

---

Published as a conference paper at ICLR 2024
wwo Gatlng
1:0:0
O:
lwo FD(.)
lwo CA(.)
MogaNel
Modules
Top-1
Params. FLOPs
Acc (%)
(M)
(G)
Baseline
76.6
4.75
1.01
+Gating brunch
77.3
1.07
SMiser
+DW Tx7
*Multi-order ()
78.0
77.5
5.14
5.17
1.09
1.10
Figure 5: Grad-CAM visualization of ablations. 1:0: 0 and
+FD(.)
78.3
5.18
1.10
O: 0: I denote only using C₁ or C₁ for Multi-order DWConv
CMixer
(CA(.)
+SF module
79.0
78.6
5.29
1.14
Layers in SA block. The models encoded extremely low- Table 1: Ablation of designed mod-
(C2) or high- (C₂) order interactions are sensitive to similar ules on ImageNet-IK. The baseline
regional textures (1: O: 0) or excessive discriminative parts (0: uses the non-linear projection and 0: I), not localizing precise semantic parts. Gating effectively DWaxs as SMixer(-) and the vanilla climinates the disturbing contextual noise (/wo Gating). MLP as CMixer(-).
feature flows into N₂ Moga Blocks, consisting of spatial and channel aggregation blocks (in Sec. 4.2 and 4.3), for further context aggregation. After the final output, GAP and linear layer are added for classification tasks. As for dense prediction tasks (He et al., 2017; Xiao et al., 2018b), the output from four stages can be used through neck modules (Lin et al., 2017a; Kirillov et al., 2019).
4.2 MULTI-ORDER SPATIAL GATED AGGREGATION
As discussed in Sec. 3, DNNs with the incompatible composition of locality perception and con- text aggregation can be implicitly prone to extreme-orde game-theoretic interaction strengths while suppressing the more robust and expressive middle-order ones (Li al., 2022a; Pinto et al., 2022; Deng et al., 2022). As shown in Fig. 5, the primary obstacle pertains to how to force the network to encode the originally ignored expressive interactions and informative features. We first suppose that the essential adaptive nature of attention in ViTs has not been well leveraged and grafted into ConvNets. Thus, we propose spatial aggregation (SA) block as an instantiation of SMixer(-) to learn representations of multi-order interactions in a unified design, as shown in Fig. 4a, consisting of two cascaded components. We instantiate Eq. (2) as:
Z X + Moga (N(Norn(X)))
(5)
where FD(.) indicates a feature decomposition module (FD) and Moga(-) denotes a multi-order gated aggregation module comprising the gating F.).) and context branch (.).)
Context Extraction. As pure ConvNet structure, we extract multi-order features with both static and adaptive locality perceptions. There are two complementary counterparts, fine-grained local texture (low-order) and complex global shape (middle-order), which are instantiated by Couvixil (-) and GAP(-) respectively. To force the network against its implicitly inclined interaction strengths, we design FD(-) to adaptively exclude the trivial (overlooked) interactions, defined as:
Y = ConvixI
Z = GELU (Y + +7 O (V (Y-OO((YY))))) GAP(Y)))
where 7 E RCx1 denotes a scaling factor initialized as zeros. By re-weighting the complemen- GYP FD(() also increases spatial feature diversities (Park &
tary interaction component Kim, 2022; Wang et al., 2022b). Then, we ensemble depth-wise convolutions (DWConv) to encode multi-order features in the context branch of Moga(-). Unlike previous works that simply combine DWConv with self-attentions to model local and global interactions (Zhang et al., 2022b; Pan et al., 2022a; et al., 2022; Rao et al., 2022). we employ three different DWConv layers with dilation ratios d E {1, 3} in parallel to capture low, middle, and high-order interactions: given the input feature X E CRCW DW 5x5,d=1 is first applied for low-order features; then, the output is fac- torized into X, E CCCIHHW Xm, E RC_XHW and X,, E RR&xHI along the channel dimension, where C + Cm + Cp C: afterward, Xm and Xx, are assigned to DW DW 5x5,d=2 and DW 7,7,d=3+ respectively, while X₁ serves as identical mapping: finally, the output of Xt, Xm. and X₁. are con- catenated to form multi-order contexts, Yc- Concat (1,,,, Y... Yx). Notice that the proposed FD(.) and multi-order DWConv layers only require a little extra computational overhead and pa- rameters in comparison to DW7x7 used in ConvNeXt (Liu et al., 2022b), e.g., +multi-order and +FD(-) increase 0.04M parameters and 0.01G FLOPS over DW7x7 as shown in Table 1.
Gated Aggregation. To adaptively aggregate the extracted feature from the context branch, we employ SiLU (Elfwing et al., 2018) activation in the gating branch, ie.. x Sigmoid(z), which has been well-acknowledged pendix C. we empirically show that SiLU in MogaNet exhibits both the gating effects as Sigmoid and the stable training property. Taking the output from FD(.) as the input, we instantiate Eq. (4):
advanced version of Sigmoid activation. As illustrated in Ap-
5

---

Published as a conference paper at ICLR 2024
Z SiLU (ion SiLU (io (Yc))
(8)
With the proposed SA blocks, MogaNet captures more middle-order interactions, validated in Fig. 3. The SA block produces discriminative multi-order representations with similar parameters and FLOPss DW7x 7x7 in ConvNeXt, which is well beyond the reach of existing methods without the cost-consuming self-attentions.
4.3 MULTI-ORDER CHANNEL REALLOCATION
Prevalent architectures, as illustrated in Sec. 2, perform chamellmmxing CMixer(-) mainly by two linear projections, e.g., 2-layer channel- wise MLP (Dosovitskiy et al., 2021; Liu et al.,
CA
2021: Tolstikhin et al., 2021) with a expand ra- tio the MLP with 3 3 DWConv in be- tween (Wang et al., 2022c; Pan et al., 2022b;a). Due to the information redundancy cross chan- nels (Woo et al., 2018; Cao et al., 2019; Tan & Le, 2019; Wang et al., 2020), vanilla MLP re- quires a number of parameters (r default to 4 or 8) to achieve expected performance, show- ing low computational efficiency as plotted in Fig. 4c. To address this issue, most current meth- ods directly insert channel enhancement mod- ule, e.g. SE module (Hu et al., 2018), into MLP. Unlike these designs requiring additional MLP bottleneck, motivated by FD(.). we introduce a lightweight channel aggregation module CA(-) to adaptive reallocate channel-wise features in nigh-dimensional hidden spaces and further extend to a channel aggregation (CA) block. As shown in Fig, 4b, we rewrite Eq. (3) for our CA block as:
2.5
tnerge
52 aes IA
66 53 ar CA
Lodyling
52 CA
c.a
CA
ca
Iwo
Figure 6: Channel energy ranks and channel saliency maps (CSM) (Kong et al., 2022) with or without our CA block based on MogaNet-S. The energy reflects the importance of the chan- nel, while the highlighted regions of CSMs the activated spatial features of each channel.
GELU (DW3x 3x3 (C (Nomm((())))
7 = Convix1 x (CA(Y)) + X.
Concretely, (-).) is implemented by a channel-reducins projection W, HC + +RIHHW and GELU to gather and reallocate shannel-wist information:
CA(X)= X + Th @ (X GFLUUXX.)).,
(10)
where %e is the channel-wise scaling factor initialized zeros. It reallocates the channel-wise feature with the complementary interactions (X GELU(X W,)). As shown in Fig. 7, CA(-) en- hances originally overlooked gamee-therrede interactions. Fig. 4e and Fig. 6 verify the effectiveness of C-(-) compared with vanilla MLP and MLP with SE module in channel-wist effiency and repre- sentation ability. Despite some improvements the baseline, the MIP w/ SE module still requires large MIP ratios (e.g., r == 6) to achieve expected performance while bringing extra parameters and overhead. Yet, our C-(-) with r 4 brings 0.6% gain over the baseline small extra cost (0.04M extra parameters & 0.01G FLOPs) while achieving the same performance as the baseline with r = 8.
4.4 IMPLEMENTATION DETAILS
Following the network design style of ConvNets (Liu al., 2022b), we scale up MogaNet for six model sizes (X- XX-Tiny, Tiny. Small, Base, Large, and X-Large) via stacking the different number of spatial and channel aggregation blocks at each stage, which has similar numbers of parameters as RegNet (Radosavovic et al., 2020) variants. Network configurations and hyper-parameters are detailed in Table A FLOPs and throughputs are analyzed in Appendix C.3. We set the channels of the multi-order DWConv layers to Ct C. C. 1:3:4 (see Appendix C.2). Similar to (Touvron et al., 2021c; Li et al., 2022a;c), the first embedding stem in MogaNet is designed as two stacked 3x3 convolution layers with the stride of 2 while adopting the single-layer version for embedding stems in other three stages. We select GELU (Hendrycks & Gimpel, 2016) as the common activation function and only use SiLU in the Moga module as Eq. (8).

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
To impartially evaluate and compare MogaNet with the leading network architectures, we conduct extensive experiments across various popular vision tasks, including image classification, object detection, instance and semantic segmentation, 2D and 3D pose estimation, and video prediction. The experiments are implemented with PyTorch and run on NVIDIA A A100 GPUs.
5.1 IMAGENET CLASSIFICATION
Settings. For classification experiments on ImageNet (Deng et al., 2009), we train our MogaNet following the standard procedure (Touvron et al., 2021a; Liu al., 2021) on ImageNet- IK (IN-IK) for a fair comparison, training, 300 epochs with AdamW (Loshchilov & Hutter, 2019) optimizer, basic learning rate of 1 x 10 and a cosine scheduler (Loshchilov & Hutter, 2016). To explore the large model capacities, then fine-tuned 30 epochs on IN-IK following (Liu et al., 2022b). Appendix A.2 and D.I provide implementation details and more results. We compare three elassical architectures: Pure ConvNets (C), Transformers (T), and Hybrid model (H) with both self-attention and convolution operations.
pre- trained MogaNet-XL. on ImageNet-21IK (IN-21K) for 90 epochs and
Results. With regard to the lightweight models, Table 2 shows that MogaNet-XT/T significantly outperforms existing lightweight architectures with a more efficient usage of parameters and FLOPs. MogaNet-T achieves 79.0% top-1 accuracy, which improves models with ~5M parameters by at least 1.1 at 224 resolutions. Using 256² resolutions, MogaNet-T outperforms the current SOTA ParC-Net-S by 1.0 while achieving 80.0% top-1 accuracy with the refined settings. Even with only 3M parameters, MogaNet-XT still surpasses models with around 4M parameters, egg. +4.6 over T2T-VVT-7. Particularly, MogaNet-TS achieves 80.0% top-1 accuracy using 2562 resolutions and the refined training settings (detailed in Appendix C.5). As for scaling up models in Table 3, Mo- gaNet shows superior or comparable performances to SOTA architectures with similar parameters and computational costs. For example, MogaNet-S achieves 83.4% top-1 accuracy. outperforming Swin-T and Con with clear margin of 2.1 and 1 MogaNet-B/L. also improves recently proposed ConvNets with fewer parameters, e.g., +0.3/0.4 and +0.5/0.7 points over HorNet-S/B and SLaK-S/B. When pre-trained on IN-21K, MogaNet-XL. XL is boosted to 87.8% top-1 accuracy with 181M parameters, saving 169M compared to ConvNeXt-XLS Noticeably, MogaNet-XL -L can achieve 85.1% at 224 resolutions without pre- tre-training and improves Conv by 0.8, indicating Mo- gaNets are easier converge than existing models (also verified in Appendix D.1).
5.2 DENSE PREDICTION TASKS
Object detection and segmentation on COCO. We evaluate MogaNet for object detection and instance segmentation tasks on COCO (Lin et al., 2014) with RetinaNet (Lin et al., 2017b), Mask- RCNN (He et al., 2017), and Cascade Mask R-CNN (Cai & Vasconcelos, 2019) as detectors. Fol- lowing the training and evaluation settings in (Liu et al., 2021; 2022b), fine-tune the models by the AdamW optimizer for 1x and 3x training schedule on COCO Irain2017 and evaluate on MMDetection (Chen et al., 2019) codebase. The box mAP (AP*)
COCO val2017, implemented and mask mAP (AP") are adopted as metrics. Refer Appendix A.3 and D.2 for detailed settings and full results. Table 4 shows that detectors with MogaNet variants significantly outperform previous backbones. It is worth noticing that Mask R-CNN with MogaNet-T achieves 42.6 AP", outperform- ing Swin-T by 0.4 with 48% and 27% fewer parameters and FLOPs. Using advanced training setting and IN-21K pre-trained weights, Cascade Mask R-CNN with MogaNet-XL achieves 56.2 AP* +1.4 and +2.3 over ConvNeXI-L and RepLKNet-3IL.
Semantic segmentation ADE20K. We also evaluate MogaNet for semantic segmentation tasks on ADE20K (Zhou et al., 2018) with Semantic FPN (Kirillov et al., 2019) and UperNet (Xiao et al., 2018b) following (Liu et al., 2021; Yu et al., 2022), implemented utors, 2020b) codebase. The performance is measured by single-scale mloU. Initialized by IN-1K or IN-21K pre-trained weights, Semantic FPN and UperNet are fine-tuned for 80K and 160K iter- ations by the AdamW optimizer. See Appendix A.4 and D.3 for detailed settings and full results. In Table 5, Semantic FPN with MogaNet-S consistently outperforms Swin-T and Uniformer-S by 6.2 and 1.1 points; UperNet with MogaNet-S/BB/ improves ConvNeXI--T/S T/S/B by 2.5/1.4/1.8 points. Using higher resolutions and IN-21K pre-training. MogaNet-XL achieves 54.0 SS mloU, surpassing ConvNeXt-L. and Repl. KNet-311.. by 0.3 and 1.6.
MMSegmentation (Contrib-
7

---

Published as a conference paper at ICLR 2024
Archisecture
Date
Type Image Param. FLOPs Top-1
Architecture
Data
Method
Param,
FLOPS
AP" APM
Size
(M)
(G)
Acc (Sc(S)
(M)
(%)
ResNet-18
CVPR 2016
C
224-
11.7
1.80
71.5
ResNer-101
CVP RetinaNet
57
315
38.5
ShuffleNetV22 2x
2018
224º
5.5
0.60
75.4
PVT-S
ICCV'2021 RetinaNet
34
226
40.4
EllcientNet-B0
ICML'2019
C
224"
5.3
0.39
77.1
CMT-S
CVPR 2022
RetinaNet
45
231
44.3
NogNe1Y-800MF
2020
C
224º
6.3
0.80
76.3
MogaNet-S
Ouss
RetinaNet
35
253
45.8
DeiT-T
ICML/2021
T
224"
5.7
1.08
74.1
RogNsT-L6G
202R Mask R-CNN
29
389
PVIT
ICCV22021
T
224
13.2
1.60
75.1
PVTT
2021
Mask R-CNN
33
208
36.7 35.1
SReT-T
T2T-VI-7
VTT
PTT:T
NIPS'2021
ICC1
2C22
ICCV
T 224
II 224
T 224"
T
224"
4.8
4.3
4.9
4.6
0.70
1.10
1.10
1.20
77.6
71.7
74.6
75.3
ConvNeXi-T
MogaNet-T
Uniformer-S
Swin-T
PVTV2-82
1022'2022
CVM 2022
Ours
10C1
2V22
Mask R-CNN
Mask R-CNN
Mask R-CNN
Mask R-CNN
Mask R-CNN
41
45
48
48
25
269
309
192
44.2
42.6 39.1
45.3 41.2
42.2 39.1
45.6 41.6
LeWT-S
ICCV
H 224"
7.8
0.31
76.6
LITV2-S
20PS
Mask R-CNN
47
261
44.9 40.8
Lit--
ICCV
II 224
5.7
1.60
77.5
FocalNet-T
21P5
R-CNN
49
45.9
Swin-IG
ICCV
H 224
7.3
1.00
77.3
MogaNet-S
Our
Mask R-CNN
45
272
46.7
42.2
MobileViT-S
ICLR'2022
1I 256
5.6
4.02
78.4
Swin-S
202T Mask R-CNN
69
44.8
40.9
Mobile
294M
CVPR 2022
H 224
11.4
0.59
77.9
Focal-S
N1P5'2021
Mask R-CNN
71
401
47.4 42.8
Con:Next-XT
CVPR'2022
C
224
7.4
0.60
77.5
ConvNeXi-S
CVPR22022
Mask R-CNN
70
348
45.4 41.8
VAN-BO
CVMJ'2023
224
4.1
0,88
75.4
HorNet-Brut
2125
Mask
68
47.
PerC-Net.S
ECCV
C
258
5.0
3.48
78.6
MogaNet-B
Ours
Misk R-CNN
63
375
47.9
43.2
MogaNet-XT
Ours
256
3.0
1.04
77.2
Swin-L'
ICCV Cascade Mask
253
53.9
MugaNet-T
Ours
224
5.2
1.10
79.0
ConvNeXo.
CVPR Cascade Mask
255
1354
54.8 47.6
MogaNet-
Ours
256
5.2
1.44
80.0
Repl KNe--311.
CVPR 2022 Cascade Mask
229
53.9 46.5
Table 2: IN-1K classification with lightweight
HorNet-L."
NIPS 2022
Casende Mask 259
1399
5600 48.6
MoL
Ouns
Cascade Mask 238
1355
56.2 48.8
models. 8 denotes the refined training scheme.
Table 4: COCO object detection and instance
Architecture
Date
Type Image Param.
FLOPs
TTop-1
segmentation with RetinaNet (1x), Mask R-
Size
(G)
Acc
Den-S
(021
T
221"
22
4.6
79.8
CNN (1x), and Cascade Mask R-CNN (multi-
Swin.T
1CCV'2021
224
4.5
81.3
scale 3x). indicates IN-21K pre-trained mod-
CSWin-T
CVPR
T
22.4
23
4.3
82.8
els. The FL.OPs are measured at 800 x 1280.
LITV2-S
NIPS'2022
224
3.7
82.0
CoaT-S
CoAtNes-O
NIPS 2021
1CCV
H
H 22.4'
224"
25
22
12.6
4.2
82.7
82.1
Method
Architecture
Date
Crop
size
Param,
(M)
FLOPs mlot.
((i)
(16
UniFormer-S
1C22
H 2212
22
3.6
82,9
PVT-S
1CCV'2021
512"
28
161
39.8
RegNecY-4GIi
CVPR 2020
C
2242
21
4.0
81.5
Semantic
Twins-S
NIPS'2021
512"
28
162
44,3
ComvNeXL-T
CVPR-2022
2244
29
4.5
82.1
FPN
Swin-T
ICCV
5122
32
182
41.5
SLaK-T
ICLR'2023
C
224"
30
5.0
82.5
(80K)
Uniformer:S
2022
512"
25
247
46.6
Hor
NIPS'2022
2244
22
4.0
82.8
LITV2-S
NIPS'2022
512"
31
179
44.3
MogaNet-S
Ours
C
224"
25
5.0
83.4
VAN-B2
CVMD 2023
512"
30
164
46.7
Swan-S
1CCV
2214
8.7
MogaNet-S
Ours
512"
29
189
47.7
Focal-S
NIPS 2021
T
22
51
9.1
83.6
DeiT-S
ICML/2021
512"
52
1099
44.0
CSWin
CVPR-2022
22.4
6.9
83.6
Swin T
ICCV'2021
512"
60
945
46,1
LITV2-M
NIPS'
T 22:4"
49
7.5
83.3
ConvNeXI-T
CVPR'2022
512?
46.7
CoaT-M
ICCY'2021
2242
9.8
83.6
UniTormesS
ICLR
5122
52
1008
47.6
CoAtNes-1
N1PS
H 22:4"
42
8.4
83.3
HorNet-Tysr
NIPS'2022
512"
52
926
48.1
UniFormer-B
ICLR'2022
H 22/12
50
8.3
83.9
MogaNet-S
512"
55
946
49.2
FAN-B-Hyybid
EfficieatNe--66
ICML/2022
[CML'2019
H 22
C 5262
43
50
19.0
11.3
84.0
83.9
UperNet
Swin-S
ConvNeXi-S
SLaK-S
ICLR
KCVV2221
CVPR22022
312"
512"
5122
81
91
82
1028
1027
10.38
48.1
49.4
48.7
RegN
CVPR
224"
39
8.1
82.2
(160K)
MogaNes-B
Ours
5122
74
1050
50.1
ConvNeXt-S
CVP2
C
22(1'
50
8.7
83.1
Swin-B
CCCV2221
312F
121
1188
49.7
FocalNet-S (LRF)
NIPS-2022
22-2
50
8.7
83.5
ConvNeXi-B
CVPR22022
5122
122
1170
49.1
HorNet-S7s7
2IPS-2022
C
221"
50
8.8
84.0
3epLKNer:31B
CVPR/2022
5124
112
1170
49.9
SLaK-S
CCLR22023
2242
9.8
St SLak-B
IC1
5122
135
1185
50.2
MogaNet-BB
Ours
224
444
9.9
84.3
MogaNet.L.
Ours
5122
113
1176
50.9
DeiT-B
CCMI/2021
22.4
17.5
81.8
Swin-L?
ICCV
640
234
2468
52.1
Swin-B
ICCY22021
T 22/1"
89
15.4
83,5
Cow-NeXI-L
CVPR'2022
646
245
2158
53.7
Focal-B
NIPS
T 22122
90
16.4
84.0
CVPR'2002
6402
207
2414
CSWin-B
CVPR'2022
T
22
78
15.0
84.2
MogaNNetXI.
Ours
6407
214
2451
54.0
DI-T
BCC2
T 22112
87
18.0
83.8
Table 5: ADE20K semantic segmentation with
BoTNeL-T7
CVPR'2021
11
2564
79
19.3
84.2
semantic FPN (80K) and UperNet (1 (160K). $ in-
CoAINet-2
NIPS 1021
H 22.4
75
15.7
84.1
BAN-B-Ilybrd
ICML/2022
II 224""
77
16.9
84.3
dicates using IN-21K pre-trained models. The
RegNetY-i6GF
CVPR'2020
C
22.4
84
16.0
82.9
FLOPs measured at 512x 2048 or 640 x 2560.
ConvNeXi-B
CVPR'2022
22-1"
189
15.4
83.8
RepLKNet-318
CVP2
C
22
79
15.3
83.5
Architecture Date
Crop
Param. FLOPs AP AP AP AR
FocalNet-B (LRF)
20PS'2022
22/F
889
15.4
83.9
size
(M)
(G)
(%)
(%)
(%) (%)
HorNet-Brur
NIPS 2022
C
22
87
15.6
84.3
RSN-T8
BCCV 2020 256 x 192
9.1
2.3
70.4
58.7 77.9 77.1
SLaK B
ICLR22033
224
17.1
MogaNet-T Ours
x
2.2
73,2
78.8
MogaNet-LL
Ours
2244
83
15.9
84.7
HRNe: W32 CVPR 2019 256 x 192
28.5
7.1
74,4
90.5
81.9 78.9
Swin-L!
ICCV
3381"
197
104
87.3
Swin-T 10CY
256 x 192
32.8
6.1
72.4
90.1
30.6
78.2
CoAINet-3t
DeiT III-1:
NIPS
20C2
T
H 13813
3844
168
304
107
191
87.7
87.6
Com CVPR 20022
PVTV2-B2 CVML-2022
Uniformer ICLR'2022
256 x 192
256 x 192
x
29.1
33.1
25.2
4.3
5.5
4.7
73.7
76,0
73.2
90.0
90.5
80.9
78.8
79.5
79.1
CVPR-2022
384
96
86.6
MogaNet-8 Ours
256 x 192
29:0
6.0
74.9
90.7
82.8
80.1
ComoNeXt-I.
CVPR'2022
C 22.4
198
34.4
84.3
Unitormar 3 2CL
381 x 288
25.2
11.1
75.3
90.6
83.4 BLT
CoavNeXi-LI
CVPR'2022
38'1"
198
101
87.5
Non-N CVPR
33.1
80.5
CoI
CVPR-2022
C
384"
350
179
87.8
MogaNet-S Ours
354 x 288
24.0
13.5
76.4
91.0
W3.3
81.4
HorNet-L
NIPS'2022
C
384"
202
102
87.7
TIRNGE W18 CVP
351 x 265
03.6
32.9
76.3
90.5
$2.0
Mo
MogaNet-XL
Ours
C 3844
2212
181
181
34.5
102
87.8
85.1
Uniformer-B ICLR'2022
MogeNet-B Qurs
Swin-L 10CV
384
384 x 288
384 x 288
203.4
47.4
24.4
86.9
14.8
77.3
76.7
76.3
91.2
84.0 82.2
83.0
81.4
814
Table 3: IN-1K classification performance with
Table 6: COCO 2D human pose estimation with Top-Down SimpleBaseline. The FLOPs are measured at 256 x 192 or 384 x 288.
scaling-up models. $ denotes the model is pre-
trained on IN-21K and fine-tuned on IN-IK.
8

---

Published as a conference paper at ICLR 2024
Interaction Strength of Order
DeiT-S
Swin-T
ResNet-50
ConvNeXt-T
MogaNet-S
Modules
Top-1
Baseline (5mal)) 23M Mogul.),
Acc (%)
Interactoo Strenghh
ConvNeXi-1
82.
-Mugs,) + CAL 2SM
Raseline
82.2
Moga Block
83.4
-FD()
83.2
(Muli-DW())
83.1
()
82.7
CA(-)
82.9
0.1 0.3
0.5
7.7 9.9 1.0
Order
Figure 7: Ablation of proposed modules on IN- Figure 8: Grad-CAM activation maps on IN- IK Left: the table ablates MogaNet modules by IK. MogaNet exhibits similar activation maps as removing each of them based on the baseline of attention architectures (Swin), which are located MogaNet-S. Right: the figure plots distributions on the semantic targets. Unlike previous Con- of interaction strength ((m,, which verifies that vNets that might activate some irrelevant regions, Moga(-) and C-(-) both contributes to learning activation maps of MogaNet are more semanti- multi-order interactions and better performance. cally gathered. See more results in Appendix B.2.
2D and 3D Human Pose Estimation. We evaluate MogaNet on 2D and 3D human pose estima- tion tasks. As for 2D key points estimation on COCO, we conduct evaluations with SimpleBase- line (Xiao et al., 201 8a) following (Wang et al., 2021b: Li et al., 2022a), which fine-tunes the model for 210 epoch by Adam optimizer (Kingma & Ba, 2014). Table shows that MogaNet variants yield at least 0.9 AP improvements for 256 x 192 imput, e.g., +2.5 and +1.2 over Swin-T and PVTV2-B2 by MogaNet-S. Using 384 x 288 input, MogaNet-B outperforms Swin-L and Uniformer-B B by 1.0 and 0.6 AP with fewer parameters. As for 3D face/hand surface reconstruction tasks on Stirling/ESRC 3D (Feng et al., 2018) and FreiHAND (Zimmermann et al., 2019) datasets, we benchmark back- bones with ExPose (Choutas et al., 2020), which fine-tunes the model for 100 epoch by Adam optimizer. 3DRMSE and Mean Per-Joint Position Error (PA-MPJPE) are the metrics. In Table 7, MogaNet-S shows the lowest errors compared to Transformers and ConvNets. We provide detailed implementations and results for 2D and 3D pose estimation tasks in Appendix D.4 and D.5.
Video Prediction. We further objectively evaluate MogaNet for unsupervised video prediction tasks with SimVP (Gao et al.,
Anchitecture
Face
3D
Video Prediction
HP. FLOPs 3DRMSD
4P FLOPs PAPPPE HP. FLOPsMSE SSIM
(M)
(G)
(M)
(G)
(mmn)
(M)
(G)
(SJ
DETES
25
6.6
252
25
4,8
7.50
46
16.9
35.2
2022) on MMNIST (Srivastava et al., 2015). where the model predicts the successive 10 frames with the given 10 frames as the in- put. We train the model for 200 epochs from scratch by the Adam optimizer and are evalu- ated by MSE and Structural Similarity Index (SSIM). Table Table 7 shows that VP with Mo- gaNet blocks improves the baseline by 6.58 MSE and outperforms ConvNeXt and Hor- Net by 1.37 and 4.07 MSE. Appendix A.7 and
Swis-T
245
30
ComNeeXi-
58
2.34
30
HorNet-T
25
5.6
2.39
25
4.3
5.23
46
16.3
29.6
93.3
MogaNet 27
6.5
2.24
27
5.0
6,08
47
16.5
25.6
94,3
Table 7: 3D human pose estimation and video pre-
diction with ExPose and SimVP on Stirling/ESRC 3D, FreiHAND, and MMNIST datasets. FLOPs of the face and hand tasks are measured at 3 x 256 and 3 224 while using 10 frames x 64 resolutions for video prediction.
D.6 show more experiment settings and results.
5.3 ABLATION AND ANALYSIS
We first ablate the spatial aggregation module and the channel aggregation module CA(-) ) in Table and Fig. (left). Spatial modules include FD(-) and Moga(.). containing the gating branch and the context branch with multi-order DWConv layers Multi-DW(.). We found that all proposed modules yield improvements with favorable costs. Appendix C provides more ablation studies. Furthermore, Fig. 7 (right) empirically shows design modules can learn more middle-order interactions, and Fig. 8 visualizes class activation maps by Crad-CAAM (Selvaraju et al., 2017) compared existing models.
CONCLUSION
This paper introduces a new modern ConvNet architecture, named MogaNet, through the lens of multi-order game-theoretic interaction. Built upon the modern ConvNet framework, we present compact Moga Block and channel aggregation module to force the network to emphasize the expres- sive but inherently overlooked interactions across spatial and channel perspectives. Extensive periments verify the consistent superiority of MogaNet in terms of both performance and efficiency compared to popular ConvNets, VITs, and hybrid architectures various vision benchmarks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Key R&D Program of China (No. 1022000111100) the National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. This work was done when Zedong Wang and Zhiyuan Chen interned at Westlake Uni- versity. We thank the Al Station of Westlake University for the support of GPUs. We also thank Mengzhao Chen, Zhangyang Gao, Jianzhu Guo, Fang Wu, and all anonymous reviewers for polish- ing the writing of the manuseript.
REFERENCES
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley value approximation. In International Conference on Machine Learning (ICML), pp. 272-281. PMLR, 2019a. 3, 23
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley value approximation. In International Conference on Machine Learning (ICML), pp. 272-281, 2019b.
Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In International Conference on Computer Vision (ICCV), 2021. 2, 34
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 607.06450, 2016. 26
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022. 2, 27, 35
Andrew Brock, Soham De, and Samuel L. Smith. Characterizing signal propagation close the per- formance gap in unnormalized resnets. In International Conference on Learning Representations (ICLR), 2021a. 26
Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. image recognition without normalization. ArXiv, abs/2102.061177 101.0677,,, 1b. 26
large-scale
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020. 33
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High-quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. ISSN 1939-3539. 7, 21
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze-excitation networks and beyond. In International Conference Computer Vision Work- shop (ICCVW), pp. 1971-1980, 2019. 6, 33
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020. 34
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference Computer Vision (ICCV), 2021. 27
10

---

Published as a conference paper at ICLR 2024
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun- jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2, 34
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu. Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. https: / gi thub. com/ cotp/openmmmiie mww.eenmmmaa////mmmmmm http//mmmeeettiiin
2019. 7, 22
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to-fine fine method for vision transformer. In AAAI Conference Artificial Intelligence (AAAT), 2023. 2, 35
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu.
Bridging mobilenet and transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 20, 21
Xu Cheng, Chuntung Chu, Yi Zheng, Jie Ren, and Quanshi Zhang. A game-theoretic taxonomy of visual concepts in dnns. arXiv preprint arXiv:2 10938, 2021. 2, 4,
Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael Black. Monocular expressive body regression through body-driven attention. In European Conference on Computer Vision (ECCV), pp. 20 40-40, 2020. 9, 22, 32
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Ad- vances in Neural Information Processing Systems (NeurIPS), 2021. 25
MMHuman3D Contributors. Openmmlab 3d human parametric model toolbox and benchmark.
https://githui: htwphhu...oommeeom www.:pw.ccoooccccccc.....e..e mm1 ab/ /mmmuman3d,, 2021. 22
MMPose Contributors. Openmmlab pose estimation toolbox and benchmark.
ccom/opent mml abllab//ii huiiaaammmposee.. 2020a. 22
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. /
com open. popert-mmile wwmmmm/m/////.mmmm wtt.////mmmeeettttteeeeeee i inn, 2020b. 7, 22
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated
data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 702-703, 2020. 21
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaug- ment: Learning augmentation strategies from data. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113-123, 2019. 21
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. International Conference on Computer Vision (ICCV), pp. 764-773, 2017. 33
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems (NeurlPS), 34: 3965-3977, 2021. 1, 2, 35
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103. 10697, 2021. 2, 35
Huiqi Deng, Qihan Ren, Xu Chen, Hao Zhang, Jie Ren, and Quanshi Zhang. Discovering and explaining the representation bottleneck of dnns. In International Conference on Learning Rep- resentations (ICLR), 2022. 2, 4, 4, 23, 24, 33

---

Published as a conference paper at ICLR 2024
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 2, 7, 20, 21, 33
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.4480,, 2018. 21, 33
Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlpnet: Hi- erarchical vision mlp with re- e--arameterrzed locality. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 578-587 2022a. 2
Xiaohan Ding, X. Zhang, Yi Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in enns. In Conference Computer Vision and Pattern Recognition (CVPR), 2022b. 2, 4, 24, 33
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer-
ence on Learning Representations (ICLR), 2021. 1, 2, 6, 26, 33, 34, 35
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weight linear units for neural network function approximation in reinforcement learning. Neural Nenworks, 107:3-11, 2018 5, 25
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In International Conference on Com- puter Vision (ICCV), pp. 6824-6835, 2021. 2
Zuen-Hua Feng, Patrik Huber, Josef Kittler, Peter Hancock, Xiao-Jun Wu, Qijun Zhao, Paul Koppen, and Matthias Rätsch. Evaluation of dense 3d reconstruction from 2d face images in the wild. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 780-786. IEEE, 2018. 9, 22
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z. Li. Simvp: Simpler yet better video prediction. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3170-3180, June 2022. 9, 23
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations (ICLR), 2019. 3, 24
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 4, 24
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In International Conference Computer Vision (ICCV), pp. 12259-12269. 2021. 24
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 27
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 35
Meng-Hau Cao Guo, Cheng Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual atten- tion network. Computational Visual Media (CVMJ), pp. 733-752, 2023. 21, 26, 33, 34
12

---

Published as a conference paper at ICLR 2024
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems (NeurIPS). 34:15908-15999 2021a. 1
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. De- mystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXiv:2106.04266 2021b. 3,33
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. 1, 20, 21, 25, 33, 34
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In International Conference on Computer Vision (ICCV), 2017. 5, 7,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In Conference on Computer Vision and Pattern Recogni- lion (CVPR), pp. 9729-9738, 2020. 35
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022. 2, 35
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian сгror linear units. arXiv preprint arXiv:1606.0844 2016. 6, 25
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 19000-19015, 2020. 4, 24
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 27
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1126-8135, 2020. 21
Qibin Hou, Cheng Lu, Mingg-Ming Mingg Cheng, and Jiashi Feng. Conv2former: A simple transformer- style convnet for visual recognition. ArXiv, 0tbs/2211.11992t 2022. 33
Andrew Howard, Mark Sandler, Grace Chu, Liang Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Inter- national Conference Computer Vision (ICCV), pp. 1314-1324, 2019. 33
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1704.04861, 2017, 33
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciratio networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7132-7141, 2018. 2, 6, 33, 34
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In International Conference on Machine Learning (ICML), 2022. 1,2, 25
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016. 21
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp- 448-456. PMLR, 2015a.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by ducing internal covariate shift, In Advances in Neural Information Processing Systems (NeuriPS), 2015b. 26
13

---

Published as a conference paper at ICLR 2024
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In Advances in Neural Information Processing Systems (NeurIPS), 20. 1, 2, 34
Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. In Advances in Neural Information Processing Systems (NeurIPS). 2021b. 2, 34
Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4401-4410, 2019. 22
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2014. 9, 22
Alexandre Kirchmeyer and Jia Deng. Convolutional networks with oriented 1d kernels. In Interna- tional Conference on Computer Vision (ICCV), 2023. 33
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6392-6401, 2019. 1, 5, 2,
Xiangtao Kong, Xina Liu, Jinjin Gu, Y. Qiao, and Chao Dong. Reflash dropout in image super- resolution. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5992-6002, 2022. 6
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60:84 90, 2012. 21, 33
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(1))62278--22222 2 1998. 33
Kunchang Li, Yali Wang, Junhao Zhang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and
Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. In Interna- tional Conference on Learning Representations (ICLR), 2022a. 1, 2, 5, 6, 9, 22, 35
Siyuan Li, Zicheng Liu, Zedong Wang, Di Wu, Zihan Liu, and Stan Z. Li. Boosting discriminative visual representation learning with scenario-agnostic mixup. ArXiv, abs/2111.1544 2021. 21
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. Openmixup: Open mixup toolbox http://hhhw.../tttteteaeeeeeeeee
and benchmark for visual representation learning. openmi xup, 2022b. 20
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, and Stan.Z.Li. Architecture-agnosie masked image mod- eling- from vit back to enn. In International Conference on Machine Learning (ICML), 2023a. 2, 4, 25
Siyuan Li, Luyuan Zhang, Zedong Wang, Di Wu, Lirong Wu, Zicheng Liu, Jun Xia, Cheng Tan, Yang Liu, Baigui Sun, and Stan Z. Li. Masked modeling for self-supervised representation learn- ing on vision and beyond. ArXiv, abs/2401 .00897, 2023b. 35
Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, S. Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Infor-
mation Processing Systems (NeurIPS), 2022c. 2, 6, 35
Zicheng Li, Siyuan Liu, Zelin Zang, Di Wu, Zhiyuan Chen, and Stan Z. Li. Genurl: A general framework for unsupervised representation learning. IEEE Transactions on Neural Networks and Learning Systems, 2023c. 35
Minghao Lin, Mengzhao Chen, Yu xin Zhang, Ke Li, Yunhang Shen, Chunhua Shen, and Rongrong Ji. Super vision transformer. ArXiv, abs/2205.1 1397, 2022. 3, 34
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014. 7, 21, 22, 23
14

---

Published as a conference paper at ICLR 2024
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 936-944, 2017a. 5
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In International Conference on Computer Vision (ICCV), 2017b. 7, 21
S. Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Constantin Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. In International Conference on Learning Representations (ICLR), 2023. 3,
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Con- ference on Computer Vision (ICCV), 2021. 1, 3, 6, 7, 20, 21, 22, 25, 34, 35
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3192-3201, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976-11986, 2022b. 1, 2, 3, 5, 6, 7, 20, 21, 25, 26, 34
Zicheng Liu, Siyuan Li, Ge Wang, Cheng Tan, Lirong Wu, and Stan Z. Li. Decoupled mixup for data-efficient learning. ArXiv, abs/2203. 10761, 2022c. 21
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision (ECCV), 2022d. 21
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 03998,, 2016. 7,21
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations (ICLR), 2019. 7, 2121,22 22
Wenjie Luo. Yujia Li. Raquel Urtasun, and Richard S. Zemel. Understanding the effecttve receptive field in deep convolutional neural networks. ArXiv, abs/1 701.04128, 2016. 33
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In European Conference on Computer Vision (ECCV), pp- 116-131, 2018. 33
Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purposs and mobile- friendly vision transformer. In International Conference on Learning Representations (ICLR), 2022. 2, 20, 35
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Hsuan Yang. Intriguing properties of vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 24
Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In Advances in Neural Information Processing Systems (NeurIPS), 2022a. 5, 6, 25, 34
Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI Conference on Artificial Intelligence (AAAI), 2022b. 2, 6, 35
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations (ICLR), 2022. 3, 5, 24
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML), 2018. 34
15

---

Published as a conference paper at ICLR 2024
Francesco Pinto, Philip HS Torr, and Puneet K Dokania. An impartial take to the transformer robustness contest. European Conference on Computer Vision (ECCV), 2022. 2,
Boris Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging. Siam Journal on Control and Optimization, 30:838 855, 1992. 21
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Conference Computer Vision and Pattern Recognition (CVPR), pp- 10425-10433, 2020.
1, 6, 33, 34
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems (NeurIPS), 34:12111-1221 2021. 2
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3, 5, 25, 33, 34
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 39:1137-1144 1149, 2015. 1
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018. 3, 20, 33
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 618-626, 2017. 9, 24
Noam M. Shazeer. Glu variants improve transformer. ArXiv, abs/2002.05202, 2020. 25
Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 5, 25, 35
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014. 24,
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. In International Conference Machine Learning (ICML), 2015. 9, 23
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. 21
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016. 21,
Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang. Zicheng Liu, Lirong Wu, and Stan Li. Openstl: A comprehensive benchmark of spatio-temporal predictive learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 23
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning (ICML), pp. 6115-6114. PMLR, 2019. 6, 33, 34
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning (ICML), 2021. 34
16

---

Published as a conference paper at ICLR 2024
Iyy O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mip architecture for vision. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2, 6, 35
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning (ICML), pp. 10347-10357, 2021a. 2, 20, 24, 27, 34
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), 2021b. 35
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Piotr Bojanowski, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Herv' e J'eggu Augmenting convolutional networks with attention- based aggregation. arXiv preprint arXiv:2112. 13692, 2021c. 6
Hugo Touvron, Matthieu Cord, and Herv'e J J'egou. Deit ii:: Revenge of the vit. In European Conference on Computer Vision (ECCV), 2022. 2, 27,34
Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychol- ogy, 12(1):97-136, 1980. 4, 24
Asher Trockman and J. Zico Kolter. Patches are all you need? ArXiv, abs/2201.09792, 2022. 3, 34
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L. Griffiths. Are convolutional neural net- works or transformers more like human vision? ArXiv, abs/2105.07197, 2021. 3, 4, 24
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017. 1, 3,
Guangting Wang. Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift oper- ation meets vision transformer: An extremely simple alternative attention mechanism. In AAAI Conference on Artificial Intelligence (AAAI), 2022a. 3, 34
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X. Yu. Orthogonal convolutional neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11502-11512,
2020. 6
Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory practice. International Con- ference on Learning Representations (ICLR), 2022b.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2019. 20
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. In Advances in Neural Information Processing Systems (NeurIPS), 2021a. 2021a, 1, 2,, 35
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision (ICCV), pp. 548-5588 2021b. 2,
9, 22, 34
Wenhai Wang. Enze Xie, Xiang Li, Deng- Ping-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media (CVMJ), 2022c. 6
17

---

Published as a conference paper at ICLR 2024
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7794-7803, 2018. 33
Ross Wightman, Hugo Touvron, and Herv Jgou. Resnet strikes back: An improved training procedure in timm. https: htt///gtttus com/ huggingface//pnneooooooooooo 2021. 2, 20, 27, 28, 29, 34
wwaccch-immigee image-models, models,
Sanghyun Woo, Jongchan Park, loon-Young Lee, and In-So Kweon. Cham: Convolutional block attention module. In European Conference on Computer Vision (ECCV), 2018. 6, 33
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In-So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Conference Computer Vision and Pattern Recognition (CVPR), 2023. 35
Fang Wu, Siyuan Li, Lirong Wu, Stan Z. Li, Dragomir R. Radev, and Qian Zhang. Discovering the representation bottleneck of graph neural networks from multi-order interactions. ArXiv, abs/2205. 07266,, 2022a.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. International Conference Computer Vision (ICCV), 2021a. 2, 35
Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. In International Conference on Machine Learning (ICML), 2022b. 24
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improv- ing relative position encoding for vision transformer. In International Conference on Computer Vision (ICCV), pp. 10033-10041, 2021b. 3,35
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference computer vision (ECCV), 2022c. 3,34
Yuxin Wu and Justin Johnson. Rethinking "batch" in batchnorm. ArXiv, abs/2105.07576, 2021. 26,
27
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In European Conference Computer Vision (ECCV), 2018a. 22
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV). Springer, 2018b. 5, 7.22
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross B. Girshick. Early convolutions help transformers see better. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2,
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1492-1500, 2017. 33, 34
Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. In Ad- vances in Neural Information Processing Systems (NeurIPS), 2022. 3, 33
Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10799-10808, 2022. 24
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT 76 minutes. International Conference on Learning Representations (ICLR), 2020. 28
18

---

Published as a conference paper at ICLR 2024
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10819-10829, 2022. 2,3, 7, 22, 23, 26, 34
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In International Conference on Computer Vision (ICCV), 2021a. 2,
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis
Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. International Conference on Computer Vision (ICCV), pp. 538-547, 2021b. 21, 25
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Interna- tional Conference on Computer Vision (ICCV), pp. 6023-6032, 2019. 21
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016. 33
Zelin Zang, Siyuan Li, Di Wu, Ge Wang, Lei Shang, Baigui Sun, Hao Li, and Stan Li. Dime: Deep local-flatness manifold embedding. In European Conference Computer Vision (ECCV), 2022. 35
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, al. Resnest: Split-attention networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2736-2746, 2022a. 33
Hao Zhang, Sen Li, Yinchao Ma. Mingjie Li, Yichen Xie, and Quanshi Zhang. Interpreting and boosting dropout from a game-theoretic view. arXiv preprint arXiv:2009. 1729, 2020. 2, 3, 23
Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Edgeformer: Improving light-weight convnets by
learning from vision transformers. In European Conference Computer Vision (ECCV), 2022b. 2, 5, 20, 21, 27
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018. 21
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmen- tation. In AAAI Conference on Artificial Intelligence (AAAI), pp. 13001-13008, 2020. 21
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision (IJV), 127:302-32 321, 2018. 7,
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng, and José Manuel Álvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning (ICML), 2022. 3,
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.00 111221111173....11 2021. 24
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning Representaatonss (ICLR), 2021. 2, 34
Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In International Conference on Computer Vision (ICCV), pp. 813-822, 2019. 9, 22
19

---

Published as a conference paper at ICLR 2024
A
IMPLEMENTATION DETAILS
A.1 ARCHITECTURE DETAILS
G-y --W
The detailed architecture specifications of Mo- gaNet are shown in Table A1 and Fig. 2, where input image of 224 resolutions is assumed for all architectures. We rescale the groups of embedding dimensions the number of Moga Blocks for each stage corresponding to different models of vary- ing magnitudes: i) MogaNet-X-Tiny and MogaNet- Tiny with emhedding dimensions of {32, 64, 96, 192] and {32, 64, 128, 256} ex- hibit competitive parameter numbers and com- putational overload recently proposed light- weight architectures (Mehta & Rastegari, 2022; Chen et al., 2022: Zhang et al., 2022b); ii) MogaNet-Small adopts embedding dimensions of (64, 128, 320, 512] in comparison other prevailing small-scale architectures (Liu al., 2021; 2022b); iii) MogaNet-Bass with embedding dimensions of {64, 160, 320, 512} in comparison to medium size architectures; iv) MogaNet-Large with embedding dimensions of (64, ,60,,20,,,,00000 640) is designed for large-scale computer vision tasks. MogaNet-X- X-Lagge with embedding dimensions of (96, 192, 480, 960) is a scaling-up version (around 200M parameters) for large-scale tasks. The FLOPs are measured for image classification on ImageNet (Deng et al., 2009) at resolution 224?, where a global average pooling (GAP) layer applied to the output feature map of the last stage, followed by linear classifier.
Stem
Norm
SMixer
Norm
CMixer
Figure A1: Modern ConvNet architecture. It has 4 stages in hierarchical, and i-th stage contains an embedding stem and N, blocks of SMixer(-) and CMixerr(-) with PreNorm (Wang et al., 2019) and identical connection (He al., 2016). The features within the i-th stage are in the same shape, except that CMixer(.) will increase the di- mension to rC, with an expand ratio r as an in- verted bottleneck (Sandler et al., 2018).
Stage
Layer Settings
MogaNet
Size
XTiny Tiny Small BaseeLarge XLarge
Convax3, stride 2, C/2
Stem
ConVax3, stride 2, C
S1
HxW
Embed. Dim.
32
32
64
64
64
96
1x
# Moga Block
3
4
4
6
MLP Ratio
Stem
ConVax3 stride 2
S2
H*W
Embed, Dim.
64
128
160
160
192
# Moga Block
3
6
6
6
MLP Ratio
8
Stem
Convax3, stride
S3
HxW 16x16
Embed. Dim.
96
128
320
320
320
480
# Moga Block
10
12
12
22
44
44
MLP Ratio
Stem
Convax3, stride 2
S4
#xW 32x32
Embed. Dim.
192
256
512
512
640
960
# Moga Block
2
2
2
3
4
ML.P Ralio
Classifier
Global Average Pooling, Linear
Parameters (M)
2.97
5.20 25.3 43.8 82.5
180.8
FLOPs (G)
0.80
1.10 4.97 9.93 15.9
34.5
Table A1: Architecture configurations of MogaNet variants.
A.2 EXPERIMENTAL SETTINGS FOR IMAGENET
We conduct image classification experiments ImageNet (Deng al., 2009) datasets. All periments are implemented on OpenMi xup (Li et al., 2022b) and timm (Wightman et al., 2021) codebases running 8 NVIDIA A100 GPUs. View more results in Appendix D.I.
ImageNet-1K. We perform regular ImageNet- IK training mostly following the training settings of DeiT' (Touvron et al., 2021a) and RSB A2 (Wightman et al., 2021) in Table A2, which are widely
20

---

Published as a conference paper at ICLR 2024
Configuration
DerT
RSB
MogaNet
Configuration
IN-2IK
IT
A2
XT
B
XL
Input resolation
224
224
224"
Input resolution
224
384
Epochs
300
300
300
Epochs
90
30
Batch size
1024
2048
1024
Batch size
1024
512
Optimizer
AdamW
LAMB
AdamW
Optimizer
AdamW
AdamW
AdamW [f,, Pa]
0.9. 0.980
0.9. 0.999
AdamW [1), (2)
0.9, 09.0.999
0.9,0.990 0.999
Learning rate
0.001
0,005
0.001
learning rate
10" 5
Learning decuy
Cosine
Cosine
Cosine
Learning rate decay
Cosine
Cosine 0.05
Weight decay
0.05
0.02
0.03 0.04
0.05 0.05 0.05 0.05
Weight decay
0.05
Warmup epochs
Warmup epochs
5
Label smoothing
0.1
0.1
0.1
Label smoothing
0.2
0.1
0.1 0.2
Stochastic Depth
0.05
0.1
0.1
0.2
0.3
0.4
Stochastic Depth
0.1
0.1 0.1
0.4
0.6 0.7 0.8
Rand Augment
9/0.5
7/0.5
7/0.5
7/0.5
9/0.5
9/0.5
9/0.5
9/0.5
Rand Augment
9/0.5
9/0.5
Repeated Augment
Repeated Augment
X
Mixup a
0.8
0.1
0.1
0.1
0.8
0.8
0.8
0.8
Mixup a
0.8
CutMix
1.0
CutMix
1.0
Erasing prob.
0.25
X
0.25
Erasing prob.
0.25
0.25
Colorliter
X
X
0.4
0.4
0.4
0.4
ColorJitter
0.4
0.4
Gradient Clipping
Gradient Clipping EMA decay Test crop ratio
EMA decay
X
X
/
X
Test crop ratio
0.875
0.95
0.90
0.90
Table A2: Hyper-parameters for ImageNet-IK train- Table A3:
Detailed training recipe for
ing of DeiT, RSB A2, and MogaNet. We use a similar ImageNet-21K pre-training (IN-21K PT) and setting as RSB for XL and T versions of MogaNet and ImageNet-IK fine-tuning (IN-1K FT) in high DeiT for the other versions.
resolutions for MogaNet.
adopted for Transformer and ConvNet architectures. For all models, the default input image res- olution is 224 for training from scratch. We adopt 256² resolutions for lightweight experiments according MobileViT (Mchta & Rastegari, 2022). Taking training settings for the model with 25M or more parameters as the default, we train all MogaNet models for 300 epochs by Adam (Loshchilov & Hutter, 2019) optimizer using batch size of 1024, basic learning rate of x 10-3 a weight decay of 0.05, and a Cosine learning rate scheduler (Loshchilov & Hutter, 2016) with epochs of linear warmup (Devlin et al., 2018). As for augmentation and regularization tech- niques, we adopt most of the data augmentation and regularization strategies applied in DeiT train- ing settings, including Random Resized Crop (RRC) and Horizontal flip (Szegedy al., 2015), RandAugment (Cubuk et al., 2020). Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), random erasing (Zhong et al., 2020), ColorJitter (He al., 2016), stochastic depth (Huang et al., 2016), and label smoothing (Szegedy et al., 2016). Similar to ConvNeXt (Liu et al., 2022b), we do not apply Repeated augmentation (Hoffer al., 2020) and gradient clipping, which are designed for Transformers but do not enhance the performances of ConvNets while using Exponential Moving Average (EMA) (Polyak& Juditsky, 1992) with the decay rate of 0.9999 by default. We also remove additional augmentation strategies (Cubuk al., 2019; Liu et al., 2022d; Li et al., 2021; Liu et al., 2022c), e.g., PCA lighting (Krizhevsky et al., 2012) and AutoAAugentt (Cubuk et al., 2019). Since lightweight architectures (3~10M parameters) tend to get under-fil fittee with strong augmentations and regularization, we adjust the training configurations for MogaNet-XTTTT following (Mehta & Rastegari, 2022; Chen al., 2022; Zhang et al., 2022b), including employing the weight decay of 0.03 and 0.04, Mixup with of 0.1, and RandAugment of 7/0.5 for MogaNet-XTTTT Since EMA is proposed stabilize the training process of large models, we also remove for MogaNet-XT/T as fair comparison. An increasing degree of stochastic depth path augmentation is employed for larger models. In evaluation, the top-1 accuracy using a single crop with test crop ratio of 0.9 is reported as (Yuan et al., b021b; Yu et al., 2022; Guo et al., 2023).
Following ConvNeXt, we further provide the training recipe for ImageNet- 21K (Deng al., 2009) pre-training and ImageNet- fine-tuning with high resolutions in Table A3. EMA is removed in pre- pre-training, while CutMix and Mixup are removed for fine-tuning.
ImageNee-22IK.
A.3 ОВJЕCT DETECTION AND SEGMENTATION COCO
Following Swin (Liu et al., 2021) and Poomerrmer (Yu et al., 2022), we evaluate objection detection and instance segmentation tasks on COCO (Lin et al., 2014) benchmark, which include 118K train- ing images (train2017) and 5K validation images (val2017). We adopt RetinaNet (Lin et al., 2017b), Mask R-CNN (He et al., 2017). and Cascade Mask R-CNN (Cai & Vasconcelos, 2019) as the stan- dard detectors and use ImageNet-IN IK pre-trained weights the initialization of the backbones. As
21

---

Published as a conference paper at ICLR 2024
for RetinaNet and Mask R-CNN, we employ AdamW (Loshchilov & Hutter, 2019) optimizer for training 1x scheduler (12 epochs) with a basic learning rate of x 10 and a batch size of 16. As for Cascade Mask R-CNN, the 3x training scheduler and multi-scale training resolutions (MS) are adopted. The pre-trained weights on ImageNet- K and ImageNet-21K are used accordingly to initialize backbones. The shorter side of training images is resized to 800 pixels, and the longer side is resized to not more than 1333 pixels. We calculate the FLOPs of compared models at 800 x 1280 resolutions. Experiments of COCO detection are implemented on MMDetect ion (Chen et al., 2019) codebase and run on 8 NVIDIA A100 GPUs. View detailed results in Appendix D.2.
A.4 SEMANTIC SEGMENTATION ON ADE20K
We evaluate semantic segmentation on ADE20K (Zhou et al., 2018) benchmark, which contains 20K training images and 2K validation images, covering 150 fine ain-ggrined semantic categories. We first adopt Semantic FPN (Kirillov et al., 2019) following PoolFormer (Yu et al., 2022) and Uni- former (Li et al., 2022a), which train models for 80K iterations by AdamW (Loshchilov & Hutter, 2019) optimizer with a basic learning rate of 2 x 10 4, a batch size of 16, and a poly learning rate scheduler. Then, we utilize UperNet (Xiao et al., 2018b) following Swin (Liu et al., 2021), which employs AdamW optimizer using a basic learning rate of 6 x 10 a weight decay of 0.01, poly scheduler with linear warmup of 1,500 iterations. We use ImageNet- 1K and ImageNet-21K pre-trained weights to initialize the backbones accordingly. The training images are resized to 5122 resolutions, and the shorter side of testing images resized to 512 pixels. We calculate the FLOPs of models at 800 x 2048 resolutions. Experiments of ADE20K segmentation are implemented on MMSegment a ion (Contributorss, 2020b) codebase and run on 8 NVIDIA A100 GPUs. View full comparison results in Appendix D.3.
A.5 2D HUMAN POSE ESTIMATION ON COCO
We evaluate 2D human keypoints estimation tasks on COCO (Lin et al., 2014) benchmark based on Top-Down SimpleBaseline (Xiao et al., 2018a) (adding a Top-Down estimation head after the back- bone) following PVT (Wang et al., 2021b) and UniFormer (Li et al., 2022a). We fine-tune all models for 210 epochs with Adam optimizer (Kingma & Ba, 2014) using a basic learning rate selected in {1 X 10 3 5 x 10-4), }). multi-step learning rate scheduler decay 170 and 200 epochs. ImageNet- IK pre-traine pre-trained weights are used as the initialization of the backbones. The training and testing images resized to 256 192 or 384 288 resolutions, and the FLOPs of models are calculated at both resolutions. COCO pose estimation experiments are implemented on MMPose (Contributors, 2020a) codebase and run on 8 NVIDIA A 100 GPUs. View full experiment results in Appendix D.4.
A.6 3D HUMAN POSE ESTIMATION
We evaluate MogaNet and popular architectures with 3D human pose estimation tasks with single monocular image based on ExPose (Choutas et al., 2020). We first benchmark widely-used Con- vNets with the 3D face mesh surface estimation task based on ExPose. All models are trained for 100 epochs on Flickr-Faces, Faces-HQ Dataset (FFHQ) (Karras et al., 2019) and tested on Stirling/ESRC 3D dataset (Feng et al., 2018), which consists of facial RGB images with ground-truth 3D face scans. 3D Root Mean Square Error (3DRMSE) measures errors between the predicted and ground- truth face scans. Following ExPose, the Adam optimizer employed with batch size of 256. basic learning rate selected in (2 x 10-1, 1 x 10-1), a multi-step learning rate scheduler decay at 60 and 100 epochs. ImageNet-IK IK pre-trained weights are adopted as the backbone initialization. The training and testing images are resized to 256 x 256 resolutions. Then, we evaluate ConvNets with the hand 3D) pose estimation tasks. FreiHAND dataset (Zimmermann et al., 2019), which contains multi-view RGB hand images, 3D MANO hand pose, and shape annotations, is adopted for training and testing. Mean Per-Joint Position Error (PA-MPJPE) is used to evaluate 3D skeletons. Notice that a "PA" prefix denotes that the metric measures error after solving rotation, scaling, and transla- tion transforms using Procrustes Aligument. Refer to ExPose for more implementation details. All models use the same training settings the 3D face task, and the training and testing resolutions are 224 x 224. Experiments of 3D pose estimation are implemented on MMHuman3D (Contributors, 2021) codebase and run 4 NVIDIA A100 GPUs. View full results in Appendix D.5.
22

---

Published as a conference paper at ICLR 2024
A.7 VIDEO PREDICTION ON MOVING MNIST
We evaluate various Metaformer architectures (Yu et al., 2022) and MogaNet with video prediction tasks on Moving MNIST (MMNIST) (Lin al., 2014) based on SimVP (Gao al., 2022). Notice that the hidden translator of SimVP is a 2D network module to learn spatio-temporal representation. which any 2D architecture can replace. Therefore, we can benchmark various architectures based on the VP framework. In MMNIST (Srivastava et al., 2015), each video is randomly generated with 20 frames containing two digits in 64 x 64 resolutions, and the model takes 10 frames as the input to predict the next 10 frames. Video predictions are evaluated by Mean Square Error (MSE), Mean Absolute Error (MAE), and Structural Similarity Index (SSIM), All models are trained on MMNIST from scratch for 200 or 2000 epochs with Adam optimizer, batch size of 16, a OneCycle
learning rate scheduler, an initial learning rate selected in {1 10 10-2, 2, x 10 10-3 111x x 10 10-3 5 x 10 1)= Experiments of video prediction are implemented OpenSTL' codebase (Tan et al., 2023) and run on a single NVIDIA Tesla V100 GPU. View full benchmark results in Appendix D.6.
B EMPIRICAL EXPERIMENT RESULTS
B. REPRESENTATION BOTTLENECK OF DNNs FROM THE VIEW OF MULTI-ORDER
INTERACTION
Multi-order game-theoretic interaction. In Sec. 3, we interpret the learned representation of DNNs through the lens of multi-order game-theoretic interaction (Zhang et al., 2020; Deng et al., 2022), which disentangles vatir-variable communication effects in a DNN into diverse game- theoretic components of different interaction orders. The order here denotes the scale of context involved in the whole computation process of game-theoretic interaction.
For computer vision, the m-th order interaction 7(m) (i..j) measures the average game-theoretic in- teraction effects between image patches and on all m image patch contexts. Take face recognition as an example, we can consider patches i and as two eyes on this face. Besides, we regard other m visible image patches included on the face. The interaction effect and contribution between the eye's patches and toward the task depend on such m visible patches as the context, which is measured as the aforementioned 7 7(m)) (i,j). If I 7(mm) (i, > 0 patches and j show positive effect under m context. Accordingly, if 7 (i. < O, we considerí and j have a negative effect under m context. More importantly, interactions of low-order mainly reflect widely-shared local texture and common visual concepts. The middle-order interactions are primarily responsible for encoding discriminative high-level representations. However, the high-order ones inclined to let DNNs memorize the pattern of rare outliers and large-scale shape with intensive global inter- actions, which can presumably over-fit our deep models (Deng al., 2022; Cheng et al., 2021), )). Consequently, the occurrence of excessively low- or high-order game-theoretic interaction in a deep architecture may therefore be undesirable.
Formally, given an input image with a set of n patches N = [l....,n} (e.g., an image with n pixels in total), the multi-order interaction (i,.j) can be calculated as:
((m) (i.j) (s - f(Su()) + f(S). S(S) indicates the score of output with patches in N S kept unchanged but replaced with the baseline value (Ancona et al., 2019a), For example, a low-order interaction (e.g., m = 0.05n) means the relatively simple collaboration between variables i,j under a small range of context, while high-order interaction (e.g., m = 0.95n) corresponds to the complex collaboration under a large range of context. Then, we can measure the overall interaction complexity of deep neural networks (DNNs) by the relative interaction strength j(m) of the encoded m-th order interaction:
(11)
where
Exen I (m)) (i,]
j(m)
(12)
ES Even Br, [ (mm (i.
where S2 is the set of all samples and 0 < m 2 n 2. Note that (mm) is the average interaction strength over all possible patch pairs of the input samples and indicates the distribution (area under
hhttps://git thub. com/ http://hhnttnnnnnnmm 9077 9000//900////// OpenSTI.
23

---

Published as a conference paper at ICLR 2024
Interaction Strength Order
Interaction Strength Order
Interaction Strength of Order
ConvNeXt (Tay), 29M
PocalNet (Ting), 29M
DelT (Smalh) 22M
ResNed0, 25M
VAN (B2), 27M
Swin (Tiny). 2RM
Interaction Steength
SLaK (Tiny), XM
Interaction Strength
MagaNNt (Small), 25M
Interaction Steength
MagaNat (Smal), 25M
MogaNet (Small), 2534
0
2
2
0.1 3 Order
5.5
0.7 0.9
1.0
0.1
3.3
0.5
7.7 0.
1.0
0
0.1
0.3
0.5
0.7
0.
0.9
1.0
Order
Order/
(a)
(b)
(c)
Figure A2: Distributions of the interaction strength ((m) for (a) ConvNets with different con- volution kernel sizes, (b) ConvNets with gating aggregations, and (c) Transformers on ImageNet- IK with 224 resolutions. Middle-order strengths mean the middle-complex interaction, where a medium number of patches (e.g., 0.2~0.8n) participate.
curve sums up to one) of the order of interactions of DNNs. In Fig. A2, we calculate the interaction strength f(m) with Eq. 12 for the models trained on ImageNet- IK using the official implementation provided by (Deng et al., 2022). Specially, we use the image of 224 x 224 resolution the input and calculate J(m) on 14 x 14 grids, i.e., n 14 X 14. And we set the model output as f(xs) P(g=y les)
log 1-P(0-y)=s) given the masked sample rs, where y denotes the ground -tunh label and P(ğ y|ar) denotes the probability of classifying the masked sample I5 to the true category. Fig. A2a and Fig, A2b compare existing ConvNets with large kernels or gating designs and demonstrate that MogaNet can model middle-order interactions better to learn more informative representations.
Relationship of explaining works of ViTs. Since the thriving of ViT's in a wide range of computer vision tasks, recent studies mainly investigate the why ViTs work from two directions: (a) Evalua- tion of robustness against noises finds that self-attentions (Naseer et al., 2021; Park & Kim, 2022; Zhou et al., 2021: Li al., 2023a) or gating mechanisms (Zhou al., 2022) in ViTs are more robust than classical convolutional operations (Simonyan & Zisserman, 2014; Szegedy et al., 2016). For example, ViTs can still recognize the target object with large occlusion ratios (e.g., only 10~20% visible patches) or corruption noises. This phenomenon might stem from the inherent redundancy of images and the competition property of self-attention mechanisms (Wang et al., 2021a; Wu al., 2022b). Several recently proposed works (Yin et al., 2022; Graham et al., 2021) show that ViTs can work with some essential tokens (e.g., 5~-00% that are selected according to the complexity of input images by dynamic sampling strategies, which also utilize the feature selection properties of self- attentions. From the perspective of multi-order interactions, convolutions with local inductive bias (using small kernel sizes) prefer low-order interactions, while self- aff-aatentions without any inductive bias tend to learn low-order and high-order interactions. (b) Evaluation of out-of-distributie sam- ples reveals that both self-attention mechanisms and depth-wise convolution (DWConv) with large kernel designs share similar shape-bise bias tendency as human vision (Tuli et al., 2021; Geirhos et al., 2021: Ding et al., 2022b), while canonical ConvNets (using convolutions with small kernel sizes) exhibit strong bias on local texture (Geirhos et al., 2019; Hermann et al., 2020). Current works (Ding al., 2022b) attribute shape or texture- bias tendency to the receptive field of self-attentio or con- volution operations, i.e., an operation with the larger receptive field or more long-range dependency is more likely to be shape-bias. However, there are still gaps between shape-bias operations and human vision. Human brains (Treisman & Gelade, 1980; Deng et al., 2022) attain visual patterns complexxt interactions to recognize objects, while self-attention or
and clues and conduct convolution operation can only encode global local features to conduct high or cowplexxtt in- teractions. As the existing design of DNNs only stacks regionality perception or context aggregation operations in cascaded way, it is inevitable encounter the representation bottleneck.
B.2 VISUAALIZAATONN OF CAM
We further visualize more examples of Grad-CAM (Selvaraju et al., 2017) activation maps of MogaNet-S in comparison to Transformers, including DeïT-S (Touvron et al., 2021a), T2T-VIT-
httpps://gi wwwpb.coooobbbaaaaaeoaaaoaaaa htw.llaaaaadi00/oaaa
24

---

Published as a conference paper at ICLR 2024
S (Yuan al., 2021b), T (Chu et al., 2021), and Swin (Liu et al., 2021), and ConvNets, including ResNet-50 (He et al., 2016) and ConvNeXt-T (Liu et al., 2022b), on ImageNet- IK in Fig. A4. Due the self-attention mechanism, the pure Transformers architectures (DeiT-S and T2T-ViT-S) show more refined activation maps than ConyNets, but they also activate some irrele- vant parts. Combined with the design of local windows, local attention architectures (Twins-S and Swin-T) can locate the full semantic objects. Results of previous ConvNets can roughly localize the semantic target but might contain some background regions. The activation parts of our proposed MogaNet-S are more similar to local attention architectures than previous ConvNets, which are more gathered on the semantic objects.
C MORE ABLATION AND ANALYSIS RESULTS
In addition to Sec. 5.3, we further conduct more ablation and analysis of our proposed MogaNet on ImageNet-IK, We adopt the same experimental settings Sec. 1.
C.I ABLATION OF ACTIVATION FUNCTIONS
We conduct the ablation of activation functions used in the proposed multi-order gated aggregation module on ImageNet- IK. Table A4 shows that using SiL.U (Elfwing et al., 2018) activation for both branches achieves the best performance. Similar results were also found in Transformers, e.g., GLU variants with SiLU or GELU (Hendrycks Gimpel, 2016) yield better performances than using Sigmoid Tanh activation functions (Shazeer, 2020; Hua et al., 2022). We assume that SiLU is the most suitable activation because owns both the property of Sigmoid (gating effects) and GELU (training friendly), which is defined Sigmoid(z))
Top-1
Context branch
Modules
Top-1
Params.
FLOPs
Acc (%)
None
GELU
SiLU
Acc (%)
(M)
(G)
None
76.3
76.7
76.7
Baseline (+Gating branch)
77.2
5.09
1.070
Gating
Sigmoid
76.8
77.0
76.9
DW7x7
77.4
5.14
1.094
branch
GELU
76.7
76.8
77.0
D55x5.t=1 + DWTx7.0=3
77.5
5.15
1.112
SiLU
76.9
77.1
77.2
DW5x5.d-1 + DW DW
77.5
5.17
1.185
Table A4: Ablation of various activation functions for the gat- ing and context branches in the proposed Moga(.) module, which
+ +Multi-order. Ci: Cm : Ch = 00:=1:::1 1:0:3
77.5
5.17
1.099
+ Multi-order, Ci: Cm Cy 00:11
77.6
5.17
1.103
+Multi-rddrr Ct: Cm Ch. 1:::9
77.7
5.17
1.104
Multi-order, C Cm : C₁ 1::34
77.8
5.17
1.102
SiLU achieves the best perfor- mance in two branches.
Table A5: Ablation of multi-order DWConv layers in the pro- posed Moga(-). The baseline adopts the MogaNet framework using the non-linear projection, DW and the SiLU gating branch as SMixer(-) and using the vanilla MLP as CMixer(-).
C.2 ABLATION OF MULTI-ORDER DWCONV LAYERS
In addition to Sec. 4.2 and Sec. 5.3, we also analyze the multi-order depth-wise convolution (DW- Conv) layers as the static regionality perception in the multi-order aggregation module Moga(-) on ImageNet-1K. As shown in Table A5, we analyze the channel configuration of three parallel dilated DWConv layers: DW 5x5.ddm DWsx5,d and DW7x7,a=3 with the channels of Ct, Cms Cs. we first compare the performance of serial DWConv layers (e.g., DW 5 +DW 777,mma3) and paral- lel DWCony layers. We find that the parallel design can achieve the same performance with fewer computational overloads because the DWConv kernel is equally applied adopt three DWConv layers, the proposed parallel design reduces Ct + C₁ and C + Omm times computations of DW 5W5 and DW 5x in comparison to the serial stack of these DWConv layers. Then, we empirically explore the optimal configuration of the three channels. We find that G : Cm : C₁ 1: 3: 4 yields the best performance, which well balances the small, medium, and large DWConv kernels to learn low, middle, and high-order contextual representations. We calcu- late and discuss the FLOPs of the proposed three DWConv layers in the next subsection to verify the efficiency. Similar conclusions are also found in relevant designs (Pan et al., 2022a: Si al., 2022: Rao et al., 2022), where global context aggregations take the majority (e.g., ~ & channels or context components). We also verify the parallel design with the optimal configuration based
all channels. When we
25

---

Published as a conference paper at ICLR 2024
MogaNet-S/B Therefore, conclude that proposed multi-order DWConv layers efficiently learn multi-order contextual information for the context branch of Moga(.).
C.3 FLOPS AND THROUGHPUTS OF MOGANET
FLOPs of Multi-order Gated Aggregation Module We divide the computation of the proposed multi-order gated aggregation module into two parts of convolution operations and calculate the FL.OPs for each part.
Conv1x1. The FLOPs of Ix1 convolution operation @gate Pcontext and Pout can be derived as:
FLOPs((ggtee 2HW C²
FLOPss (context) = 2HW
(13)
FLOPs(Com) = 2
Dipt convolution. We consider the depth-wise convolution (DW) with dilation ratio d. The DWConv is performed for the input X, where X #WCCi Therefore, the FLOPs for all DW in Moga module are:
FLOPssDW 5x5.a-1) 2HW K2x50
FLOPssDW 5x6,d=2) HW Cin K3x5:
(14)
CLOPs(DW7x7..E)) = HW HWC K2xT
Overall, the total FLOPs of our Moga module can be derived as follows:
11
(Mogss 2HW a...
K 6x5
K2 + 30in
(15)
471
HWC
60
Throughput of MogaNet We further analyze throughputs of MogaNet variants on ImageNet- IK. As shown in Fig. A3, MogaNet has similar throughputs as Swin Transformer while producing better per- formances than Swin and ConvNet. Since we add channel splitting and GAP opera- tions in MogaNet, the throughput of Con- vNeXt exceeds MogaNet to some extent.
Accuracy vs. Throughput
DeiT (VIT)
Top-1 1 Accuracy %
Swin
84.0
ConvNeXt
MogaNet (Ours)
80.0
C.4 ABLATION
78.0
OF NORMALIZATION LAYERS
For most ConvNets, BatchNorm (loffe & Szegedy, 2015b) (BN) is considered an es- sential component improve the conver- gence speed and prevent overfitting. How- ever, BN might cause some instability (Wu
76.0
250
1000 1250 1500 Throughput (Images/s)
500
750
Figure A3: Accuracy-thouughuut diagram models on ImageNet- K measured on an NVIDIA V100 GPU.
& Johnson, 2021) or harm the final performance of models (Brock et al., 202 Some recently proposed ConvNets (Liu et al., 2022b; Guo et al., 2023) replace BN by LayerNorm (Ba al., 2016) (LN), which has been widely used in Transformers (Dosovitskiy et al., 2021) and Metaformer ar- chitectures (Yu et al., 2022), achieving relatively good performances conduct an ablation of normalization (Norm) layers in MogaNet on ImageNet-IK, ble A6. As discussed in ConvNeXt (Liu et al., 2022b), the Norm layers used in each block (within) and after each stage (after) have different effects. Thus we study them separately. Table A6 shows that using BN in both places yields better performance than using LN (after) and BN (within), except MogaNet-T with 224 resolutions, while using LN in both places performs the worst. Consequently,
various scenarios. Here, we
shown in Ta-
26

---

Published as a conference paper at ICLR 2024
Input
DeiT-S
T2T-VIT-S
Twins-S
Swin-T
ResNet-50
ConvNeXt-T
MogaNet-S
Panda
Ladybug
Turtle
Blue Jay
Figure A4: Visualization of Grad-CAM activation maps of the models trained on ImageNet-
we use BN as the default Norm layers in our proposed MogaNet two reasons: (i) With pure con- volution operators, the rule of combining convolution operations with BN within each stage is still useful for modern ConvNets. (ii) Although using LN after each stage might help stabilize the train- ing process of Transformers and hybrid models and might sometimes bring good performance for ConvNets, adopting BN after each stage in pure convolution models still yields better performance. Moreover, replace BN with precise BN (Wu & Johnson, 2021) (pBN), which is optimal al- ternative normalization strategy to BN, We find slight performance improvements (around 0. %), further improve MogaNet with advanced BN. As discussed in ConvNeXt, EMA might severely hurt the performances of models with BN. This phenomenon might be caused by the unstable and inaccu- rate BN statistics estimated by EMA in the vanilla BN with large models, which will deteriorate when using another EMA of model parameters. We solve this dilemma by exponentially increasing the EMA decay from 0.9 to 0.9999 during training momentum-based contrastive learning meth- ods (Caron et al., 2021; Bao et al., 2022), e,g., BYOL (Grill et al., 2020). can also tackled by advanced BN variants (Hoffer et al., 2017; Wu & Johnson, 2021).
especially when MogaNet-S/B adopts the EMA strategy (by default), indicating that
Norm (after)
Input size
LN
LN
BN
pBN
Norm (within)
LN
BN
BN
pBN
MogaNet-T
224
78.4
79.1
79.0
79.1
MogaNet-T
2562
78.8
79.4
79.6
79.6
MogaNet-S
224
82.5
83.2
83.3
83.3
MogaNet-S (EMA)
2242
82.7
83.2
83.3
83.4
MogaNet-B
224
83.4
83.9
84.1
84.2
MogaNet-B (EMA)
224"
83.7
83.8
84.3
84.4
Table A6: Ablation of normalization layers in MogaNet.
C.5 REFINED TRAINING SETTINGS FOR LIGHTWEIGHT MODELS
To explore the full power of lightweight models of our MogaNet, we refined the basic training settings for MogaNet-XT/T according to RSB A2 (Wightman et al., 2021) and DeiT-IIIf (Touvron al., 2022). Compared to the default setting as provided in Table A2, we only adjust the learning rate and the augmentation strategies for faster convergence while keeping other settings unchanged. As shown in Table A7, MogaNet-XT// gain +0.4~0.6% when use the large learning rate of 2 x 10-3 and 3- 3-Augment (Touvron et al., 2022) without complex designs. Based on the advanced setting, MogaNet with 2242 input resolutions yields significant performance improvements against previous methods, e.g., MogaNet-T gains +3.5% over DeïT-T (Touvron al., 202 and +1.2% over Pare- Net-S (Zhang et al., 2022b). Especially, MogaNet-T with 256² resolutions achieves top-1 accuracy of 80.0%, outperforming DeiT-S of 79.8% reported in the original paper, while MogaNet-XTT with
27

---

Published as a conference paper at ICLR 2024
Architecture
Input Learning
Warmup
Rand
3-Augment EMA
Top-1
size
rate
epochs
Augment
Acc (%)
DeiT-T
224 1 10
5
9/0.5
72.2
DeiT-T
224 2 x 10-3
20
X
x
75.9
ParC-Net-S
256 1 10
9/0.5
/
78.6
ParC-Net-S
256 2 x 10-3
20
X
/
x
78.8
MogaNet-X7 XT 224 1 x 10-3
5
7/0.5
x
x
76.5
MogaNet-XT
224 x 10-3
20
X
X
77.1
MogaNet-XT
TT 256² 1 x 10
5
7/0.5
X
x
77.2
MogaNet-XT
256 2 10-3
20
X
X
77.6
MogaNet-T
224 1 x 10
5
7/0.5
X
X
79.0
MogaNet-T
2242
2 X 10-3
20
X
X
79.4
MogaNet-T
256 1 X 10-3
5
7/0.5
X
X
79.6
MogaNet-T
256
2 x 10
20
X
X
80.0
Table A7: Advanced training recipes for Lightweight models of MogaNet on ImageNet-
Architecture
Type
#P.
FLOPs
RetinaNet 1x
(M)
(G)
AP
AP50
AP75
AP$
AP
APt
RegNet-800M
C
17
168
35.6
54.7
37.7
19.7
390
47.8
PVTV2-BO
T
13
160
37.1
57.2
39.2
23.4
40.4
49.2
MogaNet-XT
167
39.7
60.0
42.4
23.8
43.6
ResNet-18
21
189
31.8
49.6
33.6
16.3
34.3
43.2
RegNet-1.6G
C
20
185
37.4
56.8
39.8
22.4
41.1
49.2
RegNet-3.2G
C
26
218
39.0
58.4
41.9
22.6
43.5
50.8
PVT-T
T
23
183
36.7
56.9
38.9
22.6
38.8
50.0
PoolFormer-S12
22
207
36.2
56.2
38.2
20.8
39.1
48.0
PVTV2-B1
187
41.1
61.4
43.8
26.0
44.6
MogaNet-T
C
14
173
41.4
61.5
44.4
25.1
45.7
53.6
ResNet-50
C
37
239
36.3
55.3
38.6
19.3
40.0
48.8
Swin-T
T
38
245
41.8
62.6
44.7
25.2
45.8
54.7
PVT-S
226
40.4
61.3
43.0
25.0
42.9
Twins-SVT-S
209
42.3
63.4
45.2
26.0
45.5
56.5
Focal-T
T
39
265
43.7
PoolForme--$36
T
41
272
39.5
60.5
41.8
22.5
42.9
52.4
PVTV2-B2
T
35
281
44.6
65.7
47.6
28.6
48.5
59.2
CMT-S
HI
45
231
44.3
65.5
47.5
27.1
48.3
59.1
MogaNet-S
253
45.8
66.6
49.0
29.1
50.1
59.8
ResNet-101
C
57
315
38.5
57.8
41.2
21.4
42.6
51.1
PVT-M
T
54
258
41.9
63.1
44.3
25.0
44.9
57.6
Focal-S
T
62
367
45.6
PVTV2-B3
T
55
263
46.0
67.0
49.5
28.2
50.0
61.3
PVTV2-B4
315
46.3
67.0
49.6
29.0
50.1
MogaNet-B
54
355
47.7
68.9
51.0
30.5
52.2
ResNeXt-101-64
C
95
473
41.0
60.9
44.0
23.9
45.2
54.0
PVTV2-B5
T
92
335
46.1
66.6
49.5
27.8
50.2
62.0
MogaNet-L
C
92
477
48.7
69.5
52.6
31.5
53.4
62.7
Table A8: Object detection with RetinaNet (1 x training schedule) on COCO val2017. The FLOPs are measured resolution 800 x 1280.
224 resolutions outperforms DeiT-T under the refined training scheme by 1.2% with only 3M parameters.
D MORE COMPARISON EXPERIMENTS
D. FAST TRAINING IMAGENET-IK
In addition to Sec. 5.1, we further provide comparison results for 100 and 300 epochs training ImageNet- KK. As for 100-epoch training, we adopt the original RSB A3 (Wightman et al., 2021) setting for all methods, which adopts LAMB (You et al., 2020) optimizer and small training res- olution of 160 We search the basic learning in {0.006, 0.008) for all architectures and adopt the gradient clipping for Transformer-basee networks. As for 300-epoch training, report results of
28

---

Published as a conference paper at ICLR 2024
Architecture
Type
#P.
FLOPs
Mask R-CNN 1x
(M)
(G)
AP
AP
AP
APm
50
AP75
50
RegNet-800M
C
27
187
37.5
57.9
41.1
34.3
56.0
36.8
MogaNet-XT
C
23
185
40.7
62.3
44.4
37.6
59.6
40.2
ResNet-18
207
54.0
36.7
51.0
32.7
RegNet-1.6G
C
29
204
38.9
60.5
43.1
35.7
57.4
38.9
PVT-T
T
33
208
36.7
59.2
39.3
35.1
56.7
37.3
PoolFormer- S12
T
32
207
37.3
59.0
40.1
34.6
55.8
36.9
MogaNet-T
C
25
192
42.6
64.0
46.4
39.1
61.3
42.0
ResNet-50
C
44
260
38.0)
58.6
41.4
34.4
55.1
36.7
RegNet-6.4G
307
62.3
37.1
39.6
PVI-S
T
44
245
40.4
62.9
43.8
37.8
60.1
40.3
Swin-T
T
48
264
42.2
64.6
46.2
39.1
61.6
42.0
MVīT-T
T
46
326
45.9
68.7
50.5
42.1
66.0
45.4
PoolFormer-S
T
32
207
41.0
63.1
44.8
37.7
60.1
40.0
Focal-T
T
49
291
44.8
67.7
49.2
41.0
64.7
44.2
PVTV2-B2
67.1
64.2
44.4
LITV2-S
44.9
67.0
49.5
40.8
63.8
44.2
CMT-S
H
45
249
44.6
66.8
48.9
40.7
63.9
43.4
Conformer- S/16
H
58
341
43.6
65.6
47.7
39.7
62.6
42.5
Uniformer-S
H
41
269
45.6
68.1
49.7
41.6
64.8
45.0
ConvNeXt-T
C
48
262
44.2
66.6
48.3
40.1
63.3
42.8
FocalNet-T (SRF)
267
45.9
68.3
50.1
41.3
65.0
44.3
FocalNet-T (LRF)
268
68.2
44.5
MogaNet-S
C
45
272
46.7
68.0
51.3
42.2
65.4
45.5
ResNet-101
C
63
336
40.4
61.1
44,2
36.4
57.7
38.8
RegNet-12G
C
64
423
42.2
63.7
46.1
38.0
60.5
40.5
PVT-M
T
64
302
42.0
64.4
45.6
39.0
61.6
42.1
Swin-S
354
66.6
44.2
Focal-S
71
47.4
69.8
51.9
66.6
PVTV2-B3
T 65
397
47.0
68.1
51.7
42.5
65.7
45.7
LITV2-M
T
68
315
46.5
68.0
50.9
42.0
65.1
45.0
UniFormer-B
H
69
399
47.4
69.7
52.1
43.1
66.0
46.5
ConvNeXt-S
C
70
348
45.4
67.9
50.0
41.8
65.2
45.1
MogaNet-B
373
47.9
70.0
46.6
Swin-B
T
107
496
46.9
69.6
51.2
42.3
65.9
45.6
PVTV2-B5
T
102
557
47.4
68.6
51.9
42.5
65.7
46.0
ConvNeXt-B
C
108
486
47.0
69.4
51.7
42.7
66.3
46.0
FocalNet-B (SRF)
C
109
496
48.8
70.7
53.5
43.3
67.5
46.5
MoggNet-L
C
102
495
49.4
70.7
54.1
44.1
68.1
47.6
Table A9: Object detection and instance segmentation with Mask R-CNN (1x training schedule) on COCO val2017. The FL.OPs are measured at resolution 800 1280.
RSB A2 (Wightman et al., 2021) for classical CNN or the original setting for Transformers mod- em ConvNets. Table A15, when compared with models of similar parameter size, our proposed MogaNet-XT/T/S/BB achieves the best performance in both 100 and 300 epochs training. Results of 100-epoch training show that MogaNet has faster convergence speed than previous architectures of various types. For example, MogaNet-T outperforms EfficientNet-BOO and DeïT-T by 2.4% and 8.7%. MogaNet-S outperforms Swin-T by 3.4%, and MogaNet-B outperforms Swin-S by 2.0%. Notice that ConvNeXt variants have great convergence speed, egg., ConvNeXt-S achieves 81.7% surpassing Swin-S by 1.5 and recently proposed ConvNet HorNet-S7x7 by 0.5 with similar param- eters. But proposed MogaNet convergences faster than ConvNet, e.g., MogaNet-S outperforms CoT by 2.3% with similar parameters while MogaNet-B/L reaching competitive perfor- mances as Bon/NeXt-B/L with only 44~50% parameters.
D.2 DETECTION AND SEGMENTATION RESULTS ON COCO
In addition to Sec. 5.2, we provide full results of object detection and instance segmentation tasks with RetinaNet, Mask R-CNN, and Cascade Mask R-CNN on COCO. As shown in Table A8 and Table A9, RetinaNet or Mask R-CNN with MogaNet variants outperforms existing models when training 1x schedule. For example. RetinaNet with MogaNet-T/S/B/L achieve 45.8/47.7/48.7 AP* outperforming PVT-T/S/M and PVTV2-BI/B2/BB by 4.7/4.6/5.8 and R-CNN with MogaNet-S/B/L. achieve 46.7/47.9/49) AP' exceeding Swin-T/S/B and Con
AP", Nask
29

---

Published as a conference paper at ICLR 2024
Architecture
Type
#P.
FLOPs
Cascade Mask R-CNN +MS 3x
(M)
(G)
Ap
AP
AP%
APm
AP
AP7
ResNet-50
C
77
739
46.3
64.3
50.5
40.1
61.7
43.4
Swin-T
T
86
745
50.4
69.2
54.7
43.7
66.6
47.3
Focal-T
T
87
770
51.5
70.6
55.9
ConvNeXt-T
C
86
741
50.4
69.1
54.8
43.7
66.5
47.3
TocalNet-T (SRF)
C
86
746
51.5
70.1
55.8
44.6
67.7
48.4
MogaNet-S
C
78
750
51.6
70.8
56.3
45.1
68.7
48.8
ResNet-101-32
96
48.1
52.4
45.2
Swin-S
T
107
838
51.9
70.7
56.3
45.0
68.2
48.8
ConvNeXI-S
C
108
827
51.9
70.8
56.5
45.0
68.4
49.1
MogaNNt-B
C
101
851
52.6
72.0
57.3
46.0
69.6
49.7
Swin-B
T
145
982
51.9
70.5
56.4
45.0
68.1
48.9
ConvNeXt-B
C
146
964
52.7
71.3
57.2
45.6
68.9
49.5
MogaNet-L
C
140
53.3
71.8
69.2
49.8
Swin-L
T
253
1382
53.9
72.4
58.8
46.7
70.1
50.8
ConvNeXt-LL
C
255
1354
54.8
73.8
59.8
47.6
71.3
51.7
ConvNeXt-XL
C 407
1898
55.2
74.2
59.9
47.7
71.6
52.2
RepLKNet-311
C 229
1321
53.9
72.5
58.6
46.5
70.0
50.6
HorNet-L
259
1399
56.0
48.6
MogaNet-XL
C 238
1355
56.2
75.0
61.2
48.8
72.6
53.3
Table A10: Object detection and instance segmentation with Cascade Mask R-CNN (3x training schedule) with multi-scaling training (MS) on COCO va/2017. denotes the model is pre-trained on ImageNet-21 The FL.OPs are measured at resolution 800 1280.
T/S/B by 4.5/3.1/2.5 and 2.5/2.5/2.4 with similar parameters and computational overloads. Notice- ably, MogaNet-XT/T can achieve better detection results with fewer parameters and lower FLOPs than lightweight architectures, while MogaNet-T even surpasses some Transformers like Swin-S and PVT-S. For example, Mask R-CNN with MogaNet-T improves Swin-T by 0.4 AP and out- performs PVT-S by 1.3 APM using only around 2/3 parameters. As shown in Table A10, Cascade Mask R-CNN with MogaNet variants still achieves the state-of-the-art detection and segmentation results when training 3x schedule with multi-scaling (MS) and advanced augmentations. For ample, MogaNet-L/XL. yield 53.3/56.2 AP' and 46. 1/4.88 APm, which improves Swin-B/L. and ConvNeXt-B/LL by 1.4/2.3 and 0.6/1.4 AP" with similar parameters and FLOPS.
D.3 SEMENTIC SEGMENTATION RESULTS ON ADE20K
addition to Sec. 5.2, we provide comprehensive comparison results of semantic segmentation based UperNet on ADE20K. As shown in Table A11, UperNet with MogaNet produces state- of-the-art performances in a wide range parameter scales compared to famous Transformer, hy- brid, and convolution models. As for the lightweight models, MogaNet-XT/T significantly im- proves ResNet- 18/50 with fewer parameters and FLOPs budgets. As for medium-scaling models, MogaNet-S/B achieves 49.2/50.1 mloUs* which outperforms the recently proposed ConvNets, e.g., +1.1 over HorNet-T using similar parameters and +0.7 over SLaK-S using 17M fewer parameters. As for large models, MogaNet-L/XL surpass Swin-B/L and ConvNeXt-B/L. by 1.2/1.9 and 1.8/0.3 msoUss while using fewer parameters.
D.4 2D HUMAN POSE ESTIMATION RESULTS ON COCO
In addition to Sec. 5.2, we provide comprehensive experiment results of 2D human key points esti- mation based on Top-Down SimpleBaseline on COCO. As shown in Table A13, MogaNet variants achieve competitive or state-of-the-art performances compared to popular architectures with two types resolutions. As for lightweight models, MogaNet-XTT XT/T significantly improves the perfor- mances of existing models while using similar parameters and FLOPs. Meanwhile, MogaNet-S/D also produces 74.9/75.3 and 76.4/77.3 AP using 256 192 and 384 x 288 resolutions, outperforming Swin-B/L. by 2.0/1.0 0 and 1 1.5/1.0 AP with nearly half of the parameters and computation budgets.
30

---

Published as a conference paper at ICLR 2024
Architecture
Date
Type
Crop size
Param.
FLOPs
m
(M)
(G)
(%)
ResNet-18
WVPR2006
C
512
41
885
39.2
MogaNet-XT
Ours
512
30
856
42.2
ResNet-50
CVPR 2016
C
512
67
952
42.1
MogaNet-T
Ours
512
33
862
43.7
DeiT-S
ICML'2021
T
512
52
1099
44.0
Swin-T
ICCV'2021
512
60
945
46.1
TwinsP-S
20P5
T
512
55
919
46.2
Twins-S
2OPS
T
5122
54
901
46.2
Focal-T
NIPS' 2021
T
5122
62
998
45.8
Uniformer-Swas W312
ICLR'2022
H 512
52
955
47.0
UniFormer-S
ICLR'
5122
52
1008
47.6
ConvNeXI-T
C222'2022
C
5122
60
939
46.7
FocalNet-T (SRF)
NIPS'2022
5122
61
944
HorNet-T7xx7
NIPS
C
512²
52
926
48.1
MogaNet-S
Ours
5122
55
946
49.2
Swin-S
ICCV'2021
T
512"
81
1038
48.1
Twins-B
NIPS'2021
5122
89
1020
47.7
Focal-S
NIPS'2021
T
5122
85
1130
48.0
Uniformer-Banh
ICLR'2022
H
512
80
1106
49.5
ConvNeXt-S
CVPR'2022
C
512
82
1027
48.7
FocalNet-S (SRF)
NIPS 2022
C
5122
83
1035
49.3
SLaK-S
ICLR'2023
5122
91
1028
MogaNet-B
Ours
5122
74
1050
50,1
Swin-B
ICCV'2021
5124
121
1188
49.7
Focal-B
NIPS'2021
T
5122
126
1354
49.0
ConwNeXt-B
CVPR'2022
512"
122
1170
49.1
R<pL
CVPR'2022
C
5122
112
1170
49,9
FocalNet-B (SRF)
NIPS'2022
5122
124
1180
50.2
SLaK-B
ICLR'2023
C
5122
135
1185
50.2
MogaNet-L
Ours
C
512"
113
1176
50.9
Swin-I.
ICCV'2021
T
6402
234
2468
52.1
ConwNeXt-L
CVPR'2022
C
6402
245
2458
53.7
RopLKNett311
CVPR'2022
C
640
207
2404
52,4
MogaNett-L..
Ours
640
214
2451
54.0
Table A11: Semantic segmentation with UperNet (160K) on ADE20K validation set. indicates
using IN-21K pre-trained models. The FLOPs are measured at 512 x 2048 or 640 x 2560 resolutions.
Architecture
Hand
Face
Type
#P.
FLOPs
PA-MPJPE
4Р.
FLOPs
3DRMSE
(M)
(G)
(mm)
(M)
(G)
MobileNetV2
4.8
0.3
8.33
4.9
0.4
2.64
ResNet-18
13.0
7.51
13.1
2.4
2.40
MogaNet-T
C
6.5
1.1
6.82
6.6
1.5
2.36
ResNet-50
26.9
4.1
27.0
5.4
2.48
ResNet-101
C
45.9
7.9
6.44
46.0
10.3
2.47
DeiT-S
23.4
4.3
7.86
23.5
5.5
2.52
Swin-T
30.2
4.6
6.97
30.3
6.1
2.45
Swin-S
T
51.0
13.8
6.50
50.9
8.5
2.48
ConvNeXt-T
29.9
4.5
6.18
30.0
5.8
2.34
NeX-NS
C
51.5
8.7
6.04
51.6
11.4
2.27
HorNet-T
C
23.7
4.3
6.46
23.8
5.6
2.39
MogaNet-S
26.6
26.7
6.5
2.24
Table A12: 3D human pose estimation with ExPose on FFHQ and FreiHAND datasets. The face and hand tasks use pre-vertex and pre-joint errors as the metric. The FLOPs of the face and hand tasks are measured with input images 2562 and 2242 resolutions.
31

---

Published as a conference paper at ICLR 2024
Architecture
Type
Crop
#P.
FLOPs
AP
AP30
AP
AR
size
(M)
(G)
(%)
(%)
(%)
(%)
MobileNetV2
256 x 192
1.6
64.6
87.4
72.3
70.7
ShuffleNetV2 2x
C
256 x 192
8
1.4
59.9
85.4
66.3
66.4
MogaNet-XT
256
1.8
72.1
89.7
80.1
77.7
RSN-18
C
256 x 192
9
2.3
70.4
88.7
77.9
77.1
MogaNet-T
256 x 192
2.2
73.2
90.1
81.0
78.8
ResNet-50
C
256 x 192
34
5.5
72.1
89.9
80.2
77.6
HRNet- W32
256 x 192
29
7.1
74.4
90.5
81.9
78.9
Swin-T
T
256 x 192
33
6.1
72.4
90.1
80.6
78.2
PVT-S
256 x 192
28
4.1
71.4
89.6
79.4
77.3
PVTV2-B2
T
256 x 192
29
4.3
73.7
90.5
81.2
79.1
Uniformer-S
256 x 192
25
4.7
74.0
90.3
82.2
79.5
-onvNeXI-T
256 X 192
33
5.5
73.2
90.0
80.9
78.8
MogaNet-S
256 X 192
29
6.0
74.9
90.7
82.8
80.1
ResNet-101
256 x 192
53
12.4
71.4
89.3
79.3
77.1
ResNet-152
256 x 192
69
15.7
72.0
89.3
79.8
77.8
HRNet-W44
C
256 X 192
64
14.6
75.1
90.6
82.2
80.4
Swin-B
256 x 192 93
18.6
72.9
89.9
80.8
78.6
Swin-I.
T
256 X 192
203
40.3
74.3
90.6
82.1
79.8
Uniformer-B
256 X 192
9.2
75.0
90.6
83.0
80.4
ConvNeXt-S
256 X 192
55
9.7
73.7
90.3
81.9
79.3
ConvNeXt-B
256 X 192
94
16.4
74.0
90.7
82.1
79.5
MogaNet-B
256 X 192
47
10.9
75.3
90.9
83.3
80.7
MobileNetV2
384 x 288
3.6
67.3
87.9
74.3
72.9
ShuffleNetV2 2x
384 x 288
8
3.1
63.6
86.5
70.5
69.7
MogaNet-XT
384 x 288
4.2
74.7
90.1
81.3
79.9
RSN-18
C
384 288
9
5.1
72.1
89.5
79.8
78.6
MogaNet-T
384 x 288
4.9
75.7
90.6
82.6
80.9
HRNet-W32
384 x 288
29
16.0
75.8
90.6
82.7
81.0
Uniformer-S
384 x 288
25
11.1
75.9
90.6
83.4
81.4
ConvNeXt-T
C
384 x 288
33
33.1
75.3
90.4
82.1
80.5
MogaNet-S
C
384 x 288
29
13.5
76.4
91.0
83.3
81.4
ResNet-152
384 288
69
35.6
74.3
89.6
81.1
79.7
HRNet-W48
C
384 x 288
64
32.9
76.3
90.8
82.0
81.2
Swin-B
384 288
93
39.2
74.9
90.5
81.8
80.3
Swin-L
T
384 x 288
203
86.9
76.3
91.2
83.0
814
HRFormer-B
T
384 288
54
30.7
77.2
91.0
83.6
82.0
ConvNeXi-S
C
384 x 288
55
21.8
75.8
90.7
83.1
81.0
ConvNeXt-B
C 384 x 288
94
36.6
75.9
90.6
83.1
81.1
Uniformer-B
384 x 288
54
14.8
76.7
90.8
84.0
81.4
MogaNet-B
C 384 x 288
47
24.4
77.3
91.4
84.0
82.2
Table A13: 2D human pose estimation with Top-Down SimpleBaseline on COCO val2017. The FLOPs are measured at 256 x 192 or 384 x 288 resolutions.
D.5 3D HUMAN POSE ESTIMATION RESULTS
In addition to Sec. 5.2, we evaluate popular ConvNets and MogaNet for 3D human pose estimation tasks based on ExPose (Choutas al., 2020). As shown in Table A12, MogaNet achieves lower regression errors with efficient usage of parameters and computational overheads. Compared to lightweight architectures, MogaNet-T achieves 6.82 MPJPE and 2.36 3DRMSE hand and face reconstruction tasks, improving ResNet- and MobileNetV2 1x by 1. 1.29/0.04 and 1.51/0 Com- pared models around 25- parameters, MogaNet-S surpasses ResNet-101 and ConvNeXt-T, achieving competitive results as ConvNeXt-S with relatively smaller parameters and FLOPss (e.g., 27M/6.5G vs 52M/11 1.4G on FFHP). Notice that some backbones with more parameters produce worse results than their lightweight variants on the face estimation tasks (e.g., ResNet-50 and Swin- S), while MogaNet-S still yields the better performance of 2.24 3DRMSE.
32

---

Published as a conference paper at ICLR 2024
D.6 VIDEO PREDICTION RESULTS ON MOVING MNIST
In addition to Sec. 5.2, We verify video prediction performances of various architectures by replac- ing the hidden translator in SP with the architecture blocks. All models use the same number of network blocks and have similar parameters and FLOPs. As shown in Table A14, Compared to Transformer- based and Metaformer-basee architectures, pure ConvNets usually achieve lower pre- diction errors. When training 200 epochs, it is worth noticing that using MogaNet blocks in SimVI VP significantly improves the SimVP baseline by 6.58/13.86 MSE/MAE and outperforms ConvNeXt and HorNet by 1.37 and 4.07 MSE. MogaNet also holds the best performances in the extended 2000-epoch training setting.
Architecture
#P.
FLOPs
FPS
200
epochs
2000 epochs
(M)
(G)
(8)
MSEL
MAEL
SSIM
MSEJ
MAE.
SSIM
ViT
46.1
16.9
290
35.15
95.87
0.9139
19.74
61.65
0.9539
Swin
46.1
16.4
294
29.70
84.05
0.9331
19.11
59.84
0,9584
Uniformer
44.8
16.5
296
30.38
85.87
0.9308
18.01
57.52
0.9609
ML.-Mixer
38.2
14.7
334
29.52
83.36
0.9338
18.85
59.86
0.9589
ConyMixer
3.9
5.5
658
32.09
88.93
0.9259
22.30
67.37
0.9507
Poolformer
37.1
14.1
341
31.79
88.48
0.9271
20.96
64.31
0,9539
Sim
58.0
19.4
209
32.15
89.05
0,9268
21.15
64.15
0.9536
ConvNeXt
37.3
14.1
344
26.94
77.23
0.9397
17.58
55.76
0.9617
VAN
44.5
16.0
288
26.10
76.11
0.9417
16.21
53.57
0,9646
HorNet
45.7
16.3
287
29.64
83.26
0.9331
17.40
55.70
0.9624
MogaNet
46.8
16.5
255
25.57
75.19
0.9429
15.67
51.84
0.9661
Table A14: Video prediction with SimVP on Moving MNIST. The FLOPs and FPS are measured
at the input tensor of 10 x 1 x 64 x 64 on NVIDIA Tesla V100 GPU.
E EXTENSIVE RELATED WORK
Convolutional Neural Networks ConvNets (LeCun et al., 1998; Krizhevsky et al., 2012; He et al., 2016) have dominated a wide range of computer vision (CV) tasks for decades. VGG (Simonyan & Zisserman, 2014) proposes modular network design strategy, stacking the same type of blocks repeatedly, which simplifies both the design workflow and transfer learning for downstream tasks. ResNet (He al., 2016) introduces identity skip connections and bottleneck modules that alleviate training difficulties (e.g., vanishing gradient). With the desired properties, ResNet and its vari- ants (Zagoruyko & Komodakis, 2016; Xie et al., 2017; Hu al., 2018; Zhang al., 2022a) have become the most widely adopted ConvNet architectures in numerous CV applications. For practical usage, efficient models (Ma et al., 2018; Howard et al., 2017: Sandler et al., 2018: Howard et al., 2019; Tan & Le, 2019; Radosavovic et al., 2020) are designed for a campleextty-accuraa trade-off and hardware devices. Since the limited reception fields, spatial and temporal convolutions strug- gle to capture global dependency (Luo et al., 2016). Various spatial-wise or channel-wise attention strategies (Dai et al., 2017: Hu et al., 2018; Wang et al., 2018; Woo et al., 2018; Cao et al., 2019) are introduced. Recently, taking the merits of Transformer- like macro design (Dosovitskiy al., 2021), modern ConvNets (Trockman & Kolter, 2022; Ding et al., 2022b; Liu et al., 2023; Rao et al., 2022; Kirchmeyer & Deng, 2023) show thrilling performance with large depth-wise convolutions (Han et al., 2021b) for global contextual features. Among them, VAN (Guo et al., 2023), FocalNet (Yang al.. 2022), HorNet (Rao et al., 2022), and Conv2Former (Hou et al., 2022) exploit multi-scale convolutional kernels with gating operations. However, these methods fail to ensure the networks learn the inherently overlooked features (Deng et al., 2022) and achieve ideal contextual aggrega- tion. Unlike the previous works, we first design three groups of multi-order depth-wise convolutions in parallel followed by double-brance activated gating operation, and then propose a channel ag- gregation module to enforce the network to learn informative features of various interaction scales.
Vision Transformers Transformer (Vaswani et al., 2017) with self-attention mechanism has be- come the mainstream choice in natural language processing (NLP) community (Devlin et al., 2018; Brown al., 2020). Considering that global information is also essential for CV tasks, Vision Transformer (VIT) (Dosovitskiy et al., 2021) is proposed and has achieved promising results on ImageNet (Deng et al., 2009). In particular, ViT splits raw images into non-overlapping fixed-
33

---

Published as a conference paper at ICLR 2024
Architecture
Date
Type Param.
100-epoch
300-epoch
(M)
Train
Test
Ace
(%)
Train Test
Ace (%)
ResNet-18 (He et at., 2016)
CVPR
C
12
160
224"
68.2
224"
224
70.6
ResNet-34 (He et at.,
2016 C
22
160
224
73.0
224
224
75.5
ResNet-50 (Hc al.,
CVP6 c
26
160
224
78.1
224a
224
79.8
ResNet-101 (Ho al., 2016)
2016 C
45
160²
224
79.9
224
224a
81.3
ResNet-152 (Hc ct al., 2016)
CVPR C
60
160²
224
80.7
224
224
82.0
ResNer-200 (He al,, 2016)
CVPR22016 C
65
160
224
80.9
224
224
82.1
ResNeX1-50 (Xie et al, 2017)
CVPR 2017
C
25
160
224
79.2
224
224
80.4
SE-ResNet-50 (Hu al., 2018)
CVPR 2018
C
28
160
224
77.0
224
224
80.1
Efficien[Net-B0 (Tan & Le, 2019)
1CML'2019
C
5
160 224
73.0
224
224
77.1
EfficientNet-B1 (Tan & Le, 2019)
ICML'2019
C
8
160 224
74.9
240
240
79.4
EfficientNet-22 (Tan & Le, 2019)
ICML'2019
C
9
1922 256
77.5
2602
260
80.1
FfficientNee-R3 (Tan & Le, 2019)
ICMI22019
C
12
224
288
79.2
3002
300
81.4
EfficientNet-44 (Tan & Le, 2019)
ICML'2019
C
19
320
380
81.2
380
380
82.4
RegNeeY-800MF (Radosavovic et al., 2020)
CVPR 2020
C
6
160
224
73.8
224
224
76.3
RegNeeY-44F (Radosavovic al., 2020)
CVPR 2020
C
21
160
224
79.0
224
224
79.4
RegNetY-8GF (Radosavovic al., 2020)
CVPR 2020
C
39
160
224
81.1
224
224
79.9
RegNetY-16GII (Radosavovic al. 2020)
CVPR 2020
C
84
160
224
81.7
224
224
80.4
ElficientNetV2nnvSSS (Tan & Le, 2021)
ICML
C
24
224
288
80.9
288
384
82.9
Eilicien/NelV--v-MM (Tan & Le, 2021)
ICML'2021
C
S3
256
384
82.3
320
384
81.9
VTT- (Desovitkkyy et al., 2021)
ICLR' 2021
T
6
160
224"
66.7
224"
224"
72.2
VIT-S (Dosovitskiy al., 2021)
ICLR 2021
T
22
160
224
73.8
224
224
79.8
VIT-B (Dosovitskiy et al., 2021)
ICLR 2021
T
86
160
224
76.0
224
224
81.8
PVT-T (Wang et al., 202
ICCV'2021
T
13
I
160
224
71.5
224
224
75.1
PVT-S (Wang et al., 2021b)
ICCV'2021
T
25
160
224
72.1
224
224
79.8
Swin-T (Liu al., 2021)
ICCV'2021
T
28
160
224
77.7
224
224
81.3
Swin-S (Liu et al. 2021)
ICCV'2001
T
50
160
224
80.2
224
224
83.0
Swin-S (1.u et al., 2021)
ICCV'2021
T
50
160
224
80.5
224
224
83.5
LITV2-T (Pan al., 2022a)
NIPS'
T
28
160
224
79.7
224
224
82.0
LITV2-M (Pan al., 2022a)
2022
T
49
160
224
80.5
224
224
83.3
LITV2-B (Pan al., 2022a)
NIPS'2022 T
87
160
224
81.3
224
224
83.6
ComvMixe:-768-132 (Trockman & Kolter, 2022)
arXiv'2022
T
21
160
224
77.6
224
224
80,2
Poollformer-S12
al., 2022)
CVPR
T
12
160
224
69.3
224
224
77.2
Poollforeer-S24 (Yu 2022)
CVPR
T
21
160
224
74.1
224
224
80,3
PoolFForer--336 (Yu 2022)
CVPR 22022
T
31
160
224
74.6
224
224
81.4
PoolFoomer- (Yu et al., 2022)
CVPR '202
T
S6
160
224
80.7
224
224
82.1
PoolFoomer:-M48 et al., 2022)
CVPR '202 T
73
160
224
81.2
224
224
82.5
Con/NeXt-T
20226)
CVPR 2022
C
29
160
224" 160
78.8
224
224
82.1
ConvNeXt-S al., 2022h)
EVPR'2022
50
224
81.7
224
224
83.1
ConvNeXt-B al., 2022h)
EVPR'2022
89
160
224
82.1
224
224
83.8
Con-NeXi-L
2022b)
CVPR 2022
189
160
224
82.8
224
224
84.3
ConvNeXI-XL (Liu et al., 2022b)
CVPR '202
350
160
224
82.9
224
224
84.5
HorNet-Trx7 (Rao et al., 2022)
NIPS'
22
160
224
80.1
224
224
82.8
HorNet-S7xz7 (Rao
2022)
NIPS'2022
C
50
160
2242
81.2
224
224
84.0
VAN-BO (Guo al., 2023)
2023
160
224
72.6
224
224
75.8
VAN-B2 (Guo al., 2023)
CVMJ'2023 2023
27
160
224
81.0
224
224
82.8
VAN-B3 (Guo al., 2023)
CVMJ'2022
45
160
224
81.9
224
224
83,9
MogaNet-XT
Ours
3
160
224
72.8
224
224
76.5
MogaNet-T
C
5
160
224
75.4
224
224
79.0
MogaNet-S
C
25
160
224
81.1
224
224
83.4
MogaNet-B
Ours
44
160
224
82.2
224
224
84.3
MogaNet-L
83
160
224
83.2
224
224
84.7
Table A15: ImageNet-1K, classification performance of tiny to medium size models (5~50M) train- ing 100 and 300 epochs. RSB A3 (Wightman et al.. 2021) setting is used for 100-epoch training of all methods. As for 300-epoch results, the RSB A2 (Wightman ResNet, ResNeXt, SE-ResNet, EfficientNet, and EfficientNerV2 as reproduced in timm (Wightman al., 2021), while other methods adopt settings in their original paper.
al., 2021) setting used for
size patches as visual tokens to capture long-range feature interactions among these tokens by self- attention. By introducing regional inductive bias, ViT and its variants have been extended to various vision tasks Carion al. (2020); Zhu et al. (2021); Chen et al. (2021), ; Parmar et al. (2018); Jiang et al. (2021a): Arnab et al. (2021). Equipped with advanced training strategies (Touvron et al., 2021a; 2022) or extra knowledge (Jiang et al., 2021b; Lin et al., 2022; Wu et al., 2022c), pure VITs can achieve competitive performance ConvNets in CV tasks. In the literature of Yu et al. (2022), the MetaFormer architecture substantially influenced the design of vision backbones, and all Traasformer-liie like models (Touvron et al., 2021a; Trockman & Kolter, 2022: Wang et al., 2022a)
34

---

Published as a conference paper at ICLR 2024
classified by how they treat the token-mixing approaches, such relative position encoding (Wu et al., 2021b), local window shifting (Liu et al., 2021) and MLP layer (Tolstikhin et al., 2021), etc. Beyond the aspect of macro design, Touvron et al. (2021b); Yuan et al. (2021a) introduced knowl- edge distillation and progressive tokenization to boost training data efficiency. Compared to Con- vNets banking
the inherent inductive biases (e.g. locality and translation equivariance), the pure VITs are more over peee-parameterieed and rely on large-scale pre-training (Dosovitskiy et al., 2021: Li al., 2023b) by contrastive learning (He al., 2020; Zang et al., 2022; Li al., 2023c) masked image modeling (Bao et al., 2022: He et al., 2022: Li et al., 2023a; Woo et al., 2023) to great ex- tent. Targeting this problem, one branch of researchers proposes lightweight ViTs (Xiao al., 2021; Mehta & Rastegari, 2022; Li et al., 2022c; Chen et al., 2023) with more efficient self-attentions vari- ants Wang et al., a)). Meanwhile, the incorporation of self-attention and convolution as hybrid backbone has been vigorously studied (Guo et al., 2022; Wu et al., 2021a; Dai et al., 2021; d' A'oolii et al., 2021; Li et al., 2022a; Pan et al., 2022b; Si ct al., 2022) for imparting regional priors to ViTs.
35


Reviews:
[{"review_id": "yM694tgjRX", "reviewer": "Reviewer_6yMg", "summary": "This paper introduces a new convolutional neural network for varies of computer vision tasks.\nSpecifically, it is motivated by the theory of  multi-order game-theoretic interaction in deep neural networks.\nThis paper finds that popular transformer-based and CNN-based networks have limited interactions on the middle-order interactions. So it introduces the MogaNet with multi-order gated aggregation to solve this problem. \nMogaNet uses convolutions with different kernel size as well as the gated aggregation, which can adatively conduct the multi-order aggregation.\n\nExperiments are perform on several popular benchmarks, such image classification, semantic segmentation, object detection, instance segmentation, and pose estimation. Results show that MogaNet achieves the SOTA performance on several popular benchmarks.", "questions": "Refer to the weakness.\n\nThe major problem is the limited novelty. Besides, there lacks a comprehensive discussion on convolutional modulations.\nThe major improvement is from the CA, not the modules claimed as major contributions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written, comprehensively introducing the motivation and method details.\n\n+ The experiments are comprehensive, covering several popular vision tasks as well as varies of network scales.\n\n+ The experimental and visualized analysis is good, helping the reviewer better understand the method.\n\n+ Code has been released, so the reproducibility can be ensured.", "weaknesses": "- Despite good experiments and visualizations, I think the novelty is limited.\nAs described in the introduction, the low-order interactions are modeling the local features, such as edge and texture. The high-order on the other hand models high-level semantic features. So multi-order feature aggreation indicates the multiscale aggregation with low and high level features. This paper implements it via depth-wise convolution with different kernel size and further adds gated operation, introducing multi-order gated aggregation.\nHowever, FocalNet exibits similar behavior, proposing hierachical gated aggregation with locality perception and gated aggregation. So I think the proposed MogaNet has similar motivation and mechanism with FocalNet [1].\n\n- Moreover,\nThe proposed method is a variant of convolutional modulation, but lacks an in-depth discussion on differences with recent CNNs based on convolutional modulations, such as VAN [2], FocalNet [1], and Conv2Former [3]. \nBesides, VAN [2] and FocalNet [1] should be added in Figure 3 for a comprehensive analysis on the interaction strength.\n\n- Regarding to the Figure 7 of ablation study, I am confused that the main improvement is not from multi-order convolution or gate, which are claimed as major contributions of this work. \nInstead, the main improvement is from the CA, which is embedded in the feed-forward network. Note that other networks do not have CA in their feed-forward network, introducing somewhat unfair comparison.\nTherefore, I think the authors should better clarify the mechanism of the CA and claim it as the major contribution, not only emphasizing the proposed multi-order gated aggregation.\n\n\n[1] Focal Modulation Networks, NeurIPS 2022\n\n[2] Visual Attention Network, CVMJ 2023\n\n[3] A Simple Transformer-Style ConvNet for Visual Recognition, ArXiv 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Bq4JxLMBK", "reviewer": "Reviewer_Koyy", "summary": "This paper proposes MogaNet, a new form of attentional aggregation mechanism across spatial dimension and channel dimension. The main motivation for this new design is to force the network against learning its implicitly preferred extreme-order interactions, and instead to learn the mid-order interactions more easily. The paper presents empirical evidence that with the proposed spatial and channel aggregation modules, the network can score higher in learning the mid-order interactions as well as achieve state-of-the-art results on multiple computer vision tasks and benchmarks.", "questions": "see questions raised above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "## originality\nThis paper presents a **novel perspective** that we should design neural networks such that it can efficiently learn **multi-order** interactions, esp. the mid-order ones. Guided by this perspective, this paper proposes a new form of **attention** mechanism (Moga Block) for both spatial and channel aggregation. While the proposed Moga Block is **not** exactly of strong novelty, the lens through which the new design is investigated and measured is very **interesting and novel**.\n\n## quality & clarity\nThis paper is **excellently presented** and backed up with extensive experiments in both the main paper and the supplementary materials. The writing is precise and concise. The figure and table layout is well thought out.\n\n## significance\nWhile the claim on the benefit of learning multi-order interactions still need to be verified with time, I believe the **strong empirical performance** achieved by the new design is of strong significance already.", "weaknesses": "There lacks a **theoretical understanding** on why the proposed Moga Block can help facilitate the learning of more mid-order interactions. There also lacks a **theoretical understanding** on why more mid-order interactions is better for the computer vision tasks. What should the **best curve** for \"interaction strength of order\" look like? Should it be a horizontal line across all the interaction orders? (If not, why should we automatically believe that more mid-order interactions will be better?)\n\nFigure 7 shows the proposed \"Moga(.)\" module and \"CA(.)\" module are helping the model to learn more mid-order interactions. But it would be also very helpful to show how the **internal design** of \"Moga(.)\" and \"CA(.)\" modules affect the curve for \"Interaction Strength of Order\".  For example, why do we choose the \"Cl : Cm : Ch = 1:3:4\" (section 4.4)? Would different mix move the curve differently? Same question for the design in Figure 4(a) and 4(b), which sub-design is the most effective component in moving the curve?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VuhR28Jmp5", "reviewer": "Reviewer_tkKT", "summary": "This paper introduces a multi-order gated aggregation network, aiming to encode expressive interactions into ConvNets and increase the representation ability of ConvNets. Particularly, a multi-order spatial gated aggregation module is introduced to encode multi-order interactions of features, while a multi-order channel reallocation module is introduced to reduce information redundancy, which can enhance middle-order game-theoretic interactions. The experiments are conducted on several vision tasks, including image classification, object detection, instance and semantic segmentation, 2D and 3D pose estimation, and video prediction.", "questions": "Additional comments:\n\n-: The compared results show that performance gains of the proposed MogaNet over existing popular architectures is not significant. So could the authors show more advantages of the  MogaNet?\n\n-: Besides parameters and FLOPs, latency is more important for practice applications. So could the authors show some results on the latency (e.g., training time and inference time) of the MogaNet?\n\n-: There exist much more efficient and effective channel attention, besides SE. Therefore, the authors would better compare more efficient channel attention methods to verify the effectiveness of CA module.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+: The experiments are conducted on several vision tasks, and the results show the proposed networks are competitive to existing popular architectures. In my opinion, extensive experiments are main strength of this work.\n\n+: The overall architectures of this work are clearly descried, and seems to be easy implement.", "weaknesses": "-: The analysis on multi-order game-theoretic interaction encourage to propose the multi-order gated aggregation network. However, in my opinion, relationship between Sec. 3 (i.e., analysis) and Sec. 4 (implementation) seems a bit loose. Specifically, I have a bit doubt on why fine-grained local texture (low-order) and complex global shape (middle-order) can be instantiated by Conv1×1(·) and GAP(·) respectively. And why three different DWConv layers with dilation ratios can capture low, middle, and high-order interactions? What are close relationship between multi-order game-theoretic interaction and multi-order channel reallocation module? Therefore, could the authors give more detailed and rigorous correspondings between analysis in Sec. 3 and module design in Sec. 4?\n\n-: From the architecture perspective, the proposed MogaNet exploit multiple depth-wise convolutions for token mixing and channel attention-based FFN for channel mixing. The idea on multiple depth-wise convolutions for token mixing was used in RepLKNet, while channel attention-based FFN was explored in LocalViT and DHVT. So I have a bit doubt on technological novelty of core ideas on designing overall architectures of MogaNet. I suggest the authors can give more detailed analysis on novelty of MogaNet from the architecture perspective.\n\n[RepLKNet]: Scaling up your kernels to 31x31: Revisiting large kernel design in CNNs. In CVPR, 2022\n\n[LocalViT]: LocalViT: Bringing Locality to Vision Transformers. arXiv, 2021.\n\n[DHVT]: Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets. NeurIPS, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yNuwCmAo7", "reviewer": "Reviewer_XNzj", "summary": "The authors introduce a new family of pure CNN architectures, dubbed MogaNet, by analyzing the effects of different-order interactions in deep neural networks. They assert that MogaNet with two new modules for spatial and channel mixing improves middle-order interaction while suppressing extreme-order interaction, yielding promising results for visual recognition. Comparisons on image classification and downstream tasks show the effectiveness.", "questions": "Primary:\n- The perspective on multi-order interactions that serves as the foundation and impetus for this work is presented in [r1]. Analysis of the factors  why existing networks results in an inappropriate composition of local and global context is still lacking.\n- It is thought that the self-attention mechanism is more adaptable than typical gate attention. Why does ViT fail to learn middle-order interactions when self-attention is present?\n- For the purpose of capturing low-, middle-, and high-order interactions, Moga Block uses three parallel branches. To confirm that the model concentrates more on the middle-order as anticipated, it would be preferable to give the gate values (or attention map).\n- The substration $Y-GAP(Y)$, which \"forces the network against its implicitly preferred extreme-order interactions,\" is the main function of FD. Why does the CA block not include this operation?\n- Wrong markup in Fig.5, i.e., two $1:0:0$.\n\nOthers:\n- Will MogaNet be less robust because low-order interation \"represents common and widely shared local patterns with great robustness\"?\n- It would be better if figures following the order they appear in the main-body text.\n- It would be interesting to know how the losses (presented in [r1]) and MogaNet pitted against (or collaborate with) each other, because both supervision signal and structure design matter for deep neural networks.\n- The two expressions before and after Equation 1 contradict each other, i.e., \"in the same shape\" and \"downsampled features\".\n\nI would be glad to raise my rating if thoughtful responses are presented.\n\n[r1] H. Deng, Q. Ren, H. Zhang, and Q. Zhang, “Discovering and Explaining the Representation Bottleneck of DNNs,” in International Conference on Learning Representations (ICLR), 2022.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The impact of multi-order interaction in neural networks provided by previous work is a fascinating starting point for network architecture design.\n- Both quantitative and qualitative experiments demonstrate adequate research work.\n- Specific training parameters are provided, which are rather important for reproduction by the community.", "weaknesses": "The core idea of this work is hard to follow. Why the proposed network design is \"an appropriate composition of convolutional locality priors and global context aggregations\" is not entirely evident. In addition to the quantitative depiction in Fig. 3, qualitative analysis of the reasons why earlier network designs failed at this point would be extremely helpful in explaining this central argument. Please refer to the QUESTIONS section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)