HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
Authors: Yongming Rao
Wenliang Zhao
Yansong Tang
Jie Zhou
Ser-Nam Lim
Jiwen Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model Architectures
Dynamic Neural Networks
Vision Transformers
Convolutional Neural Networks

Abstract: Recent progress in vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework. We present the Recursive Gated Convolution ($\textit{g}^\textit{n}$Conv) that performs high-order spatial interactions with gated convolutions and recursive designs. The new operation is highly flexible and customizable, which is compatible with various variants of convolution and extends the two-order interactions in self-attention to arbitrary orders without introducing significant extra computation. $\textit{g}^\textit{n}$Conv can serve as a plug-and-play module to improve various vision Transformers and convolution-based models. Based on the operation, we construct a new family of generic vision backbones named HorNet. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation show HorNet outperform Swin Transformers and ConvNeXt by a significant margin with similar overall architecture and training configurations. HorNet also shows favorable scalability to more training data and larger model sizes. Apart from the effectiveness in visual encoders, we also show $\textit{g}^\textit{n}$Conv can be applied to task-specific decoders and consistently improve dense prediction performance with less computation. Our results demonstrate that $\textit{g}^\textit{n}$Conv can be a new basic module for visual modeling that effectively combines the merits of both vision Transformers and CNNs. Code is available at https://github.com/raoyongming/HorNet.

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
Yongming Raol
Wenliang Zhao
Yansong Tang'
Jie Zhou
Ser-Nam Lim
Jiwen Lu
TTsinghua University
M AI
Abstract
Recent progress vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, also efficiently implemented with a convolution- based framework. We present the Recursive Gated Convolution (g"Conv) that performs high-order spatial interac- tions with gated convolutions and recursive designs. The new operation is highly flexible and customizable, which is compatible with various variants of convolution and extends the two-order interactions in self-attention to arbitrary orders without introducing significant extra computation. g"Conv can serve as a plug-and-play module to improve various vision Transformers and convolution-based models. Based the operation, we construct new family of generic vision backbones named HorNet. Extensive experiments on ImageNet classification. COCO object detection and ADE20K semantic segmentation show HorNet outperform Swin Transformers and ConvNeXt by a significant margin with similar overall archi- tecture and training configurations. HorNet also shows favorable scalability more training data and a larger model size. Apart from the effectiveness in vi- sual encoders, we also show g"Conv can be applied to task-specific decoders and consistently improve dense prediction performance with less computation. Our results demonstrate that g"Conv can be a new basic module for visual modeling that effectively combines the merits of both vision Transformers and CNNs. Code is available https:/ / //gi thub. wwwmraooongming///aoooooomoc HorNNtt.
1 Introduction
Convolutional neural networks (CNN) have driven remarkable progress in deep learning and compu- tation vision since the introduction of AlexNet [31] in the last decade. There are quite few nice properties CNNs making them naturally suitable for wide range of vision applications. Transla- tion equivariance introduces useful inductive biases major vision tasks and enables transferabiliity across different input resolutions. The highly optimized implementation makes efficient on both high-performance GPUs and edge devices. The evolution of architectures [32, 31, 49, 50, 22, 24, 51] further increases its popularity on various vision tasks.
The emergence of Transformer-based architectures [16, 52, 42] greatly challenges the dominance of CNNs. By combining some successful designs in CNN architectures and the new self aelf-attention mechanism, vision Transformers have shown leading performance on various vision tasks such as image classification [12, 42, 48], object detection [70, 41], semantic segmentation [6, 8] and video understanding [64, 18]. What makes vision Transformers more powerful than CNNs? Some efforts have been made to improve the CNN architectures by learning from the new designs vision
Equal contribution. Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

MuI
MatMul
Mui
Softmax
MatMul
(a)
(b)
(d)
Figure 1: Illustration of our main idea. We show representative spatial modeling operations that perform difterent orders of interactions. In this paper, we focus studying explicit spatial interactions between a feature (red) and its neighboring region (light gray). (a) The standard convolution operation does not explicitly consider the spatial interaction, (b) Dynamic convolution [28, 4] and SF. [25] introduce the dynamic weights to improve the modeling power of convolutions with extra spatial interactions. (c) The self-attention operation [56] performs two-order spatial interactions with two successive matrix multiplications. (d) g (CConv realizes arbitrary-onder spatial interactions using highly efficient implementation with gated convolutions and recursive deigns.
Transformers. [43] presents a thorough study to adopt the meta archilecture of vision Transformer to improve CNNs and proposes to use a large 7x7 kernel to construct a modern CNN. [46] and [14] propose to use even larger kernels to learn long- oangrange relations with global filters and up to 31x31 convolutions, respectively. [20] shows that the input-adaptive weights play a key role in vision Transformers and achieve similar performance with Swin Transformers with dynamic convolutions [4, 28]. However, the effectiveness of dot-product self-attention in vision tasks has not been analyzed from the prospective of high-order spatial interactions.
While there exists complex and often high-order interactions between two spatial locations in a deep model due to the non-linearity, the success of self-attention and other dynamic networks suggests that the explicis and high-order spatial interactions introduced by the architectural designs are beneficial to improving the modeling power of vision models. As illustrated in Figure 1, the plain convolution operation does not explicitly consider the spatial interactions between a spatial location (i.e.. the red feature) and its neighboring region (i.e., the light gray region). Enhanced convolution operations like dynamic convolution [4, 28, 20] introduce explicit spatial interaction by generating dynamic weights. The dot-product self-attention operation in Transformers [56] consists of two successive spatial interactions by performing matrix multiplication among queries, keys and values. The trend of the basic operations for visual modeling indicates that the network capacity can be improved by increasing the order of spatial interactions.
In this paper, we summarize that the key ingredient behind the success of vision Transformers is the new way of spatial modeling with input-adaptive, long-range and high-order spatial interactions performed by the self-attention operation. While previous work has successfully migrated the meta architecture [43, 20, 46, 14], input-adaptive weight generation strategy [20] and large-range modeling ability [46, 14] of vision Transformers to CNN models, a higher-order spatial interaction mechanism has not been studied. We show that all the three key ingredients can be efficiently implemented using a convolution- based framework. We propose the Recursive Gated Convolution (g" (g"Conv) that performs high-order spatial interactions with gated convolutions and recursive deigns. Instead of simply imitating the successful designs in self-attention, g"Conv has several extra favorable properties: 1) Efficien. The convolution-bal based implementation avoids the quadratic complexity of self-attention. The design that progressively increases the channel width during performing spatial interactions also enables us to achieve higher-order interactions with bounded complexity; 2) Extendable. We extend the two-order interaction in self-attention to arbitrary orders to further improve the modeling power Since we do not make assumptions on the type of spatial convolution, g c"Conv is compatible with various kernel size and spatial mixing strategies like [46, 14]; 3)
gonn fully inherits the translation equivariance of the standard convolution, which introduces beneficial inductive biases to major vision tasks and avoids the asymmetry brought by local attention [42, 34].
Based on "Conv, we construct a new family of generic vision backbones named HorNet. We conduct extensive experiments on ImageNet classification [13], COCO object detection [38] and ADE20K semantic segmentation [71] to verify the effectiveness of our models. With the same 7x7 kernel/window and similar overall architecture and training configurations, HorNet outperforms Swin

---

and ConvNeXt by a large margin on all tasks at different levels of complexity. The gap can be further enlarged by using global kemel size [46]. HorNet also shows favorable scalability to more training data and larger model size, attaining 87.7% top-1 accuracy on ImageNet, 57.9% mloU on ADE20K val and 59.2% bounding box AP on COCO val with ImageNet-22K pre-training. Apart from applying g"Conv in visual encoders, we further test the generality of our designs on task-specific decoders. By adding gConv to the widely used feature fusion model FPN [36], we develop HorFPN to model the high-order spatial relationships of features from different hierarchical levels. We observe that HorFPN can also consistently improve various dense prediction models with lower computational costs. Our results demonstrate that g"Conv can be a promising alternative self-attention for visual modeling and effectively combine the merits of both vision Transformers and CNNs.
2 Related Work
Vision Transformers. The Transformer architecture [56] is originally designed for the natural language processing tasks. Since Dosovitskiy et al. [16] show that vision models constructed only by the Transformer blocks and a patch embedding layer can also achieve competitive performance to CNNs, many new models have been proposed to modify the Transformer-basee architecture and make it more suitable for various vision tasks [42, 58, 60, 9, 66, 55J. Different from the original designs in [16], state-of-the-art vision Transformers usually utilize a CNN-like hierarchical architecture and change the global self-attention among all patches to local self-attention to avoid the quadratic complexity. In this paper, we follow the overall architecture of the previous hierarchical vision Transformers [42] and replace the self-attention sub-layer with our proposed g"Conv fairly compare with the previous Transformer-basse models.
Convolution- models. Inspired by the recent success of vision Transformers, several papers propose to adopt the Transformer-style architecture and spatial convolutions with a large kernel size to improve the performance of CNNs. Han el al. [20] replace the window self-attention Swin Transformers with large-kernel dynamic convolutions and achieve better performance. GFNet [46] proposes to perform the global spatial interactions like vision Transformers with global filters in the frequency domain, which are equivalent to depth-wise convolutions with a global kernel size and circular padding. ConvNeXt [43] thoroughly analyzes the designs in recent vision Transformers and presents strong convolutional model with 7x7 depth-wise convolutions. RepLKNet [14] explores CNN models with very large kernels (up to 31 x11,,, showing good scalability as vision Transformers. VAN [19] and FocalNet [65] use gated convolutions to perform aduu-adaptive attention and adopts large-kemel dilated convolutions and multiple successive 3 convolutions respectively to produce the weights. Previous work focuses on the meta architecture [67], large-kernel designs and input- adaptive weights to improve CNNs by learning from vision Transformers. In this paper, we offer new perspective of high-order spatial attention analyze the merits of vision Transformers. We show that the proposed HorNet that combines the ad advantages of both CNNs and vision Transformers a better architecture for various vision tasks.
Hybrid models. Combining vision Transformers and CNNs to develop hybrid architectures is a new direction various visual recognition problems. Recently, several efforts have been made to integrate the two types of blocks into unified model with a sequential [12, 29, 68, 63] or parallel [45, 11] design. Many enhanced vision Transformers also use lightweight convolutions the basic building block to efficiently capture neighboring patterns [15, 60, 17] or relax the quadratic complexity of self-attention [9. 58, 18]. Different from these hybrid models, we aim to develop a self-attention free model while combining the favorable properties of both vision Transformers and CNNs.
3 Method
3.1 g"Conv: Recursive Gated Convolutions
In this section, we will present g" Conny, an efficient operation to achieve long-term and high-order spatial interactions. The coConv is built with standard convolutions, linear projections and element- wise multiplications, but has similar function of input-adaptive spatial mixing to self-attention.
Input-adaptive interactions with gated convolution. Recent success in vision Transformers mainly depends on the proper modeling of the spatial interactions in visual data. Unlike CNNs
3

---

that simply use the static convolution kernel to aggregate neighboring features, vision Transformers apply multi-head self-attention to dynamically generate the weights to mix spatial tokens. However, the quadratic complexity w.r.t. the input size of the self-attention largely hinders the application of vision Transformers, especially on downstream tasks including segmentation and detection where higher-resolution feature maps are required. In this work, instead of reducing the complexity of self-attention like previous methods [42, 9, 57], we seek a more efficient and effective way to perform spatial interactions with simple operations like convolution and fully-connected layers.
The basic operation of our method is the gated convolution (gConv). Let x E HW# xC' be the input feature, the output of the gated convolution =gConv(x) can be written as:
IPo
HwxC
96
HW x Pin (x) E #W22
(3.1)
P1 f(qo) O Pa E
R
HWxC y dout (P1) E IR HWC
where Gin, Pont are linear projection layers to perform channel mixing and f is a depth-wise con- volution. Note that pi (5.c) Ejen, w (i.c) (f.c) where Q is the local window centered at and represents the convolution weight of Therefore, the above formulation explicitly introduce interactions among the neighboring features p(d and ( through the element-wise multiplication. We consider the interaction in gConv as l-order interaction as each Po (1) has interacted with its
neighbor feature ( only once.
High-order interactions with recursive gating. After achieving efficient l-order spatial interactions with the gConv, we then design the g"Conv, a recursive gated convolution to further enhance the model capacity by introducing higher-order interactions. Formally, we first use Oin to obtain a set of projected features Po and {q: J
Po
XCo
HWWCC, 40
HWWxC.11 xxxx 4n-1
(x) E R HW x(Coo Ck),
(3.2)
We then perform the gated convolution recursively by
Pk+1 (gk) @ (Px)/aa
k= 0=0,
(3.3)
where we scale the output by 1/a to stabilize the training. {fx} are a set of depth- convolution layers and }gk} are used to match the dimension in different orders:
Identity, Linear Ck)
k=
(3.4)
1skgn-1.
Finally, feed the output of the last recursion step 9n the projection layer dout obtain the result of the g"Conv From the recursive formula Equation (3.3), it is easy to show that the interaction-order of P& will be increased by 1 after each step. As result, we see that the g"Conv achieves n-order spatial interactions. It is also worth noting that we need only single f to perform depth- wise convolution to the concatenation of the features {q:} together instead of computing the convolution in each recursive step as in Equation (3.3), which can further simplify the implementation and improve the efficiency on GPUs To ensure that the high-order interactions do not introduce too much computational overhead, we set the channel dimension in each order as:
C₂ = 2
0 S n 1.
(3.5)
This design indicates that perform the interactions coarse-to-fine manner, where lower orders are computed with fewer channels. Besides, the channel dimension of din (x) is exactly 20 and the total FLOPs can be strictly bounded even with n increasing. can be proved that (see Appendix A):
FLOPs(g ( Conv)= < HW HW C(2K2 + 11/3 x C C+22 2),
(3.6)
where K is the kernel size of the depth-wise convolution. Therefore, our g"Conv achieves high-order interactions with similar computational cost to a convolutional layer.
Long-term interactions with large kernel convolutions. Another difference between vision Transformers and conventional CNNs is the receptive field. Conventional CNNs [49, 22] often use 3x3 convolution through the whole network, while vision Transformers calculate self-attention on the whole feature maps [16, 52] inside relatively large local window (e.g., 7x7). The large

---

g"Conv
PyTorch-style code for g"Cony
Pioj
Class geecerim Modalle):
Mul
def __.ttt_.. dis, order, dutypa):
self.arder arder
Tayer Norm
Prof,
selt dizs leia // for zengefordee) aeverae
sell peol_in in Conr Zodan, welf decoos diccurtoonn lovll dimu), Uppe-dutppe) selt proje Leo Dinvōdtoel diselti solf dimettii)) 13 for a sangelorse (21) aslf pcoj_ jott Coav2didie, dia,
Muil
Medssalien
Proj, C72
Mul
g"Conv
fick
[632
del LorvartfoelI, ab:
solf pros_in(e)
DWCone, 2C-CM
torch aplitis, Coeld disa (0), sunfoslt dima)), dimri) = salf ounono(s)
Layer Norm
[XX 7
x_ a tanch sppissis, natt dtne, dtsel)
HorBlock
Proj 2C
x- =_ssee())
for in raage order -21:
x- seld prota [63 (_.iat((11))
return self proj_sux(0)
Figure 2: Overview of the basic building block in HorNet with g"Conv. We adopt the block design of Transformers [56] and replace the self-attention sub-layer with g g"Conv develop HorNet (left). We also provide the detailed implementation g C'Covv (middle) and the Pytorch-style code for an arbitrary order (right).
receptive field in vision Transformers makes easier to capture long-term dependencies, which is also recognized as one of the key advantages of vision Transformers. Inspired by this design, there are some efforts to introduce large kernel convolutions to CNNs recently [14, 43, 46J. To make our g" g'Conv capable of capturing long-term interactions, we adopt two implementations for the depth-wise convolution f:
7x7 Convolution. 7x7 the default window/kereel size of Swin Transformers [42] and ConvNext [43]. Studies in [43] show that the kernel size produces good performance on ImageNet classification and various downstream tasks. We follow this configuration to fairly compare with representative work of vision Transformers and modern CNNs.
Global Filter (GF). The GF layer [46] multiplies the frequency domain features with learnable global filters, which is equivalent to a convolution the spatial domain with a global kernel size and circular padding. We use a modified version of the GF layer by processing half of the channels with the global filter and the other half with 3x3 depth-wise convolutions and only use GF layers in late stages to preserve more local details.
Spatial interactions in vision models. We review some representative vision model designs from the perspective of spatial interactions, shown in Figure Specifically, we interested in the interactions between feature x; and its neighboring feature X,, E Ωg. By using the tool designed for explaining the interaction effect (IE) in [33, 1], we provide an intuitive analysis of the order of explicit spatial interactions in Appendix B. Our analysis reveals a key difference between vision Transformers and previous architectures from new view, i.e., vision Transformers have higher-order spatial interactions in each basic block. The result inspires us explore an architecture that can realize more efficient and effective spatial interactions with more than two orders. As discussed above, our proposed g"Conv can achieve arbitrary-order interactions with bounded complexity. It is also worth noting that similar to other scaling factors deep models like width [69] and depth [22], simply increasing the order of spatial interactions without considering the overall model capacity will not lead to a good trade-off [51]. In this paper, we focus on developing stronger visual modeling architecture based on the analysis of the spatial interaction orders of well well-designed models. We believe a more thorough and formal discussion on the high-order spatial interactions can be an important future direction.
Relation to dot-product sell-attention. Although the computation of our g"Conv largely differs from dot-product self-attention, we will show that g"Conv also accomplishes the goal of input- adaptive spatial mixing. Let M be the attention matrix obtained by multi-head self-attention (MHSA), we write M as (mzz) since the mixing weight may vary across the channels. The spatial mixing result (before the final channel mixing projection) of the c-th channel at location i is
(3.c) THHS
Σ ((d)
Σ
Σ
(c/.c) m w (.e'))
(3.7)
jEf,
JEL, d==
5

---

where wy is the weight of the V-projection layer. Note that may obtained by the dot-product operation contains 1-order interaction. On the other hand, the output of our g"Conv (before the oous) can be written as
C h
(1.c) I CoCon -p/-)) Σ C sEA
(1.c) (c.0) 2(10) w Pmn
EE JEN,
(c.0) o 201d)
w 1,1
(3.8)
where wn-1 is the convolutional weight for fn-1, Wo... is the linear weight of Oin- and gn-1 = In-1 (P=-) is projection of Pn-1- From the formulation in Equation (3.8) we find our g" g"Conv also achieves input-adaptive spatial mixing with {hī}} } as the weights. Observing that his is computed from Pa-1 which contains n order interactions, we can regard our g"Conv an extension of the self-attention in terms of the order of the spatial mixing weight. Therefore, our g" g"Conv can better model more complex spatial interactions.
The details of gCConv and our implementation are summarized in Figure 2.
3.2 Model Architectures
HorNet.
The g"Conv can be drop-in replacement of the spatial mixing layer in vision Transform-
ers [52, 42] or modern CNNs [43]. We follow the same meta-architecture as [56, 42] to construct HorNet, where the basic block contains a spatial mixing layer and a fordafo network (FFN). Depending on the model size and the implementation of the depth-wise convolution f& in our g"Conv, we have two series of model variants named HorNet-T///B/Lxx and HorNet-T/S/B/LGF We con- sider the popular Swin Transformer [42] and ConvNeXt [43] as the vision Transformer and CNN baselines since our models are implemented based on a convolution- based framework while having high-order interactions like vision Transformers. To fairly compare with the baselines, we directly follow the number of blocks of Swin Transformers- Sittfformers-S//BLLLL [42] but insert an extra block to the stage to make the overall complexity close, resulting in [2, 3, 18, 2] blocks in each stage in all of the model variants. We simply adjust the base number channels C' to construct models with different sizes and set the number of channels in 4 stages as 2C, 4C, 8C] following common practice. We use C' = 64, 96, 128, 192 for HorNet-T/S/BBL, respectively. We set the interaction orders (i.e., the n in g" g"Conv) for each stage 2,3,4,5 by default, such that the channels of the coarsest order Co is the same across different stages.
HorFPN. Apart from using g"Conv in visual encoders, find our g"Conv can be enhanced
alternative for standard convolution that considers higher-order spatial interactions in a wide range of convolution-base models. Thus, we replace spatial convolutions for feature fusion in the FPN [37] with our g" g'Cony to improve spatial interactions for downstream tasks. Specifically, we add our g"Conv after the fusion of features from different pyramid levels. For object detection, we replace the 3x3 convolution after the top-down pathway with the g"Conv in each level. For semantic segmentation, we simply replace the 3 convolution after the concatenation of the multi-level feature maps with g" g"Conv since the final results are directly predicted from this concatenated feature. We also have two implementations called HorFPN7x7 and HorFPNGF decided by the choice of fx-
Experiments
We conduct extensive experiments to verify the effectiveness of our method. We present the main results on ImageNet [13] and compare them with various architectures. We also test our models on the downstream dense prediction tasks
commonly used semantic segmentation benchmark ADE20K [71] and object detection dataset COCO [38]. Lastly, we provide ablation studies of our designs and analyze the effectiveness of g" goConv a wide range of models.
4.1 ImageNet Classification
Setups. We conduct image classification experiments on the widely used ImageNet [13] dataset. We train our HorNet-T/S/B models using the standard ImageNet- 1K dataset following common practice. To fairly compare with previous work, we directly use the training configurations of [43, 42, 52] to train our models. We train the models for 300 epochs with 224 x 224 input. To evaluate the scaling ability of our designs, we further train the HorNet-L. models on the ImageNet-22K dataset that contains over 10x images and more categories. We follow previous practice [42, 43] to train models for 90 epochs and use similar data augmentation strategy ImageNet- 1K experiments.

---

Table 1: ImageNet classification results. We compare our models with sfate-of-the-art vision Transformers and CNNs that have comparable FLOPs and parameters. We report the top-1 accuracy on the validation set of ImageNet as well as the number of parameters and FLOPs. We also show the improvements over Swin Trasnformers that have similar overall architectures and training configurations to models. "1384" indicates that the model fine-tuned on x34 images for 30 epochs. Our models are highlighted in gray.
Image Size
Params FLOPs Top-1
Image
Params FLOPs Top-I
Model
(M)
(G)
Acc. (%)
Model
Size
(M)
(G)
Acc. (%)
ImageNet-JA IK trained models
ImageNer-11 trained models (foe-tumed at 384x 384)
EfficieniNee [4[5]]] [51]
380
19
4.2
82.9
Swin-B1384 [42]
381
89
47.1
84.5
EfficienNNetBB5 [5]]
456
30
83.6
ConyNeext-B |431
3842
85.1,000
EfficicnNNes-66 [5]]
528
43
19.0
84.0
HouNet Byst 1384
384
87
45,8
85. 85.3
EllicieaiNetV2-S
300F
HorNet-Reard Reakett--r333
384
92
85.6.i D
Repl.KNet.311 [14]
224
79
15.3
83.5
VAN-B [19]
224
82.8
ImageNNet-228 trained models (fine-tuned to ImangeNet-1K
CSWin-T1151
VAN-L [19]
221a
2242
45
4.3
9.0
83.9
82.7
R-152x4 [30]
R-101x3 [30]
3842
480²
937
388
840.5
204.6
84.4
85.4
CSWin-S [155]
CSWin-R [15]
224
224
78
15.0
83.6
84.2
VIT-L/16 [16]
VIT-B/16 [16]
3842
3812
87
305
191.1
84.0
85.2
Swin.T [42]
224
[fficienNNevV2.L[[[1]]]
350
86.8
ConvNeXt-T [43]
224
29
4.5
82. 1ua.
CSWin-L
384
173
96.8
87.5
HorNet-Trxe
221a
4.0
82.8 82.8
SwinV2-L [41]
3812
197
115.4
87.6
HorNet-TTar
224
83.0
RepLKNet-31L114
3842
172
96.0
86.6
Swin-S [42]
221a
50
8.7
83.0
Swin-T. [42]
224
197
34.5
86.3
ConvNeXi:S
224
ConyNeXt-L [43]
224"
34.4
HoeNet-S7zs7
2244
50
8.8
8/18
HorNet-Lrst
2212
195
34,8
86.8
HorNet-Sca
224
84.0 m
HorNet-Lorr
224
67.0
Swin-B [42]
2242
89
15.4
83.5
Swin-L1384 [42]
3842
197
103.9
87.3
ConvNeXitB [43]
224
88
15.4
83.8
Con-NeXi-L.138. [43]
3842
198
101.0
07.5
HorNat-Brx?
224"
15.6
84.2
HorNet-Laxx133k
3842
195
102.3
87.6,003)
HorNet-Bap
221"
88
15.5
84.3,wm0
HorNet-Loy 1384
381
202
101 8
87.7 67.7
We fine-tune the models pre-trained on ImageNet-22K or at the 224x224 resolution to ImageNet- K or/and 384x384 resolution for 30 epochs following [43]. When adapting the ImageNet-222K models to ImageNet-IK, we initialize the classifier with the pre-trained class centers to stabilize the training process. More details can be found Appendix C.
Results. The results of our ImageNet classification experiments are summarized in Table 1. We see that our models achieve very competitive performance with state-of-the-ar vision Transformers and CNNs. Notably, HorNet surpasses Swin Transformers and ConvNeXt which have similar overall architectures and training configurations by a healthy margin on various model sizes and settings. Our models also generalize well larger image resolution, larger model sizes and more training data. These results clearly demonstrate the effectiveness and generality of our designs.
4.2 Dense Prediction Tasks
HorNet for semantic segmentation. We evaluate our HorNet for semantic segmentation task on ADE20K [71] dataset using the commonly used UperNet [62] framework. All the models are trained for 160k iterations using AdamW [44] optimizer with a global batch size of 16. The image size during training is 512 x 512 for ImagNet-IP Ik (HorNet-T/S/B) pre-trained models and 640 x 640 for the ImageNet-22K pre-trained models (HorNet-L). The results are summarized in the left part of Table 2, where we report both the single-scale (SS) and multi-scale (MS) mIoU on the validation set. Both HorNet,x7 and HorNetGr models outperform Swin [42] and ConvNeXt [43] models with similar model sizes and FLOPs. Specifically, HorNetgr models achieve better results than HorNet 7 and ConvNeXt series by large margins in single-scale mloU, indicating the global interactions captured by the global filter are helpful for semantic segmentation. Notably, we find both our HorNet-Lax? and HorNet-L-ar even outperform ConyNeXt-XI. with 25% fewer FLOPPs These results clearly demonstrate the effectiveness and scalability of our HorNet on semantic segmentation.
HorNet for object detection. We also evaluate our models on the COCO [38] dataset. We adopt the cascade Mask R-CNN framework [21, 2] to perform object detection and instance segmentation using HorNet-T/S/B/L backbones. Following Swin [42] and ConvNeXt [43]. we use 3x schedule with multi-scale training. The right part of Table compares the box AP and mask AP of our HorNet models and Swin/ConnNeXX models. Similarly, we show our HorNet models achieve consistently and significantly better performance than the Swin/ConvNeXt counterparts, in both box AP and mask AP. The HorNetge series obtain +1.2~2.0 box AP and +1.0~1.9 1.9 mask AP compared with ConyNeXt.
7

---

Table 2: Object detection and semantic segmentation results with different backbones. for semantic segmentation and Cascade Mask R-CNN [2] for object detection. indicates that the model is
UperNet [62]
pre-trained on ImageNet-22K, For semantic segmentation, we report both single-scale (SS) and multi-scale (MS)
mloU. The FLOPs calculated with image size (2048, 512) for ImageNet IK pre-trate rraitraine models and (2560, 640) for ImageNet-22K pre-traing trained models. For object detection, we report the box AP and mask FLOPs are measured on input sizes (1280, 800). Our models are highlighted in gray.
Semantic Segmentation with UperNer 160K
Object Detection with Cascade Mask R CNV
mloU"
mloUpu
Parums
FLOPs
Aplus
Apuaak
Params
FLOPs
Swin-T
61M
745G
46.0
46.7
60M
939G
50.4
43.7
86M
741G
HorNet-Tix?
48.1
48.9
52M
926G
$1.7
44.8
80M
730G
HorNet-Tor
49.2
49.3
55M
924G
52.4
45.6
80M
728G
Swin-S [42]
47.6
49.5
8IM
1038G
51.9
45.0
107M
838G
ConvNoXt-SS [43]
48.7
49.6
82M
1027G
51.9
45.0
108M
827G
HorNer-Srs7
49.2
49.8
8IM
1030G
52.7
45.6
107M
830G
HorNot-Suz
50.0
85M
53.3
827G
Swin [42]
48.1
49.7
121M
1188G
$1.9
45.0
145M
982G
ConvNeXXt-B
49.1
49.9
122M
1170G
52.7
45.6
146M
9646
HooNer--rir B7
50.0
50.5
121M
1174G
53.3
46.1
144M
969G
HorNet-Bap
50,5
50.9
126M
1171G
54,0
46,9
146M
965G
Swin-L' [42]
52.1
53.5
234M
2468G
53,9
46.7
253M
1382G
ConvNeXXt-L 143]
53.2
53.7
235M
2458G
54.8
47.6
255M
1354G
Con-NoXt-XL [43)
53.6
54.0
391M
3335G
55.2
47.7
407M
1898G
HorNot-L
2473G
48.0
1363G
HorNes-L
55.0
55.2
239M
2465G
56.0
48.6
259M
1358G
Table 3: Comparisons of HorFPN with standard FPN different backbones. We UperNet 160K and Mask R-CNN 1x schedule semantic segmentation and object detection, respectively. We find our HorFPN consistently outperforms standard FPN with various of backbones on both the two tasks.
Fasion
Semantic Segmentation with UperNer 160K
Object Detection with Mark R-CNN Ix
Module
mloU
mlo
Params
FLOPS
Aphoc
Apraok
Params
FLOPs
FPN
40.7
38.2
34.7
2600
ResNet-50 [22]
HorFPNex7
41.8
44.1
60M
499G
38.7
35.1
43M
226G
HorFPNGF
43.2
44.5
60M
497G
39.1
35.5
43M
224G
EPN [37]
RosNct-101 [22]
HorkNNest
44.1
45.5
79M
577G
40.3
36.4
62M
302G
HorFFNGr
44.5
46.4
79M
574G
40.5
36.7
62M
3000
FPN [37]
47.6
49.5
8IM
1038G
45.5
40.9
69M
354G
Swin-S [=2]
HorTPN
48.0
49.2
74M
$80G
46.3
41.1
68M
325G
HorkPNoP
49.0
49.9
75M
46.8
41.9
69M
323G
PPN [3t]
49.2
49.8
81M
1030G
47.1
42.2
69M
351G
HorNet-S
HorFPNes?
49.4
50.1
74M
577G
47.4
42.3
68M
322G
HorFPNor
Again, our large model HorNet-L7x7 and HorNet;r can outperform ConvNeXt-XLL which further validates the favorable transferability with larger model size and larger pre-trained dataset.
HorFPN for dense prediction. We now show another application of the proposed g"Conv, iee,, to serve as a better fusion module that better capture the higher-orda interactions among different levels of features in dense prediction tasks. Specifically, directly modify the FPN [37] de-
Table 4: Object detection results with recent state-of-the-art frameworks. We report the single-scale Apbox and Apmask the validation of COCO. Our models are highlighted in gray.
Backbone
Framework
Aphox
Apmask
Swin-L [42]
HTC++ [3]
57.1
49.5
ViT-Adapter-L [5]
HTC++
57.9
S0.2
HorNet-LaF
HTC++ [3]
58.1
50,5
Swin-L [42]
DINO [70]
58.5
scribed Section 3.2 UperNet [62] and Mask R-CNN [21] for semantic segmentation and object detection, re-
HorNet-L-gg
[70]
59.2
Table Semantic Segmentation results with recent state-of-the- art frameworks. We report the single-scale (SS) and multi-scale (MS) mloU on the validation set of ADE20K. Our models are highlighted in gray.
spectively. We show the results Ta- ble 3, where we compare the perfor- mance our HorFPN and standard FPN on different backbones includ- ing ResNet-50/101 [22], Swin-S [42] and HorNet-S7x7z For semantic seg- mentation, we find our HorFPN can
Backbone
Framework
mloU
mloU
Swin-L [42]
Mask2Former [7]
$6.1
$7.3
Swin-L-FaPN [27]
Mask2Former [7]
56.4
$7.7
HozNet-Lag
Mask2Former
57.5
57.9
significantly reduce the FLOPs (~50%) while achieving better validation mloU. For object detection,

---

Table Ablation study and results of applying g CoCo other models/operations. We provide the ablation study of our designs in (a). [*] indicates the baseline of our model. The baseline and our final models are highlighted in gray. In (b) and (c), we apply the proposed g g"Conv to isotropic models that have similar level of complexity with ViT/DeiTTSS [16, 52] and other spatial mixing operations including the 3x3 depth-wise convolution and 3x3 pooling used in [67].
(a) Ablation study.
(b) Results on isotropic models.
Model
Params
FLOPs
Acc. (1%)
Model
FLOPs
Ace. (%)
Swin-T [42]
28M
4.5G
81.3
DeiT-S [52]
4.6G
79.8
Self-Atiention DWConvrx7 [+]
29M
4.5G
81.2
ConvNeXI-S (iso.) [43]
4.3G
79.7
SE [25]
30M
4.5G
81.5
HorNet-Srxr (iso.)
4.5G
80.6
SE + Coo-
28M
4.3G
81.7(w05)
HorNet-SaF (iso.)
4.5G
81.0
Cony
28M
4.3G
82.2
Conv
28M
4.3G
82.5
(c) g'Conv for other operations.
Conv
28M
4.3G
82.5
Couv
28M
4.3G
82.5
Model
FLOPs
Acc. (%)
8
122345)
28M
4.3G
82.6,114)
DWConvaxa
4.0G
80.7
Deeper Narrower (orNet--Txr)
22M
4.0G
82.8;-14)
& &"Convaxs
3.9G
82.1
Global Filters [46] (HorNet-Trar)
23M
3.9G
83.0,-18:
Pool [67]
3.9G
78.1
ConvNeXt [43]
28M
4.5G
82.1
poon
3.8G
79.3
(a)
64.5
[b]
Horooaa
[c)
Not
ImageNat: Top-1 Acc
$4.0
Con-Nott
64.5
Convittia
54.0
Co Naxt-
Swin
Convect: 5
Swh-3
Vel-T
NotWet-T
Swin-5
HorNet."
Swin
82.5
22.0
Convtext:T
63.5
$2.0
ConsNoot:
Swin
Conwieot
CanaNeX:
11.
Horlet
Params (M)
FLOPs
Latency (ms)
Figure 3: Comparisons of trade-offs of Swin, ConvNeXt and HorNet. We compare the trade-offs of the models via the top-1 accuracy on ImageNet w.r.t. (a) number of parameters; (b) FLOPs; (c) latency. The latency measured with single NVIDIA RTX 3090 GPU with batch size of 128.
our HorFPN can also outperform standard FPN in terms of both box AP and mask AP on different backbones with about 30G fewer FLOPs. Besides, we observe that the HorFPNGF is consistently better than HorFPN7x7. indicating that global interactions are also important when fusing hierarchical features.
Results with state-of-the-art frameworks. conduct experiments to combine our large HorNet model with recent state-of-the-art dense prediction frameworks including HTC++ [3], DINO [70] and Mask2Former [7]. For HTC++ and DINO, we train our models on COCO for 36 epochs (3x schedule) and does not introduce extra pre-training data like Object365 in [70]. We report the single-scale performance on the validation set and compared with several state-of-the the-art methods in Table 4. For Mask2Former, train our models on ADE20K with 640 x 640. We report the mloU of both single-scale and multi-scale testing on the validation set in Table 5.
further show the effectiveness our backbone, we
4.3 Analysis
Ablation study. We provide detailed ablation studies of the g"Conv and our HorNet in Table 6. We first study the model designs of our HorNet in Table 6a. Our baseline ([$]) is obtained by simply replacing the self-attention with 7x7 depth-wise convolution in Swin-T [42]. We first show that both SE [25] and our g"Conv with n (g...I...) Conv) can improve over the baseline model [*], and gl (1..I.1)) Conv is slightly better. We then perform ablations on the interaction order n for each stage and find: (1) if n is shared across the 4 stages, the accuracy will increase with larger n but saturate at 82.5 when n = 4: (2) progressively increased order (g( (onv)) can further improve the accuracy. Our final models are built on g 122.3.45) Conv by adjusting the depth and width of the networks (HorNet-T7xx7) and applying Global Filter [46] for the depth-wise convolution (HorNet-Tor). These results clearly show that our g" "onn an efficient and extendable operation that can better capture high-order spatial interactions than both self- self-attention and depth-wise convolution.

---

Figure 4:
Visualization of the adaptive weights generated by g 2'Conv. We see that the spatial mixing weights of our g" CCCnyv are adaptive both to input samples and spatial locations, which further indicates that g d'Conv shares these two desirable characteristics with the self-attention operation.
g" goConv for isotropic models. We also evaluate g goConv on isotropic architectures (with constant spatial resolutions). We replace the self-attention in DeîT-S [52] with our g"Conv and adjust the number of blocks to 13 to obtain the isotropic HorNet-S7x and HorNet-Ser- We compare DeiT-S, isotropic ConvNeXt-S and isotropic HorNet-S in Table 6b. While isotropic ConvNeXt-S cannot improve DeïT-S, our isotropic HorNet surpasses DeiT-S by large margin. These results indicate that our g Conv can better realize the functions of self-attention compared to plain convolutions and have better ability to model the complex spatial interactions.
g" g"Cony for other operations. To further demonstrate the universality of g"Conv, we use 3x3 depth-wise convolution and 3 pooling [67] as the basic operation in the g"Conv. The results in Table 6c show that g "Conv can also improve these two operations by large margins, indicating our g" goConv is potentially more powerful when equipped with some better basic operations.
trade-offs. We visualize accuracy-come complexxyyy trade-offs of Swin, ConvNeXt and HorNet series in Figure 3. For fair comparisons, we fix the input image size to 224 224 and use HorNet,xr 7 such that all the compared models are based on 7x7 local window. We see HorNet can achieve better trade-offs than the representative vision Transformers and modern CNNs with regards to model size, FLOPs and GPU latency.
Accuracy-comples
Visualization. We provide some visualizations of the adaptive weights learned by g"Conv in
Figure 4. For each sample, we show the value of с E h (see Equation (3.8) or the definition of (h)) for two random spatial locations from layer {I. 3, 5, 7, 8. 12] of the isotropic HorNet-S model. Figure 4 demonstrates that the spatial mixing weights of our g"Conv are adaptive both to input samples and spatial locations, which further indicates that g"Conv shares these desirable characteristics with the self-attention operation.
Limitations. While HorNet shows better overall latency-accuracy trade-offs, we notice that HorNet is slower than ConyNeXt with similar FLOOPs on GPU, which may be caused by the more complex designs to perform the high-order interactions. We think that developing a more hardware-friendle operation for high-order spatial interactions is an interesting future direction improve our work.
Conclusion
We have presented the Recursive Gated Convolution (g"Conv) that performs efficient, extendable, and translation-cquiivrraat high-order spatial interactions with gated convolutions and recursive deigns. g" g"Conv can serve as a drop-in replace of the spatial mixing layer in various vision Transformers and convolution-basee models. Based the operation, we have constructed new family of generic vision backbones HorNet. Extensive experiments demonstrate the effectiveness of goCony and HorNet on commonly used visual recognition benchmarks. We hope our attempt can inspire future work to further explore the high-order spatial interactions in vision models.
Acknowledgments
Jiwen was supported in part by the National Key Research and Development Program China under Grant 2017 the National Natural Science Foundation of China under Grant 621 25603 and Grant U1813218, and grant from the Beijing Academy of Artificial Intelligence (BAAD).
10

---

References
[1] Chunrong Ai and Edward C. Norton. Interaction terms in logit and probit models. Economics letters,
80(1):123-129, 2003. 5, 15
[2] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: Delving into high quality object detection. In CVPR, pages 6154 6162, 2018. 7,8, 17
[3] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, pages
4974-4983, 2019. 8,
[4] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention
convolution kernels. In CVPR, pages 11030-11039, 2020. 2
[5] Zhe Chen, Yuchen Duan. Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.086 00854444, 2022.
[6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, Masked- attention mask transformer for universal image segmentation. arXiv preprint arXiv: 12.00557,, 2021.
[7] Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, pages 1290-1299, 2022. 8,
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 34, 2021.
[9] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua
Shen. Twins: Revisiting the design of spatial attention in vision transformers. NewrIPS, 34, 2021. 3,
[10] Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPRW, pages 702-703, 2020. 16
[11] Yutao Cui, Jiang Cheng, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. CVPR, 2022. 3
[12] Zihang Dai, Hanxiao Liu, Quee V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurlPS, 34:3965-39777, 2021. 1,3
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, pages 248-255, 2009. 2, 6. 16, 17
[14] Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cans. CVPR, 2022. 2, 3,7
[15] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. CVPR, 2022. 3,7
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint
arXiv:2010.11929, 2020. 4.7,9
[17] Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In /CML, pages 2296-2296. 2021. 3
[18] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6824-6835, 2021. 1,3
[19] Meng- Hao Guo. Cheng Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.09744 09044. 2022. 3.7
Meng-Hao
[20] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXiv preprint
vrXiv:2106.000 04263, 2021. 2,
[21] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 7,8
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016. 1, 5, 16
[23] Sepp Hochreiter and Jürgen Schmidhuber. Long short-tern toot-ttrrm memory. Neural computation. (83):173-111 17800 1997. 16
11

---

[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint urXiv:/104.044
2017.
[25] Jie Hu, Li Shen, and Gang Sun. Squeeze and and excitation networks. In CVPR, pages 7132-7141, 7141, 2018. 2, 9,
18
[26] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In ECCV, pages 646-661, 2016. 17
[27] Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. Fapu: Feature- aligged pyramid network for dense image prediction. ICCV, pages 864-873, 2021.
[28] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. NeurlPS, 29,
2016. 2
[29] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Aaran Wang, and Jiashi Feng. Token labeling: Training 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint arXiv:2104 10858, 2021. 3
[30] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, pages 491-507. Springer,
2020. 7
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NewrlPS, 25:1097-1105, 2012. 1
[32] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-555 1989.
[33] Samuel Lerman. Charles Venuto, Henry Kautz, and Chenliang Xu. Explaining local, global, and higher- order interactions in deep learning. ICCV, pages 1224-1233, 2021. S, 15
[34] Yanghao Li. Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection, arXiv preprint arXiv:2203. (2220366522 /6527, 2022. 2
[35] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, pages 4804-4814, 2022. 17, 18
[36] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 21 2117-21 2017. 3
[37] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125 2017, 6,
[38] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014. 6,7
[39] Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mlps. NeurlPS, 34:92044499999 9214499211122 2021. 16
[40] Kai Liu, Tianyi Wu, Cong Liu, and Guodong Guo. Dynamic group transformer: A general vision transformer backbone with dynamic group attention. IJAAI, 2022. 18
[41] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.00 10.08883, 2021. 1,7
[42] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 2021. 1, 2, 3, 4,5, 5, 6, 7, 8, 9, 16, 17
[43] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. CVPR, 2022. 2, 5, 7, 8, 16, 17
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv: 2017. 7, 17
[45] Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. arXiv preprint arXiv:2111.144 145556, 2021. 3
[46] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In NeurIPS, 2021. 2, 5,9
[47] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Mann Manor. Imagenet-211 pretraining for the masses, arXiv:2104. 10972, 2021. 17
12

---

[48] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. NeurlPS, 34, 2021.
[49]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409 1506. 2014. 1,4
[50] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet. Scott Reed, Dragomir Anguelov. Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages
1-9, 2015. 1
[51] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114, 14, 2019. 5,
[52] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint
urXiv:2012.1287,, 2020. 1, 4, 6, 6,9,10 10
[53] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.1 17239, 2021. 16
[54] Zhengzhong Tu, Hossein Talebi, Han Zhang. Feng Yang. Peyman Milanfar. Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In CVPR, pages 5769-5780, 2022. 16
[55] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang. Peyman Milanfar, Alan Bovik, and Yinxiao Li.
Maxvit: Multi-axis vision transformer. ECCV, 2022. 3
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pages 5998-6008, 2017. 2, 3, 5, 6
[57] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear 2020.
complexity. arXiv preprint arXiv:2006.044
[58] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[59] Yan Wang, Lingxi Xie, Chenxi Liu, Siyuan Qiao. Ya Zhang, Wenjun Zhang, Qi Tian, and Alan L. Yuille. SORT: second-order response transform for visual recognition. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22 2017 pages 1368-1377 IEEE Computer Society, 2017. 16
[60] Haiping Wa, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. arXiv preprint arXiv:2103.15808 2021.
[61] Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: A general vision transformer backbone with pale-shaped attention. In AAAI, volume 36, pages 2731-2739) 2022. 18
[62] Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018. 7, 8, 17
[63] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. NeurlPS, 34:30392-3000 30400,, 2021. 3
[64] Shen Yan, Xuchan Xiong, Anurag Arna,, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. arXiv preprint arXiv:220J.042 2022.
[65] Jianwei Yang. Chunyuan Li. and Jianfeng Gao. Focal modulation networks. arXiv preprint
arXiv:2203. 2022.
[66] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers, NeurTPS, 34, 2021. 3
[67] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint axXiv:2111.11421 2021. 3, 9. 10
[68] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv:2101. 2021, 3
[69] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1605.0065 2016. 5
[70] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Hcung- Ycung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03661 2022. 1, 8,9
[71] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba Scene parsing through ade20k dataset, In CVPR, pages 633 641, 2017. 2, 6,
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See our analysis in Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We propose general framework for visual recognition. Our method not for specific applications, which does not directly involve societal issues.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We follow the common practice in previous papers, where they didn't 't report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See implementation details provided in the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "K0XufsrXDBo", "reviewer": "Reviewer_j8qU", "summary": "This paper proposes a new convolution-based deep neural architecture named $g^nConv$. Its essential operation is the gated convolution which multiplies the output of linear projection layers and depth-wise convolution layers. It also introduces high-order interactions with recursive gating. The authors conduct various experiments to show the effectiveness of the proposed results, including ImageNet classification, COCO object detection, and ADE20K semantic segmentation.", "questions": "1. It would be better if the authors could theoretically analyze why High-order interactions can improve network performance.\n2. The authors are suggested to compare with more recent works and evaluate the throughput of each method.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This is a well-written paper with carefully designed experiments.\n2. This paper combines several deep learning techniques in an interesting way to get a novel architecture for image classification, detection, and segmentation.\n\nWeaknesses:\n1. The experimental result section is weak. The authors did not compare their proposed method with the state-of-the-art methods. The authors are suggested to add more recent works for comparison, e.g., Dynamic Group Transformer [Liu etal, IJCAI 2022], MViTv2 [Li etal, CVPR 2022], Pale Transformer [Wu etal, AAAI 2022]\n2. The authors did not compare the throughput of each method. Since the proposed method involves a lot of small matrix multiplications that cannot be parallelized, and depth-wise convolutions, I guess this method should be slower than the other methods, even though their FLOPs are similar.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IX5190b0vH_", "reviewer": "Reviewer_3Snz", "summary": "This paper proposes a new operation (gnConv) and network (HorNet) to perform computer vision tasks. Motivated by the success of vision Transformers, the key idea is to build an architecture that has input-adaptive, long-range and high-order spatial interactions. The authors did so by proposing a recursive form of gated convolution, which has the capability of going even higher order explicit spatial interaction than 2. On well-established computer vision benchmarks such as image classification on ImageNet, semantic segmentation on ADE20K, object detection on COCO, the authors showed consistent improvement over Swin Transformer and ConvNeXt. \n\nPOST-REBUTTAL UPDATE:\n\nI have read the authors' rebuttal. While I am a bit disappointed by the answer to my Q2, I am satisfied with the rebuttal overall and the paper. In my opinion / experience, in this day and age, it is not an easy feat to achieve the consistent improvement as the authors have shown in this paper, and I have decided to maintain my original rating of 8.", "questions": "L316 states that HorNet currently runs slower than ConvNeXt. It would be good to clarify how many times slower. \n\nHorNet seems to outperform Swin Transformer and ConvNeXt pretty consistently. But have the authors considered comparison against models with better performance on ImageNet, such as CoAtNet [8]? Related, what are ways that will allow HorNet to achieve state-of-the-art on ImageNet classification?\n\nFinally, I think the higher-order interaction really originated from the gating mechanisms in LSTM. In vision, \"SORT: Second-Order Response Transform for Visual Recognition\" from ICCV 2017 is one of the earliest works. It would be great to add proper discussion on these.", "limitations": "I am reasonably happy with the Limitations section of the paper from L315 to L318. As I wrote earlier, it would be good to be more transparent and specific. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed architecture is very simple and effective. Higher-order interactions have been desired but have never been prevalent, possibly due to gradient problems. But the authors seem to have got it to work quite well.\n- Writing is very clear. \n- Experiments are thorough, and solid improvements are observed throughout.\n\nWeaknesses:\n- No major weaknesses. I just have a couple of questions.\n\nOverall, I think the originality, quality, clarity, and significance are all high.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-IuYWTa3tPR", "reviewer": "Reviewer_dnYU", "summary": "The paper presents the Recursive Gated Convolution (g^n Conv) that performs high-order spatial interactions with gated convolutions and recursive designs. The proposed module can serve as a plug-and-play module for both transformer and convolutional neural networks.\nBased on the proposed module, HorNet is further introduced, showing good results on ImageNet classification, COCO object detection and ADE20K semantic segmentation.\n\n", "questions": "See the Strengths And Weaknesses part", "limitations": "See the Strengths And Weaknesses part", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nPros:\n- Well organized and good writing.\n- Extensive experiments on multiple datasets and tasks.\n- The design of g^nConv is interesting and contributes much to the performance.\n- Limitations are stated in this paper.\n\n\nCons:\n- The motivation is not very clear to me. Why do we need high-order interactions?\n- Since the operator is Mul rather than MatMul, then the high-order interaction seems like an element-wise one rather than the spatial-wise interaction as stated in the paper. Or it can be recognized as a channel-wise one (element-wise multiply -> channel projection -> element-wise multiply ...)?\n- Could the author provide the complexity analysis for different n?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept