More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity
Authors: Shiwei Liu
Tianlong Chen
Xiaohan Chen
Xuxi Chen
Qiao Xiao
Boqian Wu
Tommi Kärkkäinen
Mykola Pechenizkiy
Decebal Constantin Mocanu
Zhangyang Wang
Venue: ICLR 2023 poster (2023)
Keywords: 51x51 kernel
Large kernel convolution
convolutional neural networks
sparsity
backbone

Abstract: Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO. Codes are available at https://github.com/VITA-Group/SLaK.

Published as a conference paper at ICLR 2023
MORE CONVNETS IN THE 2020s: SCALING UP KER- NELS BEYOND 51 X 51 USING SPARSITY
Shiwei Liu Tianlong Chen'; Xiaohan Chen'; Xuxi Chen', Qiao Xiao², Boqian Wu', Tommi Kärkkäinen', Mykola Pechenizkiy², Decebal Constantin Mocanu Zhangyang Wang UUniversity of Texas at Austin, "Eindhoven University of Technology,
Universitty of Twente, "University of Jyväskylä, University of Luxembourg
Codes: hwwpt:tt///cc //github.. com
ABSTRACT
Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large ker- nels motivated by the local-window attention mechanism, showing appealing perfor- mance and efficiency. While one of them, i.c. RepLKNet, impressively manages to scale the kernel size to 31 x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of ad- vanced ViTs such Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31 x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61 x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLak), pure CNN architec- ture equipped with sparse factorized 51 x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architec- tures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object de- tection on PASCAL VOC 2007, and object detection/segmmenttion on MS COCO.
INTRODUCTION
Since invented (Fukushima & Miyake, 1982; LeCun et al., 1989; 1998). convolutional neural networks (CNNs) (Krizhevsky et al., 2012a; Simonyan & Zisserman, 2015.; He et al., 2016; Huang al., 2017; Howard et al., 2017; Xie et al., 2017; Tan & Le, 2019) have quickly evolved as one of the most indispensable architectures of machine learning in the last decades. However, the dominance of CNNs has been significantly challenged by Transformer (Vaswani et al., 2017) over the past few years. Stemming from natural language processing, Vision Transformers (ViTs) (Dosovitskiy et al., 2021; d' Ascoli et al., 2021; Touvron et al., 2021b; Wang et al., 202 Liu et al., 2021e; Vaswani et al., 2021) have demonstrated strong results in various computer vision tasks including image classification (Dosovitskiy et al., 2021; Yuan et al., 2021b). object detection (Dai et al., 2021; Liu et al., 2021e), and segmentation (Xie et al., 2021; Wang et al., 2021 Cheng et al., 2021). Meanwhile, works on understanding of ViT's have blossomed. Plausible reasons behind the success of ViTs are fewer inductive bias (Dosovitskiy et al., 2021), lange dependence (Vaswani et al., 2017). advanced architecture (Yu et al., 2021), and more human-like representations (Tuli al., 2021), etc.
Recently, there is a rising trend that attributes the supreme performance of ViTs to the ability to capture large receptive field. In contrast CNNs which perform convolution in a small sliding window (c.g., 3x3 3 and 5x5) with shared weights, global attention or local attention with larger window sizes in ViTs (Liu et al., 2021e) directly enables each layer to capture large receptive field. Inspired by this trend, some recent works on CNNs (Liu et al., 2022b; Ding et al., 2022) strike back by designing advanced pure CNN architecture and plugging large kernels into them. For instance, RepLKNet (Ding et al., 2022) successfully scales the kernel size to 31 x1x31, while achieving
"Equal contribution.

---

Published as a conference paper at ICLR 2023
ConvNeXt
RepLKNet
SLak
Inger
Input
Ingur
51 S1
51 H51
SHG
61
5x51
5x5
Large depth-wise kernel (e.g., 51) paradigms of ConvNeXt, Repl.KNet, and SLak. Dark blue squares refer to the dense weights in convolutional kernels. Light blue squares refer to the sparse weights in convolutional kernels.
Figure I:
comparable results to Swin Transformer (Liu et al., 2021e). However, large kernels are notoriously difficult train. Even with the assistance of a parallel branch with small kernels, the performance of KepLKNet starts to saturate as the kernel size continues increasing, compared to the scaling trend of advanced ViTs such Swin Transformer. Therefore, it remains mysterious whether can exceed the Transformer-bent
models by further scaling the kernel size beyond 31 x31..
In this paper, we attempt to answer this research question by leveraging sparsity commonly observed in the human visual system. Sparsity has been seen as one of the most important principles in the primary visual cortex (V1) (Tong, 2003), where the incoming stimuli have been hypothesized to be sparsely coded and selected (Desimone & Duncan, 1995; Olshausen & Field, 1997; Vinje & Gallant, 2000). We extensively study the trainability of large kernels and unveil three main observations: (i) existing methods that either naively apply larger kernels (Liu et al., 2022b) or assist with structural oe-parametrrization (Ding et al., 2022) fail to scale kernel sizes beyond 31 x31; (ii) replacing one large MxM kernel with two rectangular, parallel kernels (MxN and N x,, where N <, M) can smoothly scale the kernel size up to 61 x61 with improved performance; (iii) constructing with sparse groups while expanding width significantly boosts the performance.
Built upon these observations, we propose SLaK - Sparse Large Kernel Network a new pure CNN architecture equipped with an unprecedented kernel size of 51x51. Evaluated across a variety of tasks including ImageNet classification (Deng et al., 2009), semantic segmentation on ADE20K (Zhou al., 2019), object detection on PASCAL VOC 2007 (Everingham et al., 2007), and object de- pection/semmennation on COCO (Lin et al., 2014), SLaK performs better than or on par with CNN pioneers RepLKNet and ConvNeXt (Liu al., 2022b) as well as SOTA attention-based based models c.g., Swin (Liu et al., 202 le) and Cswin (Dong et al., 2022) Transformers on ImageNet. Our analysis of effective receptive field (ERF) shows that when plugged in the recently proposed ConvNeXt, our method is able to cover a large ERF region than existing larger kernel paradigms.
RELATED WORK
Large Kernel in Attention. Originally introduced for Natural Language Processing (Vaswani et al., 2017) and extended in Computer Vision by Dosovitskiy et al. (2021), self-attention can be viewed as a global depth- wise kernel that enables each layer to have a global receptive field. Swin Transformer (Liu et al., 2021e) is ViTs variant that adopts local attention with shifted window manner. Compared with global attention, local attention (Ramachandran et al., 2019; Vaswani et al., 2021; Chu al., 2021; Liu et al., 2021d; Dong al., 2022) can greatly improve the memory and computation efficiency with appealing performance. Since the size of attention windows is at least 7, it can be seen as an alternative class of large kernel. A recent work (Guo et al., 2022b) proposes a novel large kernel attention module that uses stacked depthwise, small convolution, dilated convolution as well as pointwise convolution to capture both local and global structure.
Large Kernel in Convolution. Large kernels in convolution date back to the 2010s (Krizhevsky et al., 201 2b; Szegedy et al., 2015; 2017), not earlier, where large kernel sizes such as 7x7 and 11 x1x11 are applied. Global Convolutional Network (GCNs) (Peng et al., 2017) enlarges the kernel size to 15 by employing a combination of X + Mx1 and Mx + X convolutions. However, the proposed method leads to performance degradation 2016; 2017) allows for the utilization of varying convolutional kernel sizes to learn spatial patterns at different scales. With the popularity of VGG (Simonyan & Zisserman, 2014), it has been common over the past decade to use a stack of small kernels (1 I or 3x3) to obtain large receptive field (He et al., 2016; Howard et al., 2017; Xie et al., 2017; Huang et al., 2017). Until very recently, some
ImageNet. The family of Inceptions (Szegedy et al.,
2

---

Published as a conference paper at ICLR 2023
One trainng step
Prune-and-
Initial
No
Prune
Grow
Adjust?
Yes
activated weights
pruned weights
grown weights
Figure 2: Dynamic sparsity. Dynamic sparsity allows us to construct and train initially sparse neural networks (sparse kernels) from scratch. During training, dynamically adjusts the sparse weights by pruning the least important weights and adding new. Such dynamic procedure gradually optimizes the sparse kernels to good pattern and hence encourages
elaborate capture of local features.
works start to revive the usage of large kernels in CNNs. Li et al. (2021) propose involution with 7 large kernels that uses distinct weights in the spatial extent while sharing weights across channels. However, the performance improvement plateaus when further expanding the kernel size. Han et al. (2021b) find that dynamic depth-witg wise convolution (7 7) performs on par with the local attention mechanism if we substitute the latter with the former in Swin Transformer. Liu et al. (2022b) imitate the design elements of Swin Transformer (Liu al., 20)1e) and design ConvNeXt employed with 7x7 kernels, surpassing the performance of the former. RepLKNNet (Ding et al., 2022) for the first time scales the kernel size to 31 x31 by constructing a small kernel (c.g.. 3x3 or 5 5x5) parallel to it and achieves comparable performance to the Swin Transformer. A series of work (Romero et al., 2021; 2022) of continuous convolutional kernels can be used on data of arbitrary resolutions, lengths and dimensionalities. Lately, Chen et al. (2022) reveal large kernels to be feasible and beneficial for 3D networks too. Prior works have explored the idea of paralleling (Peng et al., 2017; Guo et al., 2022a) or stacking (Szegedy et al., 2017) two complementary Mx I and 1 X kernels. However, they limit the shorter edge to and do not scale the kernel size beyond 51 x51. Different from those prior arts, we decompose large kernel into two complementary non- non-square kernels (MxN and NxM), improving the training stability and memory scalability of large convolutions kernels.
Dynamic Sparsity. A mong-staniing research topic, recent attempts on sparsity (Mocanu et al., 2018; Liu et al., 2021 Evei et al., 2020; Mostafa & Wang. 2019; Dettmers & Zettlemoyer, 2019; Chen et al., 2021) train intrinsically sparse neural networks from scratch using only a small proportion of parameters and FLOPs (as illustrated in Figure 2). Dynamic sparsity enables training sparse models from scratch, hence the training and inference FLOPs and memory requirements are only a small fraction of the dense models. Different from post-training pruning (Han al., 2015; Frankle & Carbin, 2019), models built with dynamic sparsity can be trained from scratch to match their dense counterparts without involving any pre-training from Sparse Evolutionary Training (SET) (Mocanu et al., 2018; Liu et al., 2021 which randomly initializes the sparse connectivity between layers randomly and dynamically adjusts the sparse connectivity via a parameter prune-and-gre prune-and-gree scheme allows the model's sparse structure gradually evolve, achieving better performance than naively training a static sparse network (Liu et al., 2021c). In this paper, our target not to find sparse networks that can match the corresponding dense networks. Motivated by the principle of ResNeXt (Xie et al., 2017; Liu et al., 2022b) "use more groups, expand width", we instead attempt to leverage dynamic sparsity to scale neural architectures with extreme kernels.
dense training. Dynamic sparsity stems
scheme during the course of training. The parameter
FAILURES OF EXISTING APPROACHES TO Go BEYOND 31 31x31 KERNELS
We first study the performance of extreme kernel sizes larger than 31x31 using two existing large- kernel techniques, ConvNeXt (Liu et al., 2022b) and RepLKNet (Ding et al., 2022). We take the develood ConvNeXt on ImageNet- IK as our benchmark to conduct this study. We adopt the efficient large-kernel implementation developed by MegEngine (Meg, 2020) in this paper.
We follow recent works (Liu et al., 2022b; Bao et al., 2021; Liu et al., 2021e; Ding et al., 2022; Touvron et al., 2021b) using Mixup (Zhang al., 2017), Cutmix (Yun et al., 2019),
3

---

Published as a conference paper at ICLR 2023
Table 1: Test accuracy of 120-epoch ConvNeXt-T trained with various large kernel recipes on ImageNet- IK. "Naive" refers directly enlarging kernel size of ConyNeXt; "RepLKNet" refers to training ConvNeXt with structural re-parameterization (Ding et al., 2022). The original ConvNeXt is built with 7 7 kernels.
Kernel Size
Top-1 Acc
#Params
FL.OPs
Top-I Ace
#Params
FLOPPs
Naive
RepLKNett
7-7-7-7
81.0
29M
4.5G
31-29-27-13
80.5
32M
6.5G
81.5
32M
6.1G
51-49-47-13
80.4
38M
9.4G
81.3
38M
9.3G
61-59-57-13
80.3
43M
11.6G
81.3
43M
11.5G
Ranment (Cubuk et al., 2020), and Random Erasing (Zhong et al., 2020) as data augmentations. Stochastic Depth (Huang et al., 2016) and Label Smoothing (Szegedy et al., 2016) are applied as regularization with the same hyper-parameters as used in ConvNeXt. We train models with AdamW (Loshchilov & Hutter, 2019). is important to note that all models are trained for a reduced length of 120 epochs in this section, just to sketch the scaling trends of large kernel sizes. Later in Section 5, we will adopt the full training recipe and train our models for 300 epochs, to enable fair comparisons with offe-of-the-aa art models. Please refer to Appendix A for more details.
Liu et al. (2022b) show that naively increasing kernel size from x to 7x7 brings considerably performance gains. Very recently, KeeL (Ding al., 2022) successfully scales convolutions up to 31 x31 with structural re-parameterization (Ding et al., 2019; 2021). We further increase the kernel size 51x51 and 61 x61 and see whether larger kernels can bring more gains. Following the design in RepLKNet, we set the kernel size of each stage as [51, 49, 47, 13] and [61, 59, 57, 13]. and report test accuracies in Table 1. As expected, naively enlarging kernel size from 7x7 31 x31 decreases the performance, whereas RepLKNet can overcome this problem by improving the accuracy by 0.5%. Unfortunately, this positive trend does not continue when we further increase kernel size to 51 x51.
One plausible explanation is that although the receptive field may be enlarged by using extremely large kernels, might fail to maintain the desirable property of locality. Since the stem cell in standard ResNet (He et al., 2016) and ConvNeXt results in a 4x downsampling of the input images, extreme kernels with 51 11 are already roughly equal to the global convolution for the typical 224 x 224 ImageNet. Therefore, this observation makes sense well-designed local attention (Liu et al., 20; Chu et al., 2021) usually outperforms global attention (Dosovitskiy et al., 2021) 1) in a similar mechanism of ViTs. Motivated by this, we see the opportunity to address this problem by introducing the locality while preserving the ability to capture global relations.
A RECIPE FOR EXTREMELY LARGE KERNELS BEYOND 31x31
In this section, we introduce simple, two-step recipe for extremely large kernels beyond 31 x31:
Step 1. Decomposing large kernel into two rectangular, parallel kernels.
Step 2. Using sparse groups, expanding more width.
Decomposing a large kernel into two rectangular, parallel kernels smoothly scales the kernel size up to 61 61. Although using convolutions with medium sizes (e.g., 31 x31) seemingly can directly avoid this problem, we want to investigate if we can further push the performance of CNNs by using (global) extreme convolutions. Our recipe here is approximate the large MxM kernel with combination of two parallel and rectangular convolutions whose kernel size is M and NxM (where < M), respectively, as shown in Figure 1, Following Ding et al. (2022), we keep a 5x5 layer parallel to the large kernels and summed up their outputs after a batch norm layer.
This decomposition balances between capturing long-range dependencies and extracting local detail features (with its shorter edge). Moreover, existing techniques for large kernel training (Liu et al., 2022b; Ding et al., 2022) suffer from quadratic computational and memory overhead as the kernel size increases. In stark contrast, the overhead of our method increases just linearly with the kernel size (Figure 4). The performance of kernel decomposition with N = 5 (see Appendix E for the effect of N) is reported the "Decomposed" group in Table 2. As the decomposition reduces learnalee
4

---

Published as a conference paper at ICLR 2023
parameters and FLOPs, it is no surprise to observe network to initially sacrifice accuracy slightly compared to the original RepL at medium kernel sizes i.e. x1x31. However, as the convolution
size continues to increase, method can scale kernel size up to 61 x61 with improved performance.
Table 2:
Test accuracy of ConvNeXt-T trained with various large kernel recipes on ImageNet- IK. All the models are trained for 120 epochs.
Kerne Size
Top-1 I Ace
aParams
FL.PPs
Top- I Ace
#Parmms
FL.PP|
Top-1 I Ace
daaaams
F1.0Ps
Decomposed
Spurse groups
Spanse groups, expand more width
7-7-7-7
81,0
29M
4.5G
80,0
17M
2.6G
81.1
29M
4.5G
31-29-37-13
81,3
30M
5.0G
80.4
18M
2.9G
81.5
30M
4.8G
51-49-47-13
81.5
31M
5:4G
80.5
18M
3.1G
81.6
30M
5.0G
61-59-57-13
81.4
31M
5.6G
80.4
19M
3.2G
81.5
31M
5.2G
"Use sparse groups, expand more width" significantly boosts the model capacity. Recently proposed ConvNeXt (Liu et al., 2022b) revisits the principle introduced in ResNeXt (Xie al., 2017) that splits convolutional filters into small but more groups. Instead of using the standard group convolution, ConvNeXt simply employs depthwise convolutions with an increased width to achieve the goal of "use more groups, expand width". In this paper, we attempt to extend this principle from a sparsity inspired perspective "use sparse groups, expand more width".
To be specific, first replace the dense convolutions with sparse convolutions, where the sparse kernels are randomly constructed based on the layer-wise sparsity ratio of SNIP (Lee et al., 2019)' due to its strong performance on large-scale models (Liu et al., 2022a). After construction, we train the sparse model with dynamic sparsity (Mocanu et al., 2018: Liu et al., 2021b), where the sparse weights are dynamically adapted during training by pruning the weights with the lowest magnitude and growing the same number of weights randomly. Doing so enables dynamic adaptation of sparse weights, leading to better local features. As kernels are sparse throughout training, the corresponding parameter count and training/inference FLOPs are only proportional to the dense models. See Appendix for the full details of dynamic sparsity. To evaluate, we sparsify the decomposed kernels with 40% sparsity and report the performance as the "Sparse groups" column. We can observe in the middle column of Table that dynamic sparsity notably reduces more than 2.0 GFLOPs, despite causing temporary performance degradation.
We next show that the above high efficiency of dynamic sparsity can be effectively transferred to model scalability. Dynamic sparsity allows to computation-frientt friendly scale the model size up. For example, using the same sparsity (40%), we can expand the model width by 1.3x while keeping the parameter count and FLOPs roughly the same the dense model. This brings significant performance gains, increasing the performance from 81.3% to 81.6% with extreme 51 x51 kernels. Impressively, equipped with 61 x61 kernels, our method outperforms the previous state of the arts (Liu et al., 2022b; Ding et al., 2022) while saving 55% FLOOPs.
Large Kernels Generalize Better than Small Kernels with Our Recipe. To demonstrate that the benefits of large kernels, we also report the impact of each step for the small 7x7 kernel in Table 2. We can clearly see that the performance consistently increases with kernel size, up 51 x51, Applying each part of our proposed recipe to 7 7x7 kernels leads to either no gain or marginal gains compared to our 5 3lx51 kernels. This break-down experiment justifies our claim: large kernel the root of power, and our proposed recipe helps unleash such power from large kernels.
4.1 BUILDING THE SPARSE LARGE KERNEL NETWORK (SLAK)
So far, we have discovered our recipe which can successfully scale up kernel size to 51x without backfiring performance. Built on this recipe, we next construct our own Sparse Large Kernel Network (SLaK), pure CNN architecture employed with extreme 51 x51 kernels. SLaK is built based on the architecture of ConvNeXt. The design of the stage compute ratio and the stem cell are inherited from No The number of blocks in each stage is [3, 3, 9, 3] for SLaK-T and [3, 3, 27, 3) for SLaK-S/B. The stem cell is simply a convolution layer with 4x4 kernels and 4 strides.
We first directly increase the kernel size of ConvNeXt to [51, 49, 47, 13] for each stage, and replace each MxM kernel with combination of 5x5 and 5xM kernels illustrated in Figure 1. We find
SNIP ratio is obtained by globally selecting the important weights across layers with the highest connection sensitivity score lg @ wl| where and g is the network weight and gradient, respectively.
5

---

Published as a conference paper at ICLR 2023
that adding BatchNorm layer directly after each decomposed kernel is crucial before summing the output up. Following the guideline of "use sparse groups, expand more width", we further sparsify the whole network and expand the width of stages by 1.3x, x, ending up with SLaK. Even there could be a large room to improve SLaK performance by tuning the trade-off between model width and sparsity (as shown in Appendix D), we keep set of hyperparameters (1.3x width and 40% sparsity) for all experiments, so SLaK works simply "out of the box" " with no ad-hoc tuning at all.
EVALUATION OF SLAK
To comprchensively verify the effectiveness SLak, we compare it with various state-of-the-art base- lines on a large variety of tasks, including: ImageNet- IK classification (Deng et al., 2009), semantic segmentation on ADE20K (Zhou al., 2019), object detection on PASCAL VOC 2007 (Everingham et al., 2007), and object detection//semmetatiion on COCO.
Table 3: Classification accuracy on ImageNet-IK. For SLaK models, we report both theoretical, aparsity-aware numbers parameter & FLOPs (in black color), as well those numbers measured assuming no sparsity-aware acceleration (in blue color).
Model
Image Size
#Prram.
FLOPs
Top-1 Acc
ResNet-50 (He al., 2016)
224 x
4.1G
76.5
ResNeXt-500 50-exxx--0--224dd (Xie et al., 2017)
224
25M
4.3G
77.6
ResMLP-24 (Touvron et al., 2021a)
224x224
30M
6.0G
79.4
DeiT-S (Touvron et al., 2021b)
224x224
22M
4.6G
79.8
Swin-T (Liu al., 2021c)
224x224
28M
4.5G
81.3
TNT-S (Han et al., 2021a)
224x224
24M
5.2G
81.3
T2T-ViT,-14 (Yuan et al., 2021a)
224x224
22M
6.1G
81.7
ConvNeXt-T (I.iu et al., 2022b)
224x224
29M
4.5G
82.1
SLaK-T
224x224
30M/S0M
5.0G/8.7G
82.5
Mixer-B/U (Tolstikhin al., 2021)
224x224
59M
11.6G
76.4
ResNet- 101 (He et al., 2016)
224x224
7.9G
77.4
ResNeXt101-32x4d (Xic et al., 2017)
224x224
8.0G
78.8
PVT-Large (Wang et al., 2021h)
224x224
9.8G
81.7
T2T-ViT,-19 (Yuan al., 2021a)
224x224
39M
9.8G
82.4
Swin-S (Liu et al., 2021e)
224x224
50M
8.7G
83.0
Con (Liu al., 2022b)
224x224
50M
8.7G
83.1
SLaK-S
224x224
55M/91M
9,8G/16.7G
83.8
DeiT-Base/ (Touvron al., 2021b)
224x224
87M
17.6G
81.8
Repl. KNet-31 (Ding al., 2022)
224x224
79M
15.3G
83.5
Swin-B (Liu al,, 2021e)
224x224
88M
15.4G
83.5
Con-NeXt-B (Liu al., 20226)
224x224
15.4G
83.8
SLaK-B
224x224
95M/L58M
17.1G/28.5G
84.0
VIT-Base/16 (Dosovitskiy et al., 2021)
x84
87M
55.4G
77.9
DeiT-B/16 (Touvron et al.. 2021b)
384
86M
55.4G
83.1
Swin-B B (Liu et ul., 2021c)
384
S8M
47.1G
84.5
KNet-31BB (Ding al., 2022)
384x338
79M
45.1G
84.8
ConvNeXt-B (Liu et al., 2022b)
384x338
89M
45.0G
85.1
SLaK-B
384x384
95M/158M
50.3G/83.8G
85.5
5.1 EVALUATION ON IMAGENET-IK
ImageNet-IK IK contains 1,281,167 training images, 50,000 validation images. We use exactly the same training configurations in Section 4, except now training for the full 300 epochs, following ConvNeXt and Swin Transformer. We observed that models with BatchNorm layers and EMA see poor performance when trained over 300 epochs (also pointed by Liu et al. (2022b)), and resolved this by running one additional pass over the training data, the same used in Garipov et al. (2018); Izmailov et al. (2018). Please refer to Appendix for more details about the training configurations.
We compare the performance of SLaK on ImageNet-IK with various the state-of-the-art. in Table 3. With similar model sizes and FLOPs, SLaK outperforms the existing convolutional models such as ResNe(X)t (He al., 2016; Xie al., 2017), RepLKNet (Ding et al., 2022), and ConvNeXt (Liu et al., 2022b). Without using any complex attention modules and patch embedding, SLaK is able to achieve higher accuracy than the state-of-the-art transformers, e.g., Swin Transformer (Liu et al.,

---

Published as a conference paper at ICLR 2023
2021e) and Pyramid Vision Transformer (Wang et al., 2021b; 2022). Perhaps more interestingly, directly replacing the 7 of ConvNeXt-S to x51 is able to improve the accuracy over the latter by 0.7%. Moreover, Table 3 shows that model benefits more from larger imput sizes: the performance improvement of SL.k-B over ConyNeXt-B on x384 is twice that on 224x224 input, highlighting the advantages of using large kernels high-resolution training (Liu al., 2021d).
Moreover, we also examine if SLaK can rival stronger Transformer models - CSwin Trans- former (Dong et al., 2022), a carefully designed hybrid architecture with transformers and con- volutions. CSwin performs self-attention in both horizontal and vertical stripes. As shown in Appendix C, SLaK consistently performs on par with CSwin Transformers, and is the first pure ConvNet model that can achieve so without any bells and whistles, to our best knowledge.
5.2
EVALUATION ON ADE20K
For semantic segmentation, we choose the widely-used ADE20K, large-scale dataset which contains 20K images of 150 categories for training and 2K images for validation. To draw solid conclusion, we conduct experiments with both short and long training procedures. The backbones are pre-trained on ImageNet- IK with x24x224 input for 120/300 epochs and then are finetuned with UperNet (Xiao et al., 2018) for 80K/160K iterations, respectively. We report the mean Intersection over Union (mloU) with single-scale testing in Table 4 for comparison. The upper part of Table 4 demonstrates a very clear trend that the performance increases as the kernel size. Specifically, RepLKNet scales the kernel size of ConvNeXt-T to 31 x31 and achieves 0.4% higher mloU. Notably, SLaK-T with larger kernels (51 551) further brings 1.2% mloU improvement over ConvNeXI-T (RepL.Nee)), surpassing the performance of ConvNeXt-S. With longer training procedures, SLaK also consistently outperforms its small-kernel baseline (ConvNeXt) by a good margin, reaffirming the advantages of large kernels downstream dense vision tasks.
Table 4: Semantic segmentation on ADE20K. Models are pre-trained ImageNet- IK with 224x224 input and finetuned on with UperNet. Following Ding et al. (2022), we report mloU results with single-scale testing. FLOPss based on input sizes of (2048, 512). Methods marked with * implemented//eprrduueed by us.
Model
Kernel Size
mloU (t)
#Param
FLOPs
pre-trained for 120 epochs, finetuned for 80K iteration
Con (1.iu et al., 2022b)
7-7-7-7
44.6
60M
939G
ConyNeXt-S (Liu et al., 2022h)
7-7-7-7
45.9
82M
1027G
ConvNeXt-T (RepLKNet)* (Ding al., 2022)
31-29-27-13
45.0
64M
973G
SLaK-T
51-49-47-13
46.2
65M
936G
pre-trained for 300 epochs, finetuned for 160K iteration
ConvNeXI-T (Liu et al.. 2022b)
7-7-7-7
46.0
60M
939G
SLaK-T
51-49-47-13
47.6
65M
936G
NeN-NeXI-S (Liu et al., 2022b)
7-7-7-7
48.7
82M
1027G
SLaK S
51-49-47-13
49.4
9IM
1028G
ConvNeXt-B B (Liu et al., 2022b)
7-7-7-7
49.1
122M
1170G
SLaK B
51-49-47-13
50.2
135M
1172G
5.3 EVALUATION ON PASCAL VOC 2007
PASCAL VOC 2007 is another popular benchmark for object detection and semantic segmentation. In this section, we evaluate our model on object detection. Models are on pre-trained on ImageNet- IK for 300 epochs and finetuned with Faster-RCNN (Ren et al., 2015). Table shows the results comparing SLaK-T, ConvNeXt-T, ConvNet-T (RepLKNet), and traditional convolutional networks, i.e. ResNet. Again, large kernels lead to better performance. Specifically, ConvNeXI-T with 31 x31 kernels achieves 0.7% higher mean Average Precision (mAP) than the 7x7 kernels and TTa with 51 kernel sizes further brings 1.4% mAP improvement, highlighting the crucial role of extremely large kernels downstream vision tasks.
5.4 EVALUATION ON COCO
Furthermore, finetune Cascade Mask R-CNN (Cai & Vasconcelos, 2018) MS COCO (Lin et al., 2014) with our SLaK backbones. Following ConvNeXt, we use the multi-scale setting and train models with Adam W. Please refer to Appendix A.3 for more implementation details. Again,
7

---

Published as a conference paper at ICLR 2023
Table 5: Object detection on PASCAL VOC 2007. Faster RCNN is equipped with various backbone networks that are pre-trained for 120 epochs on ImageNet-Il I.. The pre-trained ConvNeXt-1 is obtained from its GitHub repository. FLOPPs are based on input sizes of (1280, 800). Methods marked with are implemented by
Model
Kernel Size
mAP (5) (f)
#Paara
FLOPs
ResNet-50 et al., 2016)
3-3-3-3
74.0
ResNet-100 (He 2016)
3-3.3.3
74.3
Con-NeXt-T (Liu al., 2022b)
7-7-7-7
80.6
45M
208G
Con-NeXtT (RepL.KNet)" (Ding al., 2022)
31-29-27-13
81.3
55M
207G
SLak-T
51-49-47-13
82.7
49M
205G
see an encouraging trend: the performance consistently improves with the increase of kernel size and 51 51 kernel SLaK outperforms smaller-kernel models.
Table 6: Object detection and segmentation on MS COCO. Models are pre-trained on ImageNet- 1K and finetuned using Cascade Mask-RCNN, Methods marked with * are implemented by us.
Model
Keree Aphox
AP
AP
Apmond
APp
APa
pre-trained for 120 epochs, linetumed For 1x (12 epochs)
ConvNeXi-T (Liu 2022b)
7-7-7-7
$1.5
ConvNeXi-T (NeT))NETT) (Ding et al., 2022)
31-29-27-13
47.8
66.7
$2.0
41.4
63.9
44.7
STaK-T
51-49.47-13
48.4
67.2
52.5
41.8
64.4
45.2
pre-trained for 300 epochs, finctuned for 3x (36 opochs)
Con-NeXi-T (Liu 2022b)
7-7-7-7
54.8
SLaK-T
51-49-47-13
$1.3
70.0
55.7
44.3
67.2
48.1
ANALYSIS OF SLAK
6.1 EFFECTIVE RECEPTIVE FIELD (ERF)
Constiot 3.7.1
Candlet: iessUoe 131. 13]
SLaK
49,
400
C00
200
230
A20
1000
202
820
1000
1.0
2.0
200
500
500
000
200
1000
Figure 3: Effective receptive field (ERF) of models with various kernel sizes. SLaK is not only able to capture long-range dependence but also the local context features.
The concept of receptive field (Luo al., 2016; Araujo al., 2019) is important for deep CNNs:
anywhere an input image outside the receptive field of a unit does not affect its output value. Ding al. (2022) scale kernels up x31 and show enlarged ERF and also higher accuracy over small-kernel models (He et al., 2016). As shown in the ERF theory (Luo et al., 2016), ERF is proportion (k)n). where k and refers the kernel size and the network depth, respectively. Therefore, the hypothesis behind the kernel decomposition in SLaK is that the two decomposed MxN and Nx M kernels can well maintain the ability of large kernels in terms of capturing large ERF, while also focusing on fine fine-grained local features with the shorter edge (N).
To evaluate this hypothesis, we compare the ERFs captured by SLaK and Repl.KNet. Following Kim et al. (2021); Ding et al. (2022), we sample and resize 50 images from the validation set to 1024 1024, and measure the contribution of the pixel on input images to the central point the feature map generated in the last layer. The contribution scores are further accumulated and projected to 1024x 1024 matrix, as visualized in Figure 3. In the left sub-figure, although the original ConvNeXt already improves the kernel size 7x7, its nigh-contribution pixels concentrate in the center of the input. Even the 31 x31 kernels used by RepLKNet are not sufficient for ConvNeXt to cover the whole input. In comparison, high-contribution pixels of SLaK spread in much larger ERF, and some
8

---

Published as a conference paper at ICLR 2023
high- high-contribution pixels emerge in non-center areas. This observation is in line with our hypothesis that SLak balances between capturing long-range dependencies and focusing on the local details.
6.2 KERNEL SCALING EFFICIENCY
VL.00''5 fare
#Tatamn KarreS
Resolation
Dense 51x51
Spari: 51:51
Spanc 51
64-64
243.15
245.14
58.14
128 128
900.19
950.30
234.32
23
53
100
355
22
4.3
100
Pap_ Cecompsuedd
Sparne Dwcomppoedd spenity!
Figure Left: Scaling efficiency. The number of GFLOPs (left) and parameters (right) as the kernel size in ConvNeXt-T scales up. Right: Real inference time latency (ms), Real inference time latency of depth-size convolutions with different kernel size. The results are obtained on single A100 GPU with PyTorch 1.10.0 + cuDNN 8.2.0, in FP32 precision, using special hardware accelerator.
As Section 4 mentioned, the two components of SLaK, kernel decomposition and sparse group, substantially improve the scaling efficiency of kernel sizes. To further support this, report the overhead required by various large kernel training methods in Figure 4-left. We simply replace all the kernels in stages of ConyNeXt-T with set of kernel sizes from to 151 and report the required GFLOPs and the number of parameters. One can clearly see the big gap between full-kernel scaling (yellow lines) and kernel decomposition (green lines) as the kernel size increases beyond 31 x31. Even using the ultra-large 151 151 kernels, using our methods would require fewer FLOPs and parameters, compared full-kernel scaling with 51 x51 kernels. Growing evidence has demonstrated that training with high resolution is performance booster for classification (Tan & Le, 2019; Liu al., 2021 dd) and object detection (He al., 2017; Lin al., 2017). We believe that large kernels will benefit more in this scenario.
We also report the real inference time latency (ms) of different large-kernel recipes Figure 4-right using one-layer depth-wise convolutions. The results are obtained on single A100 GPU with PyTorch 1.10. + cuDNN 8.2.0, in FP32 precision, using no dedicated sparsity-rriendly hardware accelerator. The input shape is (64, 384, R. R). In general, without special hardware support, while vanilla sparse large kernels cost similar run time to dense large kernels of the same size (sparse kernels receive limited support in common hardware), the sparse decomposed kernels yield more than 4x real inference speed acceleration than directly using vanilla large kernels.
7 CONCLUSION
Recent works on modern ConvNets defend the essential roles of convolution in computer vision by designing advanced architectures and plugging large kernels. However, the largest kernel size is limited to 31 x31 and the performance starts to saturate the kernel size continues growing. In this paper, we investigate the training ConvNets with extremely large kernels that are beyond 31 x31 and consequently provide recipe for applying extremely large kernels inspired by sparsity. Based on this recipe, we build a pure ConvNet model that smoothly scales up the kernel size beyond 51 x51, while achieving better performance than Swin Transformer and ConvNeXt. Our strong results suggest that sparsity, as the "old friend" deep learning, can make a promising tool to boost network scaling.
ACKNOWLEDGEMENT
Liu, X. Chen and Z. Wang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML). We thank Zhenyu Zhang for helping conduct many experiments in Section J. Part of this work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. NWO2021 060, FINF-2694 and EINF-2943/L.1.

---

Published as a conference paper at ICLR 2023
REFERENCES
Megengine:a fast, scalable and easy-to-use deep learning framework. https://githut. MegEnging weweenniinneMMee www.eeeeeeennneeee 2020.
GitHub repository: Convnext. https: httptacccc/aacccoooooreeeereee/a 2021.
André Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural networks. Distill, 4(11):e21, 2019.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint urXiv:2/06.08254 2021.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. URL https: http:///ppeeeiieeeeeeeeeeeeeeeee iew. net / forum d=BB MN01:=WN1CC C-
Zhaowei Cai and Nuno Vasconcelos. Cascade r-enm:
Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv: 155, 2019.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances Neural Information Processing Systems, 34, 2021.
Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. arXiv preprint arXiv:2206. 10555, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
MMSegmentation Contributors. MMSegmmentation Openmmlab semantic segmentation toolbox and benchmark. https::/gitti thubb
mmlenn-mmlaaae ab htt////msgeeeteeee atimentttaiiin.. 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengehen Liu, Lu Yuan, and Lei Zhang
Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7373-7382, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. Icee, 2009.
Robert Desimone and John Duncan. Neural mechanisms of selective visual attention. Annual review of neuroscience, 18(1):193-22) 1995.
10

---

Published as a conference paper at ICLR 2023
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:9007.04840, 2019.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1911-1920, 2019.
Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlpnet: Hierarchical vision mlp with re-parameterized locality. arXiv preprint arXiv:2112.11081, 2021.
Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. arXiv preprint arXiv:2203.06717, 2022.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped win- dows. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 12124-12134, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: /otpn//s////eeeeviieeeeee net/ forum? d=-iccbFNNTTT
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pp. 2286-2296, PMLR, 2021.
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 46629-1463,, 2020.
Paul Erdós and Alfréd Rényi. On random graphs i. Publicationes Mathematicae (Debrecen), 6: 990-297, 1959.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
M. Everingham, L. Van Gool, K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- 2007.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. In International Conference on Learning Representations, 2019. URL https:
//openrevies wwweviewww... ww..et.....e..........e net/ wwtp:eiwwwwweeeoennneeeee i -ttrrmmmm-rrrr--3------
for
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267-285. Springer, 1982.
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006. 10900, 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018
Meng-Hao Guo, Cheng-Ze Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation, 2022a. URL
https: /aaxivv.. www.iiiorrg//ooo org/ alos/ 2209.000000 08575.
Meng-Har Guo, Cheng- Ze Lu, Zheng-Ning Liu. Ming-Mine Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.0974), 2022b.

---

Published as a conference paper at ICLR 2023
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34, 2021a.
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference Learning Representations, 2021b.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: D..1109/VPR.2016.00
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi-
mensional transformers. arXiv preprint arXiv: 2019.
Duc N.M Hoang, Shiwei Liu, Radu Marculescu, and Zhangyang Wang. REVISITING PRUNING
AT INITIALIZATION THROUGH THE LENS OF RAMANUJAN GRAPH. In International Conference on Learning Representations, 2023. URL https: //tp:////e/eeeeeeeeeeeeee net forum?
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04611, 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661, Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kiliann Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-477 47084,788 2017.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method find M transposable masks. arXiv preprint arXiv:2102.08122 2021.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint 103 2018.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances Neural Information Processing Systems, 33, 2020.
Peng Jiang, Lihan Hu, and Shihui Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. In Advances in Neural Information Processing Systems.
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim. Dead pixel test using effective receptive field. arXiv preprint arXiv:2108. 13576, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012a. URL https: http:////ooocececcce/ee //tppppp//pppoeeeeeeeang www.:diiinnnnnnnuuneeeeeee neurips httpp/pppw.cpppppp/////////////e c399862d3bb 6
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2b.
12

---

Published as a conference paper at ICLR 2023
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back propaaatoggttion network. Advances in neural information processing systems, 2, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. btradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1111)):277222222 1998.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations, 2019. URL https: /tttt::://ppnrevvviie
net forum Bid- 2ooe:/ooooomma11222
Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Involution: Inverting the inherence of convolution for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12321- 12330, 2021.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21 2117-2125, 2017.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9922, a.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33(7):2589-2604 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do actually need dense prerpparameterizatoon? in-time over-parameetrization in sparse training. In Proceedings of the 39th International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021c
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, 2022a.
URL https::p//// /ttp::://ppeeeevveee net.
net forum? iiimmiddiiiii 1d=vB2J_3tz--
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint 1rXiv:2111.09888 2021d.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021e.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic.
A convnet for the 2020s. arXiv preprint arXiv:2201.03545, 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2019. URL https: /ttp::///ppeeeeeieeeee www.:/eet/ffttiieeemmemmm Cqy7
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
13

---

Published as a conference paper at ICLR 2023
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pP- 4646-4655 PMLR, 2019.
Nvidia. Nvidia a100 tensor gpu architecture.
http::/www.aadddaaccccc/oonmmeme
2020.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision research, 37(23):3311-332 1997.
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters imprrve semantic segmentation by global convolutional network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4353-4361, 2017.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in Neural Information Processing Systems, 32, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
David W Romero, Anna Kuzina, Erik Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint
2021.
David W Romero, David M Knigge, Albert Gu, Erik J Bekkers, Efstratios Gavves, Jakub M Tomezak, and Mark Hoogendoorn. Towards a general purpose cnn for long range dependencies in nd. arXiv preprint (iXXix2226603390 2022.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropa- gation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 01122889928903 2021.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14454-14463, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114, PMLR, 09-15 Jun 2019.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34, 2021.
14

---

Published as a conference paper at ICLR 2023
Frank Tong. Primary visual cortex and visual awareness. Nature Reviews Neuroscience, 4(3):119-229 2003.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint 2021a.
1:XXv:2105.03344
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference
Computer Vision, pp. 32-42, 2021c.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 459-479 Springer, 2022.
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths. Are convolutional neural networks or transformers more like human vision? arXiv preprint arXiv:2105.00719, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12894-1222 12904, 2021.
William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 287(5456)11233-12766 2000.
Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Chich Chen. Max-dceplab: End-to- end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, pp. 1-10, 2022.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEFSCVF Conference on Computer Vision and Patlern Recognition, pp. 8741-8750, 2021c
Ross Wightman. GitHub repository: Pytorch image models.
https:/ /ttpttgttttt//t/////////te.hhhst com/
rwi wttigghtnn/pytorch-hiite iaali-imaaaeeaaa maoiaggeeeeeaaae 1 1s, 2019.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pp. 418-434, 2018
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
15

---

Published as a conference paper at ICLR 2023
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint arXiv:2111.1144 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 558-567, 2021a.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2106. 13112, 2021b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pazz mixup: Beyond empirical risk minimization. arXiv preprint 171.0:171. 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n:m fine- fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021. URL
net 2id 1ob
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):30--321, 2019.
16

---

Published as a conference paper at ICLR 2023
Contents of the Appendices:
Section A. Details of experimental settings, hyperparameters, and configurations used in this paper.
Section B. Full details of dynamic sparsity.
Section C. Comparison between CSwin Transforms and SLaK on ImageNet classification.
Section D. The sparsity-width trade-off in "use sparse group, expand more".
Section E. The effect of the shorter edge N of the decomposed large kernels.
Section F. ERF quantitation of various kernel sizes.
Section G. The learning curve of various architectures.
Section H. Real inference throughput of SLaK.
Section I. Comparison among various large kernel decomposition.
Section J. Investigation into the position of large kernels.
Section K. Limitation and Broad Impact.
A EXPERIMENTAL SETTINGS
A.1 IMAGENET-IK
We share the Straiitrain settings of SLaK on ImageNet-IK this section. We train SLaK for 300 epochs (Section 5.1) and 120 epochs (Section using AdamW (Loshchilov & Hutter, 2019) with a batch size of 4096, and a weight decay of 0.05. The only difference between models training for 300 epochs and 120 epochs is the training time. The learning rate is 4e-3 with a 20-epoch linear warmup followed by a cosine decaying schedule. For data augmentation, we use the default setting of RandAAgment (Cubuk et al., 2020) in Timm (Wightman, 2019) Smoothing (Szegedy et al., 2016) coefficient of 0.1, Mixup (Zhang et al., 2017) with a = 0.8. Cutmix (Yun et al., 2019) with a 1.0, Random Erasing (Zhong et al., 2020) with p = 0.25, Stochastic Depth with drop rate of 0.1 for SLaK-T. 0.4 for SLak-S, and 0.5 for SLaK-B, Layer Scale (Touvron et al., 2021c) the initial value of le-6, and EMA with a decay factor 0.9999. We train SLaK-T with NVIDIA A100 GPUs and the rest are trained with NVIDIA V100.
Label
A.2 SEMANTIC SEGMENTATION ON ADE20K
We follow the training setting used in Ding al. (2022); Liu al. (2022b) using UperNet (Xiao et al., 2018) implemented by MMSegmentation (Contributors, 2020) with the 80K/160K -iieeitrttinoonn training schedule. We conduct experiments with both short and long training procedures. The backbones pre-trained on ImageNet- 1K with 224 x24 input for 120/300 epochs and then are fine-tuned with UperNet (Xiao al., 2018) for 80K/160K iterations. respectively. We report the mean Intersection over Union (mIoU) with a single scale. All the hyperparameters are exactly the same as the ones used in the official GitHub repository of ConvNeXt (con, 2021).
A.3 OBJECT DETECTION AND SEGMENTATION ON COCO
For COCO experiments, we follow the training settings used in BEÎT, Swin, and ConvNeXt using MMDetection (Chen et al., 2019) and MMSegmentation (Contributors, 2020) toolboxes. The final model weights are adopted (instead of EMA weights) from ImageNet-IK pre-training with 224x224 input. We also conduct experiments with both short and long training procedures. The backbones are pre-trained on ImageNet- IK with 224x224 input for 120/300 epochs and then are finetuned with Cascade Mask R-CNN (Cai & Vasconcelos, 2018) for 12/36 epochs, respectively. All the hyperparameters are exactly the same as the ones used in the official GitHub repository of ConvNeXt (con, 2021).
17

---

Published as a conference paper at ICLR 2023
A.4 OBJECT DETECTION ON PASCAL VOC 2007
We follow Liu et al. (2021e) and finetune Faster-RCNN on PASCAL VOC dataset with SLaK-T as the backbone. We use multi-scale setting (Carion et al., 2020; Sun al., 2021) which leads the length of the shorter side between 480 and 800 and the ones of the longer side at most 1333. The model trained with AdamW for 36 epochs with a learning rate of 0.0001, weight decay of 0.05, and batch size of 16.
B DYNAMIC SPARSITY
Dynamic Sparsity (Mocanu et al., 2018; Liu et al., 2021b) is a class of methods that allow for end-to- end training of neural networks with sparse connectivity, also known as sparse training. Dynamic Sparsity starts from a sparse network while jointly optimizing the sparse connectivity and model weights during training. Without loss of generality, we provide the general pseudocode of Dynamic Sparsity that can cover most of the existing methods in Algorithm.
While there is an upsurge in increasingly efficient ways for sparse training (Bellee et al., 2018; Mostafa & Wang, 2019; Dettmers Zettlemoyer, 2019; Evei et al., 2020; Liu et al., 2021c; Jayakumar et al., 2020; Chen al., 2021; Schwarz al., 2021; Jiang et al.), most dynamic sparsity techniques share three key components: sparse model initialization, model weight optimization, and sparse weight adaptation. We explain these components in more detail below.
Algorithm Dynamic Sparsity
Require: Dense neural network 0, sparse Neural Network 0,, dataset {a,, y6 > sparsity binary masks: M {M*, ..... M² where [M''...,,' refer to the binary mask from layer to L, Adaptation Frequency AT, Adaptation Rate
1: # Sparse Model Initialization
2: M + SNIP(x;; s)
Using SNIP to generate sparse masks. Applying sparse masks.
3: 0, @ M
4: for each training step f do
5: # Model Weight Optimization
6: 0, +- AdamW(a: 0,)
7:
0, 5-
0. ④ M
Applying masks.
8:
if (t mod AT) = 0 then
9:
# Sparse Weight Adaptation
10:
Pruning p percentage of parameters using magnitude pruning Growing percentage of parameters in a random fashion Update the adaptation rate p with cosine decay schedule
11:
12:
13:
end if
end for
B. SPARSE MODEL INITIALIZATION
Starting training from a sparse model is fundamental requirement of dynamic sparsity (also known as sparse training). Therefore, it is crucial to construct the sparse model in a way that ensures both trainability and capacity are preserved. One critical aspect of constructing good sparse model is choosing the appropriate layer-wise sparsity ratio. This ratio determines the computational FLOPs (floating- point operations) of the sparse model and has a significant impact on its final performance (Evci et al., 2020; Liu et al., 2022a; Hoang et al., 2023).
Mocanu et al. (2018) first introduced Erdös-Rény: (ER) (Erdös Rényi, 1959) from graph theory to the field of neural networks, achieving better performance than the standard uniform sparsity ratio. Evci et al. (2020) further extended ER to CNN and brings significant gains to sparse CNN training with the KeerseeRnny-KKemel (ERK) ratio. Other prior arts start from a uniform sparsity distribution while allowing the distribution dynamically shift towards a non-uniform one during training. Liu al. (2022a) recently discover that the layer-wise sparsity ratio learned by SNIP (Lee et al., 2019) outperforms ER-based sparsity ratios with large-scale models on ImageNet. We confirm that SNIP ratio indeed achieves better trade-off between accuracy and FL.PPs than ERK in the
18

---

Published as a conference paper at ICLR 2023
context of large-scale ConvNets. Concretely, we first calculate the SNIP scores |g for each weight using one batch of training data, where w and g is the network weight and gradient, respectively. Then, globally sort all the scores across layers and force the weights with the smallest scores to be zero with respect to the target sparsity.
B.2 MODEL WEIGHT OPTIMIZATION
After initializing the sparse model, we start to optimize the model weights minimize the loss function. Dynamic sparsity is compatible with the most widely used optimizers. In this paper, we directly inherit the default training configurations and optimizers (AdamW) from ConvNeXt (Liu et al., 2022b) without any modifications due to its already promising performance. However, we do not exclude the possibility to introduce sparsity-specific components such as optimizers, normalizations, and activation functions further enhance the performance. After each update, multiply the weights with the current binary masks to enforce the sparse structure.
B.3 SPARSE WEIGHT ADAPTATION
Sparse weight adaptation the vital factor for dynamic sparsity outperform static sparsity (Liu al., 2021c). By dynamically adapting the sparse weights during training, it enables joint optimization of sparse connections and weights for more elaborate capture of local features. Trune-and-gee the most common way to achieve weight adaptation. Removing percentage of the existing weights and regrowing the same number of new weights explores new weights while is able maintain the parameter count fixed.
is
Weight Prune: Although there exists various pruning criteria in pruning literature, we choose the most common way for SLaK, that is, removing the weights with the smallest magnitude (Mocanu et al., 2018).
Weight Grow:
The most common ways grow new weights are random-based growth (Mocanu al., 2018) and gradient-based growth (Dettmers & Zettlemoyer 2019; Evei et al., 2020). We choose to grow weights randomly sampling following (Mocanu al., 2018).
Following Liu al. (2021c), we specifically tune two factors for SLaK-T that control the strength of weight adaptation, adaptation frequency f. and adaptation rate p. Adaptation frequency determines after how many training iterations we adjust the sparse weights, and the latter controls the ratio of the weight that adjust at each adaptation. We share the results in Figure 5. We empirically find that f 100 and = 0.3 works best for SLak-T and choose the same set of hyperparameters for SLak-S/B, without further tuning.
82.5
82.47
82.5
82.44
52.41
82.4
82.45
Accuracy
82.4
82,38
Accuracy
82.3
82.2
82.14
82.35
82.3
82
82.25
81.9
0.1
0.3
0.5
0.7
1000
2000
8000
Adaptation Rate
Adaptation Frequency
Figure 5: Left: Effect of adaptation rate the performance of SLaK-T. f is set as 100. Right: Effect the adaptation frequency f on the performance of SLaK-T, p set as 0.3.
C COMPARISON BETWEEN CSWIN TRANSFORMER AND SLAK
We examine if SLaK can rival CSwin Transformer (Dong et al., 2022), a carefully designed hybrid architecture with transformers and convolutions. performs self-attention in both horizontal and vertical stripes. To produce a hierarchical representation, convolution layers are also used CSwin Transformer expand the channel dimension. As shown in Table 7, besides clearly outperforming
19

---

Published as a conference paper at ICLR 2023
the peer ConvNets backbones (ConvNeXt and RepLKNet), SLaK models also perform par with CSwin Transformers, as pure ConvNet for the first time, without any bells and whistles.
Table 7: Comparison with the CSwin Transformer. All models are trained with A for 300 epochs.
Model
Image Size
#Param.
FLOPs
Top-1 Accuracy (%)
CSWin-T (Dong al., 2022)
224x224
23M
4.3G
82.7
CSWin-S (Dong et al., 2022)
224x224
35M
6.9G
83.6
CSWin-B (Dong et al., 2022)
224x224
78M
15.0G
84.2
CSWin-B (Dong et al., 2022)
384x384
78M
47.0G
85.4
SLak-T
224x224
30M
5.0G
82.5
SLak-S
224x224
55M
9.8G
83.8
SLaK-B
224x224
95M
17.1G
84.0
SLaK-B
384x384
95M
50.3G
85.5
D TRADE-OFF BETWEEN SPARSITY AND WIDTH
The principle of "use sparse groups, expand more" in Observation #3 essentially sacrifices network sparsity for network width. Therefore, there is trade-off between model sparsity and width. To have better understanding of this trade-off,
choose 5 combinations of (Sparsity, Width factor), i.e., (0.20, 1.1 x)x ) (0.40, 1.3x), (0.55, 1.5x), (0.70, 1.9x) and (0.82, 2.5x). All the settings have roughly conducted on SLaK-T, As
5.0M FLOPs but with different network widths. The experiments expected, the model's performance keeps increasing as model width increases until the width factor reaches 1,5x, after which increasing width further starts hurt the performance apparently due to the training difficulties associated with highly sparse neural networks.
Table 8: Trade-off between sparsity and width. The experiments are conducted on SLaK-T trained for 120 epochs on ImageNet-IK.
Model
(Sparsity, Width)
#Param
FLOPs
Top-1 Accuracy (%)
(0.20, 1.1x)
29M
5.0G
81.3
(0.40, 1.3x)
30M
5.0G
81.6
SLaK-S
(0.55, 1.5x)
30M
4.9G
81.7
(0.70, 1.9x)
32M
5.0G
81.6
(0.82, 2.5x)
32M
5.1G
81.3
E EFFECT OF THE SHORTER EDGE N ON SLAK
In Table 9, we report the effect of the shorter edge on the performance of SLaK. We vary the shorter edge [3, 5, 7] and report the accuracy. All models were trained with AdamW on ImageNet-IK for 120 epochs. We empirically find that N=5 give us the best results, whereas N = 3 and N has slightly lower accuracy. We hence choose N as our default option.
Table 9: Effect of the shorter edge N on SLaK. The experiments are conducted on SLaK-T trained for 120 epochs on ImageNet-IK.
Model
N
Top-1 Accuracy (%)
SLaK-T T
3
81.5
SLaK-T
5
81.6
SLaK-T
7
81.4
20

---

Published as a conference paper at ICLR 2023
Table 10: Quantitative analysis on the ERF with the high-contribution area ratio r. A larger value suggests smoother distribution of high-contribution pixels, hence larger ERF.
Kernel Size
20%
30%
50%
99%
ResNet-101
3-3-3-3
0.9%
1.5%
3.2%
22.4%
ResNet-152
3-3-3-3
1.1%
1.8%
3.9%
34.4%
ConvNeXt-T
7-7-7-7
2.0%
3.6%
7.7%
55.5%
ConvNeXt-T (RepLKNet)
31-29-27-13
4.0%
9.1%
19.1%
97.5%
SLaK-T
51-49-47-13
6.9%
11.5%
23.4%
97.5%
F ERF QUANTITATION OF MODELS WITH DIFFERENT KERNEL SIZES
In this appendix, we quantify ERF of various models by reporting the high-contribution area ratio in Table 10 following Ding et al. (2022). refers to the ratio of minimum rectangle to the overall input area that can cover the contribution scores over given threshold Assuming an area of A the center can cover 1 30% contribution scores of a 1024x 1024 input, then the area ratio of / 30% r = (A/1024) Larger suggests a smoother distribution of high-contribution pixels. We can see that with global kernels, SLaK naturally considers a larger range of pixels to make decisions than ConvNeXt and RepLKNet.
G LEARNING CURVE
here share the learning curve of different architectures on ImageNet-IK classification Figure Again, can observe that large kernels provide significant training loss gains (yellow lines and blue lines). Increasing kernel size from 31 x31 to 51 x51 further decreases the training loss with improved test accuracy. It is worth noting that even though SLaK's ' kernel size is large x1151, it enjoys very promising convergence speed compared to the models with smaller kernel sizes.
Training
accuracy
5.00
80
4.75
75
4.25
Test accurscy
70
Training
4.00
3.75
3.50
3.25
3.00
0
20
40
60
80
100
120
0
20
40
60
80
100
120
Training
Training
ConvNeXt-T (RepLKNet))
ConvNeXL-T
SLak-T
Figure 6: Learning Curve. Learning curves in terms of training loss and test accuracy of various models on ImageNet- 1K classification.
H INFERENCE THROUGHPUT MEASUREMENT
Although paper shows promising benefits of dynamic sparsity, unable to fully translate these benefits into real speedup due to the limited support of unstructured sparsity in commonly used hardware without sparsity-awand accelerations.
To provide full picture of this aspect, we measure the real inference throughput of SLaK- -T, Swin-T, and ConvNeXt-T with an A100 GPU with PyTorch 1.10,0 cuDNN 8.2.0. in FP32 precision. The results are reported in Table 11. On common GPUs without any sparsity- aware accelerators, the throughput of Swin-T and Con-NeXt-T is around 1,8x and 2.5x larger than SLaK-T. This difference
21

---

Published as a conference paper at ICLR 2023
acceptable, considering SLaK's x.3x wider model size and >7x larger kernel size, thanks to the kernel decomposition trick. Naively scaling the kernel size of ConvNeXt to 51 x51 without kernel decomposition will result in 8.5x slowdown compared to the original 7x7 kemnl..
Table 11: Inference throughput comparisons among ConvNeXI-T, Swin-T, and SLaK-T on ImageNet-IK. The results obtained A100 GPU with PyTorch 1.10.0 + cuDNN 8.2.0. in FP32 precision, using no sparsity-awan accelerator.
Models
Image Size
Kemel Size
Throughput
ConvNeXtt-T
224x224
7-7-7-7
1779
Swin-T
224x224
1305
SLaK-T
224x224
51-19-17-13
709
To better understand the effect of different components our training recipe on real inference speed (without sparsity-aware accelerators), we break down training recipe and report the corresponding inference throughput in Table 12. Generally speaking, sparse large kernels have comparable run times to dense large kernels, with neither acceleration nor overhead, as the masking operation of sparse kernels is essentially equivalent to dense matrix multiplication, which highly efficient on GPUs. However, decomposing large kernels can significantly increase the efficiency of kernel scaling, achieving a x speedup compared to naive kernel scaling. Also due to this reason, the throughput of 1.3x wider model is still 2x larger than vanilla large kernel model.
Table 12: Inference throughput of different large-kernel recipes on ImageNet-1K. The results obtained A100 GPU with PyTorch 1.10.0 cuDNN 8.2.0, in FP32 precision, using no sparsity-aware accelerator.
Model
Kernel Size
Naive
Decomposed
Sparse Groups
Sparse Groups + 1.3 width
ConvNeXt-B
51-49-47-13
283
282
198
I COMPARISONS WITH OTHER CHOICES FOR DECOMPOSITION
In this appendix section, we study the effectiveness of our large-kernel decomposition by comparing the following four large-kernel choices:
Naively stacking 10 small kernels with 3x3 size.
Sequential convolutions with 51 x5 and 5x51 kernels.
Parallel convolutions with 51 x5 and 5x51 kernels (same as SLaK).
Dilation convolutions with similar receptive field SLaK, i.e., we the kernel size of each stage [19, 17, 17, 5] with a dilation factor of 3.
To eliminate the effect of confounding variables, dynamic sparsity is not adopted here and additional 5x5 kernel is added for all settings. We can observe from Table 13 that naively stacking multiple small kernels and dilation kemnels achieve much lower accuracy than the decomposed approaches (parallel and sequential). And the sequential decomposition performs slightly worse than the parallel one by 0.1 %. The superior performance of kernel decomposition (either sequential or parallel) over other choices also supports the effectiveness of our recipe.
Table 13: Comparisons with other choices for large-kernel decomposition. All models are trained for 120 epochs.
Model
Stacking 3x3 kemels
Dilation Convolutions
Sequential 51x5 5x51
Parallel 51x +5x11
ConvNeXi-T
$1.1
81.4
$1.5
22

---

Published as a conference paper at ICLR 2023
By combining the 51x5 and 5x51 kernels through superposition. obtain restricted 51 x51 kernel that is primarily composed of weights arranged in a crossed shape. This restricted filter offers several advantages, including computational efficiency, improved locality, and most importantly, the large effective receptive field (ERF) similar to a 51 x51 kernel. The kernel decomposition idea is related to prior work self-attention decomposition in Transformers (Ho al., 2019; Tu et al., 2022). In those models, a global self-attention mechanism is replaced with a combination of local attention and sparse global attention, with axial attention that performs attention along different image axes. These approaches aim to achieve computational efficiency while preserving the global ERF. Our empirical results provide evidence that similar decomposed structure can be extended as useful design principle for ConvNets too.
J EFFECT OF LARGE KERNEL POSITION
In the main body of our paper, we follow the design of RepLKNett (Ding et al., 2022) and respectively scale the kernel size from [31, 27, 13] to [51, 47,47,47 13] for each model stage without careful design. In this section, we question the necessity and rationality of using large kernels at every stage. For instance, since carly stages usually learn low-level features such as edges, corners, basic local shape information, large kernels like 31 x31 or x11 might not be necessary in early stages. To verify this, we conduct an ablation study by naively scaling large kernels only in one or two stages. Dynamic sparsity and decomposition are not adopted to eliminate the effect of confounding variables.
The results are presented in Table 14. As we anticipated, solely scaling the kernel size to 31 x1x31 in the first and 29 in the second stage does not yield any benefits. Instead, large kernels are more crucial at later stages. Only increasing the kernel size in the third and forth stage achieve the same accuracy as when scaling kernels in all stages. Our results indicate that scaling the kernel sizes in the later stages is a better design option for efficiency and accuracy.
Table 14: Ablation study of the position of large kernels. All models are trained for 120 epochs.
[A, B, C, D] refers the kernel size at each model stage.
Kernel Size
7-7-7-7
31-7-7-7
7-29-7-7
7-7-27-7
7-7-7-13
7-7-27-13
31-29-27-13
ConvNeXt-T
81.0
80.89
80.94
81.31
81.18
81.48
81.5
K LIMITATIONS AND DISCUSSION OF BROADER IMPACT
The main limitation of this work is that the sparse architecture is implemented with binary masks due to the limited support of sparse neural networks by the common hardware such as GPU and TPU. Therefore, the inference FLOPs reported in the main paper are the theoretical values. Traditional works on structured sparsity (Han et al., 2015; Frankle & Carbin, 2019; Gale al., 2020; Liu et al.. 202 mainly focus on finding sparse subnetworks that can match the performance of their dense counterparts. The promising results in our paper go one step further and demonstrate the large potential of unstructured sparsity for scaling modern neural architectures. Once this great potential is supported the future, it can have a significant positive impact on our planet by saving huge amount of energy and reducing overall total carbon emissions. Although not the focus of this current work, would be interesting for future work to examine the speedup of sparse large kernels, using such specialized hardware accelerators, we much improvement room of promise here. For example, at high unstructured sparsity levels, XNNPACK (Elsen et al., 2020) has already shown significant speedups over dense baselines smartphone processors. Furthermore, number of companies and researchers are considering including unstructured sparsity support in their hardware (Liu et al., 202 lb; Gale al., 2020; Nvidia, 2020; Zhou al., 2021; Hubara et al., 2021). We hope our work can provide more motivation for such advances.
increasing
While our paper is solely scientific, others could build software with negative impact using our technique, like privacy leakage and faimess issues. Moreover, the promising results of sparsity could bring benefits as cost savings, but could also possibly result in a debate between dense NN hardware and sparse NN hardware, which might lead to resource waste.
23


Reviews:
[{"review_id": "wh9AyM6X0-", "reviewer": "Reviewer_h6xx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a mechanism for scaling up the receptive field of the convolutional kernels in CNNs, and for improving the performance without extra FLOPS via sparsity. Positive results are shown for several visual tasks.", "summary_of_the_review": "The method is simple and results are encouraging. However, I have concerns with novelty. Even if the method is a natural combination of known components, the paper could clarify / analyze why the specific set of choices is the \"right\" one to use, and provide more complete empirical validation. \n\n~In the current form, I believe the paper falls short for ICLR.~\n\n*Edit 11/28:* During the discussion phase, we have identified ways for the paper to be edited to call attention to the salient aspects, specifically one around the cross-shaped filters allowing a beneficial trade-off between the parameter efficiency and the receptive field size. Raising the score from 5 to 6.", "strength_and_weaknesses": "The paper proposes a simple way for scaling up the size of the convolutional kernels, and empirically demonstrates the effectiveness of this  on several visual tasks. The kernels can be scaled to a larger size than before, while boosting the performance. \n\nThe paper proposes a mechanism for introducing dynamic sparsity, which improves the performance at no extra cost in FLOPS.\n\nThe experimental results explore different training schedules, different tasks (classification, segmentation, detection), and (positively) compare to several SOTA architectures. \n\nIt was not clear to me what the main message of the paper is, other than that it is possible to train CNNs with large kernels. For example, is it the case that the best practice is to use such models? If so, I would have liked to see experiments on more data and tasks, including ones with large-data pretraining and possibly transfer learning (as with e.g. the VTAB suite of benchmarks). Comparing to more architectures, with results perhaps presented as a pareto frontier in the cost / performance plane, would be instructive. \n\nIf the main message is the fact that sparsity is important, this should be combined with the analysis of the actual cost of sparse inference, which (as the paper mentions) cannot be measured just using FLOPS. \n\nIf the message is that a larger receptive field could be achieved by a parallel combination of non-square filters, this seems known (since at least the Inception models). I should also note that while the paper refers to 51x51 filters, it is my understanding that it considers a restricted form of such filters, which are cross-shaped. It would be useful to understand why such a shape is a good one to use -- as opposed to e.g. convolving with the two filters sequentially rather than in parallel which would give something closer to a square filter, or atrous convolutions which provide a sparse coverage of a large receptive field.\n\nI was looking forward to understanding why previous work did not use larger kernels, and the first sentence of sec. 3 seemed to promise 3 reasons for it, which I did not find.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gJNyVZHP-Ka", "reviewer": "Reviewer_oAn8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a large kernel ConvNets which scaled up the kernel size beyond 51x51 up to 61x61 upon the RepVGG-like ConvNet baseline. The authors progress the idea from ConvNeXt and show the difference between the recently published RepLKNet (CVPR 2022) with the proposed model by introducing the concept of dynamic sparsity of weights consisting of sparse initialization and sparse weight training. This goes with the weight pruning and growing (i.e., adding new ones) processes during training to reach a sparse final network. This seems to be a key idea for expanding the kernel size beyond 31x31; another is decomposing a square kernel into a rectangular one (i.e., from a single NxN kernel to two successive kernels, NxM and MxN). Finally, utilizing the model with a reduced FLOP can give an opportunity of extending the network width, so it becomes a final ingredient of the proposed kernel expansion method. The authors provide ImageNet-1k evaluation compared with ConvNeXt, ViT-based models including Swin, Cswin, and DeiT. Finetuning evaluations on ADE20k, Pascal VOC2007, and MS COCO are equipped to support the effectiveness of the proposed model. Some analyses are also provided.", "summary_of_the_review": "This paper shows the potential of using larger kernel sizes beyond 31x31 in the previous CVPR work. I'm concerned that the overall idea seems to be well organized with sub-ideas, but they are existing arts, so the novelty of this paper is limited. More specifically, one may imagine that combining a weight decomposition, training with an iterative weight pruning and growing process, and expanding the network width will naturally lead to the performance of a given network. Furthermore, the proposed method does not seem to contribute to improving the resultant model compared with the 31x31 kernel one (Table 2); but the final accuracy was reached rather leveraging the impact of the parameterization technique (Table 3). Finally, using the resultant model in practice with the trained mask is currently not straightforward for practitioners. I believe the work should need newer items and report either raised accuracies of 51x51 or 61x61 kernel models to support the main claim, so I am currently voting to reject this paper.", "strength_and_weaknesses": "Pros)\n+ Overall, the paper is easy to follow.\n\n\nCons & comments)\n- The novelty is limited. The following recipes to reach the goal are not new and widely-used prior arts:\n   - The kernel decomposition of a large kernel into smaller ones was proposed in Inception families, and many successors adopted the idea to realize efficiency or avoid overfit. \n   - Pruning and retraining (+growing) process similar to Iterative Magnitude Pruning (IMP) has also been widely used aiming the same purpose and is turned out to be very effective for maintaining precision. \n   - Increasing network width has also been a regular tweak to meet the computational demand and improve the model accuracy as well since WideResNet primitively introduced it.\n- Since the accuracy number in Table 3 looks promising, the accuracy gap between it and the one in Table 2 is quite large. I wonder if this is technically due to the parameterization trick in RepVGG and RepLKNet using (31x31 kernels). Specifically, Table 2 shows the performance boosts with the largest kernel size 61x61 are clearly marginal (compared with 31x31 and 51x51, +0, -0.1 for all the cases), so it seems that the proposed method does not produce a final network leveraging a larger kernel effect.\n- Latency or throughput is not reported. I understand the current hardware architecture may not support the kind of sparse matrix computation, but I believe this type of work should include a speed measure.\n- As the authors mention as well, the implementation of this work was done with weight masking, so using the model in practice is cumbersome, and the latency in practice may not beat the competitors'.\n- The authors claim that the lack of locality is the rationale behind the performance degradation after naive training with a large kernel size (e.g., 51x51 or 61x61). However, I respectfully disagree with the claim because there is little evidence, and a more valid reason is that the model was overfitted (in Table 1). Because the training was only done with 120 epochs which is a limited training epoch; and seemingly identical data augmentation (such as the fixed RandAug coefficients, weight decay parameters, and so on) was used for all the models, so the expanded weights could be easily overfitted under weak data augmentation training regime. I recommend the authors train it for more epochs with stronger data augmentations to justify the claim. Furthermore, further backups concerning the locality should also be provided.\n- The detailed procedure of the dynamic sparsity (in Figure 2) is not provided. How did the authors initialize the weights before training? How did SNIP use for giving sparsity - at a layer level? or channel level? How to tune the hyperparameters for scheduling the procedure, such as the interval of pruning/growing and the threshold of pruning?\n- Particularly, the dynamic sparsity may not be universally applicable for other large-kernel networks like this method.\n- How did the authors pad the feature map with such a large kernel (>56)? Zero-padded? If so, zero-padding may harm the performance due to an imbalance scale at the borders, so how did the authors handle it?\n- How did the authors use the pretrained models for finetuning? All the mask was fixed, and the corresponding weights also remained frozen?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c5YLE0MvKo", "reviewer": "Reviewer_X26i", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel convolutional network structure with very large kernel size, up to 51x51 or even larger. The key ideas is to replace the convolutional kerenl in ConvNeXt by three kernels of size mx5 / 5xm / 5xm respectively. The mx5 + 5xm provides a low-rank approximation to mxm kernel, which can be too large to be learned efficiently. The 5x5 kernel captures the local dependency more efficienctly. During the training, dynamic sparsity is used to adaptive prune-and-grow the kernel weights. Large-scale experiments on ImageNet show that the proposed network can outperform ViT models under the same params and FLOPs.", "summary_of_the_review": "This is a very strong work, with strong novelty and SOTA results.", "strength_and_weaknesses": "[Strength]\n\n* While using a large kernel size is not a novel idea, it is still the first work that scales kernel size to 51x51 without training difficulty. The most interesting design is the 5x5 small kerenl to capture locality.\n\n* The large-scale experiments show very promising results comparing to ViT-based models. Especially, it outperforms Swin-B by 1% at 384x384 resolution, only using convolutional operators.\n\n* Extensive experiments on ImageNet classification as well as multiple downstream tasks. All showing impressive improvements over baseline methods.\n\n* It is interesting that the rank-5 decomposition of the large kernel is chosen, instead of mx1+1xm. Also, the dynamic training is shown to improve the final accuracy while reducing the model size.\n\n[Weakness]\n\nAlthough this work already did lots of experiments, I would suggest a few more explorations to enlarge the impact of this work:\n\n* Can we use the proposed model in NLP problems?\n\n* What about stacking multiple 3x3 conv instead of using mx5+5mx+5x5 kernel?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IBa2gHkt8r", "reviewer": "Reviewer_YfwT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the use of large convolution kernels in CNNs, and demonstrates they can match of outperform transformer based architectures if appropriate allowances are made to reduce the parameter count (by factorisation and sparsity induction). Experiments on ImageNet classification, Semantic segmentation (ADE20K, COCO) and object detection (VOC2007, COCO) show the power of the proposed approach. ", "summary_of_the_review": "This work provides a new approach to making large-kernel convolutions in CNNs realisable. The paper is well written on the whole and the evaluation is good.", "strength_and_weaknesses": "### Strengths:\n\n- Clear idea nicely explained.\n- Good evaluation, comparing against a number of alternative state of the art models.\n\n### Weaknesses:\n\n- Some of the results might arguable lie within the bounds of error - whilst performaing a sensitivity analysis might not be possible, could the authors say something about learnability of the models? are they easier or more difficult to train than the alternatives?\n- The last paragraph of section 6.1 is probably going a bit far - whilst it is true that human vision is foveated, the way that a human looks at an image is somewhat different to how a CNN percieves it. (I'd personally just tone down the links to human vision here)\n- The statement on negative societal impacts is very naive and should be either removed or reconsidered (whilst I agree that this work is scientific, others will undoubtedly build on it and could build software with a negative impact using this technique - you should be open about this even though it is out of your control)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EwPInDWhSq", "reviewer": "Reviewer_rrkL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Recent works on modern ConvNets defend the essential roles of convolution in computer vision by designing advanced architectures and plugging large kernels. This paper pushes the extreme of kernel sizes, and proposed to build a pure ConvNet model that smoothly scales up the kernel size beyond 51×51. Both empirical results and scientific insights appear to be strong.", "summary_of_the_review": "Overall this paper looks good and the authors are encouraged to provide more discussions for the performance on real hardware.", "strength_and_weaknesses": "This is a strong empirical paper studying a highly unusual architecture. This paper did an excellent job in solidifying their approach, validating a large variety of datasets/tasks, and delivering the performance that one expects. I am hence positive overall. The main strengths of this paper are enlisted as follows:\n- The problem considered is very interesting: how much gain one could squeeze from ConvNets, by purely enlarging this kernel size. Provided with the recent trend of large-kernel CNNs, scaling kernel size up to 51x51 or even 61x61 was still rarely discussed before. \n\n- Asking this question is also scientifically meaningful: while ViTs seem to dominate vision now with superior performance, it is unclear which truly matters: the larger/global receptive field or the sophisticated adaptive attention weights per location. The observation presented in this paper, that a ConvNet (which sticks to translation-invariant kernels) with (unprecedently) large kernel size outperforms best ViTs, provides a much clearer answer to this understanding. \n\n- The authors show that naively enlarging kernel sizes won’t work, and introduce two tricks: two-way factorization, and dynamic sparsity with more width – the second one is more non-trivial and leverages the recent progress in dynamic sparse training. Their recipe seems very concise and compact, avoiding any ad-hoc architecture modification, and seemingly insensitive to hyperparameters too. Tables 1 and 2 demonstrate whether adding each step of receipt would work or not alone, so that clarifies which part contributes how much to the final performance. \n\n- The performance gains are significant and consistent on ImageNet, surpassing some of the best available models (SwinT, ConvNext, RepLKNet) across four scales (T, S, B, and B with high-resolution 384 input). It performs comparably with the strongest hybrid model of CSwin Transformer.\n\n- The authors further evaluate on three dense prediction tasks: ADE20K, Pascal VOC, and MS COCO. The gains of large kernels become more obvious on (i) dense tasks; and (ii) high-resolution images. In particular, SLaK-T with larger kernels (51×51) further brings 1.2% mIoU improvement over ConvNeXt-T (RepLKNet), surpassing the performance of ConvNeXt-S. On MS-COCO, SLaK fine-tuned with Cascade Mask R-CNN also outperforms the same fine-tuned ConvNeX-T by around 1% in all AP metrics. \n\nThe weakness is perhaps that the model is not as advantageous on real hardware. It’s encouraging to see the authors reported some real hardware measurements in Section 6.2, but those are 4x compared with the dense large-kernel networks (which are supposed to be extremely slow), not the off-the-shelf-competitors such as SwinT or ConvNext. \n\nConsidering the highly unconventional architecture that this paper innovates with, I won’t hold this point as a fatal weakness against this paper. However, I do suggest the authors to openly discuss this issue and acknowledge their real hardware speed limitation w.r.t. existing competitors, so that readers would get the full picture right for selecting models. A further interesting question is whether the authors could consider co-designing their novel NNs with some customizable hardware such as FPGA.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster