Designing Concise ConvNets with Columnar Stages
Authors: Ashish Kumar
Jaesik Park
Venue: ICLR 2025 Poster (2025)
Keywords: Convolutional Neural Networks
Columnar Stages
Input Replication
Image Classification
Detection
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A simple and accurate ConvNet backbone for resource constraints scenarios

Abstract: In the era of vision Transformers, the recent success of VanillaNet shows the huge
potential of simple and concise convolutional neural networks (ConvNets). Where
such models mainly focus on runtime, it is also crucial to simultaneously focus
on other aspects, e.g., FLOPs, parameters, etc, to strengthen their utility further.
To this end, we introduce a refreshing ConvNet macro design called Columnar
Stage Network (CoSNet). CoSNet has a systematically developed simple and
concise structure, smaller depth, low parameter count, low FLOPs, and attention-
less operations, well suited for resource-constrained deployment. The key novelty
of CoSNet is deploying parallel convolutions with fewer kernels fed by input
replication, using columnar stacking of these convolutions, and minimizing the use
of 1×1 convolution layers. Our comprehensive evaluations show that CoSNet rivals
many renowned ConvNets and Transformer designs under resource-constrained
scenarios. Pretrained models shall be open-sourced.

Published as a conference paper at ICLR 2025
DESIGNING CONCISE CONVNETS WITH COLUMNAR STAGES
Ashish Kumar
Jaesik Park*
ScoreLabsAI Atlanta, USA
Scoul National University Scoul, South Korea jaesik, parkesnu.aas ac. kr
ashi shskumaa r @egmmmll... comm com
ABSTRACT
In the era of vision Transformers, the recent success of VanillaNet shows the huge potential of simple and concise convolutional neural networks (ConvNets). Where such models mainly focus on runtime, it is also crucial to simultaneously focus on other aspects, e.g., FLOPs, parameters, etc, to strengthen their utility further. To this end, we introduce refreshing ConvNet macro design called Columnar Stage Network (CoSNet). CoSNet has a systematically developed simple and concise structure, smaller depth, low parameter count, low FLOPs, and attention- less operations, well suited for resource-constraing of CoSNet is deploying parallel convolutions with fewer kernels fed by input replication, using columnar stacking of these convolutions, and minimizing the use of 1 convolution layers. Our comprehensive evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-coostrainee http///aiiikammaaa2222222
deployment. The key novelty
scenarios. Code: https:
INTRODUCTION
In the past decade, there has been enormous study in the neural network architectures Krizhevsky cet al. (2012); (Simoyyan & Zisserman (2014), demonstrating that different information paths He et al. (2016): Huang et al. (2017); Szegedy et al. (2015) Tan & Le (019)); Xie et al. (2017) can affect the performance. However, as highlighted in recent VanillaNet Chen al. (2023), due to the increased network complexity, the primary source of runtime bottleneck would be the off-chip memory traffic apart from the main computations because GPUs constantly becoming more powerful.
The issue is prevalent in more advanced models, such as ConyNext et al. (2022). CoatNet Dai al. (2021b), VIT Dosovitskiy al. (2020), etc., due the indirect information paths the attention mechanism that requires frequent memory reordering. Hence, despite these models being far ahead of their simpler counterparts He et al. (2016); Krizhevsky al. (2012), there are still opportunities to develop concise models for better accuracy, runtime, and resource tradeoffs.
Efforts in this direction are noteworthy. For example, ReP Ding al. (2021) improves runtime via structural parameterization. ParNet Goyal et al. (2021) reduces depth by utilizing multiple shallower network modules. Recent VanillaNet while avoiding branches. These works fall in the paradigm of simplifying ConvNet models for
et al. (2023) merges layers during inference
resource-constrained scenarios, in contrast to the advanced ConvNets Dai et al. (2021b); Liu et al. (2022). or ViT Dosovitsskiy et al. (2020) focusing on state-of-the-art accuracy.
We are inspired by the utility of the former class of works, i.e., simpler and concise models. However, besides focusing on runtime or depth Chen et al. (2023); Ding et al.) (202T); Goyal et al. (202T). we also focus on other ConvNet aspects, such as FLOPs, parameters, depth, computational density, etc. To this end, we propose concise model by revisiting the fundamentals of prominent ConvNet designs and define the following key sub-objectives:
1) Reducing depth: Network depth refers to the number of layers stacked. More depth means more sequential operations, thus more latency and wastage of parallel computing elements (GPU cores). 2) Controlled parameter growth: Reducing depth to achieve lower latency leads increased number of parameters Chen et al. 2023) Goyal et al. (202T), thus necessitating parameter control
* Corresponding author

---

Published as a conference paper at ICLR 2025
3
3<3
lel. Pepars
Tel
Black
TeLPH
Do Pad IND
TCT Bestwies
[
[[]]]] Pool
Xul Doard
ne po
Tel, oepand
Inl Squesoe
fal Squkarw
Fei was
141
10
30
tel
unpuen
Scueame
(a) VGG (b) Inception
(c) ResNet
(d) ResNeXt
(e) ConvNeXi
(f) RepVGG
(z) Group Conv
Figure 1: Design of various representative architectures in the order of their development in the timeline from (a)
to (c). Each graph represents stage of a network operating particular resolution.
while having short depth.
3) Low branching: Network branching increases memory requirements to hold intermediate tensors and also increases memory access cost account for the branched operations.
4) High computational density: A layer must have a high computing density since fewer computations per layer waste the parallel computing cores, e.g., depthwise convolutions Howard al. (2017) have less computation density and high memory access cost compared to the dense convolutions Simonyan Zisserman (2014).
5) Uniform primitive operations: Maintaining a uniform convolution kernel size throughout the net- work and branches is desirable so that computations can be packed into minimum GPU transactions.
This leads to concise refreshing ConvNet design (Figure 2 that shows enhanced performance in various aspects, such as low memory consumption, low memory access costs on parallel computing hardware, smaller depth, minimum branching, lower latency. low parameter count, and reduced FLOPs. The key attributes of CoSNet-unit are parallel columnar convolutions (Sec. (322). input replication (Sec. 3.3). and shallow-deep projections (Sec. 3.7). allowing CoSNet to perform better than simple ConvNets or rival the advanced designs. The achievements of CoSNet emphasize simplicity's importance in effective ConvNet designs.
RELATED WORK
This section provides an overview of representative network designs (Figure [p. The earlier Con- vNets (Krizhevsky et al., 20012] Simonyan & Zisserman 2014) stacked dense convolutions with an increasing number of channels and decreasing resolution (Figure la]. Improved versions (He et al. 2016 Szegedy al., 2013 Xie et al. 2017) achieve higher accuracy via manually designed blocks (Figure Ic). while (Howard et al. [217] Ma et al. 2018 Sandler et al. 20188] Zhang et al., 2018), use depthwise convolutions (Sifre & Mallat) for saving computations, but they are not memory friendly
DDing et al. 2021)
Netv have also grown from branchless (Krizhevsky et al. 2012, Simonyan & Zisserman
2014)
(Figure Ta to single branch (He et al. 2016) (Figure Tcp to multi-brande (Radosavovic et al. 2020
Szegedy et al. 2016 Tan & Le 20199] Zoph et al.) 2018) (Figure [b]. These models utilize 1 x
convolutions frequently, which rapidly increases network depth (He al. 2016 Sandler et al. 2018
Tan & Le 2019 Zhang et al. 2018) (Figure Ic le). Although beneficial, both large depth and high branching tend increase the latency, memory requirements, and Memory Access Cost (MAC)
(Chen et al. 2023) due to the scrialized execution of parallel branches (Ding et al.) 2021
Srivastava
et al. 2015
Tan & Le 2019).
2

---

Published as a conference paper at ICLR 2025
1xT.
Txl, Squertoee
Txl, Squesce
Input Roplication (Swc
1x1.
Column
3x3
Deep Projection
Parallel Convolutions. each with Ni komels
Rerrove all tsyers
Insenting parallei shallowndoup projections
Merging Panilel
Squeeee
T
eccept a New (Soc.3.1)
Convoiations (fwe.
N. 3x3, gehl
36
Txl, espand
Scc3
Sx3
Project
Shallow
N. 2x3, pnM
(S
Pooi
3x3 Pool
1al, Squessee
tel
te1. 1p
Fuse Onio (Be0.3.6)
txl, expand
txt, expand
(a)
(b)
(c)
(d)
Figure 2: Design evolution flow of CoSNet-unit. (a) A ResNet (He el al. 2016) stage with three blocks. (b) removing all 1 x convolutions except the first of the first block and the last of the last block. (c) detailed design of the CoSNet-unit by integrating design ideas into '(b)', and (d) final optimized CoSNet-unit from an implementation viewpoint.
Recent RepVGG (Ding al. 2021) proposes structural parameterization (SR) resolve the branching
issue. While ParNet (Goyal et al. 2021) and VanillaNet (Chen et al., 2023) reduce depth to achieve
lower latency. Efforts reduce depth increase the parameter count (Chen et 2023 Goyal et al.)
2021 to match the accuracy of relatively deeper counterparts (He et al.
2016).
Recent Vision Transformers (ViTs) (Dosovitskiy al. 2020 Liu et al.
2021 Touvron et al. 2021b). the ((N)) )-omplllle
2023
2021 have attracted huge research interests. As outlined in (Dai et al.
attention in ViTs is notable issue from a data size and eesource-constrinng
viewpoint. This issue
continues to inspire improvements in ConvNets. For instance, Repl.KNet (Ding et al. 2022) aims to bridge the gap between ViT and CNNs by employing large kernels.
The above designs focus on limited aspects, e.g., (He et al., 2016 et al. 2017) on the accuracy, (Chen et al., 2223] Goyal et al., (02T) on runtime and depth. To address this research gap, we draw inspiration from the success of VanillaNet-style networks, and instead of pursuing large-scale models, we focus on our sub-objectives (Sec T and revisit the representative ConvNets to push the frontier of simple, concise models.
COLUMNAR STAGE NETWORK
Our approach is series of improvements motivated by representative Cosigns To understand better, we begin with ResNet (He et al.) 2016) stepping stone as done in (Liu et al. (002). We design the building block CoSNet i.e., CoSNet-unit while recalling our sub-objective: 1) reducing depth. 2) controlled parameter count, 3) high computational density, 4) uniform primitive operation, and 5) low branching.
3.1 AVOIDING 1 x 1 FOR REDUCING DEPTH
The recent works reducing depth (Chen et al. count to achieve accuracy similar deeper network. However, aim to reduce depth while avoiding a large parameter count, which is difficult objective. Hence, we handle reducing depth and controlling parameter count separately.
2023 Goyal et al. 2021) increase the parameter
3

---

Published as a conference paper at ICLR 2025
To reduce depth, we identify that 1 x convolutions in the ResNet-like designs (Figure 2a) (He et al. 2016; Liu et al. 2022) etc., form almost 66% of depth without improving receptive field due to their pointwise nature (Luo al. 2016). Hence, use only two 1 x convolutions L, and Ly in a CoSNet-unit, where L, reduces the channel squeezing while l. performs expansion (Figure 2b). Then, we stack / number of x x convolutions, forming column sandwiched between Lg and Ly.
minimize the number of these layers. Specifically,
This strategy brings two benefits. First, it reduces the overall depth at the same receptive field, e.g.. three blocks of ResNet-like design have 9 layers with three receptive-field governing 3 x layers. In contrast, the proposed design only has layers, ie.. two x and three 3 x 3 conv, indicating a notable 45% depth reduction with the same receptive field.
Second, the reduced depth results in reduced FLOPs and latency e.g., CoSNet performs better than ResNet-50 at 50% fewer layers while having relatively fewer parameters, FLOPs, and latency.
3.2 PARALLEL COLUMNAR CONVOLUTIONS FOR CONTROLLED PARAMETERS.
We propose Parallel Columnar Convolutions to handle the large parameter count originating to compensate for the lost non- -ineeiiy due to the reduced depth (Chen et al. 2023). this design, first deploy M columns in parallel (Figure 2cp. and crosstalk among columns does not exist, i.c. a convolution of a column can only feed a convolution of the same column. Then, we restrict the number of kernels in convolution layer of column to small number of N. This design affects the number of parameters less aggressively when the number of columns increases (see ablations in the supplement). This is powerful feature of CoSNet design, offering controlled growth of parameter count during network scaling. This helps CoSNet achieve higher accuracy with fewer parameters.
The idea of the parallel column is based on our hypothesis that multiple kernels with fewer channels can be better than one with large channels. Having M convolutions in parallel with a smaller number of kernels N equivalent to synthesizing multiple kernels from large kernel. On the other hand, the idea of smaller N is motivated by the fact that many parallelly operating neurons tend to learn redundant representations while being computationally taxing and causing overfitting. For the same reason, EfficientViT (Liu al.) 2023) slices the input channels in its structure. Hence, by keeping N small, we expect to decouple the data patterns learned by the different columns.
In ConvNets, a similar idea was proposed in Inception (Szegedy et al., 2013), then in ResNeXt (Xie et al.) 2017), and then abandoned later as it caused inefficiency. For instance, Inception uses different- sizere convolutions and pooling in parallel, which must be exccuted serially despite being employed in parallel. Also, Inception differs from our columnar architecture since it does not have columns deep as CoSNet.
3.3 INPUT REPLICATION
In CoSNet, all the columns are fed with replicas of the input. We achieve that via a simple Inpur Replication IR operation (Figure 2c), which transforms tensor E XX into duplicated one (M( where M denotes the desired number of the columns. In the CoSNet-unit, the IR is applied over the output of the L, layer to feed each column with the input replica.
Input replication has also been employed in the earlier ResNeXt (Xie et al. 2017). but notable differences exist. ResNeXt has multiple blocks per stage, and each block performs IR, as shown in Figure ld. Whereas CoSNet performs IR only once. In ResNeXt, IR is performed before x squeeze layer, whereas in CoSNet, done after the squeeze layer.
The parallel columnar organization may seem to overlap with widely explored group convolutions (Xie et al. 2017 Zhang et al., 2018). However, there are two key differences. First, group convolution divides the input channels, thus defying the objective of IR because now each column receives only subset of the input channels, thus less information per group, as shown in Figure Ig On the contrary, CoSNet uses IR, which feeds each column with the replica of the input, thus making the entire input information accessible to each column. This becomes one the reasons that despite infrequent fusion (Sec. 3.6). unlike group conv, CoSNet still performs better (See ablations in the supplement).
4

---

Published as a conference paper at ICLR 2025
3.4 UNIFORM KERNEL SIZE FOR HIGH COMPUTATIONAL DENSITY & UNIFORM PRIMITIVE OPERATIONS.
The parallel columns of a CoSNet-unit can be executed independently; however, this design can be optimized further if all the convolutions in all the columns have uniform kernel size. To this end, we first set the kernel size in all the convolutions to & k, where & Rg3- Then, we combine the convolutions of different columns lying at the same level, i.c., the first convolution of each column is combined into one convolution having M batches.
With this optimization, all columns (Figure 2c can be efficiently processed using GPU-based highly optimized Batched-Matrix-Me MMtrrx-MMultpple routines, leading to increased computational density, increased GPU utilization, reduced memory access cost (Ding et al., 2021). and minimized GPU load-dispatch transactions. Thus resulting in a simplified CoSNet design (Figure 2d). Moreover, since CoSNet-unit is made up mostly of x 3 convolutions, it well suits the convolution hardware accelerators because they have dedicated support for them, and more chip area can be dedicated to 3 computational units.
3.5 BATCHED PROCESSING FOR MINIMAL BRANCHING.
From the previous step, batched processing yields additional benefits, i.e., CoSNet becomes uni- branched regardless of training and testing. This reduces memory consumption and access costs, resulting in lower per-iteration training time and increased parallelization. This contrasts with RepVGG (Ding al. 202T), which has considerable training time. Regarding ASIC development, low branching in CoSNet leaves more area on the chip because of the reduced memory requirement to store intermediate tensors. This area can now dedicated to more computational units.
Although the multi-branch design is beneficial for achieving high accuracy (Ding et al. 2021) (Fig- ure Lt.. CoSNet, despite having minimal branching, effortlessly achieves high accuracy. This because the core design of CoSNet-unit posses multiple branches in the form of columns and short projections (FigurePc) However, due to batched processing CoSNet-unit mimics uni mribcrnn be- havior. In this way, CoSNet takes advantage of both worlds, i.e., eliminated train time complexity due to multiple branches and fast inference during test time without needing structural parameterization (Ding et al. 2021)
3.6
FUSE ONCE
Finally, the output of all the columns is fused by Ly. In ResNeXt (Figure Idj. the output of x convs are fused immediately via 1 x 1 conv, whereas in CoSNet, it is done much later. Our fuse once strategy different from group (Zhang et al. 2018) or depthwise convolutions (Howard et al.. 2017) that are followed by 1 x (Figure gg) to avoid loss of accuracy because each group/channel has too few connections which restrict its learning ability without frequent fusion ((Zhang et al.) 2018), Figure Ig). This increases network depth and, hence, latency. On the contrary, CoSNet free from this constraint because we increase N as we go deep in CoSNet unlike (Zhang et al. 2018). Hence, each neuron in M columns has sufficiently large number of connections that enable learning without frequent fusion. We performed an ablation (see supplement) by applying the same strategy as Figure Ig in CoSNet. We observed increased network depth, latency, and decreased accuracy.
Pairwise Frequent Fusion (PFF): Although we aim to reduce I x I layers they have high concentration of most of the network pa- rameters and FLOPS (Sec. () we propose a frequent fusion scheme via 1 1 while avoiding the parameter and FLOPs concentration issue. In this scheme. instead of fusing all the columns simultane- ously, we fuse columns only pairwise via 1 1 (Figure B This strategy essentially offers several benefits. Firstly, with pairwise fusion, I kernel incorporates only a few computations per layer due to small kernel size (fewer channels) while improving network accuracy. Secondly, the latency incorporated due to these layers does not increase the overall lateney because of the few computations per layer, hence offers better accuracy with negligible latency overhead (1 -1-20 1-2ms). See Table [ We denote all such CoSNet variants CoSNet-PFF.
SUO
541
Figure 3: Illustration of Vanilla Frequent Fusion (left) (Zhang
jtt al. 2018), Figure [g) and Pair- wise Frequent Fusion (right).
5

---

Published as a conference paper at ICLR 2025
3.7 PROJECTIONS
To facilitate better gradient flow during network training, we employ projections introduced by ResNet (He al. 2016) but slightly differently in two ways:
1) Shallow Range. These projections are formed between any two layers of a column and promote better gradient flow through the stack of layers (Figure 2c). Since such projections connect only two layers, unlike a stack of layers in ResNet-like designs, these are named shallow ranges.
2) Deep Range. These projections are formed between the input and the output of a CoSNet-unit. Specifically, the input to CoSNet-unit is projected to its output via a 3 x 3 pooling layer followed by 1 convolution Lp whose output fused with the output of Ly (Figure 2cp. The pooling operation gathers spatial context by enlarging the receptive field, which is otherwise impossible for Lp alone due to its point-wise nature. We call it deep projection because it bypasses the entire columnar structure while combining information from the previous network stages, i.e., multi-layer information fusion, and providing a short alternative path for gradient flow.
The above projection design helps achieve CoSNet better accuracy (see ablations) and is slightly different from the existing ones. First, projection in ResNet-like models (He et al. 2016 Xie et al.) 2017) is used only in the first block of stage (shallower), and projection between stages does not exist. Second, projection in these models operates at a stride 2. On the contrary, in CoSNet, the projection connects two stages (dceper) while operating unit stride and utilizing pooling to increase the receptive field.
3.8 COSNET INSTANTIATION
A CoSNet variant can be instantiated by stacking CoSNet-units (Figure 4. CoSNet does not have the notion of blocks but only has stages in the form of CoSNet-unit. This contrasts with existing ConvNets, which have stages, and each stage comprises multiple blocks (Goyal et al., 2021; He et al.) 20166] Liu et al. 2022, Xie et al. 2017 e.g.. ResNet-50 has four stages, having 3, 4, 6, and blocks respectively (Figure Ic Tep.
Favn 3e3 and
Fran, Jaler
Pail
Blage
Stage
Mocs
Rtass
Waye4, OotWe
To instantiate a CoSNet variant, we follow the tradition of five stages (He et al. 2016 Simonyan & Zisserman (2044), among which the first (stem) while the remaining are the CoSNet-units.
(a) Existing Models
CuSNet
3 convolution with stride of 2,
Figure 4: Macro design of (a)
ing networks e.g. DDing et al. (202T):
He et al. (2016) Liu et al. (202); Xie al. (2017). and (b) CoS Net. CoSNet does not have blocks in its stages.
Following ResNet (He et al. 2016). set channels of L, 64, which gets doubled at each stage, while the channels of Lp and Lf always equal & times the channels of Ly. We set { 4, following (He et al., 2016). To further simplify the instantiation, we set the depth of column, i.e.. in K CoSNet-unit equal the number of blocks in the k stage of a widely used model ResNet-50 (He et al.) (016)). Summarily, CoSNet-unit has only three hyperparameters: M.N./ which control CoSNet's parameters, depth, latency, and accuracy. Hence, different CoSNet variants can be constructed by changing them. Please refer to the supplement for CoSNet instance names and ablations on M, I.
4 EXPERIMENTS
We evaluate CoSNet on ImageNet (Deng et al. 2009) dataset consisting of 1.28M train and 50k validation images of 1000 categories. Our training methodology consistent with recent Vanil- laNet (Chen et al., 2023). We use data augmentation techniques in (Chen et al. 2023 Liu et al. See the appendix at the end of this paper for more details.
2022..
4.1 ADVANCED CONVNETS AND VISION TRANSFORMERS
CoSNet recent EfficientViT (Liu et al. 2023 As shown in Table and Figure 5 CoSNet is less deep and runs 60% faster than Efficient VT Transformer while exhibiting better accuracy, e.g.. EfffiiietVittMe vs CoSNet-AO. EfficientVit is another example of lower FL.PPs that do not guarantee

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation of CoSNet ImageNet al (2009).
Latency is measured with batch size 1. 'SR' denotes structural parameterization. 'PFF" stands for pairwise frequent fusion. See See 3.6 for details.
Architecture
Type
#Depth
#Parums
FLOPs
Latency +
Top-1 (8)
ResNet-18
He cl.
2016
ConvNet
11.6M
1.83B
4ms
250
71.1
ResNet-34
et
2016
ConvNet
34
21.7M
3.68B
8ms
125
74.1
ResNet-50
He
2016
ConvNet
25.5M
4.12B
1ims
90
76.3
ResNet-101
2016
ConvNer
44.5M
7.85B
67
ResNet-152
He
2016
ConvNet
152
60.1M
11.50B
15ms
67
77.8
ResNeXt-50 et
2017
ConvNet
25.1M
4.40B
11ms
90
77.4
ResNeXt- Xie
ConvNet
44.1M
8.108
14ms
71
78.4
EfficientNNt- 80 Tam &
2019
ComNNet
49
5.3M
0.408
8ms
125
75.1
RegNecX-12GF Radosavovic et al.
2020
ConvNet
46.0M
12.10B
13ms
77
80.5
RepVGG-A0
Ding al.
2021
ConvNet
22
8.3M
1.46B
4ms
250
72.4
RepVGG-A0 NONGGG-AOTG
Ding of al.
2121
w/o SR
ComvNet
22
9.1M
1.518
8ms
125
72.4
RepVGG-A1 Ding et
2021
ConvNet
12.7M
2.36B
5ms
200
74.4
RepVGG-AI Ding al.
W/o
ConvNer
14.0M
2.638
7ms
74.4
RepVGG B0
Ding
ConvNet
14.3M
3.408
5ms
200
75.1
RepVGG-B0 Ding a.
2021
w/o SR
ConvNet
28
15.8M
3.06B
7ms
143
75.1
RepVGG-A2
Ding cl
2021
ConvNet
25.5M
5.12B
7ms
143
76.4
RepVGGi-A2 Ding
W/o
ConvNer
RepVGG-B3 Ding
ConvNet
110.9M
26.20B
17ms
58
80.5
RepVGG B3 Ding al.
2121
w/o SR
ConvNet
28
123.0M
29.108
22ms
45
80.5
ParNet-L Goval
ConvNet
55.0M
26.70B
23ms
43
77.7
ParNet-XI (ooyl
ConvNet
85.0M
41.50B
25ms
40
78.5
DeiT-S Touyron et al.
2021
Transformer
48
22.0M
4.608
15ms
66
79.8
Swin-T et 2041
Transformer
28.0M
4.50B
20ms
50
81.1
VITAE At
Transformer
23.6M
5.60B
24ms
41
82.0
CoANNet-0 Dai at.
2021b
Hybrid
64
25.0M
4.20B
15ms
66
81.6
ConvNeXi T
2022
ConvNer
29.0M
4.508
77
ComvNextV 2.P Woo
2123
ConvNet
41
9.1M
1.37B
11ms
90
79.7
Con-Nex1V2-NX
Woo et al
0025
ConyNet
47
15.6M
2.45B
13ms
77
81.2
Con-NextV2-T
Woo al.
2025
ConvNet
59
28.6M
4.47B
16ms
62
82.5
EfficienVViT-M4
Liu
2023
Transformer
8.8M
0.30B
166
74.3
Efficien
Lau et
2025
Transformer
70
12.4M
0.60B
7ms
142
76.8
VanillaNet Chen al.
Chenn
2023
ConvNet
32.0M
6.00B
6ms
167
76.3
VanillaNer-8
212.5
ConvNer
79.1
VanillaNet-9
hen
2023
ConvNet
41.4M
8.60B
6ms
167
79.8
VanillaNet-1 10
Chen al.
2023
ConvNet
10
45.7M
9.408
7ms
142
80.5
InceptionNeXi-8
ConvNer
18ms
UniRepLKKett-S
Ding
ConvNet
56.00 OM
9.10B
23ms
43
83.9
CoSNet-AO
ConvNet
26
8.8M
1.25B
6ms
167
77.1
CoSNNt-AA
ConvNet
12.
1.70B
6ms
78.2
CoSNet-BO
ConvNet
19.8M
3.05B
7ms
79.5
CoSNet-BI
ConvNet
26
22.0M
3.50B
7ms
167
79.9
CoSNet-B2
ConvNet
26
30.0M
5,10B
9ms
111
81.3
CoSNet-CI
ConvNet
24.4M
4.12B
7ms
80.0
CooNNetCC2
ConvNet
38.9M
7.09B
90
82.1
CoSNet-A1-PEP
ConvNet
38
12.7M
1.938
7ms
143
79.7
CoSNet-Bo-PFF
ConvNet
21.8M
3.44B
8ms
80.6
CoSNet-BI-PFF
ConvNer
CoSNetB2-PFF
ConvNet
34.3M
5.918
10ms
100
82.7
CoSNet-CI-PEF
ConvNet
42
27.3M
4,75B
8ms
125
81.3
CoSNet-C2-PFF
ConvNet
44.5M
8.27B
13ms
77
83.7
lower latency. Even the CoSNet-Al-PEF variant is still relatively shallower than EfficientVit while delivering better accuracy.
CoSNet DeiT (Touvron et al. 2021 From Table T CoSNet-B1 1 is almost 50% less deep, has 23% fewer params, and runs 60% faster than DeiT Transformer while exhibiting slightly better accuracy. With PFF, CoSNet-BO-PFF performs better in terms of accuracy, depth, and runtime.
CoSNet yy advanced mid-range ConyNets and Transformers CoSNet-Bi is 72% less deeper, 55% faster, and 1.2% more accurate than the popular Swin Transformer (Liu et al. 2021 It is also 55%
7

---

Published as a conference paper at ICLR 2025
82
CaAINis-G Conpleett BEUSNIt-C2.
VITAE-S farnNeaVVA
82
CoSNe:
VITAS-S
KCSSNet82
VardiaaNe
Ca5hui-82
Accuracy (%)
78
80
Cosyet-A0
CaSNutAl
CoSNet-80+
[CSSNN-81 CoSNet PFF
DeITS
Resiex 50
Swins
ResNex- 101
VanilaNet-
MonilaaNet 10)
Parteet
ResNet-252
Accuracy (96)
78-
Venifaliet-9
VanillaNes-9
CoSNettAD
CaONut-A1
CosNe
Parket-i BesNeXt-S0
ResNext-105
Res
76
EfricentVITMM
NesNet-50
VGC-GG-A2
76
RepVEG A2
EmficlenyiTT
Ree
ResNet-101
JepVGG-B0
74
Repvss
74
Repy
Be
72
NepVGG-40
72
-RepVGG-AA
20
(a)
Parameters (M)
(b)
Parameter modulated Depth
EanoNesti22
82
CostNet 0"
ConvNests
CoSNet-22
CosNet
82
CiSNut-82
CoSNet-Cc C2
-onn-eett T-m
1CARNNet
WITE-S
Accuracy (%)
80
CoENeLBD
GoSRivAl
CosNes 81
DeITSS
VarilaNee
VendlaNee-
Accuracy (96)
80
VanillaNet-a
VanilaNect
Co5het-81 a
CoSNNtt
Reshext-101
DeiTs
Swin-T
78
ResNext 50
PepVGG Res 101
ResNet 152
78
Ao
CoSNet.40
ResNext-50
Reshett 152
Restle--10 10
76
E0's KE
Ses
76
RcpVis A2
Rep 80
Effientii
74
Diicmti Reshet-24
RopV6G
Restoe-36
72
RD
+Rpves AD
Reshett 18
RReNeet 18
0.0
2.5
5.0
7.5 FLOPS (B)
10.0
12.5
15.0
5
15 Latency (ms)
10
20
25
(c)
(d)
Figure 5: Comparing the proposed CoSNet with representative models. Models in ' and 'o refers to CoSNet and existing models respectively. CoSNet has lower parameters, lower FLOPs, while depth of CoSNet is not unnecessarily large. The size of the circle is proportional to the parameter count.
less deeper, 30% faster with slightly lower accuracy than the popular ConvNeXt (Liu et al. 2022).
Moreover, CoSNet-C2 rivals the latest ConvNext-v2-T (Woo et al. 2023) with similar accuracy but higher speed and smaller depth.
CoSNet-B2, CI, C2 models rivals advanced Transformers, such as VITAE-S (Xu et al. 2021 and hybrid models, such CoAtNet-O (Dai et al., (2021b). With similar parameter counts and accuracy, our models show faster inference speed. The competitive tradeoffs offered by CoSNet show the significance of concise models.
4.2 COMPARISON WITH STANDARD CONVNETS
We show that CoSNet achieves efficiency in multiple aspects in large spectrum of models while being simpler during training and inference and offering competitive trade-offs relative to the rival network. See Table | for the comparison. Figure the trends regarding various aspects.
CoSNet vs recent VanillaNet (Chen al., 2023). CoSNet rivals recent ConvNet design, VanillaNet. VanillaNet is shallow and mainly focuses on latency. Our CoSNet-AO shows similar latency at fewer parameters, fewer FLOPs, and high accuracy compared with VanillaNet-6 (Table I.
CoSNet vs recent ParNet (Goyal et al. 2021). CoSNet outperforms recent non-deep ParNet that focuses on lower latency (Table R4). CoSNet is umi-branched while ParNet has multiple shallow branches which serialize the computations, thus making them deeper virtually.
CoSNet vs VGG (Ding et al. 2021). RepVGG offers a plain VGG-like (Simonyan & Zisserman 2014) structure via Structural Reparameterization (SR). However, its training complexity is high due to a large number of parameters and three branches at each layer (Figure If). Hence, we show its performance with and without SR.
Compared with the RepVGG family, CoSNet offers considerably lower complexity during training and testing, thanks to its parallel columnar convolutions. In addition, CoSNet has fewer parameters
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with VannilaNet Chen al. 2023 in training.
Archilecture
#Depths #Epochs4 #Paramss #FLOPs4
Top-1 (%vf
Train Time Per Epoch Train Time 300 Epochs
VanillaNct-66 Chen et al.
2023
6
300
32.0M
6,008
76.36
minutes
40 hours
VanillaNet-88
7,70B
79.13
minutes
hours
CoSNet-BI
26
300
19.8
3,05B
79.50
5 minutes
25 hours
Table 3: CoSNet with SE-like modules Hu ct al. (2018)
Appeoach
#Epoohs
#Deeth
#Params
AFLOPs
Top- (%)
ResNet-50- + SE Hu et 2018
120
50
28. UM
4.13B
76.85
ResNet-50 CBAM
2018
120
77.34
CoNNet-B1
120
26
19 2M
3.05B
76.77
CoSNet-BI Hu
2018
ResNet-50 + AFF Dai al.
2021a
160
50
30.3M
4.30B
79.10
ReaNet-50 SKN
2019
160
50
27.7M
4.47B
79.21
CoSNet-C1 SE Hu
2018
160
28
25.0M
4.13B
79.51
and fewer FLOPs while offering similar speeds with higher accuracy. For instance, CoSNet-B2 is better than RepVGG-B3 at similar depth, 73% fewer parameters, 80% lesser FLOPs while running faster. This shows the significance of parallel columns of CoSNet that during model scaling, parameter count does not grow rapidly.
CoSNet EfficientNet (Tan & Le 2019). Although we do not aim for mobile regime in this paper, we show that having fewer parameters and FLOPPs does not guarantee faster speeds. As shown in Table EfficientNet-Bi has 50% fewer parameters and 77% fewer FLOPs, but is 50% deeper, and runs 37% slower. By exploring the design space, CoSNet can be extended to the mobile regime.
CoSNet vs ResNet (He et al. 2016) family. As shown in Table [ CoSNet-AO is 6% more accurate, has 25% fewer parameters, shows similar runtime, and shows 319 % fewer FLOPs than ResNet- 18 although CoSNet has 6 more layers. Similarly, in contrast to ResNet-34, it is more accurate by 3% with 59% fewer parameters, 66% fewer FLOPs, and 23% less layers, while it is fast by 37%. ResNet-50 is the widely employed backbone downstream tasks (Carion et al. 2020 Goyal et al.. He et al. 2017 Ren et al. (0155) due to its affordability regarding representation power, FLOPs, depth, and accuracy. Table I shows that CoSNet-Bl surpasses ResNet-50 while being 50% shallower, 22% fewer parameters, 25% fewer FLOPs, and 40% faster.
CoSNet ys bigger ResNet (He et al., 2016) and ResNeXt (Xie al., 2017) models. As shown in Table [ CoSNet-C1 is better than bigger variants of ResNet, which serves as backbones for outting-edge works (Carion et al. 2020 Li et al. 2022). Our CoSNet outperforms them in various aspects while being 72% and 82% less deep relative to ResNet-101 101 and 152, respectively. CoSNet also runs faster by 50% in 50% fewer parameters and FLOPs. In addition, despite being smaller than ResNeXt (Xie et al.] 2017), CoSNet-CI outperforms it in various aspects. Overall CoSNet-CI is 50% less deeper than ResNeXt-50 while running 50% faster at 6% fewer FLOPs, 2% fewer parameters while being more accurate. In contrast to ResNeXt- CoSNet-C2 is 75% % less deeper, 11% fewer parameters, 12% fewer FLOPs, and 35% faster at higher accuracy.
4.3 ADDITIONAL EXPERIMENTS
CoSNet has small training walltime. We provide an additional comparison with the recent ConvNet design, VanillaNet (Chen et al., 2023), under training settings. 22 that despite VanillaNet being a shallow network, has high training time. We speculate that the large number of channels in the deeper layers of VanillaNet slows down batch processing large batch sizes. In CoSNet, parallel columnar convolutions and controlled parameter growth in the deeper layers counter this issue, leading to lower training time.
CoSNet is seamlessly compatible with SE-like (Hu et al. 2018) modules. Table Table shows the results when CoSNet is used in conjunction with Squeeze and Excitation (SE) like modules (Hu et al.) 2018). outperforms recent attention mechanism (AFF (Dai et al. 2021a), SKNet (Li et al. 2019), and CBAM (Woo al. 2018)) applied to ResNet-50.

---

Published as a conference paper at ICLR 2025
Table 4: CoSNet in state-of oh-ttt-arttt Detection Transformers (DETR) Li al. 2022 @ 12 epochs setting.
Method
#Parrmm
4FPS
AP
AP5i
APr:
APs
APM
AP,.
DN-DETR-ReNNeC50
DN-DETR ResNec50 Li et C
2022
42.4
57.7
S6M
25
39.2
60.0
41.9
18.1
43.0
59.1
Image
ResNett50
CoSNet-BI
Figure 6: CAM SSrinivas & Fleuret 2019 visualizations. Notably, CoSNet attends the class regions more accurately than the baseline.
4.4 COSNET IN STATE-OF-THE-ART DETECTION TRANSFORMER
We apply CoSNet to offe--ff-the-ar object Detection Transformer, DN-DETR (Li et al., 2022) to demonstrate the effectiveness of CoSNet in the downstream task. We experiment on MS-COCO (Lin
al.
2014) benehmark and utilized DN-DETR's default training settings.
Table 4 shows that DN-DETR with CoSNet improves the inference speed and average precision com- pared to the DN-DETR with ResNet-50 backbone. By further optimizing the DETR hyperparameters, CoSNet can be configured to deliver better performance.
4.5 VISUALIZATION OF ATTENTION
To comprehend CoSNet's better performance, we investigate its class activation maps (CAM) on ImageNet (Deng et al. 2009) validation set. use CAM output from popular Full-Grad-CAM (Srinivas & Fleuret 2019) for given class. CAM visualizations of ResNet-50 and CoSNet-B) are shown in Figure 6) It can be seen that CoSNet, despite being 50% shallower than ResNet, is better at learning to attend regions of the target class relative to the baseline.
CONCLUSION
We propose CoSNet, which revisits ConvNet design based on multiple aspects for concise models. CoSNet is based on our parallel columnar convolutions and input replication concepts to be efficient parameters, FLOPs, accuracy, latency, and training duration. Through extensive experimentation and ablations, we show that CoSNet rivals many representative ConvNets and ViT's such as ResNet, ResNeXt, RegNet, VGG and ParNet, VanillaNet, DeiT, EfficientViT while being shallower, faster, and being architecturally simpler.
Future work. CoSNet is open for improvement. In this paper, we have built a simple template architecture that can further evolve like ConvNext (Liu et al. 2022). For instance, comprehensive design space of CoSNet including mobile regime can be explored, similar to RegNet [Radosavovic et al. 2020). Besides, layer merging post-training, shown in VanillarNet (Chen et al.] 2023), can be utilized to develop shallower variants of CoSNet. addition to that, CoSNet can also be married with Transformer attention mechanism like DDi al. 2021b) (Liu et al. 2023
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. Jaesik Park was supported by MSIT grant (RS-2021-112111 AI Graduate School Program at Seoul National University (5%) and 2023R1AIC20078121 (95%)
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Hanting Chen, Yunhe Wang, Jianyuan Guo, and Dacheng Tao. Vanillanet: the power of minimalism in deep learning. Advances in Neural Information Processing Systems, 36, 2023.
Yimian Dai, Fabian Gieseke, Stefan Oehmcke, Yiquan Wu, and Kobus Barnard. Attentional feature fusion. In Proceedings of the IEEE/CVF Winter Conference on Applicatiins of Computer Vision, pp. 3560-3569, 202
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965-3977, 2021b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pp. leee, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13733-13742, 2021.
Xiaohan Ding. Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernlls to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11963-11975 2022.
Xiaohan Ding, Yiyuan Zhang. Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, and Ying Shan. Unireplknet: A universal perception large-kernel convnet for audio video point cloud time-series and image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 55244, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
Ankit Goyal, Alexey Bochkovskiy, Jia Deng, and Vladlen Koltun. Non-deep networks. arXiv preprint arXiv:2110.07641, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1xXx::/776.0267. 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
Jie Hu, Shen, and Gang Sun. Squeeze-and anneeeze-anndexiiaatiiioo networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.

---

Published as a conference paper at ICLR 2025
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. CVPR, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13619-13627, 2022.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 510-519, 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420-14430, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11976-11 11966, 2022.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 4905-4915 2016.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient architecture design. In Proceedings of the European conference computer vision (ECCV), pp. 131, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perforerformnnc deep learning library. Advances in neural information processing systems, 32, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 10428-10436, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28: 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Sifre and S Mallat. Rigid-motinn scattering for image classification. arxiv 2014. arXiv preprint arXiv: 1403. 1687.
12

---

Published as a conference paper at ICLR 2025
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1555 2014.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. Advances in neural information processing systems, 32, 2019.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 10Xiv:1505.00387. 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbbgnnew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe Le. Efficientner: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16133-16142, 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in neural information processing systems, 34: 28522-28535, 2021.
Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. Inceptionnext: When inception meets con- vnext. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5672-5683, 2024.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A COSNET INSTANCES
Table
A1 shows CoSNet instances configurations mentioned in the main paper.
Table Al: CoSNet instances Configurations.
Model
P:
N
M
Deppth
#aarras
#FL.P%
Ao
256
512
1024
2048
16
32
64
128
26
8.8M
1.25B
CoSNet-AI
256
512
1024
2048
16
32
64
128
3
6
4
26
12.1M
1.77B
CoSNer BO
256
512
1024
2048
32
64
128
256
3
6
3
4
26
19.8M
3.05B
Co
256
512
CoSNer- 82
256
512
1024
2048
32
64
128
256
3
6
3
4
16
26
30.0M
5.10
CoSNet-C1 CoSNet-C2
256
512
1024
2048
48
80
144
272
4
4
28
24.4M
4.128
256
512
1024
2048
48
80
144
272
3
26
38.9M
7,09B
B ABLATION STUDY
Varying M and N. Table A2 demonstrates the effect of varying and M (RO-R5). We first fix the values of N and vary M (RO-R5), and then vary M while fixing N (RO <+ R3, R1 <+ R4, R2 < R5). For fixed N, accuracy improves by increasing M, and the same effect is seen by fixing M while varying N. It can be noticed that parameters, FLOPs can be controlled by changing the M (RI ++ R2, R4 ++ R5), which directly reflects accuracy.
Effect of PCC. We compare instances having different N.M, but have similar parameters and FLOPs budget, for instance, R1 + R2, RI + R3, Table[A2] Noticeably, R2 with 5 PCC is better by 0.36% in aceuracy, only at 1.1M more parameters relative to R1. Similarly, only at 0.8M more parameters relative to R3. It shows that multiple PCCs facilitates improved accuracy in just fraction of parameters and FL.OPs. Moreover, comparing R9 (a deeper model) with R2, R2 achieves 0.13% more accuracy in 0.2M fewer parameters and 0. 17B fewer FLOPs. It shows the advantage of having multiple convolutional modules while being shallower.
better by 0.28% in accuracy,
Varying I. The impact of varying / is shown in R9, Table A2 It can be seen that going deeper is not necessary because a shallower version with same parameters (R2) is more accurate. Moreover, increased depth causes increased latency in R9. Therefore, stick to 20 40 layers of depth.
Group Convolution or ResNext-like Xie et al. (2017) Setting We also conduct additional exper- iments where each PCC is followed by x convolution done in group convolutions while keeping depth and parameters constant. We observe a % accuracy drop. This indicates that frequent fusion similar ResNeXt is not necessary.
Effect of Shallow Projections in PCC. R6-R9, Table A2 shows this analysis. For the shallower model, the residual connection shows only minor improvement (0.09%), however, for the deeper model, the effect of residual connections is noticeable (0.70%).
Effect of Deep Projections (DP). We train an CoSNet instance in three ways: First, remove DP entirely, Second, use DP without pooling, and Third, DP with pooling. See Table A2 for the analysis. can be noticed that without DP (R10), the model suffers with heavy accuracy loss of ~ 0.54% relative to when DP is used without pooling (R11). Moreover, when using with pooling (R12), accuracy improves, i.e. 1.22% and 1.76% relative R11 and R10, respectively, because pool ing provides more spatial context to the 1 x Lp layer by summarizing the neighborhood.
Effect of using very small N to compare with Group Convs Zhang et al. (2018) and depthwise Conv-like Howard et al. (2017) structure. From Table A2 R13-14, can seen that when in deeper layers, N is restricted to very smaller value while keeping parameters or FLOPs the same, accuracy decreases considerably. This because of the reason mentioned in the main paper (Sec."Fuse "Fuse-Once") that too few connections restrict the learning ability of neurons. Hence, they need frequent fusion similar to Group and Depthwise convolution methods, but it increases depth. To avoid that, increase N go deep down in CoSNet, which does not require frequent fusion due
14

---

Published as a conference paper at ICLR 2025
Table A2: Effect of parallel columnar convolution (PCC), of kernels N. # of layers 1, # of parallel convolutions M. and Deeper projections (DP). Values M,N,I are for each four CoSNet stages, Ablations are conducted at 120 epochs.
Row
M
#Dcpth
D2
Residual in 2CC
#Params
#FLOPs
Top-1 (%)
64
8.90M
1.25B
74.45
R1
16
32
64
128
4
6
3
4
4
26
V
12.1M
1.77B
75 65
76.01
R3
64
128
256
4
6
/
<
11.3M
1.65B
75.37
R4
32
64
128
256
4
6
3
4
26
/
19.8M
3.05B
76.76
R55
32
64
128
256
6
5
22.6M
3.51B
77.01
R6
32
64
128
256
4
6
26
/
/
X
11.3M
1,658
75.28
75.37
88
+332
64
1:88
256
5
20
x
13.4M
2,128
75.18
20
13.4M
R10
32
64
128
256
4
X
8.5M
1.298
73.61
R11
+2
64
128
256
4
6
26
wfo,, Pooling
9.8M
1,448
74.15
Foo" 1ng
9.8M
1.448
75.37
R13
+33
64
128
256
4
6
3
4
4
26
19.8M
3,51B
76.76
Table A3: Effect of batch size the baselines and CoSNet in the context.
Archisecture
Type
Batch Size
#Depth
#Params
FLOPs 4
Latency
FPS f
Top-1 (S)
EtthcientNet-BU
CoavNet
256
5.3M
125
EllicieniNet- 80
Tin & Le
2019
ConvNet
2048
49
5.3M
0.40B
8mns
125
77.1
EfficientVVT.MM
Transfommer
EfficientViT-Ms
Lil
Transfomerr
2048
70
12.4M
0.60B
7ms
142
77.1
CoSNet-AO
CoavNer
256
26
8.8M
1,25B
6ms
167
77.1
CoSNet-A1-PPFF
ConvNet
256
38
12.7M
1,93B
7ms
143
79.7
ConNNeXi
Con/Net
29.0M
77
ConvNeXii-T
Lau CL
ConvNet
4096
59
29.0M
4,50B
13ms
77
#2.1
Co8Net-C2
90
CoSNet-B:2PPF
ConvNer
256
38
34.3M
5.91B
10ms
100
82.7
sufficiently large number of neuron connections. Thus, we fuse only once, eliminating the need for fusion x layers, thus smaller depth and lower latency.
C THE EFFECT OF BATCH SIZES OF THE BASELINE APPROACHES.
In the literature, some baselines are trained with larger batch sizes (above 1024), but others have been trained at much smaller batch size (256). Therefore, we retrained high batch size baselines with 256 batch sizes avoid getting biased conclusions about the effects of large batch sizes. Such results with 256 batch size are carefully reported in Table
In this section, present the results of the baselines with larger batch sizes in Table A3 widely studied, the baseline approaches Tan & Le (2019); Liu et al. 2023 2022) show improved accuracy. Interestingly,
As
be noticed that CoSNet trained with 256 batch size can compete with -htt---f-hhe-rtt art approaches trained with a larger batch size. This shows the utility of obtaining higher accuracies in -soutre-constrained training scenarios (i.e., limited memory fit 4096 batch, etc.).
state-of-ther
D ADDITIONAL RESULTS
Table A4 shows results on RetinaNet x1 Lin et al. (007) detection pipeline. can be seen that, for comparable vision transformer backbone, CoSNet performs better. We also provide semantic segmentation results for the popular PSPNet pspnet semantic segmentation framework. It can seen that CoSNet performs better than the baselines.
15

---

Published as a conference paper at ICLR 2025
Table A4: CoSNet in RetinaNet x1 Lin al. 2017) object detection framework.
Method
#Depth
#Params
AP
APs
APM
AP:
EificiietViI-M4 CaSNet-A0
2023
8.8M
35.3
26
8.8M
34.3
19.1
38.0
49.1
Table A5: CoSNet in PSPNet Zhao
2017
semantic segmentation framework.
Method
#Params
mlotI
FPS
RepVGG-B1g2 2 Ding et 2021
41 11.36M
78.88
ResNet-50
25.5M
77.17
CosNet-B1
22.0M
79.05
E TRAINING SETTING
We train models in Py Torch Paszke et al. (2019) using eight NVIDIA A40 GPUs.
F PYTORCH CODE
All codes shall be open-sourced in PyTorch Paszke et al. (2019) post the review process. Here, we provide a code snippet CoSNet-Unit. Please see until the end of this document.
InputReplicator Modulel:
3
def init init__ (self, X):
super (itputtepplicator. __nit_inittt)
number of Parailel Columnar Convolutions self.M
8
def [ssward(self, ip):
10
x ip. repeat (1, self.M, 1, 1)
12
class CosMettnit Modulel:
15
def _imit_ (self, nLip):
super (Co9eetDii, self) init
17
self.n_op_ir
512, 1024,
selr.N
self.stride 2
selr.1
6, self.M 12 4, 47
22
P_op_ia int (self.n_cp_11
self. conv_ls nn Cnnv s_og_Is, bias-False) self.bn_ll la -n.n Rn..acchKorm26 (n_
self an. SILU (True)
29
self.18 Iapurkepllicttorrteol....
Limit the op of last POC layer so that the parameteza of the 1xI expansion layer # not overly large if number of columns very big # as a rule of thumb, we set it nearly equal to n_op /
34
tr_ondda__o___ss self.M) self.M
36
caff.cooy_pccc. (n.Moduleiisz(( f nn. Mo.Moullet.itiil self,ect_poc na (t.oodueetist
38
tan, tan. Conv2d self.M, M, self.N sel1.M, 3, self. 1, groupe=seif.M, bias-False)) self pc_pppn.pp.p.pp..cc...ccc_ooob wtt.ppp.ppeedddeeeeee....p. (an, satchlo seli.N)
self
43
self.sct_n waw._ppo.....eeeeeee__eeooooooe_ htcp/ppocc.ppppee.e (tn.St10
for in renge (self,1 21:
self. www._pcc.ccccccceeeooooooo appeedd...dd.............. (nn. (nn. convid self-M, self.N self-M, 3, 2, 1,
groupa=zelf.x bios-Palsel)
celf.ba_pce..e apw..c (na. Bacchhozm2diseef..u self.x))
self.act_ pact_ppcc.pppoee appedd (an, Situ (True))
16

---

Published as a conference paper at ICLR 2025
self. wwny_peceeeeeeeeeeeeeeeee ccw...c..aaeddd
append inn. con12d selÉ.8, self.eel5.n.op.peo 3,
groupaasel f.s.e.... f.M, bias-False)
selr.bn ww_.bon.ppppppeeeeeeeeee (an. (og.:oom222211....ooo_.____.___,
self selffact__pp app.peeee.dddmmmmmmmmmmm (na. (110 (True))
Convld (aolf ___oo_pee_lare self.n_op_lf, 1, 1, bias=False) selr.bn_12 1f nn nn.astchNormad (ael [in_op_ff]]
solf.conv...
58
self.ect Sn.D (True)
self. cony_lp ion self self-n_op_Lf, 2, bias-Palse) self.bn_ig nn mo.
65
def
Iozward ipl: x self.act_lsisel lnn.aassl..comy.latttt
IRI
self.act_ppc((0 (self. ptt.eennnnppp_____co (self..ooo. (x))
range(1, self.
72
- pe] com___.oovv_cc(l (x))
self.act_pca(1) (x y)
Last peck needs to handled with caze because aLop for last pee may not match
with nLop of the previous pec layer
# and thug an !donity residual connection is not possible
words,
will used iff oLop layers
79
# is same
self.x seif.n_op.p=c_last)
idx self.1
y - self. self.bn_psclidde (self. convpeeeonpe (x1)
solf
y)
84
else:
self.1
- self. aelf.act_pec(iatt (self. (eckk (self. coc._pecccccccceccceoeee [idx] (x)
zelf delf.bo_lffsett con__
P.avg_pool2ddip. www.bin_pppsee.cooeee
93
return self.act_1fe
G COMPLETE NETWORK VISUALIZATION
We also visualize the complete architecture of CoSNet-BI variant and have put it in the context of ResNet-like like models. We have plotted ResNet-50 variant. Please Figure A1
17

---

Published as a conference paper at ICLR 2025
Tel
ResNet-50
CoSNet-BI
101
N.C
Figure A1: Illustration of (a) ResNet-50 He et al. (016)) network, and (b) CoSNet-BI. It must be noted that by merely replacing the residual bottleneck- stages of ResNet with the proposed CoSNet-unit, our CoSNet variant becomes roughly 50% less deep. has 22% fewer parameters, 25% fewer FLOPs, and 40% faster. shows the utility CoSNet design from an efficiency perspective multiple aspects.
18


Reviews:
[{"review_id": "uyE6SFkeLt", "reviewer": "Reviewer_nNEm", "summary": "This paper introduces Columnar Stage Network (CoSNet) to deploy parallel conv units with fewer kernels, and reduce the 1x1 conv layers. To optimize the model efficiency, this paper follows the design objectives to reduce depth and branching, as well as to control the parameter growth, maintain computation intensity and uniform primitive operations.", "questions": "1. what's the difference between group conv and parallel columnar conv?\n2. How much of efficiency and accuracy gain can be translated downstream to segmentation and detection work?\n3. What's the memory overhead with the input replication?\n4. How much depth is required for deep projection to have significant impact?\n5. I see some similarity between PFF and self-attention modules in transformer. What's the performance like if using some fusing all columns simultaneously?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper revisits some of the fundamental design ideas in conv net and proposed some interesting ideas. \n1. Shallow-deep projection is quite interesting. This inherits ideas from ResNet and expands to deep connection. \n2. It achieves competitive performances (accuracy and latency) with reduced network depth and parameters counts. \n3. It also introduces a pairwise frequent fusion (PFF) to fuse information across different columns.", "weaknesses": "Please refer to the questions section, where some clarity or more experiments would be great.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rh2pFkbdvs", "reviewer": "Reviewer_pdHY", "summary": "This paper proposes a simple and concise structure called CoSNet, which has smaller depth, low parameter count, low FLOPs, and attention less operations, well suited for resource-constrained deploy. The work presents a range of experiments that sufficiently support its claims. It is very  interesting for readers.\n\nOverall, it is a good read. The manuscript might get better if a few suggestions (given below) are incorporated.", "questions": "1. Some details are missing.  For example, how is the value of parallel convolution M determined? I think that different values of M will affect the performance. Please explain this details in the text. Other minor issues, such as Section 3.4 is missing in Figure 2 (c), and you should add it.\n2. How is the design like \"input replication\" to improving performance for example? Authors need to give some details in the manuscript.\n3. The related work is comprehensive. However, the authors only highlight the salient features of the previous works that they apply in their network. The manuscript can benefit from discussing shortcomings of the existing methods as research gaps in the section \"Related Work\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to read and clearly explains everything in the paper.\n2. The experimental result is good compared to the previous works. Empirically, the method seems to offer strong accuracy, compared to existing methods with similar architectures.", "weaknesses": "1. Some details are missing.  For example, how is the value of parallel convolution M determined? I think that different values of M will affect the performance. Please explain this details in the text. Other minor issues, such as Section 3.4 is missing in Figure 2 (c), and you should add it.\n2. How is the design like \"input replication\" to improving performance for example? Authors need to give some details in the manuscript.\n3. The related work is comprehensive. However, the authors only highlight the salient features of the previous works that they apply in their network. The manuscript can benefit from discussing shortcomings of the existing methods as research gaps in the section \"Related Work\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNttLJHuvE", "reviewer": "Reviewer_LJPJ", "summary": "This paper introduces a refreshing ConvNet macro design called Columnar Stage Network (CoSNet) with smaller depth, low parameter count, low FLOPs, and attention-less operations. \nIts comprehensive evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-constrained scenarios.", "questions": "As listed in Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is reasonable. \n2. The result is comparable with the state-of-the-art.\n3. The paper is easy to understand.", "weaknesses": "1. Typo: VanillaNet is published in NeurIPS 2023, instead of 2024.\n2. Lack of some new comparison methods, all models were published in 2023 and even earlier. \n   The author should provide more comparisons like InceptionNeXt[1] and UniRepLKNet [2]. \n3. The Top-1 accuracy of EfficientNet-B0 is 76.3 [3] or 77.1 [4], but the author gives a much poorer result of 75.1. \n   Similar problems also happen on ConvNeXt-T (82.1 in [5] but 81.8 in this paper) and EfficientViT-M5 (77.1 in [6] but 76.8 in this paper, and 522M FLOPs in [6] and 600M in this paper). \n\n\n[1] InceptionNeXt: When Inception Meets ConvNeXt. CVPR 2024\n[2] UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition. CVPR 2024\n[3] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019\n[4] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv 2019\n[5] A ConvNet for the 2020s. CVPR 2022\n[6] EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention. CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)