COMNET : CORTICAL MODULES ARE POWERFUL
Authors: Ashish Kumar
Laxmidhar Behera
Venue: Submitted to ICLR 2023 (2023)
Keywords: CNN Architecture
Multi-dimensional efficiencies
Cortical Modules
Columnar Structure
Real-Time Applications
Latency

Abstract: Existing CNN architectures may achieve efficiency in either one or two dimensions: FLOPs, depth, accuracy, representation power, latency but not in all. In this work, we present a pragmatically designed novel CNN architecture “CoMNet” which offers multi-dimensional efficiency at once such as: simple yet accurate, lower latency and FLOPs, high representation power in limited parameters, low memory consumption, negligible branching, smaller depths, and only a few design hyperparameters. The key to achieve the multi-dimensional efficiency is our use of biological underpinnings into CoMNet which is primarily the organization of cortical modules in the visual cortex. To realize CoMNet, a few concepts from well understood CNN designs are directly inherited such as residual learning. Our solid experimental evaluations demonstrate superiority of CoMNet over many state-of-the-art industry and academia dominant architectures such as ResNet, RepVGG etc. For instance, CoMNet supersedes ResNet-50 on ImageNet while being 50% shallower, 22% lesser parameters, 25% lower FLOPs and latency, and in 16% lesser training epochs. Code will be opensourced post the reviews.

Under review as a conference paper at ICLR 2023
COMNET: CORTICAL MODULES ARE POWERFUL
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing CNN architectures may achieve efficiency in either one two dimensions: FLOPs, depth, accuracy, representation power, latency but not in all. In this work, present pragmatically designed novel CNN architecture "CoMNet" which offers multi-dimensional efficiency at once such as: simple yet accurate, lower latency and FLOPs. high representation power in limited parameters, low memory consumption, negligible branching, smaller depths, and only few design hyperparameters. The key to achieve the multi- multi-dimensional efficiency our use of biological underpinnings into CoMNet which is primarily the organization of cortical modules in the visual cortex. To realize CoMNet, a few concepts from well understood CNN designs are directly inherited such as residual learning. Our solid experimental evaluations demonstrate superiority of CoMNet over many state-of- the-art industry and academia dominant architectures such as ResNet, Rep VGG etc. For instance, CoMNet supersedes ResNet-50 on ImageNet while being 50% shallower, 22% lesser parameters, 25% lower FLOPS and latency, and in 16% lesser training epochs. Code will be opensourced post the reviews.
INTRODUCTION
To date, a wide variety of CNN architectures exists: branchless Krizhevsky et al. (2012); Simonyan & Zisserman (2014), single branch (He et al., 2016), multi-branch and variable filter topology (Szegedy al., 2015; 2016), feature resue (Huang et al., 2017), mobile series (Howard et al., 2017: Zhang et al., 2018). Despite their architectural diversity, they all have one characteristic in common; their primary design objective dimensions are limited. For instance, (Simonyan Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017) seek to achieve higher accuracy regardless of network size and runtime, while mobile networks (Howard et al., 2017; Zhang et al., 2018) aim for fewer FLOPs at the cost of reduced representation power. Some neural architecture search based designs (Zoph et al., 2018; Tan & Le, 2019) focus on both higher accuracy and lower FLOPs but are heavily branched, discard latency, and runs slower on GPUs. More recently, (Ding al., 2021) focus accelerating (Simonyan & Zisserman, 2014) during inference phase while leaving training- maining-phase unaddressed.
In contrast, there are many other dimensions which are left unaddressed but are erucial in real-time applications, such as autonomous driving, robotics etc. Foremost is latency or per-sample runtime which still remains ignored in CNNs in favour of FLOPs or throughput. Optimizing latency is crucial because real-time applications tend to process as few one frame at time but not large batches, and also, fewer FLOPss does not mean lower latency (Ding et al., 2021). Second, neural networks are single
replacing their traditional coutnerparts, posing a huge challenge to run multiple networks device (Kumar et al., 2020), bringing in resource efficiency constraints. Third, rising complexity of the tasks (Kumar et al., 2020; Kumar & Behera, 2019) demands large networks to achieve satisfactory results, because mobile networks are insufficient due to their smaller representation power (Sec 2).
In the existing designs, the primary dimensions such as higher accuracy (He et al., 2016; Szegedy et al., 2016), fewer FLOPs (Sandler et al., 2018), and inference time FLOPs (Ding et al., 2021) are explored only individually. However, considering the above discussion, addressing multi-dimensional efficiency is current need of the time. Consequently, in this work, we aim to achieve multi-dimensional efficiency at once, while offering stronger trade-offs in some of them if efficiency in all dimensions is not feasible. To the best of our knowledge, achieving multi-dimensional efficiency has not been explored as it is a difficult task, mostly due to high correlation among dimensions, which makes it possible to be better in one but worse in another.
Towards our objective, we propose novel architecture "CoMNet", that can offer multi malei-dimensionall compatible, low benefits of lower architectural complexity, smaller depth. computing hardware, low latency, memory consumption, low memory access costs on parallel parameter efficiency. CoMNet is primarily based on our translation of biological underpinnings of cortical modules (Mountcastle, 1997) which predominantly exist in visual cortex. We particularly

---

Under review as a conference paper at ICLR 2023
refer to the structure of cortical modules in ventral stream (Tanaka, 1996) which is responsible for object recognition in mammalians. To realize our mere biological design inspirations, we inherit a few concepts from the well established CNN designs e.g. residual leaning (He al., 2016). CoMNet mostly outperforms many key CNN designs, and also offers trade-offs but minimally. Although CoMNet has its design inspired from biological studies, we neither investigate nor assert that CoMNet is functionally similar to the visual cortex.
In a brief, key take-aways of the paper are: First, the notion of multidimensional efficiency in CNNs, Second, the notion of artificial cortical modules (ACM) which helps achieving high representation in fewer parameters, controlled parameter growth, increased computational density, Third, the concept of columnar organization (Mountcastle, 1997) which helps achieve smaller depths, lower latency and FLOPs, faster convergence, Fourth, long range connections similar to pyramidal neurons (Mountcastle, 1997) which further improves the accuracy of CoMNet. We also provide detailed ablations, and a minimal design space of CoMNet to help one choosing suitable model. By using GradCAM (Selvaraju et al., 2017). we also show that CoMNet learns better data representations. Finally, we briefly discuss BrainScore (Schrimpf et al., 2020) and future prospects of CoMNet.
In the next section, we talk about the most relevant works. In Sec. 2, we brief our biological insights and their translation into the CoMNet architecture. In Sec. 5, we present rigorous experimental analysis. Finally, in Sec. 6, we provide conclusions about the paper.
RELATED WORK
Parameters and Representation Power: The earlier CNN designs (Krizhevsky et al., 2012; Si- monyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016) possess high representation power. In these designs, the deeper layers consist of a large number of channels e.g. 512 (Simonyan & Zisserman, 2014) to compensate for the reduction in resolution, leading to exponential growth in the parameters, and synaptic connections of kernel in these layers. It becomes a predominant cause of overfitting (Simonyan & Zisserman, 2014) which is alleviated via dropout (Krizhevsky et al., 2012), but at the cost of increased training time. ResNet (He et al., 2016) avoids that by reducing and expanding the number of channels via 1 X 1 conv layers and placing them before and after 3 x 3 convolutions. Xie al. (2017) restructures the residual unit of ResNet in form of groups. however. the issue of large depth, overall parameters still remains in picture. The mobile CNNs (Howard et al., 2017; Sandler et al., 2018: 8: Zhang et al., 2018; Ma et al., 2018) on the other hand employ depthwise convolutions (Sifre & Mallat) to control the parameter growth and reduce FLOPs. However representation power decreases quickly. Moreover, depthwise convs are devoid of cross channel context which is crucial for better performance (Zhang et al., 2018). Therefore such convolutions are followed by a 1 x 1 conv to intertwine cross channel information.
Depth: In the above networks, the use of 1 x 1 layers increases network depth rapidly. For example, two 1 1 for each x 3 in (He et al., 2016: Sandler et al., 2018; Zhang et al., 2018: Tan & Le, 2019), while one in (Howard et al., 2017). Despite being beneficial, these layers constitute a significant amount of depth e.g. 66% in (He et al., 2016). Moreover, due to their pointwise nature, 1 x convs do not contribute in the receptive field, which in contrast is governed by 3 x 3 layers.
Branching: With time, the CNN architectures have grown from branchless (Krizhevsky al., 2012; Simonyan & Zisserman, 2014) to single branch (He et al., 2016) to multibranch (Szegedy et al., 2016; Schneider et al., 2017). Neural architecture search has resulted in even heavily branched designs (Zoph et al., 2018; Tan & Le, 2019). Although branching helps in improving accuracy, it also raises memory access cost on parallel computing hardware (Ding et al., 2021) which directly impacts latency and memory consumption.
Latency: Both Depth and high branching increase latency despite having fewer FI.OPs. In CNNs, each layer require certain computing time and many such layers are linked serially. Output of one layer can not be computed until all outputs of the preceding layers are available, despite enough computing power remains. This dramatically increases the latency despite fewer calculations per layer. For example, 100 layers each with ims time result in 100ms of latency while shallow network of 15 layers with 3ms per layer runtime will result in 45ms of latency. The best illustration of this phenomenon is (Tan & Le, 2019) which despite having fewer FLOPSs, runs equivalent to five times bigger network (He et al., 2016). More recently. (Ding et al., 2021) proposes structural reparameterization to accelerate (Simonyan & Zisserman, 2014) during inference phase, however, the training- thining-phase network is still high in parameters, and branches even more than its predecessor (He et al., 2016), showing no improvement in training time (Table 4).
2

---

Under review as a conference paper at ICLR 2023
Convergence: Reduced representation power in (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2018; Ma et al., 2018; Tan & Le, 2019) results in overly large training schedules on ImageNet, and reduced performance on other downstream tasks. For instance, (Howard et al., 2017) requires 200 epochs on ImageNet to perform similar to one variant of (Simonyan & Zisserman, 2014) which is trained only for 75 epochs, but performs poorly on object detection. A similar case (Tan & Le, 2019) which requires 4x training schedule(400 epochs), and larger input resolutions in contrast to the earlier ones (He et al., 2016; Simonyan & Zisserman, 2014; Schneider et al., 2017) that are always trained for smaller resolution 224 x 224 and in a range of 90 120 epochs.
3 BIOLOGICAL VISUAL CORTEX
A biological visual cortex is fairly complex but consists of several interesting properties. We highlight the most relevant ones below. For more details, kindly refer to the supplementary material.
Columnar Structure. Cortical modules 1997). Modules in the shallower layers are referred as ocular dominance columns which respond to simple stimuli such as edges, lines of different orientations (Hubel & Wiesel, 1963). Whereas modules in the deeper layers are collection of neurons which respond to a complex stimuli such as face, monkey, human etc, by encoding stimuli information from different viewpoints (Tanaka, 1996). These modules are mainly present in inferotemporal cortex (IT) which is responsible for object detection, and recognition tasks (Figure la).
present all across biological cortex (Mountcastle,
Shared Input or Input Replication. Two cortical modules having different stimuli response can have a common input i.e. the input is replicated to feed the modules. This crucial because at a given location in the visual field, there can be a monkey or a car etc. Hence, module with highest similarity with the stimuli fires strongly, and signals to other parts of the cortex (Hubel & Wiesel, 1963).
Limited Synaptic Connections. Typically, a cortical column contains a small number of neurons (70 100) (Mountcastle, 1997) which results in smaller number of synaptic connections per column. is advantageous as this collection of neurons can learn a simple stimuli.
Massive Parallelization. Regardless of layers being shallower deeper, cortical module pro- cesses only a small region of retina. Multiples of such modules having similar stimuli response are replicated to span the whole retinal field. This organization facilitates massive parallelization.
Lateral Connection Inhibition. Cortical modules do not communicate with each other except at the output (Tanaka, 1996), which is achieved via fewer pyramidal neurons having large number of long range connections. Such neurons fuse cross module information (Tanaka, 1996).
4 COMNET DESIGN
While designing CoMNet, we closely pay attention to the five most crucial objectives: latency, depth, branching, FLOPs, and parameters, which are sufficient to achieve multi-dimensional efficiencies of controlled parameter growth, high representation power in fewer parameters, increased computational density, minimal branching, as other objectives such as memory consumption, memory access cost depend these. Next, we discuss our proposed biologically inspired designs, their advantage in the context of the above objectives, and their translation into CNN realm. At the end of this section, we intertwine all the individual designs to develop the fundamental computational unit of CoMNet which is used to build different instances of CoMNet.
4.1 ARTIFICIAL CORTICAL MODULES
Our first contribution is to develop IT like structure in CNNs. We achieve that by developing artificial cortical modules ACMs, and mechanism feed them similar input replication in the visual cortex. We realize input replication by defining an operator, called CM Feedee. This operator transforms a tensor E RC* HW into another tensor E (M cortical modules. This operator consumes tensor and outputs its M identical replicas (Figure 1b).
where M denotes the desired number of
Since cortical module is essentially a group of neurons, we realize its artificial counterpart (ACM) via a k X & convolution layer, = 3, having N neurons, Figure Ic. Following, the limited synaptic connection property, N is kept small. To serve as M cortical modules, M such convolutions are employed in parallel, each processing one of the replicas obtained from the CM Feeder. We refer the collection of M ACMs Collective Cortical Module (CCM). The above proposed concept of ACM helps achieving controlled parameter growth, high representation power in fewer parameters, and improved computational density discussed below.
3

---

Under review as a conference paper at ICLR 2023
X goOL.
Y c RMMsCrea
Wo : 2 gAxDsaae
Acci
W, RNNCaa
ACMI
CCM
Wa 25ADAAN
(a]
(b)
(c)
Figure 1: (a) Illustration of cortical modules in IT, (b) Feeder,, and (c) realization of ACM, CCM.
Controlled Parameter Growth. In the earlier designs, number of parameters is controlled by either increasing network width or depth which offers only a little control, and leads to exponential growth in parameters, especially in the deeper layers (Sec. 2). On the other hand, ACMs provide an absolute control over number of synaptic connections which precisely reflects the parameters count. This flexibility is crucial while scaling a CNN as per the needs.
High Representation Power We build a hypothesis that multiple neurons with lesser synaptic connections are better than a single neuron with large connections, For any stimuli, it requires certain amount of representation power or neurons be learnt In the existing CNNs, kernel operates on large number of channels, thus having large number of synaptic connections. During learning phase, it gets penalized for all the visual stimuli even it is uncorrelated with them. This is also visible by weight pruning which suggests that post training, eliminating many channels/conneccion does not impact accuracy, indicating a wastage of synaptic connections.
CoMNet addresses the issue by prioritizing smaller neurons over larger ones by limiting the number of synaptic connections. In other words, we synthesize more number of neurons out of larger one. As result, CoMNet exhibits higher accuracy (Table 1) in lesser parameters without prolonging the training duration, in contrast to (Sandler et al., 2018; Tan Le, 2019). shows that CoMNet learns better representations in fewer parameters (Figure 3).
Increased Parallelization and Computational density. Naively, each ACM in a CCM is executed independently, however all of the ACMs are operating parallely, they can be computed efficiently by using NVIDIA's highly optimized Bat ched-Matr Mat-Matrii 1er--iMulttt Multiply routines. To achieve that, we combine all the ACMs of a CCM layer into a single convolution layer having M groups. This strategy packs computations of all ACMs into single convolution, which leads to increased computational density, increased GPU utilization, and reduced memory access cost (Ding al., 2021).
Although group convs are widely explored (Xie et al., 2017; Zhang et al., 2018), there exist two key differences. First, in this approach, input channels are divided into groups i.e. slicing operation, whereas in CoMNet, inputs is replicated i.c. replication operation. Second, group convolutions has to be followed by X 1 conv to avoid loss of accuracy due to lack of inter-group communication (Zhang et al., 2018), increasing network depth, however CoMNet is free from such constraint.
4.2 COLUMNAR STRUCTURE
Our second contribution is to bring columnar organization into ACM. In a cortical column, several neurons are connected both parallely and serially (Mountcastle, 1997). The parallel connections get realized implicitly via N neurons in a module. However, to realize the serial connection, we stack several ACMs one after the another, forming column, as shown in Figure 2a. Since, ACMS are housed in CCMs, stacking essentially is performed over CCMs. Moreover to respect lateral connection inhibition, an ACM of CCM can communicate with exactly one ACM of another CCM. For example, an ACM at 4tth location in a CCM can only feed the ACM at ttth location in the subsequent CCM (Figure 2a).
We also follow the concept of pyramidal neurons which fuses the outputs of several cortical modules. We achieve that by placing 1 X 1 conv (P.) layer after the final CCM in the columnar organization. This conv layer fuses the information of multiple modules or ACMs by operating at each (h, w) E ((MxN)
Each neuron in the P. consists of a large number of connections arising due to combining the output of M * N channels. It greatly mimics the property of pyramidal neurons. The proposed artificial columnar organization facilitates eliminating x layers which results in reduced organization depth, reduced FLOPs, latency, and faster convergence of CoMNet, discussed below.
Reduced Depth, FLOPs, and Latency As mentioned in Sec. 2, 1 X layers are the major con- stituent of depth in state-of-art networks because they are based on unit/blocks which is stack of 1 x 1, x 3. and x 1 layers. Several such units are connected serially to form a stage. For instance, in stage with three units, six 1 X 1 layers present for three 3 x 3 layers. Since, receptive field is mainly governed by l x lo conv layers with & > 1 (Luo et al., 2016), the columnar organization facilitates elimination of 1 layers by stacking 3 x CCM layers, achieving equivalent receptive
4

---

Under review as a conference paper at ICLR 2023
Pawning 3 S
Core xix (Ga)
S Reeder
ccc8
ccxa
cox
Long llunge Connectione (1AC))
(a)
(h)
(c)
Figure 2: (a) Columnar organization, (b) long range connections (LRC), and (c) CoMNet unit / stage.
in just three layers. In other words, columnar organization reduces the three units into one. Notice that, It is true for any number of units. Interestingly, for this reason, CoMNet umit acts as a stage in contrast to earlier networks where a unit serves as a block, and multiple of such blocks forms stage which are sequentially connected to build a network. The elimination of 1 x 1 results in decreased depth which results in reduced FLOPss and latency because of the reason mentioned in Sec 2. As an example, CoMNet performs better than ResNet-50 at 50% fewer layers, while having lower parameters, FLOPs and latency, indicating a huge achievement.
Faster Convergence In just half' epochs i.. 60, CoMNet achieves 99.17% (76.16%) of its accuracy obtained at 120 epochs (76.76%). as compared to ResNet-50 which achieves only 97% (74.15% vs 76.32%). Primary reason behind these improvements is our belief that the 3 x 3 layers are more important than 1 x 1 because they are solely responsible for extracting spatial context, thereby governing receptive field. Hence, we can obtain equivalent receptive field without 1 x 1 by keeping the number of x 3 layers same. However, x suffers with overly large parameters for which 1 convolutions were introduced (He et al., 2016). Our CoMNet tackles this issue by employing ACMs.
4.3 ARTIFICAL LONG RANGE CONNECTIONS
Our third contribution is develop artificial long range connections. In visual cortex, these realized via pyramidal neurons which apart from communicating with the modules, also projects their input to many layers (Mountcastle, 1997). This helps in taking advantage of multi-layer information.
We achieve the same by employing 1 X 1 conv (L.) layer which is fed by pooling the input of the CoMNet unit (discussed next). L. projects this input directly toward the output of the unit while fusing it with the output of P.. This simulates the behavior of combining the cortical module information and multi-layer information. There are many CCM layers between the input and output of CoMNet unit, however, Le bypasses all of them, and directly connect input and output, thereby realizing the effect of long range connections. Although they improve accuracy significantly, they can be traded for parameters, FLOPs at the cost of reduced accuracy. See Figure 2b for visualization.
The long range connections find their structural similarity with projections in (He et al., 2016), however there are notable differences. First, projections in (He et al., 2016) are used only in the first block of a stage, while projection between stages does not exist. Second, projection conv operates with a stride of 2 in (He et al., 2016), whereas in our case, Tee is preceded by a pooling operation. Pooling serves two objectives: First, it helps gathering spatial context because Thg is a pointwise convolution, and aligns the spatial size of the input with output of the final CCM.
4.4 THE COMNET UNIT.
The above proposed CoMNet design ideas are combined to develop the fundamental computational unit of CoMNet, as shown in Figure 2c. In this unit, an input tensor T; is first passed through 1 x conv (S) which squeezes the number of channels by a factor R. Its output T, is then passed to CM Feeder which is followed by a stack of CCMs. Each CCM is connected with the previous via residual connections. The output of final CMM is fed to the P. which fuses the output of all ACMs in the final CCM. Now, T; is fed to Le after pooling (p), which projects the information from previous stage i.e. T; to the P. layer. The output of P. and L. are summed to produce an output tensor T.. Mathematically, the whole computational unit can be written as follows:
S(T.), Tem CM_ [M__Fedder(TA) (Ta); Team CCM(.)
(1)
T- = CCM @ C.M @ (CM)
=
(X,,, = Tem, Xas0 = Teem
(2)
D-....
T. = P. + L.(p(T.))
(3)
where, 3 represents function-of-functiio operator. can be seen that CoMNet is uni-branched. Although CoMNets are faster even with branch, but it can be eliminated using structural reparam- eterization (Ding et al., 2021) during inference, if desired. Fewer branching reduce the memory consumption and memory access cost regardless of training or testing. As CoMNet unit is mostly 3 x 3 convs, it is fully compatible with hardware accelerators because they have dedicated support for them 3 x 3. So far, have shed light on all of the objectives which we envisaged to achieve.
5

---

Under review as a conference paper at ICLR 2023
Table Effect of ACM, Varying M, N, L. Values of M, N are for each of the four CoMNet stages.
Row
N
M
ADepth
LRG
Residual CCN
4Params
4FL.O%%
Top-1 (9)
R0
16
32
84
128
3
1
1
26
8.90M
1.258
74.45
R1
16
32
84
128
3
6
3
4
4
4
26
12.1M
1.77B
75.65
K2
3
13.2M
1.90B
76.01
R3
82
64
128
256
3
3
1
26
/
11.3M
1.658
75.97
R4
128
/
19.8M
3.05B
76.7G
R5
32
128
256
3
5
5
26
22.6M
3.51B
77.01
R6
128
/
R7
32
64
128
258
3
26
11.3M
1.65B
75.37
R8
32
128
256
20
44
13.4M
2.128
75.18
R9
32
64
128
256
4
20
3
44
13.4M
2.128
76.88
4.5
COMNET INSTANTIATION
CoMNet variant can be instantiated by connecting many CoMNet units serially which also serve as stages. Without complicating, we follow earlier models (He et al., 2016; Simonyan & Zisserman, 2014) to keep the tradition of five stages among which first is plain x 3 convolution with stride 2. while remaining are the CoMNet units. Following (He et al., 2016), the channel dimension of S starts with 64 which gets multiplied by 2 at each stage, while channel dimension of Pe and Le always equals to R. times of the channels in S. We set R 4, following (He et al., 2016). To further simplify the instantiation process, the number of CCM layers i.e. I in t CoMNet unit is set equal to the number of blocks in the htth stage of ResNet-50 (He al., 2016). A CoMNet unit mainly has three hyperparameters: M, N, I In this work, we do not explore every permutation and combination of them, instead, the presented CoMNet models in this paper, are derived such that, they can outperform most useful and deployed statc-of-thee of-tthe-rrt models.
EXPERIMENTS
We present in- silico testing of CoMNet on ImageNet (Deng al., 2009) which de-facto standard to benchmark novel architectures. We begin with ablations where we specifically study the significance of ACMS, variation in M, N, effect of long range connection, and residual connections in CCM. We train each CoMNet variant for 120 epochs using SGD, Nesterov momentum, @ass__r=0) with cosine-annealing (Loshchilov Hutter, 2016). We also study 200 epochs regime to demonstrate efficacy of CoMNet against state-of-the-art CNNs which are evaluated for 200 epcohs. For 120 epochs, we use ResdomResized crop (Paszke et al., 2019), random flip only, while for 200 epochs, MixUp (Zhang al., 2017), AutoAugment(Cubul et al., 2019). LabelSmoothing (Szegedy et al., 2016) are also employed. We also show faster convergence by training CoMNet for 60 epochs.
5.1 ABLATION STUDY
Variation M, N Table demonstrate the effect of varying and M (RO-R5). We first fix the values of N and vary M (RO-R5), and then vary M while fixing N (RO <+ R3, R1 +> R4, R2 <> R5). It can be seen that for fixed N, accuracy improves by increasing M, and the same effect is seen by fixing M and varying N. The purpose of this ablation is to show how accuracy, parameters and FLOPs changes with these hyper-parametee precisely controlled by changing the M (R1 R2, R4 R5) which directly reflects accuracy.
It can be noticed that parameters, FLOPs can be
Effeet of ACM To analyze the significance of ACM, we compare instances having different N, M, have similar parameters and FLOPs budget, for instance, RI ++ R2, RI <> R3, Table noticeable that, R22 with 5 ACM is better by 0.36% in accuracy, only at 1.1M more parameters relative to RL. Similarly, R. better by 0.28% in accuracy, only at 0.8M more parameters relative to R3. shows that multiple ACMs facilitates improved accuracy in just a fraction of parameters, and FLOPs. Moreover, if comparing R9 (a deeper model) with R2, we see that R2 with multiple cortical modules achieves 0.13% more accuracy in 0.2M fewer parameters, and 0.17B fewer FLOPs. It shows the advantage of having multiple cortical modules while being shallower.
is
Varying "1" The impact of varying / is shown in R9, Table 1. It can be seen that accuracy definitely improves relative to shallower, but depth increases simultaneously. The increased depth causes increased latency, therefore, we stick to 20 40 layers of depth. We also carry out additional experiment where each CCM is followed by a 1 x 1 as done in group convolutions, while keeping depth and parameters constant. observe a drop in accuracy 1%.
Residuals in CCM R6-R9, Table shows this analysis for a shallower and deeper variant. For shallower model, residual connection shows only minor improvement (0.09%) however for deeper model the effect of residual connections is significant (0.7%).

---

Under review as a conference paper at ICLR 2023
Table 2: Effect of long range connections LRC module.
Row
N
M
#Depih
CCC
Residal cox
Params
4FLOPS
Top- (%)
80
32
64
128
256
3
1
26
X
S
8.5M
1.29B
73. 61
R1
32
64
128
256
1
26
w/o. Pool
9.8M
1.441
74.15
64
128
256
Pool
9.8M
1.448
70.37
Table 3: Minimal design space of CoMNet.
Model
M
*Depch
PParms
4FLOFS
Latency
#ppochs
Top 1pp-10 (%6
Co
206
512
1024
2048
16
32
64
128
3
4
6
3
1
1
26
8.8M
1.258
4ms
120
74.44
CoMNer-A1
512
2048
128
CoMNer-A2
256
512
1024
2048
16
128
3
G
13.2M
1.958
7ms
76.01
CoMNecBO
2048
32
128
256
6
CoMNec- B1
256
512
1024
2018
32
64
128
256
3
6
4
26
19.8M
3.05B
Gms
120
76.76
CoMNee-B2
256
512
1024
2018
32
64
128
256
3
4
12
4
4
4
322
23.3M
3.74B
8ms
120
77.15
CB
256
512
1024
2048
32
64
128
256
3
6
5
26
22.6M
3.51B
6ms
120
77.01
CaMNss-C
256
512
1024
2048
48
80
144
272
3
6
4
4
4
26
21.6M
3.738
6ms
120
76.93
CoMNa-C1
512
2048
CaMNeeCC
256
512
1024
2048
48
144
272
4
4
24.4M
4.1/28
7ms
200
CoMNas-C2
256
512
1024
2048
48
80
144
272
3
6
3
6
6
16
6
26
36.9M
7.098
9ms
120
78.05
CaMNes-C2
266
512
1024
2048
48
80
144
272
3
6
6
16
26
35.9M
7.098
9ms
200
79.25
CoMNe-DO
256
512
1024
2018
64
96
160
288
3
4
6
3
4
4
4
4
26
23. 6M
4.52B
8ms
120
77.25
CoMNecDD
258
512
2048
64
160
288
4
12
36
37.0M
10.5B
11ms
200
80.53
CoMNet-ED 266 512
1024
2048
64
128
266
512
3
4
6
1
3
26
18.2M
2.818
Gms
120
76.51
Effect of Long Range Connections (LRC) As mentioned previously, LRC module can be traded for parameters and FLOPs at the cost of reduced accuracy. However, it important to analyze its significance. To achieve that, we train CoMNet instance in three ways: First, remove LRC entirely, Second, use LRC without pool, and Third, use LRC with pool. See Table 2 for the analysis. It can be noticed that without LRC (RO), the model suffers with heavy accuracy loss of ~ 0.54% relative to when LRC is used without pool (R1). Moreover, when using LRC with pool, accuracy improves significantly i.c. 1.22% and 1.76% relative to R1, and RO respectively. It happens because pooling operation can provide more spatial context to the 1 x Te layer by summarizing the neighborhood.
5.1.1 COMNET DESIGN SPACE
Training many instances by varing hyperparameters of network a very time consuming process which requires a lot of computing resources and months of duration, also visible in (Schneider et al., 2017). Even the earlier (He et al., 2016; Simonyan & Zisserman, 2014) and newer ones (Ding et al., 2021) avoids exploring the whole design space due to this reason. ResNet being half decade older, its design space has only recently been explored (Schneider et al., 2017).
For this reason, despite having only three hyperparameters, we could train only a few CoMNet models that shows superiority over the existing models. However, the accomplishments of CoMNet design encourages us to push ourself to provide a minimal design space of CoMNet (See Table 3). All the CoMNet instances are diverse, and have their five most crucial attributes reported i.c. parameters, FLOPs, Latency, epochs and accuracy. These instances are crafted such that they can easily out- perform the most popular deployable networks, and can be used by the community as pretrained ones. Definitely there are always better ways to train i.e. longer training (300 epochs), more complex augmentation (Yun et al., 2019), (Huang et al., 2016) which can be used to further improve the accuracy of these instances.
5.1.2 IMAGENET CLASSIFICATION
Next we put CoMNet against existing and popular models which are quite dominant in the academia, industry and real-time applications. We shed light on the mullti-dimessonall efficiencies achieved by CoMNet. If efficiency is not possible in all dimensions, CoMNet offers stronger trade-offs relative the rival network. We show that CoMNet is simple during both training and inference while can also take advantage of recent structural reparameterization (Ding et al., 2021) during inference phase.
When comparing CoMNet with existing CNNs, if efficiency in every dimension is not feasible, we pri- oritize a few dimensions over the others by defining a dimension precedence: La Latency = Depth > Branching > FLOPs > Farameters Although parameters efficiency is important, it can be sacrificed CoMNet is better in other dimensions, while latency is kept at the highest.
Table 4 shows the comparison of CoMNet with existing models over multi-dimensional metrics. Latency is reported for RTX-2070 GPU which may vary on a different GPU device, therefore the numbers are only for reference. Below we discuss the observations in structured manner.
7

---

Under review as a conference paper at ICLR 2023
Table 4: CoMNet in the wild.
Kow
Architocture
#lDapth
Wipochs
Parrams
USR__ Pareats
#HLOPS
#SR_FLOPS
Lasency
SR Sa_Latscy
Top-1 (76)
ResNet- 18 (Hf et 2016)
18
120
11 11.6M
1.83B
4mx
71.16
RepVGG AO (Ding 2011)
22
8.30M
1.46B
6ms
72.41
ResNet-34 (Ile et 2016)
34
120
21.7M
3.68B
6ms
74.17
CoMNet-AO
26
120
8.6M
1.25M
1.258
1.25B
Time
5mn
74.45
RepVGG-A1 (Ding ol., 2021)
22
120
14.0M
12.7M
2.638
2.368
Tam
Gins
74.46
R1
EMlikntNee-BB BO (Ding ol. 20213
ReyVOG-B B0 (Ding 2021)
28
49
120
0.2GM
14.3M
0.408
3.06B
3.40B
58s
7ims
Sums
74
75.
CoMNet-A1
26
120
12:1M
12.1M
1.778
1.778
7am
Gias
75.66
ResNet-50 (Pasoke RepVCGLAA2 (Ding al., 2021)
50
25.5M
4.12B
76.30
R2
22
120
28. 1M
25.5M
5.69R
5.128
9mw
Tims
76.48
CoMNet-B1
26
120
19.8M
19.8M
3.05B
3.05B
7im
6ms
76.76
ResNet-101 (He et al., 2016)
101
120
44.5M
7.8GB
15ms
77721
R3
RosNeXt-50 (Xie et 2017)
50
120
1M
4.41
11ms
77.46
CoMNet-Ca
28
24.4M
4.128
4.128
7as
Gis
77.34
R4
ResNeXi-101
ResNet-152 (Il al., 2016)
2017)
101
152
120
44.1M
60. 1M
8.10R
11.5B
14ms
15ms
78.42
77.78
CoMNet-CC
26
120
38.9M
38. 9M
7.09B
7.09B
11ms
9ms
78.05
ReyVGG-BB (IDnng 2021)
28
200
123.00
110.9M
29. 29.18
26.28
22mms
17ms
RD
RegNetX- 12GP (Kis et al., 2017)
57
200
46.0M
12:18
13ms
80.35
CoMNet-D1
36
200
67.0M
57.0M
10.88
10.88
12ms
11mm
80.53
ResNet-50 (Paseike ol., 2019)
50
60
25. 25.5M
4.12B
10ms
74.15
RG
Co
26
60
1 19.8M
19.8M
3.05B
3.05B
7mm
6ms
76.16
CoMNet is 3.29% accurate, has 25% fewer parameters, 31% fewer FLOPs than ResNet-18 while offers similar runtime speed. Although it has 6 more layers, CoMNet offers higher representa- tion power in fewer parameters equivalent latency. Similarly, in comparison to ResNet-34, it is accurate by 0.28%, has 59% fewer parameters, 66% fewer FLOPs, 23% less deeper while runs faster by 37%. In terms of state-of-the-art RepVGG (Ding al., 2021) models, CoMNet is more accurate by 2.04% while having marginally fewer FLOPS, parameters and equivalent speed.
RO.
R1. CoMNet is strong competitor RepVGG i.e. 13% fewer parameters, 32% fewer FLOPs while offering similar speeds but higher accuracy by 1.19% for RepVGG-A1. Although we do not target mobile regime in this paper, we still show that having fewer parameters and FLOPSs do not guarantee faster speeds. EfficientNet-BoO BO has 50% fewer parameters and 77% fewer FLOPs, but is 50% more deeper, and runs 37% slower. We believe that by exploring the design space of CoMNet for mobile regime, CoMNet can offer trade-offs to EfficientNet, which we leave for future work.
R2. ResNet-50 the most widely employed backbone for many areas such as (Ren al., 2015; He et al., 2017; Goyal et al., 2017) due to its affordability in terms of representation power, FLOPs, depth and accuracy. CoMNet easily surpasses ResNet-50 while being 50% shallower, 22% fewer parameters, 25% fewer FLOPs and 40% faster. CoMNet also outperforms RepVGG-A2.
R3-R4. The corresponding experiments shows that CoMNet is better than two larger variants of ResNet which despite being older, still serves as backbones for many cutting-edge works (Carion et al., 2020; Li et al., 2022). Our CoMNet outperforms these two baselines in every aspect while surprisingly being 72% and 82% less deeper relative to ResNet-101 and ResNet-152 respectively. CoMNet also runs significantly faster by almost 50% in roughly 50% fewer parameters and FLOPs. This is quite promising in short training epochs while being shallower.
From the table, it can be also be seen that CoMNet despite being smaller then ResNeXt, outperforms in all the dimensions with marginal accuracy difference, which can be improved by matching the parameters of CoMNet with Xt. Overall CoMNet is 50% less deeper then ResNeXt-50 while runs 50% faster at 6% fewer FLOPs, 2% fewer parameters and roughly same accuracy which is huge. Moreover, CoMNet is also 75% less deeper, 11% less parameters, 12% fewer FLOPs and 35% faster at a marginal accuracy difference which can be matched by changing the CoMNet hyperparameters.
R5. CoMNet is as accurate as RegNeIX-12GF (Schneider al., 2017) at 10% fewer FLOPs, 36% less depth, and runs 15% faster while offering competitive tradeoffs the parameters. For such a shallow model, achieving beyond 80% with merely 36 layers in nulti--dimensionll efficiency setting has not been seen previously. Even RepVGG is dramatically less efficient i.e. 53% fewer parameters, 62% fewer FLOPs while rumning significantly faster. This shows promising nature of CoMNet in large network regime. However, our target in this paper is the real-time domain only.
R6. This experiment shows the faster convergence of CoMNet in half of the original training epochs i.c. 60. It can be seen that CoMNet attains 99.17% 16%) of its accuracy obtained at 120 epochs (76.76)) as compared to ResNet-50 which achieves only 97% (74.15% vs 76.32%).
8

---

Under review as a conference paper at ICLR 2023
Culdii 01, Beades 5o
Accnney
EX0
M
Deyth
PLOPS (B)
Figure 3: GradCAM visualizations
Figure 4: Multi-dimennionall efficiency plots.
Structural Reparameterization. RepVGG uses structural reparameterization (SR) during infer- ence phase which converts it into VGG-like (Simonyan & Zisserman, 2014) structure. CoMNet is also compatible with SR and can achieve lower latency but marginally. Interestingly, RepVGG offers lower architectural complexity only during inference phase, but training- very high i.e. very larger parameters, three branches at each layer (Ding et al., 2021) which leads to increased training time (Table 4). Whereas CoMNet has seamlessly lower complexity during both training and testing, because CoMNet have only one residual connection in CCM layers which can be eliminated via SR. This leads to reduction in FLOPs of element- wiie summation layers which is only marginal, however, memory access cost gets reduced due to the branch climination (Ding et al., 2021). It must be noticed that CoMNet are faster even without SR.
complexity is still
Multi-dimensional Efficiency. Figure 4 shows different plots for accuracy vs an objective dimen- sion. can noticed that CoMNet is always better regardless of the objective, except in depth relative to VGp which is only a difference of 4 layers. Despite that, CoMNet is accurate, has lower latency, lower FLOPs while has low architectural complexity, justifying the objective of this paper.
5.2 ADDITIONAL QUALITATIVE EVALUATIONS
GradCAM: Gradient based Class Activation Map. To comprehend why CoMNet performs bet- ter in multi-dimension, we investigate the class activation maps on ImageNet (Deng et al., 2009) validation set. We use GradCAM (Selvaraju et al., 2017) for this purpose, which computes class activation maps for a class label, indicating the attended regions by the network. GradCAM visualiza- tions of ResNet-50 and CoMNet-B1 (based on R2, Table 4 are shown in Figure 3). It can be seen that CoMNet significantly improves the attended regions of target class relative to the baseline. indicates high generalizability of CoMNet by emphasizing class specific parts in the image.
BrainScore. Although CoMNet follows the structure of biological visual cortex, we neither claim on CoMNet's functional similarity with brain nor do we aim to build models that are more brain- like (Kubilius et al., 2019). However, as the structure of IT cortex is our major inspiration, we BrainScore test on CoMNet-B1 out of curiosity. The BrainScore (Schrimpf et al., 2020) provides a degree to which the intermediate layers of network behave similar the layers of a visual cortex. For more details, please refer to (Schrimpf et al., 2020). The BrainScore performance largely depends on the training method. However, in our test, we found that CoMNet ranks 39 (in more than 350 entries) in the IT score without any specialized training. The models ResNet- 101 and ResNet- 152 are ranked 119 and 143 respectively the IT score. Notice that our BrainScore analysis merely provides new perspective on CoMNet, not to assert the functional similarity of CoMNet with visual cortex.
5.3 FUTURE DIRECTIONS
Achieving multi-dimensional efficiency is crucial for real-time applications and CoMNet is the first effort in this direction by incorporating biological underpinnings. Despite the significant achievements, CoMNet open for improvement. First, a comprehensive design space of CoMNet can be explored, similar to (Schneider et al., 2017). Second, mobile regime of CoMNet is yet to be explored. Third, BrainScore can serve as an additional efficiency dimension to improve CoMNet's brain-like behavior.
6 CONCLUSION
In this work, we propose notion of multidimensional efficiency in CNNs, and a novel CNN architecture "CoMNet" to achieve that. CoMNet inherits certain properties of a biological visual cortex such cortical modules, columnar organization to achieve multi-dimensional efficiency in parameters, FLOPs, accuracy, latency, training duration while having simple architecture. We provide minimal design space of CoMNet instead of only few models. CoMNet outperforms many popular networks such as Rep VGG, ResNet, RegNet, ResNeXt, while being shallower, significantly faster, and also offering competitive trade-off's when multi-dimensional efficiency is not possible.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. Icee, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 13733-13742, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint orXiv:1706. 02677, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint PrXiiv1704.04860 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. CVPR, 2017.
David H Hubel and TN Wiesel. Shape and arrangement of columns in cat's striate cortex. The Journal of physiology, 165(3)5559-568 1963.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Rov Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Ashish Kumar and Laxmidhar Behera. Semi supervised deep quick instance detection and segmenta- tion. 2019 International Conference on Robotics and Automation (ICRA), pp. 8325-8331. IEBE, 2019.
Ashish Kumar, Mohit Vohra, Ravi Prakash, and Laxmidhar Behera. Towards deep learning as- sisted autonomous uavs for manipulation tasks in gps-denied environments. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 1613-1620. IEEE, 2020.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13619-13622 2022.
10

---

Under review as a conference paper at ICLR 2023
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4905-4913, 2016.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Vernon Mountcastle. The columnar organization of the neocortex. Brain: a journal of neurology. 120(4):701-722 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, peshoperformence deep learning library. Advances in neural information processing systems, 32, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chie Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Nick Schneider, Florian Piewak, Christoph Stiller, and Uwe Franke. Regnet: Multimodal sensor registration using deep neural networks. In 2017 IEEE intelligent vehicles symposium (IV), pp- 1800-1810. IEEE, 2017.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition most brain-like? BioRxiv, pp. 407007, 2020.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Sifre and S Mallat, Rigid-motion scattering for image classification. arxiv 2014, arXiv preprint 14Xiv:1403.1687
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1400 409.1556.. 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Keiji Tanaka. Inferotemporal cortex and object vision. Annual review of neuroscience, 19(1):109-139 1996.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.

---

Under review as a conference paper at ICLR 2023
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprini 1rXiv:1710.09412. 2017.
Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
12


Reviews:
[{"review_id": "__NufUX1PAU", "reviewer": "Reviewer_fyC8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CoMNets, a biologically inspired deep network architecture that outperforms existing methods like ResNets and RepVGG in multiple respects (e.g., Latency, FLOPs, depth, etc.). Experiments are shown on ImageNet.", "summary_of_the_review": "My point 1 above in the weaknesses section is most critical. It is important to know that some naïve modification to existing ideas (like group convolutions) will not remove the need to have the CCM modules. Theoretical support would be even better. If this point is reasonably addressed, I will raise my score. Point 2 would help too but it is not as critical as point 1 above. \n\n==== UPDATE AFTER REBUTTAL ====\nI have read other reviews and author response. Authors have addressed one of my main concerns on comparison against vanilla group convolutions. Therefore, I have raised the score to 6 (marginally above acceptance threshold). Future improvements on this work can include: (1) better theoretical justifications (I am not an expert in biological explanations that authors provided during the rebuttal), and (2) deployment on constrained devices (current models are suitable for desktop-scale GPUs but it may be interesting to see how these ideas can be scaled down to MobileNet scale networks). These directions can be good future works and are not necessary for the current paper. ", "strength_and_weaknesses": "The paper has following strengths:\n\n1. I found the idea in this paper to be very interesting but I am not knowledgeable about neuroscience side of things so I cannot evaluate the accuracy of authors’ design decisions from neuroscience point of view.\n\n2. The approach is well-explained, and it was still interesting to read about the relationships with biological cortex.\n\n3. Results clearly demonstrate the superiority over RepVGG and ResNets.\n\n4. Good ablations are conducted and analysis using GradCAM, etc., is conducted to understand the feature representation power of these new networks.\n\n\nThe paper has following weaknesses:\n\n1. I was hoping to see some more theoretical (or at least empirical) study on *why* the new proposed architecture would result in better representative power than traditional blocks of the same size. For example, why would the new way of grouping (e.g., after replicating the input features) should do better (in representative power) than other alternatives (e.g., standard group convolutions when scaled up to match params/flops of CoMNet units)? Since the proposed method does look similar to Group Convolutions, one important ablation should be to replace CCM with group convolution equivalents and then scaling up the model to match params/FLOPs, and then see if we get equivalent accuracy or not. Indeed, if it could be theoretically established why the new architecture would certainly do better than these alternatives, that would be ideal.\n\n2. No comparison against newer state-of-the-art methods like ConvNexts [https://arxiv.org/abs/2201.03545] or their newer scalings shown in recent works like Restructurable Activation Networks (RANs) [https://arxiv.org/abs/2208.08562] is performed which makes it unclear if these models would actually beat the newer models on ImageNet. In any case, these newer papers should be discussed in related work to reflect the latest advances in model design for image classification.\n\n3. It is quite unclear where in the model the non-linear activation functions (e.g., ReLUs) happen. Were any batchnorms/layernoms used to stabilize training? What exact activation functions were used for CoMNet and other baselines?\n\n4. Minor -- Citation format deviates from ICLR official template.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WmtnqvvJFgw", "reviewer": "Reviewer_6Lbu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a CNN architecture based on a module performing the following sequence:\n\n(a)  1x1 dim reduce (\"S\")\n(b)  channel tiling (\"CM_Feeder\")\n(c)  stacked group conv with residuals (\"CCM\")\n(d)  1x1 combine/remap (\"P_c\")\n(e)  skip-conn residual (\"LRC\")\n\nThese steps are motivated by analogy to the columnar organization of the biological visual cortex, particularly ACM/CCM.  After training, structural reparameterization is applied before inference, leading to improved execution speed.  The method is evaluated on imagenet, finding improved speed/accuracy curves compared to ResNet and RepVGG.\n", "summary_of_the_review": "Overall, this system seems to perform well, and its performance and sizing parameters are well-measured on imagenet.  However, the main components leading to this appear to be group convs (used in ResNeXt) and structural reparameterization (from RepVGG), which are combined to create the CoMNet model.  Without a clearer focus on how this work differs and builds upon each of these, along with investigation on interactions unique to their combination (such as repeated group convs with SR), it's hard to distinguish what might be potentially unique contributions in this work, beyond these two existing works.\n", "strength_and_weaknesses": "While the resulting system appears to achieve good performance compared to recent RepVGG, the modules look fundamentally the same or similar to ResNeXt (https://arxiv.org/pdf/1611.05431.pdf), which also uses the steps (a-e) described above.  It's not clear to me what the differences are, if any, and what makes this system more performant.  The three that I can see as possibilities are: (i) stacking multiple group convs before recombining (I think resnext only uses one); (ii) structural reparameterization (from RepVGG); (iii) availability of improved group convolution implementations in cuda.  How do each of these contribute to performance?  Are there any other differences beyond these three?\n\nThe discussion of optimization on multiple efficiency fronts (flops, depth, accuracy, latency, training iters) seems like a bit of a side-track, even though it is mentioned as a main focus.  Descriptions such as in Sec 1, \"to the best of our knowledge, achieving multi-dimensional efficiency has not been explored\", suggest this paper explores explicit ways to measure combinations of these dimensions, e.g. for use in optimization or model selection, but instead the focus is architecture development with standard performance measures (see comments below).  I think it would make more sense to describe the work from this standpoint.\n\nOverall, this system seems to perform well, and its performance and sizing parameters are well-measured on imagenet.  However, the main components leading to this appear to be group convs (used in ResNeXt) and structural reparameterization (from RepVGG), which are combined to create the CoMNet model.  Without a clearer focus on how this work differs and builds upon each of these, along with investigation on interactions unique to their combination (such as repeated group convs with SR), it's hard to distinguish what might be potentially unique contributions in this work, beyond these two existing works.\n\n\n\nMore detailed questions and notes:\n\n\n* Note that the experiment varying \"l\" doesn't quite address the question of the effects of stacking group convs (\"CCMs\"), as the recombination (\"P_c\") is still applied only at the end of the stack.  To see the effect of stacking these before recombining, it would be interesting to look at different combinations of how often the recombination is applied while keeping depth or parameters approximately constant, e.g.:  tile -> CCM -> CCM -> CCM -> P, vs tile -> CCM -> P -> tile -> CCM -> P.  The second has one additional P but one fewer CCM, so similar overall depth.  What is the behavior of this, is there a speed/acc frontier that suggests a best value for number of group conv stacks before recombining and how does it vary with overall depth?\n\n* It would be good to include more comparison systems, including more efficientnet points (only B0 is mentioned in one table), resnext, resnext implemented using the same nvidia group conv library calls used by this system, and for more general comparison, transformer-based ViT variants.\n\n* fig 4:  does the latency plot in this figure use inference-time structural reparameterization?\n\n* sec 3 acm \"controlled parameter growth\":  \"AMCs provide an absolute control over number of synaptic connections\":  how do they achieve this exactly, what values are can be changed or varied in order to control the parameter count in this way?\n\n* sec 2 related work:  As mentioned above, I think ResNeXt needs to be compared to in detail.  Also, connectivity matrices controlling the number of synaptic connections are described in LeCun 1998, which could be good to mention as well.\n\n* eq 2: circled dot notation should be explained/defined in the text, this can be used for several ops e.g. concat, elemwise product, tensor product\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MAkAYSN460", "reviewer": "Reviewer_8V6y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new CNN architecture inspired by biological networks. Instead of a standard ResNet style architecture, this method duplicates the input over a set of modules that then processes all the inputs. This leads to a different way to control parameters and scale the network, resulting in networks that are less deep, but still perform well. The approach is evaluated on ImageNet.", "summary_of_the_review": "The paper proposes an interesting idea and has extensive ablation experiments. However, it is lacking comparisons to many existing works, which makes it hard to tell how effective this idea is.", "strength_and_weaknesses": "The paper is interesting and provides an alternative architecture that seems to perform well. The paper is well written and should be reproducible from the provided details. The paper has a lot of experiments to study the design of the comnet, however, it is lacking comparisons to other works. There are no comparisons to other network designs, such as group convolutions, or works that explicitly study scaling of networks, e.g., EfficientNet. The claims about controlling parameter growth are a bit misleading, as all the aspects are controllable in other networks, though they might take a bit more work to change all the settings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject