Non-deep Networks
Authors: Ankit Goyal
Alexey Bochkovskiy
Jia Deng
Vladlen Koltun
Venue: NeurIPS 2022 Accept (2022)

Abstract: Latency is of utmost importance in safety-critical systems. In neural networks, lowest theoretical latency is dependent on the depth of the network. This begs the question -- is it possible to build high-performing ``non-deep" neural networks? We show that it is. To do so, we use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, we show, for the first time, that a network with a depth of just 12 can achieve top-1 accuracy over 80% on ImageNet, 96% on CIFAR10, and 81% on CIFAR100. We also show that a network with a low-depth (12) backbone can achieve an AP of 48% on MS-COCO. We analyze the scaling rules for our design and show how to increase performance without changing the network's depth. Finally, we provide a proof of concept for how non-deep networks could be used to build low-latency recognition systems. Code is available at https://github.com/imankgoyal/NonDeepNetworks.

Non-deep Networks
Ankit Goyal 1.2 Alexey Bochkovskiy Jia Deng'
Vladlen Koltun 2.3
Princeton University
Intel Labs "Apple
Abstract
Latency of utmost importance safety-critical systems. In neural networks, lowest theoretical latency is dependent on the depth of the network. This begs the question - is it possible to build high-performing "non-deep" neural networks? We show that it is. To do so, use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, network with a depth of just 12 can achieve top-1 accuracy over 80% on ImageNet, 96% CIFARIO. 10, and 81% on CIFARI00. We also show that network with a low-depth (12) backbone can achieve an AP of 48% on MS-COCO. We analyze the scaling rules for design and show how increase performance without changing the network's ' depth. Finally, we provide proof of concept for how non-deep networks could be used to build low-latency recognition systems.
show, for the first time, that
Introduction
Latency, which is the time taken to process request, can be of utmost importance safety-critical systems that require real-time predictions. Consider an autonomous car driving at high speed. In order ensure safety, the car must be able to react within very small time window. Since deep neural networks (DNNs) are the workhorse behind lot of intelligent systems, considering the latency of DNNs is critical. Depending upon the use case, latency could take precedence over other factors like the number of parameters, memory use, and number of calculations.
In DNN, the lowest achievable latency 4. where d depth of the network and f the processor frequency. Although, this limit is hard to achieve on a general- practical applications, one build custom chips to get closer the theoretical latency. This limit for latency can also be considered for future hardware with large FLOPs or bandwidth. In such a case, the latency could only be improved by reducing depth or increasing frequency. On the frequency front, the scope of improvement is limited as the current process lithography approaching the size of the silicon crystal lattice (Sec. 3.5p. Therefore, it is worthwhile ask if one achieve high performance with "non-deep" neural networks.
processor like GPU, for
Large depth is accepted increases its representational ability and helps learn increasingly abstract features (He al. 20 2016a). In contrast, biological neural networks are expected to be much shallower given how quickly humans can perceive objects and scenes Renninger & Malik (004). This disconnect between artificial and biological neural networks, further enhances the scientific value in the question of whether is possible to design high- seghoperffnnnng "non-deep" neural networks.
essential component for high-performing networks because depth
In this work, we show that, it is indeed possible to build herf-peroogning non-deep networks. We refer to our architecture as ParNet (Parallel Networks). We show, for the first time, that classification network with depth of just 12 can achieve accuracy greater than 80% on ImageNet, 96% on CIFARIO, and 81% on CIFARIO0. We also show that detection network with a low-depth (12) backbone achieve AP of 48% MS-COCO. Note that the number of parameters in ParNet is comparable to state-of-the-art models, as illustrated in Figure I
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ParteetL (ours)
(ounc)
Hepv
BRepyoo
UiieeeiNe Net dindetto 1er11
Vininn
BeaN "
ImageNee Topl Acc
ParNii-M (ouns)
DensoNet
26
Re
VITL766 Net
Hegvi er MINask
DwnsnNWw-1211
Meahet--je
VGG
68
20
80
160
Depth (in log scale)
Figure 1: Top-1 accuracy on ImageNet vs. depth (in log scale) of various models. ParNet performs competitively to deep state-of-the-ard neural networks while having much lower depth. Performance of prior models is as reported in the literature. Size of the circle is proportional to the number of parameters. Models are evaluated using a single 224 224 crop, except for ViTB-16 and ViTB- 32 (Dosovitskyy et al. 2021). which fine-tunes at x384 and PReLU-net (He et al. 2013), which evaluates at 256. Models are trained for 90 120 epochs, except for parameter errameter-effccent models such as MNASNet (Tan et al.) 2019). MobileNet (Sandler et al., 20018) and EfficientNet (Tan & Le 2019), which are trained for more than 300 epochs. For faimess, we exclude results with longer training, higher resolution, or multi-crop testing.
Prior works have studied the problem of building seghoreeforming networks with small depth. One classical work is Wide Residual Networks by Zagoruyko & Komodakis (2016) which shows that scaling the number of channels could be an effective way to increase performance while limiting depth. We take inspiration from these exciting works and push the frontiers further. To the best of our knowledge, we are the first ones to show such high performance in the range of depth 10. (Figure [
To build ParNet, we adopt and combine techniques from the literature. One choice is the of parallel branches, where instead of arranging layers sequentially, we arrange layers parallelly. We also find that VGG-style blocks are more suitable for reducing depth as compared ResNet-st blocks. To train such block we use "structural reparamctrization' from Rep VGG Ding et al. (2021). Further. we create a variation of Squeeze-and ant--nndtdatiioo called Skip-Sqqeeze-ana and -nnd-Excttttion that allows increasing the receptive field of network while not increasing depth.
We also study the scaling rules for ParNet. Specifically, show that ParNet be effectively scaled by increasing width, resolution, and number of branches, all while keeping depth constant. We observe that the performance of ParNet does not saturate and increases tational throughput. This suggests that by increasing compute further, one can achieve even higher performance while maintaining small depth (~110) and low latency.
increase compu-
To summarize, our contributions are three-fold. First, we show for the first time, that a neural network with depth of only 12 can achieve high performance competitive benchmarks (80.7% on ImageNet, 96% on CIFAR10, 81% on CIFARIOO), Second, we show how different architccture choices like parallel branches, VGG-style blocks and Skip-Squeeze-antt -nt--annttt-Etion could be used to reduce depth while maintaining high-perfone perorperformmc... Third, we study the scaling rules for ParNet and demonstrate effective scaling with constant low depth.
Related Work
Analyzing importance of depth. There exists rich literature analyzing the importance of depth in neural networks. The classic work Cybenko al. showed that a single-layer neural network with sigmoid activations can approximate any function with arbitrarily small emor (Cybenko 1989). However, one needs to use network with sufficiently large width, which can drastically increase the parameter count. Subsequent works have shown that, to approximate a function, deep network with

---

Downompiin Bepiroi Sit
Sgasel
Augpoct PC
12
(a) ParNet
(b) The ParNet block
Figure 2: Schematic representation of ParNet and the ParNet block. ParNet has depth 12 and is composed of parallel substructures. The width of each block in (a) is proportional to the number of output channels in ParNet-M and the height reflects output resolution. The ParNet block consists of three parallel branches: 1 x convolution, 3 x3 convolution and Once the training is done, the xx 1 and 3x3 convolutions can be fused together for faster inference. The SSE branch increases receptive field while not affecting depth.
suueezee-andExciinne
(SSE).
non-linearity needs exponentially fewer parameters than its shallow counterpart (Liang & Srikant 2017). This is often cited as one of the major advantages large depth.
Several works have also empirically analyzed the importance of depth and came to the conclusion that under a fixed parameter budget, deeper networks perform better than their shallow counterparts (Eigen et al. 2013] Urban et al. (2017). However, in such analysis, prior works have only studied shallow networks with a linear, sequential structure, and unclear whether the conclusion still holds with alternative designs. In this work, we show that, contrary to conventional wisdom, a shallow network can perform surprisingly well, and the key is to have parallel substructures.
Scaling DNNs. There have been many exciting works that have studied the problem of scaling neural networks. Tan Le (2019) showed that increasing depth, width, and resolution leads to effective scaling of convolutional networks. We also study scaling rules but focus on the low-depth regime. We find that one can increase the number of branches, width, and resolution effectively scale ParNet while keeping depth constant and low. Zagoruyko & Komodakis 2016) showed that shallower networks with large width can achieve similar performance deeper ResNets. We also scale our networks by increasing their width. However, we consider networks that are much shallower depth of just 12 compared to considered ImageNet by Zagoruyko Komodakis (2016) and introduce parallel substructures.
Shallow networks. Shallow networks have attracted attention in theoretical machine learning. With infinite width, single-layer neural network behaves like Gaussian Process, and one can understand the training procedure in terms of kernel methods (Jacot et al. 2018 However, such models do not perform competitively when compared to state-ol-the-ar networks (ii ct al. 2019). We provide empirical proof that non-deep networks can be competitive with their deep counterparts.
Multi-stream networks. Multi-stream neural networks have been used in a variety of computer vision tasks such segmentation (Chen et al., 2016 2017), detection (Lin al. (007)) and video classification (Wu et al. (0166). The HRNet architecture maintains multi-resolution streams through- the forward pass (Wang al.) 2020);; these streams are fused together regular intervals to exchange information. We also use streams with different resolutions, but our network is much shallower (12 vs. 38 for the smallest HRNet for classification) and the streams are fused only once, at the very end, making parallclization easier.
Method
In this section, we develop and analyze ParNet, a network architecture that is much less deep but still achieves high performance on competitive benchmarks. ParNet consists of parallel substructures that process features at different resolutions. We refer to these parallel substructures as streams. Features from different streams are fused a later stage in the network, and these fused features are used for the downstream task. Figure 2a provides schematic representation of ParNet.

---

3.1 ParNet Block
In ParNet, we utilize VGG-style blocks (Simonyan & Zisserman, 2015). To see whether non-deep networks can achieve high performance, we empirically find that VGG-style blocks are more suitable than ResNet-style blocks (Table [). In general, training VGG-style networks is more difficult than their ResNet counterparts (He et al. 2016a). But recent work shows that it is easier to train networks with such blocks if one uses a "structural reparameterization" technique (Ding et al.) (021). During training, one uses multiple branches over the 3 x3 convolution blocks. Once trained, the multiple branches can be fused into one 3x3 convolution. Hence, one ends up with a plain network consisting of only 3x3 block and non-linearity. This reparameterization or fusion of blocks helps reduce latency during inference.
We borrow our initial block design from Rep- RGp-VGG (Ding et al. 202I) and modify it to make it more suitable for our non-deep architecture. One challenge with a non-deep network with only 3x3 convolutions is that the receptive field is rather limited. To address this, we build a Skip- (SSE) layer which is based on the snneeze-and-Excitation et al. 2018). Vanilla snneeze--nd-Exciieee of the network. Hence we use a Skip-Squeeze-Eii Excitation design which is applied alongside the skip connection and uses single fully-connected layer. We find that this design helps increase performance (Table (7). Figure 2b provides a schematic representation of our modified Rep-VGG block with the Siip-Squeeze-Excitati module. We refer to this block as the RepVGG-SSE.
Squeeze-Excitatie
(SE) design (Hu is not suitable for our purpose as it increases the depth
We also tried different activation functions for the ParNet block. We found that SiLU (Ramachandran et al. 2017) provided modest gains over ReLU and hence we used it in our network.
3.2 Downsampling and Fusion Block
Apart from the RepVGG-SSE block, whose input and output have the same size, ParNet also contains Downsampling and Fusion blocks. The Downsampling block reduces resolution and increases width to enable multi-scale processing, while the Fusion block combines information from multiple resolutions.
In the Downsampling block, there is no skip connection; instead, we add a single-layered SE module parallel to the convolution layer. Additionally, we add 2D average pooling in the 1x] convolution branch. The Fusion block is similar to the Downsampling block but contains an extra concatenation layer. Because of concatenation, the input to the Fusion block has twice as many channels as a Downsampling block. Hence, to reduce the parameter count, use convolution with group 2. Please refer to Figure Allin in the appendix for a schematic representation of the Downsampling and Fusion blocks.
3.3 Network Architecture
Figure 2a shows schematic representation of the ParNet model that is used for the ImageNet dataset. The initial layers consist of sequence of Downsampling blocks. The outputs of Downsampling blocks 2, 3, and 4 are fed respectively to streams 2, and 3. We empirically find 3 to be the optimal number of streams for a given parameter budget (Table [TP.. Each stream consists of series of RepVGG-SSe blocks that process the features at different resolutions. The features from different streams are then fused by Fusion blocks using concatenation. Finally, the output is passed to Downsampling block at depth 11. Similar to RepVGG (Ding et al., 2021), we use a larger width for the last downsampling layer. Table Al in the appendix provides complete specification of the scaled ParNet models that are used in ImageNet experiments.
In CIFAR the images are of lower resolution, and the network architecture is slightly different from the one for ImageNet. First, we replace the Downsampling blocks at depths and with Rep SSE blocks. To reduce the number of parameters in the last layer, we replace the last Downsampling block, which has large width, with a narrower 1 1x1 1 convolution layer. Also, we reduce the number of parameters by removing one block from each stream and adding block at depth 3.

---

Table 2:
Speed and performance of ParNet, ResNet and RepVGG. ParNet is distributed across GPUs. ParNet distributed on 3 GPUs is faster than similar-perte perforrgp ResNets on single GPU. RepVGG is the fastest as it only consists of 3 X convolution and ReLU.
Table I: Depth vs. performance on ImageNet. We test on images with 224x224 resolution. We rerun ResNets (He et al. 2016a) in our training regime for fairness, thus boosting their accuracy. Our ParNet models perform competitively with ResNets while having a low constant depth.
Model
Depih
Top-1 Acc.
Speed (sumplesh))
Param M)
Plops
Model
Depth (in M)
Top-1
Top-5
ResNet34
34
74.12
306
21.8
7.3
ResNet50
50
77.53
222
25.6
8.2
Acc.
Acc.
RepVGG-blge
29
77.59
376
36.1
14.6
ResNet
18
69.57
89.24
RepVGG-h2g4
29
79.38
300
55.8
22.7
P
ResNet
34
73.27
91.26
ParNet-M
12
76.00
265
35.6
17.2
ResNet-Bottlenek
50
75.99
92.98
PurNet-L.
12
77.66
249
54.9
26.7
ResNet-Botllenek
101
77.56
93.79
ParNer-XL
12
78.55
230
85.0
41.5
ResNet (ours)
18
70.15
89.55
Table 3: Fusing features and parallelizing the sub-
ResNet (ours)
34
74.12
91.89
structures across GPUs improves the speed of
ResNet- BottteBBBteeeekkkkk (ours)
50
77.53
93.87
ParNet. Speed was measured on a GeForce RTX
(ours)
101
79.63
94.68
3090 with Pytorch 1.8.1 and CUDA 11.1.
ParNet-S
12
75.19
92.29
Model
Top-1
Speed (samples/sec)
Latency (mn)
ParNet-M
12
76.60
93.02
Acc.
PanNet-L
12
77.66
93.6
(Unffusi)
112
8.95
ParNet-XL
12
78.55
94.13
PurNet-T. (Fused. Single GPU)
77.66
154
6.50
PurNet-L (Mulli MPUl
77.66
3.4
Scaling ParNet
With neural networks, it is observed that one can achieve higher accuracy by scaling up network size. Prior works ('Tan & Le (2019) have scaled width, resolution, and depth. Since our objective is to evaluate whether high performance can be achieved with small depth, we keep the depth constant and instead scale up ParNet by increasing width, resolution, and the number of streams.
For CIFAR10 and CIFAR100, increase the width of the network while keeping the resolution at 32 and the number of streams at 3. For ImageNet, we conduct experiments by varying all three dimensions (Figure
3.5 Practical Advantages of Parallel Architectures
The current process of 5 nm lithography is approaching the 0.5 nm size of the silicon crystal lattice, and there is limited room further increase processor frequency. This means that faster inference of neural networks must come from parallelization of computation.
The growth in the performance of single monolithic GPUs is also slowing down (Arunkumar et al. 2017). The maximum die size achievable with conventional photolithography is expected to plateau ~-00mm² (Arunkumar al. 2017). On the whole, plateau is expected not only in processor frequency but also in the die size and the number of transistors per processor.
To solve this problem, there are suggestions for partitioning GPU into separate basic modules that lie one package. These basic modules are easier to manufacture than single monolithic GPU on a large die. Large dies have large number of manufacturing faults, resulting in low yields (Kannan ctt al. 2015). Recent work has proposed Multi Moulti-Chip-Mon Chip-Module GPU (MCM-GPU) on an interposer, which is faster than the largest implementable monolithic GPU. Replacing large dies with medium- size dies is expected to result in lower silicon costs, significantly higher silicon yields, and cost advantages Arunkumar et al. 2017
Even if several chips are combined into single package and are located on one interposer, the data transfer rate between them will be less than the data transfer rate inside one chip, because the lithography size of the chip is smaller than the lithography size the interposer. Such chip designs thus favor partitioned algorithms with parallel branches that exchange limited data and can be executed independently for long as possible, All these factors make non-deep parallel structures ad advantagcous in achieving fast inference, especially with tomorrow's hardware.
5

---

Table 4: Non-deep networks can be used back- bones for fast and accurate object detection sys- tems. Speed is measured on a single RTX 3090 using Pytorch 1.8.1 and CUDA 11.1.
Table 5: A network with depth 12 can get 80.72% top-1 accuracy on ImageNet. We show how var- ious strategies can be used to boost the perfor- mance of ParNet.
Model
Backbone Depth
MS-COCO
Latency (in mi))
Model
Top-1 Acc.
Top-5 Acc.
AP
YOLOv4-CPP cofficial, low
46.2
21.0
ParNer-XI.
78.
94.13
YOLOvd-CS CSP (official, high res.)
47.6
21.2
LLonger Training
78.97
94.51
YOI.Dv4CCP (rermiin)
64
47.6
20.0
Train & Test Res. 320
80.32
94.95
ParNet.XL. X1. (Ours)
12
47.5
18.7
10-crop testing
80.72
95.38
PutNet-XL-CSP (Ouns)
48.0
16.4
4 Results
Experiments on ImageNet. ImageNet (Deng et al., 2009) is a large-scale image classification dataset with high- high-resolution images. We evaluate on the ILSVRC2012 (Russakovsky et (015) dataset, which consists of 1.28M training images and 50K validation images with 1000 classes. We train our models for 120 epochs using the SGD optimizer, a step scheduler with a warmup for first 5 epochs, a learning rate decay of 0.1 at every 30th epoch, an initial learning rate of 0.8, and batch size of 2048 (256 per GPU). If the network does not fit memory, we decrease the batch size and the learning rate proportionally, for example, a decrease to learning rate of 0.4 and batch size of 1024. Unless otherwise specified, the network is trained at 224 x 224 resolution. We train our networks with the cross-entropy loss with smoothed labels (Szegedy et al. 2016). We use cropping, flipping, color-jitter, and rand-augment (Cubuk et al. 2020) data augmentations.
In Table I we show the performance of ParNet on ImageNet. We find that one can achieve surprisingly high performance with a depth of just 12. For fair comparison with ResNets, we retrain them with our training protocol and data augmentation, which improves the performance of ResNets over the official number. Notably, we find that ParNet-S outperform ResNet34 by over percentage point with a lower parameter count (19M vs. 22M). ParNet also achieves comparable performance to ResNet with the bottleneck design, while having 4 to times less depth. For example, ParNet-L performs as well as ResNet50 and gets top-1 accuracy of 77.66 as compared to 77.53 achieved by ResNet50. Similarly, Par performs comparably to ResNet101 and gets a top-5 accuracy of 94.13, in comparison 94.68 achieved by ResNet101, while being times shallower.
In Table [ we compare speed and accuracy. We evaluate the speed of the models where samples are coming one at time like in robotics and autonomous driving. Hence, the batch size in all evaluations is 1. We find that ParNet performs favourably to ResNet when comparing accuracy and speed, however with more parameters and flops. Note that in this comparison, ParNet is distributed across 3 GPUs while ResNet is executed on a single GPU. We find RepVGG to be the fastest. This is because of the excellent linear design consisting only of optimized layer like 3X3 convolution and ReLU. Note that although current hardware favour RepVGG style networks, fundamentally, ParNet could be made faster with custom processor design which is exciting avenue for future research. The parallel sub-structures of ParNet are distributed across GPUs (more details below).
In T we test speed for three variants of ParNet: unfused, fused, and multi-GPU, The unfused variant consists of 3x3 and 1 xx1 branches in the RepVGG-SSE block. In the fused variant, we use the structural-reparaaettiia
trick to merge the 3x3 and 1x] branches into a single 3x3 branch (Section 3.1 For both fused and unfused versions, we use a single GPU for inference, while for the mPlti-GPU version, we use 3 GPUs. For the multi-GPU version, each stream is launched on separate GPU. When all layers in a stream are processed, the results from two adjacent streams are concatenated on one of the GPUs and processed further. For transferring data across GPUs we use the NCCL backend in Pytorch. We find that ParNet can be effectively parallelized across GPUs for fast inference. This is achieved in spite of the communication overhead. With specialized hardware for reducing communication latency, even faster speeds could be achieved.
Boosting Performance. Table 5 demonstrates additional ways of increasing the performance of ParNet, such as using higher-resolution images, longer training regime (200 epochs, cosine annealing), and 10-crop testing. This study is useful in assessing the accuracy that can be achieved by non-deep models on large-scal, scale datasets like ImageNet. By employing various strategies we can boost the performance of ParNet-XL from 78.55 to 80.72. Notably, we reach top-5 accuracy of

---

Table 6: Performance of various architectures CIFARIO and CIFAR100. Similar-sized models are grouped together. ParNet performs competitively with deep state-of-the-art architectures while having much smaller depth. Best performance bolded. The second and third best performing model in each model size block are underlined.
Architecture
Depth
Params (in Millions)
CIFARIO
CIFARIO0
Error
Error
ResNerl TO (offiiia))
110
1.7
6.61
ResNetl 10 (reported in Huang el 2016
110
1.7
6.41
27.22
ResNet (Stochastic DepthT (Huang 2016
5.23
24.58
ResNet (pre-act) (He al. 2016b
164
1.7
5.46
24.33
DenseNet (Huang el at. 2017
40
1.0
5.24
24.42
DenseNet (Botrenccc Comppiescccooooosooonn Huang et al. 2017
100
4.51
22.27
ParNet (Ours)
24.62
ResNet (Stochastic Depthi (Huang
2016
1202
10.2
ResNet (pre-act) (He et al. 2016b
1001
10.2
4.62
22.71
WideResNet (Zagoruyko & Komodakis
2016
40
8,9
4.53
21.18
WideResNet (Zagoruyko Komodakis
2016
11.0
4.27
20.43
WideResNet (SE)
al.
19.14
FractalNet [Compressed] (Larsson et
2017
41
22.9
5,21
21.49
DonseNet (Huang et al. 2017]
100
7.0
4.10
20.20
DenseNet (Batleenck+ooooppcees
(Huagg et al. 2017
250
15.3
3,62
17.60
ParNet (Ours)
3,90
20.02
WideResNNtt (Zagoruyko Komodakis
2016
28
36.5
4,00
19.25
WideResNet (Dropoul in
Zagoruyko & Komodakis 2016
28
36.5
3,89
18.85
FractalNet (Larsson et al. 2017
21
36.8
5.11
22.85
FractalNel Druppppt (Larsson al. 2017
21
4.59
23.36
DenseNet (Huang al. 2017
100
27.2
3.74
19.25
DenseNet [Bottlcnckk<Coopeesss
Huang al. 2017
190
25.6
3,46
17.18
ParNet (Ours)
12
35
3.88
18:65
95.38, which is higher than the oft-cited human performance level ImageNet (Russakovsky et al., 2013). Although this does not mean that machine vision has surpassed human vision, it provides a sense of how well ParNet performs. To the best of our knowledge, this the first instance of such "human-level" performance achieved by network with depth of just 12.
Experiments MS-COCO. MS-COCO (Lin et al., 2014) is object detection dataset which contains images of everyday scenes with common objects. We evaluate on the COCO-2017 dataset, which consists of 118K training images and 5K validation images with 80 classes.
To test whether a non-deep network such as ParNet can work for object detection, we replace the backbone of state-of-the-art single stage detectors with ParNet. Specifically, we replace the CSPDarknet53s backbone from YOLOv4-CSP (Wang al., 202 la) with ParNet-XI., which much shallower (64 12). We use the head and reduced neck from the YOLOR-D6 model, and train and test these models using the official YOLOR code (Wang et al. 2021b). We also retrain YOLOv4- CSP /Wang et al. 202 la) with our protocol (same neck, same head, same training setup) for fair comparison and it improves performance over the official model. Additionally, for fair comparison, test the ParNet-XL-CSS XL- model by applying the CSP (Wang al. (021a) approach ParNet- XL. We find that ParNet-XL and ParNet-XLL-SS) are faster than the baseline even at higher image resolution. We thus higher image resolution for ParNet-XL and ParNet-XL-CSS
In Table f we find that even on single GPU, ParNet achieves higher speed than strong baselines. This demonstrates how non-deep networks could be used to make fast object detection systems.
Experiments CIFAR. The CIFAR datasets consist of colored natural images with 32x pixels. CIFAR-10 consists of images drawn from 10 and CIFAR- 100 from 100 classes. The training and test contain 50,000 and 10,000 images respectively. We adopt standard data augmentation that is widely used for these two datasets (He et al. 201 Zagoruyko Komodakis 2016: Huang et al., 2017). We train for 400 epochs with a batch size of 128. The initial learning rate 1s 0.1 and is decreased by a factor of 5 at 30%, 60%, and 80% of the epochs as (Zagoruyko Komodakis 2016). Similar to prior works (Zagoruyko & Komodakis 2016 Huang et al. 2016), we use weight decay of 0.0003 and set dropout in the convolution layer at 0.2 and dropout the final fully-connected layer at 0.2 for all our networks on both datasets. We train each network on GPUs (a batch size of 32 per GPU) and report the final test set accuracy.
scheme (mirroring/shiftngg
7

---

77
M-s1-3320
L-51-224
79
x1-33-320
Insegpeee Ttp- Accamey (%g
+ MM-53-224
xLst
78.5
L-53- L-53-256 256
[-91-256
77
Mc-3-256
XL-51-224
L-x1-224
M-51-256
stroam 224x224
76
Scale realutionn
stroam high resolution
Mis1-224
Stale
M.5f-222
streams hich resolution
75
Scatee charnels
30
50
60
30
90
100
75.5
Namber Parmmetess (Millions)
30
35
40
50
55
60
Figure 3: Performance of ParNet increases as we increase the number of streams, input resolution, and width of convolution, while keeping depth fixed. The left plot shows that under fixed parameter count, the most effective way to scale ParNet is to use 3 branches and high resolution. The right plot shows the impact on performance by changing only one of the aforementioned factors. We do not observe saturation in performance, indicating that ParNets could be scaled further to increase performance while maintaining low depth.
of Pansmdders (Millions)
Table 7: Ablation of various choices for Par- Net. Data augmentation, SiLU activation, and Eexiittt-Exiitttion (SSE) improve performance.
Table 8: ParNet outperforms non-deep ResNet variants. At depth 12, VGG-style blocks outperform ResNet blocks, and three branches outperform single branch.
Model
Params (in M)
Top-1
Top-5 Acc.
Name
Block
Branch
Depth
Params
Top-1
Acc.
22-WWee
ResNer
12
39.0
73.52
Baseline
32.4
75.02
92.15
ResNer14 14-WideHN
BesNer-BN
14
39.0
72.06
+ Data Augmentation
32.4
75.12
92.00
ResNett 12-ee2--Wiii 2-Wide-BN
ResNer-SSF.
12
39.0
73.91
SSE
35.6
76.55
93.01
ZuNaa-M-OneSteen
RepVOG-SSE
12
36.0
75.83
SILU
35.6
76.60
93.07
ParNei-M
RepVGGG-SSR
3
12
35.9
76.5
Table 6 summarizes the performance of various networks on CIFARIO and CIFAR100. We find that ParNet performs competitively with statc-of-the-an deep networks like ResNets and DenseNets while using much lower depth and a comparable number of parameters. ParNet outperforms ResNets that are 10 times deeper on CIFAR 10 (5.0 vs. 5.23) while using a lower number parameters (1.3M vs. 1.7M). Similarly, ParNet outperforms ResNets that are 100 times deeper CIFAR1O (3.90 4.62) and CIFAR100 (20.02 vs. 22.71) while using 50% more parameters (15.5M vs. 10.2M).
ParNet performs well as vanilla DenseNet models (Huang et al. 2017) with comparable parameter counts while using 3-8 times less depth. For example, on CIFAR 00, ParNet (depth 12) achieves an error of 18.65 with 35M parameters and DenseNet (depth 100) achieves error of 19.25 with 27 27.2M parameters. ParNet also performs on par or better than Wide ResNets (Zagoruyko & Komodakis 2016) while having 2.5 times less depth. The best performance on the CIFAR dataset under given parameter count is achieved by DenseNets with the bottleneck and reduced width (compression) design, although with order of magnitude larger depth than ParNet.
Overall, it is surprising that mere depth- 12 network could achieve an accuracy 96.12% on CIFARIO and 81.35% on CIFAR100. This further indicates that non-deep networks can work as well as deeper networks like ResNets.
Ablation Studies. To test we can trivially reduce the depth of ResNets and make them wide, we test three ResNet variants: ResNet1 Wide, ResNer14-
and ResNetl2-Wide-SSE ResNet12- Wide uses the basic ResNet block and has depth 12, while 4esNee14-Wide-B1. uses the bottleneck ResNet block and has depth 14. Note that with the bottleneck ResNet block, one cannot achieve depth lower than 14 while keeping the original ResNet structure there has to be I initial convolution layer, 4 downsampling blocks (3 4 12 depth), and fully-connected layer. We find that ResNet12- Wide outperforms ResNet14 Wide-Bii block and SiLU activation to ResNet1 2--Wiee Wide to create ResNet12 2-Wide-SSE to further control for confounding factors. We find that ParNet-M outperforms all the ResNet variants which have depth 12 by 2.7 percentage points, suggesting that trivially reducing depth and increasing width is not as effective our approach. We show that the model with three branches performs better than model
with the same parameter count. We additionally add SSE
8

---

Table 9: ParNet outperforms ensemble of single-stream ParNets across different parameter budgets.
ParNet-M
Single Stream
ParNet-L.
Ensemble Single Streams)
ParNet-XT.
Ensemble Single Streams)
# Param
35.9 76.60
36
55
72
85.3 78.55
108 77.68
ImageNet Top-1
75,83
77.66
77.20
Table 10: Performance vs. number of streams. For a fixed parameter budget, streams is optimal.
of Branches (Res. 224)
# of Branches (Res. 320)
2
4
ImageNet Top-1
75.83
76.1
76.75
76.34
76.91
77.12
77.56
77.46
Table II: Accuracy vs latency trade-off with changing depth. For ParNet, we find depth 12 to perform better then other depths when considering both the latency and accuracy
Size
Size
Size XL
Depth
Latency (in ms)
ImageNet Top-1
Latency (in ms)
ImageNet Top-1
Latency (in ms)
ImageNet Top-1
3.1
73.9
3.2
75.1
3.7
76.
12
3.8
76.6
4.0
77.7
4.4
78.6
4.8
77.0
4.9
78.4
5.4
79.4
with single branch. We also show that with everything else being equal, using VGG-style blocks leads to better performance than the corresponding ResNet architecture. Architectural choices in ParNet like parallel substructures and VGG-style blocks are crucial for high performance at lower depths.
Table 7 reports ablation studies over various design choices for our network architecture and training protocol. We show that each of the three decisions (rand-aus -ugd-atgeent data augmentation, SiLU activation, SSE block) leads to higher performance. Using all three leads to the best performance.
evaluate networks with the same total number of parameters but with different numbers of branches: 1, 2, 3, and 4. We show that for fixed number of parameters, a network with branches has the highest accuracy and optimal in both cases, with network resolution of 224x224
In Table TO
and 320x320.
Ensemble of single-stream ParNets (multi-branch) ParNet. Another approach to network parallelization is the creation of ensembles consisting of multiple networks. Therefore, we compare ParNet to ensembles of single-stream ParNets. In Table we find that ParNet outperforms ensembles while using fewer parameters.
Scaling ParNet. Neural networks can be scaled by increasing resolution, width, and depth TTn & Le 2019). Since we are interested in exploring the performance limits of constant-depth networks, we scale ParNet by varying resolution, width, and the number of streams. Figure ( shows that each of these factors increases the accuracy the network. Also, we find that increasing the number of streams is more cost-effective than increasing the number of channels in terms of accuracy versus parameter count. Further, we find that the most effective way to scale ParNet is to increase three factors simultancously. Because of computation constraints, streams beyond 3, but this is not a hard limitation. Based on these charts, we see no saturation in performance while scaling ParNets. This indicates that by increasing compute, one could achieve even higher performance with ParNet while maintaining low depth.
could not increase the number of
Optimizing depth for latency and accuracy. In Table IT we show results with ParNet variants with depth 9, 12 and 15. Note that networks with size M have 128, 256 and 512 channels; size L have 160, 320 and 640 channels; size XI. have 200, 400 and 800 channels in the three branches (Fig. 1). We find that decreasing the depth of ParNet from to reduces latency but also reduces performance. ParNet-XL with depth 9 and ParNet-M with depth 12 have similar latency (3.7 vs 3.8 ms) but slightly

---

worse performance (76.1 vs 76.6). Similarly. increasing the depth of ParNet from 12 to 15 increases performance but also increases latency. ParNet-M with depth 15 both slower and less accurate than ParNet-I. with depth 12. Overall, we find that the depth of 12 to be better for ParNet when considering both the lateney and accuracy.
Discussion and Limitations
We have provided the first empirical proof that non-deep networks can perform competitively with their deep counterparts in large-scale visual recognition benchmarks. We showed how various architectural choices can be used to create non-deep networks that perform surprisingly well. We also demonstrated ways to scale up and improve the performance of such networks without increasing depth.
One limitation our work is that currently non-deep networks are not a replacement for their deep counterparts low-compute settings requiring small number of parameter and flops. Another limitation of ParNet is that although has lower depth than RepVGG, is not faster than RepVGG on GPUs. This is because: first, Rep VGG consists of consists of optimized layers like 3X3 convolution and ReLU; and second, time in spent on multi-GPU communication in ParNet. Note that although current hardware favour Rep RGG style networks, fundamentally, ParNet could be made faster with custom processor design which an exciting avenue for future research.
In summary, our work shows that there exist alternative designs where highly accurate neural networks need not be deep. Such designs can better meet the requirements of future multi-chip processors. We hope that our work can facilitate the development of neural networks that are both highly accurate and extremely fast.
References
Arunkumar, A., Bolotin, E., Cho, B., Milic, U., Ebrahimi, E., Villa, O., Jaleel, A., Wu, C.-J,, and Nellans, D. Mem-gpu: Multi-chip-module gpus for continued performance scalability. In ISCA, 2017.
Chen, L.-C., Yang, Y., Wang. J., Xu, W., and Yuille, A. L. Attention to scale: Scale-aware semantic image segmentation. In CVPR, 2016.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmenta- tion with a reduced search space. In CVPR Workshops, 2020.
Cybenko, G. Approximation by superpositions of a sigmoidal function. In Mathematics of Control, Signals and Systems, 1989.
Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, Repvgg: Making vgg-style convnets great again. arXiv:2101.0369,, 2021.
Dosovitskiy, A.. Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Eigen, D., Rolfe, J., Fergus, R., and LeCun, Y. Understanding deep architectures using a recursive convolutional network. arXiv: 12, 1 1847, 2013.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. CVPR, 2016a.
10

---

He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In ECCV, 2016b.
Hu, J. Shen, L., and Sun, G. Squcezz-andd-xxitttionn networks. CVPR, 2018.
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. In ECCV, 2016.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
Jacot, A., Gabriel, F,, and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. NeurIPS, 2018.
Kannan, A., Jerger, N. E., and Loh, G. H. Enabling interposer- based disintegration multi-com processors. In MICRO, 2015.
Larsson, G., Maire, M., and Shakhnarovich, G. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.
Li, Z.,, Wang, R., Yu, D., Du, S., Hu, W., Salakhutdinov, R., and Arora, S. Enhanced convolutional neural tangent kernels. arXiv:/91 1.00809, 2019.
Liang, S. and Srikant, R. Why deep neural networks for function approximation? In ICLR, 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. Microsoft COCO: Common objects in context. In ECCV, 2014.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, and Belongie, S. Feature pyramid networks for object detection. In CVPR, 2017.
Ramachandran, P., Zoph, B., and Le, Q V. Searching for activation functions. arXiv: 1710.0594, 2017.
Renninger, L. W. and Malik, J. When scene identification just texture recognition? Vision research, 44(19):2301-2311 2004.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. In IJCV, 2015.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., Mohamed, A., Philipose, M., and Richardson, M. Do deep convolutional nets really need to be deep and convolutional? In ICLR, 2017.
Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y. M. Scaled-yolov4; Scaling cross stage partial network. In CVPR, 2021a.
Wang, C.-Y,, Yeh, I.-H., and Liao, H.-Y. M. You only learn one representation: Unified network for multiple tasks. 2005:22005004006, 2021b.

---

Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang. X., Liu, W., and Xiao, B. Deep nigh-resolution representation learning for visual recognition. TPAMI, 2020.
Wu, Z., Jiang, Y.G.,, Wang, X., Ye, H., and Xue, X. Multi-stream multi-class fusion of deep networks for video classification. In ACM Multimedia, 2016.
Zagoruyko, S. and Komodakis, N. Wide residual networks. In BMVC, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12

---

A
Appendix
layer name
layer depth
ouiput size
Small
Medium
Large
Extra large
1x 64
1x 64
1x 64
1x 64
Downsampling
H/2
3x3,
stride 2
stride 2
stride 2
stride
1x1, 96
1x1. 128
1x1, 160
1x1, 200
Downsampling
2
W/4 x H/4
3x3, 96 stride
3x3. 128
x1
3x3,, 160
x1
3x.3, 200
x1
stride
stride 2
stride
1x
Downsampling
3
W/8 x H/8
3x3. 192
x1
3x3, 256
x1
3x3, 320
x1
3x3,, 400
x1
stride
stride
1x 384
1xl, 512
1xl,
Ix
Downsampling
4
W/16 x H/16
3x 384 stride
x1
3x3, 512
3x3, 640
3x3, 800 stride
x1
stride
strido
1x 96
Txl, 128
1x1, 160
Ix], 200
Stream
3-6
W/4 H/4
3x3, 96
3x3, 128 SSE
3x3, 160
3xc3, 200
SSE
SSE
SSE
1x 192
1x1, 256
1x1, 320
1x], 400
Streaml Dowwssamllnng
H/8
x1
256
3x33,
3xc3,
x1
stride
stride 2
stride 2
stride
1x 192
1x 256
1x 320
1x 400
Stream2
4-8
W/8 x H/8
3x3, 192
X5
3x3. 256
x5
3x3, 320
x55
3x3,, 400
x5
SSE
SSE
SSE
SSE
Stream2 Fusion
W/16 x 11/1
3x 384
x1
3x3,512
x1
3x3, 640 strido
x1
3x3, 800
x1
stride
stride
1x1,384
1x1. 512
1x1,
1x1,
Stream3
5-9
W/16 x H/16
3x3, 384 SSE
X5
3x3.512
x5
3x3, 640 SSE
3x3, 800 SSE
x5
1x 1x 384
1x1, 512
1x1, 640
1x1, 800
Stram33-usiio
10
W/16 H/16
3x3, 384 SSF
x1
3x3, 512 SSF
3x3, SSF
3xx3, 800 SSF
1x 1280
1x 2048 3x3, 2048 stride
1x 2560 3x.3, 2560
1x 3200 3x3,
Downsampling
W/32
3x3, 1280 stride
x1
x1
stride 2
stride
Final
12
1xl
average pool, 1000-d fc, softmax
Table A1: Specification of ParNet models used for ImageNet classification: ParNet-S, ParNet-M. ParNet-L., and ParNet-XI..
Batch norm
Batch norm
Concatenation
Shuffle
Avg pool 2x2
Global avg pool
Avg pool 2x2
Global avg pool
1x1 conv, groups=2
3x3 conv, s=2, groups=2
1x1 conv, groups=2
1xl
3x3 conv, s=2
1x1 conv
Batch norm
Batch nom
Sigmoid
Batch nom
Batch nomm
Sigmoid
SiL U
SIL U
Figure A1: Schematic representation of the Fusion (left) and Downsampling (right) blocks used in ParNet.
13


Reviews:
[{"review_id": "p6oduJ1LbbH", "reviewer": "Reviewer_zBDf", "summary": "The presented paper questions the importance of the depth in neural architectures to provide state-of-the-art-results. There are two main motivations for this paper, the first being that very deep networks are inherently ill-suited for real-time systems, since input data needs to be processed sequentially through each layer. The second motivation is that computations within shallow networks using parallel substructures can be parallelized, hence (theoretically) reducing the effective computation time. However, depth being a central feature allowing neural architecture to perform complex classification tasks, the main challenge addressed by this paper is to derive a class of shallow architectures whose performances remain competitive against state-of-the-art deep architecture. The resulting architecture, called ParNet, is rather flexible in the sense that one can scale its representational power by increasing the number of parallel substructure. The authors specifically discuss the importance of certain module with respect to the final performance reported in the experiment section.", "questions": "Can the author provide a setting example where they can lower-bound the time it would take for a ResNet-like or VGG-like architecture to process an input, both theoretically and empirically, and show to what extent parallelism gives a crucial advantage ?\nThroughout the paper, the authors mention situations where multi-stream shallow networks should provide much faster computations.\nFor example, table 3 compare ParNet with and without parallelism but do not compare the speed with other architectures.\nFurthermore, the authors promote the need for robust shallow architecture to reduce latency in real-time systems such as autonomous vehicles. The closest experiment addressing this issue is the MS-COCO challenge as it is a detection rather than a classification challenge, but parallelism is not employed here, thus resulting in less than 25% improvement in latency while the depth is reduced by 75% compared to the baseline. Can the authors explain why they did not include more experiments on model parallelism while strongly motivating their paper for this particular purpose ?", "limitations": "One limitation of the paper, actually discussed by the authors, is that non-deep networks are not a replacement for their deep counterparts. They motivate such claim by saying that non-deep networks still require a high number of parameter and flops. From a more theoretical side, however, the reviewer strongly wonders if there are classes of functions actually requiring a certain depth to be effectively approximated by a neural network. This last point could help identify the use cases where non-deep networks will never be an acceptable replacement for their deep counterparts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper motivates its research effort to derive efficient shallow architectures with several arguments, the first one being that shallow networks are highly desirable in real-time applications. The author also motivates their effort from two \"wall-clock\" computation arguments, the first one being that deep architectures are hardly biologically plausible given the speed of human reaction with respect to certain stimuli. The authors also try to make reasonable assumptions on the improvement that is likely to be provided by the next generation of hardware, pointing out that neural architectures enabling module parallelism (different from data-parallelism), should benefit from such improvements. Three experiments are conducted to assess the relevance of the architecture:\n1) experiments show that ParNet, an architecture of depth only 12, provides performance on par with ResNet-like and VGG-like architectures on CIFAR10, CIFAR100, ImageNet and MS-COCO. The authors also carefully discuss speed versus accuracy in the case of ImageNet.\n2) the impact of specific block design and engineering tricks found in the literature with respect to the final performance are discussed to show how they can be employed to reduce depth while maintaining high performance. The claims are backed by an ablation study.\n3) rules to increase the statistical capacity are presented, and the increased statistical capacity results in increased performance for ParNet.\n\nSome weaknesses are exposed by the authors themselves. In table 2, RepVGG was found to be faster than ParNet, however authors point out that RepVGG benefits from a highly optimized design which is coherent given that this architecture has been tested and improved on many times over. Another key weakness in the presented paper (not the contribution itself) is the lack of real-world scenario where parallelism is actively employed to reduce computation time, hence showcasing the advantage of ParNet in such setting. Table 3 attempts to show that parallelizing the streams on different GPU does reduce the effective computation time, but it is likely that neither the software nor the hardware are optimized for ParNet paradigm, since they were initially developped to tackle data-parallelism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P3d_IJLpbLD", "reviewer": "Reviewer_Zr8N", "summary": "In this paper, the authors explore how to reduce the depth of neural networks while keeping the model capacity. They propose ParNet, which has a depth of only 12 and can achieve comparable performance with deep neural networks. The authors also show some potential advantages of using non-deep networks for a smaller latency on future hardwares.", "questions": "- From my perspective, ParNet is generally comparing to models with a large computing budget like ResNet. It is interesting to see whether ParNet architecture can also achieve comparable performance with modern compact models (for example EfficientNet). Compat models are more widely used in environment that requires low-latency and some of the design philosophy might help to reduce the high computation cost of ParNet.", "limitations": "The main limitation of this work (inference speed on current hardwares) is well-discussed in the paper. In my option, figuring out a way to avoid the use of global average pooling and sigmoid while keeping the same depth/performance would be a huge technical contribution, as they are indeed slow on GPUs and hard to parallize or optimize.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The whole idea of non-deep networks is novel and not well-explored in previous works, as increasing the depth is usually a common practice for most neural network architectures to scale up themselves.\n- Achieving similar performance as 'deep networks' with a depth of 12 is pretty impressive.\n- The authors also discuss the potential advantages of using shallow networks to fully utilize the parallelism of hardware (e.g. GPUs).\n- Some previous works are reproduced under same setting for fair comparison.\n- The key limitations are well discussed.\n- The writing is good and all details are presented very clearly.\n\nWeaknesses:\n- The technical contributions are not significant, as most of the crucial components of ParNet were presented in other papers. For example, the overall ParNet structure looks very similar to HRNet [1] and the ParNet block looks like a variant of Inception [2].\n- Although the authors intend to improve the inference speed by reducing the network depth, ParNet still introduces some element-wise operations that are not parallism-friendly and hard to optimize (e.g. global average pooling and sigmoid in SSE layer and SiLU activation).\n- To reduce the network depth, ParNet sacrifces too much on the number of parameters and FLOPs (as well as inference memory usage even it's not shown in the paper).\n\n[1] Hu, Jie, Li Shen, and Gang Sun. \"Squeeze-and-excitation networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n[2] Szegedy, Christian, et al. \"Going deeper with convolutions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MWlH6RGrDKf", "reviewer": "Reviewer_t5as", "summary": "This paper presents a new CNN architecture designed for the purpose of having a relatively low depth (12 layers) while still providing results on par with (deeper) classic ResNets on various vision benchmarks, demonstrating that there may be other mechanisms than depth at play to obtain state of the art results with CNNs. The strong expressive power of the architecture is obtained thanks to parallel substructures in the network, each operating at different scale, then fusing together their information content before the final layers. In-depth ablations study are done to demonstrate the relevance of many components of the design.\nThis work is motivated by the goal of decreasing latency, which can be impacted by depth. In order to improve this metrics, it is shown that, once trained, the network can be expressed equivalently as a classic single stream CNN, increasing its speed at inference.", "questions": "* Why/How did you chose the depth of 12 ?\n* What is the effect of the depth in your new architecture ?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\nThe paper is clearly motivated and very well written. The experiments are numerous, thoroughly comparing the newly introduced CNN architecture to deeper ones on several classic vision datasets. An in-depth ablation study quantifies the impact of most parts of the new design, reporting impressive results for a 12 layers deep network. Comparison between this new design and other standard architectures in terms of latency is done, showing promising results for ParNet.\n\n$\\textbf{Weaknesses}$\n\nWhile the effect of increasing the number of parameters and the number of streams have been studied on this new architectures, the effect of the depth is suprisingly not shown while being the highlighted metric of this paper. How ParNets with a number of layers of similar order of magnitude (e.g., 10,11,13 or 14) perform ? If depth/latency was not an issue, would scaling ParNet to greater depths further increase its performances on standard benchmarks ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "43AI4ktSBQU", "reviewer": "Reviewer_momd", "summary": "Prevailing wisdom suggests neural networks must be of sufficient depth in order to achieve competitive or useful accuracy.  The authors challenge this belief and design a family of networks called ParNets, which use parallel structures that fork quickly from the input and converge again close to the output.  ParNets are built of ParNet Blocks, made up of Rep-VGG blocks with Skip-SE layers to increase the receptive field and avoid increasing depth.  The authors present results on a variety of tasks and against a variety of baseline networks.  Generally, ParNets are able to achieve competitive accuracy and latency.  Ablation studies show the impact of several strategies for boosting accuracy and how scaling various dimensions of ParNets (input resolution, number of streams, number of parameters) affects their accuracy.\n", "questions": "What happens if the streams of ParNet are made deeper?\nConversely, what if depth is decreased - is there a minimum necessary depth for ParNet?\n\nAm I missing a fundamental relationship between depth and latency that doesn't exist between other axes, like layer width, and latency?  (Formulating an equation for minimum latency seems like it should be based on FLOP count and bandwidth requirements rather than something more abstract like \"depth.\")\n", "limitations": "Yes, the limitations have been discussed adequately.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "== Strengths ==\n\nThe submission is intriguing - it boldly challenges conventional wisdom and shows that networks need not be very deep at all to achieve useful, much less competitive, accuracy.  I found the network design methodology to be interesting and grounded in reason.  I imagine much of the NeurIPS attendees and readership will find the topic and approach compelling, at least from a theoretical standpoint.\n\nThe text is not hard to follow; I rarely found myself needing to re-read a sentence due to having trouble parsing it or placing it into context with the preceding text.\n\nThe breadth of experiments is appreciated: not only are there accuracy results, but we also see latency for some experiments, comparisons with different baselines, ablations studies of all types, and different tasks are represented.\n\nFinally, the ultimate goal of showing that a shallow network can compete with deep networks is successful: the accuracy of the 12-layer networks, regardless of the number of streams, stream widths, or input size, is sufficient to make the field take notice.\n\n== Weaknesses ==\n\nFor the successes I describe above, I think the exposition and successive analysis are in need of attention before I can recommend the submission for acceptance.\n\nI found the motivation to be lacking nuance and the focus on network depth to be myopic.  \n- In the abstract, it is asserted that \"latency is fundamentally dependent on the depth of the network.\"  This is also true of the number of parameters width of each layer - wider layers require more FLOPs, increasing latency on a given piece of hardware.  Similarly, latency is fundamentally dependent on the input resolution, choice of activation function (is it a simple clamp() function, like ReLU, or does it involve a transcendental function?), and choice of layer types (3x3 convolution vs. 1x1 convolution vs. fully-connected vs. depthwise vs. …).\n- This focus on depth gets stranger with line 20's \"the lowest achievable latency is d/f\" - layer-seconds per cycle is an odd unit of latency.  I think I understand the intent, but I can't help but feel it could be stated less sensationally.\n- The focus on depth is similarly out of place in Section 3.5's Line 177-178: \"All these factors make non-deep parallel structures advantageous…\", but nothing in this section was related to network depth, just parallel operations.  \n- Line 264 suggests that \"it is surprising that a mere depth-12 network could achieve…. This further indicates that non-deep networks can work as well as deeper counterparts.\"  Strictly speaking, any surprise at the results does not indicate anything about non-deep networks' relative performance to their deeper counterparts.  What's really shown is that ParNet's structure allows for shallower networks than the other network structures studied.\n\nAn obvious experiment was left un-performed, in light of the claim that non-deep networks can work as well as deeper counterparts: what happens if the streams of ParNet are made deeper?  Does performance not improve, supporting this assertion?  Conversely, what if depth is decreased - is there a minimum necessary depth for ParNet?  Any results here would not diminish the contributions, but they would help put them into perspective.\n\nI noted several unfair or misleading comparisons in the results:\n- Figure 1 mentions that \"for fairness,\" the authors \"exclude results with longer training, higher resolution, or multi-crop testing.\"  However, the authors also seem to report their own networks' accuracy results with higher resolution in Table 4's results, as described in Line 237: \"We use this higher image resolution for ParNext-XL and ParNet-XL-CSP.\"\n- Table 2 presents latency results for several baselines networks and several ParNets, with the stated conclusion that, \"In spite of communication overhead, ParNet is faster than similar-performing ResNets.\"  This is not a fair comparison, as the ParNets are using 3x as many compute resources, detailed in Line 210-211: \"… for the multi-GPU version, we use 3 GPUs.\"  This detail is not reflected in the table, where most readers would notice.\n- Table 6 is organized such that models with similar sizes are grouped together, and then ParNets are compared only within those groups - this shows that ParNets are within the top three most accurate networks in each group.  However, this disguises other potential comparisons, such as DenseNet (Bottleneck+Compression) with a depth of 250, with the final group.  If it were included, it would show that a network with less than half as many parameters has higher accuracy than the largest ParNet model.\n\t\nI was waiting for a comparison with ensembling, which is an obvious cousin to the ParNet structure: send an input to separate networks, collect the results, and reduce to a single output.  However, the only comparison is with ensembling ParNet-M models.  A better choice might be ResNet-50; as presented in Table 2, it has higher accuracy, fewer parameters, and fewer FLOPs than ParNet-M.  Ensembling two RN50 models would have roughly the same number of parameters as a single ParNet-L.  Three RN50s would make use of the three GPUs afforded to ParNet-L in Table 2, so latency for either ensemble would be roughly identical.  If the latency is roughly the same for improved accuracy (pending the experiment!), then why would ParNets have an advantage?\n\nThe conclusion from Figure 3 on Lines 299-301 is that \"Based on these charts, we see no saturation in performance while scaling ParNets.  This indicates that by increasing compute, one could achieve even higher performance with ParNet while maintaining low depth.\"  This is in contrast to Table 10, though, which shows explicitly that scaling from a third to a fourth branch reduces Top-1 accuracy.  It seems that Figure 3 simply does not scale far enough to see the saturation shown in Table 10.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2JNlxK-qRYw", "reviewer": "Reviewer_cvGo", "summary": "This paper proposes a non-deep network for low latency. The key designs and techniques include multi-stream topology, RepVGG-style structural re-parameterization and SSE block. Reasonable results on ImageNet, CIFAR and COCO are reported. The results show that shallow models can perform competitively with deep ones.", "questions": "Suggestions:\n1. A discussion with RepVGG may highlight the contributions of this paper as well as show the differences. For example, \"this paper focus on the depth while RepVGG was more about the simplicity, .........., this paper is related to RepVGG because a RepVGG-style overall architecture is suitable for shallow model design\".\n\n2. L103: \"VGG-style block\" seems confusing. A reader may question why a single 3x3 conv is called a block. I would suggest the authors use \"VGG-style architecture\" or \"RepVGG-style design\".\n\n3. L133: missing reference. L143: missing reference\n\n4. Caption of Table 2: I would suggest the authors use 3$\\times$3 instead of 3X3.\n\n5. L311: \"consists of consists of\"\n", "limitations": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes. I appreciate the discussions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nS1. This paper shows that a 12-layer model can achieve accuracy of over 80% on ImageNet and an AP of 48% on MS-COCO. The results show that CNNs do not need to be very deep and research on shallow models is still useful, which is valuable to the literature. This is the primary reason I recommend accepting this paper.\nS2. The key techniques include RepVGG-style structural re-parameterization, multi-stream design and SSE block. The motivations are well explained and the effects are verified.\nS3. The results with custom hardware settings are particularly interesting. (For the multi-GPU version, each stream is launched on a separate GPU) The results and discussions of the communication overhead are useful.\n\nWeaknesses\nW1. The reported results are all small- and middle-sized models. It is not sure if heavy-weight models show different depth v.s. performance trade-offs. Of course, I understand the authors may have limited computing resources and did not lower my rating for lack of such large-scale experiments.\nW2. Though the discussions of the future hardware are interesting, this paper shows no clue on how to realize it. Maybe the authors can show a potential direction.\nW3. The writing can be improved. Please see the suggestions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept