Non-deep Networks
Authors: Ankit Goyal
Alexey Bochkovskiy
Jia Deng
Vladlen Koltun
Venue: ICLR 2022 Submitted (2022)
Keywords: non-deep networks

Abstract: Depth is the hallmark of deep neural networks. But more depth means more sequential computation and higher latency. This begs the question -- is it possible to build high-performing ``non-deep" neural networks? We show that it is. To do so, we use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, we show, for the first time, that a network with a depth of just 12 can achieve top-1 accuracy over 80% on ImageNet, 96% on CIFAR10, and 81% on CIFAR100. We also show that a network with a low-depth (12) backbone can achieve an AP of 48% on MS-COCO. We analyze the scaling rules for our design and show how to increase performance without changing the network's depth. Finally, we provide a proof of concept for how non-deep networks could be used to build low-latency recognition systems. We will open-source our code.

Under review as a conference paper at ICLR 2022
NON-DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Depth is the hallmark of deep neural networks. But more depth means more se- quential computation and higher latency. This begs the question - is it possible to build high-perforning "non-deep" neural networks? We show that it is. To do so, we use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, we show, for the first time, that a network with a depth of just 12 can achieve top-1 accuracy over 80% on ImageNet, 96% on CIFARIO, and 81% on CIFARI00. We also show that a network with low-depth (12) back- bone can achieve an AP of 48% on MS-COCO. We analyze the scaling rules for our design and show how to increase performance without changing the network's s depth. Finally, we provide proof of concept for how non-deep networks could be used to build low-latenee recognition systems. We will open-source our code.
INTRODUCTION
Deep Neural Networks (DNNs) have revolutionized the fields of machine learning, computer vision, and natural language processing. As their name suggests, key characteristic of DNNs is that they are deep. That is, they have a large depth, which can be defined as the length of the longest path from an input neuron an output neuron. Often neural network can be described as linear sequence of layers. i.e. groups of neurons with no intra-group connections. In such cases, the depth of network is its number of layers.
has been generally accepted that large depth is an essential component for high-performing net- works because depth increases the representational ability of network and helps learn increasingly abstract features (He et al. 2016a). In fact, one of the primary reasons given for the success of ResNets that they allow training very deep networks with as many as 1000 layers (He et al. 2016a). As such, state-of-the-arn performance is increasingly achieved by training models with large depth, and what qualifies as "ddep"" has shifted from "2 or more layers" in the early days of deep learning to the "tens or hundreds of layers" routinely used in today's models. For example, as shown in Figure competitive benchmarks such as ImageNet are dominated by very deep mod- els (He et al. 2016al Huang et al., 2017 Tan & Le 2019) with at least 30 layers, whereas models with fewer than 30 layers perform substantially worse. The pess-performin model with fewer than 20 layers has a top-1 accuracy of only 75.2, substantially lower than accuracies achievable with 30 or more layers when evaluated with a single image crop (He et al. 2013) Tan & Le 2019).
But large depth always necessary? This question is worth asking because large depth is not without drawbacks. A deeper network leads to more sequential processing and higher latency; it is harder to parallelize and less suitable for applications that require fast responses.
In this paper, we study whether it is possible to achieve high performance with "non-deep" neural networks, especially networks with 10 layers. We find that, contrary conventional wisdom, this is indeed possible. We present a network design that is non-deep and performs competitively against its deep counterparts. We refer to our architecture as ParNet (Parallel Networks). We show, for the first time, that classification network with depth of just 12 can achieve accuracy greater than 80% on ImageNet, 96% on CIFARIO, and 81 % on CIFAR I00. We also show that a detection network with low-depth (12) backhone can achieve an AP of 48% on MS-COCO. Note that the number of parameters in ParNet comparable to state-of-the-an models, as illustrated in Figure []
A key design choice in ParNet is the use of parallel subnetworks. Instead of arranging layers se- quentially, we arrange layers in parallel subnetworks. This design is "embarrassingly paralle!",

---

Under review as a conference paper at ICLR 2022
in the sense that there connec- tions between the subnetworks except at the beginning and the end. This allows us to reduce the depth of the network while maintaining high curacy, It is worth noting that our parallel structures are distinct from "widening" a network by increasing the number of neurons in layer.
ParyeiAL
Repo oe de 101
Repvoo-32
Farkot louns)
ParSee-M (oum))
MNooNee-1
RepvGc-12
ImageÑee Topl
ParNabd ovnns
Resyet BiastanNNao00
DenseNet
Repucc
1.0
DesseNet:121
24
Via--nN- T
ResNwe334
ParNet not only helps answer scientific question about the necessity of large depth, but also offers prac- tical advantages. Due to the paral- lel substructures, ParNet can be ef- ficiently parallelized across multiple processors. We find that ParNet can be effectively parallelized and out- performs ResNets in terms of both speed and accuracy. Note that this achieved despite the extra latency introduced by the communication be- tween processing units. This shows that in the future, with possibly spe- cialized hardware to further mitigate communication latency, ParNet-like architectures could be used for cre- ating extremely fast recognition sys- tems.
Vor-le
Repvoc-vo
MoNiiNNeV2
vog EN-T
70-
Res
160
Dopth log scale)
Top-1 accuracy on ImageNet vs. depth (in log scale) of various models. ParNet performs competitively to deep state-of-the-an neural networks while having much lower depth. Performance of prior models as reported in the literature. Size of the circle is proportional to the number of parameters. Models are evaluated using a single x24 224 crop, except for ViTB-16 and VITB-32 -2 (Dosovit- skiy et al. (202)). which fine-tunes at 384 384 and PReLU- net (He et al.] 2015), which evaluates at 256x Mod- els are trained for 90 to 120 epochs, except for parameter- efficient models such MNASNet (Tan al. 2019). Mo- bileNet (Sandler et al. 2018). and EfficientNet (Tan & Le) 2019), which trained for more than 300 epochs. For fairness, we exclude results with longer training, higher res- olution, or multi-crop testing.
Figure 1:
We also study the scaling rules for ParNet. Specifically, we show that ParNet be effectively scaled by increasing width, resolution, and number of branches, all while keeping depth constant. We observe that the performance of ParNet does not saturate and increases as we increase computational throughput. This suggests that by in- creasing compute further, one can achieve even higher performance while maintaining small depth (~10) and low latency.
To summarize, our contributions are three-fold:
We show, for the first time, that a neural network with depth of only 12 can achieve high performance on very competitive benchmarks (80.7% on ImageNet, 96% on CIFARIO, 81 % CIFAR100).
We show how parallel structures in ParNet can be utilized for fast, low-latency inference.
We study the scaling rules for ParNet and demonstrate effective scaling with constant low depth.
2 RELATED WORK
Analyzing importance of depth. There exists rich literature analyzing the importance of depth in neural networks. The classic work of Cybenko al. showed that a single-layer neural network with sigmoid activations
approximate any function with arbitrarily small error (Cybenko) 1989). However, one needs to use a network with sufficiently large width, which can drastically increase the parameter count. Subsequent works have shown that, to approximate function, deep network with non-linearity needs exponentially fewer parameters than its shallow counterpart (Liang & Srikant 2017). This is often cited as of the major advantages of large depth.
Several works have also empirically analyzed the importance depth and came to the conclusion that under fixed parameter budget, deeper networks perform better than their shallow counter-
2

---

Under review as a conference paper at ICLR 2022
152
552
GRED
Fan
@
Paace Angpeel FC
(a) ParNet
(b) The ParNet block
Figure 2: Schematic representation of ParNet and the ParNet block. ParNet has depth 12 and is composed of parallel substructures. The width of each block in (a) is proportional to the number of output channels in ParNet-M and the height reflects output resolution. The ParNet block consists of three parallel branches: 1 convolution, 3x3 convolution and Sxip-Squeeze-ante sanzze-and-xxiiatton (SSE). Once the training is done, the 1 x and x convolutions can be fused together for faster inference. The SSE branch increases receptive field while not affecting depth.
parts (Eigen al. 2013) Urban et al., 2017). However, in such analysis, prior works have only studied shallow networks with a linear, sequential structure, and it unclear whether the conclusion still holds with alternative designs. In this work, we show that, contrary conventional wisdom, shallow network can perform surprisingly well, and the key is to have parallel substructures.
Scaling DNNs. There have been many exciting works that have studied the problem of scaling neural networks. Tan & Le (2019) showed that increasing depth, width, and resolution leads to effective scaling of convolutional networks. We also study scaling rules but focus the low-depth regime. We find that one can increase the number of branches, width, and resolution to effectively scale ParNet while keeping depth constant and low. Zagoruyko Komodakis (2016) showed that shallower networks with large width can achieve similar performance to deeper ResNets. We also scale our networks by increasing their width. However, we consider networks that are much shallower depth of just 12 compared to 50 considered for ImageNet by Zagoruyko & Komodakis (016)) - and introduce parallel substructures.
Shallow networks. Shallow networks have attracted attention in theoretical machine learning. With infinite width, a single-layer neural network behaves like Gaussian Process, and one can understand the training procedure in terms of kernel methods (Jacot al. 2018). However, such models do not perform competitively when compared to statc-of-the-arn networks (Li et al.) 2019). We provide empirical proof that non-deep networks can be competitive with their deep counterparts.
Multi-stream networks. Multi-stream neural networks have been used a variety of computer
vision tasks such segmentation (Chen et al. 2016 2017), detection (Lin al. 2017]. and video classification (Wu et al. (2016). The HRNet architecture maintains multi-resolution streams through- the forward pass (Wang al. 2020); these streams fused together regular intervals to exchange information. We also use streams with different resolutions, but our network much shal- lower (12 vs. 38 for the smallest HRNet for classification) and the streams are fused only once, at the very end, making parallelization easier.
3 METHOD
In this section, we develop and analyze ParNet, a network architecture that is much less deep but still achieves high performance on competitive benchmarks. ParNet consists of parallel substructures that process features at different resolutions. We refer to these parallel substructures as streams. Features from different streams are fused later stage in the network, and these fused features are used for the downstream task. Figure 2a provides schematic representation of ParNet.
3.1 PARNET BLOCK
In ParNet, we utilize VGG-style blocks (Simonyan & Zisserman 2015). To see whether non-deep networks achieve high performance, empirically find that VGG-style blocks able than ResNet-style blocks (Table (). general, training VGG-style networks is more difficult than their ResNet counterparts (He al. 2016a). But recent work shows that it is easier train net-
suit-

---

Under review as a conference paper at ICLR 2022
works with such blocks if one uses "structural reparameterization: technique (Ding et al. 2021 During training, one uses multiple branches over the 3x3 convolution blocks. Once trained, the multiple branches can be fused into one 3 convolution. Hence, one ends up with a plain network consisting of only 3x3 block and non-linearity. This reparameterization or fusion of blocks helps reduce latency during inference.
We borrow our initial block design from Rep-VGG (Ding et al., [021 and modify it to make more suitable for our non-deep architecture. One challenge with a non-deep network with only 3x3 convolutions is that the receptive field is rather limited. To address this, we build Skip- Gqueeze-Excitatiin (SSE) layer which is based on the Squeeze-ane Squeeze-and--Esee and-Excitation (SE) design (Hu et al. 2018). Vanilla Squeeze-and -nxxitttttxxxitttionnnnn is not suitable for our purpose as it increases the depth of the network. Hence we use Skip-Squeeze-Excitattii design which is applied alongside the skip connection and uses single fully-connected layer. We find that this design helps increase performance (Table 77p. Figure 2h) provides a schematic representation of our modified Rep- VGG block with the Skip-Squeeze- Skip-Squeeze-Exccitatin module. We refer to this block as the RepVGG-SSE.
One concern, especially with large-scale datasets such as ImageNet, is that non-deep network may not have sufficient non-linearity, limiting its representational power. Thus we replace the ReLU activation with SiLU (Ramachandran et al. 2017).
3.2 DOWNSAMPLING AND FUSION BLOCK
Apart from the Rep VGG-SSE block, whose input and output have the same size, ParNet also con- tains Downsampling and Fusion blocks. The Downsampling block reduces resolution and increases width to enable multi-scale processing, while the Fusion block combines information from multiple resolutions.
In the Downsampling block, there no skip connection; instead, add a single-layered SE module parallel to the convolution layer. Additionally, we add 2D average pooling in the x convolution branch. The Fusion block is similar to the Downsampling block but contains an extra concatenation layer. Because of concatenation, the input to the Fusion block has twice as many channels as Downsampling block. Hence, to reduce the parameter count, we use convolution with group 2. Please refer to Figure A1 in the appendix for schematic representation of the Downsampling and Fusion blocks.
3.3 NETWORK ARCHITECTURE
Figure 2a shows schematic representation of the ParNet model that is used for the ImageNet dataset. The initial layers consist a sequence of Downsampling blocks. The outputs of Down- sampling blocks 2, 3, and 4 are fed respectively streams 1, 2, and 3. We empirically find 3 to be the optimal number of streams for a given parameter budget (Table [T). Each stream consists of a series of Rep VGG-SSE blocks that process the features at different resolutions. The features from different streams are then fused by Fusion blocks using concatenation. Finally, the output is passed Downsampling block at depth 11. Similar RepVGG (Ding al. (202T), we use larger width for the last downsampling layer. Table A] in the appendix provides complete specification of the scaled ParNet models that are used in ImageNet experiments.
In CIFAR the images are of lower resolution, and the network architecture is slightly different from the one for ImageNet. First, we replace the Downsampling blocks at depths and 2 with RepVGG- SSE blocks. To reduce the number of parameters in the last layer, we replace the last Downsampling block, which has large width, with a narrower 1 xx convolution layer. Also, we reduce the number of parameters by removing one block from each stream and adding block at depth 3.
3.4 SCALING PARNET
With neural networks, is observed that one can achieve higher accuracy by scaling up network size. Prior works (Tan & Le 2019) have scaled width, resolution, and depth. Since our objective is to evaluate whether high performance can be achieved with small depth, we keep the depth constant and instead scale up ParNet by increasing width, resolution, and the number of streams.
4

---

Under review as a conference paper at ICLR 2022
Depth performance ImageNet. We test on images with 224x resolution. We rerun ResNets (He et al. 201 our training regime for fairness, thus boosting their accuracy. Our ParNet models perform competitively with ResNets while having a low constant depth.
Table 1:
Table 2: Speed and performance of ParNet vs. ResNet. Because of parallel substructures, Par- Net can distributed across multiple GPUs. In spite of communication overhead, ParNet is faster than similar-performing ResNets.
Model
Depih
Top
Speed (samples/s)
# Param
Flops
Acc.
(in M)
(in B)
Model
Depth (in M)
Top-1
Top-5
PurNet-L
77.66
249
54,9
26.7
Acc.
Acc.
ResNet34
ResNet50
77.53
207
25.6
8.2
ResNet
18
69.57
89.24
ResNet
34
73.27
91.26
PurNet-XI XI.
78.55
230
85.0
41.5
ResNet-Botllencc
50
75.99
92.98
ResNet50
50
77.53
207
25.6
8.2
101
77.56
93.79
Table 3: Fusing features and parallelizing the
ResNet (ours)
18
70.15
89.55
substructures across GPUs improves the speed of
ResNet (ours)
34
74.12
91.89
ParNet. Speed was measured on a GeForce RTX
DesNet-Bottleneed (ours)
26
77.53
93.87
3090 with Pytorch 1.8.1 and CUDA 11.1.
ResNet-Bottlenec (ours)
101
79.63
94.68
ParNet-S
12
75.19
92.29
Model
Top-1 Acc.
Speed (samples/sce)
Latency (ms)
ParNet-M
12
76.60
93.02
ParNet-L.
12
77.66
93.6
ParNet-L (Unnfused)
77.66
112
8.95
ParNet-XL
12
78.55
94.13
ParNet-L L (Fused, Single GPU)
77.66
154
6.50
ParNet-L.
77.66
249
4.01
For CIFARIO and CIFARI00, we increase the width of the network while keeping the resolution at 32 and the number of streams at 3. For ImageNet, we conduct experiments by varying all three dimensions (Figure 3)
3.5 PRACTICAL ADVANTAGES OF PARALLEL ARCHITECTURES
The current process of 5 lithography is approaching the 0.5 size of the silicon crystal lattice, and there is limited room to further increase processor frequency. This means that faster inference of neural networks must come from parallelization of computation. The growth in the performance of single monolithic GPUs is also slowing down (Arunkumar et al., 2017). The maximum die size achievable with conventional photolithography is expected to plateau at ~800mm² (Arunkumar et al. 2017). On the whole, plateau is expected not only in processor frequency but also in the die size and the number of transistors per processor.
To solve this problem, there are suggestions for partitioning GPU into separate basic modules that lie one package. These basic modules are easier to manufacture than a single monolithic GPU on a large die. Large dies have a large number of manufacturing faults, resulting in low yields (Kannan cct al. 2015). Recent work has proposed a Multi-Chip waa--ChhipMMuuii Module GPU (MCM-GPU) on an interposer, which is faster than the largest implementable monolithic GPU. Replacing large dies with medium- size dies is expected to result in lower silicon costs, significantly higher silicon yields, and cost advantages AArunkumar al. 2017).
Even if several chips are combined into a single package and are located on one interposer, the data transfer rate between them will be less than the data transfer rate inside one chip, because the lithography size of the chip is smaller than the lithography size of the interposer. Such chip designs thus favor partitioned algorithms with parallel branches that exchange limited data and can be ecuted independently for as long as possible. All these factors make non-deep parallel structures advantageous in achieving fast inference, especially with tomorrow's hardware.
RESULTS
Experiments on ImageNet. ImageNet (Deng et al. 2009) is a large-scale image classification dataset with high-resolution images. We evaluate the ILSVRC2012 (Russakovsky et al., 2015) dataset, which consists of 1. 1.28M training images and 50K validation images with 1000 classes. We train models for 120 epochs using the SGD optimizer, step scheduler with warmup for first
5

---

Under review as a conference paper at ICLR 2022
Table 4: Non-deep networks can be used as back- bones for fast and accurate object detection sys- tems. Speed is measured on a single RTX 3090 using Pytorch 1.8.1 I and CUDA 11.1.
Table 5: A network with depth 12 can get 80. 72% top-1 accuracy on ImageNet. We show how various strategies can be used to boost the performance of ParNet.
Model
Backbone Depth
MS-COCO
Latency (in mi)
Model
Top-I Acc.
Top-5 Acc.
AP
ParNet-XI.
78.55
94.13
YOI O0..004CSPPPP (official)
46.2
21.0
Longer Training
78.97
94,51
YOLOx4-CSP (Ours)
64
47.6
20.0
ParNet XL (Ours)
12
47.5
18.7
Train & Test Res. 320
80.32
94,95
ParNet-XI. (Ours)
12
48.0
16.4
10-crop testing
80.72
95.38
epochs, a learning rate decay of 0.1 every 30th epoch, an initial learning rate of 0.8, and a batch size of 2048 (256 per GPU). If the network does not fit in memory, we decrease the batch size and the learning rate proportionally, for example, a decrease to a learning rate of 0.4 and a batch size of 1024. Unless otherwise specified, the network is trained at x24 resolution. We train our networks with the cross-entropy loss with smoothed labels (Szegedy et al. 2016). We use cropping, flipping. color-jitter, and rand-augment (Cubuk et al., 2020) data augmentations.
In Table D we show the performance of ParNet on ImageNet. We find that one can achieve sur- prisingly high performance with depth of just 12. For fair comparison with ResNets, we retrain them with our training protocol and data augmentation, which improves the performance of ResNets over the official number. Notably, find that ParNet-S outperform ResNet34 by over 1 percentage point with a lower parameter count (19M vs. 22M). ParNet also achieves comparable performance to ResNet with the bottleneck design, while having 4 to 8 times less depth. For example, ParNet-L performs as well as ResNet50 and gets a top-1 accuracy of 77.66 as compared to 77.53 achieved by ResNet50. Similarly, ParNet-XL performs comparably to ResNet101 and gets a top-5 accuracy of 94.13, in comparison to 94.68 achieved by ResNet101, while being 8 times shallower.
In Table we find that ParNet performs favourably to ResNet when comparing accuracy and speed, however with more parameters and flops. For example, ParNet-L achieves faster speed and better accuracy than ResNet34 and ResNet50. Similarly, ParNet- achieves faster speed and better ac- curacy than ResNet50, however with more parameters and flops. This suggests that depending on the use case, one can use ParNets instead of ResNets to trade off speed with more parameters and flops. Note that the high speed can be achieved by leveraging the parallel sub-structure that could be distributed across GPUs (more details below).
In Table we test speed for three variants of ParNet: unfused, fused, and multi-GPU, The unfused variant consists of 3x3 and 1 I branches in the RepVGG-SSE block. In the fused variant, we use the aimutturllreparamettiize
trick to merge the 3x3 and 1x1 branches into a single 3x3 branch (Section 3.1 For both fused and unfused versions, we use a single GPU for inference, while for the GPU version, we use GPUs. For the multi-GPU version, each stream is launched separate GPU, When all layers in a stream are processed, the results from two adjacent streams are concatenated on one of the GPUs and processed further. For transferring data across GPUs we use the NCCL backend in Pytorch.
We find that ParNet can be effectively parallelized across GPUs for fast inference. This is achieved in spite of the communication overhead. With specialized hardware for reducing communication latency, even faster speeds could be achieved.
Boosting Performance. Table 5 demonstrates additional ways of increasing the performance of ParNet, such as using higher-resolution images, a longer training regime (200 epochs, cosine an- nealing), and 10-crop testing. This study is useful in assessing the accuracy that can be achieved by non-deep models on large-scale datasets like ImageNet. By employing various strategies we can boost the performance of ParNet-XL from 78.55 to 80.72. Notably, we reach a top-5 accuracy of 95.38, which is higher than the oft-cited human performance level on ImageNet (Russakovsky et al.) 2015). Although this does not mean that machine vision has surpassed human vision, it provides sense of how well ParNet performs. To the best of our knowledge, this is the first instance of such "human-level" performance achieved by a network with depth of just 12.

---

Under review as a conference paper at ICLR 2022
Table 6: Performance of various architectures on CIFARIO and CIFARI00. Similar-sized models are grouped together. ParNet performs competitively with deep state-of-the-arr architectures while having a much smaller depth. Best performance bolded. The second and third best performing model in each model size block are underlined.
Architecture
Depth
Params (in Millions)
CITRIO
CIFAR100
Error
Error
ResNerl 10 (official)
110
1.7
6.61
ResNerl 10 (reported in Huang et
2016
110
1.7
6.41
27.22
ResNet (Stochastic Depih) (Huang et al. 2016
110
1.7
5.23
24,58
ResNett (pre-act) (He el al. 2016b
164
1.7
5,46
24.33
DenseNet (Huang al. 2017
40
1.0
5,24
24.42
DenseNet (Bottlencck+ Compression)
Huang al. 2017
100
0.8
4.51
22.27
ParNet (Ours)
1.3
5.0
24.62
ResNet (Stochastic Depth) (Huang al.
2016
1202
10.2
ResNet (pre-act) (He al. 2016b
1001
10.2
22.71
WideResNet Zugoruyko Komodakis
21.18
WideResNet (Zugorykoo Komodakas
2016
4.27
20.43
WideResNet (SL) al., 2018
32
12.0
3.88
19.14
FractalNet (Compressed) (Larsson et 2017
41
21.49
DenseNet (Huang et al. 2017]
7,0
4.10
20.20
DenseNet (Botleencs<<CCompeesmmm)) Huang et al. 2017
250
15.3
3.62
17.60
ParNet (Ours)
12
15.5
3,90
20.02
WidcResNet (Zagoruyko & Komodakis 2016
28
36.5
4,00
19.25
WideResNet (Dropout in Res- (oocBlock] Zagoruyko & Komodakis 2016
28
36.5
3.89
18.85
FractalNet Larssn et al. 2017
21
36.8
5.11
22.85
FractalNet DDropout-DDroppaahh Larsson al.
2017
21
36.8
4.59
23.36
DenseNet [Huang et al. 2017
100
27.2
3.74
19.25
DenseNet (Bottlencc<<CCooppsssion)
Huang al. 2017
190
25.6
3,46
17.18
ParNet (Ours)
35
3.88
18.65
Experiments on MS-COCO. MS-COCO (Lin et al. 2014) is an object detection dataset which contains images of everyday scenes with common objects. We evaluate on the COCO-2017 dataset, which consists of 118K training images and 5K validation images with 80 classes.
To test whether non-deep network such as ParNet can work for object detection, we replace the backbone of state-of-the-art single stage detectors with ParNet. Specifically, we replace the CSP- Darknet53s backbone from YOLOv4-CSP (Wang et al. 2021a) with Par which is much shallower (64 vs. 12). We use the head and reduced neck from the YOLOR-D6 model, and train and test these models using the official YOLOR code (Wang et al.] 20216). We retrain YOLOv4- CSP (Wang et al.] 2021a) with protocol (same neck, same head, same training setup) for fair comparison and it improves performance over the official model. Additionally, for fair comparison, test the ParNet-XI-CSP model by applying the CSP (Wang al. 202 approach ParNet- XL. We find that ParNet-XL and ParNet-XL-CSP are faster than the baseline even at higher image resolution. We thus higher image resolution for ParNet-XL and ParNet-XL-CSP.
In Table A we find that even on single GPU, ParNet achieves higher speed than strong baselines. This demonstrates how non-deep networks could be used to make fast object detection systems.
Experiments on CIFAR. The CIFAR datasets consist of colored natural images with 32x pixels. CIFAR-10 consists of images drawn from 10 and CIFAR-I00 from 100 classes. The training and test sets contain 50,000 and 10,000 images respectively. We adopt standard data augmentation scheme (mirroring/shifting) that is widely used for these two datasets (He et al.) 2016 Zagoruyko & KKomodakis 2016 Huang et al. 2017). We train for 400 epochs with batch size of T28. The and decreased by factor of
30%, 60%, and 80% of the epochs in (Zagoruyko & Komodakis, 2016). Similar to prior works (Zagoruyko & Komodakis 2016 Huang et al., 2016), we use a weight decay of 0.0003 and set dropout in the convolution layer at 0.2 and dropout the final fully-connected layer at 0.2 for all our networks both datasets. We train each network on GPUs (a batch size of 32 per GPU) and report the final test set accuracy.
initial learning rate
Table 6 summarizes the performance of various networks CIFARIO and CIFAR100. We find that PanNet performs competitively with ofate-of-the-art deep networks like ResNets and DenseNets while using much lower depth and comparable number of parameters. ParNet outperforms

---

Under review as a conference paper at ICLR 2022
Ablation of various choices for Data augmentation, SiLU acti- vation, and Skip-Squeeze-Exciin Excitation (SSE) improve performance.
Table 7:
Table 8: ParNet outperforms non-deep ResNet vari-
ParNet.
ants. At depth 12, VGG-style blocks outperform
ResNet blocks, and three branches outperform sin- gle branch.
Model
Params
Top-1
Top-5
Name
Block
Branch
Depth
Parums
Top-1
M)
Acc.
Acc.
(in M)
Apc.
Baseline
32.4
75.02
92.15
ResNett1-WWi
ResNie
12
39.0
73.52
Data Augmentation
32.4
75.12
92.00
RosNet-BN
14
39.0
72.06
ResNet12 (2--iii BN
ResNer-SSS
12
39.0
73.91
SSE
35.6
76.55
93.01
M-neSSemnn ParNet-M
RepVGGGSSE
12
36.0
75.83
SSLU
35.6
93.07
RepVGG-SSS
12
35.9
76.5
Table 9: ParNet outperforms ensembles across different parameter budgets.
ParNet-M
Single Stream
ParNet-L.
Ensemble
ParNet-XT.
Ensemble
(2 Single Streams)
(3 Single Streams)
# Parum 35.9
36 75,83
55 77.66
72 77.20
85.3
108
ImageNet Top-1
76.60
78.55
77.68
ResNets that are 10 times deeper on CIFARIO (5.0 vs. 5.23) while using a lower number of parame- ters (1.3M 1.7M). Similarly, ParNet outperforms ResNets that are 100 times deeper CIFAR10 (3.90 vs. 4.62) and CIFAR100 (20.02 vs. 22.71) while using 50% more parameters (15.5M vs. 10.2M).
ParNet performs as well as vanilla DenseNet models (Huang et al 2017) with comparable parameter counts while using 3-8 times less depth. For example, on CIFAR 00, ParNet (depth 12) achieves an слror of 18.65 with 35M parameters and DenseNet (depth 100) achieves an error of 19.25 with M parameters. ParNet also performs on par or better than Wide ResNets (Zagoruyko & Komodakis 2016) while having 2.5 times less depth. The best performance on the CIFAR dataset under a given parameter count is achieved by DenseNets with the bottleneck and reduced width (compression) design, although with an order of magnitude larger depth than ParNet.
Overall, it is surprising that mere depth-12 12 network could achieve an accuracy of 96.12% on CIFARIO and 81.35% on CIFAR100. This further indicates that non-deep networks can work as well as deeper counterparts.
Ablation Studies. To test if we can trivially reduce the depth of ResNets and make them wide, and ResNetl2-Wide-SSE
we test three ResNet variants: ResNet12-Wide, 4esNet14-Wide-Pe ResNet12-Wide uses the basic ResNet block and has depth 12, while ResNet14-Wide-Bh uses the bottleneck ResNet block and has depth 14. Note that with the bottleneck ResNet block, one cannot achieve depth lower than 14 while keeping the original ResNet structure as there has to be initial convolution layer, 4 downsampling blocks (3 x 4 = 12 depth), and fully-connected layer. We find that ResNee12-Wing
outperforms ResNet1 Widee-BN with the same parameter count. We addition- ally add SSE block and SiLU activation to ResNet Wide to create control for confounding factors. We find that ParNet-M outperforms all the ResNet variants which have depth 12 by 2.7 percentage points, suggesting that trivially reducing depth and increasing width not as effective as our approach. We show that the model with three branches performs better than a model with a single branch. We also show that with everything else being equal, using VGG- style blocks leads better performance than the corresponding ResNet architecture. Architectural choices in ParNet like parallel substructures and VGG-style blocks are crucial for high performance at lower depths.
1--Wide-SSE to further
Table 7 reports ablation studies over various design choices for network architecture and train- ing protocol. We show that each of the three decisions (rand-aud crand-agmment data augmentation, SiLU activation, SSE block) leads to higher performance. Using all three leads to the best performance.
In Table 10 we evaluate networks with the same total number of parameters but with different numbers of branches: 1, 2, 3. and 4. We show that for fixed number of parameters, network with 3 branches has the highest accuracy and is optimal in both cases, with a network resolution of 224x224 and 320x320.
8

---

Under review as a conference paper at ICLR 2022
Table 10: Performance vs, number of streams, For a fixed parameter budget, streams is optimal.
of Branches (Res. 224)
# of Branches (Res. 320)
2
3
4
2
3
4
ImageNet Top-1
75.83
76.1
76.75
76.34
76.91
77.12
77.56
77.46
80
77
Mis1-320
L-11-224
X1-63-320
Imeepee Tbp- Artoney (No
M--3-2244
Enspeaad Aocanay
76.5
L-s3-2 L-63-256
256
C-51-256
77
Mish 256
T,51-2244
Xt.51-224
M-s1-256
stream 224x/24
76
Scale realuttion
stream high resolution
Mist 224
Scate
36-8--221
streams hich resolution
75
Scate channels
30
40
50
60
70
20
90
100
73.5
Number Passmelets (Millions)
35
40
45
50
55
Number of Parmeturs (Millions)
Figure 3: Performance of ParNet increases as we increase the number of streams, input resolution, and width of convolution, while keeping depth fixed. The left plot shows that under a fixed parameter count, the most effective way scale ParNet is to use 3 branches and high resolution. The right plot shows the impact on performance by changing only one of the aforementioned factors. We do not observe saturation in performance, indicating that ParNets could be scaled further increase performance while maintaining low depth.
ParNet vs. Ensembles. Another approach to network parallelization is the creation of ensembles consisting of multiple networks. Therefore, we compare ParNet to ensembles. In Table 9 find that ParNet outperforms ensembles while using fewer parameters.
Scaling ParNet. Neural networks be scaled by increasing resolution, width, and depth TTan & Le, 2019). Since we are interested in exploring the performance limits of constant-depth networks, scale ParNet by varying resolution, width, and the number streams. Figure SaareeBshhoe shows that each of these factors increases the accuracy of the network. Also, we find that increasing the number of streams is more cost-effective than increasing the number of channels in terms of accuracy versus parameter count. Further, we find that the most effective way to scale ParNet is to increase all three factors simultaneously. Because of computation constraints, we could not increase the number of streams beyond 3, but this is not hard limitation. Based these charts, performance while scaling ParNets. This indicates that by increasing compute, one could achieve even higher performance with ParNet while maintaining low depth.
saturation in
DISCUSSION
We have provided the first empirical proof that non-deep networks can perform competitively with their deep counterparts in large-scale visual recognition benchmarks. We showed that parallel sub- structures can be used to create non-deep networks that perform surprisingly well. We also demon- strated ways scale up and improve the performance of such networks without increasing depth.
Our work shows that there exist alternative designs where highly accurate neural networks need not be deep. Such designs can better meet the requirements of future multi-chip processors. We hope that our work can facilitate the development of neural networks that are both highly accurate and extremely fast.
Reproducibility Statement. We will ooen-source our code. This would enable readers to reproduce work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Akhil Arunkumar, Evgeny Bolotin, Benjamin Cho, Ugljesa Milic, Eiman Ebrahimi, Oreste Villa, Aamer Jaleel, Carole-Jean Wu, and David Nellans. Mem-gpu: Multi-chip-mduule gpus for con- tinued performance scalability. In ISCA, 2017.
Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale- aware semantic image segmentation. In CVPR, 2016.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, 2017.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
George Cybenko. Approximation by superpositions of a sigmoidal function. In Mathematics of Control, Signals and Systems, 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. arXiv:2101.03697 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
David Eigen, Jason Rolfe, Rob Fergus, and Yann LeCun. Understanding deep architectures using recursive convolutional network. arXiv:1312.1844 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual
networks. In ECCV, 2016b.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-eecitation networks. In CVPR, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In NeurlPS, 2018.
Ajaykumar Kannan, Natalie Enright Jerger, and Gabriel H Loh. Enabling interposer-based disinte- gration of multi-core processors. In MICRO, 2015.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals. In ICIR, 2017.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kernels. arXiv:1911.00800 2019.
Shiyu Liang and Rayadurgam Srikant. Why deep neural networks for function approximation? In ICLR, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
10

---

Under review as a conference paper at ICLR 2022
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv:1710.05941, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Christian Szegedy. Vincent Vanhoucke. Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
Gregor Urban, Krzysztof Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convolu- tional nets really need to be deep and convolutional? In ICLR, 2017.
Chien- Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. In CVPR, 2021a.
Chien-Yo Yao Wang, I-Hau Yeh, and Hong- Yuan Mark Liao. You only learn one representation: Unified network for multiple tasks. arXiv:2105..m 04206. 2021b.
Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. TPAMI, 2020.
Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, and Xiangyang Xue. Multi-stream multi-class fusion of deep networks for video classification. In ACM Multimedia, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
layer name
layer depih
size
Small
Modium
Large
Extra large
1x1, 64
1x1, 64
1x
1x 64
Downsampling
x H/2
3x3,
3x33, 64
x1
3x.3,
X1
3x3.
stride
stride
stride 2
stride
128
1x
Downsampling
2
W/4 x H/4
3x3, 96
x1
3x3, 128
x1
3x3, 160
x1
3x.3, 200
x1
stride
stride
1x 192
1x1. 256
1x1, 320
1x 400
Downsampling
3
W/8 x H/8
3x3, 192
x1
3x3, 256
3x3, 320
3x3, 400
x1
stride
stride 2
stride 2
strido
1x1. 384
1x1, 512
1x1, 640
Ix1, 800
Downsampling
W/16 H/16
3x 384
x1
3x3, 512
3x3,
3x3,
stride 2
stride 2
strid 2
stride
1x1, 96 3x3,
1x1. 128
1x1, 160
1x1, 200
Stream
3-6
W/4 x H/4
3x3, 128
x4
3x3, 160
3x3,,
x4
SSE
SSE
SSE
SSE
256
Streaml Saiunl-DDwwrr Downsampling
8
W/8 x H/8
3x3. 192 stride
x1
3x3, 256 stride
x1
3x3, 320
x1
3x33, 400 stride
x1
stride 2
1x1,
1x1, 256
1x
Ix
Stream2
4.8
W/8 x H/8
3x3, 192
X5
3x3, 256
x5
3x3, 320
x5
3x3, 400 SSIE
x5
1x 384 3x
Txl, 512
Ix 640 3x3,
1x 800
Stream2. Fussinn
9
W/16 H/16
x1
3x3,
3x3,
stride
stride 2
strido 2
stride
1x 384
Lx 512
1x 640
1x 800
W/16
3x3, 384 SSE
X5
3x3, 512 SSE
x5
3x3, 640 SSE
3x33, 800 SSE
x5
1 384
1x1, 512
1x1,
1x 800
Stream3-Fuudin 3-Fusion
10
W/16 x H/16
3x33, 384 SSE
x1
3x33, 512 SSE
x1
3x3, 640 SSE
x1
3xc3, 800 SSE
x1
1x1, 1280
1x 2048 3x3, 2048 stride
1xl, 2560
1x1, 3200
Downsampling
II
W/32 x H/32
3x3,, 1280
x1
x1
3x33, 2560
x1
3x33, 3200
x
stride
strido 2
stride
Final
12
1x1
average pool, 1000-d fc, softmax
Table A1: Specification of ParNet models used for ImageNet classification: ParNet-S, ParNet-M, ParNet-L, and ParNet-XL..
Batch norm
Batch norm
Concatenation
Shuffle
Avg pool 2x2
Global avg pool
Avg pool 2x2
Global avg pool
1x1 conv, groups=2
3x3 conv, s=2,
1x1 conv, groups=2
groups=2
1x1
3x3 conv, s=2
1x1 conv
+ B Batchnomm nor
Batch norn
Sigmoid
Batch nomn
Batch norm
Sigmoid
SILU U
SIL U
Figure A1: Schematic representation of the Fusion (left) and Downsampling (right) blocks used in ParNet.
12


Reviews:
[{"review_id": "ZIXGSpyZN-m", "reviewer": "Reviewer_75Wm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- empirical result that low-depth networks can be competitive with deeper models is significant. Confirming observations in prior works, the number of parameters and computational complexity of low-depth networks need to be much higher to match the accuracy of deeper networks.\n\nWeaknesses:\n- architecture complexity: the authors claim outperforming ResNet in efficiency, but do not mention that the architecture is far more complex. It is also more complex than RepVGG. This needs to be stressed in the introduction. This is separate from computational complexity.\n- weak baseline: authors use Squeeze-and-Excitation blocks in their architecture which is known to improve ResNet performance, yet compare to ResNet without SE layers (section 4, tables 1-2)\n- missing baseline: the authors introduce parallel streams in their network, which might be working as an ensemble, known to be improving classification performance on the tested datasets. An important baseline then is treating the stream subnetworks as separate models applied on different resolutions, which is simpler than the proposed architecture and would not suffer from communication overhead. A baseline with ResNet and ParNet ensembles needs to be added.\n- statistical significance: test errors on CIFAR and ImageNet have high variance, and the authors show results of a single run for each experiment. A common practice to reduce variance is to train multiple networks and report mean and standard deviation for each experiment.\n- SiLU: the authors replace ReLU with SiLU motivated by the limited representational power of ReLU. However, SiLU was introduced to improve training of deep networks, which is not the case in the proposed architecture. Moreover, in the ablation study (table 7) it does not seem to bring a significant improvement over ReLU, there is also no statistical significance analysis of this result. Using SiLU seems like a needless complication of the proposed architecture.\n\nArguable weakness:\n- definition of depth: the approach takes advantage of the vague definition of depth in neural networks. In prior works, depth is defined as a number of layers in a single stream network. By introducing multiple parallel streams the authors effectively take a deep network and move layers around, so another, perhaps more fair, definition of depth could be counting the total number of layers in all substreams.\n\nNotes to authors:\n- table 3 is misleading because it does not show the number of GPUs used in the last row. It is also not clear if it is fused or not.", "summary_of_the_paper": "The paper proposes a new architecture of a convolutional neural network for image classification. The authors are motivated by inference efficiency and showing that networks with about a dozen of layers can be competitive in classification accuracy with 50 layers and more. So instead of growing depth, they propose to grow width and have multiple subnetworks of the same depth within one model. The proposed architecture is evaluated on CIFAR, ImageNet and COCO classification and detection tasks, and shows that networks with 12 layers can be competitive with deeper counterparts.", "summary_of_the_review": "The paper shows a significant result that a network with 12 layers can be competitive with deeper networks on well established image classification benchmarks. However, the architecture is more complex than ResNet or RepVGG, and the empirical evaluation is unsatisfactory due to weak and missing baselines, and the lack of statistical significance analysis. I hesitate between reject and weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YW6Ua1IVn6K", "reviewer": "Reviewer_pTPq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros)\n+ This paper is easy to follow.\n+ The concept of branch-parallelism in a model looks promising.\n\nCons)\n- I am agreeing that the network depth is a crucial design element towards model efficiency but the computational costs in including the number of parameters, FLOPs, and memory footprints seem to be overlooked in this paper: the proposed model has a larger # of parameters and FLOPs, so the overall computational costs are bigger than the baseline models. \n- The architectural design with the proposed building block in Figure 2 is presented without providing any intuitions or design philosophy.\n- The comparisons with the baselines in model-parallelism is not sufficient\n\nSome comments and questions)\n- The model-parallelism can be done with a single-branch model in training or inferring with multiple images (i.e., with a larger mini-batch size), then the speed gain from the branch-parallelism scheme may vanish. Please clarify a multi-branch architecture also has an advantage over the widened networks such as WideResNet in this case.\n\n- The performance comparison should be done fairly. For example in Table 2, ParNet-L is compared with much smaller networks such as R34 and R50 in terms of the computational budgets, so this is not fair. The models with widening the width while fixing depth (e.g., Wide ResNet) can achieve much better accuracy barely increase the latency.\n\n- Memory consumption would be a matter for the proposed architecture. Please specify the memory footprints when training (or inference) with the fixed batchsize.\n\n- ResNets can also be leveraged fusing methods including skip connections and BNs. Did the authors compare the latency with these fused ResNets with the proposed ParNets in Table 2?\n\n- Why vanilla SE-block is not suitable for the proposed model? Why Rep-VGG block has been adopted?\n\n- Please specify why ParNets cannot outperform DenseNet-100 even using more parameters on the CIFAR datasets in Table 6. I am just wondering whether there is a different earning behavior of a shallow network on a particular dataset.", "summary_of_the_paper": "This paper introduces a way of designing a network fixing the depth yet involving more branches. The authors try to support the claim which is that low-depth networks can achieve a good result by providing experiments on the ImageNet and the CIFAR datasets", "summary_of_the_review": "- See the main review", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UpGm5fyJKpK", "reviewer": "Reviewer_wcwX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) My major concern is that the authors argues contribution of parallel subnetworks, while I think the traditional RepVGG-wise blocks adopted in ParNet contributes significantly more to prediction accuracy. Even if ParNet adopts 1 branch only, ParNet's accuracy can still achieve 75% more, as \"ParNet-M-OneStream\" shown in Table 8. Comparison of Table 7 and Table 10 shows that the RepVGG blocks is significantly more important than adopted number of branches on improving accuracy. \n\n2) As shown in Table 2, ParNet-L/ParNet-XL (12-depth) indeed has fewer depth compared to ResNet50 (50-depth). However, ParNet-L (54.9M)/ParNet-XL (85.0M) has significantly more parameters compared to ResNet50 (25.6M). A potential problem is that if ParNet runs on low-power computing hardware with limited cuda cores, I am not sure whether ParNet still has the advantage on speed compared to ResNet50.\n\n3) The paper proposes a new block RepVGG-SSE by introducing SSE into traditional RepVGG, while ParNet's accuracy improvement from SSE is only 1.53%, as shown in Table 7. Therefore, I think contribution of RepVGG SSE is not sufficient enough.", "summary_of_the_paper": "The paper proposes to manually design a new 12-depth CNN architecture ParNet based on parallel subnetworks instead of traditionally deeply stacked blocks. Experiments show that ParNet is the first CNN achieving over 80% accuracy on ImageNet with 12 depth only. ParNet also achieves a competitive AP of 48% on MS-COCO for object detection.", "summary_of_the_review": "The authors argues contribution of parallel subnetworks, while the experiments show that the traditional RepVGG-wise blocks adopted in ParNet contributes significantly more to prediction accuracy. Therefor, my rating is \"5: marginally below the acceptance threshold\".\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4UU6UoEazZs", "reviewer": "Reviewer_Z9wZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have four considerations:\n- (a) Table 2 only shows the details for ParNet-L and -XL. It could be better to show the full details and comparisons of ParNet-S, -M, -L, and -XL.\n- (b) What are the details of speed testing in Table 2? Do the ResNet and ParNet use the same setting for speed benchmark? If ParNet uses multiple GPUs in parallel to benchmark the inference speed, e.g., 2 GPUs with 128 global batch size, does the ResNet uses 2 GPUs within the global batch size of 128 as well? It could be better to add more words in the third paragraph on Page 6 to show how to conduct the experiments. \n- (c) ParNet develops the RepVGG-SSE based on the work of RepVGG, CVPR 2021. But there is no comparison with this high-related work. It could be better to compare with RepVGG in Table 1 and Table 2.\n- (d) The paper argues that one of the advantages is parallel. But according to Table 2 and RepVGG's Table 4, ParNet has more parameters and larger FLOPs than the competitors. More parameters mean that the model will cost more memory during training and inference. This fact fades the significance of ParNet. It could be better to discuss this and highlight why parallelism is more important than the other two aspects for selling the work.", "summary_of_the_paper": "- **Motivation**. The paper argues that deep networks have several limitations\n  - (a) deep nets have a higher latency;\n  - (b) deep nets are hard to parallelize;\n  - (c) deep nets are not suitable for applications.\n\n\n- **Method**.\nMotivated by these observations, the paper aims to fill the performance gap between shallow networks and deep networks. The paper proposed a 12-layer shallow model framework, which contains proposed RepVGG-SSE blocks, fusion modules for multi-scale processing, and parallel streams.  \n\n\n- **Experiments**.\nThe proposed model, ParNet, is verified on CIFAR-10, CIFAR-100, and ImageNet for classification, MS-COCO for detection. ", "summary_of_the_review": "Please erase the above concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows the possibility to design a relatively shallow architecture, ParNet, based on parallel subnetworks, instead of traditionally deeply stacked blocks. During discussions, the reviewers pointed out two important concerns: (1) the current design heavily hinges on the recently proposed RepVGG block, whose comparison was even missed in the original submission (later added in rebuttal); (2) comparing ParNet with RepVGG, there seems no performance advantage. Although RepVGG is 2.5 times deeper than ParNet, it is still faster due to highly optimized layers.

The authors mainly argued that their contribution is to answer the scientific question “is it possible to build high-performing non-deep neural networks?” While this is indeed an interesting question, AC feels: (1) it is perhaps unfair for this paper to claim as the first work proving the feasibility. WideResNet provided similar insight much earlier, among others; (2) the presented results, with tools being not novel, are pre-mature as they display no real appeal of using ParNet, in any aspect. Probing a new question is of course valuable, but presenting an immature and novelty-lacking answer shouldn't automatically grant publication. 

In sum, the reviewers were unanimously UN-convinced by this paper's value, nor was the AC. The authors are suggested to very seriously take into account reviewers' suggestions to make improvements, before submitting their work to the next venue.
Decision: Reject