Non-deep Networks

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have revolutionized the fields of machine learning , computer vision , and natural language processing . As their name suggests , a key characteristic of DNNs is that they are deep . That is , they have a large depth , which can be defined as the length of the longest path from an input neuron to an output neuron . Often a neural network can be described as a linear sequence of layers , i.e . groups of neurons with no intra-group connections . In such cases , the depth of a network is its number of layers . It has been generally accepted that large depth is an essential component for high-performing networks because depth increases the representational ability of a network and helps learn increasingly abstract features ( He et al. , 2016a ) . In fact , one of the primary reasons given for the success of ResNets is that they allow training very deep networks with as many as 1000 layers ( He et al. , 2016a ) . As such , state-of-the-art performance is increasingly achieved by training models with large depth , and what qualifies as “ deep ” has shifted from “ 2 or more layers ” in the early days of deep learning to the “ tens or hundreds of layers ” routinely used in today ’ s models . For example , as shown in Figure 1 , competitive benchmarks such as ImageNet are dominated by very deep models ( He et al. , 2016a ; Huang et al. , 2017 ; Tan & Le , 2019 ) with at least 30 layers , whereas models with fewer than 30 layers perform substantially worse . The best-performing model with fewer than 20 layers has a top-1 accuracy of only 75.2 , substantially lower than accuracies achievable with 30 or more layers when evaluated with a single image crop ( He et al. , 2015 ; Tan & Le , 2019 ) . But is large depth always necessary ? This question is worth asking because large depth is not without drawbacks . A deeper network leads to more sequential processing and higher latency ; it is harder to parallelize and less suitable for applications that require fast responses . In this paper , we study whether it is possible to achieve high performance with “ non-deep ” neural networks , especially networks with ∼10 layers . We find that , contrary to conventional wisdom , this is indeed possible . We present a network design that is non-deep and performs competitively against its deep counterparts . We refer to our architecture as ParNet ( Parallel Networks ) . We show , for the first time , that a classification network with a depth of just 12 can achieve accuracy greater than 80 % on ImageNet , 96 % on CIFAR10 , and 81 % on CIFAR100 . We also show that a detection network with a low-depth ( 12 ) backbone can achieve an AP of 48 % on MS-COCO . Note that the number of parameters in ParNet is comparable to state-of-the-art models , as illustrated in Figure 1 . A key design choice in ParNet is the use of parallel subnetworks . Instead of arranging layers sequentially , we arrange layers in parallel subnetworks . This design is “ embarrassingly parallel ” , in the sense that there are no connections between the subnetworks except at the beginning and the end . This allows us to reduce the depth of the network while maintaining high accuracy . It is worth noting that our parallel structures are distinct from “ widening ” a network by increasing the number of neurons in a layer . ParNet not only helps us answer a scientific question about the necessity of large depth , but also offers practical advantages . Due to the parallel substructures , ParNet can be efficiently parallelized across multiple processors . We find that ParNet can be effectively parallelized and outperforms ResNets in terms of both speed and accuracy . Note that this is achieved despite the extra latency introduced by the communication between processing units . This shows that in the future , with possibly specialized hardware to further mitigate communication latency , ParNet-like architectures could be used for creating extremely fast recognition systems . We also study the scaling rules for ParNet . Specifically , we show that ParNet can be effectively scaled by increasing width , resolution , and number of branches , all while keeping depth constant . We observe that the performance of ParNet does not saturate and increases as we increase computational throughput . This suggests that by increasing compute further , one can achieve even higher performance while maintaining small depth ( ∼10 ) and low latency . To summarize , our contributions are three-fold : • We show , for the first time , that a neural network with a depth of only 12 can achieve high performance on very competitive benchmarks ( 80.7 % on ImageNet , 96 % on CIFAR10 , 81 % on CIFAR100 ) . • We show how parallel structures in ParNet can be utilized for fast , low-latency inference . • We study the scaling rules for ParNet and demonstrate effective scaling with constant low depth . 2 RELATED WORK . Analyzing importance of depth . There exists a rich literature analyzing the importance of depth in neural networks . The classic work of Cybenko et al . showed that a single-layer neural network with sigmoid activations can approximate any function with arbitrarily small error ( Cybenko , 1989 ) . However , one needs to use a network with sufficiently large width , which can drastically increase the parameter count . Subsequent works have shown that , to approximate a function , a deep network with non-linearity needs exponentially fewer parameters than its shallow counterpart ( Liang & Srikant , 2017 ) . This is often cited as one of the major advantages of large depth . Several works have also empirically analyzed the importance of depth and came to the conclusion that under a fixed parameter budget , deeper networks perform better than their shallow counter- parts ( Eigen et al. , 2013 ; Urban et al. , 2017 ) . However , in such analysis , prior works have only studied shallow networks with a linear , sequential structure , and it is unclear whether the conclusion still holds with alternative designs . In this work , we show that , contrary to conventional wisdom , a shallow network can perform surprisingly well , and the key is to have parallel substructures . Scaling DNNs . There have been many exciting works that have studied the problem of scaling neural networks . Tan & Le ( 2019 ) showed that increasing depth , width , and resolution leads to effective scaling of convolutional networks . We also study scaling rules but focus on the low-depth regime . We find that one can increase the number of branches , width , and resolution to effectively scale ParNet while keeping depth constant and low . Zagoruyko & Komodakis ( 2016 ) showed that shallower networks with a large width can achieve similar performance to deeper ResNets . We also scale our networks by increasing their width . However , we consider networks that are much shallower – a depth of just 12 compared to 50 considered for ImageNet by Zagoruyko & Komodakis ( 2016 ) – and introduce parallel substructures . Shallow networks . Shallow networks have attracted attention in theoretical machine learning . With infinite width , a single-layer neural network behaves like a Gaussian Process , and one can understand the training procedure in terms of kernel methods ( Jacot et al. , 2018 ) . However , such models do not perform competitively when compared to state-of-the-art networks ( Li et al. , 2019 ) . We provide empirical proof that non-deep networks can be competitive with their deep counterparts . Multi-stream networks . Multi-stream neural networks have been used in a variety of computer vision tasks such as segmentation ( Chen et al. , 2016 ; 2017 ) , detection ( Lin et al. , 2017 ) , and video classification ( Wu et al. , 2016 ) . The HRNet architecture maintains multi-resolution streams throughout the forward pass ( Wang et al. , 2020 ) ; these streams are fused together at regular intervals to exchange information . We also use streams with different resolutions , but our network is much shallower ( 12 vs. 38 for the smallest HRNet for classification ) and the streams are fused only once , at the very end , making parallelization easier . 3 METHOD . In this section , we develop and analyze ParNet , a network architecture that is much less deep but still achieves high performance on competitive benchmarks . ParNet consists of parallel substructures that process features at different resolutions . We refer to these parallel substructures as streams . Features from different streams are fused at a later stage in the network , and these fused features are used for the downstream task . Figure 2a provides a schematic representation of ParNet . 3.1 PARNET BLOCK . In ParNet , we utilize VGG-style blocks ( Simonyan & Zisserman , 2015 ) . To see whether non-deep networks can achieve high performance , we empirically find that VGG-style blocks are more suitable than ResNet-style blocks ( Table 8 ) . In general , training VGG-style networks is more difficult than their ResNet counterparts ( He et al. , 2016a ) . But recent work shows that it is easier to train net- works with such blocks if one uses a “ structural reparameterization ” technique ( Ding et al. , 2021 ) . During training , one uses multiple branches over the 3×3 convolution blocks . Once trained , the multiple branches can be fused into one 3×3 convolution . Hence , one ends up with a plain network consisting of only 3×3 block and non-linearity . This reparameterization or fusion of blocks helps reduce latency during inference . We borrow our initial block design from Rep-VGG ( Ding et al. , 2021 ) and modify it to make it more suitable for our non-deep architecture . One challenge with a non-deep network with only 3×3 convolutions is that the receptive field is rather limited . To address this , we build a SkipSqueeze-Excitation ( SSE ) layer which is based on the Squeeze-and-Excitation ( SE ) design ( Hu et al. , 2018 ) . Vanilla Squeeze-and-Excitation is not suitable for our purpose as it increases the depth of the network . Hence we use a Skip-Squeeze-Excitation design which is applied alongside the skip connection and uses a single fully-connected layer . We find that this design helps increase performance ( Table 7 ) . Figure 2b provides a schematic representation of our modified Rep-VGG block with the Skip-Squeeze-Excitation module . We refer to this block as the RepVGG-SSE . One concern , especially with large-scale datasets such as ImageNet , is that a non-deep network may not have sufficient non-linearity , limiting its representational power . Thus we replace the ReLU activation with SiLU ( Ramachandran et al. , 2017 ) . 3.2 DOWNSAMPLING AND FUSION BLOCK . Apart from the RepVGG-SSE block , whose input and output have the same size , ParNet also contains Downsampling and Fusion blocks . The Downsampling block reduces resolution and increases width to enable multi-scale processing , while the Fusion block combines information from multiple resolutions . In the Downsampling block , there is no skip connection ; instead , we add a single-layered SE module parallel to the convolution layer . Additionally , we add 2D average pooling in the 1×1 convolution branch . The Fusion block is similar to the Downsampling block but contains an extra concatenation layer . Because of concatenation , the input to the Fusion block has twice as many channels as a Downsampling block . Hence , to reduce the parameter count , we use convolution with group 2 . Please refer to Figure A1 in the appendix for a schematic representation of the Downsampling and Fusion blocks .

Summary: The paper proposes to manually design a new 12-depth CNN architecture ParNet based on parallel subnetworks instead of traditionally deeply stacked blocks. Experiments show that ParNet is the first CNN achieving over 80% accuracy on ImageNet with 12 depth only. ParNet also achieves a competitive AP of 48% on MS-COCO for object detection.