Non-deep Networks

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have revolutionized the fields of machine learning , computer vision , and natural language processing . As their name suggests , a key characteristic of DNNs is that they are deep . That is , they have a large depth , which can be defined as the length of the longest path from an input neuron to an output neuron . Often a neural network can be described as a linear sequence of layers , i.e . groups of neurons with no intra-group connections . In such cases , the depth of a network is its number of layers . It has been generally accepted that large depth is an essential component for high-performing networks because depth increases the representational ability of a network and helps learn increasingly abstract features ( He et al. , 2016a ) . In fact , one of the primary reasons given for the success of ResNets is that they allow training very deep networks with as many as 1000 layers ( He et al. , 2016a ) . As such , state-of-the-art performance is increasingly achieved by training models with large depth , and what qualifies as “ deep ” has shifted from “ 2 or more layers ” in the early days of deep learning to the “ tens or hundreds of layers ” routinely used in today ’ s models . For example , as shown in Figure 1 , competitive benchmarks such as ImageNet are dominated by very deep models ( He et al. , 2016a ; Huang et al. , 2017 ; Tan & Le , 2019 ) with at least 30 layers , whereas models with fewer than 30 layers perform substantially worse . The best-performing model with fewer than 20 layers has a top-1 accuracy of only 75.2 , substantially lower than accuracies achievable with 30 or more layers when evaluated with a single image crop ( He et al. , 2015 ; Tan & Le , 2019 ) . But is large depth always necessary ? This question is worth asking because large depth is not without drawbacks . A deeper network leads to more sequential processing and higher latency ; it is harder to parallelize and less suitable for applications that require fast responses . In this paper , we study whether it is possible to achieve high performance with “ non-deep ” neural networks , especially networks with ∼10 layers . We find that , contrary to conventional wisdom , this is indeed possible . We present a network design that is non-deep and performs competitively against its deep counterparts . We refer to our architecture as ParNet ( Parallel Networks ) . We show , for the first time , that a classification network with a depth of just 12 can achieve accuracy greater than 80 % on ImageNet , 96 % on CIFAR10 , and 81 % on CIFAR100 . We also show that a detection network with a low-depth ( 12 ) backbone can achieve an AP of 48 % on MS-COCO . Note that the number of parameters in ParNet is comparable to state-of-the-art models , as illustrated in Figure 1 . A key design choice in ParNet is the use of parallel subnetworks . Instead of arranging layers sequentially , we arrange layers in parallel subnetworks . This design is “ embarrassingly parallel ” , in the sense that there are no connections between the subnetworks except at the beginning and the end . This allows us to reduce the depth of the network while maintaining high accuracy . It is worth noting that our parallel structures are distinct from “ widening ” a network by increasing the number of neurons in a layer . ParNet not only helps us answer a scientific question about the necessity of large depth , but also offers practical advantages . Due to the parallel substructures , ParNet can be efficiently parallelized across multiple processors . We find that ParNet can be effectively parallelized and outperforms ResNets in terms of both speed and accuracy . Note that this is achieved despite the extra latency introduced by the communication between processing units . This shows that in the future , with possibly specialized hardware to further mitigate communication latency , ParNet-like architectures could be used for creating extremely fast recognition systems . We also study the scaling rules for ParNet . Specifically , we show that ParNet can be effectively scaled by increasing width , resolution , and number of branches , all while keeping depth constant . We observe that the performance of ParNet does not saturate and increases as we increase computational throughput . This suggests that by increasing compute further , one can achieve even higher performance while maintaining small depth ( ∼10 ) and low latency . To summarize , our contributions are three-fold : • We show , for the first time , that a neural network with a depth of only 12 can achieve high performance on very competitive benchmarks ( 80.7 % on ImageNet , 96 % on CIFAR10 , 81 % on CIFAR100 ) . • We show how parallel structures in ParNet can be utilized for fast , low-latency inference . • We study the scaling rules for ParNet and demonstrate effective scaling with constant low depth . 2 RELATED WORK . Analyzing importance of depth . There exists a rich literature analyzing the importance of depth in neural networks . The classic work of Cybenko et al . showed that a single-layer neural network with sigmoid activations can approximate any function with arbitrarily small error ( Cybenko , 1989 ) . However , one needs to use a network with sufficiently large width , which can drastically increase the parameter count . Subsequent works have shown that , to approximate a function , a deep network with non-linearity needs exponentially fewer parameters than its shallow counterpart ( Liang & Srikant , 2017 ) . This is often cited as one of the major advantages of large depth . Several works have also empirically analyzed the importance of depth and came to the conclusion that under a fixed parameter budget , deeper networks perform better than their shallow counter- parts ( Eigen et al. , 2013 ; Urban et al. , 2017 ) . However , in such analysis , prior works have only studied shallow networks with a linear , sequential structure , and it is unclear whether the conclusion still holds with alternative designs . In this work , we show that , contrary to conventional wisdom , a shallow network can perform surprisingly well , and the key is to have parallel substructures . Scaling DNNs . There have been many exciting works that have studied the problem of scaling neural networks . Tan & Le ( 2019 ) showed that increasing depth , width , and resolution leads to effective scaling of convolutional networks . We also study scaling rules but focus on the low-depth regime . We find that one can increase the number of branches , width , and resolution to effectively scale ParNet while keeping depth constant and low . Zagoruyko & Komodakis ( 2016 ) showed that shallower networks with a large width can achieve similar performance to deeper ResNets . We also scale our networks by increasing their width . However , we consider networks that are much shallower – a depth of just 12 compared to 50 considered for ImageNet by Zagoruyko & Komodakis ( 2016 ) – and introduce parallel substructures . Shallow networks . Shallow networks have attracted attention in theoretical machine learning . With infinite width , a single-layer neural network behaves like a Gaussian Process , and one can understand the training procedure in terms of kernel methods ( Jacot et al. , 2018 ) . However , such models do not perform competitively when compared to state-of-the-art networks ( Li et al. , 2019 ) . We provide empirical proof that non-deep networks can be competitive with their deep counterparts . Multi-stream networks . Multi-stream neural networks have been used in a variety of computer vision tasks such as segmentation ( Chen et al. , 2016 ; 2017 ) , detection ( Lin et al. , 2017 ) , and video classification ( Wu et al. , 2016 ) . The HRNet architecture maintains multi-resolution streams throughout the forward pass ( Wang et al. , 2020 ) ; these streams are fused together at regular intervals to exchange information . We also use streams with different resolutions , but our network is much shallower ( 12 vs. 38 for the smallest HRNet for classification ) and the streams are fused only once , at the very end , making parallelization easier . 3 METHOD . In this section , we develop and analyze ParNet , a network architecture that is much less deep but still achieves high performance on competitive benchmarks . ParNet consists of parallel substructures that process features at different resolutions . We refer to these parallel substructures as streams . Features from different streams are fused at a later stage in the network , and these fused features are used for the downstream task . Figure 2a provides a schematic representation of ParNet . 3.1 PARNET BLOCK . In ParNet , we utilize VGG-style blocks ( Simonyan & Zisserman , 2015 ) . To see whether non-deep networks can achieve high performance , we empirically find that VGG-style blocks are more suitable than ResNet-style blocks ( Table 8 ) . In general , training VGG-style networks is more difficult than their ResNet counterparts ( He et al. , 2016a ) . But recent work shows that it is easier to train net- works with such blocks if one uses a “ structural reparameterization ” technique ( Ding et al. , 2021 ) . During training , one uses multiple branches over the 3×3 convolution blocks . Once trained , the multiple branches can be fused into one 3×3 convolution . Hence , one ends up with a plain network consisting of only 3×3 block and non-linearity . This reparameterization or fusion of blocks helps reduce latency during inference . We borrow our initial block design from Rep-VGG ( Ding et al. , 2021 ) and modify it to make it more suitable for our non-deep architecture . One challenge with a non-deep network with only 3×3 convolutions is that the receptive field is rather limited . To address this , we build a SkipSqueeze-Excitation ( SSE ) layer which is based on the Squeeze-and-Excitation ( SE ) design ( Hu et al. , 2018 ) . Vanilla Squeeze-and-Excitation is not suitable for our purpose as it increases the depth of the network . Hence we use a Skip-Squeeze-Excitation design which is applied alongside the skip connection and uses a single fully-connected layer . We find that this design helps increase performance ( Table 7 ) . Figure 2b provides a schematic representation of our modified Rep-VGG block with the Skip-Squeeze-Excitation module . We refer to this block as the RepVGG-SSE . One concern , especially with large-scale datasets such as ImageNet , is that a non-deep network may not have sufficient non-linearity , limiting its representational power . Thus we replace the ReLU activation with SiLU ( Ramachandran et al. , 2017 ) . 3.2 DOWNSAMPLING AND FUSION BLOCK . Apart from the RepVGG-SSE block , whose input and output have the same size , ParNet also contains Downsampling and Fusion blocks . The Downsampling block reduces resolution and increases width to enable multi-scale processing , while the Fusion block combines information from multiple resolutions . In the Downsampling block , there is no skip connection ; instead , we add a single-layered SE module parallel to the convolution layer . Additionally , we add 2D average pooling in the 1×1 convolution branch . The Fusion block is similar to the Downsampling block but contains an extra concatenation layer . Because of concatenation , the input to the Fusion block has twice as many channels as a Downsampling block . Hence , to reduce the parameter count , we use convolution with group 2 . Please refer to Figure A1 in the appendix for a schematic representation of the Downsampling and Fusion blocks .

Summary: The paper proposes a new architecture of a convolutional neural network for image classification. The authors are motivated by inference efficiency and showing that networks with about a dozen of layers can be competitive in classification accuracy with 50 layers and more. So instead of growing depth, they propose to grow width and have multiple subnetworks of the same depth within one model. The proposed architecture is evaluated on CIFAR, ImageNet and COCO classification and detection tasks, and shows that networks with 12 layers can be competitive with deeper counterparts.