Do Deeper Convolutional Networks Perform Better?

1 INTRODUCTION . Traditional statistical learning theory argues that over-parameterized models will overfit training data and thus generalize poorly to unseen data ( Hastie et al. , 2001 ) . This is explained through the bias-variance tradeoff ; as model complexity increases , so will variance , and thus more complex models will generalize poorly . Modern deep learning models , however , have been able to achieve state-of-the-art test accuracy by using an increasing number of parameters ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) . In fact , while over-parameterized neural networks have enough capacity to interpolate randomly labeled training data Zhang et al . ( 2017 ) , in practice training often leads to interpolating solutions that generalize well . To reconcile this apparent conflict , Belkin et al . ( 2019a ) proposed the double descent risk curve , where beyond the interpolation threshhold , the risk decreases as model complexity increases . In neural networks , model complexity has thus far mainly been analyzed by varying network width . Indeed , in line with double descent , Yang et al . ( 2020 ) ; Nakkiran et al . ( 2020 ) ; Belkin et al . ( 2019a ) demonstrated that increasing width beyond the interpolation threshhold while holding depth constant can decrease test loss . However , model complexity in neural networks can also be increased through depth . In this work , we study the effect of depth on test performance while holding network width constant . In particular , we focus on analyzing the effect of increasing depth in convolutional networks . These networks form the core of state-of-the-art models used for image classification and serve as a prime example of a network with layer constraints . In this paper we answer the following question : What is the role of depth in convolutional networks ? In contrast to what has been shown for increasing model complexity through width , we demonstrate that test performance of convolutional networks worsens when increasing network depth beyond a critical point , suggesting that double descent does not happen through depth . Figure 1 demonstrates the difference between increasing width and depth in ResNets ( He et al. , 2016 ) trained on CIFAR10 . In particular , Figure 1a shows that increasing width leads to a decrease in test error even when training accuracy is 100 % . This effect is captured by the double descent curve . On the other hand , Figure 1b demonstrates that training ResNets of increasing depth but fixed width leads to an increase in test error . Since network depth is a form of model complexity , this behavior contradicts what is expected based on double descent . It is therefore critical to carefully analyze and understand this phenomenon . The main contributions of our work are as follows : 1 . We conduct a range of experiments in the classification setting on CIFAR10 and ImageNet32 using ResNets , fully-convolutional networks , and convolutional neural tangent kernels , and consistently demonstrate that test performance worsens beyond a critical depth ( Section 3 ) . In particular , in several settings , we observe that the test accuracy of convolutional networks is even worse than that of fully connected networks as depth increases . 2 . To gain intuition for this phenomenon we analyze linear neural networks . We demonstrate that increasing depth in linear neural networks with layer constraints ( e.g . convolutional networks or Toeplitz networks ) leads to a decrease in the Frobenius norm and stable rank of the resulting linear operator . This implies that increasing depth leads to poor generalization , when solutions of lower Frobenius norm ( e.g . solutions learned by linear fully connected networks ) do not generalize ( Section 4 ) . 3 . Against conventional wisdom , our findings indicate that increasing depth does not always lead to better generalization . Namely , our results provide evidence that the driving force behind the success of deep learning is not the depth of the models , but rather their width . 2 RELATED WORK . We begin with a discussion of recent works analyzing the role of depth in convolutional networks ( CNNs ) . Yang et al . ( 2020 ) study the bias-variance decomposition of deep CNNs and show that as depth increases , bias decreases and variance increases . This work observes that generally the magnitude of bias is greater than that of variance , and thus overall risk decreases . However , the focus of their analysis on depth is not on the interpolating regime . In fact , they posit that it is possible for deeper networks to have increased risk . We extend their experimental methodology for training ResNets and demonstrate that , indeed , deeper networks have increased risk . Neyshabur ( 2020 ) studied the role of convolutions , but focuses on the benefit of sparsity in weight sharing . Their work analyzed the effect of depth on fully-convolutional networks , but only considered models of two depths . Urban et al . ( 2017 ) analyzed the role of depth in student-teacher CNNs , specifically by training shallow CNNs to fit the logits of an ensemble of deep CNNs . This differs from our goal of understanding the effect of depth on CNN ’ s trained from scratch on CIFAR10 ; furthermore , the ensemble of CNNs they consider only have eight convolutional layers , which is much smaller than the deep ResNets we consider in our experiments . Xiao et al . ( 2018 ) provides initial evidence that the performance of a CNN may degrade with depth ; however , it is unclear whether this phenomenon is universal across CNNs used in practice or simply an artifact of their specific initialization designed to train deep CNNs . In fact , Xiao et al . ( 2020 ) establish that the convolutional neural tangent kernel ( CNTK ) solution approaches that of the neural tangent kernel ( NTK ) as depth increases . In our work , we analyze the generalization of the CNTK as a function of depth in Section 3.3 . We show that as depth increases , test error monotonically decreases and then increases . Lastly , Xiao et al . ( 2018 ) Figure 4a and Xiao et al . ( 2020 ) Figure 2a , b provide examples of accuracy worsening with increasing depth in CNNs , but we demonstrate this phenomenon systematically across a number of settings . Other works have aimed to understand the role of depth in CNNs by characterizing implicit regularization in over-parameterized deep CNNs . Radhakrishnan et al . ( 2019 ) characterized the inductive bias of over-parameterized autoencoders and demonstrated that with sufficient depth , these networks become locally contractive around training examples . Zhang et al . ( 2020 ) similarly studied the role of depth in autoencoders in the more restrictive setting of a single training example . Nguyen & Hein ( 2018 ) studied optimization in deep CNNs and showed that increasing depth increases representational power , while increasing width smooths the optimization landscape . While each of these works identified forms of implicit regularization which occur with depth in CNNs , they did not provide an explicit connection to generalization in CNNs used for classification , which is the focus of our work . On the other hand , previous works studying generalization via double descent have primarily focused on over-parameterization through increasing width . In particular , Belkin et al . ( 2019a ) and Nakkiran et al . ( 2020 ) demonstrated that double descent occurs when increasing the width of neural networks trained on MNIST ( LeCun et al. , 1998 ) and CIFAR10 respectively . Several theoretical works demonstrated double descent theoretically ( Hastie et al. , 2019 ; Belkin et al. , 2019b ; Mitra , 2019 ; Muthukumar et al. , 2020 ; Bibas et al. , 2019 ; Bartlett et al. , 2020 ) , but analyzed linear or shallow non-linear models with an increasing number of features . Our work performs a similar empirical analysis to Nakkiran et al . ( 2020 ) , but on the impact of depth instead of width in CNNs , thereby identifying contrasting behaviors between the two different ways of increasing model complexity . 3 EMPIRICAL EVIDENCE IN NON-LINEAR CLASSIFIERS . We now present our main set of experiments demonstrating that the test accuracy of convolutional networks decreases when increasing depth past a critical threshold . We begin with a demonstration of this phenomenon for fully-convolutional networks applied to CIFAR10 and ImageNet32 . We then demonstrate that this phenomenon holds also for ResNets applied to CIFAR10 . Lastly , we show that this phenomenon occurs for the convolutional neural tangent kernel ( CNTK ) on subsets of CIFAR10 . Our training methodology is outlined in Appendix C . 3.1 IMAGE CLASSIFICATION WITH FULLY-CONVOLUTIONAL NETWORKS . To understand the role of depth in convolutional networks , we begin with a simplified model of a convolutional network , which we call the Fully-Conv Net . The architecture of a Fully-Conv Net of depth d and width w for a classification problem with c classes is depicted in Figure 9 of the Appendix and consists of the following layers : • A convolutional layer with stride 1 , 3 input filers , and w output filters , followed by batch norm ( Ioffe & Szegedy , 2015 ) and a LeakyReLU activation ( Xu et al. , 2015 ) . • d− 1 convolutional layers with stride 1 , w input filters , and w output filters , each followed by batch norm and LeakyReLU activation . • 1 convolutional layer with stride 1 , w input filters , and c output filters . This is followed by an average pool of each of the output filters to produce a c-dimensional prediction . Crucially , this network depends only on convolutional layers , a nonlinear activation , and batch norm ; it does not depend on other components commonly found in deep learning architectures such as residual connections , dropout , downsampling , or fully connected layers . We note that this model is not designed to necessarily perform well , but rather to isolate and understand the effect of increasing the number of convolutional layers . We trained the Fully-Conv Net on 2 , 5 , and 10 classes from CIFAR10 ( Krizhevsky , 2009 ) . All experiments were performed using 5 random seeds to reduce the impact of random initialization . Models were trained using Adam ( Kingma & Ba , 2015 ) with learning rate 10−4 for 2000 epochs , and we selected the model with the best training accuracy over the course of training . We used the Cross Entropy loss , and down-sampled images to 16 × 16 resolution to reduce the computational burden . See Appendix C for a list of all classes used . The resulting train and test accuracies are shown in Figure 2 . As expected , as depth increases , training accuracy becomes 100 % . However , beyond a critical depth threshold , the test accuracy begins to degrade sharply . Furthermore , the value of this critical depth appears to increase as the number of training classes increases . In addition to CIFAR10 , we also applied the Fully-Conv Net to subsets of ImageNet32 ( Chrabaszcz et al. , 2017 ) , which is ImageNet downsampled to size 32 × 32 . We again trained on 2 , 5 , and 10 classes , using the same training procedure as for CIFAR10 . Training and test accuracies for ImageNet32 are shown in Figure 3 . Again , we observe that as depth increases past a critical value , test performance degrades . Remarks . When training to classify between 2 and 5 classes , the test accuracy continues to decrease even when increasing depth past the interpolation threshold , i.e . even after achieving 100 % training accuracy . This in contrast to double descent where increasing model complexity beyond the interpolation threshold leads to an increase in test accuracy . Interestingly , as depth increases , the test accuracy approaches that of a fully connected network . While the Fully-Conv Nets were before or at the interpolation threshold for the 10 class setting in Figures 2 and 3 , Figure 4 demonstrates that a similar decrease in test accuracy occurs also after the interpolation threshold for wider models which can interpolate the data .

Summary: This paper mainly answers a fundamental question: what is the role of depth in convolutional networks? Specifically, the authors present an empirical analysis of the impact of the depth on the generalization in CNNs. Experiments on CIFAR10 and ImageNet32 demonstrate that the test performance beyond a critical depth. My detailed comments are as follows.