Do Deeper Convolutional Networks Perform Better?
Authors: Eshaan Nichani
Adityanarayanan Radhakrishnan
Caroline Uhler
Venue:  (2021)
Keywords: Depth
Over-parameterization
Neural Networks

Abstract:     Over-parameterization is a recent topic of much interest in the machine learning community.  While over-parameterized neural networks are capable of perfectly fitting (interpolating) training data, these networks often perform well on test data, thereby contradicting classical learning theory.  Recent work provided an explanation for this phenomenon by introducing the double descent curve, showing that increasing model capacity past the interpolation threshold leads to a decrease in test error.  In line with this, it was recently shown empirically and theoretically that increasing neural network capacity through width leads to double descent.  In this work, we analyze the effect of increasing depth on test performance.  In contrast to what is observed for increasing width, we demonstrate through a variety of classification experiments on CIFAR10 and ImageNet-32 using ResNets and fully-convolutional networks that test performance worsens beyond a critical depth.  We posit an explanation for this phenomenon by drawing intuition from the principle of minimum norm solutions in linear networks.  


Under review as a conference paper at ICLR 2021
Do DEEPER CONVOLUTIONAL NETWORKS PERFORM BETTER?
Anonymous authors Paper under double-blind review
ABSTRACT
Over-parame parameterization is recent topic of much interest in the machine learning community. While over-parameterized neural networks are capable of perfectly fitting (interpolating) training data, these networks often perform well on test data, thereby contradicting classical learning theory. Recent work provided an explana- tion for this phenomenon by introducing the double descent curve, showing that increasing model capacity past the interpolation threshold can lead to a decrease in test error. In line with this, it was recently shown empirically and theoretically that increasing neural network capacity through width leads to double descent. In this work, we analyze the effect of increasing depth on test performance. In con- trast to what is observed for increasing width, we demonstrate through a variety of classification experiments on CIFARIO and ImageNet32 using ResNets and fully- convolutional networks that test performance worsens beyond critical depth. We posit an explanation for this phenomenon by drawing intuition from the principle of minimum norm solutions in linear networks.
INTRODUCTION
Traditional statistical learning theory argues that over- paraamameetee models will overfit training data and thus generalize poorly to unseen data (Hastie et al.] (200]). This is explained through the bias-variance tradeoff; as model complexity increases, so will variance, and thus more complex models will generalize poorly. Modern deep learning models, however, have been able to achieve state-of-the-ar test accuracy by using an increasing number parameters (Krizhevsky et al. 2012] Simonyan & Zisserman 2015 He et al.) 2016). In fact, while perametratttri neural networks have enough capacity to interpolate randomly labeled training data Zhang et al. (2017), in practice training often leads interpolating solutions that generalize well.
To reconcile this apparent conflict, [elkin et al. (2019a) proposed the double descent risk curve, where beyond the interpolation threshhold, the risk decreases as model complexity increases. In neural networks, model complexity has thus far mainly been analyzed by varying network width. Indeed, in line with double descent, (Yang al. (2020); Nakkiran et al. (2020); BBelkin al. (20T9a) demonstrated that increasing width beyond the interpolation threshhold while holding depth constant can decrease test loss.
However, model complexity in neural networks can also be increased through depth. In this work, we study the effect of depth on test performance while holding network width constant. In particular, we focus on analyzing the effect of increasing depth in convolutional networks. These networks form the core of sfate-of-the-a art models used for image classification and serve as prime example of network with layer constraints. In this paper we answer the following question: What is the role of depth in convolutional networks?
In contrast to what has been shown for increasing model complexity through width, we demonstrate that test performance of convolutional networks worsens when increasing network depth beyond a critical point, suggesting that double descent does not happen through depth. Figure demonstrates the difference between increasing width and depth in ResNets (He et al. 2016) trained on CIFAR 10. In particular, Figure I shows that increasing width leads to decrease in test error training accuracy is 100%. This effect is captured by the double descent curve. On the other hand, Figure Tp demonstrates that training ResNets of increasing depth but fixed width leads to increase
when

---

Under review as a conference paper at ICLR 2021
0.0125
ResNet-10
0.025
ResNet-18
0.0120
ResNet-32 34
Test MSE
ResNet-50
Test MSE
0.0115
0.0110
0.015
0.0105
0.0100
10
20
30
40
50
60
10
20
30
40
50
Depth
(a) ResNet-18 and ResNet-34 of increasing width on CIFARIO
(b) ResNet width 64 on CIFARIO.
Figure (a) As explained by double descent, increasing width in ResNets trained on CIFARIO results in decrease in test error. (b) In contrast, increasing the depth of ResNets trained on CIFARIO results in an increase in test loss (results are averaged across random seeds).
in test error. Since network depth is form of model complexity, this behavior contradicts what is expected based on double descent. therefore critical to carefully analyze and understand this phenomenon.
The main contributions of our work are as follows:
We conduct a range experiments the classification setting on CIFARIO and Ima- geNer32 using ResNets, fully-convolutional networks, and convolutional neural tangent kernels, and consistently demonstrate that test performance worsens beyond a critical depth (Section 3). In particular, in several settings, we observe that the test accuracy of convolu- tional networks is even worse than that of fully connected networks as depth increases.
2. To gain intuition for this phenomenon we analyze linear neural networks. We demonstrate that increasing depth in linear neural networks with layer constraints (e.g. convolutional networks or Toeplitz networks) leads to a decrease in the Frobenius norm and stable rank of the resulting linear operator. This implies that increasing depth leads to poor generalization, when solutions of lower Frobenius norm (e.g. solutions learned by linear fully connected networks) do not generalize (Section 4).
3. Against conventional wisdom, findings indicate that increasing depth does not always lead to better generalization. Namely, our results provide evidence that the driving force behind the success of deep learning not the depth of the models, but rather their width.
RELATED WORK
We begin with discussion of recent works analyzing the role of depth in convolutional networks (CNNs). Yang et al] (2020) study the bias-variance decomposition of deep CNNs and show that as depth increases, bias decreases and variance increases. This work observes that generally the magnitude of bias is greater than that of variance, and thus overall risk decreases. However, the focus of their analysis on depth is not on the interpolating regime. fact, they posit that it is possible for deeper networks to have increased risk. We extend their experimental methodology for training ResNets and demonstrate that, indeed, deeper networks have increased risk.
Neyshabur 2020) studied the role of convolutions, but focuses the benefit of sparsity in weight sharing. Their work analyzed the effect of depth on fully-convolutionnal networks, but only consid- ered models of two depths. |Uran et al. (2017) analyzed the role of depth in student-teacher CNNs, specifically by training shallow CNNs to fit the logits of an ensemble of deep CNNs. This differs from our goal of understanding the effect of depth CNN's trained from scratch CIFAR10; 10; furthermore, the ensemble CNNs they consider only have eight convolutional layers, which is much smaller than the deep ResNets we consider in our experiments.
Xiao et al, (2018) provides initial evidence that the performance of CNN may degrade with depth; however, unclear whether this phenomenon universal across CNNs used in practice simply
2

---

Under review as a conference paper at ICLR 2021
an artifact of their specific initialization designed to train deep CNNs. In fact, Xiao al. (2020) establish that the convolutional neural tangent kernel (CNTK) solution approaches that of the neural tangent kernel (NTK) depth increases. In work, analyze the generalization of the CNTK as a function of depth in Section 3.3 We show that as depth increases, test error monotonically decreases and then increases. Lastly, Xiao al. (2018) Figure 4a and Xiao al. (2020) Figure 2a,b provide examples of accuracy worsening with increasing depth in CNNs, but we demonstrate this phenomenon systematically across number of settings.
Other works have aimed to understand the role of depth in CNNs by characterizing implicit regular- ization in over- wverpparamttriized deep CNNs. Radhakrishnan et al. (2019) characterized the inductive bias of wver-parameterized autoencoders and demonstrated that with sufficient depth, these networks become locally contractive around training examples. Zhang et al.] (2020) similarly studied the role of depth in autoencoders in the more restrictive setting of a single training example. Nguyen & Hein (2018) studied optimization in deep CNNs and showed that increasing depth increases representa- tional power, while increasing width smooths the optimization landscape. While each of these works identified forms of implicit regularization which occur with depth in CNNs, they did not provide an explicit connection to generalization in CNNs used for classification, which is the focus of our work.
On the other hand, previous works studying generalization via double descent have primarily fo- cused over- parameterizaaioo through increasing width. In particular, [elkin et al. (2019a) and Nakkiran et al. (020) demonstrated that double descent occurs when increasing the width of neural networks trained on MNIST (LeCun et al.. 1998) and CIFARIO 10 respectively. Several theoretical works demonstrated double descent theoretically (Hastie et al. 2019, Belkin et al.) 2019b) Mitra 2019 Muthukumar al. 2020 Bibas al. 2019 Bartlett et al. 2020), but analyzed linear or shal- low non-linear models with an increasing number of features. Our work performs similar empiri- cal analysis Nakkiran al. (2020), but on the impact of depth instead of width in CNNs, thereby identifying contrasting behaviors between the two different ways of increasing model complexity.
EMPIRICAL EVIDENCE IN NON-LINEAR CLASSIFIERS
We now present our main set of experiments demonstrating that the test accuracy of convolutional networks decreases when increasing depth past a critical threshold. We begin with demonstration
1.0
Accuracy
0.9
0.9
0.8
0.7
0.7
0.7
0.0
0.6
0.5
0.5
Depth
Depth
Depth
(a) classes
(b) classes
(c) classes
0.65
0.65
Test Accuracy
0.73
0.60
0.63
0.55
0.5
0.63
0.50
0.5
2
8
Depth
Depth
Depth
(d) classes
classes
(f) 10 classes
Figure 2: Train and test accuracy of the Fully-Conv Net as a function of depth, for CIFARIO input
images. Increasing depth beyond critical value leads to a decrease in test accuracy. As depth increases, the performance of the Fully-Conv Net approaches that of a wide fully connected network (shown in red). All experiments performed across 5 random seeds.
3

---

Under review as a conference paper at ICLR 2021
of this phenomenon for collouionvllllioaall networks applied CIFARIO and ImageNet32. We then demonstrate that this phenomenon holds also for ResNets applied to CIFARIO. Lastly, we show that this phenomenon occurs for the convolutional neural tangent kernel (CNTK) subsets of CIFARIO. Our training methodology is outlined in Appendix
3.1 IMAGE CLASSIFICATION WITH FULLY-CONVOIE FILLY-CONOOLUTIONA NETWORKS
To understand the role depth in convolutional networks, we begin with simplified model of convolutional network, which call the Fully-Conv Net. The architecture of a Fully-Conv Net of depth and width w for classification problem with с classes depicted in Figure 9 of the Appendix and consists of the following layers:
A convolutional layer with stride 1, 3 input filers, and output filters, followed by batch norm (loffe & Szegedy 2013) and a LeakyReLUU activation (Xu al., (015).
d convolutional layers with stride by batch norm and LeakyReLU activation.
input filters, and output filters, each followed
convolutional layer with stride 1, w input filters, and e output filters. This is followed by average pool of each of the output filters to produce c-dimensional prediction.
Crucially, this network depends only on convolutional layers, nonlinear activation, and batch norm; does not depend on other components commonly found in deep learning architectures such as residual connections, dropout, downsampling, or fully connected layers. We note that this model is not designed to necessarily perform well, but rather to isolate and understand the effect of increasing the number of convolutional layers.
Net on 2, 5, and 10 classes from CIFARIO (Krizhevsky 2009). All performed using 5 random seeds to reduce the impact of random initialization. Models were trained using Adam (Kingma & Ba 2015) with learning rate 10 for 2000 epochs, and we selected the model with the best training accuracy over the course of training. We used the Cross Entropy loss, and down-sampled images to 16 x 16 resolution to reduce the computational burden. See Appendix C for list of all classes used. The resulting train and test accuracies
We trained the Fully-Conv
experiments
1.0
1.0
18
Train Accuracy
0.8
0.5
0.8
0.6
0.6
0.6
0.4
0.4
C4
4
Depth
Depth
Depth
(a) classes
(b) classes
(c) 10 classes
0.05
con
conv
Test Accuracy
Km
fm
0.8
0.40
0.55
0.80
0.50
0.30
0.70
Depth
Depth
Depth
(d) classes
classes
10 classes
Figure 3: Train and test accuracy of the Fully-Conv Net a function of depth, for ImageNet32 input images. Increasing depth beyond a critical threshold again leads to decrease in test accuracy. Increasing depth beyond critical value leads to decrease test accuracy. As depth increases, the performance of the Fully-Conv Net approaches that a wide fully connected network (shown in red). All experiments are performed across random seeds.
4

---

Under review as a conference paper at ICLR 2021
1.2
10
Accuracy
0.9
44
05-
014
07
05-
04-
Ōepth
δepth
(a) Cifar1O, 10 classes
(b) ImageNet32, 10 classes
Accuracy
0.65
0425
0.00
0422
933
0:75
030
Depth
Depth
(c) CitartO, 10 classes
(d) ImageNet32, 10 classes
Figure 4: Increasing depth well past the interpolation threshold leads to a decrease in test accuracy. All experiments are across 3 random seeds. (a) We trained Fully-Conv nets of increasing depth but fixed width 32 on 10 classes of CIFARIO. (b) We trained Fully-Con nets of increasing depth but fixed width 32 10 classes of ImageNet32.
shown in Figure As expected, as depth increases, training accuracy becomes 100%. However, beyond critical depth threshold, the test accuracy begins degrade sharply. Furthermore, the value of this critical depth appears to increase as the number of training classes increases.
In addition to CIFARIO, I0, we also applied the Fully-Conv Net to subsets of ImageNet32 (Chrabaszez et al., 2017), which is ImageNet downsampled to size 32 32. We again trained on 2, 5, and 10 classes, using the same training procedure as for CIFARIO. Training and test accuracies for ImageNet32 are shown in Figure Again, we observe that depth increases past critical value, test performance degrades.
Remarks. When training to classify between 2 and 5 classes, the test accuracy continues to de- crease even when increasing depth past the interpolation threshold, i.e. even after achieving 100% training accuracy. This in contrast to double descent where increasing model complexity beyond the interpolation threshold leads to an increase in test accuracy. Interestingly, as depth increases, the test accuracy approaches that of a fully connected network. While the Fully-Conv Nets were before or at the interpolation threshold for the 10 class setting in Figures 2 and 3 Figure 4 demonstrates that a similar decrease in test accuracy occurs also after the interpolation threshold for wider models which can interpolate the data.
3.2 IMAGE CLASSIFCAATIO WITH MODERN DEEP LEARNING MODELS
To understand the effect of increasing depth modern machine learning models, we analyzed variants of ResNet trained on CIFARIO. ResNet-18 and ResNet-34 consist of 4 stages, each of which are connected by a downsampling operation. Each stage is comprised of number of basic blocks, which are two layers with a residual connection. There is convolutional layer before the first stage, and fully connected layer after the last stage. ResNet-18 18 uses 2 basic blocks in each stage, while R4sNet-34 uses (3, 4, 6, 3) blocks in each stage respectively. By varying the number of blocks in each stage, we constructed variety of ResNet models of different depths; in particu- lar, by choosing (n1 12, n3, n4)) blocks each stage, we can construct a ResNet model of depth 2 + 2 (nj + n22 + n3 + R4). The width w of model is defined be the number of filters in the first stage; there are then (w, 2w, 4w, Sw) filters in each stage respectively. See Figure [0 of the Appendix for diagram. We trained models up depth 50, see Appendix C for more detailed description of the models used.
For Figure [ and Figure we trained each of our ResNet models using MSE loss random subset of 25,000 training images from CIFAR1O, and plotted the test loss as a function of depth. Our experimental methodology is based that of Yang al. (020). We trained for 500 epochs
5

---

Under review as a conference paper at ICLR 2021
0,020
0.014
0.019
Test MSE
Test
0.013
0.017
0.016
0.012
0.015
20
40
80
100
80
100
Depth
(a) Width
(b) Width 16
Figure 5: Test loss for ResNet models of width 32 (a) and width 16 (b) increases with depth.
using SGD with learning rate 0.1 and momentum 0.9, and we decreased the learning rate by factor of 10 every 200 epochs. We also use their data augmentation scheme of random crops and random horizontal flips on the training data. In the width 64 and 32 models, test loss increases beyond critical depth, and the width 16 model, test loss does not improve beyond certain depth. Plots of
the train and test losses and accuracies of all models are given in the Appendix in Figure 12
Remarks. In Appendix D we provide the training and test accuracies for the ResNets presented in this work. We also demonstrate that increasing depth in later blocks of ResNet leads to more drastic increase in test error than increasing depth in earlier blocks.
3.3 IMAGE CLASSIFICATION WITH THE CONVOLUTIONAL NEURAL TANGENT KERNEL
To remove the effect of additional factors in modern neural networks such as random initializa- tion, width, batch normalization, and down-sampling, turn to the setting of infinite width neural networks. With proper scaling as width approaches infinity, networks is given by solution corresponding kernel regression problem, where the kernel is referred to as the neural tangent kernel, or NTK (Jacot et al., 2018). Arora et al. (2019) computed the NTK for convolutional networks (CNTK); use their code for the experiments in this section.
solution given by training neural
We analyze the effect depth on generalization of the CNTK. Since we are simply rumning kernel regression, predictor perfectly interpolates the training data (i.e. 100% training accuracy and 0 training error). In Figure 6 we compute the CNTK with a training set of roughly 250 examples each of CIFAR10 planes and trucks. We then calculate the test loss and accuracy of roughly 250 planes and trucks. We observe that test error is monotonically decreasing up to a critical depth, after which it is monotonically increasing. This provides further evidence that such a phenomenon occurs more generally in convolutional networks. In Appendix D.6, we present
test
HTx
Test Accuracy
Test Error
0.18
0.99
6.66
099
6.6
15
15"
15'
Depth (a)
Depth (b)
Figure 6: Test loss and accuracy for the NTK and the CNTK of increasing depths (on log scale), trained on a subset of planes and trucks from CIFARIO. Beyond a critical depth, the test error for the CNTK increases and the accuracy decreases.

---

Under review as a conference paper at ICLR 2021
additional empirical evidence that the generalization of the CNTK worsens past critical depth by considering classification problems with a varying number of classes.
4 THE ROLE OF DEPTH IN LINEAR NEURAL NETWORKS
In the previous section, we observed that past a critical depth threshold the performance of CNNs degrades with depth. For Fully-Conv networks, depth increased, test performance appeared to approach that of fully connected network. In this section, we aim to better understand this phe- nomenon, and in particular determine whether the performance of CNNs of increasing depth does indeed approach that of a fully-connected network. We turn to the setting of linear neural networks to simplify this analysis. Linear neural networks are useful for analyzing deep learning phenomena since they represent linear operators but have non-convex optimization landscapes. Furthermore, the solution learned by single layer fully connected network is well understood as it is simply the minimum Frobenius norm solution Engl et al. (1996
In this section, we conduct experiments on linear CNNs of increasing depth both the classification and autoencoding settings. We show that linear CNNs of increasing depth consistently produce solutions of decreasing Frobenius norm, thus approaching the solution learned by a single layer fully connected network. To connect this to generalization, we provide a specific example of classification setting where the solution learned by fully-connected network, and thus linear CNNs of large depth, generalizes poorly, yet shallow CNNs generalize well. We propose that similar mechanism could also explain the decrease in test accuracy beyond a critical threshold in the non- linear setting.
4.1 PRELIMINARIES
Linear networks are models of the form ((x) Wa .. WIx, where W; R is weight matrix. CNNs are a special type of constrained neural network, where each weight matrix W; belongs to a subspace S, R of dimension т;. Since the product of linear operators is linear, these neural networks represent linear functions. In particular, let = W - W1 represent the operator of linear network. We will analyze the Frobenius norm (square root of the sum of squares of the entries) of W and the stable rank (a surrogate for the rank) of W in our analysis.
Definition 1. |Vershyin 2018 Chapter 7) Given a matrix W E IRd Xds, the stable rank of W is given by:
|W 2
E, of
where {o denote the singular values of W.
Computing the Linear Operator Efficiently. Given linear neural network, it is possible to com- pute the linear operator by first constructing the matrix representation for each layer and then mul- tiplying RRahaakisshan al. (2019). Instead, use the following simpler approach for producing the operator. Given a network implementing a map from an s X s image to Rd, we reshape the 52 8² identity matrix as an 8²
tensor (so the batch size is 8²) and feed this through the network. We then reshape the output matrix to be of size d x 5² to obtain the resulting operator.
4.2 LINEAR CONVOLUTIONAL CLASSIFIERS
The following experiment provides an example where fully connected networks do not generalize, and in which linear convolutional classifiers of increasing depth perform similarly to fully con- nected network. Consider a toy dataset of 6 x 6 color images as shown in Figure 77.. We use training examples to represent two classes: Class label has blue pixel in the upper left hand corner and class label -1 has a red pixel in the upper left hand corner. We then construct 200 test examples with 100 having red pixel and 100 having blue pixel in randomly selected location in the lower right 3 x 3 quadrant of the square.
'This assumes zero initialization for the single layer network. When the network has 1 output, the minimum Frohenius norm solution the minimum f norm solution.
7

---

Under review as a conference paper at ICLR 2021
Trinn Exeeppexx
Teai Exempie
Clss-1
Nor of Ooerator
Test
TACAAC
Depth
(A))
(b)
(6)
Figure 7: A toy example demonstrating that increasing depth in linear convolutional networks leads to operators of decreasing & norm, which manifests as a decrease in test accuracy. (a) A visualiza- tion of samples from our toy dataset. (b) The training and test performance of linear convolutional networks of varying depth across random seeds. The test accuracy of the minimum & norm solu- tion for this problem is shown dashed black line. (c) The & norm of the operator with varying depth. The norm of the minimum lz norm solution for this problem is shown as a dashed black line.
While simple, this classification setting is useful for comparing the performance of CNNs and fully connected networks. It is set up to be trivial to solve using the convolution operation, but is such that the minimum Frobenius norm solution would not be able to generalize. Indeed, as demonstrated in Figure pp. linear fully convolutional network with layers and 32 filters per layer is able to consistently get 100% test accuracy across 5 random seeds. On the other hand, performing linear regression to learn the minimum Frobenius norm solution yields 50% test accuracy.
In Figure 7p, we see that increasing the depth of the fully convolutional network leads to a degrada- tion in test accuracy. In particular, the average test accuracy across 5 random seeds is approximately 50% for networks of depth 20 or larger, which obtain 100% training accuracy. Figure 7 compare the Frobenius norm of the corresponding linear operator across depths and see that the Frobenius norm decreases with increasing depth. This simple example demonstrates that the perfor- mance of CNNs of increasing depth approaches that of fully connected network, which does not generalize for many image classification tasks. In Appendix D.7] we present an additional experi- ment with more training samples that again demonstrates similar phenomenon.
4.3 LINEAR AUTOENCODERS
Similar to the case of linear convolutional classifiers, we now demonstrate that increasing depth in linear convolutional autoencoders leads to a decrease in Frobenius norm and stable rank. As done in Radhakrishnan al. (2019); Zhang al. (2020), we begin with the simple example of linear convolutional autoencoder trained on a single image. When there is only a single layer, the authors in Radhakrishhan et al.) (2019); Zhang et al.) (2020) prove that the solution must be full rank due to the sparsity and weight sharing in convolutional layer. This is demonstrated in Figure Si.e where the depth I solution has large stable rank.
In Figure ba,.dd.e. we demonstrate that increasing depth in linear convolutional autoencoders leads solutions of decreasing Frobenius norm and stable rank. In particular, observe that the norm of the trained networks decreases to approximately that of the minimum Frobenius norm solution, which is a projection onto the training examples Radhakrishnan al. (2019): Zhang et al. (2020).
While the previous experiments have considered networks with convolutional layers, in Figure g we present an example of linear autoencoder with alternate layer constraints (Toeplitz layers) for which increasing depth again decreases the operator's Frobenius norm and stable rank. The key similarity between this layer constraint and that of a convolutional layer is that both increase repre- sentational power through depth. Indeed, the authors in Ye & Lim (2016) proved that every matrix can be decomposed as product of Toeplitz matrices, and thus, neural networks with Toeplitz layers expressive through depth. We note that showing that every matrix be written
product of convolutional layers remains open, but a simple parameter counting argument as given in Radhakr- ishnan et al. (2019) implies that representational power increases with depth. This experiment thus
8

---

Under review as a conference paper at ICLR 2021
Squared Norm of Operator
25
Depth
Depth
Depth
Convolutional, example
(b) Convolutional, examples
Toeplitz
Stable Rank of Operator
Depth
Depth
Depth
(d) Convolutional, I example
Convolutional, examples
(f) Toeplitz
Figure 8: Training layer-constrained linear autoencoders of increasing depth leads decrease in the Frobenius norm and stable rank of the resulting operator. (a,d) We train convolutional net- works of varying depth on example. (b.e) We train convolutional networks of varying depth on examples. (c.f) We train network with Toeplitz layers to autoencode 5 dimensional vector.
provides empirical evidence that the phenomenon of decreasing norm with increasing depth occurs generally for networks with layer constraints that are more expressive through depth.
DISCUSSION
In this work, we presented an empirical analysis of the impact of depth on generalization CNNs. We first demonstrated that in modern non-linear CNNs, increasing depth past critical threshold led to a decrease in test accuracy. This result is in stark contrast to the role of width CNNs, as explained by double descent. Furthermore, for Fully-Conv Nets, observed that increasing depth led to performance comparable to that of fully connected networks.
To better understand this phenomenon, demonstrated that increasing depth in these networks led to a decrease in Frobenius norm and stable rank of the learned operator. Moreover, as depth increased, we observed that the norm of the opera- tor approached that of the minimum Frobenius norm solution, which is the solution learned by fully connected network. As demonstrated by our example in Section in settings where the minimum Frobenius norm solution does not generalize, we observe that deep convolutional networks do not generalize. Understanding whether the norm of the functions learned by deep non-linear CNNs ap- proaches that of functions learned by non-linear fully connected networks is an important direction of future work that could explain the poor generalization of deep CNNs observed in this work.
analyzed the operators learned by linear CNNs and
Throughout this work, we consistently observed that increasing depth in CNNs beyond the inter- polation threshold led to decrease test accuracy. Hence, our findings imply that practitioners should decrease depth in these settings obtain better test performance. An interesting direction for future work is to understand where the critical depth threshold occurs as function of width and number of classes. Importantly, if, initial evidence in this paper suggests, the critical depth correlated with the number of classes, then practitioners should use shallow, wide convolutional networks for problems with few classes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang- On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Peter L.. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849 15854, 2019a.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. 1rXiv:1903.07571, 2019b.
K Bibas, Y. Fogel, and M. Feder. A new look at an old problem: A universal learning approach to linear regression. arXiv preprint 190.000005004708, 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative the cifar datasets. arXiv preprint arXiv:1707.08817 2017.
Heinz Werner Engl, Martin Hanke, and Andreas Neubauer. Regularization of Inverse Problems, volume 375. Springer Science & Business Media, 1996.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning, vol- 1. Springer, 2001.
Trevor Hastic, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition (CVPR), 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference in Machine Learning (ICML), 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gener- alization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In Advances Neural Information Processing Systems, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2322 1998.
Partha P. Mitra. Understanding overfitting peaks in generalization error: Analytical risk curves for 12 and penalized interpolation. arXiv preprint arXiv: nete::90600666 2019.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference Leamn ing Representations (ICIR), 2020.
10

---

Under review as a conference paper at ICLR 2021
Benham Neyshabur. Towards learning convolutions from scratch. arXiv preprint arXiv:2007.13657, 2020.
Quynh Nguyen and Matthias Hein. Optimization landscape and expressivity of deep cnns. In International Conference in Machine Learning (ICML), 2018.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Memorization in overpa- rameterized autoencoders. In ICML Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations (ICLR), 2015.
Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shenjje Wang. AbdeIrah- man Mohamed, Matthai Philipose, Matt Richardson, and Rich Caruana. Do deep convolutional nets really need be deep and convolutional? In International Conference on Learning Repre- sentations (ICLR), 2017.
Roman Vershynin. High-Dimensionas Probubility: An Introduction with Applications in Data Sci- ence, volume 1. Cambridge University Press, 2018.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How train 10,000-layer vanilla convolu- tional neural networks. In International Conference on Machine Learning (ICML), 2018.
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and gener- alization in deep neural networks. In International Conference on Machine Learning (ICML), 2020.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical Evaluation of Rectified Activations in Convolution Network, 2015, arXiv:1505.0085:
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking Birianariance Trade-off for Generalization of Neural Networks. In International Conference in Machine Learn- ing (ICML), 2020.
Ke Ye and Lek- Lim. Every matrix is a product of toeplitz matrices. Foundations of Computa- tional Mathematics, 16(3):577-598, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations (ICLR), 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparametterizatio.. In International Conference Learning Representations (ICLR), 2020.

---

Under review as a conference paper at ICLR 2021
APPENDIX
FULLY CONVOLUTIONAL NETWORK ARCHITECTURE
16x16 Image
Average Pool
3x3 conv w filters
3x3 conv w filters
3x3 conv filters
Logits
x ( 1)
Figure 9: A diagram of the Fully-Conv Net used with width w, depth d, and e classes. Each convo- lutional layer (except for the last one) is followed by batch norm and LeakyReLU.
RESNET ARCHITECTURE
Stage (32x32)
Stage (16x16)
Stage (8x8)
Stage (4x44
Fully Connected Layer
32x32 Image
3x3 con fiters
3d cony Riters
3x3 conv fiters
3x3 conv fitees
com 2w fiters
3xd conv fiters
3x3 conv fitters
3x3 comv
3x3 cony How fitter
Logits
Sw fiiters
X
X12
X
Figure 10: A diagram of the custom ResNet models used, The ith block repeated n, times in stage i. The first convolutional layer in stages 2, 3, and downsamples the input using a stride of 2, while the remaining layers have a stride of 1.
C EXPERIMENTAL DETAILS
All models were trained on an NVIDIA TITAN RTX GPU using the PyTorch library.
An anonymized repository with code used for this paper can be found here: https://
anonymous, 4open. wttppessseeeeeeeee////ffffaaeeee 565e 408400888800 ae5f 120991940019
12

---

Under review as a conference paper at ICLR 2021
Dataset
Network
Witth
Activation
Opämizer
Init.
Train Size
a Epochs
Loss
Titals
Figure
5000 /elass
Conv
Leabyflel
100%
Imageliet3
accurny
Fully-Conv
3
3
CIFAR10
Res
64
Default Pytorch
2500/class
500
MSE
3
1a. 12, 13a0
ResNitt
13t
ResNett
16
momentum
13th, 14
Incresse
ResNiet
RelU
18 schedule
from Yang
RecNer10
32
17
ResNar18
Vaned
32
500/class
2000
MSF
classification
Uniform
Convengance
Cifart0
MSE
Figure 11: Experimental details for all experiments conducted.
13

---

Under review as a conference paper at ICLR 2021
Model Depth
Blocks per Stage
10
1, I,
12
1, 2,1,,,1
14
1,2,2,
16
2,2,2, 1
18
2,2,2,
20
2,2,3,2
22
2,3,3,2
26
3,3
30
3,4,
34
3,4,6,3
38
3,4, 8,
42
3,4, 10,3
46
3, 4, 12,
50
3, 4, 14,
Table 1: Stage breakdown for all ResNet models used.
# Classes
Classes used
2
dog, cat
5
bird, cat, deer, dog, horse
10
all classes
Table 2: CIFAR10 10 classes considered in this work.
# Classes
Classes used
2
kit fox, English_ setter
5
2 classes Siberian husky, Australian _errier, English
English_springer springer
10
classes Egyptian cat, Persian cat, malamute, Great Dane, Walker hound
Table 3: ImageNet32 classes considered in this work.
14

---

Under review as a conference paper at ICLR 2021
Width=16
Width=16
Width=32
0.006
Width=64
Width=64
Train MSE
Test MSE
0.004
0.002
0.000
40
50
10
20
Depth
Depth
(a) Train Losses of ResNet models with widths (b) Test Losses of ResNet models with widths 16, 16, 32, and 64.
32, and 64.
Figure 12: Train and Test losses of the ResNet models for all widths.
Train Accuracy
3.00
028
8:0
6.0
027
837
142
025
Depth
Depth
Depth
300
25
Depth
50
200
(a) Width 64
(b) Width = 32
(c) Width 16
(d) Width
0.99
0.91
2.00
Test Accuracy
0:0.0
0.50
3.10
0.89
3.00
0.00
1.3
Depth
Depth
301
Depth
Depth
(c) Width = 64
(1) Width = 32
(g) Width
16
(h) Width =
Figure 13: Train and Test accuracies for the ResNet models of width 8, 16, 32, and 64, for increasing depths.
ADDITIONAL EXPERIMENTS
D.1 ADDITIONAL RESNET PLOTS
In Figure 12 we plot the train and losses of all ResNet models used (for widths 16, 32, 64). Additionally. Il, in Figure 13 we plot the accuracies of all ResNet models.
D.2 EFFECT OF DEPTH MODELS WITH SMALL WIDTHS
The only model in which test loss continues to increase is the width 8 model. We argue this is because the width model is not sufficiently pver-parameterizee:; in fact, in Figure 14, that the width 8 model unable to reach zero training loss, while all the other models are after sufficient depth.
D.3 EFFECT OF DOWNSAMPLING
In Figure 15 we compare ResNet model where we increase the number of blocks in the first stage versus a model where we increase the number of blocks in the third stage. We observe that the model where the third stage blocks increased performs worse. This is likely because adding block later stage, after downsampling, increases the effective depth the model more than adding a block in an earlier stage.
15

---

Under review as a conference paper at ICLR 2021
0.020
Train MSE
0.015
Test MSE
0.010
0.022
0.005
0.020
0.000
0.018
20
40
80
100
20
40
60
100
Depth
Deptn
(a) Train losses the width ResNet model, (b) Test losses width 8 ResNet model, for increasing depths.
increasing depths.
Figure 14:
Train and test losses for the width 8 ResNet model. We see that test loss decreases as model depth increases, but train loss has still not reached O, even for large depths.
Stage
0.015
Stage
Test MSE
0.014
0.013
0.012
10
Depth
Figure 15: Test losses for the width 32 ResNet model where the 1st stage blocks are increased versus the model where the 3rd stage blocks are increased.
0.0034
0.0032
0.0030
Test
0.0078
20
40
Depth
Figure 16: Test losses for the width 32 ResNet samples, trained on 500 samples per class,
D.4 EFFECT OF NUMBER SAMPLES
In Figure 16 is plot of test losses when training the width 32 ResNet model on 500 samples per class ( of CIFAR 10). Number of training epochs increased accordingly. We observe that test loss increases as depth increases, showing that this phenomenon robust to change in sample size.
D.5 EFFECT OF KERNEL SIZE
Another form of overparameteriizatonn is increasing the kernel size for convolutional filters. In Fig- ure 17 we train ResNet- 10 and ResNet-18 of width 32 and varying kernel sizes, and observe that as kernel size increases, test loss increases. This is consistent with our proposed explanation based on expressivity, since increasing kernel size increases representational power independent of depth.
16

---

Under review as a conference paper at ICLR 2021
0.025
Train MSE
0.200
0010
0.005
0.005
25
15
5
12
15
Kernel Size
Kernel Size
(a) ResNet-10
(h) ResNet-18
0.520
Test MSE
0.0.0
Kernel Size (c) ResNet-10
Kernel Size
(d) ResNet-18
Figure 17: Train and test losses for width 32 models for increasing kernel size. We observe that test loss increases as kernel size increases.
0.56-
11:4
0.81
Test Loss
034
2.83
080
0.01
26
40
60
60
100
70
45
60
80
100
20
30
70
Depth
Depth
Depth
(a) 2 classes
(b) 5 classes
(c) 10 classes
0.57
Test Accuracy
0.350
0.345
0SE
0.340
0.6
0.335
0.54
0.31
0.330
46
100
16
Depth
Depth
Depth
(d) 2 classes
(c) classes
(f) classes
Figure 18: Test loss and test accuracy for the CNTK trained on subsets of Cifar10 with 2, 5 and 10 classes. We observe that generalization improves up critical depth, after which it worsens.
D.6 ADDITIONAL CNTK EXPERIMENTS
We also train the CNTK on subsets of CIFARIO of varying number of classes. We use 100 train and 100 test examples per class, and train on both 2 classes (birds and deer), 5 classes (cats, dogs, shown
horses, bids, and deer), and 10 classes (all of CIFAR10). The test losses and accuracies in Figure I8| Again, we see that generalization is unimodel, with test loss decreasing until a critical depth and increasing afterwards, which is in agreement with main CNTK experiment in Figure
17

---

Under review as a conference paper at ICLR 2021
We note that training the CNTK for large depths is computationally prohibitive. The runtime scales quadratically the number of training samples; furthermore, training the depth 500 CNTK on GPU for 500 train and 500 test samples took approximately 2 days.
D.7 ADDITIONAL LINEAR CONVOLUTIONAL NETWORK EXPERIMENTS
120
Frobonus Norm
Min Noom
60-
Test Aocuracy
Norm Operator
55-
70-
4.5
35
Train ACC.
40-
Test Acc.
Mi Norm
10
Depth (a)
Depth
An additional toy example demonstrating that increasing depth in linear convolutional networks leads to operators of decreasing & norm, which manifests decrease in test accuracy. Instead of having only 1 training sample of each class, we now sample from each class randomly from the upper left quadrant of 6 square. Our network uses 64 filters per layer, with kernel (a) The training and test performance of linear convolutional networks of varying depth across 3 random seeds. The test accuracy of the minimum l norm solution for this problem is shown as dashed black line. (c) The & norm of the operator with varying depth. The norm of the minimum & norm solution for this problem shown as a dashed black line.
Figure 19:
size of 3, and zero padding.
D.8 TEST LOSSES FOR FULLY CONVOLUTIONAL EXPERIMENTS
Test Loss
Depth
Depth
Depth
(a) 2 classes
(b) classes
(c) 10 classes
Figure 20: Test losses for models in Figure 2 (on CIFAR10). The error for the convolutional net- works is in blue and that of full connected networks is in red.
18

---

Under review as a conference paper at ICLR 2021
Test Loss
Depth
Depth
Depth
classes
(b)
(c) 10 classes
Figure 21: Test losses for models in Figure 3 (on ImageNet32). The error for the convolutional networks is in blue and that of full connected networks is in red.
Test Loss
Depth
Depth
(a) Cifar10
(b) ImagoNet32
Figure 22: Test losses for the models Figure
19


Reviews:
[{"review_id": "fbuiJYV5G3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Post-rebuttal update:\n\nI am raising my score from 4 to 5, to recognize extensive updates to the paper experiments and numerous clarifications during the discussion, as well as to acknowledge that some of my initial concerns were not justified, e.g. asking for similar studies for MLPs (which was effectively done in the original submission), or questioning whether the Frobenius norm in Figure 7 really goes down (which was demonstrated better in the latest update).\n\nHowever, I cannot give it a higher score / advocate for acceptance because I still find section 4 (and follow-up discussion) to be more misleading/confusing than helpful when it comes to explaining the observed phenomena.\n\nPrecisely, even after the discussion, I still believe that\n\n1) Experiments in Figure 7 should be done with circular-padded convolutions (to rule out the more trivial explanation of the decreasing norm), and on small subsets of CIFAR10/Imagenet32 (to establish whether low-norm is indeed associated with poor performance on image classification).\n\n2) A precise definition of norm for nonlinear networks should be provided, and a link with the Frobenius norm of the linear networks should be established. I don't understand what exactly \"the norm in the corresponding RKHS\" means, and how the reader was supposed to infer it from the text or our earlier discussion. Further, if possible, this norm should also be evaluated on the actual nonlinear networks, and compared to other notions of norms discussed in prior literature (e.g. https://arxiv.org/pdf/1706.08947.pdf), where lower norm is typically associated with better test accuracy.\n\n3) A proper discussion about the non-monotonic dependence on depth in the context of provided intuition should be given. Current intuition can be interpreted as either predicting monotonic decrease in accuracy with depth (learning solutions of lower and lower norms), or as predicting a sharp drop in accuracy at a certain depth (point where the minimum-norm solution can be learned), but neither interpretation explains the hill-shaped dependence, which again makes me question whether this is indeed the right explanation.\n\n\nWithout section 4, the paper still has novel empirical results, but in my opinion they are neither surprising (e.g. a hill-shaped dependence of accuracy is my default expectation of any NN hyper-parameter) nor actionable (there are no hints regarding what the peak depends on / how to guess it) enough for publication at this time. \n\nA more rigorous investigation into explaining the phenomenon, or a more comprehensive empirical exploration to identify what does and what doesn't influence the best depth, would make this a great paper at a later conference.\n\nBest,\nR4.\n\n### Original review:\n\n#### Paper Synopsis:\n\nThe paper empirically studies the effect of depth on generalization in CNNs, and finds that, unlike in the case of width, generalization can decrease beyond a certain critical depth. The paper proposes an intuition for why this happens, by observing that in linear CNNs, the Frobenius norm of the trained network decreases with depth, and thus potentially converges to the minimum-norm solution, which corresponds to simple linear regression. By analogy and with some empirical evidence, the paper conjectures that CNNs converge to MLPs as they become deeper.\n\n\n\n#### Pros:\n\nApart from certain aspects that I discuss below, the paper is clear and easy to read. I especially appreciate considering a variety of experimental settings (different datasets, number of classes, vanilla-CNNs and ResNets), as well as toy examples. Code is provided, which is another strong point of the paper. The question asked in the title of the paper is interesting and worth investigating.\n\n\n\n#### Cons:\n\nMy major concerns are:\n1. The main takeaway message appears either trivial, or at least miscommunicated.\n2. Experimental evidence is unconvincing of [the stronger, non-trivial interpretation of] the key message of the paper and the proposed intuition.\n3. The proposed intuition is not convincing theoretically.\n\nI am open to be persuaded otherwise in case if I misunderstood certain claims. Below are my specific thoughts on each of the points:\n1. What concretely is the claim of the paper? Is it \n\t1. _“For each task, architecture, and training algorithm, there is a depth optimal for generalization?”_ If so, this is a truism. \n\t2. _“..., in addition, beyond this optimal depth, generalization monotonically decreases?”_ This is not supported by Figures 12.(f, g), where it remains roughly flat or even goes up, while train accuracy and loss seem to have reached the optimum per Figures 12.(b, c), and 11.(a). \n\t3. _“among CNNs with 100% training accuracy, the best depth for generalization is the smallest one”_ (per the main contribution #3, page 2)? This is not supported by Figure 1.(b), 12.(e, f), where the best network is not the most shallow.\n\t4. _“The critical depth threshold is independent of certain parameters of the task/architecture/training algorithm”_? Sadly, the paper does not answer this question, but admits this would be a good direction for future work (I agree).\n\t\n  As such, my takeaway from the paper is only that _“Double descent usually doesn’t happen through depth”_ in CNNs. Unfortunately, I find this observation quite trivial, and not particularly novel, see for instance [1, Figure 3].\n    \n2. Below, I interpret the claim of the paper as _“double descent doesn’t happen in CNNs; further, CNN performance converges to that of MLPs as they become deeper”_. Firstly, Figures 2.(b, c, e, f), and 3.(c, f) are all in the classical regime (< 100% training accuracy), and therefore somewhat unrelated to the claim of the paper, which concerns generalization trends of 100% accurate networks (to be clear, including the whole range of depths from 1 is highly appreciated, but to corroborate the claims of the paper multiple depths beyond 100% training accuracy are needed). Further, no plot shows a convincing _convergence_ of CNN performance to that of an MLP, rather, I see that in some cases the curves _intersect_ (in Figure 2, but not in Figure 3). Finally, Figures 12.(f, g) showcase settings where the double descent arguably does occur (i.e. flat or increasing test performance), which makes the claim that it does not happen through depth less robust and more hyper-parameter dependent.\n\n3. Regarding the intuition about CNN to MLP convergence. My key concern is that most image classification datasets, including both considered CIFAR-10 and ImageNet32 cannot be perfectly fit with a linear function. The linear regression solution will not interpolate the training data. Therefore, whatever mechanism is behind the degradation of performance of deep nonlinear CNNs, it cannot possibly be convergence to a linear solution, because they fit the training data perfectly, and the linear solution does not. As such, I believe the intuition that may work in the deep linear case, is _guaranteed_ to not apply to nonlinear networks. If my interpretation of the intuition is wrong, I kindly ask the authors to clarify their reasoning, explicitating precise logical steps and assumptions made.\n\n\nOther, more specific suggestions that may improve the quality of the paper and make the results more convincing:\n1. I believe the paper would be much stronger if it performed (or referenced, if applicable) equivalent studies, including toy examples, for deep MLPs, and compared/contrasted findings with CNNs. This would be especially useful to help validate claims about CNN to MLP convergence. For example, is the decreasing norm with depth specific only to deep linear networks with constraints, or to deep linear networks in general?\n2. I find the experiment in Figure 6 very interesting, but I think it can be strengthened in a few ways. Firstly, could you please provide measurements for depths < 5, as in other figures? Secondly, I assume “SAME”, zero padding was used (if not, please let me know and ignore the following point). In this case, the norm of the deep linear CNN would decrease with depth even without training, at initialization, due to convolving over more and more zeros at the edges as one goes deeper into the network. Therefore, it would be important to control for this effect (for example by using circular-padded convolutions). Finally, I believe this same toy experiment could have been conducted on a small subset of CIFAR-10 (that can be fit linearly), and the results would be more convincing, since I am otherwise not confident the observed effect is not due to some quirk of this specific toy task (e.g. would the same trend be observed if the test samples were everywhere in the image, and not at the bottom quadrant? What if train samples were in the center? What if the training set had multiple squares? etc).\n\n\n#### To summarize: \n1. am convinced by the observations that double descent _does not necessarily happen_ through depth in CNNs. \n2. I am not convinced that it never happens, and I am not convinced one can use it in practice to select optimal depth. \n3. Further, I am not convinced with the intuition and provided evidence that deep CNNs converge to MLPs. \n\nUnfortunately, I find (1) on its own not sufficiently surprising / useful / novel [1, Figure 3] for publication.\n\n[1] [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks, Xiao et al, 2018](https://arxiv.org/abs/1806.05393)", "extra_scores": {}, "extra_text": {}}, {"review_id": "dnmeD8RSciS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary:\n\nThis paper mainly answers a fundamental question: what is the role of depth in convolutional networks? Specifically, the authors present an empirical analysis of the impact of the depth on the generalization in CNNs. Experiments on CIFAR10 and ImageNet32 demonstrate that the test performance beyond a critical depth. My detailed comments are as follows.\n\n*Positive points:\n\n1. This paper is significant to understand deep neural networks and helps to develop new deep learning algorithm.\n\n2. This paper provides many empirical studies to analyze the effect of increasing depth on test performance.\n\n*Negative points:\n\n1. The importance and novelty of the research should be emphasized. Recently, there are some works [1][2][3] study the role of depth in DNN. What is the difference from these works? \n\n[1] Do Deep Convolutional Nets Really Need to Be Deep and Convolutional? ICLR 2017\n[2] Understanding intermediate layers using linear classifier probes. arXiv, 2016.\n[3] Towards Interpreting Deep Neural Networks via Understanding Layer Behaviors. 2020\n\n2. This paper analyzes the linear neural networks and demonstrates that increasing depth leads to poor generalization. However, existing works apply non-linear neural networks in real-world case. It would be better to provide analysis on non-linear neural networks.\n\n3. The authors suggest that practitioners should decrease depth in these settings to obtain better test performance. However, ResNet-101 has better test performance than ResNet-18 in practice. Could you please give more explanations?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AmiRgPK8Gw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*********\nSummary Of The Manuscript: \n*********\nThis manuscript focuses on the problem of the impact of depth in Convolutional Neural Networks (CNNs) for better generalization. To investigate the issue, the authors did an empirical analysis of the problem statement through various strategies (i.e. Deep CNNs - increasing depth vs. Fully Convolutional Networks, etc.) and provided in-depth findings via experiments. Together with experiments and evaluation on standard benchmarks such as CIFAR-10 and ImageNet32, the authors lead to the conclusion of their study that the testing performance will be poor if the increase in depth is higher. \n\n*********\nStrength Of The Manuscript: \n*********\nClarity:\n++ The paper reads very well and provides a very good description of related work and background, motivating the problem. Even outside of the contribution of this paper, I would recommend this paper to people getting started with CNNs as it provides a thorough description of the part of the pipelines it deals with. Also, all the empirical analyses have been described thoroughly and the various settings for the training and testing are performed in such a way to give better insights to novice readers.\n\nNovelty:\n++ In terms of novelty, the stand-alone contribution of the manuscript is that through various strategies authors tried to give in-depth insights on the behavior of different linear and non-linear as well as deep learning models via increasing the depth for classification task on different benchmark datasets. \n\nExperiments:\n++ There have been experiments performed across datasets with variety in terms of different models and their architectures. Additionally, the analyses provided in the manuscript is fairly consistent. Supplementary material also backs the analysis by providing the visualizations of training and testing errors. Besides, analysis/ablation studies are done on the models, by changing the widths, the effect of downsampling, and changing the kernel width, and after through all the settings finally check the performance difference.\n\n*********\nWeakness Of The Manuscript: \n*********\nOverall, apart from the contribution of the paper I have some concerns regarding the paper which are listed below.\n\n-- I believe that the authors did a tremendous job to reach the conclusion that the increase in depth is crucial for certain tasks and might lead to poor results in different situations if the depth is increased beyond the threshold. However, to back up this conclusion more effectively I believe that if the authors have performed more analysis by introducing a data-augmentation strategy,student-teacher training strategy, introducing learning rate variance and so more training strategies this manuscript will be a really a good point of start to a beginner. I encourage the authors to refer to this manuscript by Urban et al. [1].  \n-- The fact that has been mentioned corresponding to increase in depth leads to worsening the result have already been a point of view for much of the deep learning practitioners as He et al. [2] have already provided a really good analysis in the manuscript of ResNet, in contrast, the majority of the work in the current manuscript is already have been either published or have been known to the community. \n\n*********\nJustification Of The Manuscript Review: \n*********\n-- In the reviewer's opinion, in its current form, the paper provides in-depth analysis for increasing /decreasing the depth of CNNs however there are certain points mentioned in the weakness section have been mentioned which needs some clarification from the authors during the rebuttal phase, if answered thoroughly, the reviewer believes that the manuscript will be a really good point of start for novice deep learning readers/practitioners. Therefore the current rating of the paper will be 6 in reviewers' opinion as it is above the acceptance threshold marginally. \n\nReferences: \n[1] Urban, Gregor, et al. \"Do deep convolutional nets really need to be deep and convolutional?.\" In ICLR (2017).\n[2] He, Kaiming, et al. \"Deep residual learning for image recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5m0l8HJms6W", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of paper: The authors empirically study trends in neural network performance for models with fixed width but increasing depth. (Previous work has investigated how increasing \"model complexity\" from neural network width affects test loss/accuracy, with \"double descent\" behavior -- the trends observed here are very different from double descent.) \n\nThe authors consider ResNets and fully-convolutional networks (which contain only convolutional layers with a final pooling operation) on CIFAR-10 and subsets of ImageNet32. One primary finding is that the test accuracy of convolutional networks approaches that of fully-connected networks as depth increases (Fig. 2). These experiments include sweeping model complexity past the \"interpolation threshold\" (where 100% train accuracy is achievable), by analogy with experiments that have observed double descent behavior when model complexity originates from width. \n\nThe authors further study linear neural networks (with convolutional or Toeplitz constraints against fully-connected layers) where they can analyze properties of the learned solutions. In one experiment, they consider a toy problem (Sect. 4.1) of classifying the color of a single pixel in an image. This problem is chosen because the minimum Frobenius norm solution (which is learned by the fully-connected network) does not generalize, and the authors show that the norm of the solution learned by the linear convolutional network approaches this value with depth. A similar result is shown for linear autoencoders (Fig. 7). \n\nA primary implication of the authors' results is that increasing depth past the interpolation threshold may be detrimental to performance.\n\nQuality and Clarity: The quality of the work (experiments and framing) seems good, as far as I can gather, and the paper is clearly written and straightforward to read.\n\nOriginality: The primary thrust of the paper (studying the trend in loss/accuracy when model complexity originates from depth) is in close analogy to prior works that have investigated network width and not especially novel. However, I did find the comparison between learned solutions of linear convolutional vs fully-connected networks to be interesting and different from prior works I'm aware of. The class-dependence of the critical depth was also an interesting finding.\n\nSignificance: I think the work brings an interesting basic question to light -- to what extent is behavior like double descent (or the usual bias-variance tradeoff) specific to the way in which model complexity is increased in neural networks, and it provides empirical support that the trends are very different when overparameterization arises from depth.\n\nOther comments: The observation that the performance of deep convolutional networks approaches that of a fully-connected networks has appeared in one prior work that I'm aware of: https://arxiv.org/abs/1806.05393 published in ICML 2018. Fig. 3 in that work shows the test performance of ultra-deep CNNs (with an initialization scheme chosen specifically to help train ultra-deep networks) on CIFAR-10 collapses to that of fully-connected networks; Sect. 2.1.6. discusses how the behavior of signal propagation (the model prior) in random convolutional networks as they become deep approaches that of fully-connected architectures. \nSince I was familiar with this work, I am not especially surprised by the findings in this paper and hence the reason for my somewhat lower score and suggestion for incorporating a bit more (richer) content, either empirically or theoretically. I do think the results are interesting to publish in some form but am not sure the paper as it stands merits a conference rather than e.g. a workshop publication. I am open to raising my score however.\n\nI had several questions about the empirical results:\n--Some of the trends seem to fluctuate a bit rather than behave smoothly after an inflection point (e.g. Fig. 1(b)). Is there a reason for this?\n--Could the authors elaborate on why a lower Frobenius norm (in the linear networks) is a bad solution? In general I might have naively thought that low norm solutions were good for generalization.\n--In Fig. 2(e) the test accuracy of the fully-convolutional network actually dips below that of the fully-connected model. This seems a bit unexpected -- is there an understanding of why this occurs?\n\nFinally, Fig. 14 in the supplementary seems to be mislabeled (y-axis reads \"Test MSE\", caption refers to train loss).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers appreciate the numerical results presented in this paper. However, the paper needs a more rigorous theoretical investigation of the empirical phenomenon, or a more comprehensive empirical exploration to pinpoint the key factors. I recommend the authors to incorporate the suggestions from the reviewers and submit the paper to the next top conference.
Decision: Reject