Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of paper: The authors empirically study trends in neural network performance for models with fixed width but increasing depth. (Previous work has investigated how increasing "model complexity" from neural network width affects test loss/accuracy, with "double descent" behavior -- the trends observed here are very different from double descent.) 

The authors consider ResNets and fully-convolutional networks (which contain only convolutional layers with a final pooling operation) on CIFAR-10 and subsets of ImageNet32. One primary finding is that the test accuracy of convolutional networks approaches that of fully-connected networks as depth increases (Fig. 2). These experiments include sweeping model complexity past the "interpolation threshold" (where 100% train accuracy is achievable), by analogy with experiments that have observed double descent behavior when model complexity originates from width. 

The authors further study linear neural networks (with convolutional or Toeplitz constraints against fully-connected layers) where they can analyze properties of the learned solutions. In one experiment, they consider a toy problem (Sect. 4.1) of classifying the color of a single pixel in an image. This problem is chosen because the minimum Frobenius norm solution (which is learned by the fully-connected network) does not generalize, and the authors show that the norm of the solution learned by the linear convolutional network approaches this value with depth. A similar result is shown for linear autoencoders (Fig. 7). 

A primary implication of the authors' results is that increasing depth past the interpolation threshold may be detrimental to performance.

Quality and Clarity: The quality of the work (experiments and framing) seems good, as far as I can gather, and the paper is clearly written and straightforward to read.

Originality: The primary thrust of the paper (studying the trend in loss/accuracy when model complexity originates from depth) is in close analogy to prior works that have investigated network width and not especially novel. However, I did find the comparison between learned solutions of linear convolutional vs fully-connected networks to be interesting and different from prior works I'm aware of. The class-dependence of the critical depth was also an interesting finding.

Significance: I think the work brings an interesting basic question to light -- to what extent is behavior like double descent (or the usual bias-variance tradeoff) specific to the way in which model complexity is increased in neural networks, and it provides empirical support that the trends are very different when overparameterization arises from depth.

Other comments: The observation that the performance of deep convolutional networks approaches that of a fully-connected networks has appeared in one prior work that I'm aware of: https://arxiv.org/abs/1806.05393 published in ICML 2018. Fig. 3 in that work shows the test performance of ultra-deep CNNs (with an initialization scheme chosen specifically to help train ultra-deep networks) on CIFAR-10 collapses to that of fully-connected networks; Sect. 2.1.6. discusses how the behavior of signal propagation (the model prior) in random convolutional networks as they become deep approaches that of fully-connected architectures. 
Since I was familiar with this work, I am not especially surprised by the findings in this paper and hence the reason for my somewhat lower score and suggestion for incorporating a bit more (richer) content, either empirically or theoretically. I do think the results are interesting to publish in some form but am not sure the paper as it stands merits a conference rather than e.g. a workshop publication. I am open to raising my score however.

I had several questions about the empirical results:
--Some of the trends seem to fluctuate a bit rather than behave smoothly after an inflection point (e.g. Fig. 1(b)). Is there a reason for this?
--Could the authors elaborate on why a lower Frobenius norm (in the linear networks) is a bad solution? In general I might have naively thought that low norm solutions were good for generalization.
--In Fig. 2(e) the test accuracy of the fully-convolutional network actually dips below that of the fully-connected model. This seems a bit unexpected -- is there an understanding of why this occurs?

Finally, Fig. 14 in the supplementary seems to be mislabeled (y-axis reads "Test MSE", caption refers to train loss).
review_title: Empirical work that studies an interesting question -- could incorporate a bit more content

final_score: rating: 6: Marginally above acceptance threshold
review_content: *********
Summary Of The Manuscript: 
*********
This manuscript focuses on the problem of the impact of depth in Convolutional Neural Networks (CNNs) for better generalization. To investigate the issue, the authors did an empirical analysis of the problem statement through various strategies (i.e. Deep CNNs - increasing depth vs. Fully Convolutional Networks, etc.) and provided in-depth findings via experiments. Together with experiments and evaluation on standard benchmarks such as CIFAR-10 and ImageNet32, the authors lead to the conclusion of their study that the testing performance will be poor if the increase in depth is higher. 

*********
Strength Of The Manuscript: 
*********
Clarity:
++ The paper reads very well and provides a very good description of related work and background, motivating the problem. Even outside of the contribution of this paper, I would recommend this paper to people getting started with CNNs as it provides a thorough description of the part of the pipelines it deals with. Also, all the empirical analyses have been described thoroughly and the various settings for the training and testing are performed in such a way to give better insights to novice readers.

Novelty:
++ In terms of novelty, the stand-alone contribution of the manuscript is that through various strategies authors tried to give in-depth insights on the behavior of different linear and non-linear as well as deep learning models via increasing the depth for classification task on different benchmark datasets. 

Experiments:
++ There have been experiments performed across datasets with variety in terms of different models and their architectures. Additionally, the analyses provided in the manuscript is fairly consistent. Supplementary material also backs the analysis by providing the visualizations of training and testing errors. Besides, analysis/ablation studies are done on the models, by changing the widths, the effect of downsampling, and changing the kernel width, and after through all the settings finally check the performance difference.

*********
Weakness Of The Manuscript: 
*********
Overall, apart from the contribution of the paper I have some concerns regarding the paper which are listed below.

-- I believe that the authors did a tremendous job to reach the conclusion that the increase in depth is crucial for certain tasks and might lead to poor results in different situations if the depth is increased beyond the threshold. However, to back up this conclusion more effectively I believe that if the authors have performed more analysis by introducing a data-augmentation strategy,student-teacher training strategy, introducing learning rate variance and so more training strategies this manuscript will be a really a good point of start to a beginner. I encourage the authors to refer to this manuscript by Urban et al. [1].  
-- The fact that has been mentioned corresponding to increase in depth leads to worsening the result have already been a point of view for much of the deep learning practitioners as He et al. [2] have already provided a really good analysis in the manuscript of ResNet, in contrast, the majority of the work in the current manuscript is already have been either published or have been known to the community. 

*********
Justification Of The Manuscript Review: 
*********
-- In the reviewer's opinion, in its current form, the paper provides in-depth analysis for increasing /decreasing the depth of CNNs however there are certain points mentioned in the weakness section have been mentioned which needs some clarification from the authors during the rebuttal phase, if answered thoroughly, the reviewer believes that the manuscript will be a really good point of start for novice deep learning readers/practitioners. Therefore the current rating of the paper will be 6 in reviewers' opinion as it is above the acceptance threshold marginally. 

References: 
[1] Urban, Gregor, et al. "Do deep convolutional nets really need to be deep and convolutional?." In ICLR (2017).
[2] He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
review_title: Review of the manuscript Do Deeper Convolutional Networks Perform Better?

final_score: rating: 5: Marginally below acceptance threshold
review_content: *Summary:

This paper mainly answers a fundamental question: what is the role of depth in convolutional networks? Specifically, the authors present an empirical analysis of the impact of the depth on the generalization in CNNs. Experiments on CIFAR10 and ImageNet32 demonstrate that the test performance beyond a critical depth. My detailed comments are as follows.

*Positive points:

1. This paper is significant to understand deep neural networks and helps to develop new deep learning algorithm.

2. This paper provides many empirical studies to analyze the effect of increasing depth on test performance.

*Negative points:

1. The importance and novelty of the research should be emphasized. Recently, there are some works [1][2][3] study the role of depth in DNN. What is the difference from these works? 

[1] Do Deep Convolutional Nets Really Need to Be Deep and Convolutional? ICLR 2017
[2] Understanding intermediate layers using linear classifier probes. arXiv, 2016.
[3] Towards Interpreting Deep Neural Networks via Understanding Layer Behaviors. 2020

2. This paper analyzes the linear neural networks and demonstrates that increasing depth leads to poor generalization. However, existing works apply non-linear neural networks in real-world case. It would be better to provide analysis on non-linear neural networks.

3. The authors suggest that practitioners should decrease depth in these settings to obtain better test performance. However, ResNet-101 has better test performance than ResNet-18 in practice. Could you please give more explanations?

review_title: Comments

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Post-rebuttal update:

I am raising my score from 4 to 5, to recognize extensive updates to the paper experiments and numerous clarifications during the discussion, as well as to acknowledge that some of my initial concerns were not justified, e.g. asking for similar studies for MLPs (which was effectively done in the original submission), or questioning whether the Frobenius norm in Figure 7 really goes down (which was demonstrated better in the latest update).

However, I cannot give it a higher score / advocate for acceptance because I still find section 4 (and follow-up discussion) to be more misleading/confusing than helpful when it comes to explaining the observed phenomena.

Precisely, even after the discussion, I still believe that

1) Experiments in Figure 7 should be done with circular-padded convolutions (to rule out the more trivial explanation of the decreasing norm), and on small subsets of CIFAR10/Imagenet32 (to establish whether low-norm is indeed associated with poor performance on image classification).

2) A precise definition of norm for nonlinear networks should be provided, and a link with the Frobenius norm of the linear networks should be established. I don't understand what exactly "the norm in the corresponding RKHS" means, and how the reader was supposed to infer it from the text or our earlier discussion. Further, if possible, this norm should also be evaluated on the actual nonlinear networks, and compared to other notions of norms discussed in prior literature (e.g. https://arxiv.org/pdf/1706.08947.pdf), where lower norm is typically associated with better test accuracy.

3) A proper discussion about the non-monotonic dependence on depth in the context of provided intuition should be given. Current intuition can be interpreted as either predicting monotonic decrease in accuracy with depth (learning solutions of lower and lower norms), or as predicting a sharp drop in accuracy at a certain depth (point where the minimum-norm solution can be learned), but neither interpretation explains the hill-shaped dependence, which again makes me question whether this is indeed the right explanation.


Without section 4, the paper still has novel empirical results, but in my opinion they are neither surprising (e.g. a hill-shaped dependence of accuracy is my default expectation of any NN hyper-parameter) nor actionable (there are no hints regarding what the peak depends on / how to guess it) enough for publication at this time. 

A more rigorous investigation into explaining the phenomenon, or a more comprehensive empirical exploration to identify what does and what doesn't influence the best depth, would make this a great paper at a later conference.

Best,
R4.

### Original review:

#### Paper Synopsis:

The paper empirically studies the effect of depth on generalization in CNNs, and finds that, unlike in the case of width, generalization can decrease beyond a certain critical depth. The paper proposes an intuition for why this happens, by observing that in linear CNNs, the Frobenius norm of the trained network decreases with depth, and thus potentially converges to the minimum-norm solution, which corresponds to simple linear regression. By analogy and with some empirical evidence, the paper conjectures that CNNs converge to MLPs as they become deeper.



#### Pros:

Apart from certain aspects that I discuss below, the paper is clear and easy to read. I especially appreciate considering a variety of experimental settings (different datasets, number of classes, vanilla-CNNs and ResNets), as well as toy examples. Code is provided, which is another strong point of the paper. The question asked in the title of the paper is interesting and worth investigating.



#### Cons:

My major concerns are:
1. The main takeaway message appears either trivial, or at least miscommunicated.
2. Experimental evidence is unconvincing of [the stronger, non-trivial interpretation of] the key message of the paper and the proposed intuition.
3. The proposed intuition is not convincing theoretically.

I am open to be persuaded otherwise in case if I misunderstood certain claims. Below are my specific thoughts on each of the points:
1. What concretely is the claim of the paper? Is it 
	1. _“For each task, architecture, and training algorithm, there is a depth optimal for generalization?”_ If so, this is a truism. 
	2. _“..., in addition, beyond this optimal depth, generalization monotonically decreases?”_ This is not supported by Figures 12.(f, g), where it remains roughly flat or even goes up, while train accuracy and loss seem to have reached the optimum per Figures 12.(b, c), and 11.(a). 
	3. _“among CNNs with 100% training accuracy, the best depth for generalization is the smallest one”_ (per the main contribution #3, page 2)? This is not supported by Figure 1.(b), 12.(e, f), where the best network is not the most shallow.
	4. _“The critical depth threshold is independent of certain parameters of the task/architecture/training algorithm”_? Sadly, the paper does not answer this question, but admits this would be a good direction for future work (I agree).
	
  As such, my takeaway from the paper is only that _“Double descent usually doesn’t happen through depth”_ in CNNs. Unfortunately, I find this observation quite trivial, and not particularly novel, see for instance [1, Figure 3].
    
2. Below, I interpret the claim of the paper as _“double descent doesn’t happen in CNNs; further, CNN performance converges to that of MLPs as they become deeper”_. Firstly, Figures 2.(b, c, e, f), and 3.(c, f) are all in the classical regime (< 100% training accuracy), and therefore somewhat unrelated to the claim of the paper, which concerns generalization trends of 100% accurate networks (to be clear, including the whole range of depths from 1 is highly appreciated, but to corroborate the claims of the paper multiple depths beyond 100% training accuracy are needed). Further, no plot shows a convincing _convergence_ of CNN performance to that of an MLP, rather, I see that in some cases the curves _intersect_ (in Figure 2, but not in Figure 3). Finally, Figures 12.(f, g) showcase settings where the double descent arguably does occur (i.e. flat or increasing test performance), which makes the claim that it does not happen through depth less robust and more hyper-parameter dependent.

3. Regarding the intuition about CNN to MLP convergence. My key concern is that most image classification datasets, including both considered CIFAR-10 and ImageNet32 cannot be perfectly fit with a linear function. The linear regression solution will not interpolate the training data. Therefore, whatever mechanism is behind the degradation of performance of deep nonlinear CNNs, it cannot possibly be convergence to a linear solution, because they fit the training data perfectly, and the linear solution does not. As such, I believe the intuition that may work in the deep linear case, is _guaranteed_ to not apply to nonlinear networks. If my interpretation of the intuition is wrong, I kindly ask the authors to clarify their reasoning, explicitating precise logical steps and assumptions made.


Other, more specific suggestions that may improve the quality of the paper and make the results more convincing:
1. I believe the paper would be much stronger if it performed (or referenced, if applicable) equivalent studies, including toy examples, for deep MLPs, and compared/contrasted findings with CNNs. This would be especially useful to help validate claims about CNN to MLP convergence. For example, is the decreasing norm with depth specific only to deep linear networks with constraints, or to deep linear networks in general?
2. I find the experiment in Figure 6 very interesting, but I think it can be strengthened in a few ways. Firstly, could you please provide measurements for depths < 5, as in other figures? Secondly, I assume “SAME”, zero padding was used (if not, please let me know and ignore the following point). In this case, the norm of the deep linear CNN would decrease with depth even without training, at initialization, due to convolving over more and more zeros at the edges as one goes deeper into the network. Therefore, it would be important to control for this effect (for example by using circular-padded convolutions). Finally, I believe this same toy experiment could have been conducted on a small subset of CIFAR-10 (that can be fit linearly), and the results would be more convincing, since I am otherwise not confident the observed effect is not due to some quirk of this specific toy task (e.g. would the same trend be observed if the test samples were everywhere in the image, and not at the bottom quadrant? What if train samples were in the center? What if the training set had multiple squares? etc).


#### To summarize: 
1. am convinced by the observations that double descent _does not necessarily happen_ through depth in CNNs. 
2. I am not convinced that it never happens, and I am not convinced one can use it in practice to select optimal depth. 
3. Further, I am not convinced with the intuition and provided evidence that deep CNNs converge to MLPs. 

Unfortunately, I find (1) on its own not sufficiently surprising / useful / novel [1, Figure 3] for publication.

[1] [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks, Xiao et al, 2018](https://arxiv.org/abs/1806.05393)
review_title: A very interesting problem, but vague experimental results and explanations

Meta-review: Reviewers appreciate the numerical results presented in this paper. However, the paper needs a more rigorous theoretical investigation of the empirical phenomenon, or a more comprehensive empirical exploration to pinpoint the key factors. I recommend the authors to incorporate the suggestions from the reviewers and submit the paper to the next top conference.
Decision: Reject