Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work studies the representational power of Residual Networks and try to explain theoretically a desirable property of using ResNets, namely: does a residual network guarantee that stacking more residual blocks does not lead to a degradation in performance of the network. The main approach of the authors is to view the output of a ResNet as an ensemble of several shallow networks as opposed to a non-ResNet which is simply a very deep composition of non-linear functions. Given this view point, the authors identify some simplifying assumptions which make the associated Empirical Risk Minimization problem tractable: 1) introducing a sequence of "de-coupling" weights (H) which allow the weights (W) in the ensemble to be independent of each other 2) removing the non-linearity at the end of the residual block and before the final affine transformation and 3) a few assumptions on the loss function (differentiability and convexity). With these assumptions, the authors prove a few desirable properties of ResNet architectures. Firstly, the authors show that stacking more residual blocks cannot hurt in the representational sense, and can improve if the new feature is linearly independent of the other features in the ensemble basis. Secondly, the authors show that "wide ResNets" always attain the empirical risk minimization lower bound, which has an implication on the quality of local optima attained by the ResNets. The authors also prove analogous results for a related class of architectures (DenseNets), in particular showing that DenseNets may be approximated by wide ResNests.
main_review: This is a very interesting paper which studies an important problem from a theoretical angle, namely the representational power of Residual Networks, and what characterizes their empirical good performance compared to simply stacking dense functions. The work introduces some simplifications to make the problem more tractable, and show theoretically that in the setting when the final non-linearity is removed from a chain of residual blocks, we get a theoretical guarantee that stacking more residual blocks is guaranteed to not hurt performance in the optimization setting (modulo some assumptions). This seems like an important and very interesting observation, and I wonder if some of assumptions which the authors made to get to the theoretical result could lead us to design better real world architectures - for instance, would a "decoupled" ResNet lead to improvements over a regular ResNets empirically?

The paper is overall well written and easy to follow, although the main proofs of all the claims are in the Appendix which doesn't seem included. However, with the assumptions introduced by the authors, the conclusions of the theorems sound reasonable. In terms of future directions/improvements, I would encourage the authors to investigate this phenomenon empirically as well, to see if some of the assumptions hold in real world datasets, which would make the work more compelling.

Typos/Clarifications:

Line 234: It seems N_0 has not been defined before, is it some large constant we are assured the existence of?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors proposed a couple modification of ResNet in order to improve the loss landscape. The main idea behind the modifications, if I understand correctly,  is to guarantee a direct skip connection (dense matrix or product of dense matrix) from the hidden features to the logits. The authors provide a  couple theoretical claims regarding the proposed models. No experimental results are provided. 
main_review: Overall, I found the presentation of the paper unsatisfying and difficult to follow. From the abstract and introduction, I find myself difficult to extract the main theme and main contribution of the paper. Only after going thorough the papers twice, I finally realized the paper is about landscape analysis of ResNets and its variants. Regardless of the results of the paper, presentation requires substantial improvement and empirical supports are needed to support major claims. Moreover, methods/modifications that improve loss landscape of the SOTA models (e.g. ResNet) more often than not may not be useful at all, unless they could improve the original performance, or on par with it and improves the optimization, or simplifies the models and etc. As such, I do not support acceptance.   

In what follows, I try my best to summarize the main contribution and ideas of the paper. 
The main complaint about ResNet of the paper is the *non-linearity* layer before the linear classifier and the authors propose to remove it. The main benefit of doing so is the logits $y$ could be written as a linear combination of hidden features of the networks, namely, 
$$
y(x) = \sum_{i=1}^L A_i v_i(x)
$$
where $v_i(x)$ is the hidden feature and $A_i$ is an affine transform which is product of two matrices. Under some rank assumptions, the authors claim that loss function associated to this modified model is non-convex but enjoys a good property: every critical point that is not a local minimum is a saddle point (proposition 1). Obvious, the fact $A_i$ being a product of two dense matrices creates many saddle points and the author propose add a skip-connection from $v_i$ directly to the logits layer and $A_i$ could be replaced by a dense matrix $H_i$, i.e. 
$$
y(x) = \sum_{i=1}^L H_i v_i(x)
$$
The authors proceed to conclude that the new optimization problem is convex, which seems obvious since the new model is a linear function function the hidden features. They then apply similar tricks to DenseNet. 

Overall, these landscape results seem not very interesting nor significancy, at least to me. 
limitations_and_societal_impact: Theoretical understanding of ResNet is important and indeed very important! Principle-guided approach to design architecture is also equally important. However, I don't think the paper made significant advances in these two problems, in particular, without strong empirical support. I would encourage the authors to provide experimental results to demonstrate the superior of the proposed architectures.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies a variant of ResNet/DenseNet, which removes the final nonlinearity. This makes the output a linear combination of a set of basis functions, which can be viewed as features extracted by the various blocks of the architecture. Based on this formulation, results regarding the optimization minima are examined. An augmented version of the architecture in question is also analyzed.
main_review: The paper analyzes ResNEsts, which are essentially ResNets without a final nonlinearity. This characterization leads to some interesting results:
- Assuming all layers except the last one are fixed, the output can be viewed due to this formulation as a linear combination of basis functions $v_i(x)$, each outputted by the previous blocks of the network.
- This characterization also serves as an inspiration for a slightly different architecture, Augmented ResNEsts. In those, the output of each residual block is kept and used at the end of the model, multiplied by another weight matrix $H_i$.
The paper also defines DenseNEsts, as an extension of the previous architecture similar to DenseNets, and also proves that they can be seen as a sufficiently wide ResNEst.

This work serves as a theoretical analysis of the above variation of ResNets, and the theorems/propositions included are technically sound. Proofs for some training properties (namely, directions of negative curvature in saddle points, and that wide ResNEsts attain their proposed ERLBs) are also included. Nevertheless, there are some issues which limit the impact of the paper:
- Regarding ResNEsts, all theorems are done under the assumption that all weight matrices except the two last ones are fixed. Note though that this is done to keep the functions $v_i(x)$ fixed, and if I understand correctly the intent of the authors was to show via Theorem 1 that stacking extra blocks in the wide ResNEst will not decrease performance (since the A-ResNEst essentially includes ResNEsts of all previous block sizes). In this case, it may be less of a limit of novelty and more of a clarity issue.
- Similarly, regarding A-ResNEsts, due to keeping the weight matrices of the intermediate layers (and thus, the $v_i(x)$) fixed, the analysis presented in this paper is the same as the analysis that would be done for a linear predictor, which has as input all the features $v_i(x)$ and a block diagonal weight matrix, with each part being equal to $H_i$. This is a very specific case of the proposed architecture.
- Corollary 1 appears to be self-evident (as a class of models, ResNEsts already include linear predictors).

This submission is clear in general and mostly easy to follow, although I do have some questions for the authors:
- If I am not mistaken, Remark 2 seems to imply that the intended use of ResNEsts is that training is done layer-by-layer, and the result is that doing so always improves the performance of the output. If this is indeed the case, then I suggest including it more prominently in the paper.
- In lines 252-253, a condition is stated in order for Assumption 4 to be satisfied. Does this mean that Assumptions 3 and 4 can be combined?
The clarity of the text could also be improved by explicitly defining optimization problems (so that they can easily be referred to later), instead of keeping them inside the text. 

In general, the paper has several different contributions, but most of them appear limited in scope and application. I think it would be useful for the authors to highlight how the basis functions $v_i(x)$ are obtained, or if they are assumed to be given for the application of the ResNEst.

**Post-rebuttal comment**:

After the author response, I have updated my score, based on my comments below.
limitations_and_societal_impact: The authors have adequately described the limitations present in each theorem/proposition made. Depending on how the $v_i(x)$ are obtained, I believe it is worthwhile to discuss this as well.

Regarding negative social impacts, this is a purely theoretical work, and the models presented do not have any issues of such kind to be discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper defines slightly modified variants of commonly used ResNet and DenseNet architectures, and shows that these architectures have attractive theoretical properties - increasing depth does not decrease performance (which in particular means that they should be guaranteed to outperform linear models in terms of training loss).
main_review: The problem addressed (generalization properties of realistic modern neural network architectures) is important and relevant to the machine learning community, and the solution is novel and represents a substantial improvement over existing work. The paper is fairly well-written and clear.

Although this is a theory paper, it would benefit from experiments to compare standard ResNets with ResNEsts and especially A-ResNEsts (and similarly DenseNets and DenseNEsts), since these are new model architectures being proposed. Although the theoretical justification for these new architectures appears to be sound, showing that they also work well in practice would substantially improve the paper, as that combined with the theoretical guarantees would provide strong support for practitioners to actually use these architectures. Although ResNEsts are a trivial modification of standard ResNets, this is not as true for the other new architectures proposed (and even ablating the effect of the minor difference between ResNet and ResNEst would be very useful).
limitations_and_societal_impact: There is very little discussion of these. However, I do not think any negative societal impact would apply as this paper simply advances theoretical understanding.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present several theoretical properties of a ResNet variant that is similar to a standard ResNet only without one of the non-linearities. Variants of DenseNets are also suggested with improved guarantees. Such variants are much closer to the real ResNet that is used in computer vision applications than the models that were previously analyzed. 
main_review: The paper is interesting and introduces some important theoretical concepts and proofs. However, it is tough to follow and understand. Even starting at Section 2.1, I could not quite understand for sure what the ResNEst model is. This has to be improved somehow. 

Can the authors show some results regarding how ResNEst performs in simple classification tasks compared to a standard ResNet? Usually, simpler variants with guarantees perform worse than the original, and it would be interesting to see just how much ResNEst loses in classification accuracy to get these guarantees. 
limitations_and_societal_impact: The authors did not address those, but I do not see any issues or have any suggestions for improvement.
review_title: null

Meta-review: This work studies the representational power of ResNets and aims to answer the question of when adding more residual blocks in a ResNet can be guaranteed to not lead to a degradation in performance. All 5 reviewers find the paper interesting and think its contribution is a valuable addition to the literature. All reviewers were active in discussion with the authors and with the committee. No serious concerns surfaced that would invalidate the contribution made by the paper. After committee discussion the reviewers reached a consensus recommendation to accept the paper.

Multiple reviewers indicated that the author response was especially useful and served to clear up some minor concerns that existed after the first round of reviews. Authors, please integrate this discussion into the camera ready version of the paper.
Decision: Accept (Poster)