Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper demonstrates a rank diminishing behavior of deep neural networks considering the mapping from the input space to the  feature space of an increasingly deeper intermedate layer. Theoretically, it proves that the rank doesnot increase as the layer depth increases. Experimentally, it demonstrates a general decreasing trend of rank on various NN architectures. This work also empirically demonstrates that the number of major PCA components at the final feature layer is much less than its ambient dimension, which leads to feak correlation between very different categories.
strengths_and_weaknesses: Strenghs:
1. This work systematically studies the evolution of function rank throughout the layer computation and provides theoretical jusfication to the empirically observed rank diminishing behavior.
2. The finding about the independence deficit of final feature manifolds is very interesting and provides insight to the lack of robustness of DNNs.

Weakness:
1. Classification dimension estimated by the number of major PCA  components in this work is not a good indicator of the feature dimesnion. In fact, a very low dimensional manifold can have high classification dimension. Therefore, the main results about rank diminishing cannot explain the interesting finding about low classification dimension of final feature manifolds. The statement in the abstract that "independence deficit caused by the rank deficiency of deep networks" is misleading.
2. It seems that the definition of the rank of function and lemma 1 implicitly assume that the jacobian of neural network functions has a constant rank over the entire input space of R^n. This is a strong assumption that doesnot hold in general.  When this assumption holds for neural networks should be carefully discussed.
questions: 1. [related to weakness 2] What exactly is the "rank of its Jacobi matrix Jf over its input domain X" in definition of the rank of function? How the rank of function is related to the rank of its Jacobian matrix at a specific point x\in R^n?
2. [related to weakness 1] Is there any theoretical explanation about why independence deficit happens for the final feature manifold?
limitations: The authors adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims to study the rank of hidden layer representations of neural networks in relation to how deep the layer is in the network. In particular they note that the rank of the hidden layers diminish monotonically as we observe deeper layers. Numerical measures of rank are proposed and motivated. The primary theoretical concerns are the rank of the Jacobian from the input to the i-th layer of the network (essentially a linear approximation of the network mapping to that hidden layer) and the dimension of the feature space for a hidden layer. The paper further investigates the tolerance of the final hidden layer to dimensionality reduction by applying PCA to features space and projecting onto a decreasing number of eigenvectors. The number of eigenvectors remaining when a significant drop in performance is observed from the dimensionality reduction provides an approximation for the intrinsic dimensionality of the hidden layer. Finally, the paper explores the idea that it is possible to use the logits of different categories to classify another category in a dataset. One example is that by merely using -0.923 as a weight on the logit for the "triumphal arch" category it is possible to predict the "junco" category without loss of accuracy.
strengths_and_weaknesses: # Strengths
## Originality
The paper is fairly original with the primary novelty being the rank metrics used and their justification. Additionally the paper touches on some possible connections between symmetry and rank which to my knowledge have not been explored, however, these connections are mainly pointed out but not discussed or treated theoretically.

## Quality
The need for the numerical tools to measure rank is well motivated and the numerical tools themselves makes sense and are justified. The claims that are made appear correct and in-line with the evidence presented.

## Clarity
There is some variance in the clarity of the paper for various sections. The writing is clear and understandable and the mathematical notation is consistent and intuitive which helps the clarity in the earlier sections greatly. Sections 3.2 and 3.3 are examples where the notation made potentially tricky sections more manageable. Figure 4 stands out as a very helpful figure. The effort on that is definitely worth it.

## Significance
The paper touches on some significant points, like the point linking symmetries to lower ranks. The PCA experiment and the experiment on the using categories as predictors for others may be of general interest to the ML community.

# Weaknesses
## Originality
A primary concern of this work is the fact that $Rank(AB) \leq min{Rank(A), Rank(B)}$. This is even mentioned in the paper below equation 8 and is one of the primary tools for the work. This, however, is a well established principal and quite intuitive. Thus, the finding that the rank of the network decreases with layer depth is not surprising. Two possible interesting points: noise increasing network rank and structure avoiding the rank staying the same across layers are mentioned but do not form part of the analysis. The noise aspect is ignored in the theory and removed through the noise tolerant rank measures. The point on monotone decrease over equality of rank due to structure is discussed briefly.

## Quality
The various sections of the paper feel quite loosely connected. Up to Section 4 the work considers whether the rank of the network decreases monotonically. The Section 5 considers PCA just on the final feature space and is the used to point out that low dimensional feature spaces do not hold semantically meaningful features for each category in Section 6. These sections are all related to rank, however, the connections do not seem to go deeper than that. Finally, there are some points where unjustified claims are made (or the phrasing makes these claims appear unjustified). Two examples are "Theorem 5 that investigates the behaviour of all singular values of deep neural networks" when theorem 5 requires hidden layers of the same size and assumes the Jacobians have Gaussian elements (which appears to be unrealistic in its own right) and "The principle of rank diminishing describes the behavior of general neural networks with almost everywhere smooth components" where it is not clear that ReLU networks would even fit this requirement.

## Clarity
Theorem 4 and Theorem 5, which are the most technical aspects of this paper are not given enough space. The clarity of the paper could benefit greatly from a more in-depth treatment of this section. In addition, how the theory of these sections relate to Figure 1 could also be explained more. For example I acknowledge that the shape of the bottom row of Figure 1 is non-linear but to call it exponential (which Theorem 4 and 5 predict) might also be a stretch. Understanding Theorem 4 and 5 would help with interpreting Figure 1. Figure 1 could also use different colours, especially for the bottom row where distinguishing between Jacobian and Feature rank/dimension is not easy. Finally, the notation of Section 5 is not easy to follow, particularly in the meaning of the $i_j$ double subscript where it is not immediately clear what $i$ and $j$ each refer to. Figure 4 does help clarify this a lot and with space constraints fully explaining the new notation may not be feasible.

## Significance
This work appears generally significant, however, its significance is hindered by the same issues noted under the originality section. I feel that this work might spend too much time on the potentially quite obvious points of rank diminishing and on introducing the PartialRank and not enough time on the potentially significant points such as Theorem 4 and 5. My primary recommendation would be to rephrase the work more in line with those theorems.
questions: No questions as yet.
limitations: I suggest that the authors be clearer on the conditions required for their theory to help. For example saying "The principle of rank diminishing describes the behavior of general neural networks with almost everywhere smooth components," which does not seem to include ReLU networks but is described as general is unclear.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the dynamics of the rank evolution of the feature maps of a neural network as a function of its depth. By leveraging the abstract definition of rank of a function as the rank of the corresponding Jacobian matrix, the authors can study the rank dynamics in full generality (i.e. without assuming any specific architecture). This results in Theorem 1 (Principle of Rank Diminishing), that finds that the rank of neural network should never increase with depth due to its compositional nature (a neural network can be see as a composition of $L$ functions, where $L$ is the depth). Then, the authors analyze conditions under which the rank strictly diminishes (Theorem 3) and convergence of the rank to specific constants (Theorem 4-5). 

Finally, the authors apply their low rank findings to the study of the dependence and correlations between different output classes. They find that the output of some classes of ImageNet (e.g. hamster) can be predicted with a linear combination of the output for irrelevant classes (e.g. broccoli and mouse trap). The authors attribute this problem to the low rank representations of very deep network, as showed by their developed theory.
strengths_and_weaknesses: **Strengths**
1. **Generality and Importance of the Results**: the theoretical results are very general and remarkable, abstracting away from the specific architecture. The only assumption is the compositional nature of the layers, which includes most of architectures but excludes residual networks (as the author mention in the supplementary material). 

2. **Paper Organization**: the paper is very clear in explaining the abstract concepts of the first part. Until Theorem 2 (page 4), the theory is easy to digest. At first read, Theorem 1 seems trivial if one thinks about linear networks (i.e. simple product of matrices) and the famous property $\text{rank}(AB) \leq \min(\text{rank}(A), \text{rank}(B))$, but the author do a great job to generalize it to any composition of functions through ideas from topology theory. The other two theorems delve deep into the rank diminishing properties of function compositions, showing an exponential decay of the rank with depth. 

3. **Independence Deficit of Feature Manifolds**: Section 5 provides a nice application of the theory, and would probably cause follow up works in trying to understand how one can reduce this undesirable effect of strong dependences between semantically different classes. 

**Weaknesses**
1. **Inconsistency of Residual Network**: Skip connections are proposed as a tool to (partially) prevent the rank deficiency problem, and they give a brief theoretical argument in the supplementary material. However, this seems to be in contradiction with Figure 1, where an exponential decay of the rank is observed for ResNets, MLP-Mixers and Transformers, all architectures that adopt skip connections. This could be due to the fact that during training the magnitude of $\text{Res}(x^i)$ becomes large, hence lowering the rank. At initialization, the magnitude of $\text{Res}(x^i)$ can be controlled, e.g. with an appropriate factor inversely proportional to the depth (see for instance [1] for this scaling and [2] for its consequences on the rank). In any case, I found it confusing that skip connections are adopted in almost all the architectures used to exemplify the theory (skip connections that according to the authors should have an opposite effect). 

2. (minor) **Presentation Style of Structural and Implicit Impetus**: After brilliantly explaining the principle of rank diminishing, in my view  the concepts of "Structural Impetus" (due to the specific architectural modules) and "Implicit Impetus" (due to the very compositions of infinite modules) of rank diminishing could be better explained. In particular, I would invest some extra lines to better explain why normalization layer prevent rank diminishing, and maybe better introduce some concepts ( or instance "moving along directions" of Theorem 3 is not properly introduced and in general the current version of the Theorem fails to convey a simple and intuitive explanation).

[1] Hanin, Boris, and David Rolnick. "How to start training: The effect of initialization and architecture." Advances in Neural Information Processing Systems 31 (2018).

[2] Noci, Lorenzo, et al. "Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse." arXiv preprint arXiv:2206.03126 (2022). 

questions: 1. I would like to ask the authors if in their view there is a way to quantify what they call the "Structural Impetus" for different architectures and normalization layers.

2. Can (and if so, how) their results can be used to design better architectures, initializations, training procedures that better preserve the rank information?
limitations: I do not see a negative societal impact of this theoretical work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work presents some theoretical results that imply that the rank of the Jacobian between the inputs and features of deep networks is non-increasing with depth. They predict that in some settings it should in fact decrease exponentially with depth to some fixed value. 

They also develop efficient methods to estimate the Jacobian rank of real networks and show empirically that it indeed decreases with depth across a number of different architectures. 
strengths_and_weaknesses: The effects of depth on the learned representations in deep networks and their geometric structure is an important area of study. While this work contains an interesting combination of theoretical and empirical results, I believe the connection between the two would have to be made more concrete. 

The result about non-decreasing rank follows from the basic compositional structure of the network as the authors suggest, yet it is unclear that the rank must decrease. In fact, there is a vast literature on signal propagation in deep networks that approaches this question from a different angle (by studying covariance between hidden features as a function of depth, in which case convergence to certain fixed points should essentially be equivalent to the rank of the representation collapsing [1, 2]). This literature also highlights ways to avoid this phenomenon with a careful choice of initialization, and relies on modeling the dynamics of the correlations as a function of initialization hyperparameters. This allows one for example to train convnets of depth 10000 [2]. In the simplest case of a network with orthogonal weights and no non-linearities, it is clear for example that there is no decrease in rank, so there are clearly ways that it can be avoided. 

Another related issue is that the results are vague in the sense that the behavior of the rank is not connected in a quantitative way with the structure of the network (i.e. the choice of nonlinearity, initialization, etc). I think the submission would be much more compelling if the results could take these into account and make predictions about their effects on the rank. For example, how is the rank one converges to or the speed of the rank decay related to properties of the network? 

An additional, related concern is the connection between the experiments and the theory. The experiments that attempt to show exponential decay of the rank are not plotted on a logarithmic scale, which makes it hard to understand whether the decay there is indeed exponential or follows some other law. In addition, it appears that the rank decay in the case of resnets may be influenced more by the pooling layers or changes in width than any other operation, yet no mention of this is made in the text. 


[1] Poole, Ben, et al. "Exponential expressivity in deep neural networks through transient chaos." Advances in neural information processing systems 29 (2016).

[2] Xiao, Lechao, et al. "Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks." International Conference on Machine Learning. PMLR, 2018.

questions: Following my remarks in the previous section, how difficult do the authors think it would be to relate some of the predictions in a quantitative way to various aspects of the network architecture? 
limitations: Limitations have been addressed
review_title: null

Meta-review: metareview: This paper studied the "rank" of neural networks and showed that deeper network in general will have lower rank. The paper did a detailed empirical study on network rank, as well as some theoretical insights on why rank is likely to decrease as the network becomes deeper, and how the rank decrease can change with or without normalization layers. The paper also demonstrated a "independence deficit" phenomenon which happens when the rank of the output layer is too low. Overall the reviewers feel that the paper gives interesting observations and nice intuitive explanations. 
confidence: Less certain
Decision: Accept