Rank Diminishing in Deep Neural Networks
Authors: Ruili Feng
Kecheng Zheng
Yukun Huang
Deli Zhao
Michael Jordan
Zheng-Jun Zha
Venue: NeurIPS 2022 Accept (2022)
Keywords: Interpretability
Network Rank
Feature Representation
Low Rank
Learning Theory

Abstract: The rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. In particular, the assumption of low-rank feature representations led to algorithmic developments in many architectures. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear. To fill this gap, we perform a rigorous study on the behavior of network rank, focusing particularly on the notion of rank deficiency. We theoretically establish a universal monotone decreasing property of network ranks from the basic rules of differential and algebraic composition, and uncover rank deficiency of network blocks and deep function coupling. By virtue of our numerical tools, we provide the first empirical analysis of the per-layer behavior of network ranks in realistic settings, \ieno, ResNets, deep MLPs, and Transformers on ImageNet. These empirical results are in direct accord with our theory. Furthermore, we reveal a novel phenomenon of independence deficit caused by the rank deficiency of deep networks, where classification confidence of a given category can be linearly decided by the confidence of a handful of other categories. The theoretical results of this work, together with the empirical findings, may advance understanding of the inherent principles of deep neural networks. Code to detect the rank behavior of networks can be found in https://github.com/RuiLiFeng/Rank-Diminishing-in-Deep-Neural-Networks.

Rank Diminishing in Deep Neural Networks
Ruili Feng', Kecheng Zheng?.1, Yukun Huang', Deli Zhao?.3 Michael Jordan', Zheng-Jun Zha University of Science and Technology of China, Hefei, China Ant Research, 3 AAlibaba Group, Hangzhou, China
University of California, Berkeley
ruilifengustco@mail.. com, (zkeys001, keeeeesssso.k.iihheeee keviinh]@mailll astc.eduu en, zhaodeli@gmail, com, jordanecs berkeley. zhazj @ustc. edu, cn.
Abstract
The rank neural networks measures information flowing across layers. is instance of key structural condition that applies across broad domains of machine learning. In particular, the assumption of low-rank feature representations leads to algorithmic developments in many architectures. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear. To fill this gap, we perform rigorous study on the behavior of network rank, focusing particularly on the notion of rank deficiency. We theoretically establish universal monotonic decreasing property network rank from the basic rules of differential and algebraic composition, and uncover rank deficiency of network blocks and deep function couplings. By virtue of our numerical tools, we provide the first empirical analysis of the per-layer behavior of network rank in practical settings, i.e., ResNets, deep MLPs, and Transformers on ImageNet. These empirical results are in direct accord with our theory. Furthermore, reveal a novel phenomenon of independence deficit caused by the rank deficiency deep networks, where classification confidence of given category be linearly decided by the confidence of handful of other categories. The theoretical results of this work, together with the empirical findings, may advance understanding of the inherent principles of deep neural networks. Code to detect the rank behavior of networks be found in https://github..omm/ wwt.t/uu...oo/uuilliiae www.:////iiiiieeeeaaaaann--iiiin Deep-Neural-Nerreee
Introduction
In mathematics, the rank of smooth function measures the volume of independent information captured by the function [21]. Deep neural networks are highly smooth functions, thus the rank of a network has long been an essential concept in machine learning that underlies many tasks such as information compression [46, 55, 35, 53, 47], network pruning [31, 54, 6, 25, 10], data mining [7, 24, 11, 56, 19, 28), computer vision [58, 57, 30, 26, 28, 60, 59, 16, 17], and natural language processing [9, 27, 8, 12). Numerous methods are either designed to utilize the mathematical
property of network ranks, or are derived from an assumption that low-rank structures are to be preferred.
Yet a rigorous investigation the behavior of rank of general networks, combining both theoretical and empirical arguments, is still absent in current research, weakening our confidence in the being able predict performance. To the best of our knowledge, there are only few previous works discussing the rank behavior of specific network architectures, like attention blocks [15] and BatchNorms [13, 5]
"Corresponiing author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in pure MLP structures, The empirical validation of those methods are also limited to shallow networks, specific architectures, or merely the final layers of deep networks, leaving the global behavior of general deep neural networks mysterious due to prohibitive space-time complexity for measuring them. Rigorous work on network rank that combines both strong theoretical and empirical evidence would have significant implications.
In this paper, we make several contributions towards this challenging goal. We find that the two essential ingredients of deep learning, the chain rules of differential operators and matrix multiplications, are enough to establish a universal principle- that network rank decreases monotonically with the depth of networks. Two factors further enhance the speed of decreasing: a) the explicit rank deficiency of many frequently used network modules, and b) an intrinsic potential of spectrum centralization enforced by the nature of coupling of massive composite functions. To empirically validate our theory, we design numerical tools to efficiently and economically examine the rank behavior of deep neural networks. This is non-trivial task, as rank is very sensitive to noise and perturbation, and computing ranks of large networks is computationally prohibitive in time and space. Finally, we uncover an interesting phenomenon of independence deficit in multi-class classification networks. We find that many classes do not have their own unique representations in the classification network, and some highly irrelevant classes can decide the outputs of others. This independence deficit can significantly deteriorate the performance of networks in generalized data domains where each class demands unique representation. In conclusion, the results of this work, together with the numerical tools we invent, may advance understanding of intrinsic properties of deep neural networks, and provide foundations for a broad study of oow-iieesiooal structures in machine learning.
2 Preliminaries
Settings We consider the general deep neural network with L layers. It is a smooth vector-valued function F: : R - Rdd where RR. and Red are the ambient space of inputs and outputs, respectively. Deep neural networks are coupling of multiple layers, thus we write F as
F F= f" f o...of!.
(1)
For simplicity, we further write the k-th bub-network' of F as
Fx f o f';
(2)
and we use F& = (X) to denote the feature space of the k-th sub-network on the data domain X. We are more interested in the behavior of network rank in the feature spaces rather than scalar outputs (which trivially have rank 1). Thus, for classification or regression networks that output a scalar value, we will consider F = Fz as the transformation from the input space to the final feature space instead. Thus, we always have n >> and > For example, for ResNet-50 [20] architecture on ImageNet, we only consider the network slice from the inputs to the last feature layer with 2,048 units.
The rank of a function f (fi,..., Sa)"; R" + IRd refers to the highest rank of its Jacobian matrix J over its input domain X,, which is defined as Rank (f) = Ra) = max Rank (z))(xx)) ø
Rank of Function
(3)
It is well-known that the region of non-highest rank is a zero-measure set on the feature manifold by Sard's theorem [21], so we are safe to ignore them in the definition of the function ranks. The rank of function represents the volume of information captured by it in the output [21]. That is why is important to investigate the behavior of neural networks and many practical applications. Theoretically, by the rank theorem and Sard's theorem of manifolds [21], we can know that rank of the function equals the intrinsic dimension of its output feature space, as in the following lemma.
Lemma 1. Suppose that f : R - Rd is smooth almost everywhere. Let Rank = r. If data domain X is a manifold embedded in Rn and & : U + O is a smooth bijective parameterization from an open subset U C R. to O C X. then we have dim( = Rank(J <r. Thus, the rank of function f gives an upper bound for the intrinsic dimension dfm()(X)) of the output space.
PI this paper, sub-nee net-network means network slice from the input some intermediate feature layer: layer network means an independent component of the network, without skip connections from outside to it, like bottleneck layer of ResNet-50.
Due space limitation, all the related proofs attached in the Appendix.

---

It is worth mentioning that the intrinsic dimension dim(f(X)) of the feature space is usually hard to measure, so the rank of the network gives an operational estimate of it.
3 Numerical Tools
Validating the rank behavior of deep neural networks is a challenging task because it involves operations of high complexity on large-scale non-sparse matrices, which is infeasible both time and space. Computing the full Jacobian representation of sub-networks of ResNct-50, for example, consumes over 150G GPU memory and several days at a single input point. In accuracy, this is more challenging rank is very sensitive to small perturbations. The numerical accuracy of float32, 1.19c 7 [38], cannot be trivially neglected in computing matrix ranks. Thus, in this section establish some numerical tools for validating subsequent arguments, and provide rigorous theoretical support for them.
3.1 Numerical Rank: Stable Alternative to Rank
The rank of large matrices is known to be unstable: it varies significantly under even small noise perturbations [41]. Matrices perturbed by even small Gaussian noises are almost surely of full rank, regardless of the true rank of the original matrix. Thus in practice have to an alternative: count the number of singular values larger than some given threshold e as the numerical rank of the matrix. Let W Raxd be given matrix. Its numerical rank with tolerance is
Rank, (W) = # E N+ : < mn, d), 2 [[]]),
(4)
where |[ 2| is the Eg norm (spectral norm) of matrix W. o. I,............ L.., min (n, d} are its singular values, and # is the counting measurement for finite sets. We can prove that the numerical rank is stable under small perturbations. Based on Weyl inequalities [48], we have the following theorem.
Theorem 1. For any given matrix W, almost every tolerance O, and any perturbation matrix D. there exists a positive constant (c) such that 48 E [0, mc)) Rank, W is low-rank matrix without random perturbations, then there is Cmax such that for any é < Eax Rank. (W + ODD = Rank,(W] = Rank(W) for all 8 E [0, 8max(e)).
= Rank, (W).
This property of the numerical rank metric makes it suitable tool for investigating the rank behavior of neural networks. Possible small noises can be filtered out in Jacobian matrices of networks by using numerical rank. It is worth mentioning that random matrices surely under the numerical rank. Instead their rank distribution can be inferred from the well- Parcenko-Pastur distribution [33] of random matrices. So under numerical rank, low-rank matrices will be commonly seen. In this paper, we always use the numerical rank when measuring ranks.
longer have full rank almost
3.2 Partial Rank of the Jacobian: Estimating Lower Bound of Lost Rank in Deep Networks
To enable the validation of trend of the network ranks, we propose compute only the rank of sub-matrices of the Jacobian as an alternative. Those sub-matrices are also the Jacobian matrices with respect to fixed small patch of inputs. Rigorously, given a function f and its Jacobian J,, denote partial rank of the Jacobian as the rank of a sub-matrix of the Jacobian that consists of the ji-th, jz. j......--hh -th column of the original Jacobian
PartialRank(J JJ = (San((Su(JJ,.jir jkk) = (of, )axk),
(5)
where S j1 < ... < jk S n. The partial rank can efficiently computed and the variance of partial ranks of adjacent sub-networks gives a lower bound on the variance of their ranks. We demonstrate this in Appendix F.
3.3 Classification Dimension: Estimating Final Feature Dimension
Measuring the intrinsic dimension of feature manifolds is known to be intractable. So we turn to an approximation procedure. For most classification networks, a linear regression over the final feature manifold decides the final network prediction and accuracy. So we can estimate the intrinsic dimension as the minimum number of principal components in the final feature space to preserve high classification accuracy. Let cls gd 12 be the classification predictions based on the final

---

Networks
ResNet-18
ResNet-50
GluMixer-24
ResMLP-S24
Swin-T
ViT-T
ClsDim
149
131
199
196
344
109
Ambient Dim.
512
2048
384
384
768
192
Table 1: Classification dimensions (with respect to 95% classification performance of the ambient feature space IR3) and ambient dimensions of the final feature manifolds of different networks. All networks have low intrinsic dimensions for final features.
feature representation F(æ), ,, Proş be the operator that project vector to the subspace spanned by the top-k PCA components of the final feature representations, and 1 cond the indicator for condition cond. The classification dimension is then defined as
ClsDim (xX)) ming (( CIs(Pro (F(m)) (00)))==p) 1-c).
(6)
which is the minimum dimensionality needed to reconstruct the classification accuracy of the whole model.
4 Simple Principle of Rank Diminishing for General Networks
The specific designs of neural networks are vast and diverse, but most of them share two fundamental ingredients of deep networks, i.e., the chain rule and matrix multiplication. This provides us a chance to analyze the behavior of deep networks that is rccitecture-agnostie So in the theoretical aspect, we will focus on how these two intrinsic structures endow impetus of rank diminishing to neural networks.
A most straightforward observation comes from the basic rule of matrix multiplication that for any two matrices A and B. we have Rank AB) I Rann(Aan((A))) Rank(B) [22]. Taking this into the chain rule of differential of JF J J f p1-1... ft, we then have Rank (Jr, FF. Rank(J [ o ) Rank( ya then get the following principle of rank diminishing.
Jr Fv ,) S. Rank(JF, ,),k = 2, 2,.., L, which is Eq. (7). We can
Theorem 2 (Principle of Rank Diminishing). Suppose that each layer fi,i= L network F almost everywhere smooth" and data domain a manifold, then both the rank of sub-networks and intrinsic dimension of feature manifolds decrease monotonically by depth:
Rank(fi) 2 Rank(f2 o fi) 2 ... ...2 Raak( o... fi) ))== 2 Rank Cank(FL);
dim(X) 2 dim(Fi) 2 dim((F) ... 2 dim(FL).
(8)
This principle describes the behavior of generic neural networks with almost everywhere smooth components, which exhibits the monotonic decreasing (but not strictly) of network ranks and intrinsic dimensionality of feature manifolds.
A flaw of the above principle is that it does not tell whether the rank must decrease. So we need further analysis for the chance of strictly decreasing. A most direct reason support the strictly decreasing comes from the structural impetus of rank deficiency of numerous network components. Frequently used operations like pooling, downsampling. and dense layer loose ranks considerably as they explicitly decrease the ambient dimensions of feature representations, or have low rank weight matrices [34]. Specifically, we can have global criterion for whether network component will lose ranks as follows.
Theorem (Structural Impetus of Strietly Decreasing)- 5 RRuuhly speaking, if almost everywhere on the input feature manifold, there is a direction such that moving along this direction keeps the output invariant, then the intrinsic dimension of the output feature manifold will be strictly lower than that of the input. The maximum number of independent such directions gives lower bound on the number of lost intrinsic dimensions.
For example, the ReLU activation gives the same output for all inputs that only differs from each other in the negative parts. Thus the feature manifold after a ReLu activation will lose the dimension
'It means having arbitrary order gradients except for a zero measure set the input domain 'The rigorous version is given in the Appendix.

---

SwieT
ResML
Partial Rannk:
Partaa Rankk
503
165+
The E-th Layer RosNet
The -th Miker Layer MIL M.L.PMM P-Mixrr
The 1tt Layer Transformer
(a) Rank Histogram for ResVet.
(b) Rank Histogram for MT. P-Mixer
(c) Rank Histogram for Transformer
aoobi Festore
Feeture
Testure
Number FC
The Number FC Layers
Layors (1) Xavier Normal Initialization
(d) Normal 2 (0,0.02) Initialization
(c) Kaiming Normal Initialization
Figure 1: Partial rank of Jacobian matrices of CNN, MLP, and Transformer architecture networks for different layers on ImageNet (top row); rank of Jacobian matrices and feature dimensions of linear MLP network following conditions Theorem (bottom row). All the models show a similar trend of fast decreasing of ranks predicted by Theorems 4 and 5.
ChuMitue
HoNee-38
Classiiic aatttitctionnnooooo Accuraty (N)
Clacofficatonn Accurry (%)
Lastiiicaionn.
40%
12%
Percentage of principal components (%)
Percentage of principal components (%) (b) Subspace Accuracy MLP-Mixers
Percentage of principal compowents (%)
(a) Subspace Accuracy ResNets
Subspace Accuracy Transformers
Figure Classification Accuracy (top-1) of using subspaces spanned by top-k% eigenvectors (principal components) of the final feature manifolds. For all networks small percentage (see Tab. 1) of cigenvectors enough to reproduce the classification accuracy of the whole network, indicating low intrinsic dimension of final feature manifolds. Note that the r-axes are non-linear.
for those negative parts. For general linear layers (e.g., g.. pure convolution and dense layers), the feature dimensions that belong to the orthogonal complementary spaces of the weights will be lost after applying the linear transformations, the weight matrices are inactive to changes in their orthogonal complementary spaces.
5 Limiting Behavior of Network Ranks for Infinitely Deep Networks
From a theoretical perspective, of great interest to consider the limiting behavior rank of infinitely deep neural networks. However, it is infeasible to directly consider this problem and give any rigorous analysis for general cases- it is impossible to exhaust the tedious discussion of how specific structures can influence network ranks. Thus we need concentrate simplified but still representative math model this problem.
Theory Setup and Its Necessity In brief, we will assume in this section that the Jacobi matrices are random matrices independently following some distributions in fixed Fuclidean space. We then consider the limiting behavior of the singular values of series { Fr, 100 Modeling the Jacobi
5

---

2366
Bidure
5058
5000
1210
7468
7835
Te0e
8110
7100
7915
4000
4830
c000
3785
1460
3375
7900
1388
7000
Kaining
(
Saraal
Leifurm
Parameter
Tnitisllsation
Layers
Normalication Layers Histogram
(a) Dimeusioo Histagram
(b) Rank Histogram
(c) Dimension Histogram
Figure 3: PCA dimension of feature spaces and rank of Jacobian matrix for commonly seen network components under standard Gaussian inputs and randomized weights. Convolution and FC layers tend to lose rank considerably; normalization layers, like InstanceNorm (IN) [45], LayerNorm (LN) [2], and GroupNorm (GN) [51], lose rank modestly. But none can preserve rank.
matrices with random matrices allows neglect the influence of specific architectures and reveal the intrinsic property using large number law of probability. While limiting them in the same space is common practice in analyzing infinitely deep networks [42, 36, 39, 3, 32], offers the feasibility to define limits of variables. For example, if the sizes of Jacobi matrices change from time to time, some singular values may suddenly disappear appear in the series {J Jm=1 thus it unable define limits for them.
After this simplification, the problem is still representative for understanding what happens in reality. The transformer, recurrent neural networks, and some very deep CNN architectures with constant width [52] can be viewed examples of this setting. Another interesting setting is to assume that the Jacobi matrices follow the most simple distribution- Gaussian distributions. This can lead to a much stronger yet intuitive result network ranks. So
most interested in the limiting behavior when the Jacobi matrices follow weak regularized distributions and the Gaussian distributions. These two then lead Theorems 4 and correspondingly.
Theorem 4. Let the network be F == f o....... f¹ and all the ambient dimensions of feature manifolds be the same as the ambient dimension of imputs, i.e., f Ih" + IR" l,..., Suppose the Jacobian matrix of each layer ft independently follows some distribution μs, and E, max mmx(1og [ ll,, 0}] < oc. Let as denote the k-th largest singular value of Jr. Then there is an integer < n and positive constants ,rr...,, that only depend on μ. such that we have for p-allmsst everywhere,
Ok
+ 0, k .....,,, as L oc.
JF
meaning that for any tolerance e O, (ank,(F) drops below r + as L + oc.
Theorem Let the network be F = o f and all the ambient dimensions of feature
manifolds be the same the ambient dimension of inputs, i.e., fh fk: R + R k 1,... L. Suppose that y. independently follows the standard Gaussian distribution. Let σk denote the k-th largest singular value of Jp. Then almost surely
Ok + = exp, 1/(
lim L-+00
(1100/11)
< 1, -(()))
,,.
(10)
[|Je|l12)
2
where vb = D/TT" and is the Gamma function. That means for a large and any tolerance c, Rank, (F) drops exponentially with speed nCL, where C < 1 is positive constant that only depends on n.
The proofs of these two theorems rely on the advances of random matrix theory, especially the study of Lyapunov exponents of linear co-cycle systems [43, 50, 18, 40, 29] (see Appendix A for detail). Both of the two theorems reveal that infinitely deep networks have an intrinsic intention to drop ranks rapidly and centralize energy in the largest singular vectors. This intention is purely aroused by the two essential ingredients, chain rule and matrix multiplication, as here we omit the consideration of specific network designs, but focus on the large number law of coupling of massive smooth functions.

---

These two theorems point out that the ratio of non-largest singular value divided by the largest one will converge to zero with an exponential speed. As result, the numerical rank of the network will decrease quickly after the layer is deep enough. While this is not the hard rank we use in Theorem 2, we can still use singular vectors of JP corresponding to the singular values in Theorem 4, and og (Aæ/ || q.) then when the depth + oo and Ax is small, we have F(x+ Ax) = F +JrAZ + o ||
infer the low rank structure of feature representations. Let 9ı,
-n be the
on aa + o[] Aæ ss σ 2im1 ag which means the neighborhood of data æ is mapped into an r-dimensional subspace spanned by the singular vectors of the largest singular values.
0 E a;g; σi _ Lt
This can also explain why layer-wise regularization techniques are helpful for training deep networks, as those regularization techniques re-normalize the singular value distributions of deep networks, easing the trend of the ratio S becoming infinitesimal. Specifically, if the data distribution near æ is d1 assumed be standard Gaussian, then the feature distribution is (F JF. which has very low rank covariance matrix due to o, , 2 is tiny. Then applying Batch Normalization will try to pull this feature distribution back to Gaussian with identity covariance, thus ease the trend of feature dimension collapse. This has also been discussed in related work of the Batch Normalization [13, 5J. We will discuss some other techniques to remiss the rank diminishing in the Appendix.
These two theorems are also connected with the famous gradient explosion issue of deep neural networks [4, 37], where the largest singular value of the Jacobian matrix tends to infinity when the layer gets deeper. This issue could be viewed
special case of Theorem 5 that investigates the behavior of all singular values of deep neural networks. The behavior of network ranks in fact manipulates the well-known gradient explosion issue. Rigorously, we have the following conclusion. Corollary 1. Under the condition of Theorem 5, then almost surely gradient explosion happens at an exponential speed, i.e., log JI log 01 3 (log 2 + vnn/2)) oo when L, is large.
6
Validating the Rank Behavior
6.1
Validating Rank Diminishing by Depth
In this section, we numerically validate our theory in three types of architectures of benchmark deep neural networks, CNNs, MLPs, and Transformers, in the ImageNet [14] data domain. Information of those networks is listed in (Appendix) Tab. A2. For validating the tendency of network rank of Jacobian matrices, we use the numerical rank of sub-matrices of Jacobian on the central 16 x 16 x 3 image patch of input images. We report the results of other choices of patches in the Appendix. Details of the experiment setup can be found in Appendix E.
Rank Diminishing of Jacobians As is discussed in Sec. 3.2 and (Appendix) Lemma 2, the partial rank of the Jacobian is powerful weapon for us to detect the behavior of huge Jacobian matrices, which are infeasible to compute in practice. The decent value of partial ranks of adjacent sub-networks provides a lower bound to that of full ranks of them. Fig. (a.b,c) report the partial rank of Jacobian matrices of three types of architectures, where can find consistent diminishing of partial ranks in each layer, indicating larger rank losing for the full rank of Jacobian matrices.
Implicit Impetus Theorem 5 gives an exponential speed of rank decent by layers. We find that it corresponds well with practice. We investigate this exponential law in toy network of MLP-50, which is composed of 50 dense layers, each with 1,000 hidden units. The MLP-50 network takes Gaussian noise vectors of 101000 as inputs, and returns a prediction of 1,000 categories. As all the feature manifolds are linear subspaces in this case, their intrinsic dimensions can be directly measured by the numerical rank of their covariance matrices. We report the full rank of Jacobian matrices and intrinsic dimensions of feature manifolds under three different randomly chosen weights in Fig. (d.c.)). Due to the digital accuracy of float32, we stop calculation in each setting when the absolute values of elements of the matrices are lower than 1.19e - 7. We can find standard curves of exponential laws in all cases for both ranks of Jacobian and intrinsic dimensions of features.
Structural Impetus We validate the structural impetus in Fig. 3. To give an estimation for general cases, here we use Gaussian noises with the size of 128 x x8x x as inputs, and randomize weights of the network components to be validated. We plug those components into simple fully-connected (FC)
7

---

'hamntee
Acc. 99% Original Acc.
Is 'hamster" ?
'broccali" x A,,
'guinea pig' x Aig
Angora rabhit Ate
Inputs
Pretrained Classification Network
Logits
(4) Independence deficit: Clessilicatin cunfidence of some callegories are lineally decided by a few other categuries with fixed cueflicients.
ls: Angora rabbit
0.182
G: 'pooonhill'
A 0.199
4y: 'brocool'" 0,510
'eassel 0.206
triumphal arch
2 -0923
Egretta albus' Acc. 70. 0%
Acc. = 48. 5%
4. 'crane
b guinea pig'
Ag: umotrraaap''
"'uucoe Acc. 76.1%% 1%
0.764
12 Egretta caerulea'
0.123
A 0.261
_ 0.324
Original Acc. on "hamster" 48.6%
Original Acc. on 'Junco' 76.1%
Original Acc. on 'Egretta albus' 70.0%
(b) ResNet-50
(c) GluMixer-24
(d) Swin-T
Figure 4: Independence deficit. Classification confidence of some ImageNet categories are lineally decided by few other categories with fixed coefficients in the whole data domain. We illustrate this phenomenon in (a). Here we present some results from ResNet-50, GluMixer-24, and Swin-T. the (b,c.d) we illustrate the categories of i...., in (in the surrounding) to linearly decide category (in the center) and their corresponding weights } A
The classification accuracy on the validation set of using Eq. (12), instead of the true logits, predict the label is reported in blue (if tested on positive samples only, the accuracy rates are 98%, 90%, 82% for cases in (b,c.d) correspondingly). reported in green. We
For comparison, the original accuracy for the corresponding categories can find that 1) a few other categories can decide the confidence of the target category i; 2) some very irrelevant categories contribute the largest weights. For example in (c), the logits of class 'junco' the negative 'triumphal arch'. Both of them indicate a rather drastic competition of different categories for independent representations in final features due to the tight rank budgets.
layer of 8, 8,192 hidden units. As the structure is simple, we directly measure the intrinsic dimension of feature spaces and the full rank of Jacobian matrices before and after the features pass the network components to be measured. The dimension is determined by the number of PCA eigenvalues [23, 49] larger than 1.19e largest PCA eigenvalue. The batch size is set to 5,000. We find that the convolution (the kernel size 3) and FC layers (the weight size is 8, 8,192) tend lose rank considerably, while different normalization layers also lose rank modestly. But none of them can preserve rank invariant.
N max where is the number of PCA eigenvalues, and maax is the
6.2 Validating Low-Rank Terminal Spaces
This section seeks evidence to support a low-rank terminal feature space of deep networks, which can then support the significant diminishing of ranks in the previous layers. The evidence is consisted of a numerical validation that investigates the classification dimension of the terminal feature layer, together with a semantic validation that reveals the independence deficit of deep networks.
Numerical Validation To get an estimation of how many dimensions remain in the final feature representation, we measure the classification dimension in Fig. 2 and Tab. We report the classification accuracy produced by projecting final feature representations to its top 1% eigenvectors Fig. 2. We choose a threshold of such that this procedure can reproduce 95% of the original accuracy of the network. The corresponding ClsDim is reported in Tab. 1. As discussed in Sec. 3.3, this gives an estimation of the intrinsic dimension of the final feature manifold. We can find universal low-rank structure for all types of networks.

---

Semantic Validation We want to show that there are only a few independent representations to decide the classification scores for all the 1,000 categories of ImageNet. Specifically, can we predict the outputs of the network for some categories based on the outputs for few other categories, as illustrated in Fig. (a)? And if we can, will those categories be strongly connected to each other? A surprising fact is that, we can find many counter examples of irrelevant categories dominating the network outputs for given categories regarding various network architectures. This interesting phenomenon indicates a rather drastic competing in the final feature layer for the tight rank budgets of all categories, which yields non-realistic dependencies of different categories.
To find the dependencies of categories in final features, we can solve the following Lasso problem [44],
A = arg min E. ||| W F ax) [ + |||
(11)
A
where F(x) E 1000 is the slice of network from inputs to the final feature representation, is the sample from ImageNet X, and W is the final dense layer. The solution A will be a sparse vector,
with k non-zero elements Xs 2 Xa2 2 ... 2 X < 1000. We can then get
logits(æ:, i) = logits(s, (1) + ... + Ai logits( x. ic), & (fi...., 4 k < 1000, Va E X, (12)
where logits(a; i I,.... k is the logits of network for category 1,. i.e. logits(æ, it) Wi, (x)). It is easy to see that outputs for category i are linearly decided by outputs for i1...., 1 and are dominated by outputs for é1.
Fig. 4 demonstrates the solutions of Eq. (12) for three different categories in ImageNet with 7/ = 20, and network architectures ResNet-50, GluMixer-24, and Swin-T. The results are surprising. It shows that many categories of the network predictions are fact 'redundant', as they are purely decided by the predictions of the other categories with simple linear coefficients. In this case, the entanglement of different categories cannot be avoided, thus the network may perform poorly under domain shift. An even more surprising finding is that, some very irrelevant categories hold the largest weights when deciding the predictions the redundant categories, which means that the networks just neglect the unique representations of those categories in training and yield over-fitting when predicting them.
7 Related Work
Previous studies of rank deficiency in deep neural networks follow two parallel clues. One is the study of rank behavior in specific neural network architectures. Dong et al. [15] studies deep networks consisting of pure self-attention networks, and proves that they converge exponentially to rank-1 1 matrix under the assumption of globally bounded weight matrices. Daneshmand et al. [13] studies the effect of BatchNorm on MLPs and shows that BatchNorm can prevent drastic diminishing of network ranks in some small networks and datasets. Both of those works avoid directly validating the behavior of network ranks in intermediate layers due to the lacking of efficient numerical tools. An independent clue is the study of implicit seff-regularizatioo,, which finds that weight matrices tend to lose ranks after training. Martin and Mahoney [34] studies this phenomenon infinitely-wide. peer-paramttric neural networks with tools from random matrix theory. Arora et al. [1] studies this phenomenon in deep matrix decomposition. Those works focus on the theoretical behavior of rank of weight matrices induced by the training instead of network ranks.
8 Conclusion
This paper studies the rank behavior of deep neural networks. In contrast to previous work, we focus on directly validating rank behavior with deep neural networks of diverse benchmarks and various settings for real scenarios. We first formalize the analysis and measurement of network ranks. Then under the proposed numerical tools and theoretical analysis, we demonstrate the universal rank diminishing of deep neural networks from both empirical and theoretical perspectives. We further support the rank-deficient structure of networks by revealing the independence deficit phenomenon, where network predictions for a category can be linearly decided by a few other, even irrelevant categories. The results of this work may advance understanding of the behavior of fundamental network architectures and provide intuition for a wide range of work pertaining to network ranks.

---

Acknowledgement
The authors thank Prof. Dangzheng Liu of the University of Science and Technology of China for disuession. This work was supported by National Key R&D Program of China under Grant 2020AAA0105700 National Natural Science Foundation of China (NSFC) under Grants 62225207 and U19B2038, and the University Synergy Innovation Program of Anhui Province under Grants GXXT-2019-025.
References
[1] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. Adv. Neural Inform. Process. Syst. 32, 2019.
[2] J. L. Ba, J. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
[3] S. Bai, J. Kolter, and V. Koltun. Deep equilibrium models. In Adv. Neural Inform. Process. Syst., 2019.
[4] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2)157-166, 1994.
[5] N. Bjorck, C. P. Gomes. B. Selman, and K. Q. Weinberger. Understanding batch normalization. Adv. Neural Inform. Process. Syst. 31, 2018.
[6] C. Blakeney, Y. Yan, and Z. Zong. Is pruning compression?: Investigating pruning via network layer similarity, In IEEE Winter Conf. Appl. Comput. Vis., pages 114-922, 2020.
[7] X. Cai, C. Ding, F. Nie. and H. Huang. On the equivalent of low-rank linear regressions and linear discriminant analysis based regressions. In Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pages 1124-1132, 2013.
[8] B. Chen, T. Dao, E. Winsor, Z. Song, A. Rudra, and C. Ré. Scatterbrain: Unifying sparse and low-rank attention. Ady. Neural Inform. Process. Syst., 34, 2021.
P. Chen, Si, Li, C. Chelba, and C.-J. Hsich. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. Adv. Neural Inform. Process. Syst., 31, 2018.
[10] P. Chen, H.-F. Yu, Dhillon, and C.-J. Hsieh. Drone: Data-aware low-rank compression for large NLP models. Adv. Neural Inform. Process. Syst., 34:29321-29334 2021.
[11] Y. Cheng, Yin, and Y. Yu. Lorslim: Low rank sparse linear methods top-n recommendations. In IEEE Int. Conf. on Data Min., pages 90 90-99 99. IEEE, 2014.
[12] Chiu, Y. Deng,
Rush. Low-rank constraints fast inference structured models. Adv. Neural Inform. Process. Syst. 34, 2021.
[13] H. Daneshmand, Kohler, Bach, T. Hofmann, and Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Adv. Neural Inform. Process. Syst., 33: 13:18387-18898 2020.
[14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., pages 248-255. leee, 2009.
[15] Y. Dong, J.B.. Cordonnier, and A. Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In Int. Conf. Mach. Learn., pages 2793-2803. PMIR, 2021.
[16] R. Feng. Z. Lin, J. Zhu, D. Zhao, J. Zhou, and Z.-J. Zha. Uncertainty principles of encoding gans. In Int. Conf. Mach. Learn., pages 3240 3251, PMLR, 2021.
[17] R. Feng. D. Zhao, and Z.-J. Zha. Understanding noise injection in gans. In Int. Conf. Mach. Learn., pages 3284-3293. PMLR, 2021.
[18] H. Furstenberg and H. Kesten. Products of random matrices. Ann. Math. Stat., 31(2):457-469 1960.
[19] D. Goldfarb and Z. Qin. Robust low-rank tensor recovery: Models and algorithms. SIAM J. Matrix Anal. Appl., (5(11)225-22)) 2223,-25,,12 2014.
[20] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition, In /EEE Conf. Comput. Vis. Pattern Recog., pages 770-778, 2016.
10

---

[21] M. W. Hirsch. Disferential topology, volume 33, Springer Science & Business Media, 2012.
[22] K. Hoffman. Linear algebra. Englewood Cliffs, NJ, Prentice-Hall, 1971.
[23] H. Hotelling. Relations between two sets variates. Biometrika, 28(3/4):311337,, 1936.
[24] C.-J. Hsich, K.-Y. Chiang, and S. Dhillon. Low rank modeling of signed networks. In Proc. ACM SIGKDD Int. Conf. Knowl, Discov. Data Min., pages 507-515, 2012.
[25] Y.-C. Hsu, T. Hua, S. Chang. Q. Lou, Y. Shen, and H. Jin. Language model compression with weighted low-rank factorization. Int. Conf. Learn. Represent. 2021.
[26] L. Jing. L. Yang, J. Yu, and M. K. Ng. Semi-supervised low-rank mapping learning for multi-label classification. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 1483-1491, 2015.
[27] R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Adv. Neural Inform. Process. Syst., 34, 2021.
[28] M. Kheirandishfard, F. Zohrizadeh, and F. Kamangar, Deep low-rank subspace clustering, In IEEE Conf. Comput. Vis, Pattern Recog. Worksh., pages 864-865, 2020.
[29] J. F. C. Kingman. Suhadditive ergodic theory. Amn. Probah., pages 883-899, 1973.
[30] S. Kong and C. Fowlkes. Low-rank bilinear pooling for fine-grained classification. In IEEE Conf. Comput.
Vis. Pattern Recog- pages 365-374, 2017.
[31] M. Lin, R. Ji, Y. Wang. Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In IEEE Conf. Comput. Vis. Puttern Recog., pages 1529-1538, 2020.
[32] Liu, K. Kawaguchi, B. Hooi, Y. Wang, and X. Xiao. Eigun: Efficient infinite-depth graph neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors. Adv. Neural Inform. Pro- cess. Syst., volume 34, pages 18622-18773. Curran Associates, Inc., 2021. URL https: //tppceeeeic...... neurips.com ctcpiip..ccpppppppee http:cccppppeee//eee http:///00/eeeeeeeeeeeeeeeeeeeee
htte////eeeeeeeeeeeeeeeaaaaaaaaa 939c1 Paper pdf.
[33] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Math. USSR Sh., 1(4):457, 1967.
[34] C. H. Martin and M. W. Mahoney. Implicit reff-regularization in deep neural networks; Evidence from random matrix theory and implications for learning. J. Mach. Learn. Res., 22(165): 2021.
[35] J. Mu, R. Xiong, X. Fan, D. Liu, F. Wu, and W. Gao. Graph-based non-convex low-rank regularization for image compression artifact reduction. IEEE Trans. Image Process., 29:5374-5385. 2020.
[36] A. Nazaret and D. Blei. Variational inference for infinitely deep neural networks. In Int. Conf. Mach. Learn., pages 0447-164611 PMLR, 2022.
[37] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Int. Conf. Mach. Learn., pages 1310-1318 PMLR, 2013.
[38] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen. Z. Lin, N. Gimelshein. L. Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Adv. Neural Inform. Process. Syst., 32, 2019.
[39] S. Peluchetti and S. Favaro. Doubly infinite residual neural networks: diffusion process approach. J. Mach. Learn. Res., 22(175):1-48, 2021. URL http: htwp///rrrrrrr//reeeeeeeeeeeeeee wttp::t..ppppppaaeeeeee wwtgppapers//2220-02122 html.
[40] Y. B. Pesin. Characteristic Lyapunov exponents and smooth ergodic theory. Russ. Math. Surv, 32(4):55, 1977.
[41] A. Quarteroni, R. Sacco, and F. Saleri. Numerical mathematics, volume 37. Springer Science & Business Media, 2010.
[42] S. Sonoda and N. Murata, Transport analysis of infinitely deep neural network. J. Mach. Learn. Res., 20 (1):31 82, 2019.
[43] R. Temam. Infinite-dmme difinie-dimessioall dynamicul systems in mechunics and physics, volume 68. Springer Science & Business Media, 2012.
[44] R. Tibshirani. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Series B Stat, Methodol., 58(1):267-288, 1996.

---

[45] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprini arXi::1607.08022 2016.
[46] T. Vogels, S. P. Karimireddy. and M. Jaggi. Practical low-rank communication compression in decentralized deep learning. Adv. Neural Inform. Process. Syst. 33:14171-144 1-1181,, 2020.
[47] H. Wang and N. Ahuja. Rank-r approximation of tensors using image-as-mait mnntrii representation. In IEEE Conf. Comput. Vis. Pattern Recog,, volume pages 346-353. IEEE, 2005.
[48] H. Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialglrichungen. Math. Ann., 71:441-479, 1912.
[49] S. Wold, K. Eshensen, and P. Geladi. Principal component analysis. Chemom. Intell. Lab. Syst, 2(1-3): 37-52, 1987.
[50] A. Wolf, J. B. Swift, H. L. Swinney, and J. A. Vastano. Determining Lyapunov exponents from time series. Physica 16(3):28-3177, 1985.
[51] Y. Wu and K. He. Group normalization. In Fur. Conf. Comput. Vis., pages 3-19, 2018.
[52] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and mean field theory of CNNs: How to train 10,000 layer vanilla convolutional neural networks. In J. Dy and A. Krause, editors, Int. Conf. Mach. Learn., volume 80 of Proceedings of Machine Learning Research, pages 5393-5402. PMLR, 10-15 2018.
[53] Ye. Generalized low rank approximations of matrices. Mach. Learn., 61(1):167-191, 2005.
[54] X. Yu, T. Liu, X. Wang, and D. Tao. On compressing deep models low rank and sparse decomposition. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7370-7379, 2017.
[55] Z. Zha, X. Yuan, B. Wen, J. Zhou, J. Zhang, and C. Zhu. From rank estimation to rank approximation: Rank residual constraint for image restoration. JEEE Trans. Image Process., 09:3254-3269, 2019.
[56] M. Zhan, S. Cao, B. Qian, S. Chang, and J. Wei. Low-rank sparse feature selection for patient similarity learning. In IEEE Int. Conf. Data Min., pages 1335-1340. IEEE, 2016.
[57] T. Zhang, B. Ghanem, S. Liu, C. Xu, and N. Ahuja. Low-rank sparse coding for image classification. In Int. Conf. Comput. Vis., pages 281-288, 2013.
[58] Y. Zhang, Z. Jiang, and S. Davis. Learning structured low-rank representations for image classification. In IEEE Conf. Comput. Vis. Pattern Recog. pages 776-683, 2013.
[59] J. Zhu, R. Feng, Y. Shen, D. Zhao, Z.-J. Zha, J. Zhou, and Q Chen. Low-rank subspaces in gans. Adv. Neural Inform. Process. Syst. 34:16644 16658, 2021.
[60] H. Zou, T. Hastic, and R. Tibshirani. Sparse principal component analysis. J. Comput. Graph. Stat., 15(2): 265-286, 2006.
12


Reviews:
[{"review_id": "zgEK6m68Fkz", "reviewer": "Reviewer_4Q14", "summary": "This work presents some theoretical results that imply that the rank of the Jacobian between the inputs and features of deep networks is non-increasing with depth. They predict that in some settings it should in fact decrease exponentially with depth to some fixed value. \n\nThey also develop efficient methods to estimate the Jacobian rank of real networks and show empirically that it indeed decreases with depth across a number of different architectures. ", "questions": "Following my remarks in the previous section, how difficult do the authors think it would be to relate some of the predictions in a quantitative way to various aspects of the network architecture? ", "limitations": "Limitations have been addressed ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The effects of depth on the learned representations in deep networks and their geometric structure is an important area of study. While this work contains an interesting combination of theoretical and empirical results, I believe the connection between the two would have to be made more concrete. \n\nThe result about non-decreasing rank follows from the basic compositional structure of the network as the authors suggest, yet it is unclear that the rank must decrease. In fact, there is a vast literature on signal propagation in deep networks that approaches this question from a different angle (by studying covariance between hidden features as a function of depth, in which case convergence to certain fixed points should essentially be equivalent to the rank of the representation collapsing [1, 2]). This literature also highlights ways to avoid this phenomenon with a careful choice of initialization, and relies on modeling the dynamics of the correlations as a function of initialization hyperparameters. This allows one for example to train convnets of depth 10000 [2]. In the simplest case of a network with orthogonal weights and no non-linearities, it is clear for example that there is no decrease in rank, so there are clearly ways that it can be avoided. \n\nAnother related issue is that the results are vague in the sense that the behavior of the rank is not connected in a quantitative way with the structure of the network (i.e. the choice of nonlinearity, initialization, etc). I think the submission would be much more compelling if the results could take these into account and make predictions about their effects on the rank. For example, how is the rank one converges to or the speed of the rank decay related to properties of the network? \n\nAn additional, related concern is the connection between the experiments and the theory. The experiments that attempt to show exponential decay of the rank are not plotted on a logarithmic scale, which makes it hard to understand whether the decay there is indeed exponential or follows some other law. In addition, it appears that the rank decay in the case of resnets may be influenced more by the pooling layers or changes in width than any other operation, yet no mention of this is made in the text. \n\n\n[1] Poole, Ben, et al. \"Exponential expressivity in deep neural networks through transient chaos.\" Advances in neural information processing systems 29 (2016).\n\n[2] Xiao, Lechao, et al. \"Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks.\" International Conference on Machine Learning. PMLR, 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UJMt8Qm8I0R", "reviewer": "Reviewer_Dh79", "summary": "This work aims to study the rank of hidden layer representations of neural networks in relation to how deep the layer is in the network. In particular they note that the rank of the hidden layers diminish monotonically as we observe deeper layers. Numerical measures of rank are proposed and motivated. The primary theoretical concerns are the rank of the Jacobian from the input to the i-th layer of the network (essentially a linear approximation of the network mapping to that hidden layer) and the dimension of the feature space for a hidden layer. The paper further investigates the tolerance of the final hidden layer to dimensionality reduction by applying PCA to features space and projecting onto a decreasing number of eigenvectors. The number of eigenvectors remaining when a significant drop in performance is observed from the dimensionality reduction provides an approximation for the intrinsic dimensionality of the hidden layer. Finally, the paper explores the idea that it is possible to use the logits of different categories to classify another category in a dataset. One example is that by merely using -0.923 as a weight on the logit for the \"triumphal arch\" category it is possible to predict the \"junco\" category without loss of accuracy.", "questions": "No questions as yet.", "limitations": "I suggest that the authors be clearer on the conditions required for their theory to help. For example saying \"The principle of rank diminishing describes the behavior of general neural networks with almost everywhere smooth components,\" which does not seem to include ReLU networks but is described as general is unclear.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n## Originality\nThe paper is fairly original with the primary novelty being the rank metrics used and their justification. Additionally the paper touches on some possible connections between symmetry and rank which to my knowledge have not been explored, however, these connections are mainly pointed out but not discussed or treated theoretically.\n\n## Quality\nThe need for the numerical tools to measure rank is well motivated and the numerical tools themselves makes sense and are justified. The claims that are made appear correct and in-line with the evidence presented.\n\n## Clarity\nThere is some variance in the clarity of the paper for various sections. The writing is clear and understandable and the mathematical notation is consistent and intuitive which helps the clarity in the earlier sections greatly. Sections 3.2 and 3.3 are examples where the notation made potentially tricky sections more manageable. Figure 4 stands out as a very helpful figure. The effort on that is definitely worth it.\n\n## Significance\nThe paper touches on some significant points, like the point linking symmetries to lower ranks. The PCA experiment and the experiment on the using categories as predictors for others may be of general interest to the ML community.\n\n# Weaknesses\n## Originality\nA primary concern of this work is the fact that $Rank(AB) \\leq min{Rank(A), Rank(B)}$. This is even mentioned in the paper below equation 8 and is one of the primary tools for the work. This, however, is a well established principal and quite intuitive. Thus, the finding that the rank of the network decreases with layer depth is not surprising. Two possible interesting points: noise increasing network rank and structure avoiding the rank staying the same across layers are mentioned but do not form part of the analysis. The noise aspect is ignored in the theory and removed through the noise tolerant rank measures. The point on monotone decrease over equality of rank due to structure is discussed briefly.\n\n## Quality\nThe various sections of the paper feel quite loosely connected. Up to Section 4 the work considers whether the rank of the network decreases monotonically. The Section 5 considers PCA just on the final feature space and is the used to point out that low dimensional feature spaces do not hold semantically meaningful features for each category in Section 6. These sections are all related to rank, however, the connections do not seem to go deeper than that. Finally, there are some points where unjustified claims are made (or the phrasing makes these claims appear unjustified). Two examples are \"Theorem 5 that investigates the behaviour of all singular values of deep neural networks\" when theorem 5 requires hidden layers of the same size and assumes the Jacobians have Gaussian elements (which appears to be unrealistic in its own right) and \"The principle of rank diminishing describes the behavior of general neural networks with almost everywhere smooth components\" where it is not clear that ReLU networks would even fit this requirement.\n\n## Clarity\nTheorem 4 and Theorem 5, which are the most technical aspects of this paper are not given enough space. The clarity of the paper could benefit greatly from a more in-depth treatment of this section. In addition, how the theory of these sections relate to Figure 1 could also be explained more. For example I acknowledge that the shape of the bottom row of Figure 1 is non-linear but to call it exponential (which Theorem 4 and 5 predict) might also be a stretch. Understanding Theorem 4 and 5 would help with interpreting Figure 1. Figure 1 could also use different colours, especially for the bottom row where distinguishing between Jacobian and Feature rank/dimension is not easy. Finally, the notation of Section 5 is not easy to follow, particularly in the meaning of the $i_j$ double subscript where it is not immediately clear what $i$ and $j$ each refer to. Figure 4 does help clarify this a lot and with space constraints fully explaining the new notation may not be feasible.\n\n## Significance\nThis work appears generally significant, however, its significance is hindered by the same issues noted under the originality section. I feel that this work might spend too much time on the potentially quite obvious points of rank diminishing and on introducing the PartialRank and not enough time on the potentially significant points such as Theorem 4 and 5. My primary recommendation would be to rephrase the work more in line with those theorems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OKKaDv7xCU6", "reviewer": "Reviewer_dzFU", "summary": "The paper studies the dynamics of the rank evolution of the feature maps of a neural network as a function of its depth. By leveraging the abstract definition of rank of a function as the rank of the corresponding Jacobian matrix, the authors can study the rank dynamics in full generality (i.e. without assuming any specific architecture). This results in Theorem 1 (Principle of Rank Diminishing), that finds that the rank of neural network should never increase with depth due to its compositional nature (a neural network can be see as a composition of $L$ functions, where $L$ is the depth). Then, the authors analyze conditions under which the rank strictly diminishes (Theorem 3) and convergence of the rank to specific constants (Theorem 4-5). \n\nFinally, the authors apply their low rank findings to the study of the dependence and correlations between different output classes. They find that the output of some classes of ImageNet (e.g. hamster) can be predicted with a linear combination of the output for irrelevant classes (e.g. broccoli and mouse trap). The authors attribute this problem to the low rank representations of very deep network, as showed by their developed theory.", "questions": "1. I would like to ask the authors if in their view there is a way to quantify what they call the \"Structural Impetus\" for different architectures and normalization layers.\n\n2. Can (and if so, how) their results can be used to design better architectures, initializations, training procedures that better preserve the rank information?", "limitations": "I do not see a negative societal impact of this theoretical work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. **Generality and Importance of the Results**: the theoretical results are very general and remarkable, abstracting away from the specific architecture. The only assumption is the compositional nature of the layers, which includes most of architectures but excludes residual networks (as the author mention in the supplementary material). \n\n2. **Paper Organization**: the paper is very clear in explaining the abstract concepts of the first part. Until Theorem 2 (page 4), the theory is easy to digest. At first read, Theorem 1 seems trivial if one thinks about linear networks (i.e. simple product of matrices) and the famous property $\\text{rank}(AB) \\leq \\min(\\text{rank}(A), \\text{rank}(B))$, but the author do a great job to generalize it to any composition of functions through ideas from topology theory. The other two theorems delve deep into the rank diminishing properties of function compositions, showing an exponential decay of the rank with depth. \n\n3. **Independence Deficit of Feature Manifolds**: Section 5 provides a nice application of the theory, and would probably cause follow up works in trying to understand how one can reduce this undesirable effect of strong dependences between semantically different classes. \n\n**Weaknesses**\n1. **Inconsistency of Residual Network**: Skip connections are proposed as a tool to (partially) prevent the rank deficiency problem, and they give a brief theoretical argument in the supplementary material. However, this seems to be in contradiction with Figure 1, where an exponential decay of the rank is observed for ResNets, MLP-Mixers and Transformers, all architectures that adopt skip connections. This could be due to the fact that during training the magnitude of $\\text{Res}(x^i)$ becomes large, hence lowering the rank. At initialization, the magnitude of $\\text{Res}(x^i)$ can be controlled, e.g. with an appropriate factor inversely proportional to the depth (see for instance [1] for this scaling and [2] for its consequences on the rank). In any case, I found it confusing that skip connections are adopted in almost all the architectures used to exemplify the theory (skip connections that according to the authors should have an opposite effect). \n\n2. (minor) **Presentation Style of Structural and Implicit Impetus**: After brilliantly explaining the principle of rank diminishing, in my view  the concepts of \"Structural Impetus\" (due to the specific architectural modules) and \"Implicit Impetus\" (due to the very compositions of infinite modules) of rank diminishing could be better explained. In particular, I would invest some extra lines to better explain why normalization layer prevent rank diminishing, and maybe better introduce some concepts ( or instance \"moving along directions\" of Theorem 3 is not properly introduced and in general the current version of the Theorem fails to convey a simple and intuitive explanation).\n\n[1] Hanin, Boris, and David Rolnick. \"How to start training: The effect of initialization and architecture.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[2] Noci, Lorenzo, et al. \"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse.\" arXiv preprint arXiv:2206.03126 (2022). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FGa8IibP0Y1", "reviewer": "Reviewer_Gfbz", "summary": "This paper demonstrates a rank diminishing behavior of deep neural networks considering the mapping from the input space to the  feature space of an increasingly deeper intermedate layer. Theoretically, it proves that the rank doesnot increase as the layer depth increases. Experimentally, it demonstrates a general decreasing trend of rank on various NN architectures. This work also empirically demonstrates that the number of major PCA components at the final feature layer is much less than its ambient dimension, which leads to feak correlation between very different categories.", "questions": "1. [related to weakness 2] What exactly is the \"rank of its Jacobi matrix Jf over its input domain X\" in definition of the rank of function? How the rank of function is related to the rank of its Jacobian matrix at a specific point x\\in R^n?\n2. [related to weakness 1] Is there any theoretical explanation about why independence deficit happens for the final feature manifold?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghs:\n1. This work systematically studies the evolution of function rank throughout the layer computation and provides theoretical jusfication to the empirically observed rank diminishing behavior.\n2. The finding about the independence deficit of final feature manifolds is very interesting and provides insight to the lack of robustness of DNNs.\n\nWeakness:\n1. Classification dimension estimated by the number of major PCA  components in this work is not a good indicator of the feature dimesnion. In fact, a very low dimensional manifold can have high classification dimension. Therefore, the main results about rank diminishing cannot explain the interesting finding about low classification dimension of final feature manifolds. The statement in the abstract that \"independence deficit caused by the rank deficiency of deep networks\" is misleading.\n2. It seems that the definition of the rank of function and lemma 1 implicitly assume that the jacobian of neural network functions has a constant rank over the entire input space of R^n. This is a strong assumption that doesnot hold in general.  When this assumption holds for neural networks should be carefully discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept