The Low-Rank Simplicity Bias in Deep Networks
Authors: Minyoung Huh
Hossein Mobahi
Richard Zhang
Brian Cheung
Pulkit Agrawal
Phillip Isola
Venue: ICLR 2022 Submitted (2022)

Abstract: Modern deep neural networks are highly over-parameterized compared to the data on which they are trained, yet they often generalize remarkably well. A flurry of recent work has asked: why do deep networks not overfit to their training data? In this work, we make a series of empirical observations that investigate the hypothesis that deeper networks are inductively biased to find solutions with lower rank embeddings. We conjecture that this bias exists because the volume of functions that maps to low-rank embedding increases with depth. We show empirically that our claim holds true on finite width linear and non-linear models and show that these are the solutions that generalize well. We then show that the low-rank simplicity bias exists even after training, using a wide variety of commonly used optimizers. We found this phenomenon to be resilient to initialization, hyper-parameters, and learning methods. We further demonstrate how linear over-parameterization of deep non-linear models can be used to induce low-rank bias, improving generalization performance without changing the effective model capacity. Practically, we demonstrate that simply linearly over-parameterizing standard models at training time can improve performance on image classification tasks, including ImageNet.

Under review as a conference paper at ICLR 2022
THE LOW-RANK SIMPLICITY BIAS IN DEEP NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern deep neural networks are highly over parraaaaatteized compared to the data on which they are trained, yet they often generalize remarkably well. A flurry of recent work has asked: why do deep networks not overfit to their training data? In this work, make series of empirical observations that investigate the hy- pothesis that deeper networks are inductively biased to find solutions with lower effective rank embeddings. We conjecture that this bias exists because the volume of functions that maps to low effective rank embedding increases with depth. We show empirically that our claim holds true finite width linear and non-linear models and show that these are the solutions that generalize well. We then show that the low-rank simplicity bias exists even after training. using a wide variety of commonly used optimizers. We found this phenomenon to be resilient to initializa- tion, hyper- parameters, and learning methods. We further demonstrate how linear seee-parameterization of deep non-linear models can be used to induce low-rank bias, improving generalization performance without changing the effective model capacity. Practically, we demonstrate that simply linearly over parameterizing standard models at training time can improve performance on image classification tasks, including ImageNet.
INTRODUCTION
It has become conventional wisdom that the more layers one adds, the better deep neural network (DNN) performs. This guideline is supported, in part, by theoretical results showing that deeper networks can require far fewer parameters than shallower networks to obtain the same modeling "capacity" (Eldan & Shamir (2016)). While it is not surprising that deeper networks are more expressive than shallower networks, the fact that state-of-the-re art deep networks do not overfit, despite being heavily over anee-parameterieed" defies classical statistical theory (Geman et al. (1992); Zhang al. (2017); Belkin al. (2019)).
The belief that preenetraettiiittooonn via depth improves generalization is used axiomatically in the design of neural networks. Unlike conventional regularization methods that penalize model complexity (c.g., Ei/lz penalty), prcctearamtttiiitiionn does not. Yet, like explicit regularization, wver-parameterizztion appears to prevent the model from over-fitting (Belkin al. (2018); Nakkiran et al. (2019a)). Why this implicit regularization works is still an ongoing area of research. Even in the zero-training error regime, it is commonly observed that increasing the number of parameters improves generalization performance. Currently, the status quo explanation for this phenomenon is that gradient descent in over- peeemetrmmttee models acts as nuclear norm regularizer (Gunasekar et al. (2017); Arora et al. (2019a); Li et al. (2020)).
This work provides new set of observations that expand the growing body of work on over- parameterization and highlights the central role of depth in finding solutions that map to low effective rank embeddings for both linear and non-linear networks. Mainly, we make series of empirical observations that indicate deep networks have inductive bias to find lower effective rank em- beddings. First, we observe that random deep networks are biased to map data to a feature space whose Gram matrix has low effective rank. Next, find that this low effective rank phenomenon exists even after training with gradient descent. We further observe that the bias towards low effetive rank embeddings exists in wide variety of common optimizers even optimizers that do not use gradient descent. Moreover, we find that regardless of the initialization, the effective rank of the
c.g. Dosovitskiy al. (2020) trains 632 million parameter, 200+ layer model, 1.3 million images.

---

Under review as a conference paper at ICLR 2022
converged solution is largely dependent the depth of the model. This set of observations leads us to conjecture that deeper networks are implicitly biased to find lower effective rank embeddings because the volume of functions that map to low effective rank embeddings increases with depth. We then leverage our observations to demonstrate how one could use "depth" as a practical regularizer to achieve better generalization performance standard benchmarks such CIBAR (Krizhevsky et al. (2009)) and ImageNet (Russakovsky et al. (2015)).
PRELIMINARIES
2.1 NEURAL NETWORKS AND OVER- PARAMETERIZATION
Simple linear network A simple linear neural network transforms input x E Rax1 to output ÿ E R=x1 with a learnable parameter matrix W Rmxa
y = Wx.
For notational convenience, we omit the bias term.
prramterrmetrr linear networks One can over-parameterize a linear neural network by defining d matrices {W:) and multiplying them successively with input x:
ý WaWa-1
Wiz
We
where We II W, As long as the matrices are of the correct dimensionality - matrix Wa has m columns, Wi has n rows, and all intermediate dimensions {dim (W:) 2 min(m, n) then this over-paramin preeterratttiizttionn expresses the same set of functions as single-layer network. We disambiguate between the collapsed and expanded set of weights by referring to { {W;} as the over-parameterized weights and Wa as the
or the effective weights.
Non-linear networks For non-linear network, activation function & (e.g. ReLUU is interleaved in between the weights matrices:
W WWaaWWWWa1-- (Wi()))
In contrast to linear networks, non-linean models become more expressive as more layers are added.
2.2 EFFECTIVE RANK
We characterize the rank of matrix using continuous measure known as the effective rank:
Definition (Effective rank; Roy & Vetterli (2007)). For any matrix A E Rmxx the effective rank p is defined as the Shannon entropy of the normalized singular values:
min(n.m)
p(A)
Σ
og log(ō.),
where o, =,, By are normalized singular values. such that Σ, - Also referred to as the spectral entropy. Without loss of generality, we drop the exponentiation for convenience.
This measure gives us meaningful representation of "continuous rank", which is maximized when the magnitude of the singular values are all equal and minimized when a single singular value dominates relative to others. The effective rank provides us with metric that summarizes the distribution envelope. Effective rank has been used in prior works (Arora et al. (2019a); Razin & Cohen (2020); Baratin al. (2021)) and have also found that our observations are consistent with closest definition of rank in which we threshold the smallest singular values after normalization (Appendix C). Similar to prior works, refer to low effective rank as low rank here on out.
this measure extensively throughout
work. We
2.3 EMBEDDING MAPS
parameteric function f(W) E Fw is neural network parameterized with set weights {W} (Wi,.., Wa) that maps the input space the output space X' -> V. For training dataset of
2

---

Under review as a conference paper at ICLR 2022
linear network
non-linear network (ReLU)
normal distribution
uniform distribution
normal distribution
uniform distribution
unnomalized aalliiddd P(p(K)
-5-
25-
35
30
30-
30-
50-
25
25
25
25
20
20
20
20
unnoma
15
15
15
15
10-
10-
10-
10-
5-
5-
1.0
20 effective rank p(K)
1.2
14 1.6
18
22
1.0
1.2
14
16
1.8
20 2.2
effective rank p(k)
effective rank p(K)
effective rank plk)
deeph
depth
depch 4
depth 8
depth 16
Figure 1: Deep networks are biased toward low-rank: The approximated probability density function (PDF) the effective rank p over the Gram matrix is computed from features the networks. The Gram matrix is computed with 256 random inputs, and we use 4096 network parameter samples to approximate the cumulative distribution function. The CDF is used to compute the PDF via the finite difference method. We apply Savitzky Golay (1964) filter smoothen the approximation. There exists probability mass for lower-rank rank embeddings when adding more layers. The experiment is repeated for both normal and uniform distributions.
size ạ. the input and output data is X E Rexy and Y E Rmx4 Then, the predicted output is Y = (+) = f[w)(X) where & gn on'xg is the last-layer embedding and Wa guxn' is the last layer of the network.
We analyze the embedding space by computing the effective rank the Gram/kernel matrix K RPxp where p is the size of the test set. The ij-th entry of the Gram matrix corresponds to a distance kernel Ki n(tb, Φ5) where o; corresponds to the i-th column of क We the model's intermediate features before the linear classifier and use cosine distance kerne common method for measuring distances in feature space (Kiros et al. (2015); Zhang et al. (2018)). Since the dimensionality of the Gram-matris does not depend on the model parameters, we can compare neural networks with different modeling capacities in the zero training error regime.
Gram matrices often used analyze optimization and generalization properties of neural net- works (Zhang et al. (2019); Du et al. (2018; 2019); Wu et al. (2019); Arora et al. (2019b)). natural data, often assumed that we are trying to discover a low-rank relationship between the input and the label. For example, a model that overfits to every training sample without inferring any structure on the data will generally have a test gram-matre that is higher rank than that of model that has learned parsimonious representations. Lower rank on held-out data indicates less excess variability and indicative for studying generalization and robustness. The intuition becomes clearer in linear networks, since the rank of Gram matrix depends on the rank of the linear transformation computed by the network. We illustrate this empirically in Appendix I., where we see that there is tight relationship between the rank of the linear weight matrix and the resulting Gram matrix.
2.4
LEAST SQUARES
Given a dataset X, Y generated from W*, the goal is regress parameterized function f(w) (-) to minimize the squared-distame
[If[w (X) Yl1.. The rank(W") measure of the "intrinsic dimensionality" of the data and we refer to it as the task rank. In this work, we exclusively operate in the under-determined regime where we have fewer training examples than model parameters. This ensures that there is more than one minimizing solution.
THE INDUCTIVE PARAMETERIZATION BIAS OF DEPTH
Given that our models can always fit the data, what are the implications of searching for the solution in the over- parameterized model? In linear models, this is equivalent to searching for solutions in (W,) versus directly in W., One difference is that the gradient direction V (w.,)& C((W,)) is usually different than Vw. L(W)) for a typical loss function (see Appendix D. The consequences this difference have been previously studied in linear models by Arora et al. (2018; 2019a), where the
0.07
cosine kemel: x()r,
[
3

---

Under review as a conference paper at ICLR 2022
over-parameterized update rule has been shown accelerate training and encourage singular values to decay faster, resulting in low nuclear-norm norm solution. Here we motivate result from the perspective of parameter volume space.
Conjecture 1. Deeper networks have a greater proportion of parameter space that maps the inpul data lower-rank embeddings; hence, deeper models are more likely converge to functions that learn simpler embeddings.
We now provide a set of empirical observations that supports conjecture. Our work and existing theoretical works on gradient descent biases are not mutually exclusive and are likely compliment. We emphasize that do not make any claims on the simplicity of the function, but only the simplicity - lower effective rank of the embeddings.
3.1 LOW-RANK SIMPLICITY BIAS OF DEEP NETWORKS
Observation Randomly initialized deep neural networks are biased correspond Gram matrices with low effective rank.
When sampling random neural networks, both linear and non-linear, we observed that the Gram matrices computed from deeper networks have lower effective rank. We quan- tify this observation by computing the distribution over the effective rank of the Gram matrix in Figure 1. Here, the
unnormalized plp(k]
16-
depth
14
depth
12
depth
10
depth
depth
weights of the neural networks are initialized using uni-
form W ~ (((,-) or Normal distributions W; ~ (---). The input, output, and intermediate dimensions are 32, giving parameters {W;} E x2dx32x32 for a network with d layers. We draw 4096 random parameter samples and compute the effective rank on the resulting Gram matrix. We see that the distribution density shifts towards the left (lower effective rank) when increasing the number of lay- These distributions have small overlap and smoothen out with increased depth. This observation shows that depth correlates with lower effective rank embeddings.
0.5
1.0
1.5
2.0
2.5
effective rank plk)
Figure 2: Distribution convergence:
Rank distribution after training the network with gradient descent. The dotted line indi- cates the initial distribution. the solid line indicates the converged distribution, and the green line indicates the task rank.
The low lonk bias becomes more intuitive in linear models as there a simple way to relate the Gram matrix the weights of the model K Rs (Wa-1:1 x)² (Wa-1:1 XX). Intuitively, if any constituent matrices are low-rank, then the product of matrices will also be low-rank - the product of matrices only decrease the rank of the resulting matrix: rank (AB) S min (rank( A), rank B)) (Friedberg et al. (2003)). In Appendix L, we show that as the depth of the model increases, both the effective rank of the Gram matrix and the weights decrease together. Another way to interpret our observation is that for linear models, over- parameterization does not increase the expressivity of the function but re- re-weights the likelihood of subset of parameters the hypothesis class. For non-linear models, we cannot make the same claims.
Although uniformly sampling under the parameter distribution is an unbiased estimator of the volume of the parameter space, it is certainly possible that sub-space of the parameters is more likely to be observed under gradient descent. Hence, by naively sampling networks, we may never encounter model parameters that gradient descent explores. In light of this, repeat our experiment above by computing the PDF on randomly sampled parameters after taking n gradient descent steps.
Observation 2. Deep neural networks trained with gradient descent also learns to map data to simple embedding with low effective rank.
Figure illustrates the change in distribution as we train our model to convergence using gradient descent. Each randomly drawn network sample is trained minimize the least-squares error. The initial distribution is plotted with dotted lines, and the converged distribution is plotted with solid lines. As the model is trained, the distribution of the rank shifts towards the ground-truth rank (green
4

---

Under review as a conference paper at ICLR 2022
line). Training the model with gradient descent results in distribution that is still largely dependent on depth: this reaffirms that the role of optimization does not remove the parameterization bias in deep models. In fact, if the bias stems from the model's 's parameterization the same bias must also exist under other common and natural choices of optimizers. We investigate this claim in the next section.
3.2 IS THE LOW-RANK BIAS SPECIFIC TO GRADIENT DESCENT?
Deep neural networks trained with common and natural choices of optimizers also exhibit the low-rank embedding bias.
Observation 3.
The low-rank bias of deep networks has been primar- ily studied under the context of first-order gradient de- cent (Arora et al. (2018; 2019a)): how and why does gra- dient descent converge low nuclear norm solution. In contrast, our conjecture focuses on the bias of parameter- ization of the network and not on the bias introduced by the gradient descent. Since parameterization bias exists regardless of the optimizer choice, we would expect to observe the low-rank simplicity bias on wide range of optimizers. We directly show this in Figure 3 by ablat- ing across various popular choices of optimizers on least- squares with linear networks. Here, we compare against Nesterov (Nesterov (1983); momentum), ADAM (Kingma & Ba (2015); hessian approximator), L--BGS (Liu & Nocedal (1989); second-order), CMA-ES (Hansen al. and random search. All mod-
depth depih
depth
effective rank convergen gen
3.2
3.0
2.8
2.6
2.4
2.2
2.0
1.8
sgd
nesterov adam bige
cma-ds
optimizers
Figure 3: Low-rank bias & optimizers: Least-squares trained on lincar neural net- works using various optimization methods. The rank of the converged Gram matrix correlated with depth of the network. The experiment is repeated 5 times.
(2003) evolutionary-searge els were trained to zero training error except for random- search. For random search, we randomly initialize the network 100, 000 times and take the best performing sam- ple. As we have seen with gradient descent, the experiment indicates that even when we train with wide suite of commonly used optimizers, the solution obtained by these models depends on how the model was originally parameterized.
3.3 CAN THE SIMPLICITY BIAS BE EXPLAINED SOLELY BY THE INITIALIZATION?
The previous set of experiments indicate that deeper networks are biased towards low-rank embedding at both initialization and convergence. In these experiments, the random settings of neural networks had different initial distributions. This happens because, even if the individual weights are normally distributed, the weights constructed from series of matrix multiplications result in distribution normally distributed weights
that has a high density around zero. For example, the product becomes symmetric x-squared distribution, with degrees of freedom. Hence, one could argue that the converged solutions are low-rank because of the initialization bias.
Observation 4. Deep neural networks are biased towards learning low-rank embeddings regard- less of the initialization.
To test whether the initialization of the model affects the rank of the converged solution. We optimize our network W e xxx32x32 on seast-squares where the task-rank is set to 24. All models are trained for 4000 epochs, using gradient descent, and the best learning rate is chosen for each depth. In Figure (left), for models using default initialization, we show that increasing the number of layers decreases the effective rank of the Gram matrix at convergence. We repeat the experiment in Figure (right) by initializing the over-parameten parammeeettte models with the distribution associated with the 32-layer linear network. Following a similar trend to that of default initialization, deeper models learn embeddings that are a lower rank than the shallower counterparts. Although initialization is not insignificant. we that the depth of the model has tight control over the solution in which the model explores. For deeper networks, the majority of the parameter volume is mapped to low-rank embedding (Observation 1), and therefore it is expected that typical search algorithm
observe that
5

---

Under review as a conference paper at ICLR 2022
default initialization
low-rank initialization
rinn
Paslattrr
coneeged
aB
sair
Intelbutte
02
loss
effective
loss
effective
50
10
depth
depth
depth
depph
Figure 4: Bias of parameterization: The effective rank of the Gram matrix from initialization to convergence on various depth. For each depth, we train a linear network using gradient descent on least squares regression. We repeat our experiments times with different seeds, and report the median of these runs. The rank initialization and convergence is indicated by white and colored dots, respectively. For deeper models, the effectiiv-rankk is lower at initialization because the product of normally distributed weights is no longer normally distributed. On the right, we initialize the networks with the same low-rank distribution of weights as the 32 32-layer model. We observe that shallower networks tend to converge to higher rank embeddings.
would likely encounter parameters that map to low-rank embeddings regardless of initialization. Similarly, for a shallower network, it would be easier to find solution with higher rank embeddings.
3.4 RELATION TO RANDOM MATRIX THEORY
In linear models, we have a special case in which the low-rank embedding corresponds to low-rank weights. This enables us make natural connection to existing theoretical work from random matrix theory (RMT), which studies the spectral distribution under matrix multilpications (Akemann al. (2013a;b); Burda et al. (2010)). We leverage the results from Pennington et al. (2017); Neuschel (2014) to show the following:
Theorem 1. Let p be the effective rank measure defined in Definition 1. For a linear neural network with d-layers, where the parameters drawn from the same Normal distribution {W.).- ~ W, the effective rank of the weights monotonically decreases when increasing the number of layers when dim(W) DC,
P (W.Wa-1-.. WT) S P (W W.)
Proof. See Appendix G.
The theory is preliminary and can only be understood in the context of an infinite width model with random matrices. Yet, we have found in practice that the empirical spectral distribution of finite-width models is well approximated by theory (see Appendix A). The main contribution of our work is on the empirical theory the low-rank bias of deep networks; nonetheless, we show that there is a natural theoretical connection to RMT to stimulate future research.
OVER- PARAMETERIZATION AS REGULARIZER
Thus far, we have observed that depth acts bias for finding functions with low-rank embeddings. As one could imagine, this inductive bias of depth could be used to help but also hurt generalization performance. Our observations indicate that the low-rank simplicity bias helps when the true function we are trying to approximate is low-rank. On the contrary, if the underlying mapping is a high-rank, or the network made too deep, depth could have converse effect on generalization. Ample evidence from prior works (Szegedy et al. (2015); He et al. (2016)) suggests that pvee-paramettiizationn of non-linear models improves generalization on fixed datasets, but blindly increasing the number of layers without bells & whistles (c.g., batch-norm, residual connection, etc.) hurts (He et al. (2016)).
Fortunately, networks are trained on natural data, where often the goal to discover a low-rank relationship between the input and the label. Hence, the inductive bias of depth acts prior rather than a bug. As noted by Solomonoff (1964) theory of inductive inference, the simplest solution is often the best solution, suggesting that low-rank mapping in neural networks can be used to improve generalization and robustness to overfitting. However, increasing the number of non-linear layers also increases the modeling capacity, thereby making it difficult isolate the effect of depth.

---

Under review as a conference paper at ICLR 2022
original
weer-parameterized
1.0
1.0
monuuEeu angeBou
singul ar values
occunacy
0128
0128
0231
30
150 0251
0.0
30
60 90
120
150
0.0
training epochs
training epachs
Figure Training dynamics: Singular values of the Gram matrix for both original (left) and linearly over-parameterized (right) model throughout training. The models are trained on CIFAR100 10 using SGD. Since the first few singular values dominate the distribution, we plot the negative log magnitude of the normalized singular values to better visualize how the intermediate singular values change. The singular values sorted from largest smallest σ; 0;-1 (top bottom in the figure) where blue means large and red means small. original and the over-parameterized models are functionally equivalent and use same colorbar and scale. The dotted lines (--) indicate the learning step schedule, and train and test accuracies are overlayed on top of the distribution. The over parameterized model learns lower rank embedding and exhibits less overfitting, and has better generalization. See Figure 1l and Figure 12 the Appendix for the dynamics of the individual weights.
Nevertheless, since non-linear network is composed of many linear components, such as fully connected and convolutional layers, we can over-parameterize these linear layers to induce low- rank bias in the model without increasing the modeling capacity. The details of our linear over- parameterization method are in Appendix B. We observe that such linear wver-parameterization improves generalization performance classification tasks. Furthermore, regularization outperforms models trained with several choices of explicit regularization. Guo et al. (2020) made similar empirical observation in the context of model compression where linear wwer-parameterization improves generalization, but why it works is unexplored.
find that such implicit
4.1
IMAGE CLASSIFICATION WITH OVER-PARAMETERRAATIOO
Using the linear expansion rules in Appendix B, we over-parameterize various architectures and evaluate suite of standard image clas- sification datasets: CIFAR10 0, CIFAR100, Tma Tma. All models trained using SGD with a momentum of 0.9. For data augmen- tation, we apply a random horizontal flip and random-resized crop. We follow standard train- ing procedures and only modify the network architecture (see Appendix E).
Expansion
CIFARI0
CIFARI00
Factor
PC
Conv
accuracy
guin
accuracy
guin
1x
86.9
57.0
R022
5K.4"
2x
87.8
+0.9
61.0
+4.0
2x
/
7
89.1
+2.2
61.2
+4.2
7
87.3
40.4
5977
+2.7
4x
89.1
+2.2
61.3
+4.3
89.0
42.1
63.5
+6.5
83.9
7.D
58.8
88.5
+1.6
61.6
+4.6
88.0
+1.1
61.5
+4.5
In Figure 5, we compare a CNN trained without (left) and with (right) our prramttttttii (expansion factor d 4) on CTFAR100 00. The CNN consists of convolutional layers and fully connected layers; the architecture details are in Appendix E. We overlay the dynamics of the singular values of the Gram matrix throughout training. The spectral distribution is normalized by the largest singular value and are sorted in descending order (AAA) 2 ( for i < 1 s min(m, n). We observe that both the individual effective weights and the Gram matrix of the over- parameterized model is biased towards low-rank weights. Unlike the original, the majority of the singular values of the peeepaarameeer model are close to zero. When we take a closer look at the weights of the model, both the original and linearly wver-parameterized models first exhibit rank contracting behavior throughout training, and then the rank starts to increase again to the best of our knowledge, this is an unexpected training behavior in larger models that are not explained in prior works, possibly because the isometric, balanced initialization, and infinitesimal assumptions made in prior theoretical works do not hold in practice (visualized Appendix D).
Table 1: Over-parameterizzaionn ablations: A nonlin- ear CNN with 4 convolution and 2 linear layers trained CI PARIO 0 and C CIFAR 00 with various degrees of linear preepearamttiizzaionn...
7

---

Under review as a conference paper at ICLR 2022
ImageNet over-param
We further quantify the gain in performance from linear over- over-paramtterization in Table The learning rate is tuned per configuration, and we report the best test accuracy throughout the training. We try various configurations and find an expansion factor of 4 to be the sweet spot, with gain of +6.3 for CIFAR1 00 and +228 for CIFAR10, O. The opti- mal expansion factor depends on the depth of the original network, and in general, we observe consistent improvement for over-paramtterizing models with < 20 layers on image classification.
architecture
original
gain
AlexNet (2x)
57.3
59.1
+1,8
ResNet10 (2x)
62.8
63.7
+0,9
parammteeettiiiooonnnn
ResNet18 (2x)
67.3
67.7
+0.4
Table 2: ImageNet: We show on existing architectures that linear over-parameteriiation can improve generaliza- tion performance. The benefit plateaus off when using deeper models. We did not see noticeable improvement starting from 34sNet
We scale up our experiments to ImageNet, a large-scale dataset consisting of 1.3 million images with 1000 classes, and show that our findings hold in practical settings. For these experiments, we use standardized architectures: AlexNet (Krizhevsky et al. (2012)) which consists 8-layers, and ResNet10 / ResNet18 (He et al. (2016)) which consists of 10 and 18 layers, respectively. If our prior observations hold true, we would expect the gain in performance from over- peee-parameterization to be reduced for deeper models. This is, in fact, what we observed in Table 2, with moderate gains in AlexNet and less for Re and even less for ResNet18. In fact, starting from ResNet34, we observe linearly weerpparamtterized models perform worse than the original. These experiments support our claim that adding too many layers can over-penalize the model.
To find out whether explicit regularizers can approximate the advantages of over- parameterization, we directly compare the per- formance in Table 3 on CIFAR. These regular- izers include popular & and & norm-based reg- ularizers and commonly-used pseudo-measures of rank. These pseudo-men measures rank, such as effective rank and nuclear norm, require one to compute the singular value decompo- sition, which is computationally infeasible large-scale models. Although we found explicit rank regularizers to help, observed over- parameterization to out-perform models trained with explicit regularizers. Moreover, we found that combining norm- corm-based regularizers with over-parameterizaaton further improves perfor- mance. This discrepancy between implicit and explicit regularization may stem from the fact that over-parameterizzaton receives a combined effect of both gradient descent's implicit bias and model parameterization's inductive bias. Therefore, one may need to jointly consider both biases to approximate its effect as an explicit regularizer correctly. Another reason could that regu- larizers are inherently different than over weer-praamtterizatonn (Arora et al. (2018)). For example, model trained with a regularizer will have a non-zero gradient, even at zero training loss, while the over-parameterized model will not.
CIFARIO
CIFAR100
regularization
accuracy
gain
accuracy
gain
none (haseline)
86.9
$7.0
low-rank initialization
-0.1
57.2
+0,2
Eg nöri
"87.2
A1.
513)
4 norm
87.4
+0.5
60.0
+3.0
nuclaar norm
87.0
+0.3
58.7
FCr
effective rank
86.9
+0.0
$7.2
+0.2
stable rank
87.6
+0.9
58.3
+1.3
frobenius norm
87.0
+2.2
over-parim (2R)
18.1
+2.2
612
F4.2"
over-param (2x) +
89.6
+2.7
61.1
+4.1
over-param (2x) +
89.7
+2.8
63.3
+6.3
Table 3: Explicit regularizers: Comparison of models trained with various regularizers. Over preemaarmtttriitooonn consistently outperforms explicit regularizers.
DISCUSSION
We conclude by discussing several implications of our work:
Parameterization matters One of the main ingredients in any machine learning algorithm the choice of hypothesis space: what is the set of functions under consideration for fitting the data? Although this critical choice, how the hypothesis space is also parameterized matters. Even if two models span the same hypothesis space, the way we parameterize the hypothesis space can ultimately determine which solution the model will converge to - recent work has shown that networks with the better neural reparameterizations can find more effective solutions (Hoyer et al. (2019)). The automation of finding the right parameterization also has relationship to neural architecture search (Zoph& & Le (2017)), but architecture search typically conflates the search for better hypothesis spaces with the search for better parameterizations given hypothesis space. In this work, we have explored just one way of reparameterizing neural nets stacking linear layers which does not
8

---

Under review as a conference paper at ICLR 2022
change the hypothesis space, but many other options exist (see Figure 7 and short extension to residual networks Appendix H). Understanding the biases induced by these reparameterizations may yield benefits in model analysis and architectural designs.
Deep parammtteettiinooonnn vs. regularization We have argued that depth acts as an inductive bias toward simple (low-rank) solutions because the relative volume of simple solutions in deeper parameter space is larger, extending prior arguments by Valle-Perez et al. (2019). This effect is qualitatively different than explicit regularization in that it does not change the objective being minimized. Occam's 's razor states that the best solution is the simplest that fits the data (as has been formalized in various notions of optimal inference (Solomonoff (1964))). Too strong an explicit regularizer can prefer simple solutions that do not fit the data, which is incorrect when the true solution is, in fact, complex. Picking a parameterization in which simple solutions are more prevalent but not enforced is an alternative way to approximate the idea of Occam's 's razor. Our experiments, which show that explicit rank regularization underperforms compared to deep ver-parraeeterizaionn, suggest that this implicit regularization may have advantages.
6 RELATED WORKS
Linear networks Linear networks have been used in lieu of non-linear networks for analyzing the generalization capabilities of deep networks. These networks have been widely used for analyzing learning dynamics (Saxe et al. (2014)) and forming generalization bounds (Advani et al. (2020)).
Notable work from Arora et al. (2018) shows that aver-parameteriizaton induces training acceleration which cannot be approximated by an explicit regularizer. Furthermore, Gunasekar et al. (2017) shows that linear models with gradient descent converge to a minimum nuclear norm solution on matrix factorization. More recently. Li et al. (2020) demonstrated gradient descent acts as a greedy rank minimizer in matrix factorization, and Bartlett et al. (2020; 2021) argues that gradient descent in werrpaaraeetrrize models lead to benign overfitting. Although mainly used for simplifying theory, Bell-Kligles et al. (2019) demonstrate the practical applications of deep linear networks.
Low-rank bias Deep linear neural networks have been known to be biased towards low-rank solutions. One of the most widely studied regime is on matrix factorization with gradient descent under isometric assumptions (Tu et al. (2016); Ma et al. (2018); Li et al. (2018)), and further studied on least-squares (Gidel et al. (2019)). Arora et al. (2019a) showed that matrix factorization tends to low nuclear-norm solutions with singular values decaying faster in deeper networks. Note that these work focuses on why gradient descent finds low-rank solutions. Pennington et al. (2018) showed that the input-outpur Jacobian's 's spectral distribution is determined by depth. For non-linear networks, understanding the biases has been mostly empirical, with the common theme that over- parameterization of depth or width improves generalization (Neyshabur et al. (2015); Nichani et al. (2020); Golubeva et al. (2021); Hestness et al. (2017); Kaplan et al. (2020)). These aforementioned theories have also been adopted for auto-encoding Jing et al. (2020) and model compression, Guo et al. (2020). More recently, Pezeshki et al. (2020) have observed that SGD learns to capture statistically dominant features which leads to learning low-rank solutions, and Baratin et al. (2021) observed that the alignment of the features act as an implicit regularizer during training.
Simplicity bias Recent work has indicated that gradient descent in linear models find max margin solutions Soudry et al. (2018); Nacson et al. (2019); Gunasekar et al. (2018). Separately, in the perspective of algorithmic information theory, Valle-Perez et al. (2019) demonstrated that deep networks' parameter space maps to low-complexity functions. Furthermore, Nakkiran et al. (2019b) and Arpit et al. (2017) have shown that networks learn in stages of increasing complexity. Whether these aspects of simplicity bias are desireable has been studied by Shah et al. (2020).
Complexity measures A growing number of work has found matrix norm to not be a good measure for characterizing neural networks. Shah et al. (2018) shows that minimum norm solution is not guaranteed to generalize well. These findings are echoed by Razin & Cohen (2020), which demonstrates that implicit regularization cannot be characterized by norms and proposes rank as an alternative measure.
Infinite-widie networks Analyzing the spectral properties of deep networks has also been studied under infinite width neural networks. Aitchison et al. (2021) have observed that deep kernel processes with fixed, non-learned kernels exhibit a lower-rank structure, where the kernel follows power-law structure with depth. Yang & Salman (2019) shows that NTKs also exhibit this simplicity bias. For Gaussian processes, Aitchison (2020); Zavatone- Veth et al. (2021) further demonstrates that the rank of the "output Gram matrix" is restricted to the dimensionality of the output space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of general- ization error in neural networks, Neural Networks, 132:428-4444 2020.
Laurence Aitchison. Why bigger is not always better: finite and infinite neural networks. In International Conference on Machine Learning, pp. 156-164. PMLR, 2020.
Laurence Aitchison, Adam Yang, and Sebastian W Ober. Deep kernel processes. In International Conference on Machine Learning, pp. 130-140. PMLR, 2021.
Gernot Akemann, Jesper R Ipsen, and Mario Kieburg. Products of rectangular random matrices: singular values and progressive scattering. Physical Review E, (055),052188, 2013a.
Gernot Akemann, Mario Kieburg, and Lu Wei. Singular value correlation functions for products of wishart random matrices. Journal of Physics A: Mathematical and Theoretical, 46(27):275205, 2013b.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In ICML, 2018.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019a.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019b.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242. PMLR, 2017.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pp. 2269-2277. PMLR, 2021.
Peter L. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117744):30033-3007) 2020.
Peter I, Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. arXiv preprint arXiv:2 103.09/77, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. arXiv preprint 1812: 11118, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learnin practice and the classical bari-ver trade-off. Proceedings of the National Acudemy of Sciences, 116632):1844-11555 2019.
Sefi Bell- Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 2019.
7dzistaw Burda, Andrzej Jarosz, Giacomo Livan, Maciej Nowak, and Artur Swiech. Figenvalues and singular values of products of rectangular gaussian random matrices. Physical Review E, 82 (6):0611 14, 2010.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /929, 2020.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp. 1675-1685. PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2022
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.0205,, 2018.
Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pp. 907-940. PMLR, 2016.
S.H. Friedberg, A.J. Insel, and L.E. Spence. Linear Algebra. Featured Titles for Linear Algebra (Advanced) Series. Pearson Education, 2003. ISBN 9780130084514, URL https://bbooo google com/ books wit
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In Advances Neural Information Processing Systems, pp. 3202-3211, 2019.
Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. Are wider nets better given the same number of parameters? In International Conference on Learning Representations, 2021. 20211
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations, 2015.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization matrix factorization. In Advances in Neural Information Processing Systems, pp. 6151-6159, 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp. 2018
Shuxuan Guo, Jose M Alvarez, and Mathieu Salzmann. Expandnets: Linear over-parameeerivaatoon to train compact convolutional networks. In Advances in Neural Information Processing Systems, 2020.
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). 1xXiv:6006.00 08415, 2016.
arXiv preprint
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.004009 2017.
Stephan Hoyer, Jascha Sohl-Dicksteii and Sam Greydanus. Neural reparameterizzation improves structural optimization. arXiv preprint arXiv: 10XX::1999004240, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International conference on machine learning, pp. 448-456. PMLR, 2015.
Li Jing, Jure Zbontar, et al. Implicit rank-minimiznng autoencoder. In Advances in Neural Information Processing Systems, volume 33, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint a0Xiv:2001.088.0 2020.
11

---

Under review as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv.org/ abs / 212.. 6980.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pp. 2-47. PMLR, 2018.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09833 2020.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming. 45(1):503-577 -28. 1989.
Cong Ma, Kaizheng Wang. Yuejie Chi, and Yuxin Chen. Implicit regularization in nonconvex
statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion. In International Conference on Machine Learning, pp. 3345-3354, PMLR, 2018.
RV Mises and Hilda Pollaczek-Geiringgg Praktische verfahren der gleichungsauflösung. ZAMM- Journal of Applied Mathematics and Mechanics/ZZitschriff für Angewandte Mathematik und Mechanik, 9(1):58-77, 1929.
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. -4288. PMLR, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv: wxXiv:192.00229) 2019a.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. arXiv preprint arXiv: 1604, 2019b.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of conver- gence (1/k* 2). In Doklady an ussr, volume 269, pp. 543-547, 1983.
Thorsten Neuschel. rlanche formulae for average characteristic polynomials of products of ginibre random matrices and the fuss-catalan distribution. Random Matrices: Theory and Applications, (001))1450003, 2014.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In International conference on machine learning, 2015.
Eshaan Nichani, Adityanarayanan Radhakrishnan, and Caroline Uhler. Do deeper convolutional networks perform better? arXiv preprint arXiv:2010.096111 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peeformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and
12

---

Under review as a conference paper at ICLR 2022
R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, In Advances in neural information processing systems, 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 1924-12221 1932. PMLR, 2018.
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. arXiv preprint arXiv:201 1,09666,, 2020.
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In Advances neural information processing systems, 2020.
Lior Rokach and Oded Maimon. Clustering methods. In Data mining and knowledge discovery handbook, pp.
Springer, 2005.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007
15th European Signal Processing Conference, pp. 606-610. IEEE, 2007.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
Abraham Savitzky and Marcel JE Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry, 36(8):1627-163,, 1964.
Andrew M Saxe, James I. Mcclelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning deep linear neural network. In In International Conference on Learning Representations. Citeseer, 2014.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint arXi::2006.07710 10, 2020.
Vatsal Shah, Anastasios Kyrillidis, and Sujay Sanghavi. Minimum norm solutions do not always generalize well for over-parameterized problems. stat, volume 1050, pp. 16, 2018.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7(1):1-22, 1964.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2872. 2018
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning, pp. 964-973. PMLR, 2016.
Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-functie map is biased towards simple functions, In International Conference on Learning Representations, 2019.
13

---

Under review as a conference paper at ICLR 2022
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Xiaoxia Wu, Simon S Du, and Rachel Ward. Global convergence of adaptive gradient methods for an over-parameterized neural network. arXiv preprint arXiv:1902.077 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Greg Yang and Hadi Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv: /907. 10599, 2019.
Jacob A Z Abdulkadir Canatar, Ben Ruben, and Cengiz Pehlevan. Asymptotics of representation learning in finite bayesian neural networks. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over- parametereee neural networks. In Advances in Neural Information Processing Systems, pp. 8082-8093, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
14

---

Under review as a conference paper at ICLR 2022
APPENDIX
A RANDOM MATRIX THEORY IN FINITE MODELS
normalznd singullo maguiorrr voluee
082112
tormalized singularr volios
0.8
0.
0.6
probebility
0.6
06
0.6
0.4
02
0.0
0.0
200
value (o)
singular values
singular
(a) Thecretical Pennington et al. (2017); Neuschel (2014)
(b) Empirical W E 3232x32
(c) Empirical W 256
Figure Theoretical and empirical singular-value distributions: show that even on finite matrices, singular-valuc distribution matches that of the theorctical distribution. This implies that deeper finite-width linear neural networks should have lower effective rank in practice.
Random matrix theory makes an infinitely large matrix assumption to derive deterministic spectral distribution of random matrices. This assumption equivalent to making an infinite width neural network assumption. In Figure 6, we show that the empirical singular-value distribution closely follows that the theoretical distribution. Even when using a very small weight matrix of size W E 332x332 and more on larger weight matrices W I2 256x256 the singular values sharpen when increasing the number of layers reaffirming our Theorem and our conjecture.
15

---

Under review as a conference paper at ICLR 2022
B EXPANDING A NON-LINEAR NETWORK
F
deep non-linear neural network with layers is param-
eterized by set of weights W [WI,... W..... WiJ. The output of the j-th layer is defined as Φ = (oi-1),, for some non-lineat function 1b and input feature Φ--1. The initial feature map is the input do x, and the output the final feature map y = (r. We can expand a model by depth by expanding all linear layers, i.e. redefining Jw; + ww o Sw3 j & (1,...,)) 1). We illustrate this in Figure 7. We describe this operation for fully connected and convolutional layers.
Fell
Reparameterize
Lawer
Linser
Linear
Figure Linear reparameterization: For a model F, we can reparameterize any linear layer to another functionally equivalent layer (shown in the box below). In this work depth.
Fully-connected layer A fully-connected layer is pa- rameterized by weight W mm One can over- parameterize W series of linear operators defined I W,, For example, when d 2, W + W2W1, where W2 Rmxh and W1 E Rhxn for some hidden dimension h. The variable h is referred to as the width of the expansion and be arbitrarily chosen. In exper- iments, we choose h = n unless stated otherwise. Note that h < min(m, n) would result in rank bottleneck and explicitly reduce the underlying rank of the network.
mainly explore reparameterization Batch norm and any other running-statieo statistics driven normalization layers are linear only at test time.
Convolutional layer A convolutional layer is parameterized by weight W m=xxxxxk; where m and n are the output and input channels, respectively. and &: is the dimensionality of the convolution kernel. For convenience, we over-pore ever-parameterize by adding
convolution operations. Wa Wa-1 W1. where Wa E R.mxhxix,, Wa-1, .... W2 E IR xxxxxxxx and W1 E Rhxnxkkk Analogous to the fully-connected layer, we choose = n to avoid rank bottleneck.
The work by Golubeva et al. (2021) explores the impact of width h. Similar to their findings, we observed using the larger expansion width to slightly improve performance. We use h 2n our ImageNet experiments.
16

---

Under review as a conference paper at ICLR 2022
COMPARISONS OF RANK MEASURES AND KERNEL DISTANCE FUNCTIONS C
Depth
Depth
Depth
threho dinkk
effccive ra%k
Manihe ratic
traning epoch
traniigg epoch
training spoch
training epach
Figure 8: Comparing rank-measures: Comparison between various pseudo-metrics of rank when varying the number of layers. The threshold is
0.01 for threshold rank.
The rank of matrix which defines the number of independent basis - in practice can often be sub-optimal measure. For deep learning, fluctuations in stochastic gradient descent and numerical imprecision easily introduce noise that causes matrix to be full-rank. In addition, simply counting the number of non-zero singular values may not indicate what we care about in practice: the relative impact of the i-th basis compared the j-th basis. In typical image classification setup, we observed that the norm of the matrix often increases during training. This is highlighted by the nuclear norm in Figure 8. Coupled with numerical imprecisions, we found that the weights of the matrix are often always full rank.
rank measure closest the true definition of rank would be thresholded-rank, where the smallest singular values are thresholded after normalization (re-weighting the singular values based on relative contribution). However, thresholded rank is very sensitive to the threshold value one chooses (shown below); hence we used effective rank to avoid this issue.
Definition (Effective rank). Roy & Vetterli (2007)
For any matrix A E mR the effective rank is defined as the Shunnon entropy of the normalized singular values:
min(n.m)
P(A) =
Σ
5y log(ō.),
2=1
where 8, = = E, J are the normalized singular values such that Σ, o. = 1. It follows that p(A) < rank(A). This measure is also known as the spectral entropy.
The effective rank has been previously used surrogate measure for measuring the rank of neural network weights (Arora et al. (2019a)). We now state other various metrics that have been used as pseudo-measure of matrix rank. One obvious alternative is use the original definition of rank after normalization:
Definition 3 (Threshold rank). For any matrix A E Rmxxn, the threshold rank R-Rank is the count of non-small singular values after normalization:
min(n.m)
R-Rak((A)
Σ
16.
1=1
where 1 is the indicator function, and T E (0,1) is the threshold value. ō; are the normalized singular values defined above.
It is worth noting that not normalizing the singular values results in the numerical definition of rank. As stated before, the threshold rank depends largely on the threshold value and therefore a drastically different scalar representation of rank can emerge. Potentially, better usage of threshold rank is to measure the AUC when varying the threshold.
Related to the definition of the threshold rank, stable rank operates the normalized squared-singular values:
17

---

Under review as a conference paper at ICLR 2022
Train test loss
Effective rank
Thresholded rank
Threshold r- 0.00
Threshold a015
Threshold 0.01
High task rank
Elective gnii
Ronk
Depth
0.00
Threshold
Trreshold
Low Medium rank
LoG0
Threshold 0.03
Threshold
Threshold
Loss
Eflecdive
Farkkc
Depth
Figure 9: Least-sqaures ablation: Least-squares experiment using both effective-rank and thresholded rank measure. We run the experiments on various ranks 30, 16, For thresholded rank, we use various threshold values of T = {0.001, 0.005, 0.01) } and show that it correlates well with effective rank. The thresholded rank has downside of being sensitive the threshold values, and one has to subjectively tune the suitable threshold, making it suhoptimal choice. The figure shows that depending on the rank of the task, the generalization performance depends on depth. When the task rank is high, shallower models perform better, and when task rank is low, deeper models perform better. This aligns with our observation that the model parameterization biases the hypothesis search space in neural networks even if the models are effectively the same and span the same set of functions.
Definition 4 (Stable rank). Vershynin (2018)
For any matrix, A Rmxx, the stable rank is defined as: ||A| ||A12
Σ o 22 moz
SRank(A)
Where o; are the singular values of A.
Stable-rank provides the benefit of being efficient approximate via the power iteration Mises & (1929). In general, stable-rank is good proxy for measuring the rank of the matrix and has been used in prior works such as (Nichani al. (2020)). This not necessarily true when the singular values have long tail distribution, which under-emphasizes the small singular values un proprpportiooaellyy due to the squared-operator. We observed that the largest singular values often get over exaggerated in neural networks and hence we often found that SRrank converges to values close to 1, making insightful observations impractical.
Pollaczek-Geirine
Lastly, the nuclear norm has been considered as the de facto measure of rank for the task of matrix factorization/completiion with low nuclear-norm indicating that the matrix is low-rank:
Definition (Nuclear norm). For any matrix A E gmxa the nuclear norm operator is defined as:
min(n..))
= -allAAr
Σ a,(A)
Where are the singular values of A.
Nuclear norm, however, has obvious flaws of being an-normalized measure. The nuclear norm is dictated by the magnitude of the singular values and not the ratios. Therefore, the nuclear norm can be made arbitrarily large small without changing the output distribution.
18

---

Under review as a conference paper at ICLR 2022
Train test loss
Linear kernel
Casine kernel
Correlation kemee
Iileeiive rank
Effective rank
Effectivv rank
-
Depth
Figure 10: Kernel ablation: We ablate our least-squares experiments by using various kemel distance functions. Cosine kernels normalized version of linear kernels, and pearseon-correlation kernels are another way of normalizing linear kernls.. We can see that kernels show the same behavior.
The comparisons these metrics are illustrated Figure 8 where effective rank has the closest behavior to that of the thresholded rank. The metrics are computed on the end-to-end weights throughout the training. We use linear over-parameterized models with various depths on least- squares.
In Figure 9, we repeat our least-squares experiments from our main paper using thresholded rank with various threshold values {0.001, 0.005, 0.01). We show that the effective rank indeed correlates well with the thresholded rank. As stated above, we observe that the rank drastically changes depending on the threshold value. We also run the same experiment varying task-ranks of 30, 16, and 4. Although all models span the same set of functions (same effective weight dimensionality). the resulting generalization performance differs depending on the depth of the model. In a high task-rank setting, the generalization error increases with depth, while generalization error decreases with depth low rank-rank setting. This indicates the parameterization of the model determines the hypothesis space the model explores during training, which aligns with our conjecture and our observations. This further highlighted in medium and low task-rank settings, where all models reach zero-training error, yet the test-loss differs.
Kernel distance functions In our work we used cosine kernels to construct the Gram matrices. Cosine kernels are normalized linear kernels and found it to produce cleaner results. Cosine kernels has been commonly used as distance function to measure similarity between features (Zhang al. (2018), 8)). We further show in Figure 10 that Gram matrices constructed with kernel distance functions such as linear kernels and correlation kernels also exhibit the low-rank simplicity bias.
19

---

Under review as a conference paper at ICLR 2022
D SINGULAR VALUE DYNAMICS OF WEIGHTS
Unnormalized singular values
Normalized singular values
Original
Over-parameterieed (4x)
Original
Overparameterried (4x)
Conv1
indion Aesnna
Conv2
Conv3
Convd
FC1
FC2
Figure 11: Dynamics per layer: The singular values of the individual weights during CIFARICO training. On the left we have the unnormalized singular values, and on the right the distributions are scaled by the largest singular values. We uniformly subsample 24 singular values for the visualization. The cross sections are provided to help visualize the distribution at that specific epoch. The individual lines track the singular values a, over time.
In Figure 11, we visualize the singular values of the individual weights when training on CIFAR1 CO image classification for the first 120 epochs. The cross-sections indicate the singular value distribution at that specific epochs. For the parametteeete model, the effective rank is computed on the effective weight. On the left, we plot the unnormalized singular values and observe that the norm of the singular values increases throughout training for all layers except for the last classification layer in the over-paraan wee-pparameteiiee model. When we normalized the distribution by the largest singular value oo (right), we observed that the distribution becomes sharper early in training but does not change much throughout.
20

---

Under review as a conference paper at ICLR 2022
1.0
Normalized magnitude
Initial distribution
Singular values first decreuses
Singular values slowly increases
[poch
(Epoch
[Epoch 180)
Original
0.4
0.2
0.0
Over-parameten (4x)
100
200
300
400
500
100
200
300
400
100
200
300
400
1.0
Normalized magninude
Initial distribution
Singular values first decroases
Singular values slowly increases
0.8
(Epoch
(Epoch 8)
(Epoch 180)
0.6
500
100
200
300
400
500
100
200
300
400
500
Singular value indicies
Singular value indicies
Singular value indicies
Figure 12: Dynamics overlay: We overlay the singular values of the Conv4 weights. We observed that the effective rank first rapidly decreases early on in the training and then bounces back up slowly throughout the rest.
To get a better sense of how the distribution evolves over time by overlaying the distribution on top of each other. In Figure 12, overlay the distribution top of each other for Conv4 weights and observed that the effective-rank first decays rapidly and then slightly increases throughout the rest of the training. This dynamical behavior, our knowledge, and could highlight the dissonance between the assumptions made in theory do not fully describe behaviors observed in practice.
explained in prior theoretical works
21

---

Under review as a conference paper at ICLR 2022
E TRAINING DETAILS AND MODEL ARCHITECTURE
All models for image classification are trained using PyTorch Paszke et al. (2019) with RTX 2080Ti GPUs. We use stochastic gradient descent with momentum of 0.9. For CIFAR experiments, the initial learning rate is individually tuned (0.02 for most cases), and we train the model for 180 epochs. We use step learning rate scheduler at epoch 90 and 150, decreasing the learning rate by factor of 10 each step. For all the models, we use random-horizonte www--hoooozzoifiii and random-resize-crop for data augmentation. The training details for ImageNet can be found in https: / thub. com/ /ttppttttccc//cttoooo caccc/exaaee /examp I es / /bllbbllbbb.ooo/coooo/oooo ollommma ma ster/ / mmagenete When lin- early over-paramettrrzing our models, we bound the variance of the weights using Kaiming initializa- tion He et al. (2016), scaled Normal distribution. This allows to have the same output variance, regardless the number of layers we pacameeraeterii our models by. We found this to be critical for stabilizing our training. We also found it important to re-tune the weight decay for larger models on ImageNet. The architecture used for the CIFAR experiments is:
CIFAR architecture
RGB image E R33233233
Convolution 3 - 64, MaxPool, ReLU Convolution 64 + 128, MaxPool, ReLU Convolution 128 > 256, MaxPool, ReLU
Convolution 256 + 512, ReLU
GlobalMaxPool
Fully-Connected 512 + 256, ReLU
Fully-Connected 256 -> num classes
We tuned the learning rate per model as deeper models (8x expansion or more) become sensitive to the initial learning rate. This was critical for the least-squares experiments but not so much for CIFAR and ImageNet experiments (since we used up to 8x expansion). The one hyper-parameter that we found that needed tuning was the weight decay in ImageNet classification. A typical 2x or 4x expansion does not require much tuning at all. The learning rate scheduler was originally tuned to the baseline and was held fixed. The learning rate decay for basclines with explicit regularizers was tuned.
22

---

Under review as a conference paper at ICLR 2022
F DIFFERENTIAL EFFECTIVE RANK
To analyze the effective rank as a function of the number of layers, we define a differential variant of the effective rank. This formulation allows us use the fact that the eigen/singular-valee spectrum assumes probability distribution in the asymptotic case.
Definition 6 (Differential effective rank), For any matrix A E RM as min(m, n) + oc the singular values assume probability distribution p(o)). Then, we define the differential effective rank as:
((A) -
J. log( (p(o)do
where p(o) is the singular value density function and ap(a))o is the normalization constant.
G PROOF OF THEOREM
To prove Theorem 1, we leverage the findings from random matrix theory, where the singular values assume probability density function. Specifically, use the density function corresponding to the singular values the matrix W composed of the product L individual matrices W WL,,.. Wi. where the components of the matrices W1 to WL drawn i.i.d from Gaussian. Characterizing such density function is, in general intractable, or otherwise very difficult. However, in the asymptotic case where dim(W) - and W is square, the density function admits the following concise closed-form (Eq. 13 of Pennington et al. (2017) derived from Neuschel (2014)):
2 (in)(() sin sin 2 π sin L- ((L+ 1)
sin sin + 1)Ø)
p(a(o))
o - sin( sin sin (Lv)
(5)
where denotes singular values (parameterized by ø E [0, E41 and p denotes the probability density function of for [0, Tmax]. and d L L- 1) The parametric probability density function spans the whole singular value spectrum when sweeping the variable ø.
We interested in computing the effective rank of W. Using the above density function, write it in the form:
p(W) = -- mmax σ ( p(o)
(6)
c
We now write this integral in terms of & as the integration variable, such that we can leverage the density function in 5g 5. Using the change of variable, we have:
LL = -- Lt ((d) log o(0) (- do.
(-0)
where o') d (Ø). Note that the integral limits [0, maax] respectively translate" into 0] on Φ. Computing the inner probability term, we get:
-p(a (0) 2mt (1 + 2 L(L 1) cos(2s) (L + (+-1)coso c2LL) L cos(((1 + L)e) ese (LØ).
(Lo)
; L) decreases in T, if and only if de < 0. Since integration and differentiation are w.r.t. different variables, they commute; we can first compute the derivative of the integrand w.r.t. L and then integrate w.r.t. á and show that the result is negative.
As pWW L) is differentiable in I., p(W:
ZI
The integrand can be expressed as:
( log a - (p hu² log(lg
note that the direction of integration needs flip (by multiplying by -1) to account for flip of the upper and lower limits.
23

---

Under review as a conference paper at ICLR 2022
where the variables h, and are defined as:
[1+ 1)(1 cos(26)) (1 L) cos(2Lo) + L cos (2 +)6)
4
csc(Le)
(T. 1)4+1 (sc( sin (Lo) sin ++ + 1)6).
(10)
Now it is straightforward to differentiate the integrand and show that it has the following form:
d dL ho log(u) v(h(2 + log(u))ou' + 2 log(u) (uf + +2ww))
(11)
2/u
We should now integrate the above form w.r.t. & from 0 to I+I and examine the sign of the integrated form. The Eqn. 11 is positive for smaller & and negative for larger ø. Furthermore, the area under the positive region is smaller than the negative one; hence, the total value of the integral is negative. This completes the proof that the singular values decrease as a function of the number of layers.
The proof here considers the asymptotic case when dim(W) the probability distribution of the singular values. Although we do not provide proof for the finite case, results demonstrate that it holds empirically in practice (see Figure 1).
This limit case allowed us to
24

---

Under review as a conference paper at ICLR 2022
H EXTENSION TO RESIDUAL CONNECTIONS
This work concentrates our analysis on depth and its role in both linear and non-linear networks. Yet, the ingredients that make up what we know as state-of- the-art models today are more than just depth. From cardinality Xie et al. (2017) to normalization loffe Szegedy (2015) and residual connections He et (2016), numerous facets of parameterization have become a fundamental recipe for a successful model (see Figure 7). Of these, residual connections have the closest relevance to our work.
3.5
3.25
feed foward netchroont
media Qurch-oorr
2.75
2.25
1.78
10
15
20
25
depth
Figure 13: Residual connections: The effective rank of linear models trained with and without residual connection on low-rank eaut-squares problem. Contrary to feed-forward networks, resid- ual networks maintains the effective rank of the weights even when adding more layers. Residual networks without nommaliommaiiztionn suffer from unstable output variance after 16 layers.
What is it about residual connections that allow the model to scale arbitrarily in depth? while vanilla feed-forward networks cannot? One possibility is that beyond a certain depth, the rank of the solution space reduces much that good solutions longer exist. In other words, the implicit rank-rogularization of depth may take priority over the fit training data. Residual connections are essentially "skip connec- tions" that can be expressed I is the identity matrix (Dirac tensor for convolutions). There are two interpretations of what these connections do: one is that identity preservation prevents the rank-collapse of solution space. The other interpretation is that residual connections reduce the effective depth - the number of linear operators from the input to the output (c.g., ResNet50 and ResNet101 have the same effective depth), which prevents rank-collapse of the solution space. Results in Figure 13 confirm this intuition. ResNets, unlike linear networks, do not exhibit a monotonic rank contracting behavior and the effective rank plateaus after 8 layers, regardless of using Patch-normalization preliminary experiments on least-squares using linear residual networks indicate that the effective rank of the solution space is also bounded by the number of layers in the shortest and longest path from the inputs to the outputs. A thorough study on the relationship between residual connections and rank left for future work.
- W where
not. Furthermore,
25

---

Under review as a conference paper at ICLR 2022
I LEAST-SQUARES LEARNING DYNAMICS
The learning dynamics of a linear network change when over-parameterized. Here, we derive the effective update rule least-squares using linear neural networks to provide motivation why they have differing update dynamics. For a single-layer linear network parameterized by W, without bias, the update rule is:
wi++) + -(()) voo ((W(() ,2,
(12)
W( 7 wio -(u- - w))
(13)
= ( 7WW(9 "0.." - yx²"
(14)
Where ry is the learning rate. Similarly, the update rule the two-layer network y = War W2W10 can be written as:
W((+1)] w/O) ((WW))(WW))) -yx") Wh(t1) - W/O -nW!! xn - yx yxx- (W)(1)
(15)
(16)
(17)
Using short hand notation for VL W:( xxT yx., we can compute the effective update rule for the two-layer network:
w((+1) ((( (((++)(((t))))
(18)
first order (1)
second order O(1)²)
W(()) (W v VLO + (t) ( w(()) (
(19)
W(t) (P3V + VVC(pT
(20)
Where p( w(th w:(OT are the preconditioning matrices. The higher order terms can be ignored if the step-size is chosen sufficiently small.
(General case) For a linear network with d-layer expansion, the update for layer S S d is:
weights >
original gradicnt
weights <
(((+1) +- w(()) (w(l) w( (W(t)r"" yx") (w:2) W(
141)
(21)
Denoting W W, ... W We for j > i, the effective update rule for the end-to-end matrix is:
W(t+1)
II W ((+1)
II leied
(W; mw (OT (0T
(22)
d:tt
leied
((t))
Σ
W WT
di++
W + +..
+..
(23)
Ieied
~ ((t))
Σ Isied
W Was-
WT da+1
(C)
W
W1-11
(24)
original gradient
left precondition
right precondition
The update rule for the general case has much more complicated interaction of variables. For the edge i and = p the left and right preconditioning matrix is an identity matrix respectively.
26

---

Under review as a conference paper at ICLR 2022
J RANK-LANDSCAPE
We visualize the effective rank landscape the effective weights in Figure 14 and Gram matrices in Figure 15. We use single and two-layer linear networks for effective-rank landscape. We use two-layer, and four-layer ReLU networks for the Gram matrix and are constructed from 128 randomly sampled input data. For both methods, all the weights are sampled from the same distribution. The landscape is constructed by moving along random directions u, v. We observe that over wee--paameteriiee linear and non-linear models almost always exhibit a lower-rank landscape than their shallower counterparts.
maghitude effectivv rank p(W,)
(M)
1.5
7o 2t
Co
de magnitudk
magnitude of
25 magnitude of
single-layer
two-layer
Figure 14: Rank landscape: The landscape of the effective rank p of a linear function W. parameterized either by single-layer network (W, W') two-layer linear network (W. W2W1). The visualization illustrates simplicity bias depth. where the two-layer model has relatively more parameter volume mapping to lower rank Wa. Both models are initialized to the same end-to ind-to-end weights W. at the origin. Motivated by Goodfellow et al. (2015), the landscapes are generated using 2 random parameter directions v to compute f(a, 9) p(W + u- 8-2) for the layer model and (a, 3) p((W2 + + B-22) (W1 + +a --v)) for the two-layer model (u [wi, u2]. [V1, v2)).
effecttvve rank p(k)
WecttWo rank 0000
A3
magniitude
35
do magnitade
65 magnitude of
magnitude of
25
55
two-layer
four-layer
Figure 15: Kernel rank landscape: The landscape of the effective rank p computed on the kernels constructed from random features.
27

---

Under review as a conference paper at ICLR 2022
K
GRAM MATRIX AND NON-LINEARITIES
linear
tanh
relu
leaky relu
selu
gelu
siren
2005
rolu
2
2.1
Ficar
tarh
effective rank
2.0
1.9
depth
1.8
1.7
1.6
1.5
1.4
16
depth
Figure 16: Gram matrices of networks: Gram matrices of neural networks trained with various non- linearities and depth. The Gram matrix computed using the cosine-dine distance the features of the test-set training loss. Increasing number of layers decreases non-linear activation functions. Gram matrix hierarchically clustered (Rokach Maimon (2005)) for visualization. We observe the emergence of block structures in the Gram matrix as we increase the numher of layers, indicating that the embeddings become lower rank with depth.
effective rank the Gram matrix on variety of
In Figure 16, we further visualize the learned Gram matrices when varying the depth of the model. The Gram matrices trained with various non-linear activation functions also emit the same low-rank simplicity bias. These actívation functions include standard functions such as ReLU and Tanh as well as recently popularized non-linear functions such as GeLU (Hendrycks & Gimpel (2016)), and the sinusoidal activation function from SIREN (Sitzmann et al. (2020)). By hierarchically clustering Rokach & Maimon (2005) these Kernels, we can directly observe the emergence of block structures in the Gram matrices increase the number of layers, implying that the embeddings become lower rank with depth.
28

---

also the Under review as a conference paper at ICLR 2022
L
RELATIONSHIP BETWEEN WEIGHT AND EMBEDDINGS
Rank relation kernel and weight: Each point represents randomly drawn net- work. For each network, compute the rank the ef- fective weight linear kernel. The kernel is constructed from the MNIST dataset. The rank of the kernels weights have linear rela- tionship.
Figure 17:
effective rank of kemel
14
effective rank of weight
We show that there is almost one-to-one relationship between the effective rank of the weights and the effective rank of the Gram matrices in deep linear models. The figure plots this relationship for random deep linear networks applied to random subsets of the MNIST dataset. Moreover, it becomes apparent that the number of layers dictates the rank of the embedding as well as the weights.
29


Reviews:
[{"review_id": "w8ki5NdMOEy", "reviewer": "Reviewer_Jihb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well-written and I enjoyed reading it.\n- It starts with a conjecture and then validates that through experiments.\n- Using the effective rank as defined in Def. 1 is smart and it is continuous.\n- I liked it very much that the authors investigate that whether the bias is due to initialization or the training procedure itself.\n- This paper also does a good job at proving an overview of previous related work.\n\nWeaknesses:\n- I found most of the results to be discovered in related work in one form or another. As acknowledged by the authors, in several works by Arora et al. (2018 a,b; 2019 a,b) as well as Saxe et al. (2014) such a bias has been noted before.\n    - I appreciate it if the authors can pinpoint the exact differences and help us identify the significance of this work.\n\n- Figure 5 which seems to present a very important result of the paper is very hard to read.\n    - The paper says: *train and test losses are overlayed on top of the distribution*. I believe it is not the loss but the accuracy. But still it is not clear to me whether the scale of both plots are the same or not. It would be better to add another set of ticks on the right side of the plots.\n    - The paper says: *Both original and linearly over-parameterized models first exhibit rank contracting behavior throughout training, and then the rank starts to increase again - to the best of our knowledge, this is an unexpected training behavior in larger models that are not explained in prior works.* \\\nI cannot read any decrease or increase from the distribution heatmap. It would be best to show the effective rank through out the training.\n    - I also do not know what the y-axis is. Is it the indices of the singular values? It is somewhat misleading that those with larger \"negative log magnitude\" are smaller.\n\n- Finally, the std must be reported in tables 2 and 3 so that allows the reader to know how significant the gains are.\n\nOther minor remarks:\n- In Def. 1, it says \"Without loss of generality, we drop the exponentiation for convenience.\" Which exponentiation?\n- Throughout the paper, 'p' and 'q' are used to denote to size of dataset while 'n' and 'm' are dimensions. I found it somewhat unconventional.\n- On page 3, the paper says: *Since the dimensionality of the Gram-matrix depends on the dataset size, we can compare neural networks with different modeling capacities in the zero training error regime.* \\\nI found this a bit confusing. I believe the reason is that \"it does **not depend** on the number of parameter\".\n", "summary_of_the_paper": "This paper conduct several controlled experiments and arrive at the following results:\n- Deep networks (both linear and non-linear) are biased towards learning low-rank embeddings at initialization.\n- This low-rank bias exists even after the training is done regardless of the initialization or training algorithm.\n\nThis paper also proposes a new regularization method which is adding extra linear layers which does not increase the network's expressivity but biases the network more towards learning low-rank solutions.", "summary_of_the_review": "As mentioned above, in summary,\n\n(+) The paper is very well-written and easy to follow.\\\n(+) The finding are interesting and well supported by the experiments.\\\n(-) Most of the findings have been discovered before in related work. However, I might have missed something and am looking forward to hearing the authors opinion.\n(-) Figure 5 is not clear at all. However, I believe it can be fixed easily and I would like to encourage the authors to modify it.\n(+) Depth as a regularizer has been using by practitioners however. Nevertheless, increasing the depth by adding linear layers is still interesting.\n\nFinally, I would like to let the authors know that I am willing to increase my score if the authors could kindly answer the questions above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e4wnbDp5vU", "reviewer": "Reviewer_utsM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "By this point, we know quite alot about the biases towards low or high rank in finite and infinite neural networks, and the paper lacks alot of this nuance.\n\n1. A priori, we know that finite width networks do exhibit a lower-rank structure than the corresponding infinite-width network, due to the low-rank bias in the Wishart distribution [1] (Appendix Fig. 5).  But this isn't exactly a point that has been emphasised in past work, so I wouldn't necessarily regard it as novelty-destroying, especially in light of the random matrix theory.\n2. Infinite neural networks don't have a bias towards lower rank as they get deeper.  The kernel has a power-law structure which appears to remain consistent with depth [2,3].  \n3. Low-rank fixed kernels aren't particularly likely to improve performance (which is why the lack of low-rank kernels in deep infinite neural networks is a good thing).  If the kernel became low-rank, then it assigns high probability only to linear combinations of a few output functions.  If the true function lies outside this class, then we expect poor performance.  Of course, if the true function lies within this class, then we expect a performance boost, but typically that only happens after training.\n4. While trained networks indeed appear to have a low-rank bias [3], the bias is really towards the \"output Gram matrix\" formed by the product of the one-hot labels [3,4].  Typically, (for small label sets) this Gram matrix will be low rank.  But it doesn't have to be, and it doesn't represent a generic bias towards low-rank.  This is much more accurately described as just representation learning [3].\n\nThus, conjecture 1 requires _alot_ of nuance.  In particular, work comparing finite and infinite networks would lead us to expect a simpler top-layer kernel for a single, fixed, random initialisation of the weights.  But obviously we don't just use a single, fixed, random initialisation of the weights.  If you averaged over many different random initialisations, you'd end up with something much more like the infinite kernel, which isn't low-rank.  And if you trained, you'd drag the top-layer kernel towards the output Gram matrix, which is typically low-rank, but doesn't have to be.\n\nAs a final note, if the bias towards the low-rank output Gram matrix is baked into the objective, it is unsurprising that it occurs with any choice of optimiser.\n\nI'd be willing to increase my score if these results are built in to the discussion.\n\n[1] Aitchison, Yang, Ober.  Deep kernel processes.  ICML (2021).\n\n[2] Yang, Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10599\n\n[3] Aitchison. Why bigger is not always better: on finite and infinite neural networks.  ICML (2020).\n\n[4] Zavatone-Veth, Canatar, and Pehlevan. \"Asymptotics of representation learning in finite Bayesian neural networks.\" arXiv preprint arXiv:2106.00651 (2021).", "summary_of_the_paper": "The paper argues that modern deep neural networks are effective in part because they incorporate a bias towards low-rank Gram matrices (where the Gram matrix is the product of the features with themselves).", "summary_of_the_review": "Missing the understanding of these issues built up in the infinite neural network literature, and in the literature relating infinite to finite networks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8Akc-LKgE-K", "reviewer": "Reviewer_cVHG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the claims are clear. However I'm not sure that the main claim is not trivial, after all, when matrices are multiplied, rank can either decrease or stay the same. Whether a low effective rank of the embeddings helps generalization is not very clear to me, the observed benefits of the proposed overparametrization might be just due to increased capacity. Notably, most modern deep architectures use skip connections, and as shown in appendix C, skip connections correspond to identity matrices which preserve the rank.\n\nThe connection to Solomonoff's theory of universal induction that the authors make seems far-fetched: in the Solomonoff ensemble shorter programs consistent with the data are preferred, while in deep learning deeper neural networks, which are longer programs, are preferred, which suggests that neural networks operate far from the idealized regime of Solomonoff induction.\n\n", "summary_of_the_paper": "The paper explores the effect of depth on a continuous estimate of rank (effective rank) of the data embeddings computed by feed-forward neural networks.\n\nOn artificial datasets, the paper shows that at initialization (with common uniform or normal weight distributions), the effective rank of both linear and non-linear MLPs decreases with depth, and this is preserved after training. The results does not extend to network architectures with skip-connections such as ResNets.\n\nThe authors claim that this low-rank bias of the embeddings help generalization on natural datasets, and show that a reparametrization of common architectures where the linear layers are replaced by products of linear layers improves test accuracy on CIFAR-10 and CIFAR-100.\n", "summary_of_the_review": "The paper presents somewhat interesting observations on the algebraic properties of feed-forward neural networks, however their relevance to the generalization power of such models is not entirely clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2cT9CZ8brr", "reviewer": "Reviewer_Yhy3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n- Analyzing the properties of parameterization in neural networks is an important aspect. (But, often this is sidelined in literature by overly focussing on, e.g.,  'implicit bias' of gradient descent)\n- The experiments and ablations are, in general, well thought of (as they would be natural questions) and properly executed.\n- The paper is nicely written and the figures are very elegant (so much so that I am inclined to ask for the code used to make them :))\n- Linear over-parameterization is a nice strategy to boost performance. This also has the advantage that at inference one can just collapse the involved weight matrices into a single equivalent weight matrix, thereby leading to compact networks for deployment.\n\n**Cons:**\n- Continuous rank measure: (a) I understand the motivation behind using a continuous measure --- the usual, discrete rank is sensitive to measure --- but if careful (i.e., using high precision) rank can still be useful. To make claims about low-rank solutions, it is quite important that the authors actually illustrate what goes on with the true rank, or even say numerical rank (which thresholds at some level). Then, once having presented the case there, you can move on to continuous approximations. (b) Next, the choice of the particular continuous measure is not justified, except for the statement that \"our observations are consistent with alternative measures\" --- but I did not find where this is shown (the appendix only contains how other measures of rank behave during training). (c) Finally, in light of the above arguments, I would say it is rather sloppy to say \"we use rank and effective rank interchangeably\" --- without first having justified it. Also, given this is just one additional word, it will help to make your point precise, which otherwise only serves to confuse what is actually being used. \n\n- Link to generalization: Figure 4 is the quintessential figure behind my criticism. We can clearly observe that the generalization gap first decreases and then starts increasing with increasing depth. In contrast, effective rank has an unaffected monotonic decrease with increasing depth. It is also not the case that the task rank is high, but is lower than that of input or output. This raises the question how strongly the claim that 'lower effective rank, better generalization' is justified? A similar fact is also evident in the Tables 1-3 --- the expansion factor needs to be tuned to a sweet spot. All this only weakens the presented claims. (Note: I am not against linear over-parameterization as I also mention in the 'Pros' above, over here the concern is the claim about the benefit of low-rank inductive bias of depth.) As a result, there are likely some other missing factors in 'true story' concerning the benefit and inductive bias of depth, and thus the authors should actually reduce their claims in this regard. \n\n- Empirical aspects: (a) It is not a great practice to report the best test accuracy during the training (I know a lot of other works do it, but one is ideally not supposed to see the test performance except for a handful of occasions). (b) Plus, it is not clear how critical LR tuning is to reap the benefits of linear over-parameterization. What happens when used with default settings? (c) Do you also tune the LR decay for explicit regularizers in Table 3? (d) In all the comparisons, are the different networks at a comparable loss value? Since it is that what makes things comparable, not necessarily the same number of epochs.  \n\n- Choice of the kernel: The cosine kernel seems quite harmless at first glance, however, it carries within it the normalization by the norms of the feature maps. If the scales are anything except for constant factors, then the value of the effective rank used here can also vary. This raises the question if simply the linear kernel is used, would the empirical observations about low-effective rank with increasing depth remain the same?\n\n- Related work: Overall, this is covered nicely in the paper, but I do still have some minor comments. (i) Does the cited work by Arora et al show that solutions reached with increasing depth also have lower nuclear norm? I would guess this work only claims about increasing convergence rate. Can you clarify? (ii) The recent work of [1] also makes related claims the neural networks are inductively biased towards low-rank solutions, but with low-rank in the sense of the Hessian. Also, see therein the benefit of true, discrete rank to obtain precise theoretical statements and formulas. \n\n[1] Singh et. al. 2021, Analytic Insights into Structure and Rank of Neural Network Hessian Maps, https://arxiv.org/abs/2106.16225\n\n- Miscellaneous: \n(i) Another thing which I would be curious to see is that if you compare the effective rank of the gram matrix across layers from the same network itself. What kind of trend does it have? If the low-rank claims with depth are considered to be true, I would expect it to hold within the layers of the network itself? (empirically, since the authors use the same width for all layers, the comparison should not be a problem)\n(ii) Fig 5, \"rank training dynamics\": I believe it would make the figure complete if the authors add the plot for the effective rank as well. In any case, I guess the rank dynamics similar to that in Figure 9?\n\n", "summary_of_the_paper": "This work proposes the hypothesis that deeper networks are inductively biased to find low-rank solutions (i.e., in the sense of gram matrix of features being low-rank) and presents empirical evidence to support the same. The authors thoroughly show that this inductive bias is a property of the parameterization of neural networks, and so remains resilient to changes in optimizer or initialization distribution. Based on this, they suggest linear over-parameterization as a way to further bias the network to low-rank solutions, which is then linked to increasing generalization performance. \n\n", "summary_of_the_review": "The paper investigates an interesting hypothesis concerning the parameterization of neural networks, in that increasing depth leads to solutions that have lower effective rank. This is backed by various experiments and ablations, and is in itself convincing. However, a significant part of the paper focuses on consequently linking this to generalization --- but which remains questionable. This is because their empirical observations and theoretical statement (in the deep linear network case) indicate that increasing depth results in a monotonic decrease in the effective rank, but in practice, they find this holds up to some moderate depth and beyond which the performance deteriorates. This weakens their connection of low-rank gram matrix solutions to generalization. But not only that, there are some critical yet unexplained choices/aspects in their framework, such as: whether effective rank is the right measure of rank, what happens with the actual (discrete) rank, choice of the kernel, etc. which raises additional questions and further serves to dilute the claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experimentally investigate the inductive bias of deep neural networks tending to produce low rank embeddings of data, which is important to explain why over-parameterized DNN can generalize. In particular, the paper empirically finds that deeper networks are more likely to produce lower rank embedding through thorough numerical experiments with different network architectures, hyperparameters and so on. The authors also proposed a linear over-parameterization technique to induce low-rank bias and empirically justifies its effectiveness. 

Overall, this paper is well written, and the numerical experiments are carefully executed. However, the main drawback of the paper is that the low-rank inductive bias itself is a well known phenomenon and this paper gives a kind of additional confirmation to it. I acknowledge that there are several differences from existing papers, but overall we see rather limited insight from the results. Indeed, some of existing studies gave theoretical analyses to understand "why it happens", but this paper does not give a sufficiently novel insight to reveal the reason. 

To summarize the decision, this paper lacks deeper insight compared with existing work although the authors did a good job to execute through experiments. Therefore, it is a bit below the acceptance threshold.
Decision: Reject