The Low-Rank Simplicity Bias in Deep Networks

Abstract: Modern deep neural networks are highly over-parameterized compared to the data on which they are trained, yet they often generalize remarkably well. A flurry of recent work has asked: why do deep networks not overfit to their training data? In this work, we make a series of empirical observations that investigate the hypothesis that deeper networks are inductively biased to find solutions with lower rank embeddings. We conjecture that this bias exists because the volume of functions that maps to low-rank embedding increases with depth. We show empirically that our claim holds true on finite width linear and non-linear models and show that these are the solutions that generalize well. We then show that the low-rank simplicity bias exists even after training, using a wide variety of commonly used optimizers. We found this phenomenon to be resilient to initialization, hyper-parameters, and learning methods. We further demonstrate how linear over-parameterization of deep non-linear models can be used to induce low-rank bias, improving generalization performance without changing the effective model capacity. Practically, we demonstrate that simply linearly over-parameterizing standard models at training time can improve performance on image classification tasks, including ImageNet.

Reviews and author responses:

 Thank you for the feedback. Indeed we missed some prior work on infinite width networks and fixed, non-learned kernels; we appreciate the pointers and will add discussion of these works and the nuances they provide, especially regarding Conjecture 1. However, we think the reviewer has misinterpreted our main finding and contribution, which is a set of empirical results on finite trained networks which cannot be explained by a bias toward the “output Gram matrix”. 

The core of the issue is conveyed in the reviewer’s point #4. At first glance, it may seem unsurprising that trained deep networks converge to low-rank kernels -- of course, this should be true if the target labels are low-rank and we have trained the net to match the targets well! However, the point of our paper is to show that this is not the only reason trained networks exhibit a tendency toward low-rank embeddings. Rather, deep networks have an inductive bias -- distinct from the training objective and data -- toward low-rank embeddings. Our paper shows this in three distinct ways:  
1.  Randomly initialized, untrained networks are biased toward low-rank kernels (this point overlaps with prior work, which we will discuss more below)  
2. Even when two architectures parameterize the exact same hypothesis space, the deeper architecture tends to result in lower-rank kernels after training. This cannot be due to the deeper architecture searching over more expressive functions that better fit the output (because the hypothesis spaces are identical). Moreover, the training error is slightly higher for the deeper networks (but near zero for all networks) -- again, this rules out that the deeper networks fit the (low-rank) training data better.  
3. Even when the data is high-rank, the deeper the net, the lower the rank of the solution found after training (Figure 4). When the data is low-rank, deeper networks perform better on the test set than shallower networks even when they have reached the same zero training error (Appendix D Figure 9)

We constructed our experiments so as to tease apart the low-rank effect of pulling toward the training data and the low-rank biases from other sources. We did so by studying untrained networks (1), architectures of different depth but equal capacity (2), and training on high-rank data (3). In the revision, we will make clearer that this design intended to rule out the alternative explanation the reviewer brought up. Please let us know if you see any gaps in the logic that could be improved.

Given that our focus is on the analysis of trained, finite networks (as networks are used in actual practice), we believe the other three concerns the reviewer brought up are more tangential -- they deal with infinite width nets and nets with fixed (non-learned) kernels. Nonetheless, we respond to these concerns below:

**Concern #1:** We make no claims about the relative rank of finite versus infinite width nets. Can the reviewer please clarify which part of our paper motivated this concern?   

**Concern #2:** From our initial understanding, [2] seems to suggest the existence of a weaker version of the simplicity bias that depends on depth. [2] further shows that NTKs also have a sweet spot for depth, which is consistent with our findings for finite models. For [3], could you clarify where it implies the kernel has a power-law structure that is independent of depth?  

**Concern #3:** We agree with these points and find them to be consistent with the message in our paper: when the true function is low-rank (as is typically the case with natural data), the bias toward low-rank kernels is beneficial as training will tend to find a good fit that also matches the true structure of the data (and therefore generalizes well). When the true function is not low-rank, the bias will be detrimental, as we have shown in Appendix D, Figure 9, Table 1, Figure 4, Section 4 first paragraph.

We appreciate the references the reviewer provided us with and agree they add nuance to the background theory. We will add them to the paper and discuss related results on infinite-width nets and nets at initialization. We will use this discussion as an opportunity to clarify in the paper that this is not the regime that we are studying and that the prior theory does not directly predict our empirical findings. Although the suggested works operate on an infinite width regime and are somewhat tangential in terms of contributions, we have found these suggested works valuable as it expands the scope of our knowledge and help reaffirm that our findings are consistent. We hope to incorporate these discussions into our manuscript. We have begun iterating over the related works section in the updated pdf, but as we are not experts in deep kernel processes or Bayesian networks, please let us know if there are improvements to make on the related work section and we will update it appropriately.

This paper conduct several controlled experiments and arrive at the following results:
- Deep networks (both linear and non-linear) are biased towards learning low-rank embeddings at initialization.
- This low-rank bias exists even after the training is done regardless of the initialization or training algorithm.

This paper also proposes a new regularization method which is adding extra linear layers which does not increase the network's expressivity but biases the network more towards learning low-rank solutions. Strengths:
- The paper is very well-written and I enjoyed reading it.
- It starts with a conjecture and then validates that through experiments.
- Using the effective rank as defined in Def. 1 is smart and it is continuous.
- I liked it very much that the authors investigate that whether the bias is due to initialization or the training procedure itself.
- This paper also does a good job at proving an overview of previous related work.

Weaknesses:
- I found most of the results to be discovered in related work in one form or another. As acknowledged by the authors, in several works by Arora et al. (2018 a,b; 2019 a,b) as well as Saxe et al. (2014) such a bias has been noted before.
    - I appreciate it if the authors can pinpoint the exact differences and help us identify the significance of this work.

- Figure 5 which seems to present a very important result of the paper is very hard to read.
    - The paper says: *train and test losses are overlayed on top of the distribution*. I believe it is not the loss but the accuracy. But still it is not clear to me whether the scale of both plots are the same or not. It would be better to add another set of ticks on the right side of the plots.
    - The paper says: *Both original and linearly over-parameterized models first exhibit rank contracting behavior throughout training, and then the rank starts to increase again - to the best of our knowledge, this is an unexpected training behavior in larger models that are not explained in prior works.* \
I cannot read any decrease or increase from the distribution heatmap. It would be best to show the effective rank through out the training.
    - I also do not know what the y-axis is. Is it the indices of the singular values? It is somewhat misleading that those with larger "negative log magnitude" are smaller.

- Finally, the std must be reported in tables 2 and 3 so that allows the reader to know how significant the gains are.

Other minor remarks:
- In Def. 1, it says "Without loss of generality, we drop the exponentiation for convenience." Which exponentiation?
- Throughout the paper, 'p' and 'q' are used to denote to size of dataset while 'n' and 'm' are dimensions. I found it somewhat unconventional.
- On page 3, the paper says: *Since the dimensionality of the Gram-matrix depends on the dataset size, we can compare neural networks with different modeling capacities in the zero training error regime.* \
I found this a bit confusing. I believe the reason is that "it does **not depend** on the number of parameter".
 As mentioned above, in summary,

(+) The paper is very well-written and easy to follow.\
(+) The finding are interesting and well supported by the experiments.\
(-) Most of the findings have been discovered before in related work. However, I might have missed something and am looking forward to hearing the authors opinion.
(-) Figure 5 is not clear at all. However, I believe it can be fixed easily and I would like to encourage the authors to modify it.
(+) Depth as a regularizer has been using by practitioners however. Nevertheless, increasing the depth by adding linear layers is still interesting.

Finally, I would like to let the authors know that I am willing to increase my score if the authors could kindly answer the questions above.

 We thank the reviewer for the comments and suggestions. We agree that some of our choices in the paper could have been better motivated. We clarify any misunderstandings and have updated our manuscript accordingly. 

**Concern 1:** The use effective-rank is unjustified  
Effective rank was first popularized by Arora et al. [1] and has since been commonly used for analyzing the spectral properties of linear layers in neural nets ([2, 3, 4]).
The impractical use of true rank is further exemplified by the fact that in a typical image classification setup, where we observed that the norm of the matrix often increases during training [2] (see Figure 8 and Figure 11). Coupled with numerical imprecisions, we found that the weights of the matrix are often always full rank. The closest rank measure that would circumvent these issues is the use of threshold rank, where we threshold the smallest singular values after normalization (re-weighting the singular values based on their relative contribution). However, threshold rank is very sensitive to the threshold value chosen; hence we avoided this issue by using effective rank. We added a new figure in Appendix D, that repeats the least-squares experiments in Figure 4 using threshold rank with various threshold values and found these measures behave very similarly to effective rank. We agree that the explanation on the use of effective rank and its relation to prior work could be better elaborated and we hope that our new experiments and extended discussion in the Appendix clarify this issue. 

**Concern 2:** Over-parameterization does not always improve generalization  
There may have been a misunderstanding of our paper. In our work, we did *not* claim that over-parameterization will always improve generalization. Like *any regularizers*, over-regularizing your model would hurt performance. This is what we stated and observed in our experiments (see the first paragraph of Section 7).
The main contribution of our work was to highlight that existing theory on the implicit bias of gradient does not provide a full picture and that model parameterization has an important inductive bias that largely influences the generalization performance (discussed in Section 8).
We further highlight this in our new experiment in Appendix D figure 9, where over-parameterization only helps when the desired target task rank is low-rank.

**Concern 3:** Training details and parameter tuning  
It is a known fact that stable learning rates change as a function of depth for neural networks [6]. We tuned the learning rate per model as deeper models (8x expansion or more) become more sensitive to the initial learning rate. This was critical for the least-squares experiments but not so much for CIFAR and ImageNet experiments (since we used up to 8x expansion). The one hyper-parameter that we found that needed tuning was the weight decay in ImageNet classification. A typical 2x or 4x expansion does not require much tuning at all. The learning rate scheduler was originally tuned to the baseline and was held fixed. The learning rate decay for baselines with explicit regularizers was tuned. We wanted to avoid concerns that we may have trained the baseline models poorly. Our results highlight the best performance one could arrive at after tuning the learning rate. 

[continued on the next comment]

---
[1] Arora et al., Implicit Regularization in Deep Matrix Factorization, NeurIPS 2019.   
[2] Razin et al., Implicit regularization in deep learning may not be explainable by norms, Neurips 2020  
[3] Boyarski et al., Deep matrix factorization with spectral geometric regularization, MSML 2021  
[4] Baratin et al., Implicit regularization via neural feature alignment, PMLR 2021  
[5] Gunasekar et al., Implicit Regularization in Matrix Factorization, ITA 2018   
[6] Saxe et al., Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, ICLR 2014  


 **concern 1 continued**  
4. Arora et al. 2019b - *Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks*  
- Constructs a gram matrix for nonlinear networks and tracks the dynamics.  
- Provides a convergence rate characterization of nonlinear networks.  
- Shows that the sample complexity to fit the data is independent of the network size.  
- Further shows the family of functions a 2-layer ReLU network using gradient descent can approximate.  

This work mainly focuses on analyzing the observations of Zhang et al. (Understanding deep learning requires rethinking generalization) by answering the question of why models converge faster on meaningful labels, and what complexity measure is able to differentiate a model trained on random labels and meaningful labels. The work shows that the convergence rate depends on the eigen-vectors of the Gram matrix. This work studies the learning dynamics of 2-layer relu networks and not on the bias of parameterization. This work is complementary in explaining generalization properties but is tangential in terms of our findings.

**Concern 2**: Suggestions on improving Figure 5 is  
Yes, there was a typo and it should have said “accuracy”, thank you for pointing this out. They use the same scale as indicated by a single color bar. The y-axis is indices of the singular values, and the singular values are sorted from the largest to smallest (largest on the top, smallest on the bottom). We have updated the figure caption to clarify the concerns raised. To highlight how the intermediate singular values are evolving, we decided to use negative log magnitude. We wish to make this figure as clear as possible so if you have any further suggestions please let us know and will try our best to update it.  
We have added a separate visualization in Appendix D Figure 11 to show this phenomenon on individual weights. The dipping and increase in the effective rank and threshold rank is a minor yet noteworthy phenomenon, which can be better observed in Appendix D Figure 12 (newly added) and also Figure 9. 

**Concern 3:** Reason for omitting standard deviation for CIFAR classification  
In supervised classifications, the standard deviations are very small. All the experiments in our work have less than < 0.3% standard deviation. For the sake of making the tables readable, we have decided to omit the standard deviations in the tables. If requested, we will elaborate our choice for this reason in the manuscript or add them separately with the standard deviations in the appendix.   

**Concern 4:** minor comments and clarifications  
(1) The exponentiation is dropped from the original definition of effective rank is $e^{\rho(A)}$.   
(2) If you have suggestions and references on how to improve the dimensionality notation please let us know.  
(3) Our statement on gram-matrices implies that it does not depend on the number of parameters. The dimensionality of the gram-matrix depends on the number of samples used (the rank; however, is bounded by the bottleneck dimension).  

 We thank the reviewer for the helpful suggestions and we hope to incorporate these changes into our paper. Please let us know if you have any further suggestions

**Concern 1:** Differences to prior work (Arora et al. (2018; 2019 a,b), Saxe et al. (2014))  

The aforementioned prior works study the learning dynamics in over-parameterized models. To our knowledge, our work is the first to focus on the parameterization bias of deep networks for both linear and non-linear models. Our work highlights that this is not specific to gradient descent dynamics and further demonstrates how it can be used as a regularizer in deep non-linear models. We highlight how our work differs from the related works below. We apologize in advance if we have missed some of their contributions.  

1. Saxe et al. 2014 - *Exact solutions to the nonlinear dynamics of learning in deep linear neural network*  
- Provides analytical theory that explains nonlinear learning dynamics in linear networks. The types of nonlinear behaviors mentioned are (faster convergence, lower error, and better transfer performance)  
- The learning dynamics of the input-output map (similar to Gram matrices) are derived from gradient descent dynamics under specific assumptions such as infinitesimal learning rate, orthogonal input, and more.  
-  Demonstrates dynamical isometry can be achieved via orthogonal initialization.  
- Depending on the initial conditions, the training dynamics, such as the convergence rate, change.  
- Studies how training time scales with depth and also the learning rate.  
- Shows that initial conditions that preserve dynamic isometry allow the network to have depth-independent learning times.  

Saxe et al. focus on understanding the nonlinear learning dynamics of deep linear networks. Under specific conditions, they provide an exact gradient flow dynamics of a linear network. They further highlight how deep networks evolve over time, and how initial conditions affect depth independent learning. Although this work has served as pioneering work in deep learning theory, the work of Saxe et al. is tangential to our as it does not study the simplicity bias of model parameterization.

2. Arora et al. 2018 - *On the optimization of deep networks: Implicit acceleration by overparameterization*  
- Observe that by having extra scalar parameters in the linear network, one can rewrite the gradient descent dynamics with the one that is composed of past gradients (mimicking the momentum term that is often used in SGD).  
- Under the dynamical isometry assumption on the weights, they extend this theory to linear networks. Using gradient flow dynamics, over-parameterized models encourage movement along directions already taken by the optimizer.  
- They further demonstrate that an explicit regularizer cannot approximate the implicit bias of over-parameterization.  

Arora et al. demonstrate that deep linear models, under dynamical isometric assumptions, can accelerate training. This work does not study the simplicity bias of deep networks and only the bias of gradient descent for over-parameterized models. This work, however, has become a foundational work on understanding the bias of overparameterization.

3. Arora et al. 2019a - *Implicit regularization in deep matrix factorization*
- The observation is that deeper linear models can better solve low-rank matrix factorization problems using gradient descent.  
- Under isometric assumptions, provides a theory on how depth plays a role in gradient descent learning dynamics in linear networks.  
- Shows the dynamics of the singular values during training. Where the rate of decay in the singular values depends on the depth.

Arora et al. extend the conjecture of Gunasekar et al. (Implicit regularization in matrix factorization) that gradient descent in linear matrix factorization results in low nuclear norm solution. They demonstrate that this tendency is encouraged by depth. This work is specific to deep linear matrix factorization and is focused on the bias of gradient descent. The proposed theory sheds light on our observations but does not fully explain our observations. In our work, we show that both linear and non-linear networks are biased towards low-rank embeddings even at initialization (Figure 1, 2). We show that the models converge to low-rank embeddings on a variety of optimizers beyond gradient descent (Figure 3). We also show that the depth of the network can sometimes be detrimental when the task is high rank (Figure 4, Table 1-2, Figure 9).   

[continued on the next comment]

 
**Concern 4:** Are comparisons made at comparable loss value?   
Yes. Note that only the experiments for Figures 3-4 are important to have comparable losses. Our experiments assume the models have reached zero training error at convergence. This is stated in the paper (Section 3.2, and Figure 3). The other figures have the same modeling capacity and do not need to be compared at the same training error as long as they have been correctly tuned. To highlight this further, we have added an additional experiment in Appendix D Figure 9 (see medium rank, low rank), where we repeat the least-squares experiment with various task rank settings. The comparisons are made at zero training error but the generalization performances and converged effective rank of the weights changes with depth. 

**Concern 5:** Would the results be consistent with linear kernels?  
Originally we used linear kernels but found cosine kernels to produce less noisy results. Linear kernels work as well as cosine kernels and have been a common choice in measuring distances in the Kernel space (as noted in Section 2.3). We omitted a direct comparison in the initial draft but we have included a new experiment to highlight this in Appendix D Figure 10. 

**Concern 6:** Differences to Arora et al.
Arora et al. [1] study the implicit bias of depth in over-parameterized models. Contributions of their work include:  
- Extends the conjecture of Gunasekar et al. (Implicit regularization in matrix factorization) that gradient descent in linear matrix factorization results in low nuclear norm solution.  
- The observation is that deeper linear models can better solve low-rank matrix factorization problems using gradient descent.  
- Provides theory, under isometric assumptions, how depth plays a role in the learning dynamics of gradient descent in *linear* networks. “The dynamics promote solutions that have a few large singular values and many small ones, with a gap that is more extreme the deeper the matrix factorization is”.   

Our work differs in many ways. We highlight these differences below:  
- We extend the observation on simplicity bias to linear and non-linear finite networks. Our observations do not depend on any isometric assumptions, and we show that it holds even in practical problem setups.  
- We show that the parameterization bias exists regardless of training. That is, even with or without optimization, models are biased towards low effective rank mappings (Fig 1, 2).   
- We show that implicit bias exists beyond gradient descent, whereas prior theory only applied to models optimized with gradient descent (Fig 3).  
- Even at zero-training error, models with different depths but the *same* capacity will converge to different solutions, and ultimately exhibit different generalization properties (Fig 4, Fig 5, Tbl 1-2, Fig 10, Fig 11).  
-  In summary: while prior works have primarily studied the bias of over-parameterization under the context of gradient descent, our work focuses on bringing attention to another missing piece in understanding why over-parameterized models converge to low-rank solutions -- a phenomenon which is often credited to why deep networks generalize [1, 2, 5].

---
[1] Arora et al., Implicit Regularization in Deep Matrix Factorization, NeurIPS 2019.   
[2] Razin et al., Implicit regularization in deep learning may not be explainable by norms, Neurips 2020  
[3] Boyarski et al., Deep matrix factorization with spectral geometric regularization, MSML 2021  
[4] Baratin et al., Implicit regularization via neural feature alignment, PMLR 2021  
[5] Gunasekar et al., Implicit Regularization in Matrix Factorization, ITA 2018   
[6] Saxe et al., Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, ICLR 2014  

 We believe there has been a misunderstanding of the contributions in our work.    

**Concern 1:** “I'm not sure that the main claim is not trivial, after all, when matrices are multiplied, rank can either decrease or stay the same.”  
While this is true and well-known for linear models with discrete rank, our Theorem 1 is making a more subtle statement that the spectral distribution of the weights becomes sharper as models are made deeper --- the entropy of the spectral distribution decreases. In addition, our empirical findings are not directly predicted by the fact that multiplying matrices reduces rank: the networks also exhibit this effective rank reducing behavior at initialization and also at convergence. This has been a common interest within the community as it is currently conjectured to be why overparameterized models generalize.

**Concern 2:** “benefits of the proposed overparametrization might be just due to increased capacity”  
Throughout our paper, we discuss that even when the model does not have increased capacity, linearly over-parameterized models improve generalization (See Figure 4, Figure 5, Table 1, Table 2,  Figure 9). Further, even when all models achieve the same training error, model parameterization biases the hypothesis space the optimizer searches over. This ultimately results in different generalization properties (See Figure 9).  

**Concern 3:** “The connection to Solomonoff's theory of universal induction that the authors make seems far-fetched”
We stated Solomonoff's theory briefly in light of Occam’s razor to motivate why low-rank functions could lead to better generalization when the underlying input-output relationship is simple. This is not the main contribution of our work and was only mentioned briefly in Section 4 and Discussion (few lines total). However, we maintain that connected our work to Solomonoff’s theory is valid. The reviewer makes an incorrect statement: “deeper neural networks, which are longer programs, are preferred.” This statement confuses the description length of the hypothesis space with the description length of the learned function; the statement that short programs are more likely to generalize applies to the function, not the hypothesis space. The function computed by a deeper net may be a shorter program than the function computed by a shallower net.

This work proposes the hypothesis that deeper networks are inductively biased to find low-rank solutions (i.e., in the sense of gram matrix of features being low-rank) and presents empirical evidence to support the same. The authors thoroughly show that this inductive bias is a property of the parameterization of neural networks, and so remains resilient to changes in optimizer or initialization distribution. Based on this, they suggest linear over-parameterization as a way to further bias the network to low-rank solutions, which is then linked to increasing generalization performance. 

 **Pros:**
- Analyzing the properties of parameterization in neural networks is an important aspect. (But, often this is sidelined in literature by overly focussing on, e.g.,  'implicit bias' of gradient descent)
- The experiments and ablations are, in general, well thought of (as they would be natural questions) and properly executed.
- The paper is nicely written and the figures are very elegant (so much so that I am inclined to ask for the code used to make them :))
- Linear over-parameterization is a nice strategy to boost performance. This also has the advantage that at inference one can just collapse the involved weight matrices into a single equivalent weight matrix, thereby leading to compact networks for deployment.

**Cons:**
- Continuous rank measure: (a) I understand the motivation behind using a continuous measure --- the usual, discrete rank is sensitive to measure --- but if careful (i.e., using high precision) rank can still be useful. To make claims about low-rank solutions, it is quite important that the authors actually illustrate what goes on with the true rank, or even say numerical rank (which thresholds at some level). Then, once having presented the case there, you can move on to continuous approximations. (b) Next, the choice of the particular continuous measure is not justified, except for the statement that "our observations are consistent with alternative measures" --- but I did not find where this is shown (the appendix only contains how other measures of rank behave during training). (c) Finally, in light of the above arguments, I would say it is rather sloppy to say "we use rank and effective rank interchangeably" --- without first having justified it. Also, given this is just one additional word, it will help to make your point precise, which otherwise only serves to confuse what is actually being used. 

- Link to generalization: Figure 4 is the quintessential figure behind my criticism. We can clearly observe that the generalization gap first decreases and then starts increasing with increasing depth. In contrast, effective rank has an unaffected monotonic decrease with increasing depth. It is also not the case that the task rank is high, but is lower than that of input or output. This raises the question how strongly the claim that 'lower effective rank, better generalization' is justified? A similar fact is also evident in the Tables 1-3 --- the expansion factor needs to be tuned to a sweet spot. All this only weakens the presented claims. (Note: I am not against linear over-parameterization as I also mention in the 'Pros' above, over here the concern is the claim about the benefit of low-rank inductive bias of depth.) As a result, there are likely some other missing factors in 'true story' concerning the benefit and inductive bias of depth, and thus the authors should actually reduce their claims in this regard. 

- Empirical aspects: (a) It is not a great practice to report the best test accuracy during the training (I know a lot of other works do it, but one is ideally not supposed to see the test performance except for a handful of occasions). (b) Plus, it is not clear how critical LR tuning is to reap the benefits of linear over-parameterization. What happens when used with default settings? (c) Do you also tune the LR decay for explicit regularizers in Table 3? (d) In all the comparisons, are the different networks at a comparable loss value? Since it is that what makes things comparable, not necessarily the same number of epochs.  

- Choice of the kernel: The cosine kernel seems quite harmless at first glance, however, it carries within it the normalization by the norms of the feature maps. If the scales are anything except for constant factors, then the value of the effective rank used here can also vary. This raises the question if simply the linear kernel is used, would the empirical observations about low-effective rank with increasing depth remain the same?

- Related work: Overall, this is covered nicely in the paper, but I do still have some minor comments. (i) Does the cited work by Arora et al show that solutions reached with increasing depth also have lower nuclear norm? I would guess this work only claims about increasing convergence rate. Can you clarify? (ii) The recent work of [1] also makes related claims the neural networks are inductively biased towards low-rank solutions, but with low-rank in the sense of the Hessian. Also, see therein the benefit of true, discrete rank to obtain precise theoretical statements and formulas. 

[1] Singh et. al. 2021, Analytic Insights into Structure and Rank of Neural Network Hessian Maps, https://arxiv.org/abs/2106.16225

- Miscellaneous: 
(i) Another thing which I would be curious to see is that if you compare the effective rank of the gram matrix across layers from the same network itself. What kind of trend does it have? If the low-rank claims with depth are considered to be true, I would expect it to hold within the layers of the network itself? (empirically, since the authors use the same width for all layers, the comparison should not be a problem)
(ii) Fig 5, "rank training dynamics": I believe it would make the figure complete if the authors add the plot for the effective rank as well. In any case, I guess the rank dynamics similar to that in Figure 9?

 The paper investigates an interesting hypothesis concerning the parameterization of neural networks, in that increasing depth leads to solutions that have lower effective rank. This is backed by various experiments and ablations, and is in itself convincing. However, a significant part of the paper focuses on consequently linking this to generalization --- but which remains questionable. This is because their empirical observations and theoretical statement (in the deep linear network case) indicate that increasing depth results in a monotonic decrease in the effective rank, but in practice, they find this holds up to some moderate depth and beyond which the performance deteriorates. This weakens their connection of low-rank gram matrix solutions to generalization. But not only that, there are some critical yet unexplained choices/aspects in their framework, such as: whether effective rank is the right measure of rank, what happens with the actual (discrete) rank, choice of the kernel, etc. which raises additional questions and further serves to dilute the claims. 

The paper explores the effect of depth on a continuous estimate of rank (effective rank) of the data embeddings computed by feed-forward neural networks.

On artificial datasets, the paper shows that at initialization (with common uniform or normal weight distributions), the effective rank of both linear and non-linear MLPs decreases with depth, and this is preserved after training. The results does not extend to network architectures with skip-connections such as ResNets.

The authors claim that this low-rank bias of the embeddings help generalization on natural datasets, and show that a reparametrization of common architectures where the linear layers are replaced by products of linear layers improves test accuracy on CIFAR-10 and CIFAR-100.
 The paper is well written and the claims are clear. However I'm not sure that the main claim is not trivial, after all, when matrices are multiplied, rank can either decrease or stay the same. Whether a low effective rank of the embeddings helps generalization is not very clear to me, the observed benefits of the proposed overparametrization might be just due to increased capacity. Notably, most modern deep architectures use skip connections, and as shown in appendix C, skip connections correspond to identity matrices which preserve the rank.

The connection to Solomonoff's theory of universal induction that the authors make seems far-fetched: in the Solomonoff ensemble shorter programs consistent with the data are preferred, while in deep learning deeper neural networks, which are longer programs, are preferred, which suggests that neural networks operate far from the idealized regime of Solomonoff induction.

 The paper presents somewhat interesting observations on the algebraic properties of feed-forward neural networks, however their relevance to the generalization power of such models is not entirely clear.

The paper argues that modern deep neural networks are effective in part because they incorporate a bias towards low-rank Gram matrices (where the Gram matrix is the product of the features with themselves). By this point, we know quite alot about the biases towards low or high rank in finite and infinite neural networks, and the paper lacks alot of this nuance.

1. A priori, we know that finite width networks do exhibit a lower-rank structure than the corresponding infinite-width network, due to the low-rank bias in the Wishart distribution [1] (Appendix Fig. 5).  But this isn't exactly a point that has been emphasised in past work, so I wouldn't necessarily regard it as novelty-destroying, especially in light of the random matrix theory.
2. Infinite neural networks don't have a bias towards lower rank as they get deeper.  The kernel has a power-law structure which appears to remain consistent with depth [2,3].  
3. Low-rank fixed kernels aren't particularly likely to improve performance (which is why the lack of low-rank kernels in deep infinite neural networks is a good thing).  If the kernel became low-rank, then it assigns high probability only to linear combinations of a few output functions.  If the true function lies outside this class, then we expect poor performance.  Of course, if the true function lies within this class, then we expect a performance boost, but typically that only happens after training.
4. While trained networks indeed appear to have a low-rank bias [3], the bias is really towards the "output Gram matrix" formed by the product of the one-hot labels [3,4].  Typically, (for small label sets) this Gram matrix will be low rank.  But it doesn't have to be, and it doesn't represent a generic bias towards low-rank.  This is much more accurately described as just representation learning [3].

Thus, conjecture 1 requires _alot_ of nuance.  In particular, work comparing finite and infinite networks would lead us to expect a simpler top-layer kernel for a single, fixed, random initialisation of the weights.  But obviously we don't just use a single, fixed, random initialisation of the weights.  If you averaged over many different random initialisations, you'd end up with something much more like the infinite kernel, which isn't low-rank.  And if you trained, you'd drag the top-layer kernel towards the output Gram matrix, which is typically low-rank, but doesn't have to be.

As a final note, if the bias towards the low-rank output Gram matrix is baked into the objective, it is unsurprising that it occurs with any choice of optimiser.

I'd be willing to increase my score if these results are built in to the discussion.

[1] Aitchison, Yang, Ober.  Deep kernel processes.  ICML (2021).

[2] Yang, Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10599

[3] Aitchison. Why bigger is not always better: on finite and infinite neural networks.  ICML (2020).

[4] Zavatone-Veth, Canatar, and Pehlevan. "Asymptotics of representation learning in finite Bayesian neural networks." arXiv preprint arXiv:2106.00651 (2021). Missing the understanding of these issues built up in the infinite neural network literature, and in the literature relating infinite to finite networks.

Meta-review: This paper experimentally investigate the inductive bias of deep neural networks tending to produce low rank embeddings of data, which is important to explain why over-parameterized DNN can generalize. In particular, the paper empirically finds that deeper networks are more likely to produce lower rank embedding through thorough numerical experiments with different network architectures, hyperparameters and so on. The authors also proposed a linear over-parameterization technique to induce low-rank bias and empirically justifies its effectiveness. 

Overall, this paper is well written, and the numerical experiments are carefully executed. However, the main drawback of the paper is that the low-rank inductive bias itself is a well known phenomenon and this paper gives a kind of additional confirmation to it. I acknowledge that there are several differences from existing papers, but overall we see rather limited insight from the results. Indeed, some of existing studies gave theoretical analyses to understand "why it happens", but this paper does not give a sufficiently novel insight to reveal the reason. 

To summarize the decision, this paper lacks deeper insight compared with existing work although the authors did a good job to execute through experiments. Therefore, it is a bit below the acceptance threshold.

Decision: Reject