Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies a new training strategy for finding a low-rank sub-network by solving matrix differential equations and tools of matrix decompositions. The idea is novel and the implementation is convincing. However, if this method can be applied to much large scale and frequently used networks instead of current test settings, this work can be definitely more compelling.
strengths_and_weaknesses: 
### Originality
The idea of using matrix differential equations to capture the underlying low-rank weights is novel and attractive. Researchers in many domains have wondered about efficient methods to train the network weights under the factorization of SVDs. So this work may have a broad audience in the machine learning domain. The formulation and implementation of this method also seem to have a rather solid background in numerical analysis and pdes.

### Quality
The basic idea is good, however, the theoretical analysis is a bit limited, and the empirical study is too narrow. 
1. The authors should give clear theorems about the convergence and convergence analysis of the proposed algorithm, without which I cannot fully judge the soundness of this work. I am eager to such results in the response of the authors.
2. The authors should also give theorems on what $S_k$ learns during the training, and what is the geometry or intuitive meaning of the final decomposition $USV$ since $S_k$ is not a diagonal matrix,  and $USV$ is not an SVD.
3. Is there any theoretical results that support the universal approximation ability of networks decomposed by this algorithm?
4. Is there any theory analysis on the relationship between hyper-parameters of this method and the test accuracy or approximation error?
5. I am also wondering why the author did not test this method in ResNet, Transformer, or deep generative networks? This seems no barrier to doing so.

### Clarity
The clarity needs many improvements. The authors should summarize the major conclusions in the context into theorems, which can significantly improve the readability and soundness of this work. The background knowledge of this method is also missed, especially the minimum preliminary knowledge for readers to understand the `DLRA' and `KLS'.

### Siginificance
The paper lacks clear theorems to support it theoretical soundness and significance. The numerical results are also a bit narrow for practical usage. There are also some previous methods[3] for leveraging SVD in the training of deep networks, the authors may consider comparing the proposed method with them.

### My suggestions:
1. In Line 105-107, I strongly recommend the authors cite some papers on this topic to enhance the argument. The most relevant two papers could be [1][2]. [1] gives both theoretical and empirical evidence of low rank weight matrix in over-parametric deep networks with tools from recent advances in random matrix theory. Using the theory of dynamic systems and Lyapunov exponents, [2] further describe how chain rules, i.e., large amounts of matrix multiplications can lead to low-rank results and network gradient, which exactly support the motivation here. Besides, [3],[4], and [5] also give support for low-rank networks from different aspects.
2. In line 122-123, I think it would be better to say `$W_k(\tau)\in\mathcal{M}_{r_k}$' for some $\tau>0$. Most networks are initialized with Gaussians, thus initially they have full rank, but there is evidence that training will eventually converge to some low-rank status.

[1]Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning. Charles H. Martin Michael W. Mahoney
[2] Rank Diminishing in Deep Neural Networks.  Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, Zheng-Jun Zha
[3] Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li

questions: 1. Eq 6, what is `min!' ?
2. The description to Fig. 2 is not clear. The timings for what? A gradient descent step, a feed forward pass for a batch of samples?
limitations: The main limitations are two:
1. Lack theorems or enough theoretical analysis to rigorously support the soundness of the algorithm. Readers may not be familiar with the classical results in related backgrounds. Some very important theory results, like convergence and error analysis, should be clearly stated.

2. I do not understand why the authors do not apply this method in large networks like ResNet50 on ImageNet.

Apart from these two points, I think this is good work. If the authors can successfully address my concerns on these two issues, I will raise my rating.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a dynamic adaptive low-rank training algorithm for deep neural networks that is able to find efficient low-rank subnetworks during training time. It achieves improvement in both memory and computation efficiency. Experimental results on MNIST show that the proposed approach outperforms some other pruning and low-rank methods.
strengths_and_weaknesses: Strengths
* I really appreciate this work for exploring low-rank training. I think the overall novelty is above the line.
* This paper has detailed equation derivation and algorithms.
* This paper is well organized with sufficient materials to support the idea, especially the sections of computational cost and limitations analysis.

Weaknesses
* Is there a typo at line 298 (LNRR)?
* Why only evaluate on MNIST? How about the performance on more complicated networks and datasets?
* Is there any other related works that research on low-rank manifold and can be compared with in this paper? It would be better if more such comparison is included.
questions: No other specific questions. Please refer to weaknesses above.
limitations: Yes. The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a Dynamic Low Rank Training Scheme (DLRT) algorithm, which aims at attaining low-rank networks during training directly for faster computation efficiency and lower memory footprint. Low-rank training is formulated via gradient workflow, then via incorporating the dynamics of the model weights, the proposed DLRT algorithms achieve adjusting the layer ranks dynamically (via explicit optimization). The complexity and efficiency of the DLRT algorithms are discussed. Empirical simulation is also provided to justify the effectiveness and efficiency of the proposed DLRT algorithm.
strengths_and_weaknesses: Strengths:
Overall the proposed idea makes sense. The DLRT algorithm is a viable algorithm for attaining low-rank neural networks (presumably for faster inference after training). The simulated experiments support the major claims to some extent.

Weaknesses:
My major concern is that it is not clear if the proposed method can serve as a useful method in practice. The paper is not a theory paper and comes with no theoretical analysis, thus I would justify it mainly from the application perspective. My detailed concerns are summarized below:
1. It is not clear if the DLRT algorithm is designed for faster training or faster inference. For faster training, given the evidence in the current draft, I am not convinced that DLRT is even faster than normal training, e.g., DLRT requires conducting extra computation (in lines 8-21 of Algorithm 1, which requires conducting a full SVD per iteration). Not to mention the other baselines, e.g., [1-2], which claim much faster end-to-end running time compared to vanilla training.
2. Memory saving is also claimed in the paper, but I do not believe the low-rank-based method can save memory costs largely during training. Low-rank networks have fewer (or much fewer number of parameters), but the bottleneck of memory cost comes from the layer activations, not model weights [3]. And low-rank neural networks can not reduce the activation size (during training). During inference, low-rank networks can efficiently reduce the memory cost as no activations are required to be saved. The authors should make the claims more clear.
3. The paper does not compare against powerful enough baselines outside the low-rank domain. For instance, quantization+sparsification [5] and learned structured pruning [6] (as low-rank methods are versions of structured pruning after all). 
4. The experimental results are too weak, DLRT generates either larger networks than LRNN or with worse accuracy. Then why people should use DLRT in practice given that LRNN method can also adaptively adjust ranks?
5. It has been shown in [2], that the low-rank method won't scale to large tasks, e.g., ImageNet without the proposed warm-up and hybrid network tricks. How does DLRT perform on the ImageNet dataset or even BERT language modeling tasks?
6. It is claimed that DLRT automatically selects ranks, however, it seems the final ranks of the layer are controlled by another hyper-parameter \tau, which is still required to be tuned in practice.

[1] https://arxiv.org/abs/2105.01029  
[2] https://arxiv.org/abs/2103.03936  
[3] https://arxiv.org/abs/1904.10631  
[4] https://ieeexplore.ieee.org/document/9157223  
[5] https://arxiv.org/abs/1510.00149  
[6] https://openreview.net/forum?id=BJxsrgStvr  
questions: Please refer to the "Weaknesses" part for answering my major questions. In summary:

1. Is DLRT designed for efficient training or efficient inference, or both?
2. How does DLRT compare against DeepCompression and EarlyBird Ticket?
3. Why can't DLRT outperform LRNN? If not, why should people use DLRT in practice?
4. DLRT scale to larger networks, e.g., ResNet-50, EfficientNet, ViT, BERT on larger tasks, e.g., ImageNet, WikiText, GLUE benchmark?
limitations: As discussed thoroughly in the section "Strengths And Weaknesses" and "Questions", the paper is not a theory paper. But in practice, the proposed DLRT method has a few flaws, e.g., computation efficiency, and training-time memory cost is not clear. The empirical results of this paper are weak, the experiments are only conducted on one toy dataset (even though, DLRT is off compared to at least one of the baseline methods).
review_title: null

Meta-review: metareview: This paper proposes a Dynamic Low Rank Training Scheme (DLRT) to optimize the neural network weight matrix imposing low rank constraint and hence yields better computational and memory efficiency. The obtained solution becomes low rank and thus it can realize factorization of the weights by its nature. The optimization procedure can be equipped with an adaptive rank selection scheme. The proposed method is mainly justified by numerical experiments. 

The optimization method is basically derived from the gradient flow along the low rank matrix manifold. This paper gives a rather theoretically solid optimization scheme.
The presentation is overall good. It is well organized and contents required to understand the contribution are appropriately presented. 

There are still some weakness. First, the paper would benefit from adding some more related topics and giving more discussions about connection to them. Second, it would enhance the paper if the additional numerical experiments on larger data and models (Imagenet/Resnet50) as presented in the rebuttal phase would be included. 

In summary, although there are some weakness, this paper gives a novel and solid methodology to obtain a low rank weight matrices. So, I recommend acceptance.  
On the other hand, I strongly recommend the authors to address the issues pointed out by the reviewers in the final version. 

confidence: Certain
Decision: Accept