Title: Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture
Venue: NeurIPS 2025
Accepted: no
Keywords: matrix completion

implicit regularization

training dynamics

low rank
Abstract: Gradient descent for matrix factorization is known to exhibit an implicit bias toward approximately low-rank solutions. While existing theories often assume the boundedness of iterates, empirically the bias persists even with unbounded sequences. We thus hypothesize that implicit bias is driven by divergent dynamics markedly different from the convergent dynamics for data fitting. Using this perspective, we introduce a new factorization model: $X\approx UDV^\top$, where $U$ and $V$ are constrained within norm balls, while $D$ is a diagonal factor allowing the model to span the entire search space. Our experiments reveal that this model exhibits a strong implicit bias regardless of initialization and step size, yielding truly (rather than approximately) low-rank solutions. Furthermore, drawing parallels between matrix factorization and neural networks, we propose a novel neural network model featuring constrained layers and diagonal components. This model achieves strong performance across various regression and classification tasks while finding low-rank solutions, resulting in efficient and lightweight networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIntroduces a new UDV matrix factorization model $X = UDV^\\top$ (and a symmetric $X = UDU^\\top$ variant), inspired by the implicit bias towards low rank in matrix factorization literature. The norm of the $U$ and $V$ components are constrained and $D$ is diagonal. Experiments demonstrate that the UDV parameterization exhibits an implicit bias towards low rank for a wider range of learning rates and initialization sizes compared to the classical Burer-Monteiro factorization (BM; $X = UV^\\top$, without norm constraints). Furthermore, the utility of the UDV factorization is evaluated as a replacement for linear layers in existing non-linear neural network architectures. The corresponding layers in the modified neural networks are shown to be low rank after training, enabling low rank pruning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n---\n1. The paper is well-written and generally easy to follow. The experimental settings are described in sufficient detail, and the background motivating the design of the UDV factorization is clearly presented.\n\n2. While a formal analysis of the UDV (or UDU variant) is not provided, the theoretical insight section, along with the accompanying experiments in Appendix A.3 are useful for comprehending why a low rank bias is observed.\n\n3. The implicit low rank bias of BM factorization, or its deep variants where multiple unconstrained linear layers are multiplied, often heavily relies on a small initialization size. This creates a tradeoff between the low rank bias and convergence speed, since when the initialization is small the gradient (and higher-order derivatives for deeper factorizations) vanish. I find it interesting that the low rank bias in the UDV factorization does not require a small initialization, and so is not at odds with convergence speed.\n\n\nWeaknesses\n---\n1. As the paper does not include a formal treatment of the UDV factorization, the significance of its contributions relies primarily on empirical validation. However, I find the evaluation falls below standard for empirical works in the field. While there are quite a few experiments included, the considered datasets are either non-standard tabular datasets or MNIST. It is not clear why the authors chose specifically these datasets and did not consider more relevant settings. Even if compute is a major restriction (which does not seem to be the case since experiments were run on a A100 GPU according to Section 3.1.1), surely going beyond MNIST for classification should be possible.\n\n2. Furthermore, the neural network results do not show that the UDV factorization brings noticeable improvements in terms of resulting performance (Table 1). The low rank bias can still be considered as a positive due to its potential use for pruning. But, while the idea makes sense, its evaluation is again limited to simplistic settings. There is also no comparison to alternative neural network pruning and compression techniques.\n\nTo be clear, the critique above is not regarding the quantity of experiments (there are plenty, including the appendix), but rather on their quality — how decisive are they and do they substantiate the significance of the proposed UDV factorization. The paper lacks convincing evidence of the factorization being useful in practice. I believe it would have been much better off if many of the existing experiments (especially those in the appendix) would have been replaced with a few convincing experiments in more realistic settings.\n\nOverall, despite the limitations above, my assessment of the work is positive due to its clarity and the potential interest that the UDV factorization may attract, both from theorists and perhaps practitioners. Unfortunately, it is difficult for me to wholeheartedly recommend acceptance of a paper whose main contributions rely on experiments in such simplistic settings. Thus, the borderline accept rating.\n\nMore minor comments:\n---\n1. In Figure 1, the convergence rate of the UDU factorization is not impacted by the change in initialization scale. However, looking at the gradient updates (below Equation 5), they should shrink the smaller the initialization scale is. Why then is the convergence rate not affected? Is it because the initialization scale is not taken to be small enough?\n\n2. In lines 42-43, it is stated that the key insight behind this work is that the implicit bias in the classical BM factorization is driven by divergent dynamical behavior. However, the paper does not contain any evidence supporting this claim. Rather, it focuses on the newly proposed factorization. I strongly recommend rephrasing this sentence as it is misleading and can confuse readers.\n\n3. Similar to the comment above, I find the remark in the subsequent line 44 to be unclear and not substantiated in the paper. Why would “standard (convergent) dynamics” be associated necessarily just with data fitting?\n\n4. Typo in line 107: “here matrix completion” should probably be “here is matrix completion”.\n\n5. In Equation 6, what does it mean for $\\phi (x)$ to be approximately equal to $y$? It is not clear where this $y$ comes from or what is the purpose of this notation.\n\nQUESTIONS:\n--\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-06"}

comments: SUMMARY:
This paper tackles a matrix completion problem solved with projected gradient descent and extends the well-known BM factorization in the algorithm. Orthogonality constraints are replaced with Frobenius norm-constraints and a positive diagonal matrix that multiplies between the matrices (like SVD) is introduced. Extensive experiments along with fixed point analysis show that this extension results in low-rank solutions for a wide range of hyperparameters investigated in the literature. When used for training fully-connected layers of deep neural networks, this approach also facilitates pruning while not significantly impacting the prediction performance.

STRENGTHS AND WEAKNESSES:
The factorization approach for matrix completion is unique. Extensive experiments including variants, comparisons to weight decay and applications in computer vision and NLP frameworks strengthen the results. The paper is clearly written.

QUESTIONS:
Are validation and test sets used interchangably (Table 1 vs. Section 3.1.2) or are they separate sets? they should be separate since it sounds like learning rate etc. are optimized over the validation set.

How is the singular value threshold for pruning selected in practice?

LIMITATIONS:
limitations on theoretical analysis including convergence are noted in the Conclusion section
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel matrix factorization UDV that utilizes norm constraints on the outer factors and a diagonal term in the middle. The authors demonstrate through numerical simulation on synthetic PSD matrix completion that solutions found through this factorization are exactly low-rank compared to BM factorization. This factorization is evaluated for linear regression and replacing FC layers in image classifiers, where it results in compressed low-rank representations.\n\nSTRENGTHS AND WEAKNESSES:\nThe numerical simulations are convincing of the low-rank bias of the proposed factorization, although the practical implications are difficult to assess as the experiments are conducted on toy datasets. The theoretical insights on page 5 are greatly appreciated, but I was left wanting a more convincing explanation of why low-rank bias arises (see questions below). A major strength of the paper is its clear and detailed exposition. Even though the evaluations are quite limited, the proposed method seems quite promising for more classical directions (low-rank recovery) as well as more modern ones (compressing DNNs), and to my knowledge is novel. The current drawbacks of the paper IMO are the analysis and lack of more modern evaluations. If these could be addressed, it would help increase my score.\n\nQUESTIONS:\n+ From my understanding of the analysis section, the columns of U for which the corresponding lambda is nonzero are forced to be zero eigenvectors of the gradient, but this doesn't necessarily explain why this forces low-rank solutions. Are you assuming that the gradient is full-rank?\n+ My intuition for why columns of U go to zero is that with the matrix norm constraint, it is better to increase the magnitude of columns of U aligned with the top subspace of the gradient (and shrink the other ones), but then this would just result in a rank-one matrix? Overall, if the analysis could be made more rigorous and distinguish the behavior of this factorization (as opposed to BM or weight decay) this would help with appreciating the proposed factorization. \n+ I feel that rather than being a new neural network, this factorization can help compress dense matrices in modern architectures (e.g., transformers). Have the authors explored this direction, and if so, what is the reason it was not included?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a matrix factorization model, $UDV^\\top$, where $U$ and $V$ are explicitly constrained to have a bounded Frobenius norm, and where D is a diagonal matrix with (explicitly enforced) nonnegative entries. The authors motivate this decomposition through the implicit bias of gradient descent to low-rank solutions, and report that it finds truly low-rank solutions, whereas a standard $UV$ factorization only finds approximately low-rank solutions. Based on these results, a new neural network architecture is proposed. Experiments show that low-rank solutions are obtained, resulting in a natural pruning strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is in general well-written, and key ideas are presented in a simple manner.\n- It is interesting that the proposed factorization has an implicit bias to low-rank solutions, even for large step sizes and large distances to the origin, and when used in neural network architectures. \n\nWeaknesses:\n- The theoretical motivation for, and analysis of the factorization are limited. The authors provide no theoretical guarantees for converging to low-rank solutions, although the `theoretical insights’ in section 2.3 provide some loose justification based on a fixed-point analysis. Moreover, the motivation for even considering nonnegative matrices $D$ is unclear. \n- The significance and impact of the proposed architecture is not sufficiently motivated. The main advantage of the proposed architecture is that it converges to low-rank solutions. It should be better argued why this is desirable. The only motivation given in the paper is that this results in a natural pruning strategy, without compromising significant performance. However, the authors do not sufficiently substantiate this last claim, as they do not compare to any existing pruning technique.\n\nTypos:\n1. Line 107: A notable example here matrix completion -> A notable example here is matrix completion\n2. Inconsistency: subj. to in eq. (1), s.t. in eq. (4)\n3. Inconsistency: step-size versus step size. In Lines 247-248 you first use step sizes and then LRs\n4. Inconsistency: Lines 671-672 write 1e-3 and 1e-4, whereas the rest of the paper uses 10^{-3} and 10^{-4}\n\nQUESTIONS:\n1. Can you provide a motivation for considering the nonnegativity constraints on $D$? Do you have an explanation for why it works better than without nonnegativity constraints?\n2. Are there other practical advantages to having an architecture that converges to low-rank solutions, besides for pruning? \n3. Do you think gradient clipping, which entails a modification to the optimizer instead of the architecture itself, can be related to your approach?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}