Title: Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Implicit Bias

Implicit Regularization

Muon

Spectral Descent

Adam

Multiclass
Abstract: Different gradient-based methods for optimizing overparameterized models can all achieve zero training error yet converge to distinctly different solutions inducing different generalization properties. We provide the first complete characterization of implicit optimization bias for p-norm normalized steepest descent (NSD) and momentum steepest descent (NMD) algorithms in multi-class linear classification with cross-entropy loss. Our key theoretical contribution is proving that these algorithms converge to solutions maximizing the margin with respect to the classifier matrix's p-norm, with established convergence rates. These results encompass important special cases including Spectral Descent and Muon, which we show converge to max-margin solutions with respect to the spectral norm. A key insight of our contribution is that the analysis of general entry-wise and Schatten p-norms can be reduced to the analysis of NSD/NMD with max-norm by exploiting a natural ordering property between all p-norms relative to the max-norm and its dual sum-norm. For the specific case of descent with respect to the max-norm, we further extend our analysis to include preconditioning, showing that Adam converges to the matrix's max-norm solution. Our results demonstrate that the multi-class linear setting, which is inherently richer than the binary counterpart, provides the most transparent framework for studying implicit biases of matrix-parameter optimization algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves implicit bias results for linear multi-class classification on linearly separable data with respect to various matrix norms. Specificially, it considers norms given by entry-wise p-norms and Schatten norms (p-norms of the set of singular values of the matrix) with and without momentum. In each of these cases, the parameters converge in direction to a max-margin solution with respect to the corresponding norm.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The results are comprehensive and consider a variety of different optimization methods, unifying them under the umbrella of normalized steepest descent under different matrix norms.\n- Quantitative rates of how quickly the margin shrinks in the number of iterations of gradient descent.\n- The paper is well-written and easy to follow. It includes proofs/sketches of intermediate steps within the main text.\n- The paper describes how specific quantities tracked in the proof, such as $\\mathcal{G}(\\mathbf{W}),$ are modified from the univariate case. This intuition could be helpful for future results for multiclass classification.\n\n### Weaknesses\n- The dataset is assumed to be linearly separable and only linear models are considered. The types of functions that get learned by a neural network for linearly separable versus non-linearly separable data are qualitatively different, so this limits how much we can interpret from the results. But this is a standard assumption when studying the implicit bias of gradient descent for classification, so this is not a major weakness.\n\nSmall typos:\n- In table 1: \"Refernce\" -> \"Reference\".\n- On line 136, no comma after \"algorithms\".\n- On line 154, $\\mathbb{S}'(\\mathbf{v})$ does not match its original definition.\n- In equation (6), $o(\\|\\Delta\\|_F^3)$ -> $O(\\|\\Delta\\|_F^3)$.\n\nQUESTIONS:\n- Isn't Assumption 4 satisfied automatically since the dataset is finite? Is the purpose of this assumption just to introduce the notation $B = \\max_i \\|h_i\\|_1$?\n- I find it surprising that the learning rate needs to decrease over time to prove convergence in direction, as the proof of implicit bias for linear models with binary data works with a constant step size. Do you have any intuition as to which part of your setting makes this necessary?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyzes the implicit bias of normalized steepest descent NSD (with and without moment. the version with moment is abbreviated as NMD) family of optimization algorithms for multiclass classification. as the authors point out, even for linear classification The multiclass setting is more rich due to the linear classifiers being matrices rather than vectors.\nThe NSD/NMD families encompass a broad range of gradient-based optimizers currently being studied by the community. \n\nThe authors focuses on spectral-descent, which contains Muon and Shampoo.\nThey show that normalized steepest descent w.r.t either an entrywise matrix norm or a Schatten matrix norm (Lp norm of the spectrum of the matrix) has implicit bias toward that same norm.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe authors nicely fill in a gap in the multiclass analysis.\nThus, there are richer set of norms/regularizers on these matrix linear classifiers, compared to the binary case.\nThe analysis is clean and slick.\n\nWeaknesses:\nTheorem 1 gives the convergence in margin. How about convergence in direction? Does that also hold?\nThere can be a better comparison with the two previous works Zhang et al. [63] and Xie and Li [59] on line 284 and line 289. How does this work's result compare to those obtained in the two cited papers? Are they better/worse/not comparable?\n\nQUESTIONS:\nEqn (8) looks quite different from the binary one cited on line 172. could you help elucidate/reconcile this difference, at least on an intuitive level?\n\nTheorem 1 gives the convergence in margin. How about convergence in direction? Does that also hold?\n\nSuggestions:\nin the paragraph beginning on line 158, the word \"former\" is used twice and is a bit hard as a reader to parse. consider replacing the word with the actual things the word \"latter\" were intended to reference.\n\nconsider adding a brief comment about how line 211 elementary fact is used to get Property (ii) in Lemma 3\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a comprehensive theoretical characterization of the implicit bias of a wide family of normalized steepest descent (NSD) and momentum descent (NMD) algorithms in the setting of multiclass linear classification with cross-entropy loss. It includes special cases such as Spectral Descent and Muon, both of which are shown to converge to the maximum-margin classifier with respect to the spectral norm. The authors present a unified analysis framework using a carefully constructed proxy function $G(W)$ to track both the loss and margin convergence, and they further extend their results to Adam. Experimental results validate the theoretical claims.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Despite being set in a relatively simple setting, the paper develops an elegant and reusable framework via a proxy function  $G(W)$ that enables generalization across a broad family of optimization algorithms.\n\n2. Muon has recently become popular, and the authors provide new insights into the implicit bias of spectral descent and Muon in multiclass linear settings.\n\n3. The paper is well-organized, with a clear exposition of the background. The related work is thoroughly cited, the theoretical results are solid, and they are supported by experiments. The writing and logical flow are excellent.\n\n**Weaknesses**\n\n1. Strong Linearity Assumption: All analysis is restricted to linear classifier models, which limits the applicability to real-world nonlinear neural networks. There is no discussion of whether or how the results might extend to nonlinear models.\n\n2. Empirical Results Are Limited: The experiments are only conducted on synthetic linearly separable datasets. Whether the approximate implicit bias predicted by theory still holds in more general settings remains an open question worth further exploration. That said, I believe the theoretical contributions of this paper are already substantial.\n\nQUESTIONS:\n1. In Figure 2, your results show that Muon and Spectral-GD converge to the same solution $V_{\\text{spec}}$, right? Within your theoretical framework, what is the implicit bias of the momentum term? Is it possible that the linear model setup is too simple to reveal any difference? As we know, in general nonlinear networks, momentum-based methods like Adam and SGD often prefer different solutions.\n\n2. During optimization, we typically do not constrain the norm of the parameter matrix. So in Theorem 1, is the margin gap result obtained by normalizing with $\\|W_t\\|$? In the caption of Figure 1, how exactly is $\\gamma_{\\bar{W}_t}$ computed? Is it also a normalized result?\n\n3. For Adam, you consider the version without the stability constant. In practice, Adam with $\\varepsilon$ is more common. Why was $\\varepsilon$ omitted, and how would including it affect the implicit bias?\n\n4. Do you expect your convergence results for NSD/NMD to generalize to nonlinear models (e.g., two-layer MLPs)? Can the proxy function idea be extended to such cases?\n\n5. How sensitive are your results to non-separable data? Is there still an implicit bias guarantee when separability only approximately holds (e.g., with margin slack)?\n\nOther minor comments:\n\n1. Table 1: \"Refernce\" → should be \"Reference\".\n\n2. The symbol $e_{y_i}$ in Equation (2) appears to be a standard one-hot vector, but this is not explained upon its first appearance.\n\nLIMITATIONS:\nThe experimental scope is limited—restricted to small-scale, linearly separable synthetic data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors characterize the implicit bias of p-norm normalized steepest descent and its momentum variant for multiclass linear classification with cross-entropy loss, proving that both families of algorithms converge in direction to maximum-margin classifiers defined by the same norm. They derive explicit non-asymptotic convergence rates of $O(1/\\sqrt{t})$ for NSD and NMD. They further show that Adam converges to the $\\ell_\\infty$-margin solution under an appropriate step size.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper’s key contributions stand out in its combination of the finite‐time analysis, a unifying optimization framework, and empirical confirmation:\n\nStrengths:\n- The authors provide the first non‐asymptotic implicit‐bias guarantees for normalized steepest descent (NSD) and its momentum variant (NMD) on linearly separable multiclass classification tasks.\n- The authors consider a broad algorithmic family—p-norm normalized steepest descent (NSD) and its momentum variant (NMD)—which includes practical optimizers such as Muon and  Adam, unifying their implicit-bias dynamics under the steepest descent with unit balls. \n- They provide explicit non-asymptotic convergence rates of $O(1/\\sqrt{t})$ for NSD and NMD.\n- They present rigorous, non-asymptotic proofs that connect the algorithm's convergence directly to maximum-margin classifiers.\n- They also empirically verify that signGD, NGD, spectral descent, and Muon converge in direction to their corresponding maximum-margin solutions. \n\nWeaknesses:\n- Analysis is limited to linear models with cross-entropy loss; extensions to non-linear or deep architectures are not addressed.  \n- No empirical validation is provided to confirm the theoretical convergence rates or practical impact on real datasets.\n\nQUESTIONS:\n1. How sensitive are the convergence rates and maximum-margin solutions to the choice of p in the p-norm normalization? \nCan you specify how the hidden constants in the $O(1/\\sqrt{t})$ convergence rates depend on the choice of p?\n\n2. Have you conducted any empirical validation to confirm that the theoretical convergence rates or implicit bias manifest in real datasets?  \n\n3.  What intuition can you provide on which norm normalization is most beneficial under different data or task conditions?\n\n4. How do the results change when we deal with non-linear models in experiments?\n\nLIMITATIONS:\nWhile this work provides finite-time implicit-bias guarantees for p-norm normalized steepest descent (NSD) and its momentum variant (NMD), its scope is limited. The analysis is confined to fully linear, multiclass separable problems trained with exact, full-precision gradient updates and cross-entropy loss, leaving open the question of whether these results extend to deep or non-linear architectures, as well as stochastic or mini-batch training.  I understand the complexity of the theoretical analysis with more complicated settings, but it would be nice to conduct the experiments to explore whether these results can be extended and made more practically relevant.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}