Understanding Adam Requires Better Rotation Dependent Assumptions
Authors: Tianyue H. Zhang
Lucas Maes
Alan Milligan
Alexia Jolicoeur-Martineau
Ioannis Mitliagkas
Damien Scieur
Simon Lacoste-Julien
Charles Guille-Escuret
Venue: NeurIPS 2025 poster ()
Keywords: Optimization
Adam
Deep Learning
Rotation Dependency
Primary area: deep_learning
TL;DR: We observe that Adam’s performance in training transformers degrades differently under different types of random rotations of the objective function, highlighting the need for new, basis-dependent theory to fully understand Adam’s success.

Abstract: Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature and find that they fall short in explaining Adam's behavior across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam’s basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks that better explain its empirical success.

Understanding Adam Requires Better Rotation Dependent Assumptions
Tianyue H. Zhang422
Lucas Maes 1,2,1 M.e
Alan Milligan
Alexia Jolicoeur-Martinear'
Ioannis Mitliagkas
Damien Scieur2.1
Simon Lacoste-Julien' 1.2,4,5
Charles Guille-Escuret 1.2
1 Mila, Quebec AI Institute University British Columbia
2 Université de Montréal Samsung SAIL Montreal
Canada CIFAR AI Chair
Archimedes Unit, Athena Research Center
Abstract
Despite its widespread adoption, Adam's advantage over Stochastic Gradient De- (SGD) lacks comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's ss performance training transformers degrades under random rotations of the pa- rameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional -nvvrrannnnv assumptions are insufficient to cap- ture Adam's 's advantages theoretically. To better understand the rotation- -epeeden properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-depender assumptions the literature and find that they fall short explaining Adam's s behaviour across various rotation types. In contrast, verify the orthogonality of the update as promising indicator Adam's basis sensitivity, suggesting may be the key quantity for developing rotation-dependent theoretical franeworks that better explain its empirical success.
1 Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities as their scale grows [Brown et al., 2020, Kaplan et al., 2020]. However, this unprecedented growth in model scale has led to a proportional increase in the economic [Dong and Xie, 2024, Sharir et al., 2020, Varoquaux et al., 2024] and environmental [Luccioni et al., 2023, 2019] costs associated with their training. Despite this clear motivation, Adaptive Moment Estimation (Adam) [Kingma and Ba, 2015] has persisted the go-to optimizer especially for language models, with only minor modifications such as AdamW [Loshchilov and Hutter, 2019] becoming widely adopted since Adam's 's inception. This success has prompted extensive research to provide theoretical justification for Adam's 's perfor- mance. While the original convergence proof for Adam was later found to be flawed [Rubio, 2017], recent studies have proposed rigorous convergence proofs under plausible assumptions [Li et al., 2023b, Chen et al., 2019, Défossez et al., 2022].
However, these proofs do not elucidate Adam's advantages over SGD when training transformer models (Vaswani et al., 2017]. Numerous works attempted to explain Adam's superiority, employing
Equal Contribution, correspondence to: tianyue. wwwngmmilaaaaaammmmaamm quebee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

31000
Rotation Liser-Wise
Global Rotation
2.75
Layer
Training Loss
2,0875
Output-Nise
Ingut-Wise
Training Loss
2.50
Input-Wise
Wiiiee
No Rotation
2.25
No Rotation
2.00
23M25
1.23
24700
100 1.03
20
40
63
20
60
50
100
Iteration (k)
Epoch
(a) GPT2 (124M)
(b) VIT/S (22M)
Figure 1: Adam's performance degrades under certain random rotations of the parameter space, demonstrating its dependence the standard basis. (a) For GPT2, global rotations lead to a 16% slowdown in training. (b) ViT experiences a more dramatic 96% slowdown under global rotations. Performance preserved under output-wise rotations but progressively worsens with input-wise, laser-wise, and global rotations, revealing Adam's ' increasing sensitivity to coordinate changes of broader scopes. Experimental details are provided in Section 3.1.
diverse assumptions and analytical frameworks [Zhou al., 2024, Pan and Li, 2022, Zhang al., 2024, Kunstner et al., 2024). The heterogeneity of these approaches has led to a lack of consensus on which theoretical explanations most accurately capture the fundamental mechanisms underlying Adam's 's improved performance. For instance, Zhang et al. [2020] suggests it stems from enhanced robustness to heavy-tailed noise, while Kunstner al. [2023] argues it plays no role.
This study focuses on a fundamental distinction between Adam and SGD: Adam's dependency on the coordinate system. SGD is rotation- equivariant, meaning if the loss landscape is rotated, the resulting optimization trajectories from SGD will be the same up to that rotation. In contrast, Adam produces substantially different trajectories. Although the sensitivity of adaptive methods rotations of the parameter space is well established [Duchi et al., 1). it remains unclear what properties of a rotation benefit or hinder performance, particularly in practical neural network training.
Our experimental investigation reveals that Adam's performance when training transformers em- pirically degrades when the objective function undergoes random rotations (Figure 1). This result demonstrates that Adam's 's effectiveness crucially depends on canonical basis, challenging the adequacy of many existing theoretical frameworks used to analyze Adam's 's performance. Assump- tions employed in the literature (Appendix F) are typically ooaatio-invvriian. and thus the resulting frameworks are agnostic to the basis, preventing them from fully capturing justifying Adam's empirical advantages.
Beyond theoretical motivations, recent studies have shown that applying rotations to the canonical basis can significantly enhance the performance of Adam and other optimizers [Vyas al., 2025, Gupta et al., 2018, Jordan et al., 2024]. However, these rotations are often designed based on intuition and heuristics rather than systematically understanding their impact. By identifying the key properties that make a basis advantageous for Adam, we can develop more principled approaches to constructing optimal rotations that may outperform the existing otation-based methods.
To understand the relationship between basis orientation and Adam's performance, we address two key questions:
Q1. How do various types of rotations influence Adam's performance in practice?
We investigate Q1 by conducting experiments in Section 3, examining Adam's convergence when rotating specific regions of the parameter space. We also identify some rotations that preserve or enhance Adam's performance. These findings provide more nuanced picture of Adam's 's adaptive behaviour and its dependency on the basis.
Finally, as a itation-invarian theoretical framework cannot fully capture Adam's advantage, we turn to rotation -edenddend assumptions existing in the literature, and seek to answer:

---

Original Quadratic Function
Rotated Quadratic Function
SGD-M
SGD-M
46.5
Adam
Adam
29
66.9
28.0
16.7
46.5
46.5
-4
91.1
119.0
150.6
186.0
225
Figure 2: Trajectories of SGD-M and Adam on quadratic under two different rotations. SGD-M maintains the same trajectory up to rotation; Adam does not.
Q2. What rotation-dependers assumptions adequately capture Adam's behaviour under rotations?
Section 4 examines three rotation- deeeedend assumptions used in literature in this context: Loc bounded gradients, Hessian diogo-diagn,lltty and -mooothnesss Our analysis reveals that none of these conditions fully capture Adam's 's behaviour under rotations. Recently, Muon [Jordan et al., 2024] achieved strong performance by approximating orthogonalized gradients for each layer. Inspired by this, we measure the orthogonality of Adam's 's weight matrix updates for each layer (up to scalar multiplication) and find closely aligns with performance across various rotations.
We summarize our key contributions:
Analysis Adam's sensitivity to various scopes of parameter space rotations neural network training: We conduct in Section 3.1 an empirical study demonstrating Adam's sensitivity to random parameter space rotations in practical training. We found a clear correlation between rotation scope (c.g.. global. layer-wiss wise) and performance, where broader scope leads to greater degradation. In Section 3.2 we also employ a structured, SVD>bbaeed rotation inspired by Zhao al. [2024] that improves Adam's convergence.
2. Challenging existing rotation-dependen assumptions: We assess the applicability of rotation-dependen properties in the literature by examining them jointly with our exper- imental study, and find that existing theoretical frameworks are not properly equipped to understand the beneficial properties of Adam.
3. Verifying a better rotation-dependent quantity: Given that SVD-based rotations improve performance, we analyze the singular values the layer updates and find that their orthogo- nality is a strong predictor of Adam's performance across different bases. We also draw connection to the Muon optimizer [Jordan al., 2024], which approximates orthogonalized updates, and provide additional empirical support for its underlying intuition.
2 Preliminaries
Let f: Rd R be the loss of neural network with d trainable parameters. Stochastic optimization algorithms approximate arg wir f(w) by only accessing independent stochastic functions fn that depend stochastic minibatch B following some data distribution D such that V/w Rd EE-DD//((ww = f(w).
Our study examines the optimization process under rotations of the parameter space. More formally, let (O(d) be the set rotation matrices,
sO(d) = ( R Rdd R RR det
Instead of directly optimizing f. we consider its rotated counterpart (R), -> (RRTw) R E SO(d). This transformation rotates the coordinate system while preserving the geometry of the optimization landscape.

---

2.1 Rotational Equivariance of SGD
We say that an optimizer is rotation equivariant if, after a rotation of the parameter space, its trajectories equally rotated.
Definition (Rotational equivariance). Consider an optimization algorithm A applied the function f. generating iterates Wi+1 A(Lwi)i=o....ts f,t). We say that the optimization algorithm is rotation equivariant if it satisfies,
VR E SO(d), Rw =
((R),
where f(R) - fRR w) is the rotation of f.
Proposition 1. Stochastic Gradient Descent with momentum is rolation-equiivariam..
The rotation equivariance of SGD-M is straightforward result as the gradient operator is rotation equivariant; we provide the proof in Appendix for clarity. contrast, Adam not rotation equivariant, due to its element-wise division (Figure 2).
2.2 Training Neural Networks in Rotated Parameter Spaces
crucial aspect of our study is the empirical evaluation of Adam's performance under parameter space rotations. Our approach (Figure 3) maintains the weights w, the standard basis and performs Adam's 's optimization steps in the rotated space. This allows us to leverage existing neural network frameworks for forward and backward propagation while examining Adam's behaviour under rotation.
III Backpropagation
Gradients
(
Rotated
14/
Weights
Gradients
WF
(V) Update
(ii) Adam
Global Rotation
Layer-vése Ratation
Inputwise Rotation
Outpupille
Rotation
Step
(i)
Rotated
Illustration of different rota- tion scopes for a model with weights
Figure 4:
Step
Figure 3: Methodology train neural networks un- der parameter space rotations. (i) Forward and back- ward passes in the standard space to retrieve the gradients. (ii) The gradients are rotated using R. (iii) Adam receives the rotated gradients and produces an update (R) in the rotated space. (iv) (w is rotated back to the original space using R (v) The parameters are updated with R Aw
W {W1, W2, Wa}- Global rotation rotates the entire parameter space at once, layer-wise only performs rotations within each layer subspace, and imput-wise (resp. output-wise) rotates within the weights originating from same input neuron (resp. leading to same output neuron).
Rotations in high dimension. It is computationally intractable to operate with full d x d rotation matrices due to the size of modern neural networks. We employ a composite approach that combines block-diagonal rotations with strategic permutations to cireumvent this limitation while preserving the essential characteristics of uniformly sampled rotations, effectively emulating the statistical properties of full-scale rotations. A detailed description and ablation studies provided in Appendix A.
Numerical considerations. Neural network training is sensitive numerical precision [Li al., 2018, Wang et al., 2018, Sun et al., 2022], and it is crucial to ensure that rounding errors from rotations do not significantly confound the impact of the rotation. In particular, we apply rotations in single precision, and we refrain from using FlashAttention [Dao et al., 2022), which was found to increase numeric deviations [Golden et al., 2024]. We validate our methodology with ablations on SGD rotation equivalence, various rotation dimensions, and the use of FlashAttention in Appendix A.

---

3 Influence of Rotation on Adam's Efficiency
This section examines the effect random rotations from neuron-wise to global (Section 3.1), showing that broader rotations degrade performance, while smaller-scale rotations have little to no impact. We then demonstrate that specific structured rotations can enhance Adam's performance.
3.1 Random Rotations
We first study the effect of four types of random rotations Adam's performance (Figure 4): Global (entire parameter space), Layer-wise (per-layer subspaces; in transformers, keys/queries/valuee are treated separately). Output-wise (rotate weights where connections terminate at the same neuron in the subsequent layer), and Input-wise (rotate weights originating from the same neuron).
Experimental setting. We conduct experiments across three distinct settings spanning both trans- former and non-transformer architectures, and language and vision tasks. Technical details and hyperparameters provided in Appendix B.
Language modeling (GPT-2, Fig. la): 124M-parameter decoder-only Transformer [Rad- ford al., 2019] trained on OpenWebTex [Gokaslan and Cohen, 2019].
Image classification (VIT, Fig. 1b): 22M-parameter Vision Transformer (ViT/S) [Dosovit-
skiy al., 2021] evaluated on ImageNet- IK [Deng et al., 2009].
Image classification (ResNet, Fig, B): ResNet-50 [He et al., 2016] on ImageNet- IK, where SGD often outperforms Adam [Keskar and Socher, 2017, Wilson et al., 2017].
Results. We make several key observations:
1. Adam's performance degrades under global rotations across all settings, confirming that the standard basis possesses advantageous properties.
2. The performance further degrades with broader rotation scopes. Layer-wise rotations, which preserve basis structure, consistently outperform global rotations, highlighting the importance of local coordinate alignment.
3. ResNets exhibit minimal performance degradation under rotations. This reduced sensitivity suggests Adam obtains limited benefit from the standard basis structure in ResNets, which possibly explains its historically smaller marginal gain in training these networks.
4. Ouiput-wise rotations show no degradation across all settings, with GPT2 even slightly im- proving. This suggests that Adam's adaptivity within output neurons is minimal, supporting recent approaches to reduce redundancy in Adam's 's second moments [Zhang et al., 2025].
Previous works [Zhang et al., 2024, 2025] have highlighted the heterogeneity across different parameter block types in Transformer architectures. Appendix C, we restrict rotations specific parameter types and study their individual impact on Adam's 's rotation sensitivity.
3.2 Investigating the Performance-Improiin Rotations
Inspired by GaLore [Zhao et al., 2024], which uses low-rank Singular Value Decomposition (SVD) to compress optimizer states, extend this concept to full-rank SVD to rotate the parameter space. Our approach decomposes the gradient matrix G of each layer into G = USVT This decomposition yields natural rotation of the parameter space through the transformation G - UT GV, which corresponds to an output-wise rotation (via U and an input-wise rotation (via V). We train a GPT2 model under the same conditions as in Section 3.1, but in this SVD-rotated space. We update the SVD decompositions every 250 steps.
Figure 5 shows that Adam's performance improves under SVD-based rotations, with low computa- tional overhead. These results highlight the potential of more principled understanding of how basis orientation affects optimizer behaviour. Instead of relying on intuition or heuristics, we advocate for theory-driven design grounded rotation sensitivity.
approaches and motivate a
To further understand this behaviour, Figure 6 shows second moment distributions after training under various rotations. Global random rotations yield more concentrated second moments, implying less variation in effective learning rates and reduced adaptivity, which explains Adam's 's degraded
5

---

2,3000
Giobal Rotation
No Rotstion
1.2
Global Botation Rotation
Training Loss
3,0077
OutputNiae Sidd
Output-Wise
Froquency
3,0150
2,0422
Iteration (k)
Log second moment values
Figure 5: Performance of GPT2 trained with Adam in SVD-rotated space, without rotations, with random output-wise rotation and with ran- dom global rotation. The rotations computed with SVD lead sizeable improvement.
Figure 6: Distribution of second-moment values for the final checkpoint of a GPT2 model trained with Adam in various rotated spaces. Second moments are concentrated under ran- dom rotations, indicating reduced adaptivity.
performance, However, the benefits of SVD rotations are not apparent from second moments alone, suggesting a more subtle relationship between the parameter space and Adam's adaptive behaviour.
4 Examining Rotation Dependent Assumptions
We have established that Adam's performance depends heavily on the choice of basis. Rotation- invariant analyses yield identical guarantees for all bases, failing to capture performance gaps between rotations. A rotation-dee dettrinn-deeedeent assumption is necessary to explain Adam's practical advantage. In this section, we examine rotation- -eetenndde assumption adequacy jointly with our GPT-2 experiments.
4.1 Adequacy of existing assumptions in theoretical frameworks
While rotation-invariant assumptions dominate optimization literature, some frameworks incorporate rotation-dependent properties. This subsection examines three existing assumptions and whether they adequately capture Adam's rotation dependency. In particular, we focuses on two aspects: (i) Practical Feasibility. The assumption must be realistic in practical settings. (ii) Alignment with Adam's Performance. An adequate property should have favourable constants under rotations that improve performance and break down (or have unfavourable constants) under rotations that hinder performance. An ideal theoretical convergence analysis should be based on realistic assumptions that relate the problem's characteristics the optimizer's performance, where faster theoretical rates correspond to better practical performance.
-mooohhtsss and (1, -oorm. -mooothnss was recently shown to guarantee the conver- gence of Adam, and presented as potential key property of the basis [Xie et al., 2025, Balles et al., 2020]. We first remember its definition.
Definition 2. A function f is C-smooth wrt. 1ox -
V/ S Cl|x-yyll V x, E Rd.
Given the challenges in directly estimating the Loe smoothness constant, Xie al. [2025] proposed using the (1,1)-norm of the Hessian as surrogate, defined as:
M
N
H =
Σ
Σ
H
VIL m=1
where H represents the element at the m-th and n-th column of the Hessian matrix. Notably, they observed degradation in their estimate of [H] under global random rotations. However, remains unclear whether this degradation is a universal phenomenon for all rotations of the parameter space or if it specifically correlates with Adam's 's performance. To investigate this, we estimate the (1, 1)-orrmmmmm by averaging the L1 norm of Hessian rows, sampled using the methodology described in Section 4.1. Figure& 8 illustrates the change in H (1.1) under global, SVD, and output-wise rotations.
Under global rotations, we confirm the (1, norm degradation reported in [Xie al., 2025], while SVD rotations improve it in line with Adam's performance gains, suggesting a link between this

---

Man
Man
0.02
Change Train Loss
50
Change la (1. (1.1)-Nom
Change Caose
Change 1rato
0.01
0.01
0.00
0.0
0.00
-0.01
-0.01
-25
0.02
0.02
Global
SVD
Output
Global
Output
Rotation Type
Type
Figure 7: Empirical Lo gradient bound Č over 1000 stochastic gradients the last checkpoint for Global, SVD, and output-wise rotations, pre- sented as differences from the non-rotated base- line. The trend disagrees with Adam's 's perfor- mance, especially under global rotation.
Figure Estimated (1, (1,1)-m) norm of the Hessian and final accuracy for Global, SVD, and output- wise rotations, presented as differences from the non-rotated baseline. The (1, norm corre- lates with Adam's performance on global and SVD rotations, but not output-wise.
Loo bounded gradients. Reddi et al. [2018], Kingma and Ba [2015] assume bound on the Loo norm stochastic gradients,
VWER",
[VfB(w)ll ( C
almost surely.
The constant C' depends on the basis, as the Loo norm is not preserved under rotations. To evaluate this assumption, compute the empirical bound the rotated gradients,
CR (R) == max B.
V (wR)lloos
where WR denotes the last checkpoint obtained by running Adam under rotation R. The maximum over 1000 stochastic minibatches B;, across different rotations R (see Section 3). Figure 7 reveals that Ö significantly decreases under random global rotations, predicting better performance, but we observe degradation in Adam's convergence. This discrepancy shows that the Loo gradient bound fails to capture the beneficial properties of the basis for Adam.
geometric property and optimizer efficiency. Output-wise rotations yield slight performance gains but reduced (1, n)-mrmm indicating that the metric does not capture all relevant factors. Overall, the (1, 1) -)-mor shows promise as -eatiinnsensiiive indicator, especially under global and SVD rotations, but its limitations motivate the development of refined or complementary measures.
Block-diagonality of the Hessian. A common hypothesis for understanding Adam's behaviour is that the Hessian is well- -pproooxoiite by block-diagonal matrix [Zhang et al., 2024]. Then, random rotations likely disrupt -iagoaiiygyyl and hinder convergence, while rotations within diagonal blocks preserve the structure, explaining the stable performance under output- wise rotations.
To examine this assumption's validity, we sample rows of the Hessian of (R) at checkpoint wa.
ri (R) = E 2 (
(WR) Eラ e; TRV2 fB, RW
where e; denotes the 7th canonical basis vector. The vector r; represents the average of the i-th of the stochastic Hessian over & minibatches. As k increases, converges to the true Hessian row. We is 5000 in the experiments, and use efficient dessian-vector products [Dagréou et al., 2024].
We partition the indices of the Hessian row I; corresponding to weight w; into three disjoint subsets:
ri = rips]] + ripp.] + rilz/1 where
In are indices of weights leading to the same output neuron as w,,
Jr are indices of other weights from the same layer.
are indices of weights other layers.
and ri[ss] (resp. 1r and 1 in the indices in IN (resp. 1i and I and zero elsewhere.

---

No Rotation
Global Rotation
Outpul-wise
SVD Rotation
Absoltue Valunn (Lagg)
13
tort
23m
Io-
Iom
Nwaron
Layor
Other
Neuron
Layer
Other
Nauron
Layer
Other
Neumn
Layer
Other
Figure 9: Distribution of Hessian values within output neuron, layer and non-layer in the second Transformer block attention projection layer. In no rotation, values within the neuron are of magnitude higher than others, presuming a possible block-diagonal structure. The structure is preserved in SVD and output- wise rotations, and lost in global rotation.
Figure presents the distribution of absolute values for each subset. Our findings show that entries in In and, lesser extent, Z2, are significantly larger than those from L. supporting an approximate doocoaliaoonal Hessian structure.
Given this approximately block-diagond Hessian structure, previous work [Zhang et al., 2024] proposes a strict block-diagone approximation, assuming that off-diagonal elements are negligible. We further investigate whether this simplification can accurately reflect the local geometry by assessing the practical implications the block diagonal Hessian structure. We evaluate how each block contributes to gradient variations for given small direction dw via:
fW dw) [(()]] = e. V f(w)(ww ri[[x] ów Fi(e)] ow + Tilr Ów.
Table I quantifies these contributions random direction or update direc- tion. Surprisingly, our results reveal that Hessian values outside the block are the primary drivers of gradient evolution, despite their smaller mag- nitude. This finding challenges the strict block-diagonal Hessian assump- tion in theoretical analyses. While the diagonal blocks contain larger values, their limited size compared to the full parameter space means that off-diagonal elements collectively play a crucial role in shaping the loss landscape's geometry. Neglecting off off-diagonal elements is oversimplification, making the approximation inadequate and potentially misleading downstream results.
dw direction
Random
Update
Fi[Tx] 5w (Neuron)
2.86
10-5
-4.6 10-10
[i[G.]" 5w (Layer)
-8.71 x 10
1.30 x 10
Fifr 5w (Other)
1.48 x 10-4
2.02 x 10-7
Table 1:
Contribution Fi(z] 6w(z of Hessian values in block to the variation of the i-th gradient component in direction 5w. Averaged over multiple Pw, off-diagonal blocks con- tribute significantly both random and update directions.
4.2 Orthogonality layer updates up to scalar factor
In Section 3.2 we find that SVD-based rotations improve Adam's performance. We also discuss in Appendix E connections with the recently proposed Muon optimizer [Jordan et al., 2024], which achieves strong performance by performing updates in the orthogonalized first moment direction.
Scaled jothoonothog Since orthogonality defined only for square matrices, we adopt relaxed notion for rectangular matrices. Specifically, we say that rectangular matrix is orthogonal if (this notion is commonly referred as semi-orthogonality
all its singular values are either [Abadi and Magnus, 2005]). Moreover, we say that matrix is a scaled orthogonal matrix if its cigenvalues are either a or where a is the scaling parameter. To measure the scaled orthogonality of matrix A, we will use the coefficient of variation of its singular values
CV(s;) μ, min, n5 V (s; -a)
where μs and are the mean and standard deviation of the s,'s, respectively. We discuss in appendix C.4 other measures of scaled orthogonality matrix
Scaled orthogonality of the layer update. We resume training from a checkpoint for each rotation
type for the next 500 steps to measure the orthogonality of the update w!!!) (!)) where w!(
++1

---

Wg
Wy
MLP W.
1.3
0.60
0.50
Singular Value Vartation
1.0
0.9
0.40
0.35
20100
30200
Wk
20420
50000
20020
50100
30203
Wo
20002
50600
52300
50020
MLP
0.7
1.2
1.1
1.0
0.5
0.9
0.8
20100
30200
20020
50000
50100
30203
30600
32320
50020
32120
30502
20600
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 10: CV of singular values of layer updates over 500 steps, averaged over depth. SVD rotation consistently yields lower CV and more orthogonal layer updates, whereas full rotation shows the opposite. Downward spikes in the SVD rotation occur when the rotations recomputed.
represents the weights of layer at time t. For simplicity, we omit the layer index (1). To separate the
effect of step size and weight decay, we measure the update as A R. (. /(V ( +c).
Observations. Overall, we find that SVD consis- tently yields lower CV and more orthogonal updates under the coefficient of variation measure, and output- wise rotation behaves similarly to the no-rotation baseline. Full rotation consistently results in the least orthogonal updates. This ranking aligns clearly with the observed performance of Adam under rotations (Figure II), making a promising quantity to under- stand Adam's 's behaviour.
Change Update Orthoponnllll
Ain
0.02
0.01
Change
-0.01
-0.02
In Figure 10, show the CV across time for each layer type, averaged over the depth. See Ap- pendix C.4 for full results per layer and over time. Notably, CV drops right after the SVD rotation is recomputed every 250 steps. This offers insight into the frequency tradeoff of SVD rotations, where more frequent updates improve performance but introduce computational overhead. Overall, our analysis sug- gests that update orthogonality strongly correlates with optimizer performance, supporting the approaches of Muon [Jordan et al., 2024] and SOAP [Vyas et al., 2025] and opening new avenues for rotation-aware theoretical frameworks.
Global
SVD
Output
Retation Type
Figure 11: Each panel shows the difference in loss and average CV of singular values relative the non-rotated baseline. The or- thogonality of layer updates closely aligns with performance under rotation.
While robust theoretical analysis of Muon's 's update rule remains open, attempts have been made to characterize in what context an orthogonal update is optimal. Bernstein [2025] argues that for linear layers, the orthogonalized update controls the scale at which the weight matrices can scale features. They argue this encourages stable optimization and limit the need for normalization layers. As we discuss in Appendix E, after simplification and SVD rotation the Adam update simplifies to dividing the singular values by their magnitude. When rotated back, this results in the same update Muon through a different mechanism, which is consistent with Muon's 's often superior performance.
5 Related Work
Optimization under rotations. Shampoo [Gupta et al., 2018] first demonstrated the benefits of optimizing under rotations by running AdaGrad under regularly updated rotations, identifying the singular values the gradient matrices. More recently, SOAP [Vyas et al., 2025) improved Shampoo by applying it to Adam and appropriately updating the moments each change of basis. Muon

---

[Jordan et al., 2024] concurrently explores a similar approach, but removes the need to explicitly store rotation matrices and instead orthogonalizes gradient matrices with a fast matrix iteration. While these methods show promising empirical improvements via heuristics, our work highlights the importance of developing better theoretical tools to understand their success.
On the theoretical side, we consider [Xie et al., 2025] to be the closest related study, showing that Adam converges more slowly with a randomly rotated loss landscape. They provide convergence analysis based on Loo geometry, demonstrating that this yields better empirical smoothness constant for GPT-2 models. While their work offers valuable theoretical insights, our study takes a more experimental stance. We aim to paint comprehensive picture of Adam's 's behaviour under spectrum of rotations, from random structured transformations. and evaluate how existing assumptions correlate with Adam's 's performance. Notably, Balles et al. [2020] also provides relevant insights through the lens of sign gradient descent.
Understanding Adam. Our work casts light on the critical interactions between Adam and the coordinate system, contributing to a growing body of research on Adam's 's behaviour and convergence. Recent works have attributed Adam's 's success to the heterogeneous block-diagonal structure of its Hessian [Zhang et al., 2024], though we find this assumption be unrealistic. Others have improved convergence guarantees: Défosse7 et al. [2022] and Guo et al. [2022] offered simplified and novel derivations, Zhang et al. [2022] argued that vanilla Adam converges without modification, Zhou et al. [2024] provided a general convergence analysis for adaptive methods in non-convex settings, and Li et al. [2023b] proposed convergence proof for Adam without relying on globally bounded gradients. Li et al. [2023a] developed a convergence analysis based on generalized smoothness conditions, and Hübler et al. [2023] proposed parameter-agnostic convergence results under these relaxed conditions. Finally, lower bounds for non-convex optimization were established by Arjevani et al. [2019], with Wang et al. [2023] addressing the gap between upper and lower bounds for Adam's ss iteration complexity.
Adam's ad advantages over SGD. Prior works have attempted to justify Adam's advantages over SGD. Zhang et al. [2020]. Zhou et al. [2024] suggest SGD suffers more from heavy-tailed noise, with Adam converging faster when gradients are sparse. However, Kunstner et al. [2023] found that noise reduction through larger batch sizes benefits Adam but not SGD. Additionally, Kunstner et al. [2024] ties Adam's 's advantage in language models to ill-conditioning caused by heavy-tailed class imbalance, and Pan and Li [2022] to directional sharpness.
6 Discussion
Linitations. (i) Our purpose of the SVD rotation is not to introduce a new practical optimizer, but to demonstrate the existence of more beneficial rotation and provide insights into the relationship between Adam and the standard basis. (ii) While our results reveal an alignment between the semi-orthogonality of layer updates and Adam's 's empirical performance, we do not offer theoretical guarantees, and further work is needed to formalize formalize this quantity and incorporate it into theoretical analysis. (iii) We relate our findings the Muon optimizer and discuss the theoretical motivation for scaled orthogonality; however, there is still a lacking of rigours understanding of why Adam under SVD rotations produces more orthogonal updates than under the canonical basis, and why this quantity leads to improved performance. (iv) Finally, although the observed gap appears smaller in recent experiments, more evidence is required confirm whether the superior performance of SGD on ResNets primarily comes from reduced sensitivity to rotations.
Conclusion. In this work, we have conducted comprehensive investigation into Adam's 's sensitivity to rotations of the parameter space, revealing key insights into its optimization dynamics. We demon- strated that some rotations possess advantageous properties, opening new avenues for algorithmic contributions to adaptive algorithms. Our study demonstrates that Adam's 's performance is intricately tied the choice of basis, a relationship that existing theoretical frameworks struggle to capture adequately. This investigation highlights the limitations of current explaining Adam's 's behaviour, and identifies update orthogonality as a promising theoretical tool. As the field evolves, we hope these findings will spark new avenues of research, potentially leading to more robust optimization algorithms and deepening our understanding of the fundamental principles underlying successful deep learning optimization.
iaation-invarian assumptions in
10

---

Acknowledgments and Disclosure of Funding
This research was partially supported by the Canada CIFAR AI Chair program (Mila) and Samsung Electronics Co., Ltd. Simon Lacoste-Julies is CIFAR Associate Fellow in the Learning in Machines Brains program and acknowledges support by NSERC Discovery grant (RGPIN- 2GGPI-2005--05123) We also acknowledge that this research was partly enabled by computing resources, software, and technical assistance provided by Mila and the Digital Research Alliance of Canada. Ioannis Mitliagkas We thank Adam Ibrahim
acknowledges support by an NSERC Discovery grant RRGPIN-2019-0657 for his helpful comments and insights, and Ayoub Echchahed, Frederik Kunstner, Mark Schmidt, Pedram Khorsandi, Ryan d' O'Orazio, and Vitória Barin Pacela for their valuable feedback.
References
K. M. Abadir and J. R. Magnus. Matrix Algebra. Cambridge University Press, Cambridge, 1st edition, 2005. See Section 4.23 on Semi-orthogonality.
Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake E. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming. 199:165-214, 2019.
Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent, 2020.
Jeremy Bemstten. Deriving muon, 2025. URL https:// https:://errmybeenste.... wttp:/sttttettiiiieeeieeee ting/ deriving-muon.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- 1k. arXiv:2205.015800 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. In ICLR, 2019.
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. In NeurlPS, 2022.
Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How to compute hessian-vector products? In ICLR Blogposts, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NeurIPS, 2022.
Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. simple convergence proof of adam and adagrad. TMLR, 2022.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Haiwei Dong and Shuang Xie. Large language models (Ilms): Deployment, tokenomics and sustain- ability. arXiv:2405. 17147, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
John Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
11

---

Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019.
Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Is flash attention stable? arXiv:2405. 020.02233. 2024.
Baptiste Goujaud, Adrien Taylor, and Aymeric Dieuleveut. Optimal first-order methods for convex functions with quadratic upper bound. arXiv:2205.15033, 2022.
Charles Escuree Baptiste Goujaud, Manuela Girotti, and Ioannis Mitliagkas. A study of condition numbers for first-order optimization. In AISTATS, 2020.
Charles Guille-Escuree, Adam Ibrahim, Baptiste Goujaud, and Ioannis Mitliagkas. Gradient descent is optimal under lower restricted secant inequality and upper error bound. In NeurlPS, 2022.
Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras, and Ioannis Mitliagkas. No wrong turns: The simple geometry of neural networks optimization paths. In ICML, 2024.
Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. A novel convergence analysis for algorithms of the adam family and beyond. arXiv:2 2022.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In ICLR, 2018.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Florian Hübler, Junchi YANG, Xiang Li, and Niao He. Parameter-agnostic optimization under relaxed smoothness. In Neurips workshop OPT: Optimization for Machine Learning. 2023.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08366, 2020.
Andrej Karpathy. NanoGPT, 2022.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam sgd. arXiv:1712.07628, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In ICLR, 2023.
Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. In NeurIPS. 2024.
Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimization under generalized smoothness. In NeurIPS, 2023a.
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. In NeurIPS, 2023b.
Huan Li and Zhouchen Lin. On the T174 convergence rate of rmsprop and its momentum extension measured by E norm. arXiv:2402. 240..0999,, 2024.
a
12

---

Zhaoqi Li, Yu Ma, Catalina Vajiac, and Yunkai Zhang. Exploration of numerical precision in deep neural networks. arXiv:1805.01074 2018.
Jingyuan Liu and Jianlin et al. Su. Muon is scalable for Ilm training, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, 1 176b parameter language model. JMLR, 24(253):1-15, 2023.
Sasha Luccioni, Victor Schmidt, Alexandre Lacoste, and Thomas Dandres. Quantifying the carbon emissions machine learning. In NeurlPS Workshop on Tackling Climate Change with Machine Learning, 2019.
Zhi-Quan Luo and Paul Tseng. Error bounds and convergence analysis of feasible descent methods: general approach. Annals of Operations Research, 46(1):157-178, 1993.
Francesco Mezzadri. How generate random matrices from the classical compact groups. Notices of the American Mathematical Society, 54:592 - 604, 2007.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate o(1/k?). Proceedings of the USSR Academy of Sciences, 269:543-544 1983.
Maris Ozols. How to generate a random unitary matrix, 2009. Technical report.
Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than SGD for transformers. In NeurlPS Workshop OPT: Optimization for Machine Learning, 2022.
Boris T. Polyak. Gradient methods for the minimisation of functionals. USSR Computational
Mathematics and Mathematical Physics, 3(4):864 878, 1963.
B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018.
David Martinez Rubio. Convergence analysis of adaptive method of gradient descent. University of Oxford, Oxford, M. Sc. thesis, 2017.
Or Sharir, Barak Peleg. and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv:2004.08900, 2020.
Yuxin Sun, Dong Lao, Ganesh Sundaramoorthi, and Anthony Yezzi. Surprising instabilities in training deep networks and a theoretical analysis. In NeurlPS, 2022.
Gaäl Varoquaux, Alexandra Sasha Luccioni, and Meredith Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in ai. arXiv:2409.14 141600 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira. David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In ICLR, 2025.
Bohan Wang. Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of adam's iteration complexity. In NeurlPS, 2023.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan, Training deep neural networks with 8-bit floating point numbers. In NewrIPS, 2018.
13

---

Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods machine learning. In NeurIPS, 2017.
Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $tell_ Stell_ -innity-eggooeetty of loss landscape via coordinate-wise adaptivity. In ICLR, 2025.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In NeurIPS, 2020.
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhii-Quan Luo. Adam can converge without any modification on update rules. NeurIPS, 2022.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li. Ruoyu Sun, and Zhi-Quar Luo. Why trans- formers need adam: A hessian perspective. In NeurlPS, 2024.
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. ICLR, 2025.
Jaawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- efficient LLM training by gradient low-rank projection. In ICLR Workshop on practical ML for limited/low resource settings, 2024.
Dongruo Zhou, Jinghui Chen, Yuan Cao, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. TMLR, 2024.
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for conver- gences of adam and rmsprop. In CVPR, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction are supported by the following sections:
Justification:
In Section 3.1, we show that Adam's performance in training transformers degrades under random rotations of the parameter space. In Section 3.2, we demonstrate that applying SVD-based rotations improves empirical performance.
In Section 4.1, we examine three existing quantities and show that they fail to explain the performance changes under rotation. In Section 4.2, we show that update orthogonality better aligns with performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section 3.2, we clarify that the purpose of the SVD rotation is not to propose new practical optimizer.
In Section 4.2, note that the proposed quantity opens the door for future work to formalize and incorporate it into theoretical analysis.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
15

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: There are no major theoretical results. For Section 2.1, we provide proof in Appendix D for clarity and illustrative purposes, although this is not novel result, as stated in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We use standard architectures and datasets commonly used previous work, stated in Section 3. The experimental details and hyperparameters needed to reproduce the main results are provided in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets we use are publicly available and standard. The code to reproduce our experiments will be made publicly available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httppp..ccpubbicc//e www.////g///e//eeeeeoeo/eeeaeee/ /CodeSubmia ssi assionPolli onPolicy) iyl for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are publicly available and follow standard practices. We provide the full details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments. The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Although the large experiments were too computationally expensive to run over multiple random seeds, we provide several ablation studies in Appendix C, examining the dimension of the rotation matrices, numerical stability, and verifying that the same procedure confirms the rotation invariance of SGD. For the analysis in Section 4, each experiment sampled from a specific number of steps from checkpoint, as described in the corresponding subsections.
17

---

Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources required are documented in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: pettps://neurppll wwt.eurrps...ppuulrieeeeececeeee wwtppcccc//eeeeeeeeceeec htthi//tihihisuieeeeeeeiiiiieaie
Answer: [Yes]
Justification: The authors confirm that the research conducted conforming to the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
18

---

This paper focuses on foundational research aimed at understanding the behaviour of generic algorithms used to optimize neural networks. It is not tied any specific application, and we do not foresee direct path to social impact at this stage.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Section 3.1 and Appendix B includes references for the datasets, models, and code used this project, along with citations to the original papers and URLs where available.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, anperswithcod com/coo/////ooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu-
tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ettps:/nnet http///eeipp.///// /2//oonferencce/2055/// /LLM) for what should or should be described.
21

---

Understanding Adam Requires Better Rotation Dependent Assumptions
(Appendix)
Introduction
2
Preliminaries
2.1 Rotational Equivariance of SGD
2.2
Training Neural Networks Rotated Parameter Spaces
3 Influence of Rotation on Adam's Efficiency
3.1
Random Rotations
3.2
Investigating the Performance- Performance-mprroinng Rotations
4
Examining Rotation Dependent Assumptions
4.1
Adequacy of existing assumptions in theoretical frameworks
4.2
Orthogonality of layer updates up to scalar factor
5
Related Work
Discussion
10
A
Sampling Random Rotations in High Dimension
23
A.1
High-Dimensional Rotations
23
A.2 Reflections and Sampling From The Haar Measure
A.3 Rotation Residual
24
A.4 Overall Validation and Impact of FlashAttention
24
Experimental Details
24
B.1 Rotations Design Choices
................................
25
B.2 Architectures
25
................................
B.3 Assumptions Estimation
26
Additional Results
26
C.I Main Experiments...
26
C.2 Architecture Aware Rotation.
27
................................
C.3 Hessian Rows
28
C.4 Update orthogonality
30
D Optimization Algorithms with Rotations
32
SVD Rotations and Muon
40
F Common Assumptions in First-Order Optimization Theory
41
22

---

A
Sampling Random Rotations in High Dimension
This section explains our method of sampling random rotations for high-dimensional spaces and the implementation details.
A.1 High-Dimensional Rotations
Even small modern machine learning models typically have millions of parameters. Consequently, storing a d x d rotation matrix is often intractable, let alone performing the dot product required to rotation matrix R,, with < d
rotate the gradient vector. To address this issue, sample uniformly (in the sense the Haar measure) from the special orthogonal group (n)),, and a random permutation ı of 0, O,...,d- 1. For now, we assume d E N, see appendix A.3 for a general case. To rotate gradient g. compute:
d/n + Rn
g (R.
1
π
o (rog)
(1)
i=1
Rn
Rn
0
-T-
Rn
(T o g),
(2)
0
R
where Q denotes the direct sum operation, producing a block-diagonal matrix with d/n blocks R.,- This procedure effectively computes a rotation by blocks of size picked from a random partition of indices, constituting a valid rotation.
Intuitiiell, if n is sufficiently large, we expect this procedure to approximate well the effect of random rotations sampled uniformly from (O)d), due the law of large numbers homogenizing geometric properties across coordinates. To confirm this intuition, we perform an ablation study in Figure 12, finding that the impact on Adam's performance saturates well below our operational values.
Our approximation reduces the memory cost from (dd) to O(n² + d), and the computational cost from ((d) to O(nd). Since batch matrix multiplications required for the rotation can be performed efficiently on modern GPUs, the final overhead of applying rotations extremely small.
3.400
Rotation dimension
512
Training Loss
3.325
768
1024
no rotation
3.250
3.175
3.100
1.0
6.8
12.6
18.4
24.2
30.0
Iteration (k)
Figure 12: Training loss of GPT-2 when training with different rotation dimension n. The loss of performance is consistent across n at our range.
23

---

A.2 Reflections and Sampling From The Haar Measure
To sample Rn uniformly from SO(n) with respect to the Haar measure, we employ the QR decompo- sition trick [Mezzadri, 2007, Ozols, 2009], which samples from the Haar measure μ of the orthogonal group (n). Let consider the projection π O(n) - SO(n), such that π (R) ) is R when
R. (n),, and π (R) simply multiplies the first column of by when R E ( (.S(n).. The push forward of ja by π is the Haar measure on (O(n). Since Adam is reflection equivariant, rotating with π (R) and with will lead to identical performance for any R E O(n). Thus we can omit to apply and simply sample from j using the QR decomposition method.
Similarly, Adam is permutation equivariant; thus we omit to apply the inverse permutation before providing the rotated gradients to Adam, and to apply the permutation before rotating the update, as removing these two steps does not affect performance.
A.3 Rotation Residual
Based on the type of rotation the chosen dimension n, the number of blocks may not divide evenly, i.e., d á N. To address this issue, we introduce an additional rotation matrix, which we refer n to as the residual matrix, complete the missing dimensions. More formally, let d represent the dimensionality of the parameter space, and let n denote the block dimensions of the rotation. We define b - [ as the number of complete blocks. The residual matrix R. then sampled from (p))), where p A nb. Therefore, eq. (I) becomes
9 R... = π - ([B @ RJ
(T g)),,
(4)
B
(T o g),
(5)
R
R
R
=T-
(x g)
(6)
0
R,
R
where B + R..
A.4
Overall Validation and Impact of FlashAttention
In Figure 13, we present the training loss when training GPT-2 with SGD without rotations, with global random rotations using Fttentntention, and with global random rotations without FlashAttention. In particular, we confirm two important observations:
Without FlashAttention (the setting we use for our experiments) the performances of SGD under global random rotation and under rotations are identical. This validates that our experimental setting is behaving as expected.
When use FlashAttention with rotations, observe slight difference in performance. As explained in Section 2.2, this is due to FlashAttention amplifying numerical errors from the application of the rotation. Interestingly, likely due slight regularization effect, it increases training performance.
B Experimental Details
This section provides additional details about the hyperparameters used for the architecture mentioned in the paper, well their optimizer and rotations.

---

3.7
SGD
no rotation
global rotation, no flash attention
Training Loss
3.6
global rotation, flash attention
3.4
3.3
52
84
Iteration (k)
Figure 13: SGD performance when applying global random rotations, with and without FlashAtten- tion.
B.1 Rotations Design Choices
By default, for random rotations we fix the dimension of our rotation matrix R., at 768 (which hidden dimension and thus makes residual rotations unnecessary for most rotation types). The matrix sampled at the start of training, remains fixed throughout, and shared across blocks the entire training process.
SVD Rotation. Following [Zhao et al., 2024], this is the only rotation that is dynamic rather than static. Specifically, we compute the full-rank SVD decomposition of the gradient for each layer every 250 steps (recommended frequency in [Zhao et al., 2024]).
Rotation in Transformers. By default, many implementations store the query, key, and value pa- rameters within single linear layer. Thus, we split them to treat them as separate layers, reflecting the fundamental differences in how their parameters PyTorch stores parameters as tensors in the shape (output_ding stored as lookup tables in the shape finput_dimg output_diil For output neuron and input neuron rotations to behave intuitively, we thus transpose embedding layers before and after rotations.
involved in forward computations. Additionally,
input_dim) , but embeddings
B.2 Architectures
GPT2 (Transformer). We trained GPT-2 model with 124M parameters the OpenWebText dataset [Gokaslan and Cohen, 2019] using a configuration designed for efficient pretraining. The model architecture includes 12 layers, 12 attention heads, and 768-dimensional embedding space, with no bias in LayerNorm or Linear layers. We employed the AdamW optimizer with a peak learning rate of 6 10-4 81 0.9, 832 0.95, and weight decay of 0.1, applying gradient clipping of 1.0. Training ran for 100,000 iterations (or 30,000 for some smaller ablations), with learning cosine rate decay starting after a 2.000-iteration warm-up. decaying to a minimum of 6 x 10-5, We used sequence length of 1024 and micro batch size of 12 with gradient accumulation steps to simulate an effective batch size of 480 sequences. We additionally tried tuning 83 by using values 0.9 and 0.99. We found that the base AdamW W showed slightly better performance, but the globally rotated model's performance decreased with both of these values, meaning further tuning will not close the gap with the base model. All experiments were performed on four A 100 80GB GPUs, leveraging mixed precision. Unless otherwise specified, all optimizer hyperparameters were shared across experiments and set the default values specified in Karpathy [2022].
VIT (Vision Transformer). We trained a Vision Transformer (ViT) model on the ImageNet- dataset [Deng et al., 2009] using the VimpleViT architecture [Beyer et al., 2022]. The model consists
25

---

of 12 layers, 6 attention heads, a hidden dimension of 384, and an MLP dimension of 1536, with patch size 16 and input image size of 224. The AdamW optimizer was employed with learning rate of 0.001, 31 0.9, f2 0.999, = 10-$, and weight decay of 0.1. We used cosine learning rate schedule with warm-up epochs. The training was conducted for 100 epochs with a batch size of 1024. All experiments were performed with mixed precision.
ResNet-50 (CNN). We trained ResNet-50 model [He al., 2015] on the ImageNet-IK dataset [Deng et al., 2009] using the Adam optimizer. The optimizer was configured with learning rate of 0.001, 31 0.9, 32 0.999, 10 and a weight decay 0.0001. We employed cosine learning rate schedule with 5 warm-up epochs. The training ran for 100 epochs with batch size of 256.
B.3 Assumptions Estimation
We now outline how we computed empirical estimations of assumptions in Section 4.
Lee bounded gradient. Algorithm 1 describes the process estimate the bound constant C' of stochastic gradients under Loc norm, as detailed in section 4.1.
Algorithm Empirical Gradient Bound Estimation for Adam
Require: T: total number of iterations (1000)
Require: WR: last checkpoint obtained by running Adam under rotation R
1: Initialize Ć C-+0 0
Maximum infinity norm of gradients
2: for t + 1 to T do
3: Sample minibatch B;
Select minibatch
4: gB; V V (WR)
b Compute gradient for minibatch Compute infinity norm of the gradient Update the maximum gradient bound
5:
Č" - l|B llcc
6: Č max C, C*)
7: end for
8: return C
Return the estimated gradient bound
(1, 1)-Norm. Using the Hessian rows sampled from GPT-2 checkpoints that were trained under various rotations Section 1. we estimate [| by averaging the L, norm of sampled rows. While this could induce a large variance from the sampling of rows, we find that variations of the L norms from rotations are fairly homogeneous across rows.
d
C
Additional Results
C.1 Main Experiments
We provide additional results from our main line of experiments.
VIT/S (ImageNet). Figure 14 extends the results from Figure 1b with validation loss and accuracy
Skee telaite
Giabal Butatinn
Training Loss
Wat
Mas
No Retatio
No Reratoo
Validation
Valldttion
-2
LaperWhin
lapskWaa
Rondiiss
Epoch
Epoch
Egoch
Figure 14: Simple Imagenet training loss, validation loss and top-1 validation accuracy
26

---

ResNet50 (ImageNet). Figure 15 demonstrates that Adam maintains its performance well under rotational transformations for ResNets. This robustness to rotation implies that Adam gains little advantage from the standard basis structure in this setting. This finding aligns with the fact that SGI with extensive tuning can outperform Adam when training these networks.
Eiiba Raziiv
Tmining Loos
Mike
Outped Wike No bubidion
No Ricoidin
Validation
Valléation
boret Wane
Outpoe wao No Reeatien
Epoch
Epoch
Epoch
Figure 15: Training loss, validation loss and top 1 % validation accuracy, when training ResNet-50 with Adam on ImageNet across different scopes rotations.
C.2 Architecture Aware Rotation.
We seek to identify whether certain transformer layer types are more sensitive to rotations than other, and contribute more to the overall performance degradation observed in Figure la when using layer-wise rotations.
Figure 16 shows the loss curves when rotating only one layer type at a time. We find that the performance degradation induced by layer-wise rotations is small for most layer types, seemingly balanced across these layers, with the exception of value and embedding layers.
Layerwise rotation of value layers seem impact the loss more noticeably than with other layer types. In Figure 18, we find that reducing the scope of rotations to output neuron wise does not improve the performance when rotating value layers.
The biggest drop of performances is observed for embedding layers, which we conjecture to be linked to the discrepancy in frequency across tokens. Figure 17 shows indeed that when rotating the embedding layer by output neuron (i.e., within weights corresponding to a same token) the degradation becomes unnoticeable.
3,2000
rolation
embedding
layer
Training Loss
3.1625
key
query
value
attention projection
3.1250
mlp
mlp projection
3.0875
Iteration (k)
Figure 16: Layer-wise rotation applied to only specific layer types
27

---

32000
mitatin caibelding layer wide embedddnng outpuiziise
robatinn value laycr vise
vallus outpul--oo
Training Loss
5.1925
Training Loss
3.1925
3.1250
3.1250
3,0873
3,0875
3,500
3,8000
10
18
22
26
30
14
18
22
26
Iteration (k)
Iteration
Figure 17: Layer-wise rotation and output-wise
Figure 18: Layer- wise rotation and output-wise rotation attention values only
rotation embedding layers only
C.3
Hessian Rows
We use the end checkpoint of GPT-2 to sample rows from the Hessian in different rotated parameter spaces (see Section 4.
From Figure 19 to Figure 25, we present the same figure as in Figure 9, but for rows taken from different layer types, confirming that the behaviour we observed is consistent across parameter types. Except for embeddings, rows are always taken from the second Transformer block.
No Rotation
Glubal Rolation
Outpul-wise
SVD Rolation
Aboalte Value (Log)
82
10
10
70
20m
10r1
10r1
28m
32:19
10-19
10:1
10-17
32:12
Neuron
Layer
Neuron
Layer
Neuron
Figure 19: Hessian value distribution of a row the embedding layer.
No Rotation
Global Rotation
Output-wise
SVD Rotation
Absolue Value (Log)
52.
10
30-1
33
10
10-1
33
12
10-
10:91
16-1
12-9
10:9
18:1
12:0
53
10-44
30-4
10
10-4
Neuron
Layer
Other
Neuron
Layer
Other
Nearon
Layer
Other
Neuron
Layer
Other
Figure 20: Hessian value distribution of a row the second Transformer layer norm layer.
No Rotation
Global Rotation
Output-wise
SVD Rotation
Absolue Value (Log)
10
1n
Io-4
18
10
10mm
Ipon
10-h
1pex
10-9
30-0
12-0
10-4
30-4
1-
10-4
Neuron
Layer
Other
Neuron
Laysrr
Other
Neazon
Layer
Other
Nouron
Layer
Other
Figure 21: Hessian value distribution of a row in the second Transformer key layer.
28

---

Rotation
Global Rolation
Output-
SVD Rotation
Absolue Value (Log)
10-4
Lo
10-
Lone
Layer
Layer
Layer
Nouroa
Layer
Figure 22: Hessian value distribution of row in the second Transformer query layer.
Global
Output-wise
Rotation
Absolue Value (Log)
10-4
Neuron
Layer
Other
Neuron
Layer
Other
Nouron
Layer
Other
Neuron
Layor
Other
Figure 23: Hessian value distribution of in the second Transformer value layer.
Global
Output-wise
Rotation
Absolue Value (Log)
In-4
38-t
19
10:91
38-11
19
Neuron
Layer
Other
Nouron
Layer
Other
Nouron
Layer
Other
Neuron
Layer
Other
Figure 24: Hessian value distribution of in the second Transformer mlp layer.
Output
Absolue Value (Log)
J8-1
1354
16-91
Jp-
1p-
10
Jp-n
10-m
13
Meuron
Layer
Other
Neuron
Layer
Other
Neuron
Layer
Other
Neuroa
Layer
Other
Figure 25: Hessian value distribution of in the second Transformer mlp projection layer.
Figure 26 shows row in the attention projection layer of the 8-th transformer block, showing our observations seem also consistent across depth.
No Rotation
Global Rotation
soluue Valuue (Log)
12--4
I0-
Ipan
33-
Ipen
33.
1p-lt
33-44
10-R
32-4
Nauron
Layer
Neurrnn
Layyer
Figure 26: Hessian value distribution of a row in the eighth Transformer attention projection layer.
29

---

Figure 27 uses checkpoints trained with the same rotations as the one applied to the Hessian. We find the same behaviour for no rotations, global and output-wise, but we find that with the SVD-rotated checkpoints, there is increased variance in the Hessian values outside of the layer.
No Rolation
Global Rotation
Output-wise
SVD Rolation
Absolue Value (Log)
10- 10-
33-1
13-
10
3am
10m
10.
1m-44
10-10
10:13
19:14
10:44
10-11
10-0
12-44
10/11
10-14
16-M
12:44
Neuron
Layer
Other
Neurom
Layrr
Other
Neuron
Layer
Other
Neuroa
Layer
Other
Figure 27: Hessian value distribution of a row in the second Transformer attention projection layer from checkpoints that trained with different rotations.
C.4 Update orthogonality
We aim to measure the orthogonality of the update. Since weight decay is applied directly in parameter space rather than the gradient, the update rule (ignoring the bias corrections) becomes:
M.
W - W. = R.R
aLWW,,
(7)
Vi+c
where
M. M -1 -1- (1-B))RVZ(W)
Ve 32 V.Vi + 81) (RVL(W) RVL(W)).
To isolate the effective update direction, we rewrite the expression as:
W r+1 a,
M.
R
=
(8)
at
VV
Coefficient of variation. Suppose matrix A has singular values {s;}. For a scale-invariant measure of orthogonality, we minimize the normalized root mean square deviation between singular values and constant,
1
min
Σ 8: a)) P
where μa n Σ 8; is the mean {s:) The normalization facilitates the comparison between data with different scales. The solution for this objective is the coefficient of variation (CV), defined as
CV(s:)
where ds the standard deviation of the singular values. CV captures the relative dispersion of the singular values and is invariant uniform rescaling of A. This makes it especially suitable for comparing updates or transformations that differ in magnitude but share underlying structure. Since an orthogonal matrix has all singular values equal to 1, lower CV indicates that the singular values are more tightly clustered, suggesting that is closer to being orthogonal up to a global scaling.
In Figure 10, we provided a mean variation of the singular values in the update for each layer type, averaged across the depth of the network using GPT-2. We now expand this and show the variation for each individual weight matrix in the network. We display the coefficient of variation of the singular values of each linear layer's 's update. Each of the GPT-2 model's 's twelve encoder layer has six linear layers, four in the attention layer after splitting (to compute Q K, V and the output projection O) and two in the MLP block (labeled W, and W2). We compute the SVD of each update, and compute the coefficient of variation of the singular values and show the results in Figure 28. For completeness, we also show this result the beginning and end training in Figure 29 and Figure 30 respectively. The trend is clearer and steadier at the end of training.
30

---

Other metrics. In our experimentation, we considered other metrics to measure orthogonality, with each finding similar results.
One alternative method to measure how far the scaled singular values are from
10 u8s - 1)2
==1
where a is a scaling factor shared across the singular values s; of the update. For a set of singular values .s....., S, the optimal scaling factor is
We display measures at the start, middle, and end of training in Figure 31 respectively. We found this to be strongly correlated with the coefficient variation of {s:}-
Similarly, another measure of semi-orthogonality is how close AA to the identity matrix I. The objective being
1 |AAT all
aA CV()),
min 1
where À = of are the eigenvalues of AA T and σд and are the standard deviation and mean of X.
To see this, since A T is symmetric, so it has an eigendecomposition by the Spectral theorem, i.e.
AAT T = QAQ where Q is an orthogonal matrix. Then,
QAQ al
QAQ - -aQQ"
QAQ - -AQ-
Q(A aIQQ
Then, using the circulant property of the trace, we have
|| Tr ((Q(A - - allQ
Tr (Q al) aI)
QI) al)Q
Tr (Q(A - -a) (
Tr (Q- Q(A al) (A aI)
(A
Tr ((A - al) (A al)
al))
Σ a)² t=1
no
Substituting optimal a*
The optimal a in this case is μxa The normalization by mean assures scale invariance, and yields coefficient of variation of A. We show this metric in Figure 32.
Instead, the scaling factor is used on the symmetric matrix AAT T, we have the objective
min
aAAT
I|
which is already scale invariant and yields the optimal scaling factor a En
5TH X. We also
d
plot this metric in Figure 33.
31

---

Additionally, Liu and Su [2025] use metric inspired by the signal processing literature called the SVD entropy study Muon updates, which is defined follows.
H(s) (ng
log
j==
We also computed this metric, and it again strongly correlated with the coefficient of variation. We display measures at the start, middle, and end of training Figure 34 and note that larger is better for this metric, in contrast to the others we consider. While all metrics showed roughly the same trends, we focus on the coefficient of variation of singular values for simplicity.
D Optimization Algorithms with Rotations
remind here the SGD-M algorithm Algorithm 2, AdamW algorithm (pseudocode) in Algo- rithm 3, and provide rotated version in Algorithm 4.
Algorithm SGD Momentum Optimization Algorithm
Require: a: stepsize
Require: 8: momentum parameter
Require: weight decay coefficient
Require: f(0): stochastic objective function with parameters 0
1: Initialize 0g. f 4- 0
2: while 0, not converged do
3: + +
4: g: (
Get gradients w.r.t stochastic objective timestep t
5: 0, - 01-1 agt + 3(0,-1 0(-2) - a20,-1
Update parameters
6: end while
7:
return 0,
D Return the final parameters
Proof of section 2.1. Using the notation of Section consider SGD with momentum with learn ing rate V), momentum parameter (3, and fixed batches B;:
W++1 A(( (...).)... f.t) W. fVB (w)) B(w, wi-1).
By the chain rule, V ()(R) (R)
Br (w,) RV B. ( Wt), hence:
w [+1
(R.
= A([Rw) Ji=o....t ((R).t)
Rw: RVVB, (R Rw:)) + B(R,, Rw--1)
Rw; - RRV SR, (we) + (.(RR.: (.W::WW)) wr-1)
Rw
matching Definition 1.
While the SVD rotation we use in the AdamW algorithm can be represented as in Algorithm mathematically for specific choice Rt for clarity and to match our implementation, we write SVD rotated Adam W in Algorithm 5. In our experiments, the SVD update frequency f was set to 250 steps. note that while the other rotations we study are written as matrix-vector products, the SVD rotation is written a left and right matrix product the gradient matrix. These be shown to be mathematically equivalent, but we clarify given the standard practice of writing the gradient as vector.
We consider the heavy ball formulation here [Polyak, 1964], but the same would hold for Nesterov Accelerated Gradient [Nesterov, 1983].
32

---

Lanez
Singular
E H E.
1000 SOW
LONUSIER MOO son
Step
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 28: Singular value variation during training. We measure the variation of the singular values of each update of AdamW under various rotations for every linear layer in each GPT2 encoder block. While not universal, we find that majority of the time, the SVD rotations leads the lowest variation, while the global rotations leads to the highest. Additionally, we see that recomputing the SVD matrices (at 50000 and 50250 steps) leads downward spike the variation.
33

---

W/g
We
14
a2 X
EEEERT
Stogulor
Layse
Lowe
Lieee
Layer11
Layer
010
200
Steg
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 29; Singular value variation at the beginning of training. While the variation increases
initialization, we see it begin to trend downwards, notably after the SVD is recomputed.
34

---

We
Wy
MI.P MLP
Lays1
Lays
Lopve
Lapet
Laper 6
Stopular Laye:
Loye
Lase
Laye
Laya 11
Laya 12
ctse Oade come
wase mon come
dooe From waw 1ee
Step
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 30: Singular value variation the end training. Except for few layers, we see the variation hold relatively stably aside from when the SVD is recomputed.
35

---

Wo
Wv
MLP W1
1.0
0.8
0.8
0.88
0.6
Singular Value Variation
0.66
0.4
0.2
0..4
200
400
100
200
400
100
Wx
Wo
MLP W
1.0
0.9
0.8
0.8
0.6
0.6
0.7
0.4
0.6
0.2
0.5
0.2
400
100
200
400
180
Wo
MLP
0.60
0.55
0.20
Singular Value Variation
0.50
0.5
0.45
0.40
0.10
50000
50100
50200
50000
50600
50500
50020
50100
50200
50300
50400
50500
50000
50100
50000
50300
50600
50500
Wo
M W
0.60
0.55
0.55
0.50
0.50
0.45
0.45
0.15
0.49
0.40
Wg
Wy
MLP W1
0.6
0.6
Singular Value Variation
0.4
0.10
100000
L00100
100200
J00300
100400 100500
100000
100100
100200
100800
300000 100500
100000
100100
100200
100300
100600 100500
MLP W
0.55
0.55
0.50
0.3
0.50
0.15
p Mmmaam
0.45
0.40
0.40
100000 100100 10ID00 101DIDD 300000
100501
100000 100100 100200 101000 100500 100500
100020 100105 100200 100300 1006DO 101800
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 31: An alternative variation metric for singular values described in Appendix C.4 throughout training, averaged over network depth. We see similar results to the coefficient of variation.
36

---

Wo
Wy
MLP W1
20
20
15.
20
15
10
10
Eigenvalue Variation
100
2:00
400
Wx
Wo
MLP
25
15
10
15
10
10
1.00
1.00
2,00
200
400
Wg
MLP
5
4
Eigenvalue Variation
1.0
2
1.5
50000
50190
59209
Wx
52300
50400
50500
50000
50100
50200
Wo
50500
50400
50500
50000
50150
50200
MLP W22
50000
50000
50500
3.0
2.5
2.0
2
Wo
Wy
MLP W
2.0
3
Eigenvalue Variation
2
100000 100100 100200 100300 300400 L00500
100000
100100
100200
100300 100600 100500
100000
100100 100300,
100300
100400 100500
W2
3.0
2.5
2.0
1.5
100000 100120 130200 100300 30DA0D 101500
100020
100100 100200 100300 100300 102507
150000 100100 300200 100300 100400 103500
No Rotation
Rotation
Output Rotation
SVD Rotation
Figure 32: The coefficient of variation of the eigenvalues deseribed in Appendix C.4 throughout training, averaged over network depth. We see similar results to the coefficient of variation.
37

---

Wo
MLP
1.0
1.00
1.0
0.9
Eigenvalue Variation
0.85
0.5
0.880
100
200
300
400
500
100
230
300
400
500
300
400
500
Wx
Wo
1.0
0.95
0.7
0.85
0.8
0.6
0.7
0.5
Wg
Wy
MIP W1
0.95
0.90
0.9
0.7
0.85
Eigenvalue Variation
0.75
0.4
0.70
50000
50100
50200
50000
50600
50500
50000
50100
50200
50300
50400
505.00
53080
501000
50000
00300
50000
50500
Wo
MLP W2
0.90
0.9
0.85
0.80
0.75
0.70
0.4
53000
50200
392U3
SCON
50400
50500
30000
30100
50200
SUSOU
30600
30500
50000
50102
50200
50200
Wo
Wy
MLP W
0.9
0.8
Eigenvalue Variation
0.7
0.7
0.4
0.3
200300
Wr
MLP
0.90
0.85
0.80
0.8
0.75
0.70
0.7
0.65
100000 105100 10100O
100360 130400
102500
100000
100100 100200 10100O 100400 100320
100000 1000D0 100200 100300 102-DO 101500
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 33: The variation eigenvalues of the scaled AAT described in Appendix C.4 throughout training, averaged over network depth. We see similar results to the coefficient of variation.
38

---

Wo
Wv
MLP W1
0.8
0.6
0.6
0.4
0.4
SVD Entropy
Wx
100
Wg
MLP W
0.8
0.8
0.6
0.6
0.6
0.4
0.22
0.2
100
200
300
400
500
100
2000
300
4.00
500
1.0
200
300
400
500
Wg
Wy
MLP W1
0.85
0.85
0.80
0.80
0.75
SVD Entropy
0.70
0.90
50000
50300
50200
50300
50600
50500
50000
50200
50000
50000
50500
50000
50100
50200
50000
50600
WK
Wo
MLP W
0.85
0.950
0.925
0.80
0.80
0.900
0.75
0.75
0.875
0.850
30500
Wo
Wy
MLP
0.96
0.85
0.80
0.92
0.75
0.90
SVD Entropy
0.70
100000 L00100 100200
Wr
100300
100400 Lo0G00
100000
100100
100200
100300
100400 100500
100000
100100
LO0200
MLP W
100300 100400 100500
0.95
0.875
0.850
0.90
0.80
0.825
0.75
0.85
0.800
0.70
0.775
0.80
No Rotation
Full Rotation
Output Rotation
SVD Rotation
Figure 34: The SVD Entropy metric described in [Liu and Su, 2025] throughout training, averaged over network depth. We see similar results to the coefficient of variation, noting that higher is better for this metric.
39

---

Algorithm 3 AdamW Optimization Algorithm
Require: a: stepsize
Require: [1, f2 E [0, 1): exponential decay rates for moment estimates
Require: A: weight decay coefficient
Require: e: small constant for numerical stability
Require: f(0): stochastic objective function with parameters 0
Initialize 0,. ma + O, Vo <- 0, +
2: while 0, not converged do
3: f- ++1
4: ge + Vole(Or--)
Get gradients w.r.t stochastic objective timestep t Update biased first moment estimate D- Update biased second raw moment estimate D Compute bias-corrected first moment estimate Compute bias-correeted second raw moment estimate Update parameters
5:
m; f 3,m,
+ (1-Bi)
6:
+ f2V:-1 + +(1 82)gi
7:
m, - m,/(1 (13))
8:
Vi vi/(1 (15)
9:
O₂ < 00-1 -am-amiiiivve -
10: end while
11: return Oz
Return the final parameters
Algorithm 4 AdamW Optimization Algorithm with Rotation
Require: stepsize
Require: B1, .2 E [0, 1): exponential decay rates for moment estimates
Require: A: weight decay coefficient
Require: e: small constant for numerical stability
Require: f(0): stochastic objective function with parameters
I: Initialize 0g. me + O, Vo < O, t +0
2: while not converged do
3: t + t 1
4: g: Vofi
Get gradients w.r.. stochastic objective timestep b Apply rotation to gradients Update biased first moment estimate D. Update biased second raw moment estimate D Compute bias-corrected first moment estimate Compute bias-corrected second raw moment estimate Update parameters
5:
ge = Rg:
6:
m, t -mm
(1 (1-Bi)
7:
+ 32V;-1 + +11 (1- 82)gi
8:
m; m,/(1 -B9)
9:
V. v B3)
10:
0, - 01-1-aR aR (aR-imm/ (m./(vVi <)) a10,-1
end while
12: return O,
Return the final parameters
E SVD Rotations and Muon
Recently, Jordan et al. [2024] proposed Muon, optimization algorithm for the internal linear layers of neural networks. This algorithm departed from various modifications of Adam on favour using an ""rthhooonaized"" matrix update. We write the Muon algorithm in Algorithm 6. We note that simpler version of the algorithm described in [Jordan et al., 2024], however, their implementation and the description in subsequent work is described in Algorithm 6. We additionally make notational switch to emphasize that Muon acts only on the matrices of internal layers, Jordan et al. [2024] recommends using different update scheme (e.g., Adam W) vector-valued parameters such as bias vectors or NoyerNorm parameters (along with the embedding and prediction layer in transformers). We write parameters at time as O and their gradients as G₁-
Muon's normalization and orthogonalization step aims to drive the singular values of the update towards one. That is, if B' has the Singular Value Decomposition USV MMon aims to have the update approximate UV The approximation is computed through an iterative algorithm inspired by the Newton-Schulz method. We show that under simplifications, this is the same update recovered by SVD Rotated Adam(W), If we let 3 32 update with rotation, the numerator and denominator terms become the singular values the gradient matrix, which cancel, and the rotation back to the original basis leaves with UVT, Mathematically,
O) in Adam(W) and compute single step
40

---

Algorithm 5 AdamW Optimization Algorithm with SVD Rotation
Require: a: stepsize
Require: 81, B2 E [0, 1): exponential decay rates for moment estimates
Require: A: weight decay coefficient
Require: e: small constant for numerical stability
Require: F:a frequency at which to update the SVD matrices
Require: f(0): stochastic objective function with parameters
I: Initialize 0b. ma t- 0, Vo <- 0, t +
2: while not converged
3: t 1+1
4: gr - Va (0_1)
Get gradients w.r.. stochastic objective timestep
5:
i mod F 0 then
6:
U.S. V - SVD(g:)
P Calculate the Singular Value Decomposition of gr
7:
end if
8:
g. = ==UT.. g: V
b Apply rotation to gradients Update biased first moment estimate Update biased second raw moment estimate > Compute bias- corrected first moment estimate Compute bias-corrected second raw moment estimate Update parameters
9:
m: 81m4-1
+ (1 3:)
10:
Vi 32V1-1
32)gi
11:
m. t mi((1 1 89)
12:
Vi vi((1 85)
13:
02 + 01-1 (
aA0,-1
14:
end while
15: return 0,
Return the final parameters
Algorithm 6 Muon Optimization Algorithm
Require: stepsize
Require: ju: Nesterov momentum parameter
Require: A: weight decay coefficient
Require: 6: small constant for numerical stability
Require: f(O): stochastic objective function with parameters O
I: Initialize Og. Bo + 0, + 0
2: while O, not converged do
3:
t --tt 1+1
4:
G, + @
Get gradients w.r.t. stochastic objective timestep
5:
B. < μBB-1 + G2
Update momentum buffer Apply Nesterov Momentum
6:
B' < μB, + G.
7:
B. <- B +()
D Normalize the update Approximately orthogonalize the update Update parameters
8:
O, + NewtonSchulz5(B)
9:
θ, t 0,-1 -0; a
10: end while
11: return θ
Return the final parameters
let USVT be the SVD of gradient G. Then, the SVD Rotated Adam(W) numerator becomes M U GV UTUSV
Similarly, the denominator is the entry-wise square of the rotated gradient, which leaves us with V S². Then, computing the update M/v leaves us with S/S which is the identity. The final step in the algorithm is to rotate this update back, which is done by UIV leaving us with the Muon update.
While this setting is oversimplification (the momentum parameters often crucial for perfor- mance), does offer an interesting connection between Adam's 's update in a different basis and more recent algorithms like Muon or Shampoo [Gupta et al., 2018).
F Common Assumptions in First-Order Optimization Theory
We present non-exhaustive summary of common assumptions used in theoretical works for first- order optimization, see Table 2. For each assumption, we indicate whether it is rotation invariant.
41

---

Assumption
Inaation-Invarant
(Strong-) Convexity
Polyak-Lojasiewicz [Polyak, 1963]
Convexxx [Guille-Escuret al., 2020]
Quadratic Growth [Goujaud et al., 2022]
L-Smoothness (I-g norm) [Défossez et al., 2022, Zhou al., 2024]
Gradient Growth Condition [Zhang et al., 2022]
Bounded Expected Gradient Squared Norm [Zou al., 2019]
(Lo, L mSmoothnesss [Li et al., 2023b]
Restricted Secant Inequality [Guille-Escuree
al., 2022]
Error Bound [Luo and Tseng, 1993, Guille-Fsscuret et al., 2024]
L-smoothness (Loo norm)[Guo et al., 2022]
Coordinate-wise (Lo, L-momoothhness [Crawshaw et al., 2022]
Coordinate-wise "Affine" Variance Noise [Li and Lin, 2024]
Bounded Gradient (Loz) [Reddi al., 2018]
Table 2: Common assumptions involved in first-order optimization algorithm, indicating whether Rotation-dependent assumptions are comparatively rare in the literature.
they are
42


Reviews:
[{"review_id": "24NXz18oZD", "reviewer": "Reviewer_z9Dc", "summary": "This paper empirically studies how different types of parameter space rotations—such as global, output-wise, and SVD-based rotations—affect the performance of Adam. Motivated by these observations, the authors revisit several rotation-dependent assumptions from prior work and demonstrate their limitations in explaining Adam’s behavior under various rotations. To address this, they propose a new metric: the scaled (semi)orthogonality of the update. They show that this quantity aligns well with Adam’s performance across different rotation settings.", "questions": "1. For the SVD-based rotation, it appears that running SGD with momentum in the rotated space is equivalent to the Muon update rule. Is this interpretation correct? Additionally, how does running Adam in the SVD-rotated space compare to Muon in terms of performance?\n2. Could the authors provide theoretical motivation or intuition for why scaled orthogonality should be considered a meaningful metric for understanding Adam’s behavior under rotation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\nThe paper conducts systematic and well-controlled experiments examining how different types of rotations—including but not limited to global rotation—affect Adam’s performance. This broad empirical scope is a valuable contribution.\n\n**Weaknesses**\n1. **Limited Novelty:** The core findings largely overlap with prior work, especially Xie et al. (2025), which shows that global rotation degrades Adam’s performance and provides a theoretical explanation via $ \\ell_\\infty $-geometry and convergence analysis. While the observation that SVD-based rotation improves Adam is interesting, it is unsurprising given that it effectively reproduces Muon’s update rule.\n2. **Lack of Theoretical Justification:** The proposed metric, scaled orthogonality, is presented as a better predictor of Adam’s behavior under rotation than existing metrics such as $ \\ell_\\infty $-smoothness or the $ (1,1) $-norm. However, this claim lacks theoretical support. In contrast to existing metrics, scaled orthogonality is not accompanied by any convergence analysis or formal guarantees on optimization performance.\n\n---\n**Reference**\n\nXie et al., Adam Exploits $ \\ell_\\infty $-geometry of Loss Landscape via Coordinate-wise Adaptivity, ICLR 2025.", "was_revised": true, "final_justification": "My original concerns were the limited novelty compared to Xie et al. (ICLR 2025) and the lack of theoretical development. After the authors communicated with the PC/SAC, I was informed to treat this work as concurrent to Xie et al., so I do not consider this limitation in my final evaluation. During the rebuttal, the authors provided additional discussion on the connection between Adam on an SVD-rotated subspace and Muon, which offers some theoretical insight but does not introduce new theory. Overall, the empirical observations are well controlled and interesting. However, I agree with Reviewer Giz7 that the ultimate utility of these results for theory-building or optimizer design remains unclear, and I therefore lean toward a borderline acceptance.", "rebuttal": "We thank the referee for their consideration of our work. We would like to bring up some crucial points in response to the referee’s concerns:\n\n---\n\n### **W1: Novelty with respect to [Xie et al., 2025]:**\n\nWe have communicated to the PCs evidence showing that our work could be treated as concurrent work to the ICLR 2025 reference (and their previous arXiv versions), and thus our novelty evaluation should not be negatively impacted by their paper. To stay compliant with the anonymity requirements, we leave it to the PCs to continue communicating on the topic with the reviewer and AC.\n\nBesides the above, the referee is concerned that our core findings overlap with [Xie et al., 2025]. On the contrary, we would like to point out that our work directly contradicts the findings from [Xie et al., 2025]. Indeed, their main claim is “the exploitation of nice $L_\\infty$-geometry is the key advantage of Adam over SGD”. They support their claim empirically by showing in [Table 1] that Adam’s performance is better when the (1,1) norm is lower. Although we find their work very valuable, we show experimentally that this correlation doesn’t generalize to other bases than the ones they considered, with random rotations by output neurons increasing the (1,1) norm while simultaneously improving Adam’s performance. Our work investigates an alternative quantity to link Adam’s performance to the properties of the basis.\n\nThe novelty of our work includes not only the study of Adam’s behavior under SVD-based rotations, but also the study of Adam’s behavior under different scopes of random rotation (layer wise, input wise, output wise), which brings significant differences (described in great details in Figures 18 to 33), and also the evaluation of several existing theoretical quantities under each of these rotations, leading to different conclusions from prior works. \n\n### **Q1: Link between Adam in SVD-based rotations and Muon:**\n\nThe closest existing method to running Adam in SVD-based rotations like we do is GaLore, but without rank compression. Although there are indeed some links between Muon and our approach in that they both tend to orthogonalize the update (similarly to SOAP and Shampoo), there are significant differences:\nMuon orthogonalizes the first moment at each step, while our approach updates SVD every T steps\nMuon uses Newton-Schulz (an approximative method) to compute UV, whereas we use SVD to compute U and V separately so that we can run Adam in the latent space.\nMuon orthogonalizes the first moment, whereas we compute SVD and apply it to the gradients\nUsing Adam in the SVD rotated space only orthogonalizes the update if you assume Adam behaves exactly like sign descent\n\n### **W2/Q2: Theoretical motivation for scaled orthogonality:**\n\nRegarding the theoretical motivation for scaled orthogonality, we agree with the referee that it should be explicitly mentioned in the main paper. We discuss the connection with Muon, but do not explicitly remind or point to Muon’s theoretical justification, which directly justifies scaled orthogonality as a powerful theoretical tool for studying the optimization of neural networks. We will update our submission to explicitly refer to the theoretical origins of Muon (and therefore the justification for scaled orthogonality), and remind them in an appendix section. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly. Here is a quick summary of the justifications:\n\n* Due to the sequential nature of operations in Deep Learning, the function value and the gradient with respect to higher layers only depend on the output of layer $L$, not directly on the weights of layer $L$. Thus, when updating the weights $W$ of layer $L$, one should control how much the update $dW$ impacts the possible outputs, rather than the weights themselves. In other words, instead of constraining the Frobenius norm of dW (like SGD does), we should seek to control the $L_2$ norm of the output shift $dy=(W+dW)x - Wx = dWx$. This is achieved by constraining the operator norm of dW instead of its Frobenius norm. It turns out that if the gradient $G$ has SVD decomposition USV, minimizing $dW^\\top G$ (i.e., the linear approximation of the loss) under a constraint on the operator norm of $dW$ has a closed form solution: $-\\eta UV$, which is the scaled orthogonalization of the gradient. It is the update that maximizes the reduction of the loss (under linear approximation) while constraining the worst-case shift of output. \n\nThis gives a theoretical justification to orthogonalizing the gradient like Muon does, and directly relates the scaled orthogonality of the update to optimization performance. Since Adam’s update orthogonality depends on the basis, it could be an explanatory factor of Adam’s variation of performance across bases, which we find to be empirically supported.\n\n---\n\nIn light of the above clarifications and changes, we hope the referee will reconsider their assessment of our work’s significance and novelty."}, {"review_id": "LRBQ7FxmCo", "reviewer": "Reviewer_c8DY", "summary": "This paper studies the effect of rotations of the parameter space on the performance of AdamW. They include four types of random rotations: global (of the entire parameter as a vector), layer-wise, output- and input-wise. They demonstrate empirically that all types except output-wise slow down training of GPT-2 on OpenWebText and ViT / ResNet-50 on ImageNet-1K, with global rotation hurting the most. They also propose a type of non-random SVD-based rotation which strongly speeds up the AdamW training (as measured in terms of the speed of the train loss decrease as a function of epoch). In a degenerate case (Adam with no memory or $\\epsilon$), this happens to coincide with Muon with no momentum or Shampoo with no momentum. \n\nA possible source of this speed-up is that the layer-wise update is closer to a semi-orthogonal matrix. This is investigated by correlating a \"closeness to semi-orthogonality\" metric with AdamW's training speed for different rotations: the SVD type makes updates closer to semi-orthogonal, whereas the global rotation makes them the least semi-orthogonal. \n\nIn addition, there are different assumptions in the literature used to investigate AdamW's performance or prove convergence, specifically, $\\|\\cdot\\|_\\infty$-bounded gradients, small $(1, 1)$-norm of the Hessian, near block-diagonality of the Hessian. These assumptions are investigated in the context of rotations.", "questions": "1. Why is \"Epoch\" replaced with \"Iteration (k)\" in Fig. 13? (Was it intended?)\n2. When comparing, say, $A A^T$ with a rescaled identity matrix (e.g. in Appendix C.4), is $A$ assumed to be wide and full-rank? If so, why is it a reasonable assumption for the matrices whose near-semi-orthogonality is checked?", "limitations": "Although limitations are discussed and referenced in the relevant section of the checklist, I would suggest adding them in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* I am not aware of such a fine-grained analysis of the effect of rotations on AdamW's training speed, and rotation-dependent assumptions used in the literature. The study includes rotations of different scopes, and both language and vision tasks are included. \n* Existing rotation-dependent assumptions are challenged, and a new metric is introduced which is the variability of the singular values of layer-wise updates (how close these updates are to semi-orthogonal matrices). This new metric appears original and promising. \n* The paper is clearly written. \n\n## Weaknesses\n\n* Only the typical hyperparameters of Adam, and one choice of batch size, are included. It would be interesting to see how robust the results are with respect to the $\\beta_2$ parameter. I understand that it may be unconventional to change $\\epsilon$, but technically, just as an illustration, the results should NOT be robust to $\\epsilon$ because changing it can transform Adam into SGD (e.g., https://arxiv.org/pdf/1910.05446). Maybe tuning $\\epsilon$ is too much, but looking at a couple of $\\beta_2$ values and batch sizes would not be unreasonable. Relatedly, statistical significance of the training speed comparisons is not really clear. \n* The performance of Adam is evaluated only in terms of the training speed: how quickly the training loss decreases as a function of the iteration number. The outperformance of Muon is typically reported in terms of time until reaching a certain validation loss [Jordan et al., 2024] \n* I think more evidence is needed to claim that better typical performance of SGD compared to Adam when training ResNets is explained by reduced sensitivity to rotations, whereas typical outperformance of Adam when training Transformers is explained by using the features of the standard basis. (At least it would be nice to carefully compare to SGD.)\n* The paper would benefit from careful proofreading. For example, I doubt that a matrix is called semi-orthogonal if all singular values are 1 or 0. (E.g., the zero matrix is not usually called semi-orthogonal.) The third picture in Figures 13 and 14 seems exactly the same which is likely a small bug.", "was_revised": true, "final_justification": "I think this paper provide a significant empirical contribution connecting to theoretical work. I further commented on the discussion with other reviewers in my response to the authors.", "rebuttal": "We thank the reviewer for recognizing our contributions, and we address their concerns below.\n\n---\n\n### **W1: Hyperparameter robustness:**\n\nRegarding robustness to the $\\beta_2$ hyperparameter, we have conducted an additional sensitivity analysis on the base AdamW run and under global rotation. We fixed all other hyperparameters but $\\beta_2$, with the additional runs using $\\beta_2=0.9$ and $\\beta_2 = 0.99$ to compare to the original $\\beta_2= 0.95$. \n\nIn the case of AdamW under global rotation, both increasing and decreasing $\\beta_2$ lead to inferior performance of the optimizer. This indicates that our original choice of $\\beta_2$ was near optimal with all other hyperparameters fixed for the global rotation. \nSpecifically, the globally rotated AdamW with $\\beta_2=0.95$  completed training with training loss 2.912, whereas with $\\beta_2$ being set to $0.9$ and $0.99$, the training completed with loss 2.9165 and 2.9167, respectively.\nFor the base AdamW, we found that decreasing $\\beta_2$  led to worse performance than the original run, while increasing it led to superior performance. While this does mean that base AdamW could have been tuned more, in all configurations of $\\beta_2$, the base AdamW showed superior performance to all configurations of the globally rotated AdamW. From our experimentation, we would expect further hyperparameter tuning to only widen this gap. \nSpecifically, our original run with $\\beta_2=0.95$ ended with a loss $2.896$, whereas $\\beta_2=0.9$ and $\\beta_2=0.99$ led to a loss of $2.905$ and $2.888$, respectively. \nWe regret not being able to visualize these results in the rebuttal due to Neurips' policy of not allowing images and PDF links, but the ordering of these loss values remains consistent throughout the 100 epochs. We plan to add these visualizations in future revisions.\n\nFor the rebuttal period, we prioritize budget for searching over $\\beta_2$ as we felt it would be more likely to alter performance. Regarding batch size, we refer the reviewer to other recent works [1] and [2] that primarily study the impact of batch size on the performance gap between Adam and SGD. For our focus on Adam’s performance under different coordinate bases, we do not expect the change of gradient variance to have a strong interaction with the rotations. However, it is possible that a larger batch size beyond a certain threshold would better recover to full batch properties that are basis dependent. We will investigate the impact of batch size in future revisions.\n\n[1] Teodora Srećković, Jonas Geiping, Antonio Orvieto. (2025) Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. arxiv/2506.12543\\\n[2] Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, Micah Goldblum. (2025) Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful. arxiv/2507.07101\n\n### **W2: Comparison metrics**\n\nOur aim in comparing Adam under various rotations was strictly to understand Adam’s dependence on the basis. While our SVD Rotated Adam algorithm does offer some improvements as a function of iteration number, we are not proposing this as an effective replacement for Adam or a competitor to Muon (although the two are connected, as we note in Appendix E). As we mentioned in our response to the reviewer wWFQ, we are planning to add a direct comparison to Muon for further clarity, but again are focusing on comparing training speed rather than wall clock time. As a quick note on the distinction between training and validation loss, the GPT experiments are conducted by doing a single pass over the training set, so the validation and training loss are equivalent in theory. That said, for completeness, we do observe the same patterns in the validation loss, where the final validation loss of global rotation is $2.929$ and the base run is $2.909$.\n\n### **W3: Re: ..more evidence is needed to claim that better typical performance of SGD compared to Adam when training ResNets is explained by reduced sensitivity to rotations…**\n\nWe agree that more evidence would be required if one were to make the strong claim that the only factor leading to a gap or lack of gap in optimization performance is sensitivity to the basis being used. Instead, we claim that the basis can have a significant impact, as shown in our experimentation. While it is certainly possible and likely that other factors, such as properties of the datasets and architectures, would also impact optimization performance differences, we feel our results showing that the choice of basis can both degrade and improve performance make a meaningful contribution.\n\n### **Mistakes/edit suggestions:**\n\nWe thank the reviewer for their careful read. We will fix the following in the next revision:\n* W4: The zero matrix should not be considered as semi-orthogonal, \n* W4: There is a mistake in Figure 13/14 where we uploaded the wrong figures for ViT validation loss and accuracy. Although Figure 13 (left) is correct, Figure 13 (mid) representing the validation loss is wrong, and Figure 13 (right) is just a duplicate of Figure 14 (right), which is for ResNet50, not ViT. Additionally, the \"Epoch\" replaced with \"Iteration (k)\" in Fig. 13 is also part of this mistake. We invite the reviewer to read our response to [Reviewer Jnyt](https://openreview.net/forum?id=KD4wgunbhO&noteId=QTuhcsNZQg) for a more detailed summary correcting this mistake,  and we will certainly replace the incorrect figures in future revisions.\n* Limitation 1: We will also move the limitation discussion to the main text and further discuss the possibility of other factors contributing to optimization performance differences.\n\n### **Q2: Re: Is A assumed to be wide and full-rank?**\n\nWe don’t make any assumptions on the matrix shape or rank, given that the parameter matrices vary in shape and the training process cannot guarantee any rank structure."}, {"review_id": "XYEqSxGWyg", "reviewer": "Reviewer_Giz7", "summary": "This paper investigates Adam's sensitivity to different rotations of the parameter space. First, by studying the effect of different types of rotations (global, layer-wise, output-wise, input-wise) on the performance of Adam and SGD in three settings (training a GPT-2 model on the OpenWebText dataset, a ViT/S on ImageNet-1K and a ResNet50 on ImageNet-1K), it shows that broader rotation scope leads to more degradation in Adam’s performance. Output-wise rotations don’t show significant degradation, and a structured SVD-based rotation, inspired by Zhao et al. (2024) improves performance. Next, the paper studies common assumptions used in the theoretical analysis of Adam in prior work, such as $\\ell_\\infty$-bounded gradients, Hessian block-diagonality and $\\ell_\\infty$ smoothness, and shows that they are not always correlated with Adam’s performance. Based on these observations, it then proposes a coefficient of variation in the singular values of the gradient metric which measures its scaled semi-orthogonality and correlates with Adam’s performance.", "questions": "Please see above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "In terms of strengths, the paper studies an important direction of understanding why Adam is effective via a novel perspective of looking at the effect of different types of rotation on its performance. It is well-written and easy to follow, and offers interesting insights about insufficiency of existing assumptions such as Hessian block-diagonality in analyzing Adam and proposes a new metric that measures scaled semi-orthogonality of the gradient. \n\nOn the other hand, the paper has two main weaknesses:\n\n- The experiments are conducted with relatively smaller model sizes. Given that one of the paper's contributions is a presenting a comprehensive empirical analysis on the effect of different rotations on Adam’s performance, it’s important that the experimental section is thorough. In the three settings considered in the paper, the hyperparameters are also kept fixed and the results don’t show any error bars. This should be addressed to make the results and conclusions more convincing.\n- In the current state, the work presents justification for optimizers like Muon, showing that more orthogonalized updates can lead to faster convergence. The proposed metric and its correlation with the performance looks like another way to quantify the observation that SVD-based rotations don’t cause degradations in Adam’s performance. While it can be useful for theoretical analyses for future work, it is unclear how it contributes new insights into motivating the design of other optimization algorithms. Based on the development of SOAP and Muon, it is evident that some type of orthogonalization helps speed-up convergence. So, analyzing this for Adam doesn’t seem to give new insights other than showcasing the difference and quantifying it.", "was_revised": true, "final_justification": "After going through the discussions and revisiting parts of the paper, some of my concerns were addressed and I lean toward acceptance now. I appreciate the paper's contribution (hence, learning toward accept). However, as mentioned in the comment to authors, I am unclear on the ultimate utility of this result for either theory-building or optimizer design (hence, not strongly supporting accept).", "rebuttal": "We thank the reviewer for recognizing our novelty, and we address their concerns below.\n\n---\n### **W1.1 Scale of experiments**\n\nWhile we agree that the models we train do not reach the scale used in industry, this work was conducted within the computational constraints of an academic institution. However, we argue that the scale of the models we train is sufficiently representative. Notably, the GPT2-Small (124M parameters) style model we use (NanoGPT) has been the basis for several previous and concurrent works throughout the literature on explaining the performance of Adam and related optimization methods. In particular, [1] and [2] use similar GPT2-style models while using a smaller dataset (Wikitext-103), and [3] uses the slightly smaller GPT-Neo variant. The same model is also used in [4], where the authors additionally conduct a wide array of experiments on other models that show similar results to their GPT2-Small model. The same is true of [5], as mentioned by Reviewers z9Dc and Jnyt. Given this trend of prior and concurrent work, we feel that the largest model we train is sufficient, but if the reviewer is interested in seeing any particular experimental setting, we would be happy to try to accommodate.\n\n[1] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie (2020a). “Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity”. In: International Conference on Learning Representations (ICLR).\\\n[2] Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang (2022). “Robustness to Unbounded Smoothness of Generalized SignSGD”. In: Neural Information Processing Systems (NeurIPS).\\\n[3] Yan Pan and Yuanzhi Li (2023). Toward Understanding Why Adam Converges Faster Than SGD for Transformers. NeurIPS 2022 Workshop on Optimization for Machine Learning. arXiv/2306.00204.\\\n[4] Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems, 2024.\\\n[5] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $\\ell_\\infty$-geometry of loss landscape via coordinate-wise adaptivity. ICLR 2025\\\n\n### **W1.2 Hyperparameters:** \n\nWithin the rebuttal budget, we conducted a small set of hyperparameter sensitivity tests on the choice of $\\beta_2$ as suggested by reviewer c8DY (See [“W1: Hyperparameter robustness”](https://openreview.net/forum?id=KD4wgunbhO&noteId=ZHQ8pWB4Pa) response). We regret not being able to visualize these results in the rebuttal due to the NeurIPS policy of not allowing images and PDF links. Overall, the results are consistent with what we observe in the original experiments. We will add these additional experiments to the next revision.\n\n### **W.2 Significance of contribution:**\n >..it is unclear how it contributes new insights into motivating the design of other optimization algorithms. \n\nTheoretical assumptions are often central to the design and analysis of new algorithms in both classical and deep learning optimization. In this work, we argue that the assumptions made in several prominent theoretical arguments on the superior performance of Adam do not correlate with that superior performance. These works often conduct theoretical analysis to justify the effectiveness of a new algorithm. This is the case in roughly half of the works we mention in Table 2, and a much more extensive list of proposed (and largely forgotten) optimization algorithms can be found in Table 2 of [1]. It is often the case that algorithms are developed based on vague intuitions of what may work in a specific setting, but lack performance guarantees and fail to generalize to other settings (for example, consider the 160 optimizers in Table 2 of [1]). Given the current scale and cost of training large models, we argue that rather than continue this “brute force” approach, newer theories and assumptions are needed.\n\nWe aim to remind the community that for their theory to predict the effectiveness of a new optimization algorithm, the assumptions they use in their analysis should be able to capture the differences between gradient descent and Adam. Our findings show that many common assumptions do not capture this difference, or do not correlate with downstream performance. Our first contribution to the development of new methods is a reminder of the insufficiency of these common assumptions when conducting analysis and development, and the second is to look for some problem characteristics that correlate with performance. We conducted a preliminary investigation into update orthogonality as a characteristic that we find to correlate. We additionally note that it is still often the case that theoretical analysis leads to new practical optimizers, for example, Muon builds off ideas introduced in previous work, such as [2] and [3].\n\n>Based on the development of SOAP and Muon, it is evident that some type of orthogonalization helps speed up convergence. …  analyzing this for Adam doesn’t seem to give new insights…\n\nWhile “orthogonalization” has been shown to help in very recent optimizers like Muon, there are many important questions left unresolved. There are limited theoretical explanations for why orthogonalization is beneficial (which is itself an interesting question to theoreticians), how it improves upon Adam, whether it is the best quantity to optimize for, or in which parts of the network it is helpful, and so on. The Muon optimizer specifically suggests still using Adam’s update in embedding, prediction, and layer-norm layers; it is unclear what the correct method is for regrouping and orthogonalizing tensors beyond two dimensions, such as in convolutional networks.\n\nAs Adam has been proven over time to be effective and widely used in practice, we contend there is still significant value in the study of its performance and specifically its dependency on the basis, and make the connection to the idea behind new methods like Muon. Our work highlights that the coordinate basis being used (represented through different rotations) impacts how “close” to an orthogonal update. In Appendix E, we additionally note that under simplification, AdamW with the SVD rotation is equivalent to Muon. We argue that insights like this can directly lead to the development and proper usage of new optimization methods. For example, SOAP [4] is equivalent to Adam running on a rotated basis, highlighting the importance of studying the impact of rotations on Adam.\n\n[1] Robin M. Schmidt, Frank Schneider, and Philipp Hennig (2021). “Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers”. In: International Conference on Machine Learning (ICML). Vol. 139, pp. 9367–9376.\\\n[2] Jeremy Bernstein and Laker Newhouse. “Old optimizer, new norm: An anthology.” arXiv preprint arXiv:2409.20325 (2024).\\\n[3] Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. arXiv:2405.14813, 2024.\\\n[4] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. ArXiv, abs/2409.11321.\n\n---\nWe kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available and happy to address any further questions during the discussion period."}, {"review_id": "msy3F02tMb", "reviewer": "Reviewer_wWFQ", "summary": "This paper shows that Adam’s well-known advantage over SGD is not a universal property of the optimizer itself but strongly depends on the coordinate basis in which training is performed. Through extensive experiments on GPT-2, ViT and ResNet-50, the authors demonstrate that random rotations of the parameter space can severely slow Adam, whereas SGD remains unaffected due to its rotation-equivariance. Conversely, a structured rotation obtained by computing a full-rank SVD of each layer’s gradient and running Adam in the SVD basis accelerates convergence beyond the standard baseline. Classic rotation-invariant theoretical assumptions and three popular rotation-dependent ones fail to predict these behaviours. The study identifies the (scaled) orthogonality of the layer updates—quantified by the coefficient of variation of their singular values—as a reliable indicator of Adam’s efficiency across bases, linking the findings to recent orthogonal-update optimizers such as Muon and SOAP.", "questions": "1.\tGiven that the success of optimizers like Muon and SOAP already highlights the importance of update orthogonality, can the authors provide a more intuitive explanation for why this property is so beneficial? And a more intuitive explanation about what is the underlying mechanism that allows SVD rotation to enhance Adam's performance?\n2.\tI found the discussion about SVD Rotations and Muon is interesting, could the authors further add Muon to the performance comparison in Figure 5 to show how it performs against SVD-rotated Adam?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength：\n1.\tThis paper is well written and provides clear empirical demonstrations of basis sensitivity.\n2.\tIt provides a systematic evaluation of existing assumptions, shows why three popular rotation-dependent hypotheses are inadequate.\n3.\tThe main idea of this paper can explain the benefit of recent good optimizers (Muon, SOAP)\n\nWeakness：\nThis is an overall interesting paper to deliver a good point with detailed experiments. However, the paper's conclusion lacks significant impact. Despite conducting numerous experiments, their connection to the main claims can be weak(refer to Q1). For example, the experiment on Hessian rows doesn't directly explain why the SVD-rotated space performs better. Additionally, the interesting finding in Figure 6 regarding the second-moment distributions is presented without sufficient analysis or discussion or conclusion.", "was_revised": true, "final_justification": "My questions about the emperiments on Hessian rows was addressed. I suggest the authors add some of the explanations to the intuition and figure 6 in the next version. However, considering the paper can only give limited intuition and no experimental comparisons to Muon, I still think its technical contribution is somehow limited to the community. I will keep my score 4 unchanged.", "rebuttal": "We thank the reviewer for recognizing our contributions and relevance to recent new optimizers, and we address their concerns below.\n\n---\n\n### **W: Significance of contribution:**\n\nRegarding the significance of impact, we claim the success of Muon and SOAP justifies more work studying how the choice of basis affects the performance of various optimization algorithms. Many optimizers are proposed and justified theoretically (for example, see Table 2 of [1]), but are rarely used in practice. We believe this is partially because the typical assumptions that are used in theory are not predictive of what makes a good deep learning optimizer. We find that the basis does matter, and therefore, many works basing their derivations on rotationally invariant assumptions are unlikely to predict good performance. In a more practical sense, recent works show that choosing a good basis can enhance performance in practice. SOAP [2], for example, “is equivalent to running Adam in a rotated space” as per the authors. We additionally note that the basis SOAP uses is different from that of the SVD rotation, while both improve performance, meaning there is likely room for further research into other bases that could lead to even superior performance.\n\n> The experiment on Hessian rows doesn't directly explain why the SVD-rotated space performs better. \n\nThe reviewer is correct that the Hessian row experiments do not explain the improved performance of SVD rotation. Our purpose with the experiment on Hessian rows is to 1) verify the block-diagonal dominance of Hessian, a rotation-dependent characteristic that is mentioned in existing literature; and 2) identify whether simplifying the Hessian to a strict diagonal is a sufficient representation. Our finding shows that even though the Hessian is block-diagonal dominant, the off-block diagonal elements still play an important role and should not be ignored, hence arguing this assumption is inadequate in explaining Adam’s behaviour under rotation, including under SVD rotation.\n\n>Figure 6 regarding the second-moment distributions is presented without sufficient analysis or discussion, or conclusion\n\nFigure 6 acts as a complementary observation that global rotations concentrate Adam’s second moments, leading to less variation in effective learning rates across dimensions. This loss of adaptivity directly accounts for the performance drop observed with global rotations.  While we cannot directly interpret why the second moment values are much more widely distributed under the SVD rotation, we can see that it leads to a much more varied effective step size across dimensions. We suspect this basis is allowing the algorithm to be “more adaptive,” but leave the underlying mechanism for future work.\n\n### **Q1: Intuition on SVD rotation enhances performance, and update orthogonality is potentially the correct metric:**\n\nA full explanation for SVD rotations enhancing performance is an active area of research, but we appeal to SVD Rotated Adam’s connections to Muon as discussed in Appendix E. Given this connection, an intuition for explaining Muon’s effectiveness should extend here. For a summary of the importance of scaled orthogonality in theoretical frameworks, which we will add to the appendix and refer to in the main paper, we refer to our reply to [reviewer z9Dc Q2](https://openreview.net/forum?id=KD4wgunbhO&noteId=24NXz18oZD), who raised a similar concern. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly. \n\n### **Q2: Add Muon to the performance comparison:**\n\nWe are also interested in the performance of Muon as compared to the various Adam rotations, but our initial runs have shown subpar performance relative to Adam. We suspect a more thorough hyperparameter search is necessary. Hyperparameter choices often used in GPT speedrunning ([3]) are optimized for a slightly different model (modded-nanogpt) and dataset (fineweb10B), in contrast to our use of NanoGPT and Openwebtext. We are currently conducting this hyperparameter search and hope to be able to report results during the discussion period.\n\n[1] Robin M. Schmidt, Frank Schneider, and Philipp Hennig (2021). “Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers”. In: International Conference on Machine Learning (ICML). Vol. 139, pp. 9367–9376.\\\n[2] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. ArXiv, abs/2409.11321.\\\n[3] https://github.com/KellerJordan/modded-nanogpt/tree/master"}, {"review_id": "vgmiSoOEER", "reviewer": "Reviewer_Jnyt", "summary": "In this paper, the authors empirically compare the training loss of Adam when the parameters are conducted with random rotations with respect to different spaces, and report a degradation of performance when rotations are involved. Based on this observation, the authors design some experiments to examine some assumptions that are commonly adopted in recent theoretical studies regarding Adam. They found that these assumptions can not explain the degradation caused by random rotations. In contrast, they propose that the scaled(semi-) orthogonality, alignment with the degradation caused by random rotations.", "questions": "Could the authors explain why they introduce two rotation matrices $\\mathbf{R}$ and $\\mathbf{R}'$ before line 192 and line 220. I do not find any discussions regarding why two matrices are required. In addition, there might be a typo in the definition of $\\mathbf r_i$. The current formula might indicate the $i$-th column instead of the $i$-th row.", "limitations": "See weaknesses.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\nBased on the recent success of Muon, it is interesting to investigate optimizers from the perspective of the spectral distribution and the orthogonality of layers. The empirical observation that the training with Adam would be correlated with scaled(semi-) orthogonality proposed in this paper is somewhat insightful.\n\n**Weakness:**\n- I do not feel that one of the main conclusions that the rotation degrades the performance of Adam is convincing enough. \n    - Compared to the training loss, people concern more about the generalization of the models obtained from the training. However, as demonstrated in Figure 13 and 14, there are no significant distinctions among different curves representing the validation loss and the validation accuracy. \n    - Even when considering only the training loss, the current experimental results(Figure 1) are insufficient to fully support the claim. **It is evident that after a brief initial phase, all training loss curves exhibit nearly identical patterns, appearing almost parallel.** This indicates that the loss reduction rate is consistent across conditions, with or without rotation. Furthermore, the losses do not appear to have converged after 100 epochs. Should training be extended over a longer duration, the initial stage's instability would likely have a diminished impact on the entire training required, thereby dramatically reducing the claimed ratio of training iteration slowdown mentioned in the caption of Figure 1. In addition, I am also concerned about the visual interpretation of Figure 1(a) due to the chosen scale of the y-axis. Given the very narrow range of the y-axis (approximately 2.87 to 3.16), differences that appear visually distinct are, in absolute terms, less than 0.02. This narrow scaling potentially exaggerates what might otherwise be considered negligible differences in training loss, making them appear more substantial than they are.\n\n- The foundational premise of discussions regarding Adam's assumptions, specifically that rotation degrades its performance, appears unconvincing. Consequently, this casts doubt on the validity of all conclusions and discussions presented in Section 4. However, even overlooking this fundamental issue, the basic logical framework of Section 4 itself lacks persuasiveness. Given my limited familiarity with the third assumption, my focus will primarily be on the first and second assumptions. Based on my understanding of theoretical studies concerning Adam (e.g., [1]), these assumptions are typically employed to provide theoretical convergence guarantees, reflecting worst-case behavior rather than an exact convergence rate. Furthermore, establishing an exact convergence rate for Adam remains exceedingly challenging; to my knowledge, no theoretical studies currently present such results, even under specific assumptions regarding data distributions. Therefore, to challenge certain theoretical assumptions solely based on observed loss convergence seems arbitrary, especially considering that theoretical results commonly accommodate discrepancies up to a constant factor, a phenomenon that exactly appears to be precisely reflected in your Figure 1.\n\n- While the concept of scaled (semi-)orthogonality proposed in this paper offers some insight, its rigorous translation into a beneficial assumption for establishing a robust theoretical framework remains unclear. More importantly, the methodology for constructing such a theoretical framework based on this concept is not articulated. Therefore, this point, in its current presentation, may not suffice as the major contribution of the paper.\n\n[1] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $\\ell_\\infty$-geometry of\n loss landscape via coordinate-wise adaptivity. In ICLR, 2025", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for their consideration of our work. We would like to bring up some crucial points in response to the referee’s concerns:\n\n---\n\n### **W1: Concerns about rotations degrading Adam’s performance:**\n\n### W1.1 Impact on validation:\nWe thank the reviewer for making us realize our mistake with the figures: we made an error in the labels used for Figure 13, which led to the wrong Figures being loaded. Although Figure 13 (left) is correct, Figure 13 (mid) representing the validation loss is wrong, and Figure 13 (right) is just a duplicate of Figure 14 (right), which is for ResNet50, not ViT. \n\nWe sincerely apologize for that mistake, which wrongly indicates that the differences in training loss were not extended to validation. Though openreview does not allow sharing figures, we will fix this error. The final validation accuracy for ViT is about 69% for original Adam, against 64% for Adam with global rotations. The original Adam reaches 64% validation accuracy in only 20/100 epochs, 5 times faster than with global rotations, an even wider gap than for training loss. \n\nTo summarize the validation accuracy after fixing the error in Figure 13:\n* For GPT, we train for much less than a full epoch, and training samples are never seen multiple times. Therefore, the training loss has the same distribution as the validation loss and shows a clear difference in performance.\n* For ViT, we observe a significant difference in training and validation loss, and an even greater difference in validation accuracy.\n* For ResNet, we see no significant difference with rotations. This supports our claim that Adam’s advantages over SGD are tied to Adam leveraging the properties of the canonical basis. Indeed, Adam is not superior to SGD for training ResNet, and precisely does not seem to be affected by the basis in this setting. Adam is sensitive to the basis in the settings where it performs better than SGD (ViT and GPT).\n\n### W1.2 Parallel training curves, non-convergence at the end, small loss delta: \nWe provide evidence that these concerns raised by the referee are natural artifacts of empirical training of transformer models at this scale, well documented in this type of setting, and do not contradict a large degradation of performance under global rotations: In fact, they make optimizer performances appear visually smaller than they are. \n\nThe training curves reported in the original blog post (Muon: an Optimizer for Hidden Layers in Neural Networks) by Keller Jordan (Figures 1 and 2) display the same ‘parallel curves’ that the referee is concerned about. Similarly, they also seem to have been stopped before converging, which, as explained below, is only in appearance. More strikingly, we invite the referee to consult Figure 2 of [2], which shows almost perfectly parallel lines, with a final difference between Muon and AdamW training losses of around 0.02. Given the clear success of Muon and the striking similarities with our experiment results, we hope the referee will reconsider the significance of these observations. We detail below the reasons for the artifacts noted by the referee.\n\nThe apparent non-convergence after 100k iterations (the x-axis is in thousands of iterations, our training does not complete a single epoch of the dataset) is a typical artifact of learning rate decay in LLM training. A large portion of the loss improvement is driven by the decay of the LR (allowing more fine-grained representation), but the LR reaches 0 by the end of training. There is no way to pursue training any further, and keeping the LR constant before it reaches 0 would lead to a near-stagnation of the loss. A striking example can be seen in Figure 10 of [1]: although the training curves appear not to have converged at 200k steps, increasing the training duration to 400k steps leads to a near identical final training loss. The referee can also refer to Karpathy’s NanoGPT Github repository, which our experiments are based on, as it provides some insightful comparison, including a training curve of GPT2 (almost identical setting to ours – except we improved hyperparameter tuning) trained over 400k steps. The range of the y-axis in this baseline training is the same as ours, and the referee will notice that the final loss is 2.905, which is worse than ours, even though we trained for ‘only’ 100k steps. As such, our final training loss after 100k steps indicates a strong final loss that would be hard to significantly improve simply by training longer. The same phenomenon also leads to training curves having near-parallel lines due to the similar impact of the LR annealing.\n\nAdditionally, evaluating the impact of a fixed shift in training loss can be misleading for LLMs, due to cross entropy and the naturally high noise in the data. For instance, from iterations 120k to 400k, Kartpathy’s GPT2 goes from loss ~3.0 to ~2.9 over 280k steps, and even this shift is largely driven by LR annealing. A shift of .02 at the same point of the LR schedule is thus significant. We refer to Figure 10 of [1] in which the optimizer introduced by the authors as superior to Adam achieves a .01 loss improvement (in an almost identical setting!) over AdamW, over 200k steps. We observe a degradation twice as large in half the training steps compared to the sizable gain claimed by the authors.\n\nAs a side note, Xie et al. [3] concurrently made the same observation as us: “Adam on the original loss converges the fastest as expected. But convergence of Adam on a rotated loss is much slower [...]”\n\n[1] Sophia: A Scalable Stochastic Second Order Optimizer For Language Model Pretraining, H. Liu et al., 2024, arXiv 2305.14342 v4\\\n[2] Muon is Scalable for LLM Training, J. Liu et al., 2025. arXiv 2502.16982 v1\\\n[3] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $L_\\infty$-geometry of loss landscape via coordinate-wise adaptivity. In ICLR, 2025\n\n\n### **W2: Logical framework of Section 4:**\n\nThe referee brings an important point. The goal of Section 4 is not to reject theoretical assumptions or their usefulness. These assumptions have their purpose in theoretical frameworks. The goal of Section 4 is instead to assess whether existing assumptions that are rotation-dependent (and thus capable of discriminating bases) can explain the variations in performance of Adam observed across bases. The minimal requirement is that when varying only the basis, a basis with the worst property should lead to the worst performances and vice-versa.\n\nInterestingly, the paper cited by the referee, Xie et al. [3], precisely uses the same reasoning as us to support the validity of the $(1,1)$ norm, and, by proxy, of the $L_\\infty$ smoothness that it lower bounds. See Table 1 of [3], which they use to justify the effectiveness of their quantity by observing that its values align with Adam’s performance in the canonical and randomly rotated bases. While we also observe this alignment for the canonical basis and global rotation, it breaks under a type of basis that [3] had not considered, obtained via output-wise rotations. For the same reason, they supported the effectiveness of their assumption by observing this alignment; our observation that this alignment does not generalize contradicts its effectiveness. \n\nVery importantly, this does not imply that metrics such as $L_\\infty$ smoothness are not useful. It simply invalidates its ability to capture the properties of the basis that Adam leverages to achieve its advantages over SGD.\n\n### **W3: Scaled (semi-)orthogonality as a beneficial theoretical assumption:**\n\nWe agree with the referee that this aspect should be explicitly discussed in the main paper. We discuss the connection with Muon, but do not explicitly remind or point to Muon’s theoretical justification, which directly justifies scaled orthogonality as a powerful theoretical tool for studying the optimization of neural networks. We will update our submission to explicitly refer to the theoretical origins of Muon (and therefore the justification for scaled orthogonality), and remind them in an Appendix section. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly. \n\nFor a summary of the importance of scaled orthogonality in theoretical frameworks, we refer to our reply to [reviewer z9Dc Q2](https://openreview.net/forum?id=KD4wgunbhO&noteId=rNhxJeyPDz), who raised a similar concern. We will also add this to the appendix and refer to it in the main paper.\n\n### **Q1: Difference between $R$ and $R’$:** \nThe reason for the distinction is to be able to distinguish the basis that is used for measuring a quantity, and the basis that is used during training. For instance, if $R’=$None and $R$=Global, we would use the checkpoints from a normal Adam training run, and measure, such as the $(1,1)$-norm of the Hessian in a globally rotated basis. The initial purpose was to separate the impact of training on a basis from the impact of measuring on a basis. However, we found that the significance of the results obtained with $R!=R’$ was not well justified. Reporting results that have no clear interpretation or meaning was confusing, so we removed them. We thank the referee for this remark, and we will remove this separated notation by a single dependency on R since we only report with $R=R’$. \n\n### Typo in $r_i$: \n\nThe reviewer is correct, and we will fix this for clarity, though amusingly, with the Hessian being symmetric, its rows and columns are the same anyway (up to transpose).\n\n---\nWe thank the reviewer for helping us improve the paper. We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available and happy to address any further questions during the discussion period."}]

Author rebuttal:

Meta-review: The paper considers the problem of understanding the performance of Adam. Given the popularity of Adam and a lack of rigor around it, the problem is clearly very important and of interest.

The paper shows two results:

R1. Across different architectures, Adam's performance is dependent on the basis.
R2. Across different metrics that have been proposed to understand Adam, orthogonality of updates has the most promise to explain the speedup that Adam provides relative to SGD.

Strengths:

1. The paper does a fine-grained and thorough empirical analysis of Adam.
2. While Adam being basis-dependent is known, the analysis points out various subtleties such as the type of rotation.

I will now go through some of the concerns that were raised:

1. The paper uses the final loss to compare setting, the rate of convergence appears similar for different settings: During discussion phase, reviewers were mostly satisfied with this, since it is standard practice in LLM optimization (even though there is some gap from usual theoretical analysis). Therefore, I will mainly ignore this for decision making.

2. Comparison with prior work: Xie. et al, 2025 was regarded as concurrent work. However, there is still the point about comparison with Muon. Perhaps the most important takeaway of the paper, about the orthogonality of the updates, is already incorporated in Muon, and is the main motivation for Muon.

3. A more rigorous link between rotations and Adam's performance: The claim that "the models for which Adam does not depend on the basis are precisely the models for which Adam is not superior to SGD" is not that well-supported. There seems to be a correlation, but a general claim about a model class (esp. about Adam vs SGD) that does not take other factors such as the properties of the data (such as heavy tailed behavior) into account can potentially be misleading. Therefore result R1 could have been better supported.

Overall, I feel this paper is borderline, though given the current lack of understanding of Adam and the potential to inspire future work, I lean in favor of acceptance.
Decision: Accept (poster)