SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training
Authors: Sahar Rajabi
Nayeema Nonta
Sirisha Rambhatla
Venue: NeurIPS 2025 poster ()
Keywords: large language models
memory-efficient training
low-rank optimization
gradient projection
Primary area: deep_learning

Abstract: Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of optimizer states. While recent work has aimed to reduce memory consumption, such efforts often entail trade-offs among memory efficiency, training time, and model performance. Yet, true democratization of LLMs requires simultaneous progress across all three dimensions. To this end, we propose SubTrack++ that leverages Grassmannian gradient subspace tracking combined with projection-aware optimizers, enabling Adam’s internal statistics to adapt to subspace changes. Additionally, employing recovery scaling, a technique that restores information lost through low-rank projections, further enhances model performance. Our method demonstrates SOTA convergence by exploiting Grassmannian geometry, **reducing training wall-time by up to 65%** compared to the best performing baseline, LDAdam, while preserving the reduced memory footprint.  Code is at https://github.com/criticalml-uw/SubTrack.

SubTrack++: : Gradient Subspace Tracking for Scalable LLM Training
Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla
Critical ML, Department of Management Science and Engineering, University of Waterloo (srajabi, nnonta, srambhatla] @uwaterloo.ca
Abstract
Training large language models (LL.M) is highly resource-intensive due their massive number of parameters and the overhead of optimizer states. While recent work has aimed to reduce memory consumption, such efforts often entail trade-off's among memory efficiency, training time, and model performance. Yet, true democ- ratization of LLMs requires simultaneous progress across all three dimensions. To this end, we propose SubTrack++ that leverages Grassmannian gradient subspace aware optimizers, enabling Adam's 's internal statistics to adapt to subspace changes. Additionally, employing recovery scaling, a technique that restores information lost through low-rank projections, further enhances model performance. Our method demonstrates SOTA convergence by ex- com-
tracking combined with projection-awee -wware
ploiting Grassmannian geometry, reducing training wall-time by up pared to the best performing baseline, LDAdam, while preserving the reduced mem- ory footprint. Code is at wtlhttp::// //ggth/bb...t../// com/ www.aaallmwwuuutaaieeeeeaaaeeeee
Introduction
LLMs have demonstrated state-of-the-art performance across wide range of tasks and are rapidly
growing in popularity. However, training and fine-tuning these models require significant resources,
including extensive hardware and time, which limits their practicality for many applications and
increases their environmental impact and carbon footprint. [Zhao et al., 2024 Jaiswal al. [2024]
Muhamed et al. 2024 Miles et al. 2024 Modoranu et al. 2024| Hao et al. 2024 Li et al. 2024]]
Several techniques have been proposed to mitigate memory bottlenecks (Chnn et al. 2016 Rajbhandari
al. 2020]. LoRA (Hu al. [02T] and other low-rank adaptation methods Dettmers al. 22244]]
Hu et al. al. 2021 Yaras et al. 2024 [ialin et al. [2023, |Rnduchintala et al. 2024 Xia et al. [0024] Miles
ct al, 2024] have gained popularity by optimizing reduced set parameters. Such approaches often
assume low-rank parameter space, which can lead to suboptimal performance. In addition, methods
like BAdam [Luo et al. 2024] and Block-LLM [Ramesh et al. 2024], utilize block coordinate descent
to optimize parameter subsets, achieving memory savings at the cost of reduced accuracy.
However, memory requirements extend beyond trainable parameters, with a significant portion consumed by the optimizer's 's states [Zhao et al. 2024]. Recent efforts have focused on reducing
this space while targeting full parameter training [L et al. 2023 Anil et al.) 2019 Lv al., 2024
Dettmers et al.] 2022 Zhang et al.) 2024 Modoranu et al. 2024 Zhao et al. 2024| Muhamed et al.
2024 Leveraging the low-dimensional nature of gradients during gradient descent (Gur-Ari et al.
2018, Schneider et al.) 2024 Yaras et al. 2023] Gal.ore (Zhao et al. (2024) reduces memory usage by projecting gradients into a low-rank subspace and periodically updating this approximation singular value decomposition (SVD). While SVD offers optimal low-rank approximation [Robert] cct al. 2025), it can pose several challenges. First, it is computationally intensive, and alternatives which use random projections [Zhu et al. 2025 approximation methods to estimate dominant
singular values Robert et al. 2025 Liang et al. 2024]. match or outperform SVD in practice.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Evaluation
Memory (GB) 1 Pira ctsl.,
Wall-Time
Fira [Chan di 3025]
Fim 2005)
Sub Trackecke+ [Ours]
SubTrackt+
Suh Track++
LDAdam
LDAdem (Buhent al, 2025)
LDAdam (Rebent 2025]
(Rubatt sl., 2025)
APOLLO [2t al, 2005]
APOLLO [2hu al., 2015]
APOLLO
12 at.
GoLorefZlaa 2024)
GaLore iZla 2024]
GoLore 2004)
(a)
(b)
(c)
Figure 1: We compare baselines on pre- pre-training 1B- parameter model. (a) SubTrack++ achieves the lowest evaluation loss across all methods. (b) Its peak memory usage is significantly lower than APOLLO and LDAdam, and on par with GaLore and Fira. (c) In terms of wall-time, SubTrack++ incurs minimal overhcad relative to APOLLO and is markedly faster than GaLore, Fira, and LDAdam. Overall, SubTrack++ outperforms all baselines in evaluation loss while matching exceeding them memory and runtime efficiency.
Moreover, SVD is sensitive noise (Vaswani et al. 2018 He et al. 2025 and tends to degrade in late training stages when gradients are small, often hindering convergence (He et al. 2025].
Geometry- based methods have shown strong performance in various machine learning applications (Zhang et al., 2018 Balzano et al., 2011 He al., 2011 Blocker al. 2023]; Grassmannian the manifold of all subspaces of dimensions in a space of dimensions d, and using Grassmannian for
subspace tracking has led structurally embedded information, lower computational complexity, and improved performance [Zhang et al. 2018 Balzano et al.) 2011 He et al.] 2011 Blocker et al. 2023
Chakraborty et al. 2017 Zhang and Balzano 2016). This line of work has demonstrated robustness and efficiency in high-dimensional. noisy environments [Zhang et al. 2018 Balzano et al. 2011
He et al., 201T Balzano et al.) 2018 Chakraborty et al., 2017]. Their natural robustness against perturbations and strong theoretical guarantees make them particularly well-suited to tracking the evolving gradient subspaces encountered in LLM training.
To this end, we employ subspace tracking on Grassmannian geodesics (Figure to develop geometry-based, time- and memory-efficien training method. This way, instead of recon- structing low-rank approximations via expen- sive SVD, we can efficiently leverage previ- ously computed subspaces and the estimation error to better adjust the projection. We also incorporate subspace shifts into Adam's 's first and second momentum update rules, ensuring proper alignment with coordinate changes ing a Projection-Aware Optimizer [Robert et al.) 2025). Additionally, recover and scale (Re- covery Scaling) the gradient information lost during low-rank projection by utilizing the scal- ing information of the low-rank, state-full opti- mizer Chen al. 2025). Zhu al. 2025).
Subspace Update Step
Projected Gradients First Subspace
Subspace
2nd Subepace
Projected Gradients
Second Subspace
Cork
Geodesic Line
Tangantnt Vector)
Spms,
New Point on the Grasamana Manifold
Spe-S... Geodesic
Visualization of Grassmannian subspace tracking: Between subspace updates, gradients are pro- jected onto a fixed subspace. The tangent vector V is computed via the derivative of loss function, measur- ing the subspace estimation error. The subspace is then updated by moving along the corresponding geadesic, determined by V minimize estimation error.
Figure 2:
To summarize, SubTrack++ is projection-awe aware geometry-based approach that supports full- parameter training and incorporates recovery scaling, offering superior time efficiency compared to SVD or PowerSGD methods (c.g., GaLore [Zhao et al., 2024 [hen et al. 2025 Robert et al. 2025), while maintaining GaLore's memory footprint. It also outperforms online PCA subspace tracking methods [Liang et al., 2024) and achieves SOTA convergence and evaluation loss across all strong baselines; with comparison provided in Figure and Figure
2 SubTrack++
Challenges of Low-Rank Optimization. Projecting gradients into a low-rank subspace reduces memory footprint and enables scalable LLMM training, but introduces important trade-offs. First, gradient subspaces require adaptive tracking; while $VD-based methods can capture these shifts [Zhao et al., 22024, Chen al. 2025). they are computationally expensive. To address this, recent
work has explored cheaper approximations and random projections [Robert et al. 2025 Liang et al.)

---

2024 Zhu et al. 2025 He et al. 2025 Furthermore, optimizers like Adam assume fixed coordinate system, and subspace changes must be reflected in their internal states for a consistent momentum updates. Finally, low-rank projections inherently discard some gradient components, that recovering and utilizing these discarded signals can boost performance [Chen et al. 2025| Zhu et al.. 2025).
Overview. SubTrack++ a memory- and time-efficient method, embeds geometric insights into low-rank optimization, and improves efficiency and performance via three core components: 1) Grassmannian subspace tracking that refines projections via estimation error and subspace history; 2) projection-awad aware optimizer which adapts Adam's 's state to account for evolving subspaces; and 3)
recovery scaling that restores lost information by scaling discarded gradient components.
Subspace Tracking. We frame subspace estimation selecting point on the Grassmannian, the manifold of all d-dimensional subspaces in an n-dimensional space [Bendokat et al.] 2024). This
perspective offers three key benefits: 1) It refines the subspace using prior subspaces and estimation error, avoiding full reinitialization as done in Zhha et al. (2024), [Chen et al.] (2025). He et al. (025).
2) Updates rely on lightweight algebraic operations. 3) Controlled subspace shifts improve robustness against noise and abrupt changes. Similar to GaLore [Zhao et al., [024]. our proof in Theorem 3.2
shows that applying the projection linear layers of LLMs preserves convergence while significantly reducing optimizer state memory. The initial subspace is computed using SVD as shown in (
Gg E mm*" is the gradient matrix at step 0; U. S, and V are its SVD components, and is the specified rank.
Go USV
(I)
At each step, gradients are projected onto the subspace of left singular vectors if m I n, and right singular vectors otherwise; optimizing memory usage [Zhao al. loss of generality, so the subspace is represented by S, ER
2024). We assume S n without (So [u,..... U,)) an orthonormal basis spanning the top-r directions. The gradient is projected as G. = S. E R=*" and the optimizer operates in this reduced space, significantly lowering memory and state overhead.
To account for subspace drift, the core subspace is updated every k steps (the subspace update interval) by minimizing the cost function (2) which measures its Euclidean distance the current gradient.
F(S.) = min S, 1
A
where A is the solution to the least squares problem. The derivative of 2 with respect to S, is given in 33), and the residual Rt = G. S.A lies in the orthogonal complement of S.. To update the subspace, we calculate the tangent vector F on the Grassmannian, as shown in (4) based on Edelman et al. [1998] where the second equality holds because R is orthogonal to S,S,T.
aF Ss.
G = -2RA = 2(S
(3)
VF= -I-IIISSA S.S. aF OF 2RA N U
(4)
as oS,
For optimizing the loss function 2 the subspace should moved in the direction of VF to reduce the estimation error. However, to control subspace changes, SubTrack++ computes rank- 1 approximation of F.. determined by its largest singular value and the corresponding singular vector obtained from its SVD, represented as ÜzÊ.V VT This approximation is then used for subspace update. As shown by Edelman al. [1998]. Bendokat et al. [2024), we can move along Grassmannian geodesic guided by rank-1 estimation of VF, with a step-size as presented in (5).
S (1) = VS.Vr ŪF) ( cos EEn sin
(- Epr
DFT + S.(1 - VF
(5)
This update rule preserves the orthonormality of St+1, ensuring it remains the Grassmannian. The last term in (5), projects the previous subspace onto the orthogonal complement of Vr, ensuring that the portion of S. which has not been updated in this step is still included.
Projection-Aware Optimizer. In Adam, the first and second momentum update rules are as shown in 6 and respectively (reminder: G is the projection of gradient into low-rank subspace).
M, < Bi M 1 + (1 - (1) G.
(6)
V4 B2 V1-1 (1 - 32) G
(7)
3

---

Algorithm +ubTrack++
( Subspace Tracking. Projection-Aware Optimizer Recovery Scaling
Regular Adam
Require: W.. G. Rm** with & (w.lo.g.), learning a, decay rates 81 and Bz, SubTrack+ step-size
T). rank r, subspace update interval recovery sealing limiter factor Ç. We use @ to denote Hadamard division.
So - U(;; : r]. where U,S, + SVD(Gu) for =
(Initializing First Subspace)
.., .,... do
if f mod k == then
Ga. arg A 1(S.-1A-G))) Go)l and R G. St-1Gir
VF = -2GG = V.
cos Epn sin
S. (S Ür)
V.T + St-1(I Vr
-in Spnn
Mi 81 (87 S M + (1 31)-
G S Ge: low-rank projection of G.
Vi 32 14.111 11 S1-1)² (Vi-1 _M3-)) (S) S. M 1M ) (2) Gi
else
= S
Mi +- B1 -M-M- + 31) G
Vi + 32 Va-1 + (1 - (1) GG
end if
T. Me 0 +e.
G, S.G0
(G optimizer's output, G,: projected-hack gradients )
(Gt),
de
At = di (Gt) S:G.)
We use 2 to denote Hadamard division.) )
[
if N
A Ç then As f-
|
Wi+ W a.c. Gr A
end for
Inspired by methods such LDAdam [Robert et al. 2025 we emphasize the importance of updating
optimizer states in projection- awarre manner to account for shifting subspaces; otherwise, misaligned projections distort the optimizer's performance. To address this, at each subspace update step, modify Adam's ' original update rules in (6) and (7). replacing them with projection-awant counterparts represented in (8) and (9), which reflect subspace changes into optimizer statisties (Robert al. 2025). Further details regarding these projections can be found Appendix C
Mf + 31 (ST St-1 Mc-1) + (1 81)- G.
V. V 82 (1- 82 (s. S.-1)² (V-11 M² SS- M,z)
S--1
82) द
(9)
These projection-awars update rules enables Adam optimizer to track optimization dynamics precisely as the subspace evolves, achieving significantly better practical performance.
Recovery Scaling, Optimizer outputs To = M, 2 Vv, + e ('O' denotes Hadamard division), which is then projected back via G. O to be used in weight update; however, low-rank projections inevitably discard some information in the full gradient matrix, that could enhance performance if properly utilized. Fira [Chen et al., (0255 observed that adaptive optimizers like Adam exhibit consistent scaling behaviour in low-rank and full-rank regimes. This suggests that the scaling information of the low-rank optimizer can be used to recover and rescale the discarded components of the gradient. A similar method is also employed in APOLLO (Zhu et al. 20025.. Consequently, an additional correction term is added to the standard weight update rule, as formalized in (1O).
W. + Wt-1 a G, a (G) S,G.)
(10)
Here is the learning rate and (G,) is the column-wiss scaling factor computed based the low-rank gradient representation, G, and the optimizer's processed output, Go as:
||G....
d. (G.), |.Gr.all
(11)

---

Following Fira's ' observation [Chen et al. 2025], we employ piandieen-clipppinnnnsiiiii mechanism
to stabilize training. Specifically, we limit the growth rate of A, (G(G))(G SS,;,) by factor Ç, and apply a correction whenever it exceeds this threshold as:
Aa
A
-11AA-1ll
(12)
A&]
By incorporating a geometry-aware perspective into low-rank optimization and applying this approach across all components of the training pipeline, SubTrack++ demonstrates robust and stable training, and achieves state-of-the-art performance while maintaining minimal memory footprint and wall-time. The overall flow of operations in SubTrack++ is illustrated in Algorithm
3 Theoretical Analysis
In this section, we analyze the convergence Grassmannian Subspace Tracking applied Sub- Track++ using theoretical analysis. To begin, the general weights update rule is follows:
d'=1-1
Wi Wo
Σ
G.
(13)
As previously mentioned, use left projection if m S n, where m and are the dimensions of the gradient matrix, and vice versa. Thus, Gp can be computed as shown in (14)
(S, if S Pr otherwise
Ge
(14)
Here, Sy is the projection matrix that projects the gradient onto the subspace, and Pe is representing the entry-wise regularizer used in the optimizer. If we use the full projection, then Ge will be computed as shown in (15;; where S% and Sr. are the rank-r left and right projection matrices.
G = SS T S
(15)
Definition 3.1 (L-continuity). A function ((X has Lipschitz-continiity (L-continuity) for any X1 and X2, [ f(Xi)ll I 2 - Xille
Theorem 3.2 (Convergence of Grassmannian Subspace Tracking). Suppose gradient has the following form with functions AG, B, and C₁ being L-continuous as per 3.7 with constants LA LB, and Le wrt. weight matrix We and |[, || I M; where We denotes the weight matrix at step and M is a sealar value,
G G= G=ΣA:
A: + +.D+BB.W B,W..
Now, define = ( B.W))) and Ct,t -- - (ST,) T C.WWW))... where S and ST are the rank-r left and right projection matrices; B; B.(,)) and C,(W)) denote the dependence of B; and C2 on the weight matrices W.. Further letting P; = S G.S,, and KG = E. Amin (m (C,,). where Xmin(.) (-) denotes the minimum eigenvalue over each batch, and N representing the number batch. Assuming that the projection matrices remain constant during the training. Then for learning-roo
of samples raaee rate and men((K)) (LA + M2),, subspace tracking. with pt 1 (the
element wise regularizer of the optimizer) satisfies:
Pelle $ [1 p( LA - [
That is, Pt 0 and it converges.
The proof of Theorem is provided in Appendix A based on Zhao et al. [024]. While both GaLore and SubTrack++ assume the subspace remains unchanged for the proof of convergence, GaLore must limit these updates to ensure convergence, as each update can potentially change the entire subspace. In contrast, SubTrack+ leverages rank-1 updates to the subspace, preventing drastic changes with each update. While a deeper analysis of slowly changing subspaces and their impact on convergence remains an open problem, in practice, this allows SubTrack+ perform more frequent updates.
Here we investigate the Grassmannian update rule presented in (5), which is direct application of Grassmann geometry [Edelman et al. [998 Bendokat et al. 2024
5

---

Table compare evaluation loss (4) for pre-training Llama-based architectures on the dataset over iterations. SubTrack++ oulperforms all other baselines in nearly every configuration. The best results are marked in bold, with the second- boosd-bbstt performance underlined. "LDAdam could not be run on the 7B configuration duc out-of of-meorry error with available resources.
60M
130M
350M
1B
3B
7B
128
r=256
r=256
r=512
r=512
1=1024
Full-Rank
3,41
3.25
3.40
4.61
4.52
4.30
Gal.ore Zhao ct al.
2024)
4.02
3.61
3.62
6,53
6.57
5.55
BAdam [uo et al.
2024
7.86
7.08
7.62
7.28
7.12
6.76
Online Subspace Deseent
Liang et al.
2024
4.18
3.88
4.09
6.79
6,85
5.69
LDAdam Robert et
2025
3.52
3.44
3.67
4.70
4.39
OOM"
Fira [Chen 2025
3.80
3.55
3.56
6.31
6.50
6.83
SubTrack++ (Ours)
3.43
3.24
3.29
4.52
4.50
4.63
Definition 3.3 (Exponential Map). The exponential map expp T,M + M on a Riemannian
manifold M is a mapping that assigns the point y(1= E M to each tangent vector A E T,M. where T.M is the tangent space of M at p. and is the unique geodesic originating at with initial velocity A. This map establishes relationship between geodesics and the Riemannian exponential, such that
(() = exp,(1A) for & E R.
Definition 3.4 (Stiefel Manifold). The Stiefel manifold Sl(n,p), parametrizes the set of all n x orthonormal matrices U, each representing a rank-y subspace of ik".
Definition 3.5 (Grassmann Manifold). The Grassmannian manifold Gr(n, p) parametrizes the set all p-dimensional subspaces R", Each point can be represented by a projection matrix P = UU where U E pl(n,p).
Theorem 3.6 (Grassmann Exponential). Let P UUT E Gr(n.p) be point on the Grass- mannian, where E St(n, p) the orthonormal basis of the corresponding subspace. Consider tangent vector A E TyGr(n.P) p), and let Aly denote the horizontal lift of A to the horizontal space at U in the Stiefel manifold Si(n,p). Suppose the thin SVD of A given by A =EVV where Q E St(n,r), min(p, P). and V St(p, r). The Grassmann exponential map, representing the geodesic emanating from P in the direction A, is given by:
(tagg contains the nonzero singular values of Air with
Expr (12) [UV cos(IE))V + V +UVIV]],
where V1 E TRpx(x-r) any orthogonal complement V.
The proof of Theorem 3.6 be found in Appendix B Leveraging this theorem and our notation in section 2 one can easily verify that the subspace update rule is as follows:
[osSErn (-sin (
cos Epn VT + + VE
S44( (S.V
ÜF)
This update rule generally converges to stable subspace if the step size 17 decreases time [alzano et al. 1) However, decreasing step size can impair the ability to accurately track and adapt subspace changes. Consequently, SubTrack++ subspaces. This approach does not hinder convergence, as proved in Theorem 3.2 which guarantees convergence long as changes are controlled to maintain the stable subspace assumption.
constant step size effectively adjust
4 Experiments and Results
We evaluated SubTrack++ across diverse models and datasets through pre-training and fine-tuming, measuring key metrics critical to LLM democratization.
Pre-Training Experiments. We pre-trained several Llama-based models on the C4 dataset, with results in Table I To ensure fair comparison, benchmarked against diverse set of baselines.

---

APOLLO [2hu 2025)
11
APOLLO (2hu et 2022)
Fini 2005]
BAdas [Lus atal, 2004|
10
Fixx [Chen etall, 2025]
BAdam (Lae 2124
Onloore [Zhso st, 2004]
GaLone
2004]
GoLane (Hl 2025]
Gelore et al. 2025]
LDAdem (Robert al, 2025]
[Lisng 2024)
Loss
8-
O&D 2024]
LDAdam (Robert 2025
SubTrack+ [Ous]
7-
SubTinke [Ohin]
5-
5000
10000
15000
20000
2000
4000
6000
8000
10000 12000
14000
Step
Relative Time (Wall) [s]
(a) Loss vs. Training Steps.
(b) Loss vs. Relative Wall-Time.
Figure 3: Comparison of baselines in pre rre-training Llama- 1B architecture. (a) shows training loss (L) versus training steps. (b) shows same runs against wall-time. SubTrack++ outperforms baselines; substantially reducing wall-time, especially compared I.Dddam, the top-performing haseline.
While all compared methods aim for memory-efficiet training, their architectural principles yield distinct computational and convergence trade-offs. BAdam (Luo al. 2024) achieves strong memory and time efficiency, yet its partial parameter update strategy compromises final performance. GaLore low-rank subspace estimated via periodic SVD, an
(Zhao et al., [024] constrains gradients approach sensitive to noise and costly to compute, while discarding information residing in the orthogonal complement. Fira [Chen et al. 2025 introduces norm- norm-based recovery scaling to mitigate this information loss, but its reliance on frequent SVD still leads to substantial wall-time overhead. LDAdam [Robert al.. 2025) replaces SVD with PowerSGI--bssed iterative updates and projection- aware optimizer that synchronizes internal states with evolving subspaces, improving convergence and stability but incurring high per-step costs due continual subspace updates. It also adds an extended arror-feedback mechanism to compensate for both gradient and optimizer compression to target GaLore's shortcomings. Online Subspace Descent (OSD) [Liang al. 2024), our tracking-based baseline, further reduces complexity by employing Online PCA for subspace tracking.
SubTrack+ departs from these formulations by treating subspace evolution a geometric tracking problem on the Grassmannian. performs efficient rank-1 geodesic updates that reuse historical subspace information, inherently preserving consistency and avoiding the instability of discrete subspace resets. Simultaneously, it aligns Adam's 's internal states through projection-avee aware optimizer and leverages recovery scaling to reintegrate the lost information. This unified approach yields new training paradigm that retains the memory efficiency of low-rank methods, while achieving on-par runtime efficiency compared fastest baselines like APOLLO (Zhu al. 2025). SubTrack++ sets new state-of-the-an results across model scales, running 43% faster than LDAdam, the strongest prior baseline, on the 1B model, and 67% faster the 3B model (see Ta.le
Figure demonstrates the pre-training of a IB 1B-parameter Llama model across several basclines. SubTrack++ has the fastest convergence in both training steps and wall-time, highlighting the effectiveness of geometry-aware optimization in improving performance and reducing resource consumption. To further assess its generalization in longer training regimes and larger models, we extend training to 100k steps and compare SubTrack++ with GaLore. As shown in Figure [ SubTrack++ converges substantially faster, achieving evaluation loss of 3.37 (a significant improvement over the 10k-step setting), while GaLore reaches 4.64 under the same conditions.
As shown in Table 11 SubTrack++ occasionally outperforms full-rank training. This effect may be attributed to the implicit regularization introduced by low-rank projections, which can enhance generalization in overparameterized models. Similar trends have been reported in other studies [Robert cct al. 2025 Zhu et al. 2025 Chen et al. 2025). addition, to further validate that convergence on the projected gradient reflects convergence of the original full-gradient, we measured both norms during Llama- IB pre- pre-training. The -al-dgaadent norm drops from 0.46 - 0.08, while the projected- gradient norm drops from 0.45 0.05, following nearly identical trajectories. This confirms that the optimization progress observed on the projected gradient accurately mirrors convergence in the original gradient space, consistent with prior low-rank gradient findings.
Hyperparameters of pre-training experiments are provided in Appendix E with detailed runtime and
memory reports in Appendix F
7

---

GaLerre (Zhao et al. 2024]
GeLore (2hao al. 2024]
Greemennian Sulspure Dracking (GST)
Gresmaniio Sulspace Trokkng (OSI)
10
10
GST Projection Awaze Opsimizer
GST Projection Awaze Optimizer
GST + Recovery Scaling
OST Recwery Sealing Track
SubTracket
Loss
7
5000
12500
20000
Relative Time (Wall)
(a) Loss vs. Training Steps.
(b) Loss vs. Relative Wall-Time.
Figure 4: Ablation study comparing pure Grassmannian subspace tracking with incremental additions of the projection-aware optimizer and recovery scaling, leading to SubTrack++. While Grassmannian tracking alone almost matches GaLore's 's step-wise convergence (a), it significantly reduces wall-time (b).
Ablation Studies. We conducted an ablation study to assess the individual and combined contributions of the projection-awan aware optimizer and recovery scaling, integrated with Grassmannian subspace tracking, on the IB Llama model. Experimental settings are summarized in Table As illustrated Figure bb, Grassmannian subspace tracking alone substantially reduces wall-time compared to frequent SVD updates. Both the projection-awan optimizer and recovery scaling independently provide notable performance gains over baseline subspace tracking, lowering the loss from 6.53 to 5.43 and 5.28, respectively. Their combination, SubTrrckk further improves the loss to 4.51, surpassing all baselines. Importantly, these improvements are achieved with only minimal increases in runtime and memory overhead, thanks to the efficiency of Grassmannian subspace tracking.
In addition, we conducted ablations the subspace update rank and update frequency, As shown in Figure 53. more frequent updates can further improve performance, and the computational efficiency of SubTrack++ enables higher update frequencies with minimal overhead. However, excessively frequent updates may impede convergence. Figure b shows that rank-1 updates achieve the best performance among all tested values. This suggests that making controlled, small adjustments the underlying subspace helps maintain stability, while updating the subspace along the most informative direction prevents stagnation low-rank region and enhances generalization.
Fine- Tuning Experiments. RoBERTa-Baee Base and RoBERTa-Large are fine-tuned on GLUE Wang al. 2019) and SuperGLUU [Sarlin et al. 2020) tasks; with the results presented in Table 7 and 8 respectively. We also conducted supervised fine-tuning of the Llama-2-7B-chat-h model for one epoch on the Alpaca (Taori al., 2023 dataset. In this experiment, SubTrack++ achieved 36% lower
wall-time compared to GaLore and 65% compared to LDAdam. The results are presented in Table g
More details and hyperparameters are provided in Appendix G
Time and Space Complexity. Table provides memory requirements of the optimizer states and the time complexity of the subspace update step considering m gradient matrix with S GaLore (Zhao et al.] [024)] and Fira [Chen et al.) 2025) periodically perform SVD to estimate the underlying subspace, while LDAdam [Robert al. 2025) relies on the faster PowerSGD to update the subspace at every iteration. In con- trast, SubTrack+ employs Grassmannian--aan subspace tracking at the same frequency as Ga- Lore and Fira. Comparing the time complexities of these methods, highlights why SubTrack++ significantly more efficient than SVD-based methods. A breakdown of the subspace update time complexity for SubTrack++ is shown in Appendix D Additionally, the memory required for storing optimizer states in SubTrack++. is equivalent to GaLore and other basclines.
Table 2:
The optimizer's state parameter count and subspace update time complexity across baselines, given a gradient matrix of dimension m x and projection rank where r & m S n. "LDAdam updates the subspace every iteration, while other methods update it every l steps.
Optimizer Mem.
Subspace Update Time
Adam
LDAdum"
mr + 2nr
O(mnr)
GaLore, Fira
mr + 2nr
O(nm²)
SubTrack++
mr + 2nr
O(nnr)
8

---

Evaluttin Loss Converged
Evaluation Loss
5.50
Evaluation Loss
5.25
5,00
4.75
X
Not Conveged Eodiationn
4.25
100
150
200
500
1k
2k
x-1
16
32
Update Interval
Rank of Updates
(a) Loss vs. Subspace Update Frequency.
(b) Loss Subspace Update Rank.
Figure 5: Ablation results on (a) update frequency: decreasing the update interval (i.e.. increasing the frequency) improves evaluation performance up to point, but overly frequent updates hinder training convergence. (b) update rank: increasing the rank of updates degrades model performance, and beyond certain threshold, can prevent convergence. These results emphasize the importance of controlled subspace adjustments.
Robust Subspace Tracking. Relying SVD for subspace updates makes methods sensitive to noise and abrupt changes [He et al. 2025]. Figure [6 compares Grassmannian subspace tracking with GaLore's SVD the Ackley function, highlighting how SVD causes erratic jumps, while our subspace tracking ensures robust optimization. GaLore struggles to reach the global minimum within 100 steps scale factor 1, and although increasing the scale factor to 3 improves perfor- mance, it amplifies jumps that hinder convergence in non-convex settings, revealing sensitivity to hyperparameters, noise, and abrupt changes. To empirically measure the robustness of SubTrack++ in tracking evolving subspaces, we quantified subspace drift using the norm of the tangent vector VF, which reflects the deviation between the projected gradient and the original gradient matrix. On Llama-350M pre- rre-training, this norm rapidly decays from 0.06 to below 0.0002 within 5-7 subspace updates and remains near zero thereafter, indicating highly stable and well-aligned subspace tracking. This quantitative stability supports our qualitative observations in Figure 6 and aligns with prior findings in Grassmannian optimization.
Related Works
Parameter-Efficien Training. Several works aim to improve the efficiency of training LLMs, addressing a growing demand as their popularity rapidly increases. Popular LoRA (Hu al. 2021 significantly reduces memory requirements for fine-tuning LLMs by leveraging
low-rank trainable
low-rank matrices. Dettmers et al. (2024) employ quantization techniques and paged optimizers to further reduce memory usage. Additionally, Yaras et al. (2024) introduce Deep LoRA address overfitting issues and reducing the need for precise tuning of the rank parameter. Several other works
have also extended LoRA to enhance the efficiency of training and fine-tuning LLMs [Lialin et al.
2023 Renduchintala et al.) 2024 Xia et al., 2024 Pan et al. 2024]. Miles et al. (2024) propose compressing intermediate activations and reconstructing them during backpropagation
enhance
memory efficiency. Yen et al. (2025) propose adjustments to the LoRA factorization that promotes balanced training and correspondingly update the optimizer's internal states (i.e., first and second moments) to remain consistent under the change of basis. Additionally, Hao et al. [2024] demonstrate that full- parameeer fine-tuning feasible by random projections on the gradient matrix, showing that LoRA essentially performs a down- wown-projection of the gradient. BAdam [Luo et al. 2024 leverages the block coordinate descent framework to reduce memory consumption while maintaining capabilitics comparable Adam.
Gradient Low-Rank Projection. Several approaches aim to reduce optimizer states, optimizers like Adam [Kingma and Ba 2017 account for a significant portion of memory footprint [Li et al.
2023 Anil et al.) 2019 Lv al.] 2024 Dettmers et al. 2022]. MicroAdam [Modoranu et al.] 2024
tackles this by compressing the gradient space and utilizing the compression error through feedback loops. Adam-mini [Zhang et al. 2024) partitions model into blocks, assigning a single learning rate to each block to preserve performance while saving memory. Gur-Ari et al. [2018] Schneider et al. (2244) Yaras et al. (2023) suggest that a substantial portion of gradients lies within a largely
consistent subspace. GaLore ZZhao et al. 2024 leverages this fact to reduce the optimizer's memory

---

Start Point Endppint
Saut Point Endpoint
Start Point Endpoint
Sout Point Endpoiut
(a) Ours, SF=1
(b) GaLore, SF=1
(c) Ours, SF=3
(d) GaLore, SF=3
Figure 6: Comparison of Grassmannian subspace tracking (Ours) (a, c) and GaLore's SVD (b, d) on the Ackley Function over 100 optimization steps, with subspace update interval of 10. SF stands for scale factor; with scale factor of I, GaL.ore fails to reach the global minimum due to abrupt jumps. a scale factor of 3, while the minimum is reached, the jump length increases. This demonstrates SVD's sensitivity to noise and abrupt changes, highlighting the robustness of our subspace tracking method with its controlled subspace updates.
by projecting gradients into a low-rank subspace and then projecting them back for full parameter tuning. This approach has been integrated with other methods regarding efficient L.LM training [L] al. 2024]. However, not all layers' gradients evolve within stable low-rank subspace. [Jaiswal et al.
[204]] identify layers where gradients evolve within a low-dimensional subspace and fine-tune only
those layers, freezing the others to avoid inefficient low-rank updates. Grass [Muhamed et al. 2024
reduces memory usage by applying sparse projection matrices to the gradient. |Ramesh et al.
2024
dynamically select and update a subset of parameters, for a fast and eeemory-effiiient training. Fira
[Chen et al. 2025 utilize norm-based scaling method along with GaLore to maintain performance comparable to full-rank training. GoLore [He et al., 2025) addresses GaLore's ' convergence issues and employ random projection in latter steps solutions. I.Adam (Robert et al. 2025 performs optimization within liver-dimensiona subspaces, incorporating a projection-aware optimization
update rule and generalized error feedback mechanism. Projection-Aware APOLLO (Zhu et al. 2025) approximates channel-wis. learning rate scaling based on random projection. Also, Liang et al. 2024 introduce dynamically evolving projection matrix updated via online PCA, enhancing the model's 's ability to navigate the parameter space efficiently without relying on expensive SVD.
Geometric Subspace Updates. A common approach in working with sighndimensional data is to project the data into nower-dimenniona. space, and many studies focus on tracking these subspaces as they evolve. Balzano et al. (201T) introduce on the Grassmannian when the data is partially observed. ZZhang and Balzano [2016] and Kasai [201]] propose methods handle noise effect in tracking these subspaces. Furthermore, Blocker et al. 2023) present a method for evolving geodesic- based data in the Grassmannian for updating the subspace effectively. Mo et al. 2025 propose LORO, a low-rank pretraining method that performs Riemannian optimization on the manifold of fixed-rank matrices, enabling parameter- and memory-efficient training by updating low-rank factors via manifold-aware gradients and retractions.
incremental method for updating subspaces
6 Discussion and Conclusion
We propose SubTrack++ time- and memory-efficien approach that projects gradients into low-rank subspace and uses Grassmannian subspace tracking to preserve the computed subspace while incorporating gradient components from the orthogonal complement. By integrating projection- aware optimizers that reflect subspace changes in Adam's 's internal statistics and utilizing the gradient information lost during low-rank projection, SubTrack++ achieves state-of state-of-the-arr tfethhrrrt convergence and accuracy across all baselines. While Grassmannian subspace tracking integrates seamlessly with various optimizers as a plug-and-play module, extending projection-awarr optimization beyond the Adam family requires further design and investigation. Also benchmarking on models with more than 7B parameters was not feasible regarding our limited time and resources.
Acknowledgments and Disclosure of Funding
Sirisha Rambhatla would like to acknowledge support of the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, RGPIN-2022-03512.
10

---

References
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memoorye Ilm training by gradient low-rank projection, 2024. URL https:
saaxiv.org/absee.
03507
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients, 2024. Utt.ttss:://artt/tt
org/ abs/2407 11239
Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, and Virginia Smith. Grass; Compute efficient low-memory Ilm training with structured sparse gradients, 2024. URL https://arxiv org/ 17660
Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections, 2024. URL /...ttpps//arra.ooo//a/////.. 17991
Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtarik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence, 2024. URL
/b////////244/ 2405 15593.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024. BtRL/tttts///rii..ooo org/ abs/ http://wwwww.r///////22.../
03293
Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. Owlore: Outlier weighed layerwise sampled low-rank projection for memory-efficient I1m fine-tuning, 2024, URL /ttp:/tss///ar wrg/ /bs/ 2405. 1 18380
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016. URL https: httpttpps.//.r....o/////////////
06174
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arxiv. htwpg///a////..///////// 1ttps/9110000004444
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685. 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in Neural Information Processing Systems. 36, 2024.
Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparame- terized low-rank learning & adaptation. arXiv preprint arXiv:2406.04) /2, 2024.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira. and Anna Rumshisky, Relora: High-rank training through low-rank updates, 2023. URL https:////arriioorrr wttp:/wwwww.gg/////.......
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-LoRA: Enhaneing parameter efficiency of LoRA with weight tying. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8694-8705, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. URL httphttpss///alattoooogooooo 2rtpo/hooolgyy.//22222444 http:/2222nnaaa11... www.naaalllloo -log. 481
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning, 2024. URL
2401. 04151
Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter optimization method for large language models, 2024. URL httph:ttt//att
////////2//44444 02827
Amrutha Varshini Ramesh, Vignesh Ganapathiraman, Issam H. Laradji, and Mark Schmidt. BlocklIm: Memory-effiiiet adaptation of lims by selecting and optimizing the right coordinate blocks, 2024. URL https: /
//tr//aaaa //trxi/xvvv///////////a////aaavr org/ wtwp 2

---

Bingrui Li. Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states, 2023. URL https://arxiv.. wtt.:oggg/bb.gg.o/e//e/// 01507
Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory--fficient adaptive optimization,
2019.
abaaabaa//990 11150
Kai Lv, Yuqing Yang. Tengxiao Liu, Qipeng Guo, and Xipeng Qiu, Full parameter fine-tuning for large language models with limited resources. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Bangkok, Thailand, August 2024. Association for org/ 2204./224..2/ acl long. 445
(Volume I: Long Papers), pages Computational Linguistics. URL
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization, 2022. URL httpps/////xxii.oorrr http://awavv..r////////1111....2
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more, 2024. URL https:://rxiii org axg/abs/2400 16793
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace, 2018.
URL /ttp:ii/////v/.e//eeeeaeeeseaaaa htttp:://arriivorr//bb/b//12 04754
Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel Häufle, Bernhard Schölkopf, and Dieter Büchler. Identifying policy gradient subspaces, 2024. BRRLtttp:://aar http///xxxvooor/aaba org/aba/
2401.06604
Can Yaras, Peng Wang, Wei Hu, Zhihui Zhu, Laura Balzano, and Qing Qu. Invariant low-dimensional subspaces in gradient descent for learning deep matrix factorizations. In NewrIPS 2023 Workshop Mathematics of Modern Machine Learning. 2023.
Thomas Robert, Mher Safaryan, Ionut-Vias Modoranu, and Dan Alistarh. LDAdam: Adaptive optimization from low-dimensional gradient statistics. In The Thirteenth International Conference on Learning Representations, 2025. URL h
GuHerF
Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance, 2025. URL https::::: //atpps://rxxiv.. org/ abs 2412. 05270
Kaizhao Liang, Bo Liu, Lizhang Chen, and qiang liu. Memory-efficient LLM training with online subspace descent. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpast//pppereeiiir mett/forummidd
Namrata Vaswani, Thierry Bouwmans, Sajid Javed, and Praneeth Narayanamurthy. Robust subspace learning: Robust pea, robust subspace tracking, and robust subspace recovery. IEEE Signal Processing Magazine, 35(4):32 55, July 2018. ISSN 1558-0792. doi: 10. 10111000.mm......... 2826566.
URL http::/ttp:..../// oog.oor.o..o.oooooooooooo.ooo... /0////1..110 1109/MSP 2018. 2826566
Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimiztion for large language models with convergence guarantees, 2025. URL /tLltttpp://ppeeeriiee net/forumm
id=udtrtwkvk5
Jiayao Zhang, Guangxu Zhu, Robert W Heath Jr, and Kaibin Huang. Grassmannian learning:
Embedding geometry awareness in shallow and deep learning, arXiv preprint vrXiv:1808.022) 2018.
Laura Balzano, Robert Nowak, and Benjamin Recht. Online identification and tracking of subspaces from highly incomplete information, 2011. UtL.pttps:/arxxv.orrr// Saarrg/abs/0000 4046
Jun He, Laura Balzano, and John C. S. Lui. Online robust subspace tracking from partial information, 2011. httptttts///axiiiia org/ abs//109..... 3827
Cameron Blocker, Haroon Raja, Jeffrey A. Fessler, and Laura Balzano. Dynamic subspace estimation with grassmannian geodesics, 2023. URL //tp////xvvvv////////a/ org/ abs /2003 14851
12

---

Rudrasis Chakraborty, Søren Hauberg, and Baba C. Vemuri. Intrinsic grassmann averages for online linear and robust subspace learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 801-809, 2017. doi: 10.1109/CVPR.2017.92.
Dejiao Zhang and Laura Balzano. Global convergence of grassmannian gradient descent algorithm for subspace estimation, 2016. URL httptttts//aaiii.rrrrgeg www.oorr////aa/.oooe 07405
Laura Balzano, Yuejie Chi, and Yue M Lu. Streaming pea and subspace tracking: The missing data case. Proceedings of the IEEE, 106(8):1293-133) 2018.
Xi Chen, Kaituo Feng. Changsheng Li, Xunhao Lai, Xiangyu Yue. Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of LLMs under low-rank constraint?, 2025. URL https
//openreview wtw.forruneeeootmmeoemmmm 2
Thomas Bendokat, Ralf Zimmermann, and P.-A. Absil. grassmann manifold handbook: basic geometry and computational aspects. Advances in Computational Mathematics, 50(1), January 2024. ISSN 1572-9044. doi: 00.10077/10444-023- 10090-8. URL http://dx doi 1o0. 1007/s10444 023-444--2221111 10090-8
Alan Edelman, T. A. Arias, and Steven T. Smith. The geometry of algorithms with orthogonality constraints, 1998. URL Sttthttp::// //apa/////xivv arrivv org/ http///bbbyycca//000000
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. 2019. URL
https://arxiv.. org/ abs/ 1804. 07461
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks, 2020. /Rtpsstsss//arxi/ abs/
1911. 11763
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followingg Ilama model. 2023.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024.
URL http::tttt:///tttt /ttpss/////xvvvv/ee www.xxiv..or/abbe org/ http://bb.//23...../ 17919
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu. Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsich, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for loRA optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevier: wtt.:/roo....ttooooooooo/////ila
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
htttp://arxiv..or httg/abs/11140
Hiroyuki Kasai. Fast online low-rank tensor subspace tracking by cp decomposition using recur- has////b////
sive least squares from incomplete observations, 2017. URL
10276
Zhanfeng Mo, Kang-Kai Huang, and Sinno Jialin Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Representations, 2025, URL http:: httpp::/ppenreveev.eeeeeee /ttppp/rrrreerreee//ooooooo/od/0
13

---

A Convergence of SubTrack++
Theorem 3.2 (Convergence of Grassmannian Subspace Tracking). Suppose gradient has the following form with functions Ag, Bi and C₁ being L-continuous as per D with constants LA. LB, and Le w.r. weight matrix W and [[, S M; where W, denotes the weight matrix at step t, and M is scalar value,
G=Σ A + Σ B.WC,-
Now, define B., = (S (W.,)) and Ct = ( C.,WW)) where S and Si are the rank-r left and right projection matrices; (.(,)) and (W,) denote the dependence of B, and C₁ on the weight matrices W Further letting Pt S G and Kit = 'N E. Amin (Br,)) (C,,). where (m)n(-) (-) denotes the minimum eigenvalue over each batch, and N representing the number of samples
batch. Assuming that the projection matrices remain constant during the training.
Then for learning- rrate and mān((s)) > (La + 2Ln LcM2), subspace tracking, with pr 1 (the element-wit wise regulurizer of the optimizer) satisfies:
[1 p(kt-1 LA 2L ML
That is, P. and il converges.
proof. To demonstrate that SubTrackt+ converges to the global minimum during training, we begin by deriving the recursive form of the gradients.
Let @ denote the Kronecker product. Then, vec(AXB) (BT @ A))e
By applying vec to the gradient form given in the theorem, we obtain:
gt = vec(G.) = vec( Σ A++ Σ B,WC,) = - - Drue
(16)
where gr vec(G.), wy vec(Wi), a: == N Σ vec(Au.)) and D. E, C. 8 Br.-
As defined in the theorem, let P: = S G.S. Its vectorized form can be expressed using the Kronecker product as follows:
P: vec(PP) = vee(s G.S,) = (5, & S )ecc(G.)
(17)
= -S, @ S vec(G,) = (S; 8 S
Now recalling G, from
15 it can be written as:
= Stst G.S.S
Thus, its vectorized form will be:
rec(GG)) J. vec(S G.S.S, ST
(18)
= -S; @ St)vec(P.) (S) & S)P.
This is where the constant subspace assumption becomes necessary. To derive the recursive form of gt. we assume that the projection matrices remain fixed throughout training, i.e., S = S" and S S'. Consequently, we can restate equations (17) and 18) as follows:
Pt (S" ② @S)
S ge
(19)
g. (S" & S S')p
(20)
Then we can write the recursive form of ge:
St a; DDW,, (a, - 1)-1) + (D,-1 D,)w; + a:-1 - Di-1
(21)
= + ap-1- - -D.-1 + 1 et + -t-1 - JDD=-1i--
where E = (ar a(-1) + (Di-1 D.Yer
14

---

Note that in deriving 21 we utilized the general form of the weight update rule, w1+1 = We pge. which can be rewritten as w, 1++1 + ug.- By applying this rule along with (T6), we arrive the second equality in 21 as follows:
9: at D,w, = at D,w, - 9t-1 9t-1
Diw - 01-1
4;-1 - Di-1
at
Dr00 0t11 + Di- (w, + (1t-1))111)100 0=-1 Dt-1(w; 19h-1)
= - Diw: 1 + D--1 Wi + 1D: + D1-1W; pDr-19t-1
at at-1 + (D;-1 w; + + Dz-1
To obtain from this recursive formulation, we can left-multiply by (5" 69 St) as shown in 20
Pt = (S" & S
-t-1 - 115" 3 S5S D1-10 Ce + (5" @ 5*) T
(22)
Now, based
19 and
20
Pa be written as:
Pt (5" @ S*)
c P - MS 3 (') D5-1(5" & s')p-11
(23)
Let define:
(5 8 T Dr(S" 05)= = N Σ (8" 8SS (Ci, @ (5,)(5" 8S
(24)
E(St Ci, ( (S BB,
Then we can expand 23 and show that:
(1 DD. (S" @0 Te
(25)
Note that St and S are orthonormal matrices. This is ensured because the subspace is initialized
using the SVD of Go. and the Grassmannian update rule provided in preserves the orthonormality of the subspace matrices throughout training. Since St and S are orthonormal, we have st s =
and Sr S" I. Consequently, we can bound the norm of the second term in (55) as follows:
I1(S St) T etll2 = ||vec(S' E,S") ||s E. $ ||E||L
(26)
Here Eg is the matrix form of and as declared before, == (ar a--1) + (D,-1 10)1--i)u, thus:
E, N - A1--1)+ N E(BLL B,,.W.C..)
(27)
Next, we need to find an upper bound for the norm of each term in (7 to establish an upper bound for [e F. Based on the assumptions of the theorem, A,, B;, and C, exhibit L-Lipschitz continuity with constants LA. LB, and Lc, respectively. Additionally, ||Well is bounded by a scalar M. We have:
A A < LAIlW, W.-ille = μL FLa S PL
(28)
In the first equality, we apply (13), while the last equality holds due 20) and the orthonormality of the projection matrices. The subsequent two inequalities can be derived similarly using these equations.
- (..) < LnIlW, - W,
(29)
= =
[B, S ( Wille
(30)
= -MI
We can now derive the bound for ||E|| as follows:
|| S pL &
M2 /tlp
(31)
= =u(AA 2L
15

---

To calculate the norm bound for the first term in (25). we first need to establish the bounds for D.. This involves estimating the minimum eigenvalue of Dr.
If define Timéniit ((s' Bi, @ Cc. inequality for every unit vector 7:
( (S B., ( Ci,,5"), then it follows that Imin,it. Consequently, Dr will satisfy the following
Σ T (IS & (ST 8 Σ
Tminns.
(32)
this actually provides lower bound for eigenvalues of Dr, thus:
Amax (1 (11=-D,))) 1) S N Σ
7min,i,t-1
(33)
considering the definition of fi in the theorem, we can now easily show that:
P $ [1 p(xx-1 LA - 2L
and completing the proof.
While SubTrack++ utilizes right/left projections to reduce memory consumption, the proof is pre- sented using both projection matrices ensure generality. Here, we demonstrate how the proof proceeds under the assumption m S n (without loss of generality), which allows the use of the left projection matrix.
Using the left projection matrix, the current formulation of Pt, defined as P. = St GGST, simplifies to P. S Gt- Similarly, G, S.SS S; reduces to G S.S, G.. From this point, the proof continues by substituting ST with the identity matrix, allowing the derivation of the vectorized forms of St. Gt. Pt, and related terms.
The remainder of the proof remains largely unaffected. can be readily verified that the recursive formulation of 9r is unchanged. Although the definition of P. is modified, it continues to satisfy the bounds required for convergence, ensuring that P. converges to when the left projection matrix is used.
B Grassmann Exponential
Theorem 3.6 (Grassmann Exponential). Let P UUT E Gr(n.p) be a point on the Grass- mannian, where St(n. p) the orthonormal basis of the corresponding subspace. Consider tangent vector A E TpGr(n, p), and let Airr denote the horizontal lift of À to the horizontal space at U in the Stiefel manifold St(n, p). Suppose the thin SVD of Avr is given by A =QVV where Q E St(n,r). Σ diag(o1..... min(p, n - p), and V E St(p, r). The Grassmann exponential map, representing the geodesic emanating from P in the direction A, given by:
contains the nonzero singular values of hotr with
Ex
[UV cos(tE)VT + sin(tV + +UV1V]]
where V, E (p-r) any orthogonal complement of V.
proof. Using Grassmannina mathematics, we know that every A E TpGr(n,y) is of the form
0
B 8 )) QT =
B BT) Q O))TPP)
A Q
(34)
B
Then the lift of A TpGr(n
to Q (U U1) can also be calculated explicitly follows:
Abr A PlQ Q (g )
(35)
To resume proof, we need to define the orthogonal group and specifying its tangent space.
16

---

Definition B.1 (Orthogonal Group). The orthogonal group O(n) is defined as the set of all
matrices Q over I& such that Q Q QQQ = n, where Q is the transpose of Q and I is the n identity matrix:
( = {Q xx 1QQQ= = ="Q=I=QQ']
Then the tangent space of the orthogonal group (n) a point Q, denoted Tn)(n), is defined as the set of matrices of the form QΩ, where 52 R is skew-symmetric matrix, i.e., Ω -2:
TnO(n) {QΩ Ω E R Ω² -22].
The geodesic from E (nn) in direction QN E (n) is calculated via
(xP2( = Q expm (tf2),
(36)
0
- -B²
If P E Gr(n. and A TyGr(n.p) with A
Q
). tthe geodesic in the Grassman-
B
0
nian is therefore
Expy (tA) = DG (QeSP...
(BB
0
B
t
(37)
0
where π OG is the projection from (n) to Gr(n, p). If the thin SVD of B is given by
B = U A UI
with W = U E Stin - p,r), E E R V E Sl(p, Let WI, V1 be suitable orthogonal completions. Then,
cost Σ)
0
sin
0
VT
0
(0 0
B
V
V1 0
-p-r
VI
0
expm
0 1O1
B
0
W
W1
sin
0
cos (2)
0 In-p-r/
0
W
0
0
WI
which leads to the desired result when inserted into (37 For more mathematical details, you can refer Edelman et al. [1998), [Bendokat al. [2024), or other useful resources on Grassmann geometry.
C Projection-Aware Optimizer
When projecting into lower-dimensional space and tracking coordinate changes, we typically use orthonormal projection matrices to represent the subspaces and their multiplications to effect change of basis. While this works well for purely linear operations, Adam's 's updates also incorporate non-linear elements.
Suppose the subspace changes at step t, transitioning from the subspace spanned by the orthonormal matrix S to that spanned by S. Since both matrices are crthonormal-preseeeee mannian update rule in (5) the matrix S. S.-1 represents the change of basis between the two subspaces. In other words, if - = (cl-1--.., < and E, = 1 and f, respectively, then the ith column of matrix X transforms under for the subspaces at steps the change of basis X Σ (ej x where x is the jth column of X based on the basis of the time step f. 1.
by the Grass-
ee)') are orthonormal bases
Eg [-] denotes the exponential tine-weighted expectation at time with decay rate 3. Following the reinterpretation by Robert al. 2025), Adam's first and second moment estimates can be expressed as M, E, G2 and V2 E,,o, (G,) where G, denotes the low-rank representation of the gradient time step As shown (38). the first moment estimate can be transformed under change basis using the change-of-basee matrix, STT S,-1. Notably, (Ge, et) gives the ith column of G when the subspace has the basis E, We use the superscripts to indicate a column of a matrix.
Ex,o, [G.,e ct}
Σ (e of_ Er,s, (( = Σ of (s. S.
(38)
17

---

Following the same approach, we can change the basis for the second moment estimate as described in 339
(()
E(e (e 2/11 B2 (iin
E,
=
Σ le et () (1), En M kylh
(39)
Σ (el 10
of lepi
In transitioning from the first equality to the second, we assume independence among the gradient coordinates. This enables estimates. This assumption is often reasonable in practice because we compute the SVD of the gradient and maintain an orthonormal subspace projection matrix, updating along the Grassmannian geodesic to track the optimal subspace. Since SVD tends to diagonalize the covariance, the off- diagonal entries are typically negligible. Additionally, we clip any negative values to zero to ensure valid (non- (nonnnegative) variance estimates. Moreover, to rewrite the second term in the final equality of (39). we employ the following equation:
approximate the covariance using product of first-order moment
ΣΣ
(40)
(et e 1/2
M₂
+
kgl
Given these, we can rewrite 33) follows:
Er,02
(G
Σ (ch (()))
Σ (ei i Mr - Σ (et oi (M²)
(41)
Σ (eh, )² 136 3)-((Tr) ((61.
((ST (sT) 1 1 mrr) )
By applying (38) and [4 we can directly derive the update rules for the projection-aware optimizer,
as expressed in and
D Time Complexity Analysis
Table 3 presents the time complexity breakdown for the subspace update step in the SubTrack++ algorithm assuming m x n gradient matrix and rank r projection matrix, where & m S n. As outlined in Algorithm| []] the subspace update step begins by solving the least squares problem (2) to estimate the optimal update for S,. the m orthonormal matrix. This operation has complexity of O(mr²). Computing the residual and the partial derivative with respect to S, requires O (mrn) and ((mer) time respectively. This is because the solution to the least squares problem, A, has shape r x n which is multiplied by S, in the residual R G. S.A, resulting in time complexity (mmr)). The following operation for the partial derivative is -2RAT, where the matrix multiplication has (mmrr) complexity. The tangent vector computation (4) which involves an identity transformation and matrix multiplication has time complexity of O(mr).
18

---

CaLore (Zhan et 2024]
OaL [2hao all 2014
SubiTrcck- [Utrs]
Prack [Ours]
10
Loss
100000
200000
300000
400000
100000
200000
300000
400000
Step
Relative Time (Wall)
(a) Loss vs. Training Steps.
(b) Loss vs. Relative Wall-Time.
Figure 7: Comparison of pre-training Llama-77 7B architecture for 100k iterations. (a) shows training loss (4) versus training steps. (b) shows the same rums against wall-time. SubTrack++ outperforms GaLore; substantially reducing wall-time.
Table 3: Time Complexity for SubTrack++ Subspace Update
The rank-1 approximation step uses largest sin- gular value from the SVD of the m r tangent
Computation Step
Time
vector, and has time complexity of O Ommr?) Fi- nally, the update rule as shown in (13) which has time complexity of (mmr2). The over- all complexity of the algorithm is dominated by the matrix multiplication calculations of time complexity (mmr). However, unlike GaLore, since avoid computing SVD operation on the m x n gradient matrix, which has complexity of O(nm²) ) the overall update step in SubTrack+ is still more efficient with respect to time com- plexity.
Cost function
(mm²))
Residual
(mrn)
Partial derivative
((mrr)
Tangent vector AF
O(m²r)
Rank-1 1 approximation of AF
Ommr))
Update rule
(
Overall
0 (mmrr)
E Pre-Training Llama-Base Architectures
We pre-trained all six Llama-based architectures for 10k iterations using hyperparameters reported in Table 4 To demonstrate the generalizability of the proposed method, we also present results from pre-training the 7B architecture with both SubTrack++ and GaLore for 100k iterations, as shown in Figure SubTrack++ maintains its advantage in terms of faster convergence and superior performance. The hyperparameters of this run are identical to those reported in Table except for the number of iterations which 100k. Also the bar-chart version of Figure i]s represented in Figure
F
Memory and Time Comparison
Table 5 presents the the peak memory consumption measured to compare SubTrack++ and other basclines. It shows that +ubTrack++ requires on-par or better memory compared to GaLore (Zhao) al. 2024 while getting state-of-the-al results. As detailed in Table 2 all basclines except BAdam Luo et al. 2024] use the same number of optimizer parameters; therefore, any differences in their peak memory consumption stem from variations in their runtime parameters.
Additionally, presents the wall-time consumed by each baseline across all model architectures during pre-training. Each run is configured to include exactly 10 subspace updates for Sub + and other baselines employing periodic subspace updates. Specifically, models ranging from 60M to 3B use subspace update interval of 200, resulting in 2,000 total iterations, while the 7B models use an interval of 500, yielding 5,000 iterations. Experiments for the 60M 3B models are conducted on an NVIDIA A A10 GPU, while the 7B model experiments are run NVIDIA RTX A6000.
19

---

264
17.5
1735
Evaluatonn Lo03
Pook Memory (015
15.0
12.9
Time
10.0
Giaar
APOUL
Londio
WPLLO
Gdber
WPRLO
Pe
Lo
Soffaak
(a)
(b)
(c)
Figure 8: We compare baselines on pre- wre-training a 1B- 1B- B-parameter model. (a) SubTrack++ achieves the lowest evaluation loss across all methods. (b) Its peak memory usage is significantly lower than APOLLO and LDAdam, and on par with Gal ore and Fira. (c) In terms of wall-time, SubTrack++ incurs minimal overhead relative APOLLO and is markedly faster than Gal.ree Fira, and I.DAdam. Overall, SubTrack+ outperforms all baselines in evaluation loss while matching or exceeding them memory and runtime efficiency.
Table 4: Hyperparameters of pre-training Llama-based architectures.
60M
130M
350M
1B
3B
7B
Architectural Parameters
Hidden
512
768
1024
2048
2560
4096
Intermediate
1376
2048
2736
5461
6848
I 11008
Heads
12
16
24
32
32
Layers
12
24
32
32
32
Shared Parameters
Learning Rate
le-3
le-3
le-3
lc-4
le-4
lc-4
Batch Size
128
128
Gradient Accumulation
Iterations
10k
Gradient Clipping
1.0
Warmup Steps
1000
scale
0.25
dtype
bfloat16
Low-Rank Optimizer
Rank Subspace Update Interval
128
256
512
512
1024
Methods Parameters
200
200
200
200
200
500
SubTrack++ Step-Size
10000
BAdam Parameters
Block Swithch Interval Switch Mode
100
Random
G Fine-Tuime Fine-Tuning Experiments
As described, examined SubTrack++ on fine-tuning RoBERTa-base and RoBERTa-large models to evaluate them on GLUE and SuperGLUE benchmarks. results for GLUE task is summarized in T and SuperGI.UF. in
The hyperparameters for fine-tuning RoBERTa-base are detailed Table 10 matching those reported in the GaLore [Zhao et al.) 20244 for rank-8 subspaces, with a subspace update interval set at 500 iterations. We also fine-tuned RoBERTa-Large on SuperGLUE tasks using the hyperparameters from Luo et al. [004]], as detailed in Table [1 with the exception that we fine-tuned each task for 30 epochs.
The results of supervised fine-tuning of the Llama-2-7B-cha--hf model on the Alpaca dataset on an Nvidia-H100 GPU presented in Table The fine-tuning was performed for epoch, and the corresponding hyperparameters are listed in Table [2]
20

---

Table 5: Peak memory consumption of pre-training Llama-based architectures C4 dataset. The 7B models trained using the 8-bit Adam optimizer, except for the marked with * SubTrack++ demonstrates better or on-par memory compared other low-rank methods that allows paramettr training.
60M
130M
350M
1B
3B
7B
1=
r=256
r=256
1=11
1=512
r=1024
Full-Rank
16.86
25.32
28.67
18.83
34.92
50.50
BAdam [Luo et al.
2024
13.34
20.01
16.45
9.18
14.75
22.35
GaLore Zhao et al.
2024
16.89
23.52
27.85
14.74
26.03
36.00
Online Subspace Descent
Liang et al. (2024)
16.61
23.86
28.76
18.45
32.57
33.10
LDAdam [Robert et al.
2025
17.18
25.94
28.03
18.45
32.57
OOM*
Fira [Chen et al.
2025
16.39
24.99
27.33
14.68
25.62
47.84*
Sub +racTrac++ (Ours)
16.40
25.06
27.42
15.22
25.54
49.82"
Table 6: Wall-tine comparison of pre-training Llama-based architectures on the C4 dataset. The number of iterations is to 10 subspace updates for methods using periodic subspace adjustments. + achieves the lowest wall-tims among all baselines that support full- parameter training on large models. The 7B models are trained using the 8-bit Adam optimizer, except for the runs marked with Since this can impact wall-time comparisons, it is more appropriate to compare runs within the same cluster.
60M
130M
350M
1B
3B
7B
r=128
r=256
r=256
r=512
r=51
r=1024
Full-Rank
524.0
1035.1
1396.4
974.9
1055.9
12726.2
BAdam Luo al.
2024
511.3
779.2
961.6
798.6
1004.1
7283.7
GaLore Zhao al.
2024
547.8
1094.2
1589.0
1729.5
2715.5
21590.4
Online Subspace Descent
Liang el al. 2024
662.8
1228.2
1818.6
1438.7
1676.9
18221.9
LDAdam Robert 2025
639.9
1342.2
2083.4
2780.9
4625.4
OOM*
Fira [Chen al. 2025
635.3
1180.8
1729.7
1938.5
2898.4
22554.3*
SubTrack++ (Ours)
627.6
1140.6
1593.2
1304.3
1517.6
16491.7
Table 7: Evaluating the performance of SubTrack++ and other baselines when fine-tuning RoBERTa-Base GLUE tasks for 8. The performance is measured via Accuracy (t) for SST-2 and RTE tasks, F1 (t) for MRPC, Pearson Correlation (f) for STS-B, and Matthews Correlation (f) for COLA. The best results marked in bold, with second-best performance underlined.
COLA
STS-B
MRPC
RTE
SST-2
Full-Rank
62.57
91.03
91.32
77.98
94.27
BAdam [Lo al.
2024
54.44
89.01
91.35
68.59
94.15
GaLore Zhao et al
2024
58.5
90.61
91.30
74.37
94.50
LDAdam Robert et al.
2025
58.81
90.90
92.22
76.53
94.27
SubTrack++ (Ours)
58.55
90.95
92.04
78.34
90.02
21

---

Table 8: Evaluating the performance SubTrack++ and other baselines when fine-tuning RoBERTa-Large SuperGLUE tasks with r The performance measured via Accuracy (1) for COPA, WIC, WSC, BoolQ. and AX, tasks, and (?) for CB. The best results are marked bold, with the second best performance underlined.
BoolQ
CB
COPA
WIC
WSC
AX,
Full-Rank
85.96
90.33
76.00
71.79
63.46
96.30
GaL.on
Zhao et al.
2024
85.44
88.85
80.00
71.47
63.46
100.00
BAdam
Luo et al.
2024
82.51
53.28
59.00
70.38
60.58
51.85
LDAdam
Robert al.
2025
85.75
56.16
80.00
71.00
64.42
70.37
Sub Track++ (Ours)
85.38
83,96
82.00
70.70
62.5
100.00
Table 9: Comparing final evaluation loss (L), wall-time, and memory of fine-tuning 1aima-2-7B-hheee on
Alpaca dataset. cd
Method
Evaluation Loss
Wall-Time (min)
Memory (GB)
Gal.ore Zhao et al.
2024
0.88
178
59.4
LDAdam Robert et al.
2025
0.85
342
66.1
SubTrack++ (Ours)
0.88
117
62.5
Table 10: Hyperparameters of fine-tuning RoBERTa-Bass GLUE tasks.
SST-2
MRPC
CoLA
RTE
STS-B
Shared Parameters
Batch Size
16
16
32
16
16
# Epochs
30
Max Seq. Len.
Low-Rank Optimizer Methods Parameters
Learning Rate
2E-05
2E-05 3.0
IE-05
2E-05 15.0
3E-05
SubTrack Step-Size
10.0
Subspace Update Interval
500
Rank Config
BAdam Parameters
Learning Rate
2F-05
2F-05
1F-05
2F-05
3F-05
Block Switch Interval Switch Mode
100
Random
Table 11: Hyperparameters of fine-tuning RoBERTa-Large SuperGLUE. tasks.
BoolQ
CB
COPA
WIC
WSC
AX,
Shared Parameters
Batch Size
16
# Epochs
30
Learning Rate Max Seq. Len.
le-5
512
Low-Rank Optimizer SubTrack++ Step-Size Subspace Update Interval Rank Config.
10.0
10.0
100.0
1.0
Methods Parameters
500
100
100
500
250
100
BAdam Parameters
Block Switch Interval Switch Mode
100
50
50
100
50
50
Random
22

---

Table 12: Hyperparameters of fine-tuning Llama-2-7B-chat-hf on Alpaca dataset.
Parameter
Value
Subspace Update Interval Rank
500
1024
Q
0.25
Target Modules
mlp
Batch Size
8
Epoch
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We included the overview of method, achieved results, and the scope of experiments in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

The limitations and future directions is included in the Discussion section. Also the computational complexity is reported.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical proofs are aligned with well-known prior works while showing improvement over them, and backed up by strong sources for Grassmannian manifolds geometry.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
25

---

Justification:
All resources and hyperparameters used for testing the proposed method and baselines are provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will include the code in the main paper after submission, and the anonymized repository supplemental material of this version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public/guided Codeeuumissiooooolee.. aisiynooicyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
26

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details regarding resources and hyperparameters for the proposed method and baselines included in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: As the experiments of these paper are highly resource- and consuminn performing statistical test was not possible for this version. However, variety of experi- ments are included ensure generalization.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
27

---

Answer: [Yes]
Justification:
All the necessary information concluded in supplemental material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt.rppp....plllll/o/cccccclaaaa
Answer: [Yes]
Justification: Yes, totally conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is foundational research, and further investigation for preserving safety is left for future work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and funetioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
28

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
is a foundational research and do not poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow com/datasets: has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code that is included anonymously on supplemental material is docu- mented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
29

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This term does not apply to our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This term does not apply to our research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: does not apply to our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standardd components.
Please refer to LLM policy lhttps://neurips) /aaeerrppp.cc//ooofeeeee//////00 for what should or should not be described.
30


Reviews:
[{"review_id": "XSkNHVgU8k", "reviewer": "Reviewer_hLxD", "summary": "The authors study memory efficient large language model training (LLM) through low rank projections. They propose Sub Track++, which combines Grassmannian gradient subspace tracking, Projection-aware optimization, and recovery scaling. They provide theoretical guarantee on the convergence of the subspace and the empirical evaluation shows the improved performance over the compared baselines.", "questions": "My questions are related to the weakness listed. Please provide some insights on why the proposed method should perform better than the compared baselines. Additionally, please explain why in the experiments larger models seem to perform worse than the smaller ones and why subtrack++ can outperform full rank training.", "limitations": "The authors have some discussion in the conclusion section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The strength of the paper is that the proposed method by itself is decent. All the proposed mechanisms are reasonable and accompanied by theoretical guarantees and empirical improvements.\n\nThe main weakness is the presentation and the discussion/comparison with the related works. While the proposed method itself is reasonable, it is hard for readers to understand why it should work better than the existing ones. I summarized my issues as follows.\n\n1. Figure 1 looks like a very unconventional way to present the statistics the authors desire to present.\n\n2. There seems to be no in-depth discussion even with the baseline methods mentioned in the paper. This is concerning especially when combined with the ablation results. It seems like Grassmannian gradient subspace tracking by itself does not perform better than Galore, while Projection-aware optimization and recovery scaling are inspired by LDAdam and Fira respectively.\n\n3. There are also some related works in the literature that can be discussed. For example LoRo [1] also utilizes Riemannian optimization for subspace tracking, while LoRA-RITE [2] also utilizes Projection-aware optimization on the momentum with a seemingly different formulation.\n\n4. Some empirical results look counterintuitive. Specifically, the larger models perform worse than the smaller one, which might indicate that suboptimal settings are being used. Additionally, SubTrack++ sometimes outperforms the full-rank training, which also requires some explanation. \n\n\n\n[1] Mo, Z., Huang, L. K., & Pan, S. J. (2025). Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Representations.\n\n[2] Yen, J. N., Si, S., Meng, Z., Yu, F., Duvvuri, S. S., Dhillon, I. S., ... & Kumar, S. (2024). LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. The Thirteenth International Conference on Learning Representations (ICLR 2025)", "was_revised": true, "final_justification": "The rebuttal addresses my key concerns. Especially, the results that show improvement over Galore with only Grassmannian tracking and the promise to add a more detailed description on the comparison with the other baselines.", "rebuttal": "We thank the reviewer for their detailed feedback and for recognizing SubTrack++ as a solid and reasonable approach. We appreciate their constructive feedbacks especially regarding presentation and discussion. We have addressed the concerns and will reflect them in the camera-ready version.\n\n---\n> **Weakness 1:**  *Figure 1 looks like a very unconventional way to present the statistics the authors desire to present.*\n\nThank you for the feedback. We chose radar plots to enable compact comparison across loss, memory, and wall-time, consistent with prior work [1, 2]. We are happy to switch to bar charts or other formats if it helps improve clarity and is preferred by reviewers in the camera-ready version.\n\n> **Weakness 2:** *There seems to be no in-depth discussion even with the baseline methods mentioned in the paper. This is concerning especially when combined with the ablation results. It seems like Grassmannian gradient subspace tracking by itself does not perform better than Galore, while Projection-aware optimization and recovery scaling are inspired by LDAdam and Fira respectively.*\n\n> **Question1:**  *Please provide some insights on why the proposed method should perform better than the compared baselines.*\n\nWe agree that Fig. 3 underrepresents the impact of Grassmannian tracking. However, **Fig. 6** in the Appendix shows that this alone **accelerates GaLore’s [3] convergence by 29%** (and up to **44%** in the 3B model) in wall-time. We will move this to the main paper to clarify its significance. Grassmannian tracking is the basis of both **efficiency and robustness** of our method; by preserving subspace history and applying rank-1 updates, it reduces SVD's noise [8] and provides smooth optimization dynamics **(Fig. 5)**.\n\nUnlike FiRA’s [5] costly SVDs, our method **lowers wall-time by 33% and final loss by 29%** (1B). Compared to LDAdam’s [4] per-step PowerSGD updates, SubTrack++ performs rank-1 updates every k steps, achieving **50% faster convergence and higher accuracy**. \n\nSubTrack++ achieves superior performance by efficiently tracking subspaces and the synergy of its components. While the Related Work section (lines 309–317) summarizes the baselines, we agree that a more explicit comparison would be beneficial. We will revise the main text to clearly highlight these differences and better attribute the sources of SubTrack++'s improvements.\n\n> **Weakness 3:**  *There are also some related works in the literature that can be discussed. For example LoRo [1] also utilizes Riemannian optimization for subspace tracking, while LoRA-RITE [2] also utilizes Projection-aware optimization on the momentum with a seemingly different formulation.*\n\nThank you for pointing out LORO and LoRA‑RITE. While both highlight valuable ideas, geometric structure (LORO) and transformation-invariant updates (LoRA‑RITE), they are both LoRA-based and not directly comparable to gradient-based low-rank methods like ours. We appreciate the suggestion and will include them in the related work for completeness. \n\n> **Weakness 4:** *Some empirical results look counterintuitive. Specifically, **(a)** the larger models perform worse than the smaller one, which might indicate that suboptimal settings are being used. Additionally, **(b)** SubTrack++ sometimes outperforms the full-rank training, which also requires some explanation.*\n\n> **Question 2:** *Explain why in the experiments **(a)** larger models seem to perform worse than the smaller ones and why **(b)** SubTrack++ can outperform full rank training.*\n\nThank you for the observation. **(a)** The trend arises from the compute budgets, which is also employed in prior works [6]: as Table 1 shows, all methods, including baselines, experience the same trend as model size increases. To confirm this, we trained the 7B model for 100K steps and observed loss **drop from 4.61 to 3.37**, outperforming GaLore’s [3] 4.64 under identical conditions. We will include this result in the camera-ready version.\n\n**(b)** Regarding SubTrack++ occasionally outperforming full-rank training, we believe this stems from implicit regularization introduced by low-rank projections, which can enhance generalization in overparameterized models. Similar trends have been observed in LDAdam [4], APOLLO [7], and FiRA [5]. We will include a discussion of this.\n\n---\n\nWe hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed. If the concerns are resolved, we would be grateful if the reviewer considers updating their score accordingly. \n\n---\n\n[1] Qin et al., 2024. *InFoBench: Evaluating Instruction Following Ability in Large Language Models.*\n\n[2] Qi et al., 2023. *Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!*\n\n[3] Zhao et al., 2024. *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.*\n\n[4] Robert et al., 2025. *LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.*\n\n[5] Chen et al., 2024. *Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?*\n\n[6] Liang et al., 2024. *Memory-Efficient LLM Training with Online Subspace Descent.*\n\n[7] Zhu et al., 2025. *APOLLO: SGD-like Memory, AdamW-level Performance.*\n\n[8] He et al., 2025. *Subspace Optimization for Large Language Models with Convergence Guarantees*"}, {"review_id": "J9X06tHHGI", "reviewer": "Reviewer_bnSQ", "summary": "The authors tackle the problem of online subspace tracking by formulating it as minimizing a reconstruction cost on the Grassmann manifold. They introduce a lightweight rank-1 geodesic update that approximates the full tangent by its leading singular component, then moves the current subspace along the corresponding principal plane via a closed-form rotation. This approach avoids heavy matrix exponentials or full SVD at each step, yielding an algorithm that is both computationally efficient and easy to implement. Empirical results on synthetic and real-world streaming datasets demonstrate that the proposed method matches or outperforms state-of-the-art trackers in accuracy, while reducing per-update cost.", "questions": "1. For the subspace tracking step, I think you want to use the gradient method to minimize the cost function (2) on the Grassmann Manifold. However, the algorithm performs an SVD on $\\nabla F$ rather than on $-\\nabla F$? It seems like the algorithm is doing a gradient ascent on the Grassmann Manifold. Could the authors clarify why the SVD is taken on $\\nabla F$ instead of $- \\nabla F$?\n\n2. Why there exists a term $1 - \\beta_2^{t-1}$ in (9)? Suppose that $S_t = S_{t-1}$, the update rule for the first-moment Projection-Aware Optimizer step is reduced to regular adam like update, but it is not the same case for the second-moment. What caused such inconsistence?\n\n3. Since the proposed method is much more efficient that full SVD of the gradient, why the authors don't try a more frequent update to the subspace or a higher rank update?", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear derivation of the rank-1 geodesic step with full orthonormality proof on Grassmann. Solid empirical evaluation across multiple benchmarks; ablation shows robustness to step-size and noise.\n\n3. Well-written exposition of manifold basics and geodesic construction. Pseudocode aligns closely with the released PyTorch code.\n\nWeakness:\n\n1. Some details of the update rule is not well discussed (see Questions below).\n\n2. No ablation study on the subspace update frequency and the rank chosen for subspace update.", "was_revised": true, "final_justification": "Most of my questions and concerns are resolved.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and for recognizing SubTrack++’s solidity, efficiency, and robustness. Below, we address each of the raised concerns.\n\n---\n\n> **Weakness 1:** Some details of the update rule is not well discussed (see Questions below).\n\nWe thank the reviewer for pointing out these sources of confusion. We clarify them here and will ensure the final camera-ready version addresses them clearly to prevent any ambiguity.\n\n> **Question 1:** For the subspace tracking step, I think you want to use the gradient method to minimize the cost function (2) on the Grassmann Manifold. However, the algorithm performs an SVD on $\\nabla F$ rather than on $- \\nabla F$? It seems like the algorithm is doing a gradient ascent on the Grassmann Manifold. Could the authors clarify why the SVD is taken on $\\nabla F$ instead of $- \\nabla F$?\n\nSince the Grassmannian exponential map follows a geodesic with step size $\\eta$, gradient descent corresponds to moving along $-\\nabla F$, which can be expressed either via a negative $\\eta$ or a sign change in the sine term in Eq 5. The SVD computations of $\\nabla F$ can be translated to $-\\nabla F$ with sign flips in either the left or right singular vectors. We combine notation from [6, 7], but can see why this is confusing and will update the notation for consistency in the camera-ready version.\n\n> **Question 2:** Why there exists a term $1 - \\beta_2^{t-1}$ in (9)? Suppose that $S_t = S_{t-1}$, the update rule for the first-moment Projection-Aware Optimizer step is reduced to regular adam like update, but it is not the same case for the second-moment. What caused such inconsistence?\n\nAdam maintains unbiased estimates of the first and second moments. When the subspace shifts, we linearly rotate the (biased) first-moment accumulator into the new basis and then resume the standard Adam update; no extra scaling is needed because the first moment transforms linearly. For the second moment, we first compute a projection-aware statistic from the unbiased moments using a covariance approximation, then re-bias it by scaling with $1-\\beta_2^{t-1}$ before continuing the updates. This re-biasing is necessary because Adam’s second-order state is nonlinear. Also, between subspace updates, these rules are reduced to standard Adam. A similar strategy is used in LDAdam [1].\n\n> **Weakness 2:** No ablation study on the subspace update frequency and the rank chosen for subspace update.\n\n> **Question 3:** Since the proposed method is much more efficient that full SVD of the gradient, why the authors don't try a more frequent update to the subspace or a higher rank update?\n\nFollowing the reviewer’s suggestion, we conducted ablations on subspace update frequency and rank using the LLaMA 1B model and we will include the results in the camera-ready version. \n\nAs demonstrated in the following table, the results show that more frequent updates improve performance, while omitting updates leads to noticeably higher loss. However, we have used the setting proposed by GaLore [2] and also adopted by our other baselines [3-5] for a fair comparison. \n\n| Update Interval |  50  | 100  | 150  | 200  | 500  | 1K   | 2K   | no update |\n|:----------------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:---------:|\n| Evaluation Loss  | 4.32 | 4.39 | 4.47 | 4.52 | 4.71 | 4.88 | 5.08 |   5.79    |\n\nFor rank, we find that rank-1 updates strike the best balance. Unlike full SVD, which is more sensitive to noise, rank-1 updates allow us to preserve prior subspaces and make controlled adjustments, leading to more stable optimization. Adaptive rank selection across layers remains a promising direction for future work.\n\n| Rank of updates | r = 1 | r = 2 | r = 4 | r = 8 | r = 16 | r = 32 | full-rank     |\n|------------------|--------|--------|--------|--------|---------|---------|----------------|\n| Evaluation loss  | 4.52   | 4.52   | 4.55   | 4.61   | 4.91    | 5.40    | Not converged  |\n\nThese findings confirm that SubTrack++ outperforms baselines even under conservative settings, highlighting the robustness of our design.\n\n---\n\nWe hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed. If the concerns are resolved, we would be grateful if the reviewer considers updating their score accordingly.\n\n---\n\n[1] Robert et al., 2025. *LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.*\n\n[2] Zhao et al., 2024. *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.*\n\n[3] Chen et al., 2024. *Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?*\n\n[4] Liang et al., 2024. *Memory-Efficient LLM Training with Online Subspace Descent.*\n\n[5] He et al., 2025. *Subspace Optimization for Large Language Models with Convergence Guarantees.*\n\n[6] Balzano et al., 2010. *Online identification and tracking of subspaces from highly incomplete information.*\n\n[7] Bendokat et al., 2023, *A Grassmann Manifold Handbook: Basic Geometry and Computational Aspects.*"}, {"review_id": "K79Q7tsPkg", "reviewer": "Reviewer_qFVw", "summary": "This paper proposes SubTrack++, a novel memory-efficient algorithm for LLM training using gradient projection. SubTrack++ is composed of several key components: i) the novel subspace tracking mechanism based on the Grassmannian manifold; ii) the momentum inheritance mechanism similar to LDAdam; iii) the recovery scaling mechanism similar to Fira. Combining the above techniques, SubTrack++ can not only surpass existing algorithms in training performance, but also be more memory and computation-efficient.", "questions": "In the provided source code, I did not find codes that support the error feedback mechanism in LDAdam (Maybe I've just missed it). It seems that the code for reproducing the results regarding LDAdam (or other baselines not supported by the provided codes) is not provided, so I wonder if they are really not provided. If so, have you reproduced their algorithms or used the publicly reported data?", "limitations": "The authors have addressed the limitations, including the constraints on using the Adam family and the scale of their experiment.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\ni) The proposed algorithm exhibits great performance in the experiment, both in training performance and in training efficiency.\n\nii) The overall paper is well presented and easy to follow.\n\niii) The effectiveness of the proposed subspace tracking mechanism over GaLore's method is well addressed by illustrative examples.\n\nWeakness:\n\ni) While the experimental performance is impressive, this paper somehow lacks more incentive analysis on why it can beat existing algorithms. The proposed analysis is based on a strong assumption that the gradient matrix is as simple as a linear transform of the weight matrix, and the result only accounts for convergence on the projected gradient, which may not be so informative as a convergence result on the original gradient.\n\nii) The model used in the fine-tuning experiment is pretty small. I acknowledge this may follow the same setting as GaLore's paper; however, in my experience, existing algorithms (using gradient projection) behave poorly in fine-tuning large models such as LLaMA-7B. Given the promising performance of SubTrack++, it should be a good choice to validate its advantage in this scenario.", "was_revised": true, "final_justification": "I have no unresolved concerns and retain my recommendation.", "rebuttal": "We thank the reviewer for recognizing the novelty of our method and acknowledging that its combination of components contributes to improved efficiency and performance. Below, we address the concerns raised by the reviewer in detail.\n\n---\n\n> **Weakness 1: (a)** While the experimental performance is impressive, this paper somehow lacks more incentive analysis on why it can beat existing algorithms. **(b)** The proposed analysis is based on a strong assumption that the gradient matrix is as simple as a linear transform of the weight matrix, and **(c)** the result only accounts for convergence on the projected gradient, which may not be so informative as a convergence result on the original gradient.\n\n**(a)** SubTrack++ improves over baselines by addressing their key bottlenecks. As shown in **Fig. 6 (Appendix)**, Grassmannian subspace tracking alone **accelerates GaLore’s [1] convergence by 29%** (up to 44% on the 3B model) in wall-time. By avoiding costly and noisy SVDs [4], reusing prior subspaces, and applying rank-1 updates, SubTrack++ reduces noise and enables smoother, more stable optimization (Fig. 5).\n\nCompared to FiRA’s [3] expensive SVDs, our method **cuts wall-time by 33% and final loss by 29%** (1B model). Against LDAdam’s [2] per-step PowerSGD updates, SubTrack++ **achieves 50% faster convergence** and higher accuracy using lightweight rank-1 updates every k steps—all while maintaining low computational complexity.\n\n**(b)** The assumption of the linear structure of the gradient is standard in the literature, notably in GaLore, which demonstrates its validity across linear and complex architectures like reversible networks [1], and methods such as FiRA [3], GoLore [4], and LDAdam [2] leveraging it, especially in large-scale LLMs. Exploring more complex and non-linear architectures (e.g., VLMs) remains a promising direction for future work.\n\n**(c)** Like other gradient project methods, we rely on the low-rank structures of the gradient, established in [1, 6]. This assumption has been empirically established by low-rank gradient-based methods, including SubTrack++ and our baselines [1-5], which occasionally even surpasses full-rank training. \n\nTo verify this, we also conducted an experiment measuring $G_t$ and the projected counterpart during LLaMA‑1B pre‑training: the **full‑gradient norm drops from 0.46 → 0.08**, while the **projected‑gradient norm drops from 0.45 → 0.05**, showing nearly identical trajectories. These results will be included in the camera-ready version.\n\n> **Weakness 2:** The model used in the fine-tuning experiment is pretty small. I acknowledge this may follow the same setting as GaLore's paper; however, in my experience, existing algorithms (using gradient projection) behave poorly in fine-tuning large models such as LLaMA-7B. Given the promising performance of SubTrack++, it should be a good choice to validate its advantage in this scenario.\n\nThis is an interesting point! We considered the standard experimental structure for our evaluations. Per reviewer’s recommendation, we are currently running those experiments. And we will add those in the paper if there are any notable insights. \n\n> **Question 1:** In the provided source code, I did not find codes that support the error feedback mechanism in LDAdam (Maybe I've just missed it). It seems that the code for reproducing the results regarding LDAdam (or other baselines not supported by the provided codes) is not provided, so I wonder if they are really not provided. If so, have you reproduced their algorithms or used the publicly reported data?\n\nWe used their own publicly available source code for the baselines to reproduce their results, including LDAdam.\n\n---\n\nWe hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed. \n\n--- \n\n[1] Zhao et al., 2024. *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.*\n\n[2] Robert et al., 2025. *LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.*\n\n[3] Chen et al., 2024. *Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?*\n\n[4] He et al., 2025. *Subspace Optimization for Large Language Models with Convergence Guarantees.*\n\n[5] Zhu et al., 2025. *APOLLO: SGD-like Memory, AdamW-level Performance.*\n\n[6] Gur-Ari et al., 2018. *Gradient Descent Happens in a Tiny Subspace.*"}, {"review_id": "5XyPqrwANl", "reviewer": "Reviewer_Y7Ku", "summary": "This paper proposes SubTrack++, a memory- and time-efficient method for scalable low-rank optimization in large language model (LLM) training. SubTrack++ addresses the memory bottleneck in LLMs by combining three key components: (1) Grassmannian subspace tracking for adaptively estimating low-rank projection spaces without relying on computationally expensive SVD, (2) a projection-aware optimizer that incorporates subspace changes into the momentum update rules of Adam, and (3) a recovery scaling mechanism to recover information lost during low-rank projections. The paper provides theoretical convergence guarantees and validates the method through extensive experiments across LLaMA-based pretraining, GLUE/SuperGLUE fine-tuning, and synthetic robustness benchmarks. Results show that SubTrack++ outperforms existing low-rank methods like GaLore and FIRA in terms of loss, memory, and wall-time, while approaching full-rank performance. The method is also demonstrated to be robust to abrupt subspace shifts and scale changes.", "questions": "1. Have you considered applying SubTrack++ to decoder-only or multimodal architectures?\nThe current experimental scope is focused on encoder-only LLaMA models and text classification tasks (e.g., GLUE/SuperGLUE). Given the rising importance of decoder-only and multimodal models (e.g., LLaVA, Flamingo), it would be useful to understand whether SubTrack++ can be seamlessly applied to these architectures. Any clarification or additional discussion on its architectural flexibility would strengthen the applicability claim.\n2. Why not include a quantitative analysis of projection error or subspace drift over time?\nThe paper emphasizes the robustness of SubTrack++ in tracking subspace changes, yet it lacks quantitative measures such as geodesic distance or angle between subspaces to support this claim. Could the authors provide an empirical analysis of subspace drift over the training trajectory? This would help substantiate the claimed robustness benefits and differentiate SubTrack++ from SVD-based methods more clearly.", "limitations": "While the paper presents strong empirical and theoretical results, there are several limitations that merit discussion:\n\nLimited Architectural Coverage:\nAll experiments are conducted on encoder-only architectures, including LLaMA and RoBERTa. The method's performance on decoder-only models or multimodal systems—common in generative modeling and vision-language tasks—remains untested, raising questions about its generality across architecture types.\n\nScalability Beyond 7B Models:\nSubTrack++ is evaluated on models up to 7 billion parameters. However, its behavior at larger scales—such as memory overhead, subspace tracking stability, and optimization dynamics in 13B or 65B models—has not been explored. These aspects are crucial to assess its viability for full-scale deployment in industrial LLM settings.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper presents a well-motivated, efficient, and robust method for scalable LLM training, combining geometry-aware subspace tracking, optimizer adaptation, and recovery scaling, and validates its effectiveness through rigorous theoretical and empirical analysis.\n\nWeaknesses:\n\n1. Limited architectural and task diversity in experiments\nMost experiments are conducted on encoder-only models (e.g., LLaMA-1/2) and primarily within language modeling and GLUE-style tasks. The method has not been tested on decoder-only architectures, vision-language models, or instruction tuning scenarios. This limits the conclusions one can draw about its general applicability to a wider range of model types and tasks.\n2. Unverified scalability beyond 7B models\nAlthough SubTrack++ is shown to be effective up to 7B parameter models, the paper does not investigate its behavior on larger-scale models such as 13B or 65B. Without insights into numerical stability, memory behavior, and performance at such scales, the practicality of the method for the largest LLMs remains uncertain.", "was_revised": true, "final_justification": "All concerns have been resolved. I would keep my score since it is already positive.", "rebuttal": "We thank the reviewer for their valuable points and recognizing SubTrack++ efficient and robust. In the following we will answer the raised concerns.\n\n---\n\n> **Weakness 1:** Have you considered applying SubTrack++ to decoder-only or multimodal architectures? The current experimental scope is focused on encoder-only LLaMA models and text classification tasks (e.g., GLUE/SuperGLUE). Given the rising importance of decoder-only and multimodal models (e.g., LLaVA, Flamingo), it would be useful to understand whether SubTrack++ can be seamlessly applied to these architectures. Any clarification or additional discussion on its architectural flexibility would strengthen the applicability claim.\n\n> **Weakness 2:** Unverified scalability beyond 7B models Although SubTrack++ is shown to be effective up to 7B parameter models, the paper does not investigate its behavior on larger-scale models such as 13B or 65B. Without insights into numerical stability, memory behavior, and performance at such scales, the practicality of the method for the largest LLMs remains uncertain.\n\n> **Question 1:** Have you considered applying SubTrack++ to decoder-only or multimodal architectures? The current experimental scope is focused on encoder-only LLaMA models and text classification tasks (e.g., GLUE/SuperGLUE). Given the rising importance of decoder-only and multimodal models (e.g., LLaVA, Flamingo), it would be useful to understand whether SubTrack++ can be seamlessly applied to these architectures. Any clarification or additional discussion on its architectural flexibility would strengthen the applicability claim.\n\nWe would like to gently correct the reviewer, we test our method on decoder-only (LLaMa-based) for pre-training experiments, and encoder-only (RoBERTa-based) for fine-tuning tasks. Notably, such analysis and experimental set-up is standard in the literature, including GaLore [1], FiRA [3], LDAdam [2], and GoLore [4]. Exploring more complex architectures (e.g., VLMs) remains a promising direction for future work. \n\nLikewise, all of the recent methods (LDAdam [2], GaLore [1], FiRA [3], APOLLO[5], and GoLore [4]) also evaluate models up to 7B parameters, and our results show consistent improvements under these standard evaluation settings. To the best of our knowledge, none of the recent SOTA works report empirical studies for 13B or 65B. Our results at 7B already align with the current state of large-scale experimentation and provide a solid basis for future work at even larger scales.\n\n> **Question 2:** Why not include a quantitative analysis of projection error or subspace drift over time? The paper emphasizes the robustness of SubTrack++ in tracking subspace changes, yet it lacks quantitative measures such as geodesic distance or angle between subspaces to support this claim. Could the authors provide an empirical analysis of subspace drift over the training trajectory? This would help substantiate the claimed robustness benefits and differentiate SubTrack++ from SVD-based methods more clearly.\n\nWe thank the reviewer for the insightful suggestion. In response, we now empirically quantify subspace drift by measuring the norm of the tangent vector $\\nabla F$, which captures the deviation between the projected gradient and the original matrix. Our analysis on Llama with 350M parameters shows that this norm quickly decays from 0.06 to less than 0.0002 within 5–7 updates and remains extremely close to zero, indicating rapid convergence and stable subspace tracking. This supports SubTrack++'s robustness and alignment with the underlying subspace, consistent with prior results on Grassmannian optimization [6–11]. We will this analysis to the camera-ready version. \n\n---\n\nWe hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed. If the concerns are resolved, we would be grateful if the reviewer considers updating their score accordingly.\n\n---\n\n[1] Zhao et al., 2024. *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.*\n\n[2] Robert et al., 2025. *LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.*\n\n[3] Chen et al., 2024. *Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?*\n\n[4] He et al., 2025. *Subspace Optimization for Large Language Models with Convergence Guarantees.*\n\n[5] Zhu et al., 2025. *APOLLO: SGD-like Memory, AdamW-level Performance.*\n\n[6] Balzano et al, 2012. *Handling missing data in high-dimensional subspace modeling.*\n\n[7] Balzano et al., 2018. *Streaming PCA and subspace tracking: The missing data case.*\n\n[8] Balzano et al., 2010. *Online identification and tracking of subspaces from highly incomplete information.*\n\n[9] He et al., 2011. *Online robust subspace tracking from partial information.*\n\n[10] Chakraborty et al., 2017. *Intrinsic grassmann averages for online linear and robust subspace learning.*\n\n[11] Blocker et al., 2023. *Dynamic subspace estimation with Grassmannian geodesics.*"}]

Author rebuttal:

Meta-review: This was a rare case where the author response made some reviewers better appreciate the contribution and some reviewers appreciate the contribution less. So my uncertainty is high here, but all reviewers ultimately gave a recommending score so that seems like the most logical recommendation.
Decision: Accept (poster)