Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Authors: Ipsita Ghosh
Ethan Nguyen
Christian Kümmerle
Venue: NeurIPS 2025 poster ()
Keywords: Low-Rank Optimization
Model Compression
Low-Rank Adaptation
Parameter-Efficient Pre-training
Iteratively Reweighted Least Squares
Log-Determinant
Primary area: deep_learning
TL;DR: We propose the novel low-rank regularization methodology Q3R which enables robust pre-training of low-rank models for the first time in the literature.

Abstract: Parameter-efficient training, based on low-rank optimization, has become a highly successful tool for fine-tuning large deep-learning models. However, these methods fail at low-rank pre-training tasks where maintaining the low-rank structure and the objective remains a challenging task. We propose the Quadratic Reweighted Rank Regularizer dubbed Q3R, which leads to a novel low-rank inducing training
strategy inspired by the iteratively reweighted least squares (IRLS) framework. Q3R is based on a quadratic regularizer term which majorizes a smoothed log determinant serving as rank surrogate objective. Unlike other low-rank training techniques, Q3R is able to train weight matrices with prescribed, low target ranks of models that achieve comparable predictive performance as dense models, with
small computational overhead, while remaining fully compatible with existing architectures. In experiments, we are able to truncate 60% of the parameters of a ViT-Tiny parameters with marginal loss in CIFAR-10 performance and up to 80% with only 4% accuracy drop. The efficacy of Q3R is confirmed on Transformers across both image and language tasks, including for low-rank fine-tuning.

Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Ipsita Ghosh"
Ethan Nguyen
Department of Computer Science University of Central Florida ipsita. ghosh@ucf...mmm edu
Department of Computer Science
University of North Carolina Charlotte ethan nguynnge- .n
Christian Kümmerle
School of Data, Mathematical and Statistical Sciences
Department of Computer Science
University of Central Florida
Cnemmerle@uch edu
Abstract
Parameter-efficient training, based on low-rank optimization, has become highly successful tool for fail low-rank pre-training tasks where maintaining the low-rank structure and the objective remains challenging task. We propose the Quadratic Reweighted Rank Regularizer dubbed Q3R, which leads to novel low-rank inducing training strategy inspired by the iteratively reweighted least squares (IRLS) framework. Q3R is based on a quadratic regularizer term which majorizes a smoothed log determinant serving rank surrogate objective. Unlike other low-rank training techniques, Q3R able to train weight matrices with prescribed, low target ranks of models that achieve comparable predictive performance dense models, with small computational overhead, while remaining fully compatible with existing architectures. In experiments, ViT-Tiny parameters with marginal loss in CIFAR- performance and up to 80% with only 4% accuracy drop. The efficacy of Q3R is confirmed Transformers across both image and language tasks, including for low-rank fine-tuning.
large deep-learning models. However, these methods
able truncate 60% of the parameters of
The code is available at httttpp://ggitnni hwtp:///ooo/aatt////eeea/
Introduction
Modern deep learning architectures continue to grow in size and complexity (RWC+19). creating growing demand for efficient training methodologies. Low-rank regularization has emerged as a powerful paradigm for addressing these challenges by explicitly constraining the parameter search space through matrix factorization. This approach builds on the empirical observation that neural networks exhibit inherent low-dimensional structure in their weight matrices during training
Practical implementations face three key challenges: (1) performance degradation compared to full-rank baselines, (2) optimal rank selection across layers, and (3) maintaining training stability. Prior work addresses these through spectral initialization (GK (22, orthogonality regularization
(YYT-20 20).
* Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent advances in parameter-efficien fine-tuning (PEFT) have expanded the low-rank training paradigm through methods like Low-Rank Induced Training (LoRITa) (AZW). These approaches maintain the original model architecture during inference while inducing low-rank structure through strategic layer overparamettriization during training. LoRITa specifically decomposes weight matrices W; into products I W during optimization, enabling implicit rank reduction through singular value truncation post-training (AZW). This methodology demonstrates that explicit rank constraints can be replaced by training dynamics that naturally favor low-rank solutions.
Despite their promise, existing low-rank training approaches present several notable limitations. Traditional low-rank methods often suffer from performance degradation relative to full-rank baselines 22 YYT 22)). Methods such as LoRA and LoRITa, while effective at reducing trainable parameters, can struggle to capture the full structure making it difficult to generalize to complex tasks (AZW). Furthermore, PEFT techniques introduce additional hyperparameters (such rank and scaling factors) whose optimal values may not generalize across architectures, datasets, or downstream tasks, often requiring extensive re-tuning and experimentation. In multilingual or low-resource settings, PEFT methods like LoRA have been observed to yield inconsistent results, sometimes improving language-specific generation at the expense of reasoning abilities (JPP*55) 25). Combining multiple PEFT modules for multi-task or continual learning can also lead to increased memory usage and system complexity, offsetting some of the intended efficiency gains. Overall, it can be observed that the advances in LoRA-type parameter-efficien training methods have not yet been able to be translated enable robust low-rank pre-training.
generalization
2 Contribution
We propose Quadratic Reweighted Rank Regularization (Q3R), which solves the pre-existing problems by introducing an optimizer--ooppabble regularization framework based on smoothed log- oog-determmnant rank surrogates outlined in Section 4.1 which is specifically designed for low-rank pre- rre-training Our approach is theoretically grounded in saddle-escaping second-order optimization methods, and it comes with little computational overhead compared to unregularized training despite its efficacy for promoting low-rank neural network weight matrices. Additionally, propose the Adam variant AdamQ3R in Section 4.2 which is tailored to optimizing Q3R-regularized loss functions and which improves the performance of training Q3R-regularized models.
Numerical experiments show that Q3R is able to reduce the number of parameters in ViT models by 60% during pre-training on CIFAR-10, with only around 1.3% accuracy drop. We validate the performance of Q3R for low-rank fine-tuning with experiments fine-tuning RoBERTa and Llama3 on GLUE tasks, for which Q3R achieves comparable performance compared to dense fine-tuning and state-of-the art low-rank PEFT methods. Compared to state-of of-the-art low-rank training methods such as LoRA (HSW 22b), LoRITa (AZW), Q3R consistently produces models with better generalization at high truncation levels, without requiring overparameterization or full-rank warmup phases.
In Section 4 we elaborate the methodology of the proposed work, which is further discussed with detailed derivation in the Supplementary material Appendix [A] In Section[5] we empirically show the performance of Q3R, in comparison to other state-of-ther -frt----t-e-rtttt methods. We continue with more experimental evaluations in the Supplementary material in Appendix D Appendix
includes discussions of the computational aspects of our methodology. In Appendix L.] we demonstrate the robustness of Q3Rto different hyperparameter variations. We briefly discuss the limitations of our work in Section 6
3 Related Work
Parameter-Etttiien Fine-Tuning (PEFT) Parameter-efficient fine-tuning the concept of modi- fying only parts of a fully parametrized pre-trained model to excel at specific task of interest. PEFT methods such as adapters 19) and LoRA (HSW+22b) introduce small, trainable modules into a frozen pretrained model, drastically reducing the number of parameters to be updated. These techniques often match full fine-tuning performance with only a tiny fraction of trainable parame- ters. However, the low-rank constraints that make LoRA-style methods efficient for downstream tasks also limit capacity if applied during pre-training. Training from scratch with only low-rank adapters factorizations (instead of full-rank weight updates) tends to underperform, it restricts
2

---

optimization to a Now-dimensionas subspace (4) LoRA assumes well-formed pretrained weight W plus a low-rank perturbation of rank-r adapter matrices A R and B Brxd such that AW - AB; without strong initial W, such updates struggle to capture the full complexity needed for learning from scratch. KronA replaces LoRA's product with a Kronecker factorization for better rank-paramee parapmarreeer trade-offs (ETK 225). DoRA decouples update magnitude and direction via learnable scaling factor, improving upon LoRA's expressivity (L) Compacter uses shared, low-rank, parameee--rr adapters across layers, matching standard adapters with only 0.05 % extra parameters (MHR2I)
Low-Rank Training in Neural Networks. Neural networks often exhibit implicit low-rank struc- ture during training, as optimization dynamics like SGD with weight decay tend to bias models toward low-rank solutions (GSGP25) HM)+23). This observation has motivated a range of explicit low-rank training methods that constrain parameter matrices directly. A common approach factor- izes weights and trains the factorized weights instead, reducing compute and memory costs with minor accuracy loss (TTM2TT. Techniques like LoRA (WMPG24b) and its extensions (LSMR23) inject low-rank updates into pretrained Transformer weights, enabling parameter-efficient adaptation. However, pre-training directly under low-rank constraints remains more challenging. (W (WMPG224)
shares a similar motivation to ours ours- studying the limitations of LoRA-style low-rank pre-training and proposing an alternative regularization-driven approach to induce low-rank structure during training. Although we approach the problem through a different optimization framework, their analysis and framing of the limitations of adapter-based methods are highly relevant and can guide refinement of both the positioning and justification of our method. Regularization -based approaches use nuclear norm or log- -eterrmmmnnn surrogates to promote low-rank solutions (SZCT23) while others apply orthogonality constraints and adaptive rank pruning (YYT* t 20) YCS + 20). In Transformers, low-rank parameterizations have achieved 2-5 x compression with minimal performance drop (AZW), and Cuttlefish (W3 automates rank selection by monitoring stable ranks during a warmup phase. Still, many methods rely on post-hoc truncation or overparameterrizaionn which do not minimize rank during training. Our work addresses this gap by directly optimizing for low-rank solutions via reweighted least squares, promoting compact representations throughout pre-training. However, many of these methods rely on overparameterization or post-hoc truncation and do not directly minimize rank during training. In contrast, our approach promotes low-rank structure directly via optimization, using a principled regularization technique rooted in reweighted least squares.
Spectral Low-Rank Regularization. A related line of work studies algorithms that impose low- rankness of neural network matrices based the nuclear norm, Schatten-p quasi-norm or direct rank regularization. In particular, (AS17) proposed a proximal stochastic gradient descent applied to the nuclear norm. Methods that apply spectral truncation (c.g.. via truncated SVD) during training post-training (YTW* 20; XLZ*20) can also be understood within this framework. A downside of such approaches is the computational overhead: they require at least truncated SVD at every iteration, which quickly becomes computationally prohibitive for larger networks. Moreover, arguably, such aggressive, discontinuous rank regularization interferes with the continuous gradient-based training process of the network.
In contrast, our proposed Q3R regularizer imposes low-rankness more gradually by reweighting at periodic intervals, at which a smoothing parameter is updated as well, which we find to be sufficient for convergence while significantly reducing the computational overhead. This "soft" imposition of low-rank structure aligns with insights from IRLS- -aaee methods (see below). A spectral, Schatten-p regularization is also the core of the motivation of LoRITa (AZW); however, in this case, the spectral regularization can be seen as a justification of an (unweighted) squared Frobenius norm regularization on factor matrices, whereas Q3R does not work with factor matrices and considers reweighted quadratic term.
Rank Regularization and IRLS. In a line of work that significantly precedes the interest in low-rank techniques for deep learning, the problem of identifying or learning low-rank matrix from noisy,
linear measurements has been studied for decades in control theory (FB03 DCM22), recommender systems (Kor09) KBV09) and compressed sensing (RFP1O DR16). Even in this setting, which is fundamentally linear unlike the training of deep neural networks, the minimization of a rank objective subject to the constraints NP-hard (Nat95) RFPI0) motivating surrogate formulations relaxations. Convex relaxations using the nuclear norm (CR09) RFP10
3

---

DRI6: CW18) have been popular for a long time due to their strong recovery guarantees under suitable assumptions and their ability be tackled using the machinery of convex optimization (BV04), but fail to result in convex formulation in the deep learning setting. Even disregarding computational limitations of convex regularizations (CC18), non-convex rank surrogates such as the log-determinant penalty (FHB03) lead to algorithms which are more data-efficient as a evidenced for a variety of low-rank matrix recovery problems (FHB04 CESVI3 KS 88 KM23).
If combined with suitable smoothing strategy, the non-smooth optimization framework of Iteratively Reweighted Least Squares (IRLS), originally pioneered by Weiszfeld (Wei37; [P09] BST5), has emerged as a leading algorithmic framework to optimize non-convex rank surrogates (FRW II MF10 KMV2I GTK24) as it provides good trade-offs between scalability, eattaeefficeecy, saddle-poin evasion (present due to inherent non-convexity) and fast convergence. On a high level, IRLS solves sequence of weighted Frobenius-norm problems that progressively suppress smaller singular values. The proposed rank regularization term Q3R (detailed in Section 4.1 builds on recent improvements on low-rank IRLS weight operator formulations (KV221) [221]TTKK (TK244) (or reweighting strategies), which, unlike older formulations (FRWII MF10), allow for fast saddle-point evasion and locally quadratic convergence rates. To the best of our knowledge, IRLS-type low-rank regularization, which is at the core of Q3R, has not been explored in the literature in the context of deep learning so far. While providing an interesting perspective on older IRLS formulations TRRWIIMFIO from an average gradient outer product perspective, the recent work (RBD25) does not provide insights towards the derivation of quadratically convergent IRLS methods (KMV21 GTK24). nor does it extend the framework towards low-rank training of deep networks.
In the language of the low-rank recovery literature, LoRA-type HSW approaches are known under the name of CLC19 [ZC222, XSCM23, SZZ25). While popular in applications (KBV09) RKZK22) due to their scalability, is known that they can be outperformed by IRLS or Riemannian optimization approaches in more challenging setups involving, e.g., limited data (ZN22 LHLZ24), which is one motivation of
(BM03) matrix factorization methods (6L16) ZL13 MWCC20
our work.
4 Methodology
In this section, we provide a detailed derivation and definition of the Quadratic Reweighted Rank Regularizer Q3R in Section 4.1 before we embed it into training scheme train low-rank weights of deep learning models in Section 4.2
4.1 Low-Rank Regularization via Q3R
Given a neural network with K weight matrices θ (W; W; is weight matrix, I,..., K}. functionally ideal regularization term to add to the loss function of the network for the promotion of a low-rank weight is simply the rank of W. However, rank( () W is non-convex and not continuous, and thus hard to incorporate into a gradient-base based training methodology.
In the following, we consider the non-convex, but continuously differentiable rank surrogate F.(-) called e-smoothed log-determinant, defined as
(22 (log(o) - log(t)) + 2e2, F((W)= = Σ fo(o,(W)), where fe(o) = 1302,
i 3 6, <
(I)
where oi (W) is the i-th singular value of W. and which is parametrized by a smoothing parameter > O. The definition of follows (KM23) and is related to the log- degerreernn heuristics log det( W + +e) nd 1- (og(),(W)++ E) defined for positive semi-definite matrices W e dxd of (FHBO3 CESVI3 RBD25), Compared to other log-determinant type functions, F.(-) from ( has few advantages: It is lower bounded by 0 for any and has 1-Lipschitz gradient, making the objective compatible with gradient-baad based optimizers without extensive step-size adaptation. Furthermore, its smoothing parameter regulates the non-convexity of its optimization landscape and recovers the well-known squared Frobenius norm as (, = W in the case of a o (W) < c.
The rank regularizer we study in this paper, however, is not simply F((:): If we were to work directly with the e-smoothed fogedeterm,nant, its gradients F.()W) would require a full spectral decomposition of W at each training iteration (see supplementary material).

---

Instead, we consider, given an expansion center point W (which may correspond to the current weight matrix during a neural network training dynamies), the quadratic model Qe(- (c()))) defined as
Qc(W W F((W)) (W, Rw 1 (W', ,w (W"))
where Rw (+) IR xd2 - chth xd2 is positive definite, so-called reweighting operator KMV21 GTK24), defined in Definition 4.1 below.
Definition 4.1 (Reweighting Operator (KM23)). Let E O and W' gd xda be matrix with singular value decomposition w Uw (lag));(WW)))) w" where U E Rd ( .) and V dddxx((",d) () are matrices of the leading r r(W', c) left and right singular vectors satisfying Uw = Ui]E e (gdixdi and V [V Vi] ] RRdaxd, and
[U
r( .c) = Ki ......,,inn(dd, (2)) :o,(W)) > c]]
is the number of singular values of w' larger than Then we define the reweighting operator Rw Rd xda - [Rd xd2 associated to the matrix W' and smoothing parameter as
,w Uw Lnd, Uw WVw. w'
where De,d diag(max(o; (W")/e, 1)) e gdxd for e (d1,d2).
The reweighting operator satisfies the following simple properties (shown in the supplementary material), which makes working with it computationally feasible.
Lemma 4.1. For e > 0 and W', let U E IRd xdixr(W".)) :s) V E Rdixr(W.e)) ,c) and Rwi : Redixda
Rd xd2 be as in Definition 4.1 Then the following statements are true:
1. .e)) is a positive definite operator with respect to the Frobenius inner product (A,B) tr( (A(A)) UB), ie., (W, (Rw,((W) > 0 for all non-zero W E R.d xd2
The image .w.((W) of any E Rdd xd2 w.r. the reweighting operator can be computed as
Rw)((W) = 2 U WV + -UX-IU-W(I-V)
(4)
+ + UUTJUUU-VV + +I- (U-UU")(I-vV))
where Σ = d (W") (W".))
" ( ())((" is the diagonal matrix containing the largest
r(W',e ,e) singular values of W"
3. The quadratic model of (2) satisfies, for all W, W' E Idd xd2 that
Q.(W| W') = (W)) + (WF.WW), W W') ( WW, Rw (W W')).
(5)
We note that the quadratic model Q.(-) W (2) defined by Rw a majorizing quadratic model that satisfics Qe (W (W) > F.(W (W) for all W ch x It is different from a recond-order Taylor expansion of the Fc(-) about W which would only be an approximation, but no majorization due to the non-convex nature of Fe(.). The quadratic model can still be related to a second-order Taylor expansion of F. via (5) each generalized Hessian (HUSN84) F²F. (W') of F, satisfies 22 (W") 3 Rw in the Loewner order.
We observe that in the quadratic model (W W') of (2) the only term that depends W is the second summand. Thus, to obtain a simple, differentiable regularizer term that can be incorporated into deep learning framework, we define the Quadratic Reweighted Rank Regularizer Q3R of a neural
network weight matrix W E Rd x with
given W E Rd xd2 and O, Q3R ggd xd R
1 2
Q3R (W)
W. Rw R. (W)).
(6)
As see in the next section. it is simple and tractable to compute its gradient V Q3Rw (W) d xd2 which can be used by any gradient- based optimizer.
"This majorization property is implicitly postulated in (KMV21 KM23), but without proof. While proving this property is beyond the scope of this paper, we believe that the statement true.
5

---

Algorithm 1 Update Reweighting Operator Ro - Rw.
1: Input: NN weight matrix W x target rank V'arget prev. smoothing parameter cold-
2: Output: Updated new, reweighting operator Rw.e. (via E, U, V), envelope rank l'av
3: Compute [U, E, VJ= SVD nou (W) ) of W, where SVD (-) computes partial singular value decomposition of its input up order r(; Eola) (see (3) as well rruset + (W).
4: fnew min (Fold: +1))++((W::
b- UPDATE SMOOTHING (7)
5: Tenv r( W. Cnew)
P- UPDATE RANK ENVELOPE
6: Set U = U 1:r V= Vll E = 21rra,, B RESTRICT PART. SVD MATRICES
7: return Reweighting operator Rw. implicitly defined by Ue XTen
VeR X
E & Enew-
Furthermore, we periodically (but not at each training iteration) update the reweighting operator of
Q3R (and thus, the underlying quadratic model Q+ W")) by setting W - W and re-compute
Rw for which a truncated SVD of W is sufficient due to (4) Additionally, whenever updating
R. .c. apply the non-increasing update
min(c, Trupe (W))
the smoothing parameter e, which uses a target rank parameter Taagee as an input. The rationale of this smoothing parameter update is two-fold: first, this choice gives partial control on the expected rank of the weight matrix after training, as the value of Rw',, (W) tends to 0 if follows the dynamics (f in the case of e - 0 for matrices W whose row and column spaces are both orthogonal to the columns of U and V, respectively. Second, this choice increases the non-convexity the e-smoothed log- deg-determnnat F, underlying Q3R gradually (YATC20), facilitating a fast convergence to true low-rank solutions without becoming trapped high-rank local minima (KMV21 KM23). We summarize computational steps of a reweighting operator update in Algorithm !
4.2 Neural Network Training via AdamQ3R
Let now y [d xd] x Re xdf x I da be the input-output mapping of deep neural network that depends on weight parameter matrices θ = {Wx E Rd xay : We is weight matrix, = 1, 1...., KJ. For Transformer-based architecture such as Vision Transformer (BBKea21), the weight matrices include square layer- and head-wise query, key and value weight matrices W g: We, W, E Rddd as well rectangular projection and MLP layer weight matrices. Given pairwise loss {(.,)) such as cross entropy and a training dataset ( - 1. we can define the (unregularized) network loss C(0) = - ((y(O, 2x,), y:).
Algorithm 2 I Low-Rank Training via AdamQ3R
In order to gradually impose low-rank weights dur- ing training, propose to optimize instead the Q3R-regularized total loss Lqaa(O) := L(O) + A A
Input: Minibatch size B, reweighting period T, Q3R param- eter X, learming rate a = 0.001, 31 = 0.9, B2 = 0.999, = 10 10-*, y = 3, target rank Varge-
2=k-1:00 1:033 is active for W, Q3Rw, (Wa). where
1: Initialize parameter Wa. eg and reweighting operator RWe.co
0 a regularization parameter and the Q3R (.) are as in (6) the (W2) are ini- tially set to the initialization weights, and en = oo for each A: 1,..., K. We observe that due to the definition of Q3R. the gradient with re- spect the regularizer terms can be computed as Vw Q3R c. (W&) = Rwi (W&) for each k. i.e., by computing the image of Wa with respect to the reweighting operator R (f) of
2: for 1. 1. do
3: if mod T then
4: Update reweighting operator [(+(()
Rw....)) (.) and ct end if
USE ALGORITHM
Sample minibatch S ((,,, 1.)1
got1 -WwCs
COMPUTE BATCH
m.+1 -,,,, + (1 31)83-1
GRADIENT OF L
9:
Ve+1 -.2v, (1 32)ge+1
10:
m - -m.+1(11 (1 -s9 36+1)
11:
Ve+i - -V++1/(1 18
Definition 4.1
12:
R, - R R[+ (.) COMPUTE Q3R GRADIENT (4)
The QSR-regularized loss LasR can now be used in conjunction with any optimizer suitable for the neural network architecture such as minibatch stochastic gradient or Adam (A+14). To ensure that the quadratic models underlying Q3R match the E smoothed deg-determinant rank surrogates
13: W W, o(
Veni AR,
ami41
14: end for
15: return W,

---

Fez (1) closely, we update the reweighting operators ee (-) via Algorithm 1 on a fixed iteration schedule of every T training iterations-we call this parameter T the reweighting period.
However, instead of using generic adaptive gradient optimizer such as Adam on Luss, we observe that the Q3R terms already possess accurate second-order information of underlying regularization surrogate, which means that including the Q3R terms into the adaptive part of am Adam-like optimizer is likely to be suboptimal. For this reason, we propose use dedicated adaptive optimizer to optimize Lasa. dubbed AdamQ3R, which is detailed in Algorithm 2 AdamQ3R extends the observation of Adam W (LH19) that decoupling of regularization term (in that case, squared Frobenius norm regularization) and network loss improves generalization performance to Q3R regularization, avoiding distortion of the loss landscape. A validation of the benefits of using AdamQ3R vs. standard Adam applied to LqRR can be found in the supplementary material.
Computational Aspects. Following the low-rank training framework of Q3R, for example, via AdamQ3R, introduces a limited computational overhead compared to unregularized deep learning. Every T training iterations, a truncated singular value decomposition of order Tew (see Algorithm I of each weight matrix W, E Rgdixds to which Q3R is applied is required, which has time complexity (HMTII) Similarly, calculating Q3R gradient R, in Algorithm 2 imposes a total cost of O (d d2Teew + (d1 + d2)r emy + + Since the smoothing parameter update rule (7) designed to relate Tenv with the target rank Furget such that Tew & " l'rgge. the additional time complexity is somewhat proportional to the target rank. To obtain significant parameter reductions in the trained network weight matrices, it chosen such that I' Fuaget < min( d1, d2), which limits the computational overhead of Q3R in the most interesting use cases. Additional memory requirements amount to Femo(d, + d2 + 1) per weight matrix as the reweighting operator information needs to be stored via U, V and E.
of d2d + (di + d2) Dr2m
5
Experiments
We explore the ability of Q3R to obtain favorable trade-offss between model performance and parameter- efficiency across diverse architectures and data distributions experimentally. To this end, we compare different low-rank training methodologies across range of architecture-dattse pairs: pre-train ViT-Tiny (DBKea21 SKZ 22) on CIFAR-10, ViT-Base (DBKea21) on CIFAR-100 followed by post-training low-rank truncation (AZW); further, fine-tune BERT-Large (DCLT19) on GLUE benchmark tasks (without truncation).
5.1 Low-Rank Pre-Training
We compare the accuracy of models trained by Q3R against baselines LoRITa AZW). LoRA (HSW+22a) and a model trained without low-rank regularization, after post-training truncation. After training, truncate each layer weight matrix WR E θ using a truncated SVD to obtain factor matrices A, B with inner dimension r and W 2 AB for a range of ranks r corresponding to different parameter retention percentages p. Depending on the experiment, we apply low-rank regularization to different subsets of weights [W+).
ViT-Tiny Trained on CIFAR-10. We train ViT-Tiny on CIFAR- for 100 epochs using a learning rate of 0.00004. We enable low-rank training for all Transformer blocks, accounting for 96% of ViT ViT-Tiny's 's parameters. We conduct hyperparameter sweep across different configurations, and Figure la shows the best performance achieved by each training method when rank regularization is applied to the MLP and QKV weights. From Table I we find that AdamQ3R retains 42.4% of the original parameters with only 1.22% performance drop, and retains 23.2% of parameters with a 4.4% performance drop, while LoRITa consistently underperforms in comparison. As shown in Figure Tb despite various hyperparameter configurations À and for LoRITa, AdamQ3R consistently outperforms LoRITa upon truncation.
ViT-Base Trained on CIFAR-100.
To demonstrate the performance of the low-rank pre-training methods more challenging dataset and larger model, we train the 86M parameter ViT-Base from scratch for 100 epochs on CIFAR-100 with data augmentation and a 0.0001 (SKZ+22)) In line with practice for large-scale Transformers LLX+244 we apply low-rank training techniques solely to
7

---

ModellParameter Retetion p
%%
10% %
15%
20%
30%
40 %
100
Vanilla VIT-T
0.1 0.1475
0.1252
0.1350
0.1213
0.1624
0.1524
0.6840
LoRA
0,3546
0.3655
0.3576
LoRITa D=2, A=10-1
0.0989
0.1433
0.1543
0.2125
0.3247
0.4523
0.7142
LoRITa D=3, A=10-1
0.2258
0,3861
0.4466
0.5035
0.6368
0.6740
0.7273
LoRITa D=3, x=10-3
0.1338
0.2136
0.3839
0.4560
0.5973
0.6253
0.7449
Q3R, farget = 10%, A=10
0.2322
0,4085
0.5606
0.6295
0.6526
0.6654
0,6843
Q3R, target = 15%, A=10
0.1796
0.4758
0.5883
0.6215
0.6455
0.6555
0.6737
Q3R, Marget 20%. A=10
0.1998
0.4737
0.6175
0.6511
0.6749
0.6833
0.6990
Q3R, Ftarget = 10%. A=10-2
0.2041
0.4387
0.6115
0.6449
0.6707
0.6771
0.6889
Q3R, Ttarget 15%, A=10-2
0.1313
0.3896
0.6158
0.6550
0.6689
0.6801
0.6982
Q3R, Farrget 20%, A=10-2
0.1870
0.4335
0.6123
0.6496
0.6744
0.6868
0.6962
Table I: MLP truncation performance of ViT-T, rank regularization is applied to both attention (QKV) blocks
and MLP blocks. For LoRA, factor ranks are adaptive to
Vanilla VIE-B
LoftTe Best
03R r-B ..... LoRTh D-2 ...... A-
03R Best
03R mi:4
LoRTb 0.001
231
Truncated Model Tast Accuracy
0.0
Accuracy
2.0
0.5
0.4
0.3
0.1
2.3
P..
15%
40
Percentage of Parameters Retained
Percentage Parameters Retained
(a) Upper Bounds Comparison
(b) AdamQ3R LoRITa Variants
Figure 1: Performance curves on CIFAR-10 10 with rank regularization applied to MLP and QKV blocks: (a) Best performance across methods, (b) AdamQ3R vs. LoRITa variants.
the multi-head self attention blocks (QKV, but not to the MLP blocks). Despite the additional size and complexity of ViT-Base compared to ViT-Tiny, Q3R remains robust and exhibits larger performance advantages with 0.40-0.44 test accuracy at 20% parameters retained, whereas LoRITa models do not exceed accuracy of 0.25 at the same truncation level despite their substantial overparametrization (Figures 2a and 2b
ViT-Tiny with Low-Rank Attention Weights. We train ViT-Tiny for 100 epochs on CIFAR- 10 (Kri09) from scratch with learning rate 0.0004, with low-rank regularization applied only to attention weights. We evaluate the methods for larger set hyperparameters as shown in Figure 3b using layer-wise truncation levels with retained parameter percentages (5%, 10%, 15%, 20%, 30%, 40% 50%, 60%, 70%, 80%, 90%]. and present results in Figure Fig- ure shows that Q3R experiences almost no performance drop up to p 30% for most parameter choices, exceeding the performance of reference methods. Figure FigureBblshows shoss that the worst performing Q3R model still outperforms any LoRA, LoRITa, vanilla ViT-Tiny below 60%, showcasing the method's robustness.
ViT-Base on ImageNet-1k. We train ViT-Base on ImageNet- Ik using Automatic Mixed Precision (MNA+ 18) with Augmentttoooioon (CZM+19) for 100 epochs. Training is conducted with learning rate of a x 10-5 a batch size of 384, and gradient clipping (ZHST20) across L40S GPUs. We observe that Q3R consistently outperforms the baseline model while utilizing fewer

---

maR Q3R LoRTa
Thuncated Model Test Accuracy
0.40
Truncated Modell Test Accuracy
0.30
0.00
0.5
0.0
0.1
Percentage Retained
Percentage
(a) Upper Bounds Comparison
(b) AdamQ3R vs. LoRITa Variants
Figure 2: Performance curves on CIFAR-100 with rank regularization applied to QKV blocks: (a) Best
performance across methods, (b) AdamQ3R vs. LoRITa variants.
Vanilla VIFT
LoRITA Best LoRA Best
CSR m00.5
Dog
Ama-0.01
G.R Bixst
Truncated Model Test Accuracy
R
0.05
Truncated Model Test
0.55
04
0.45
104255
30% 20
4276
50%
00%
50%
14
210
88
30
86
Percentage of Parameters Retained
Percentage Parameters Retained
Rank
Upper Comparison
(b) AdamQ3R LoRITa
Figure 3:
Performance curves on CIFAR-10 with rank regularization applied QKV blocks: (a) Best perfor- mance across methods, (b) AdamQ3R vs. LoRITa variants.
parameters. This performance advantage holds under two truncation paradigms: attention matrices only, and entire Transformer blocks. In both cases, Q3R maintains performance comparable to the full baseline model, as seen in Table
Optimizer Transformer Modules
0.15
0.2
0.3
0.4
0.5
AdamQ3R I QKV, MLP Adam QKV, MI.P
0.0138
0.1439
0,3376
0.421
0.4458
0,4556
0,5816
0.0193
0.0976
0.2950
0.399
0.4311
0.4523
0.5179
AdamQ3R QKV Adam QKV
0.1713
0.3016
0.4623
0.4895
0.4952
0.4975
0.5816
0,2552
0.3366
0.4551
0.4882
0.4882
0.4937
0.5179
Table 2: ViT-Base on ImageNet- Ik validation accuracy post- cost-truncation on the last epoch
5.2 Low-Rank Fine-Tuning
Q3R not only induces a low-rank structure during pre-training in memory-effiiien manner, but also extends naturally to compact fine-tuning. We fine-tune pre-trained RoBERTa models on the GLUE benchmark using AdamQ3R with the proposed Q3R regularizer, and compare against full fine-tuning and LoRA (HSW (HSW+22b) 22h). We impose Q3R on the weight matrices that are added to the full-rank pretrained weight matrices. For LoRA, we adopt the hyperparameters from (HSW+221 and for Q3R we cross-validate the learning rate and regularization hyperparameter X. As shown

---

Table 3: GLUE Benchmark Scores
Method
MRPC
RTE
CoLA
STS-B
SST-2
QQP
MNLI
QNLI
Average
Dense Fine-tuning
91.9
77.62
62.3
90.19
94.04
90.2
87.3
91.49
85.88
LoRA rank=4
89.04
73.55
56.25
89.86
94.3
90.11
87.00
92.5
84.58
Q3R rank=4
92.24
77.23
63.50
90.19
92.2
91.60
87.2
90.20
85.86
Q3R matches or exceeds LoRA's accuracy most tasks and exhibits a performance closer dense fine-tuning. These results demonstrate that Q3R can serve as a unified, low-rank training strategy both for pre-training and fine-tuning of Transformer models. We discuss additional fine-tuning experiments in Appendix D.2
in Table 3
Limitations
While experiments showcase robust post-truncation accuracy of Q3R-trained Transformers vision and natural language tasks in small-to-medinn scale settings that exceeds (or in the case of fine-tuning, matches) the one of other relevant low-rank training paradigms, the viability of Q3R is yet to be established across diverse architectures and large-scale problems. Fundamentally, Q3R relies on suitable choice of the regularization strength hyper parameter A, well on suitable choice of the target rank Fiarget- We provide ablations about these values in Appendix [.. While Q3R exhibits vulnerability to elevated values of due to a convergence
trivial, very low-rank matrix, this is easily detectable by monitoring the tail ratio T(X,r) -i 1
models. In practice, have
observed stable behavior within the range À E [0.001, 0.01]. The target rank Tmrget remains insensitive underestimation because of the direct computation of epsilon resulting in large e, and due the monotonicity of the smoothing parameter update function (7) remains within a reasonable bound. We note that for weight matrices and iterations with large e, the effect of AdamQ3R resembles the one of Adam W with weight decay parameter À (see also (6).
Arguably, limitation of this work is also the fact that while the final weights after training are (for appropriate parameters) low-rank, AdamQ3R still handles dense weight matrix variables throughout training. which does not allow reduction of the parameter budget during training, unlike recent work (MHP25). More elaborate post-training postprocessing (e.g.. inspired by (WAUc+23)) might lead to further performance improvements.
7 Conclusion
We introduced Quadratic Reweighted Rank Regularization (Q3R), principled, aptimizer-compatibe framework for inducing low-rank structure deep neural network weights through explicit, contin- uous regularization. By majorizing a smoothed log-determinant surrogate with quadratic model embedded in the AdamQ3R optimizer, Q3R trains weight matrices to achieve target ranks with mini- mal accuracy loss. This enables model compression with negligible performance degradation under reasonable parameter reductions, decreasing deployment costs and increasing throughput. Our ex- perimental results demonstrate that Q3R generalizes across modalities and training regimes, with its design being particularly suitable for low-rank pre-training. Reducing Q3R's computational overhead, for example via low-rank subspace projections, remains to future work.
Acknowledgement
We would like to thank Tonmoy Hasan and Arkaprava Sinha for their assistance in setting up the LLM experiments. I.G. and C.K acknowledge the support of the NSF Grant CCF-2549926 for this work.
10

---

References
[A+14] K. D. Adam et al. A method for stochastic optimization. arXiv preprint arXiv: 1412(6), 2014.
[AS17] J. M. Alvarez and M. Salzmann. Compression-aware training of deep networks. Advances in neural information processing systems, 30, 2017.
[AZW] I. Alkhouri, X. Zhang, and R. Wang. Structure- Network Compression Via Low-Rank Induced Training Through Linear Layers Composition. Transactions on Machine Learning Research.
[Bec17] A. Beck. First-Ordes Methods in Optimization. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2017.
[BM03] S. Burer and R. D. Monteiro. A nonlinear programming algorithm for solving semidefi- nite programs via low-rank factorization. Mathematical Programming, 95(2):329-357, 2003.
[BS15] A. Beck and S. Sabach. Weiszfeld's Method: Old and New Results. J. Optim. Theory Appl., 164(1):1-400 2015.
[BV04] S. Boyd and Vandenberghe. Convex optimization. Cambridge university press, 2004.
[CC18] Y. Chen and Y. Chi. Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation: Recent Theory and Fast Algorithms via Convex and Nonconvex Optimization. IEEE Signal Process. Mag., 35-4):14-31, 2018.
[CESVI3] Candès, Y. Eldar, T. Strohmer, and V. Voroninski. Phase Retrieval via Matrix Completion. SIAM J. Imag. Sci., 611):199-225, 2013.
[CLCI9] Y. Chi, Y. M. Lu, and Y. Chen. Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview. IEEE Trans. Signal Process., 67(20):5239-5200 2019.
[CR09] E. J. Candès and B. Recht. Exact matrix completion via convex optimization. Found. Comput. Math., (6):717-772, 2009.
[CWI8] J.-F. Cai and K. Wei. Exploiting the Structure Effectively and Efficiently in Low-Rank Matrix Recovery. Processing, Analyzing and Learning of Imuges, Shapex, and Forms, 19:21 pp., 2018
[CZM*19] 19] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q V. Le. AutoAugment: Learning Augmentation Policies from Data. 2019.
[DKKaa1]] A. Dosovitskiy, L. Beyer, A. Kolesnikov, and et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
[DCLT19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
Burstein, C. Doran, and
[DCM22] F. Dörfler, J. Coulson, and I. Markovsky. Bridging direct and indirect data-driven con- trol formulations via regularizations and relaxations. IEEE Transactions on Aulomatic Control, 68(2):883 897, 2022.
[DR16] M. A. Davenport and J. Romberg. An Overview of Low-Rank Matrix Recovery From Incomplete Observations. IEEE J. Sel. Topics Signal Process., 10:608-221 2016.
[TTK+25] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. Clark, and M. Rezagholizadeh. KronA: Prrameter-Efficiee Tuning with Kronecker Adapter. In Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, pages 49. Springer, 2025.

---

[FHB03] M. Fazel, H. Hindi, and S. P. Boyd. Log-det heuristic for matrix rank minimization with applications Hankel and Euclidean distance matrices. In Proceedings of the American Control Conference, volume 3, pages 2156-2162, 2003.
[FHB04] M. Fazel, H. A. Hindi, and P. Boyd. Rank Minimization and Applications in System Theory. In Proceedings of the American Control Conference, number 4, pages 3273-3278, 2004.
[FRW11] M. Fornasier, H. Rauhut, and R. Ward. Low-rank Matrix Recovery via Iteratively Reweighted Least Squares Minimization. SIAM Optim., 21(4):6614-164) 2011.
[GK 22] A. N. Gomez, S. R. Kamalakara, et al. Exploring Low Rank Training of Deep Neural Networks. arXiv preprint arXiv:2209. 13569, 2022.
[GSGP25] T. Galanti, Z. S. Siegel, A. Gupte, and T. A. Poggio. SGD with weight decay secretly minimizes the ranks your neural networks. The Second Conference on Parsimony and Learning (CPAL 2025), 2025.
[GTK24] I. Ghosh, A. Tasissa, and C. Kümmerle. Sample-Efiiieet Geometry Reconstruc- tion from Euclidean Distances using Non-Convex Optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems, volume 37, pages 77226-77268. Curran Associates, Inc., 2024.
[HGJ 19] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Ges- mundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In International conference machine learning, pages 2790-2799, PMLR, 2019.
[HMT11] N. Halko, P.-G. Martinsson, and A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-285 2011.
[HMZ* 23] M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The Low-Rank Simplicity Bias in Deep Networks. Transactions on Machine Learning Research, 2023.
[HSW 22a] E. Hu, Y. Shen, P. Wallis, Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.
[H5 Hu, Y. Shen, Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[HUSN84] J.-B. Hiriart-Urruty, J.-J. Strodiot, and H. Nguyen. Generalized Hessian matrix and second-order optimality conditions for problems with C data. Appl. Math. Optim., 11(1):33-56, 1984.
[KBV09] Y. Koren, R. Bell, and C. Volinsky. Matrix Factorization Techniques for Recommender Systems. Computer, 42(8):30 37, 2009.
[KJP* 25] O. Khade, S. Jagdale, A. Phaltankar, G. Takalikar, and R. Joshi. Challenges in Adapting now-Resource Languages using LoRA PEFT Tuning. In
Multilingual LLMs Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 217-222, Abu Dhabi, UAE, 2025. International Committee Computational Linguisties.
[KM23] Kümmerle and Maly. Recovering simultaneously structured data via non-convex iteratively reweighted least squares. Advances in Neurul Information Processing Systems (NeurIPS 2023), 36:71799-7183: 2023.
[KV22]] C. Kümmerle and C. Mayrink Verdun. A scalable second order method for ill- conditioned matrix completion from few samples. ICML. 2021, pages 5872-5883, 2021.
12

---

[Kor09] Y. Koren. The BellKor Solution to the Netflix Grand Prize. Netflix prize documentation.
81 81(2009):1-00, 2009.
[Kri09] A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. BibSonomy, pages 32-33, 2009.
[KS18] C. Kümmerle and J. Sigl. Harmonic Mean Iteratively Reweighted Least Squares for Low-Rank Matrix Recovery. J. Mach. Learn, Res., 19(47): 2018.
[KTMF21] S. Kamalakara, N. Tenenholtz, L. Mackey, and N. Fusi. Initialization and Regu- larization of Factorized Neural Layers. In International Conference on Learning Representations (ICLR 2021), 2021.
[Lew95] A. S. Lewis. The Convex Analysis of Unitarily Invariant Matrix Functions. J. Convex Anal., 2(1-2): 1700113,,, 1995.
[LFX*24 24] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. DeepSeck-V2 V3 Technical Report. CoRR, abs/2412.19437, 2024.
[LH19] I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations (ICLR 2019), 2019.
[LHLZ24] Y. Luo, W. Huang, X. Li, and A. Zhang. Recursive importance sketching for rank con- strained least squares: Algorithms and high-order convergence. Operations Research, 72(1):237-256, 2024.
[LS05] S. Lewis and S. Sendov. Nonsmooth Analysis of Singular Values. Part I: Theory. Set- Valued Analysis, 13(3):213-241 2005.
[LSMR23] V. Lialin, N. Shivagunde, S. Muckatira, and A. Rumshisky. ReLoRA: High-Rank Train- ing Through Low-Rank Updates. In Workshop on Advancing Neural Network Training at 37th Conference on Neural Information Processing Systems (WANT@ NeurIPS 2023), 2023.
[LWY+24] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight decomposdd low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[MF10] K. Mohan and M. Fazel. Reweighted nuclear norm minimization with application to system identification. In Proceedings of the American Control Conference, pages 2953-2959, IEEE, 2010.
[MHP25] Z. Mo, L.-K. Huang, and S. Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025.
[MHR21] R. K. Mahabadi, J. Henderson, and S. Ruder. COMPACTER: efficient low-rank hypercomplex adapter layers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc.
[MNA* + 18] P. Micikevicius, S. Narang. J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed Precision Training. 2018.
[MWCC20] Ma. K. Wang. Y. Chi, and Y. Chen. Implicit Regularization in Nonconvex Statis- tical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution. Foundations of Computational Mathematics, 20:451-632, 2020.
[Na15]] B. Natarajan. Sparse Approximate Solutions to Linear Systems. SIAM J. Comput., 24(2):227-234, 1995.
[RBD25] A. Radhakrishnan, M. Belkin, and D. Drusvyatskiy. Linear Recursive Feature Ma- chines provably recover low-rank matrices. Proceedings of the National Acudemy of Sciences, 122(13))2241132512,, 2025.
13

---

[RFPIO] B. Recht, M. Fazel, and P. A. Parrilo. Guaranteed Minimum-Rame Solutions of Linear Matrix Equations via Nuclear Norm Minimization. SIAM Review, 52(3):471-501, 2010.
[RKZK22] S. Rendle, W. Krichene, L. Zhang, and Y. Koren. Revisiting the Performance iALS Item Recommendation Benchmarks. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 427-435. Association for Computing Machinery, 2022.
[RWC* 19] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language Models are Unsupervised Multitask Learners. OpenAl Blog. 2019. Technical report.
[SKZ* 22] A. P. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your VIT? Data, Augmentation, and Regularization in Vision Transformers. Transactions
Machine Learning Research, 2022.
[SL16] R. Sun and Z. Q. Luo. Guaranteed Matrix Completion via Non-Convex Factorization. IEEE Trans. Inf. Theory, 62(11):6535-6579 2016.
[SZ25] D. Stöger and Y. Zhu. Non-convex matrix sensing: Breaking the quadratic rank barrier in the sample complexity. N. Haghtalab and Moitra, editors, Proceedings of the 38th Annual Conference Learning Theory, volume 291 of Proceedings of Machine Learning Research, pages 1-2. PMLR, 2025.
[SZCT23] D. Savostianova, E. Zangrando, G. Ceruti, and F. Tudisco. Robust Low-Rank Training via Orthonormal Constraints. In Advances in Neural Information Processing Systems (NeurlPS), 2023.
[WAUc 23] H. Wang, S. Agarwal, P. U-chupala, Y. Tanaka, E. Xing, and D. Papailiopoulos. Cuttlefish: Low-Rank Model Training without All the Tuning. In Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems (MLSys 2023), volume pages 578-605. Curan, 2023.
[Wei37] E. Weiszfeld. Sur le point pour lequel la somme des distances de points donnés est minimum. Tohoku Mathematical Journal, First Series, 43:355-3380 386,, 1937.
[WMPG244] Wei, S. Moalla, Pascanu, and C. Gulcehre. Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers. In Advances in Neural Information Processing Systems (NeurIPS 2024), volume 37, pages 4689-4717 2024.
[WMPG24b] X. Wei, S. Moalla, R. Pascanu, and C. Gulcehre. Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis. In ICML 2024 Workshop on "Next Generation of Sequence Modeling Architectures" "* 2024.
[WP09] E. Weiszfeld and F. Plastria. On the point for which the sum of the distances n given points is minimum. Ann. Oper. Res., 167(1):7-41, Mar 2009.
[XIZZ+0]] Y. Xu, Y. Li, Zhang, W. Wen, B. Wang, Qi. Chen, W. Lin, and H. Xiong. TRP: Trained Rank Pruning for Efficient Deep Neural Networks. In C. Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 977-983. International Joint Conferences on Artificial Intelligence Organization, 7 2020. Main track.
[XSCM23] X. Xu, Shen, Y. Chi, and Ma. The power of preconditioning in overparameterized low-rank matrix sensing. In International Conference on Machine Learning, pages 38611-38654, PMLR, 2023.
[YATC20] H. Yang, P. Antonante, V. Tzouma, and L. Carlone. Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers Global Outlier Rejection. IEEE Robotics and Automation Letters, 5(2): 127-11144, 2020.
14

---

[YCS 20] F. Yang, X. Chen, W. Shao, S.-F. Chang, and W. Liu. Learning Low-Rank Deep Neural Networks via Singular Vector Decomposition. In JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 954-963, 2020.
[YTW=20] H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning Low-Rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[0Y F. Yan, Y. Yang, Y. Tang, et al. Learning Low-Rank Deep Neural Networks via Singular Vector Decomposition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW). 2020.
[ZCZ22] J. Zhang. H.-M. Chiu, and R. Y. Zhang. Accelerating SGD for Highly Ill-Conditioned Huge-Scate Online Matrix Completion. NeurlPS, 35:37549-377 -35562, 2022.
[ZHSI200 J. Zhang, T. He, S. Sra, and A. Jadhabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity, 2020.
[ZL15] Q. Zheng and J. Lafferty. A Convergent Gradient Descent Algorithm for Rank Min- imization and Semidefinite Programming from Random Linear Measurements. In Advances in Neural Information Processing Systems (NIPS), pages 2015. 2015.
[ZN22] P. Zilber and B. Nadler. GNMR: a provable one-line algorithm for low rank matrix recovery. STAM J. Math. Data Sci., 4(2):909-934, 2022.
[Z] J. Zhao, Z. Zhang, Chen, Z. Wang, A. Anandkumar, and Y. Tian. GaLore: Memory- Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.
Supplementary material for Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
In this supplementary material, we first provide theoretical justifications of the relationship between Q3R and the smoothed objective, expanding on Section 4.1 in Appendix A The derivation of Q3Rvalue evaluation algorithm is provided in Appendix B The expression used in AdamQ3R is derived in Appendix [ In Appendix[Di we discuss more experimental results in both pre-training and fine-tuning, and we discuss the computational aspects. In the concluding part of this supplementary material, in Appendix D we demonstrate the robustness of Q3R to hyperparameter variation.
A Relationship between Smoothed Log-Determinant and Q3R
In this section, we expand on the relationship between the e-smoothed log-determinant surrogate objective F-(-) defined in (1) Part of this material is covered in (KM23) Section B.2) in different context.
A.1 Properties of Smoothed Log-Deteriminant
We focus first on some basic properties of the c-smoothed log-determinant F. Rd xd2 - R, which, xd2 as
as we recall from (1) was defined for any W d1dxxxa
F.(W)= Σ f((o,(W)), where fe(o) 6² (log(o) log(e)) 2e,, if e, if <
102
given E > 0.
As seen by its definition, F.(-) is spectral function, i.e., it only depends on the singular values (1(W), O2( W),... of W, but not on any singular vector information.
Let now d == min(d, (d. d2). More precisely, we can define, following (Lew95) Beel 17 LS05), function functtion F IR dddx xd2 R as a function for which there exists function f Rd R for
15

---

which Fo o, where Rdixd - R W -a(W) mapping matrices in R xd2 to its singular value vector o (W) A key towards understanding the derivative structure that we can obtain an explicit formula for the gradient ((W of F. at W if the function f in the spectral function definition is absolutely (permutation) symmetric (Bec 17 Section 7.3) according to Definition A.1 It is easy to check that Le from the definition of the e-smoothed -eg-determmnant (-).) satisfies this definition.
is the function
Definition A.1 (Absolutely permutation symmetric functions).
1. Leix R We call r(x)
TRd the non-increasing rearrangement of x if it holds that
r(x)1 32 r(x)2 3- ... 3 r(x)a
and there is permutation matrix P Pd such that r(x); for all i [d].
2. We say that a function f : -R R is absolutely permutation symmetric if
f(x) f(r((x))
(8)
for any x E Rdd"
For case notation, given a vector v E R", we define dg(v)= E Rd xdxxda matrix such that for E Id and any e (1...., d1}.3 E {l..... :,....:d)). d2).
xd2 be the rectangular diagonal
Vis if = j.
dg(v)
0, else.
Next, we cite a key result about the differentiability of spectral functions which is due to
Proposition A.1 (Differentiability of Spectral Functions (LSO5) Section 7)), Let F Rd R be spectral function F = fo a with an associated function f IR - R that is absolutely permutation symmetric. Then, F is differentiable at W E R xd2 if and only if J is differentiable (W) R
In this case, the gradient V F of F at W is given by
VF(W)= Ug (V)
if W Ug (T) is singular value decomposition of W with orthogonal matrices U e IR xdi and V e d
Using Proposition A.T we can characterize the derivative of the F. for arbitrary c 0, as established in the following lemma.
xdz Then F. is differentiable with 1-Lipschitz gradient V RR.xxx Xd2
Lemma A.2. Let > 0 and F. xd - R be the e-smoothed log-determinant of Equation (1)
Rdh x that is given by
(W)
VF.(W) Uw dg max
V
(9)
(W)/e, 1)2
i=1
for any matrix W with singular value decomposition Uw dg VW.
W
Uw dg (o(W))
Proof of Lemma A.2 For the differentiability of Fz, as per Proposition A. is sufficient to show that
the function [((r).... oa)) Σi= Se(a,) with f. R2z0 R as defined in (1) is differentiable at
any (a (d)....,oa) IRd Due the sum structure of f. this will follow if fa is itself differentiable
any o > 0.
To this, observe that for any 0, # e, we have that fe is differentiable at with derivative
E ifa
f(()) = a. ifoeoce.
Since [[(a) lima (o), it follows that Se is also differentiable at with f((e) 6, and thus, differentiable on the entirety of its domain. The formula (9) follows then directly from Proposition A.I
D
16

---

Remark A.3. It is well-known that the optimization of convex functions (BV04) is from many perspectives less challenging than the optimization of non-convex functions. In Section 4.1 we have claimed that the e-smoothed log-determinant surrogate is not convex. This can indeed by shown directly by invoking (LS05) Proposition 6.1), which states that a spectral function
f for is convex if and only if f is convex. Indeed, it is easy to see that fe(-) is not convex due to its logarithmic dependence on the input for large inputs, which shows that Fe(.) is not a convex function.
As mentioned in Section 4.1 we see from t9htha that computing V (W)) given the matrix W E IR4x xd2 indeed would require a full singular value decomposition that includes at least d leading singular val-
ues. Defining r( (W.c)= ==
e (1..... d} of (W) > e]] as in (3) and Σ = diag (W)); (=1
R (r((We))r(( (e) as in () we obtain for U and V defined from the r r(W,e) leading columns of
Uw [U U.] R 1 and Vw IV Vile R that
xd1
VF((W) = L -V + E
(10)
inserting the formula from (9) with the notation that Σ; dg (a, (W) (W)) Fundamentally, this is the key reason why direct inclusion of the smoothed log-determinant objective into a gradient-based optimization algorithm is computationally inefficient.
Finally, we conclude with the observation that F-(.) becomes convex i E >> 0 is chosen large enough. In particular, it holds for any W E Rd xd2 that
d (W) Σ (W() = Σ
d 1 o7
(W) (W)
= W 3
2
if additionally the largest singular value ai a (W) of W satisfies ai (W) < e. Here, we used in the last equality that the Frobenius norm of matrix is the 22-norm if its singular values.
A.2 The Quadratic Model Function Underlying Q3R
We proceed by justifying the claims made in Section 4.1 about the relationship between the e-
smoothed log-determinant (-).), the Q3R-regularizer Q3R (-), the quadratic model Qe W') of
(2) and the reweighting operator (-). .. To this end, we show the first statements of Lemma A.2 that characterize the reweighting operator (-).
Proof of Lemma 4.1 1.
Let W didida be arbitrary with singular value decomposition W Recall from Definition 4.1 that
U' dg(a:(W"))
Zw',(W) = U' c,d, U WV' c,d, V
where Σe, = (iag(max((a)))) (W' //e, 1)); e Rdxd for e (d1, d2).
To show that Rw _dd xd2 - d xda is positive definite operator, we consider any W E IR xd2 such that W * O, which implies that |/ O. Defining Z == U WV', we see that
(W, RRw,e(W) = tr (W Rww.((W) = tr (w U'S c,d, T WV'E 6, V
U WV
= tr z T Ded Z2 e,d2)
= tr
cdd,
c,d
Ir (E 2) 2) = (,
2) (2 2
cddi
d
Σ ã.ō, Zi
with ã. max (ax')):W')//e. 1) for ie (1, .l..... max(di, d2)), with using the cyclicity of the trace in the third equality. Since ō. > 0 for all i, we can establish the lower bound
di da
max(d; ,d))
di
d2 max(di db)
Rw W)
Σ
min ==1
Σ
zt
min ==1
2
1-1
max(di
max(d
min
7: wV'1
min
0 W 2 > 0.
==1
17

---

Due to the definition of Q3R (6) an implication of this is that
Q (W)
- <W. W. (ww.((W)
and Q3Rw . (W) - 0 e> W 0, i.e. the value of Q3R is always non-negative and positive for non-zero matrices.
We proceed with the proof of the second statement Lemma 4. which provides an explicit formula for the reweighting operator that only requires partial SVD of W".
Proof of Lemma 4.7 If W = Uw dg(or ( V w is full singular value decomposition of W' with Uw
[U
U.] E xd and Vw = [V Vi] 22 recall that the
image matrix W Rd x with respect Rw is defined (see Definition 4.1 as
(ww)((W) = L U. W c,ds Vw:
(11)
using the definition for We,d from the proof of Lemma above.
With a similar argument as made in [10] we can see that
Uw Ded c,d,
Uw cTT-'UT + ULU = -U +I-UU
with Σ diag(o, (W)) W .0) (rr(Wa))rr(W. .c) U (d and the identity matrix In
the last equation, used that U_ U is the projection operator onto the subspace that orthogonal to the one spanned by the columns of U. Analogously, we obtain that
Vw VW =VE ' VLVI =VE + +I-VV
where Rddx e) Inserting these two equations into (11) we obtain
Rw (W) = Uw cdd w WVw --1 Vw
c,dg
-EUNN' I- UU") w (eVE +I --V''
+ +CUS U VVT)
=
+ +c - UU" WVE -U--Ul] w w (I-VV))
where the last equality shows the statement of Lemma 4.1
As a preparation for the proof of the last statement Lemma we formulate the following lemma which relates the gradient of F. at W with the reweighting operator.
Lemma A.4 (Gradient Condition). Let e > For any W E xdg the reweighting operator
Rw,, Rd, xd
RRd xd2 satisfies
Rw., (W) VF.(W),
(12)
where (W() is the gradient of the e-smoothed log-determinant at W.
Proof. If W = Uw dg(a, W)) V a singular value decomposition of W with Uw =
[U U1] E xdid and Vw [V
Vi]e E ddd we observe that
(w.((W) Uw E-1
U (Uw dg(a: E- c.dy
V
c.d,
Uw dg(o:(W)) (1)) VW
a (W)
Uw dg
V VF((W),
max 0r W) l/c, 1)2
using the gradient formula Lemma A.2 in the last equality.
18

---

As corollary of Lemma A.4 we see that for W W, the gradient of the Q3R regularizer satisfies
V VQ3Rw (W)
VF.(W).
This is this a direct implication Lemma A.4 since
Vw Q3R (W)
Vw
W, Rw (..w.((W)) (W))
using the self-adjointness (see, e.g., KM23 Appendix B)) of Rw.
The gradient condition (12) enables us to equate the definition of the quadratic model function (2)
Qe(- W') (which is, up constant that depends on and W' the same the value of Q3R) with the standard quadratic model form of (5)
Proof of Lemma 4.1 3. Let W, W E Rdd xd2 be arbitrary. To show the equation (5)
we start with
its right hand side. By inserting [2) we obtain
F.(W)) + (VF(WW), W w') W W', Rw (W- -W')
F((W') + (Rwi. (W'), W W-W>>
KW Rw. (W)) (Rw',. W)
2
(WW, (w")(W"))
<W', =F(WW') + W. Rw (W))) 3/2 W', Rw
=: Q. W")
where we also use the self-adjointness of Rw ..)) in the first equality. The last expression corre- sponds the definition of the quadratic model Qe(- W') of F-(-) given the expansion point W This concludes the proof.
From this proof, becomes clear that the quadratic model Qe W') is pure quadratic model with vanishing linear term. This implies that, for example, Qe(-W W W") = (W W") for all W, which reflects the geometry of the smoothed log determinatt F.(.) better than mixed quadratic model function as it likewise satisfies F. F-(-W) F((W).
B Computation of Q3R value
In this section, we provide an implementable algorithm for evaluating the Q3R regularizer Q3Rw (W)) as defined in (6) defined in Algorithm below.
We note that strictly speaking, evaluating Q3Rw (W) ) is never necessary a training scheme such as AdamQ3R; however, evaluating Q3Rw (W) might be insightful to keep track of the extent of the regularization.
First, we decompose the reweighting operator image such that
+ T Ti - WVE T = -U CU W VVT),
Rw W) = 7 +
T5 +
T5
with
T = = (I-UU WVE Vy.
T (I- - UU W [ - VVT).
Defining
I = (,, TT) = P U UWVE
4 = (,, T5) = Cu(W"Ur-'U- U vVJ),,
(W, TΦ) =cu((W UUT) W WVE
4 = (W, Tς) = tuw (-- W (I (I-v))))
19

---

we can write
(W, (W.) = + 13 4.
Apply eyclicity to /, 12; Is.
4 = é tr WU "'V-')
w) tr(W -.--U'VVV')
1
13 = [I(WTWVE-VV) UU WVV I((TTU--WV--VV))
Expand Ta.
4 ( w) twWWW tu w) (WW'Uu'
Group terms.
W, (Ww,((W) tr(W W)
+ vi W
+ =et(("TU--UU'W - UU UU UU'WUU-VVVV
+etr(W -V
tr(W WVV) uu(WUU w) + ((W'
Then by rearranging each pair of trace-terms we arrive
(W, Rw (W)) tr(W W)
+ t W W U (CE---II)
IV WV (cE-1-1) - -
In
For each iteration we calculate the quadratic regularizer (W, Rw (W) for weight matrices For algorithmic simplicity, We now re-arrange the summand of our Q3R regularizer and show the simplified expression for this inner product Algorithm 3 later show in Eqquation (13) where we derive the expression of this inner product and we show that this matches the one proposed in Algorithm
(r W) W u + ((U'WWTs
4 T = WV, B=WU, C-U'V,, M = /// CS122
t
S c -E-1_
Using tr (r(ATA) = | and cyclicity:
4 tr(W W) W
4 (U (V) |M
f ((S(W))[(WV)) T 1/2 F:
f (<(S UJ (W) = BS 1/2 1/2
Putting it all together, with R 1/2
f(W.U.V) W
+ M - + TR + BRR
R (EE - 1/2
(13)
20

---

Algorithm 3 Computation of the Q3R function value Q3Rw ,e (W)
1: Input: W E Rdixda xd2, U Rdirr, V Rd S Rx
2: Output:e J(W,U, U, V [
M + 3 + 2
+
3: 1. Compute projections
4: T < WV
O(did2r)
5:
WT
O(d1d2r)
6: 2. Form intermediate products
7: C < U
O(d,r
8: Compute symmetric square-root of S: RR S
O
9: M < RCR R.
O(ri)
10: 3. Evaluate Frobenius norms
11: h W
O(did))
12: 22 - M
0(r²)
13 [ R
O d(r) O(d2r² + d2r)
14: 4 - B Ri
return
12
1/
+
4
To summarize, we obtain a total FLOP count of
Tiotal = 2d1d2r + (d1 + d2)r² + 2 + (d1d2 + (d1 + d2)r + 1 = (d1d22 + (d1 + d2)r + r2)
for evaluating Q3 (W).
C
Computation of the Gradient of Q3R
From Equation (4) we rearrange the summand for algorithmic simplicity
Rw (Ww,(WW 2 --- WV CUE-'U W(I-VV))
e
(I- (I-UU (
Using --1 I+ S, we rewrite each term:
Ti UE- WVE 'VT U(++S)U WV
TH = -U- W VV = U(+ W[--VV)
TV (I- W WV = (I-UU" WV
Ti = (I- W - VVT).
Collecting like-terms in powers of S gives the compact form
Rw W + USU W
WVSV + USU WVSV
T
with S -
Now, we deduce the gradient in the following algorithm that is used in line of Algorithm We explain the step-by-step computation of our Rw (W) stated below.
21

---

Algorithm 4 COMPUTATION GRADIENT OF Q3R: Compute Rw (W)
1: Input: WeR UERdixr, VeR singular values o e R threshold
2: Output: = (W)
3:
4: <- 1/max(o, 1)
t> r element-wise comparisons create diagonal matrix of shape x + subtractions on diagonal
5: S diag(s) s)
6: Ssift + cS
7:
8: - U
- r x 1 x d2 - d2 (cost: rd1d2) d1 (cost: di d2r)
9: B WV
d1 d2 by d2
10:
11: C C A.V
x d2 by d2 x
(cost: rd2r) t> r x r triple product (cost: 2r
12: E - Sain C Sahit
for i= ,...., do
14: for 1...., r do
Eg ( Ciy (S
16: end for end for
elementwise scalar products (cost: r2)
17: T- - U. VT
U-
E: d1 x byr d1 r (cost: d3r²) Then V d1 by d2 + d1 d2 (cost: d1d2r)
18:
19: Ti W
t- copy identity operation (di d2)
20: T3 - B Sain VT
B Sain: dj x r by x d1 x (cost: d2r²)
x
Then VT. T dj by d2 + di d2 (cost: d1d2r)
21: Sain
> T by d2 -r d2 (cost: r² d x by d2 - di d2 (cost: d1rd2)
22: T4 - U-
24: gradient - 7i T2 T3 +
elementwise addition of d1 d2 matrices
25: return gradient
D Experimental Results
In this section, we provide details regarding the experimental training methodology of Section well as some additional data of these experiments.
D.1 Experimental Protocol
the experiments, we compared the training of unregularized models, as well as models regularized by Q3R, LoRITa (AZW) and LoRA HHSW+22al Due to the limitations of some techniques not providing strong truncation guide lines, we truncate each method at various truncation ranks r. Ensuring such that is chosen to always ensure that the factor matrix pairs always have less original parameters, to minimize a practical environment. Table we find that running with = .20 or = 19 and X 0.001 performs best given the truncation and accuracy trade-offs.
VIT Hyperparameter Selection We selected each model's learning rate based on the perfor- mance of the unmodified ViT model for each respective dataset. All ViT models were configured with an input resolution of 224x224 pixels and utilized patch size 16 for tokenization. LoRITa hyperparameter optimization was conducted through grid search with regularization parameter ε {10-1, 10-2, 10-3 10-4) and rank parameter d e {1,2, 2,3. 3] to ensure optimal performance. The best-performing configuration from three independent runs was selected for evaluation across additional datasets. LoRA was evaluated across various target ranks, with the proportion param- eter p selected from E (0.05, 0.15, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). AdamQ3R underwent grid search optimization with regularization parameter E {10-5 10-2 10-1) and proportion parameter The regularization strength remained con- stant regardless of matrix dimensions. The QKV projection matrices selected the target
r (0.05, 0, 0.15, 0.2), with a stable period

---

for modification due to their prevalence in the literature candidates for model compression and adaptation.
CIFAR-100 Data Augmentation For CIFAR-100 experiments, we applied comprehensive data augmentation during training, including random cropping with 4-pixel padding from the original 32x32 images, random horizontal flipping. and subsequent resizing 224x224 pixels match the ViT input requirements. Images were normalized using channel-wise means of (0.4914, 0.4822. 0.4465) and standard deviations of (0.2023, 0. 0.1994, 0.2010), corresponding to the CIFAR-101 dataset statistics. The test set underwent only resizing to 224x224 pixels and the same normalization procedure, without augmentation.
Fine- Tane-T Experimental Details We fine-tuned the pre-trained RoBERTa- model (from Hugging Face) on all nine GLUE tasks using a maximum sequence length of 512 and batch size of 16 (30 for CoLAA). We run all fine-tuning experiments with target rank as low as The best performing setup had rreeighttin period" of 3 and À E (1.5, 2). As shown in Table 3 we compare Q3R with LoRA and dense fine-tuning. The tasks and their metries are summarized below:
Single-Sentence Classification
CoLA:: 8 train k test; linguistic acceptability; Matthews correlation
-SST-2: 67 k train 1.8 k test; sentiment classification; accuracy
Similarity & Paraphrase
MRPC: 3,7 k train k.7 test; paraphrase detection; accuracy Fl
- -STS-B: 7 train 1.4k test; sentence similarity; Pearson Spearman correlation
-QQP: 364 train / 391 k test; question paraphrase detection; accuracy FI
Natural Language Inference
MNLI: k train / k matched 20 k mismatched test; entailment classification; accuracy
QNLI: 105 k train / 5.4 k test; question-answer entailment; accuracy
RTE: 2.5 train / 3 test; textual entailment; accuracy
WWNL: 634 train / 146 test; coreference-based inference; accuracy
D.2 Fine-tuning Experiments
To assess the potential of Q3R for fine-tuning large language models, we conducted an ad- ditional experiment on Llama 3.2-3B using an NVIDIA A5000 GPU. Since the original neta-llama/llmma-3.2 2-3B checkpoint provides only the pre-trained language model parameters, we instantiated the linear projection, commonly referred to the classification head, top of the final hidden-state representation. This head, whose weight matrix is registered as score. weight (and optionally score.bias. is absent from the checkpoint and was therefore initialized with random Gaussian values.
module, which attaches an additional
The fine-tuning setup involves training this weight matrix as well as additive weights for the Q, K, and V layers.
With Q3R (A = 0.0001, target rank = 4, for 100 epochs), we achieved an Fl score of 81.89% on the MRPC dataset of the GLUE benchmark, whereas full fine-tuning (effectively corresponding to setting À = 0) resulted in an Fl score of 80.7% after the same number of epochs.
To further compare our proposed method, we conducted experiemnts Llama3.2 2 IB subset of the GLUE tasks. Table 4 provides a comparison our method's performance with dense fine-tuning and LoRA.
We achieved the performance as mentioned in Table with single value of without any extensive hyperparameter search. While we acknowledge that better performance for GLUE tasks like RTE can often obtained by starting from more finely tuned initialization, these results demonstrate that the proposed method is effective even for the fine-tuning of large-scale LLMs.
23

---

Table 4: GLUE Benchmark Results for Llama3.24 2-BB
Model
MRPC
SST-2
RTE
CoLA
STS-B
Dense
86.3
95.3
77.3
47.09
90.5
LoRA rank=4
87.3
95.7
80.9
61.8
89.1
Q3R rank=4
87.8
94.4
64.7
51.8
87.04
D.3 Appendix Tables
Table 5 and Table 6 demonstrate the low rank induction techniques on ViT models. We train ViT-B on CIFAR-100 in Table 6 and ViT-T on CIFAR-IO Table 5 respectively.
Table 5: Performance varying percentages of parameters saved on ViT-T when regularizer applied only
attention blocks.
Model
15%
%
30%
40%
%
60%
70%
80%
90%
100%
Vanilla VIT-T
0.4687
0.5430
0,5892
0.6170
0,6542
0.6694
0.6726
0.6788
0.6892
0.7027
Q3R rank=4, A=0.01
0.6032
0.6630
0.7043
0.7060
0,7084
0.7080
0.7079
0.7079
0.7077
0.7152
Q3R rank=9, 0=0.01
0.4852
0.5616
0.6668
0.6772
0.6831
0.6835
0.6832
0.6829
0.6831
0.7034
Q3R rank=14, A=0.01
0.4576
0.5812
0.6866
0.6970
0.6988
0.7002
0.6997
0.6996
0.7003
0.7104
Q3R rank=19, X=0.01
0.6208
0.6792
0.6913
0.6977
0.6978
0.6970
0.6970
0.6967
0.6966
0.7061
Q3R rank=4, X=0.001
0.5694
0.6629
0.7017
0.7053
0.7084
0.7091
0.7086
0.7086
0.7079
0.7158
Q3R ramk=9, >=0.001
0,4214
0,5304
0.6467
0.6680
0.6691
0.6692
0.6691
0.6696
0.6695
0.6955
Q3R 14n 0.6040
0.6364
0.6642
0.6695
0.6695
0.6694
0.6698
0.6701
0.6704
0.6819
LoRITa 0=0.1 0.1154
0.1217
0.1234
0.1091
0.1502
0.2310
0,3377
0.4197
0,5027
0.7061
LoRITa -==.0==0000
0.1576
0.1512
0.1602
0585
0.1902
0.2456
0.2955
0.3221
0.4056
0.7086
LoRITa D=1, 0=0.01
0.1474
0.1514
0.1539
0.1644
0,2486
0.2968
0,4150
0.4857
0.5326
0.6911
LoRITa D=2, 0=0.01
0.2478
0.2559
0.3115
0.4065
0.5127
0.5641
0.5997
0.6456
0.6720
0.7287
LoRITa D=2, 0=0.1
0.2041
0.2589
0.3639
0.5397
0.5982
0.6408
0.6727
0.6910
0.7025
0.7595
LoRITa D=2, 0=0.001
0.2195
0,2774
0.4163
0.5115
0.5461
0.5675
0.6137
0.6417
0.6614
0.7393
LoRITa D=3, a=0.01
0.3794
0.5074
0,5907
0.6385
0.6598
0.6761
0.6843
0.6916
0.6959
0,7462
LoRITa D=3, a=0.001
0.3951
0,5147
0,5952
0.6308
0.6639
0.6676
0.6798
0.6847
0.6833
0.7367
LoRA rank=4
0,3443
0,3443
0.3443
0.3443
0,3443
0.3443
0.3443
0.3443
0.3443
0.3443
LoRA rank=14
0.5859
0.5859
0.5859
0.5859
0.5859
0.5859
0,5859
0.5859
0.5859
0,5859
Table 6: CIFAR- 100 Performance with ViT-B Attention Block Truncation
Model
5%
10%
15%
20%
30 %
40%
100%
Vanilla
ViT-B Best
0.2773
0.3355
0.3659
0.3909
0.4172
0.4327
0.4686
Q3R, rank=19, A=0.01
0.3238
0.3979
0,4172
0.4301
0.4411
0,4485
0.4625
Q3R, rank=19, A=0.001
0.3174
0.3790
0.3945
0.4050
0.4130
0.4197
0.4408
Q3R, rank=14, A=0.000 A=0.001
0.3250
0.3978
0,4149
0.4240
0.4351
0.4429
0.4613
Q3R, rank=14, A=0.01
0.3465
0.3950
0.4062
0.4172
0.4305
0.4389
0.4526
LoRITa D=2, a=0.1
0.0152
0.0444
0.0883
0.1404
0.2428
0.3043
0.4021
LoRITa D=3, a=0.001
0.0607
0.1375
0.2001
0.2380
0.2975
0.3269
0.4103
LoRITa D=3, 0=0.1
0.0570
0.1369
0.2003
0.2401
0.2946
0.3188
0.4111
D.4
Computational Aspects
For few experiments like Q3R, used NVIDIA A5000 to train the ViT models. The rest of the experiments were performed on NVIDIA V100 with 32GB memory. The fine-tuning experiemtns were all performed in NVIDIA A5000 GPUs.
D.5 Computational Overhead of Methodology
In Algorithms and 4, we provide the detailed stepwise FLOP count of our method. Following the experiments testing the influence of the reweighting period on the truncation, we report the average training time of the first 5 epochs below. Note that regularization was only applied on the QKV matrices of ViT-Tiny Transformer.
24

---

Table 7: Average training time epochs) and reserved GPU memory for ViT-Tiny with QKV-only regularization.
Model
Variant Setting
Avg. Time (s)
Reserved GPU Mem (GB)
Base
150.91
6.26
AdamQ3R
7-500
205.34
6.28
T==10
207.39
6.31
T-50
209.29
6.62
T-20
217.90
8.22
T=10
218.88
8.62
LoRA
R-4
124.83
11.89
R-14
144.51
11.87
R-28
145.28
11.89
Depth baseline
157.29
8.35
Based on the theoretical computational overhead outlined in Algorithm 3 Algorithm along with the results above, we expect Q3R to incur additional computational and memory overhead. The added memory originates from the additional weight matrices stored in Q3R to perform the weight least squares minimization. We attribute the lower Reserved GPU Memory from implementation differences in the optimizer.
E
Ablation Studies
Table & presents the comparison between the regularization term (6) and Algorithm evaluated across varied truncation levels and hyperparameters. Initially AdamQ3R presents competitive performance against Q3R; however, Q3R provides superior truncation performance the validation set at lower truncation values (20% and below). The impact of the regularization parameter is notably small, with 0.001 generally providing superior performance.
Table Performance Vit-T with AdamQ3R Q3R across different truncation level on CIFAR10.
Model Name
5%
10%
15%
20%
30%
40 40%
100% %
AdamQ3R, rank 0.05*
0.1904
0.4600
0.6032
0.6630
0.7043
0.7060
0.7152
AdamQ3R, rank 0.10*
0.2081
0.3591
0.4852
0,5616
0.6668
0.6772
0.7034
AdamQ3R, rank 0.15*
0.1266
0.3970
0.4576
0.5812
0.6866
0.6970
0.7104
AdamQ3R, rank 0.20*
0.1740
0.4717
0.6208
0.6792
0.6913
0.6977
0.7061
AdamQ3R, rank 0.05
0.2701
0.4608
0.5694
0.6629
0.7017
0.7053
0.7158
AdamQ3R, rank 0.10*
0.1066
0.1741
0.4214
0.5304
0.6467
0.6680
0.6955
AdamQ3R, rank 0.15
0,2467
0.4993
0.6040
0.6364
0.6642
0.6695
0.6819
Q3R, rank = 0.04*
0.2476
0.5171
0.6544
0.6862
0.6884
0.6874
0.6920
Q3R, rank = 0.09*
0.1959
0.5040
0.6789
0.6789
0.6862
0.6870
0.6827
Q3R, rank 09
0.2774
0.4317
0.6337
0.6697
0.6818
0.6896
0.6901
Q3R, rank = 0.04
0.2828
0.4699
0.6004
0.6569
0.6801
0.6801
0.6934
Q3R, rank 0.09*
0.3202
0.5527
0.6827
0.7024
0.7024
0.7079
0.7081
Q3R, rank -
0.3119
0.4440
0.6137
0.6670
0.6839
0.6877
0.6885
Legend: RRegularization parameter À 0.01.
Regularization parameter À 0.001
E.1 Robustness to Hyperparameter Variations
Empirically, we found Q3R to be quite robust to its hyperparameters within reasonable ranges. Below, we are providing few empirical evidences from Table Tpp ViT-Tiny. However, similar results can be conducted across other datasets and backbones as in Table 4 (CIFAR- 100 on ViT-Base). Generally, when choosing
viable rule is easy to tell the choice lambda too small by monitoring if the Q3R value increases within the first few epochs. We recommend value of X that is slightly larger than the lower bound the divergence threshold, as determining the À value too large remains challenge.
25

---

Table 9: Effect of regularization strength À accuracy.
Parameter Retention
Accuracy (%)
46.08-46.30
5%
63.40-63.66
10%
66.30-66.40
25%
70.68-70.74
50%
71.04-71.09
100% (no trunc.)
71.95-71.52
Beyond the 20% retention point, the absolute accuracy gap never exceeds 0.3 percentage points, confirming that AdamQ3R largely insensitive to within the 0.001-0.01 range in this practical operating regime.
Table 10: Effect of target rank Turget accuracy.
Retention (%)
Accuracy (%)
1%
0.5683
5%
0.6568
10%
0.7068
25%
0.6671
0.6457
100%
0.6663
Once 30% of the parameters are retained, the choice of rank changes accuracy by < 0.3%, confirming low sensitivity to rank in this regime. We here scale the target rank by layer dimensions so that single syper-parameter works for networks of any size.
E.2 Merits of Ban-kank Initialization
In our experiments, we implemented supervised initialization where the regulated weight matrix has rank greater than equal to the target rank hyperparameter specified in Q3R. Our findings support the hypothesis that low-rank initialization in Q3R imposes strong constraint on the optimization landscape when the rank is low, thereby limiting the model's capacity to explore more expensive solutions during training. This hypothesis is further supported by the fine-tuning results on RoBERTa where Q3R is applied to the pretrained model and continuously tuned the GLUE benchmark tasks. We achieve notable results that support the general intuition that models tend to learn better representation through expensive subspaces during the training process. Due to the time constraint of the evaluation period, we could experiment on only the smaller data we observe that LowRank initialization is unable surpass the performance of Q3R without such constraint the initialization. However, the accuracy is within 1 range the full rank initialized model which proves that our proposed method can be implemented in resource constrained setups well. We provide the example of the empirical evidence performed on CIFAR-I0 on ViT-Tiny with 0.05 and a = 0.01.
E.3 Choice of Reweighting Period
We observe in Table 12 that higher reweighting periods (1) (300, 200, 100) result in underperformance
comparison to the lower reweighting periods(T) (25, 5). While longer reweighting periods provide some computational performance gains based upon formulation Algorithm| [ we observe superior performance for faster intervals which corresponds to the IRLS--Aaoooaa maj.oiiaaoiiiiioon of the logdet.
Code The code is available athtt:p://ggihhhb.cmm/aaaaat//
26

---

Table 11: Parameter retention effect accuracy (AdamQ3R LowRank).
Parameter Retention
Accuracy (%)
1%
46.08
5%
63.40
10%
66.30
25%
70.68
50%
71.04
100% (no trunc.)
72.19
Table 12: Model performance under different truncation percentages. Best value per column is bolded. Fach model trained with À 0.001, 0.2, Trained for 30 epochs
Model Name
5%
10%
15 %
20%
30%
40 %
100 %
AdamQ3R,7 7 300
0.2999
0.5609
0.6651
0.6801
0.6847
0.6838
0.6827
AdamQ3R, 200
0.2601
0.5914
0.6519
0.6766
0.6869
0.6885
0.6871
AdamQ3R, T= 100
0.2764
0.4871
0.6623
0.6776
0.6869
0.6885
0.6936
A3amQ3R, T' 25
0.3729
0.5813
0.6555
0.6725
0.6734
0.6778
0.6790
AdamQ3R, T'=
0.1740
0.6838
0.6828
0.6949
0.6995
0.7000
0.7031
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error hars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used").
general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
. Do not modify the questions and only use the provided macros for your answers.
1. Claims
27

---

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract includes the main contribution of the paper and to highlight it properly, there is separate subsection in the Introduction that discusses on the main contribution of the paper
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes Yes]
Justification:
We discuss the limitations of the paper in a separate section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
28

---

The Methodology section contains detailed description of the proposed training strategy and the related algorithms. The supplementary material contains additional formulae and explanations.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The Experiment section includes all the hyperparameter details and model variants. Further discussion on the choice of hyperparameter is included in the supplementary
material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
29

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: In the supplementary material we provide an anonymized GitHub repository of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcceeeoeeooocp gwwcc//guuese//eee hutee/ggguudeee/eeeuuuubiiiiiiii www.anssssonppoo ssiniioiscoyycy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The Experiments section includes all the hyperparameter choice and the Methodology section includes the detailed description of the algorithm.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The Experiments section discusses the accuracy and performance of the models in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details about the experimental resources are mentioned in the Supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
cc http://ppppppppppppcccccccccccaa
Answer: [Yes]
Justification:
The NeurlPS Code of Ethies http::/neerrips.cc/pbbb.cc
Ethi cahhcideieeeees is followed while writing the paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper fall under the category of optimizing neural networks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The data, models used in the paper are all publicly accessible.
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All the existing work mentioned in the paper are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppppappphhaoocc thcode. com com/datasets
32

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
33

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification: LLMS are used for any important component of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtaddard components.
Please refer to our LLM policy (https: for what should or should not be described.
34


Reviews:
[{"review_id": "4V7A5yySiE", "reviewer": "Reviewer_aasu", "summary": "Authors introduce Q3R, a new method for training deep neural networks that enables effective model compression by promoting low-rank weight structures during training. Unlike methods like LoRA that modify only parts of a pretrained model, Q3R starts with standard full-sized (dense) weight matrices and gradually encourages them to become low-rank through a specially designed regularization term. This regularizer gently pushes the smaller singular values of weight matrices to shrink over time, encouraging a low-rank structure. This allows the model to be effectively compressed after training without significant accuracy loss. For improved convergence of the process, authors also present a custom optimizer called AdamQ3R that helps apply this regularization efficiently. Experiments on ViT and language (RoBERTa) models show that Q3R achieves large reductions in model size, up to 80% fewer parameters, with minimal drops in performance. With this, the presented approach is able to outperform other low-rank methods especially at high compression levels.", "questions": "It would be great if the authors can address the following questions:\n\n1. Have you considered or experimented with initializing from low-rank approximations or progressively reducing rank during training to improve training-time efficiency, especially for resource-constrained settings? And if you compare this setting with the dense full-scale setting as starting point, do you see performance difference?\n\n2. Could the authors quantify the additional training-time cost introduced by periodic SVD and reweighting operations, and are there strategies (e.g., using approximate or less frequent updates) that might reduce this burden while retaining performance?\n\n3. How robust is Q3R to variations in regularization strength and target rank? Would the authors consider providing heuristics or adaptive mechanisms to guide these choices across tasks and model scales?\n\n4. Since Q3R promotes but does not guarantee a specific rank, could the authors clarify if post-training truncation is required in practice? Would adding an explicit rank constraint during optimization improve rank control without harming performance?\n\n5. Have the authors considered evaluating Q3R on more challenging benchmarks such as ImageNet or large-scale language tasks beyond GLUE to better demonstrate the method’s scalability and generalization potential?", "limitations": "Yes, however, there is no discussion towards social impact, especially the bias that could be introduced by this compression.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and and easy to understand, It has several strengths:\n1. Proposed approach can be applied during both pre-training and fine-tuning. This makes it more flexible than methods like LoRA that are typically limited to fine-tuning.\n2. The method is based on a solid theoretical foundation using smoothed log-determinant surrogates and quadratic upper bounds.\n3. The approach of shifting a matrix towards low-rank is interesting and designed very intelligently. It avoids overparameterization and post-hoc truncation by encouraging low-rank weights directly during training.\n4. A custom optimizer, AdamQ3R, is designed that helps separate the effects of regularization from standard gradients, improving stability and generalization.\n5. Experimental results across both vision (ViT) and language (RoBERTa) tasks show strong performance, with up to 80% parameter reduction and minimal accuracy loss, outperforming prior methods at high compression levels.\n6. Q3R is compatible with standard architectures and can be integrated without structural changes.\n\nWhile the work is very exciting and relevant for the scientific community, there are several weaknesses as well.\n1. Since the method starts from full-rank systems, the benefit can clearly be considered to be for inference-time and not training time. This also implies that unlike LoRA type approaches, this approach is not suited for budget-constrained systems for training.       \n2. Periodic SVD and reweighting computations introduce extra training-time overhead, especially for larger models.\n3. The approach requires tuning hyperparameters like regularization strength and target rank, which may not transfer well across datasets or tasks.\n4. The experimental evaluation is limited to relatively small and easy benchmarks (CIFAR-10/100 and GLUE), which may not fully reflect the performance and scalability of Q3R on more complex, real-world tasks or larger models.\n5. If I understand correctly, the method promotes low-rank structure but does not strictly enforce a final rank, meaning post-training truncation may still be needed for exact control.", "was_revised": true, "final_justification": "Authors have adequately responded to my questions, and being proactive with the literature in the field, I fully understand the significance of this approach. I also took into account the reviews from other reviewers before adjusting the score, and I couldn't find any strong drawbacks. Thus, I have moved my suggestion from borderline accept to accept.", "rebuttal": "Thank you for your thoughts about our submission.\nWe are delighted that you found the paper clear and easy to follow, and we greatly appreciate your recognition of  the solid theoretical grounding via smoothed log-determinant surrogates and quadratic upper bounds and the method’s plug-and-play compatibility with standard architectures. We agree that the principled low-rank-shifting strategy that circumvents over-parameterisation and post-hoc truncation \n\nPlease consider our responses regarding your assessment:\n### **Q1: Merits of Low-Rank Initialization**\n\nIn our experiments, we implemented spectral initialization where the regularized weight matrix has a rank greater than or equal to the target rank hyperparameter specified in Q3R. Our leading hypothesis for the poor validation accuracy following initialization is that Q3R imposes a strong constraint on the optimization landscape when the rank is low, thereby limiting the model’s capacity to explore more expressive solutions during training. This hypothesis is further supported by the finetuning results on RoBERTa where Q3R is applied to the pretrained model and continuously tuned on the GLUE benchmark tasks. We achieve notable results that support the general intuition that models to tend to learn better representation through expressive architectures and training regimes.\nDue to the time constraint of the rebuttal period, we could experiment on only the smaller data we observe that Lowrank initialization is unable to surpass the performance of Q3R without such constraint on the initialization. However, the accuracy lies within $1\\%-2\\%$ range of the fullrank initialized model which proves that our proposed method can be implemented in resource constrained setup as well. \nWe provide one example of the empirical evidence performed on CIFAR-10 on ViT-Tiny with r = 0.05 and $\\lambda = 0.001$.\n\n| Parameter retention | AdamQ3R + LowRank| AdamQ3R (Q3R, no LowRank) |\n|-------------------------|------------------------------|---------------------------|\n| 15 % | **65.62** | 64.30 |\n| 20 % | 68.30 | **69.60** |\n| 30 % | 68.90 | **70.60** |\n| 40 % | 69.26 | **70.84** |\n| 100 % (no trunc.) | 70.02 | **72.19** |\n\n### **Q2: Computational Overhead of Methodology**\n\nIn algorithms 3 and 4, we provide the detailed stepwise FLOP count for our method.  Following the experiments testing the influence of the reweighting period on the truncation, we report the average time of the first 5 epochs below, note that regularization was only applied on the QKV matrices of a VIT-tiny Transformer\n\n**Average training time (first 5 epochs) and reserved GPU memory\n(ViT-tiny, regularization on QKV only)**\n\n| Model       | Variant / Setting | Avg. Time (s) | Reserved GPU Memory (GB) |\n| ----------- | ----------------- | ------------- | ------------------------ |\n| **Base**    | —                 | 150.91        | 8.84                     |\n| **AdamQ3R** | $T$ 500           | 205.34        | 6.28                     |\n|             | $T$ 100           | 207.39        | 6.31                     |\n|             | $T$ 50            | 209.29        | 6.21                     |\n|             | $T$ 20            | 217.90        | 6.28                     |\n|             | $T$ 10            | 218.88        | 6.22                     |\n| **LoRITa**  | depth 3           | 190.83        | 11.89                    |\n|             | depth 2           | 174.51        | 10.37                    |\n|             | depth 1           | 157.29        | 8.85                     |\n\nBased on the theoretical computational overhead outlined in Algorithm 3, along with the results above, we expect Q3R to incur additional computational and memory overhead. The added memory overhead stems from the additional weight matrices stored in Q3R to perform the weight least squares minimisation. We attribute the lower Reserved GPU Memory from implementation differences in the optimizer.\n\n### **Q.3 Robustness to Hyperparameter Variations**\n\nEmpirically, we found Q3R to be quite robust to its hyperparameters within reasonable ranges. Below, we are providing few empirical evidences from Table 5(CIFAR-10 on ViT-Tiny), however, similar results can be concluded across other datasets and networks like in Table 4 (CIFAR-100 on ViT-Base).\n\n Generally when choosing a lambda value it is easy to tell if the choice of lambda is too small by monitoring if the Q3R value increases within the first few epochs.  We recommend lambda that is slightly larger than the lower bound of the divergence threshold, as determining if the $\\lambda$ value is too large remains a challenge.\n\n####  Choice of Regularization Strength $\\lambda$\nFor CIFAR-10 trained on ViT-Tiny we can conclude the following with reference to Table 5(CIFAR-10 on ViT-T with AdamQ3R applied to only attention blocks). With the target rank held constant at **0.05**, lowering the regularization strength from **λ = 0.01** to **λ = 0.001** leaves accuracy virtually unchanged once more than ~15 % of the parameters are kept:\n\n| Parameter retention | λ = 0.001 | λ = 0.01 \n|---------------------|-----------|----------\n| 10 % | **46.08 %** | 46.00 % |\n| 15 % | 56.94 % | **60.32 %** |\n| 20 % | 66.29 % | 66.30 % | \n| 30 % | 70.17 % | **70.43 %** | \n| 40 % | 70.53 % | **70.60 %** | \n| 100 % (no trunc.) | **71.58 %** | 71.52 %|\n\nBeyond the 20 % retention point, the absolute accuracy gap never exceeds **0.3 percentage points**, confirming that **AdamQ3R is largely insensitive to λ within the 0.001–0.01 range in this practical operating regime**.\n\n#### Choice of Target Rank \nFurthermore, we can see below that it is  target rank $r_{\\text{target}}$ \nFor the other hyperparameter, if we refer to Table 1, for a fixed $\\lambda = 0.001$,\n| Parameter retention | ρ = 0.10 | ρ = 0.15 | ρ = 0.20 |\n|---------------------|---------|---------|---------|\n| 5 %  | **0.2322** | 0.1796 | 0.5606 |\n| 10 % | 0.4085 | **0.4758** | 0.5883 |\n| 15 % | 0.5606 | 0.4737 | **0.6175** |\n| 20 % | 0.6295 | 0.4387 | **0.6511** |\n| 30 % | 0.6526 | 0.3896 | **0.6749** |\n| 40 % | 0.6654 | 0.4335 | **0.6833** |\n| 100 % (no trunc.) | 0.6843 | 0.6737 | **0.6990** |\n\n> Once ≥ 30 % of the parameters are retained, the choice of rank changes accuracy by **≤ 3.8%**, confirming low sensitivity to rank in this regime. \n\nHere we **scale the target rank by the layer dimensions** so that a single\nhyper‑parameter *$\\rho$* works for networks of any size.\nFor a weight matrix of shape \\(m \\times n\\) we set target rank ( $r_{\\text{target}}$ ) = $\\rho\\frac{m+n}{2mn}$\nThis definition makes the choice of *r* directly transferable across layers\nwith different values of m and n.\n\n#### Choice of Reweighting Period $T$ \nWith λ = 0.001 and the target rank fixed at 0.2, shortening the re‑weighting period markedly improves accuracy once more than ~10 % of the parameters are retained. From Table 6 of our paper, we can conclude that **Frequent re‑weighting (period 5)** yields the top accuracy for every retention level ≥ 10 %. Once ≥ 30 % of the parameters are retained, switching from period 300 to period 5 adds up to **1.6 %**—a modest but consistent gain, indicating diminishing returns as the network grows denser.\n\n\n### **Q4: Strict Rank Constraint During vs. Post-Training**\nIt is true that our method does not impose a specific hard constraint during training, which is intentional that the proposed regularizer promotes low-rank weights rather than imposing a hard constraint during training.  This design choice is intentional. As reported in Section 5.1, we perform a one‑shot SVD at the end of training and keep only the leading singular vectors, which is in line with the post-processing present in other works on low-rank training, such as [ZAW24 \"LoRITa\"].  This step (i) guarantees the final rank exactly matches the budget (ii) yields the parameter counts shown in Tables 3–6. \n\nAdding an explicit rank constraint during training of Q3R will harm the stability of the method as the model's total loss might spike; furthermore, it would be easy to truncate at the wrong rank. Overall, however, we expect that that an adaptive truncation strategy based on stable rank notions similar to [WAUc'23] will improve performance after truncation for a fixed parameter budget, the exploration of which remains for future work.\n\n### **Q5: Performance in Large-Scale Settings**\nTraining from scratch on ImageNet-1k or LLM-scale corpora requires substantial compute time and resources, which were beyond the scope of our academic resource constraints. Reporting results on Transformer models trained on CIFAR-10 and CIFAR-100 is in line with the scale of experiments in the reference paper [ZAW24 \"LoRITa\"], where ImageNet was only used to train CNNs, but not Transformer models. In Section 5.2, we provide the somewhat larger-scale GLUE fine-tuning experiments which indicate competitive performance of Q3R for such settings as well.\n\nHowever, in order to test the potential of our method, we trained/finetuned our model on ImageNet with a different initialization using pretrained weights from an unregularized model. The parameters used for that training are $\\lambda = 0.001$ and $0.01$. The training was performed for 15 and 3 epochs, respectively, with a target rank scale of $\\rho = 0.2$. We observe that the test accuracy after retention of only $20\\%$ of the parameters is $76.5\\%$ and $77.4\\%$, respectively. On the other hand, an unregularized ViT-B model exhibits a test accuracy of $62.7\\%$ when only $20\\%$ parameters are retained. These preliminary results show the potential of our method on larger and more complex setups. We will provide comprehensive ImageNet results in the camera-ready version of the paper.\n\nIf you feel that we have addressed your questions and concerns adequately, we would appreciate if you can consider updating your score. We warmly welcome any further questions that you may have. Thank you!"}, {"review_id": "JuDlN8vOOv", "reviewer": "Reviewer_KK56", "summary": "This paper introduces Q³R (Quadratic Reweighted Rank Regularizer), a novel regularization framework designed to directly promote low-rank structure in deep neural network weights during training. Unlike traditional parameter-efficient fine-tuning (PEFT) approaches that rely on low-rank updates (e.g., LoRA, LoRITa) or post-hoc truncation, Q³R integrates a smoothed log-determinant surrogate into training via a majorizing quadratic model. The method is computationally tractable through a periodically updated reweighting operator and is embedded into a custom optimizer, AdamQ³R, which decouples the regularization from the adaptive gradient updates. The paper demonstrates that Q³R achieves strong performance on both image classification and NLP tasks, outperforming prior low-rank methods in aggressive compression settings (e.g., ViT-Tiny on CIFAR-10, RoBERTa on GLUE), and unifies low-rank pretraining and fine-tuning in a single framework.", "questions": "1. Can the authors provide wall-clock training time comparisons between Q³R, LoRA, and dense baselines? \n\n2. Can authors provide more discussion and tuning results regarding hyperparameters? How sensitive they are?\n\n3. Can authors evaluate the method in large-scale setting such as LLM fine-tuning or pre-training?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The use of iteratively reweighted least squares (IRLS) for low-rank deep network training is novel. Prior methods rely on factorized updates or heuristic truncation, whereas Q³R directly optimizes a smooth rank surrogate via majorization.\n\n2. Nice theoretical understanding. The paper provides a rigorous and detailed derivation of the Q³R regularizer and its integration into training via quadratic majorization.\n\n3. Strong empirical experiments: Experiments on ViT-Tiny/ViT-Base (CIFAR-10/100) and RoBERTa (GLUE) convincingly show that Q³R outperforms or matches state-of-the-art baselines (LoRA) especially at high compression ratios.\n\nWeaknesses\n\n1. Limited computational efficiency discussion. While the paper argues that Q³R introduces limited overhead, no explicit wall-clock time comparisons or FLOP breakdowns are provided. Since the method involves periodic SVDs and reweighting operations, this omission makes it hard to assess practicality in large-scale settings.\n\n2. Hyperparameter Sensitivity: Although the authors discuss stability of the regularization weight, the method still requires tuning of it, rank targets, and reweighting periods. The paper does not fully explore how sensitive performance is to these choices.\n\n3, Limited scope of the experiment: The method is tested only on moderate-scale models and datasets (CIFAR). Its applicability to large-scale LLMs (e.g., GPT-class models) remains unknown, such as fine-tuning and pre-training LLM on tasks such as C4.", "was_revised": true, "final_justification": "The authors have addressed my concerns and questions, and thus I would like to raise my score accordingly.", "rebuttal": "We thank the reviewer for the positive feedback. We appreciate that you value the rigorous derivation and quadratic-majorization view of IRLS that clarify why the regularizer works as well as our empirical results on ViT-Tiny/ViT-Base and RoBERTa that match or surpass state-of-the-art baselines. \n\nPlease consider our thoughts and clarifications for your questions: \n\n### **Q1. Wall-clock training time comparisons**\nPlease refer to our response to Q.2 of Reviewer \"aasu\". Additionally, in comparison to the LoRa run time, we observe the following results.\n**Average training time (first 5 epochs)\n(ViT-tiny, regularization/factorization on QKV only)**\n\n| Model       | Variant / Setting | Avg. Time (s) |\n| ----------- | ----------------- | ------------- |\n| **Base**    | —                 | 150.91        |\n| **AdamQ3R** | $T$ 500           | 205.34        |\n|             | $T$ 100           | 207.39        |\n|             | $T$ 50            | 209.29        |\n|             | $T$ 20            | 217.90        |\n|             | $T$ 10            | 218.88        |\n| **LoRA**  | R=4             | 124.83        |\n|             | R=14           | 144.51        |\n|             | R=28           | 145.28        |\n\n\n### **Q2.Robustness to Hyperparameter Variations**\n\nPlease refer to our response to Q.3 of Reviewer \"aasu\".\n\n### **Q3.  Performance in Large-Scale Settings**\n\nPlease refer to our response to Q.5 of Reviewer \"aasu\".\nAdditionally, we fine‐tune pre‐trained RoBERTa models on the GLUE benchmark using Adam\\QER with the proposed \\QER regularizer, and compare against full fine‐tuning and LoRA. We conclude from Table 2 that Q3R matches or exceeds LoRA’s accuracy on most tasks, showing performance closer to dense fine-tuning.\n\nIf you feel that we have addressed your questions and concerns adequately, we would appreciate if you can consider updating your score. We warmly welcome any further questions that you may have. Thank you!"}, {"review_id": "d1NWEk34PA", "reviewer": "Reviewer_9KVH", "summary": "The paper proposes to use a regularizer which reduces the rank of the weight matrices during neural network training. The regularizer is a smooth approximation of the log-determinant of the weight matrix and optimized using the iteratively reweighed least squares (IRLS)\nalgorithm. This leads to an elegantly modified AdamW algorithm, which instead of the regular weight-decay receives a \"modified weight\ndecay\" which reduces the rank of the weight matrix. The method is evaluated in a small-scale setting on CIFAR-10, 100 and NLP tasks.", "questions": "- What are the numbers in Table 1, is it accuracy or error? The text and figure caption only talk about performance, which makes it hard to assess the results. \n- Is there an inherent overhead in the method which prevents to run it on larger models, such as Llama series, as seen in LoRA papers?\n- An alternative approach to low-rank regularization are proximal algorithms using e.g. proximal operators of the nuclear norm or its nonconvex variants. Is there an inherent advantage of the IRLS approach over proximal algorithms?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The paper was easy to follow, and the mathematical derivations are rigorous building on recent works on IRLS from applied mathematics.\n- The proposed modification of AdamW is elegant, easy to implement and comes with little overhead.\n\nWeaknesses:\n- The experiments seem preliminary and small scale, making it is hard to assess whether the proposed method will work well for larger and realistic settings. The accuracies (?) reported in table 1 are very low for CIFAR-10, a simple LeNet CNN gives over 80%.\n- As the method is motivated as a competitor to LoRA, it would be more convincing to see similar experimental setups as in LoRA papers, for instance, fine-tuning Llama 3 or Qwen rather than RoBERTA models. \n- The use of IRLS for rank regularization is not new and a standard technique from applied mathematics. Given that there is limited novelty in the method, this is a mostly empirical work which requires larger scale experiments to be valuable to the community.", "was_revised": true, "final_justification": "The authors rebuttal cleared some of my concerns, while others about the experiments still remain. I have updated my score, but still lean slightly towards rejection of the paper.", "rebuttal": "Thank you for your comments and thoughts. We appreciate your mention on the proposed modification of AdamW being elegant and easy to implement.\n\nWe would like to propose our thoughts and justifications to your questions as the following: \n\n### **Q1: Numbers in Table 1**\n\nThe numbers in Table 1 report the test accuracies of our method on CIFAR-10 on ViT-Tiny when $5\\%-100\\%$ parameters are retained. In this setup, we apply our regularizer to the attention blocks as well as the MLP layers. \n\n### **Q2: Relationship to LoRA Models & Fine-Tuning**\nWe think that there might be a misunderstanding here: The proposed Q3R is a _low-rank training_ method, unlike LoRA, which is a _low-rank fine-tuning_ method which, due to its introduction of the additive product $B\\cdot A$ of matrices $B$ and $A$, is inherently _unable_ to reduce the number of model parameters. For this reason, we would like to point out our competitive post-truncation accuracy results presented in Tables 1, 3 and 4, in which we show that we obtain significantly improved post-truncation accuracies of ViT models trained on CIFAR data, e.g., with models with only 20% retained parameters. As we mentioned in our related works section, previous works such as [LSMR23] and [HCB'24] already reported a sub-par performance of a naive LoRA extention to low-rank pretraining (e.g., by choosing the frozen weight matrix to be the zero matrix). For completeness, we have provided an experiment in Table 3 in which LoRA (after hyperparameter tuning) achieves only $0.5859$ test accuracy on CIFAR-10 (low-rank training only on attention blocks) at $20%$ retained parameters compared to $Q3R$, which achieves a test accuracy of $0.6792$ for the same number of retained parameters.\n\nThe fine-tuning experiments we provided in Section 5.2 can be seen rather as a proof-of-concept and sanity check for the methodology's applicability for low-rank fine-tuning, which is arguably an easier problem than low-rank pre-training. Due to academic experimental constraints, we restricted this study to a RoBERTa model. There is no fundamental reason why Q3R cannot also be used for larger models. We quantify the additional computational burden in lines 251-262. In its current implementation (which focus was not computational efficiency), Q3R imposes a noticable, but managable computational overhead for the Transformer models that we used in our experiments; see the discussion about computational overhead in our reply to Reviewer aasu for additional details. \n### **Q3: Q3R vs. Proximal Methods for Rank Regularization**\nWe appreciate the idea of using proximal algorithms to induce low-rank regularization, for example for similar rank surrogates as the (smoothed) log-determinant. A well-known work that follows this idea is [Alvarez, Salzmann, \"Compression-aware Training of Deep Networks\", NIPS 2017], which applies proximal stochastic gradient descent to a nuclear norm regularizer. An important downside of such approaches is that they require a full SVD at _every_ iteration, which becomes computationally prohibitive for larger networks. In contrast, our Q3R approach only requiresd an SVD each $T$ iterations (reweighting period). The reweighting period $T$ is chosen as $T \\in \\{3,5 ,25\\}$ to obtain optimal post-truncation performance in our ablations; furthermore, as discussed in our answer to Q2 of Reviewer KK56, the performance does not significantly deteriorate even for large reweighting periods such as $T=200$.\n\nAnother downside of such proximal-inspired methods is that they tend to resemble closely iterative truncation of the singular value vectors of the weight matrices at each iteration at certain levels in the case of non-convex objectives such as the smoothed log-determinant. In contrast, the proposed Q3R regularizer imposes low-rankness in a \"softer\" manner as it only gradually imposes low-rankness during training, and thus, allows for a better exploration of the space of weight matrices before convergence. \n\n### **Novelty of Q3R**\nWe respectfully disagree with your point about the novelty of Q3R. While the regularizer of Q3R corresponds to an IRLS objective as discussed in our related work, we use the particular definition of the reweighting operator that has been only used in [KMV21,GTK24] and significantly differs from older IRLS formulations as defined in [FRW11,MF10] - using an analogue of Q3R based on those IRLS formulations will lead to a significantly slower convergence to low-rank weights. Furthermore, we would like to point out that to the best of our knowledge, IRLS-type rank-regularizations have not been explored in the literature, and an extention to low-rank pretraining of deep neural networks necessitates several innovations such as the particular choice of the smoothing parameter update, the reweighting period, and the integration into the optimizer via AdamQ3R, to name a few.\n\nIf you feel that we have addressed your questions and concerns adequately, we would appreciate if you can consider updating your score. We warmly welcome any further questions that you may have. Thank you!"}, {"review_id": "7Hy4Bi67BC", "reviewer": "Reviewer_krCR", "summary": "This work introduces a parameter-efficient training method using a quadratic regularizer that majorizes a smoothed log-determinant as a rank surrogate. The approach enables effective low-rank pretraining and fine-tuning of deep models, demonstrating competitive performance across both vision and language tasks.", "questions": "1, $\\textit{Line 283}$, what is the another tuning hyper-parameter?What is the other tuning hyperparameter mentioned here, in addition to $\\lambda$? Please clarify.\n\n2, The use of $r_{target}$ raises concerns about the sensitivity of the model to the predefined $r_{target}$, the step size, and the weight initialization. \nWhile the sensitivity to $r_{target}$ is briefly addressed in the limitations section, could the authors elaborate on the impact of the other two factors—especially how step size and initialization might affect training stability and final performance? \nFor instance, is it beneficial for the initial weight matrices to have a rank close to $r_{target}$ , or is random initialization sufficient?", "limitations": "See Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "The proposed method explicitly induces low-rank structure in deep neural network weights via continuous regularization. It is also robust to the choice of the target rank hyperparameter $r_{target}$, reducing the need for extensive hyperparameter tuning during initialization.\n\nWeaknesses: The experiments on language tasks are limited to fine-tuning only, which may not be sufficient to fully support the authors’ claim.\n Based on the paper’s title and abstract, one would expect competitive performance in pre-training across vision and language domains.\nWhile it may be impractical to train large language models from scratch, a more direct application of the proposed method to pre-trained weights—rather than using a LoRA-style side path—would better support the claim of unified low-rank training.\nThis does not imply that the experiments are poorly designed, but rather that the scope of the evaluation falls short of the expectations set by the paper.", "was_revised": true, "final_justification": "My concerns have been resolved and my rating has been raised to borderline accept.", "rebuttal": "Thank you for your review. We appreciate you noting that the proposed method reduces the need for extensive hyperparameter tuning during initialization and training. We have provided more empirical evidences in support of that statements below. Please consider the following responses regarding your assessment.\n\n### **Usage of LoRA-style Additive Weights for Fine-Tuning**\n\nIn our parameter-efficient fine-tuning (PEFT) experiments we adhered to standard practice by freezing the pretrained backbone, inserting low-rank trainable adapters, and then applying Q3R to those adapters; this provides a clean, apples-to-apples comparison with other PEFT baselines. From our observation, when using the pretrained weights as initialization without an additive side-path, Q3R does not lead to competitive performance of the dense model, which we were most interested in for fine-tuning, presumably due to Q3R's strong low-rank compression bias, which is here problematic given the small size of the fine-tuning data compared to the model size. We still expect Q3R to do better than other low-rank fine-tuning methods which target weight parameter reduction (unlike LoRA).    \n\n### **Q1: Hyperparameter of Line 283**\nWe apologize for the typo and thank you for pointing this to us. The hyperparameters mentioned in line 283 should be regularization Strength $\\lambda$, Target Rank (related $\\rho$) and Reweighting Period $T$.\n\n### **Q2. Robustness to Hyperparameter Variations**\nPlease refer to our response to Q.3 of Reviewer \"aasu\".\n\n\nIf you feel that we have addressed your questions and concerns adequately, we would appreciate if you can consider updating your score. We warmly welcome any further questions that you may have. Thank you!"}]

Author rebuttal:

Meta-review: The reviewed work applies iteratively reweighted least squares to the low-rank training and fine tuning of large deep learning models. The reviewers appreciate the method as well-motivated and grounded in a solid theoretical perspective. The concerns of the reviewers regarding the sensitivity to hyperparameter tuning were largely alleviated during the rebuttal. A remaining weakness of the methodology is that its main benefit of being applicable to both training and fine-tuning is somewhat undermined by the requirement to work with full dense matrices, which prevents it from leveraging low-rank structure to accelerate training. In summary, this is a somewhat borderline paper that I am in favor of accepting.
Decision: Accept (poster)