Retraction-free optimization over the Stiefel manifold with application to the LoRA fine-tuning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:04:14

Abstract: Optimization over the Stiefel manifold has played a significant role in various machine learning tasks. Many existing algorithms either use the retraction operator to keep each iterate staying on the manifold, or solve an unconstrained quadratic penalized problem. The retraction operator in the former corresponds to orthonormalization of matrices and can be computationally costly for large-scale matrices. The latter approach usually equips with an unknown large penalty parameter. To address the above issues, we propose a retraction-free and penalty parameter-free algorithm, which lands on the manifold. A key component of the analysis is the convex-like property of the quadratic penalty of the Stiefel manifold, which enables us to explicitly characterize the penalty parameter. As an application, we introduce a new algorithm, Manifold-LoRA, which employs the landing technique and a carefully designed step size strategy to accelerate low-rank adaptation (LoRA) in fine-tuning large language models. Numerical experiments on the benchmark datasets demonstrate the efficiency of our proposed method.

Main paper: 
 introduction: 
 Optimization over the Stiefel manifold has attracted considerable attention in the context of machine learning, e.g., RNN [3] , batch normalization [10] , and distributionally robust optimization [8] . The mathematical formulation of this class of problems is:EQUATIONwhere r ≤ d and f : R d×r → R is a continuously differentiable function. The most popular methods for solving (1) are retraction-based algorithms, which have been extensively studied in the context of manifold optimization [2, 23, 6] . Recently, to alleviate the possible computational burden of the retraction operator, some retraction-free methods have been developed in [19, 18, 41, 1] . The ideas in these papers are based on a combination of the manifold geometry and a penalty function for the manifold constraint, which involves an unknown but sufficiently large penalty parameter. For largescale machine learning applications, retraction-free algorithms are preferred. However, designing retraction-free algorithms with a known penalty parameter for solving (1) remains a challenge.Another motivation for studying retraction-free methods arises from its application in the fine-tuning of large language models (LLMs). Recently, LLMs have revolutionized the field of natural language processing (NLP), achieving unprecedented performance across various applications [33, 32] . To tailor pretrained LLMs for specific downstream tasks, the most common approach is full fine-tuning, which requires prohibitively large computational resources due to the need to adapt all model weights, hindering the deployment of large models. As a result, parameter-efficient fine-tuning (PEFT) has gained widespread attention for requiring few trainable parameters while delivering comparable Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.or even superior results to full fine-tuning. This paradigm involves inserting learnable modules or designating only a small portion of weights as trainable, keeping the main model frozen [21, 26, 44] .Among fine-tuning methods, low-rank adaptation (LoRA) [22] has become the de facto standard among parameter-efficient fine-tuning techniques. It assumes that the change in weights lies in a "low intrinsic dimension", thereby modelling the update ∆W ∈ R d×m by two low-rank (not greater than a small integer r) matrices A ∈ R r×m and B ∈ R d×r , i.e., ∆W = BA. Since r ≪ d, the requirements on both storage and computation are significantly reduced. Due to its decompositional nature, there is redundancy in the representation of ∆W . Traditional optimization methods for LoRA do not exploit this redundancy, which consequently undermines model performance. Instead, we reformulate LoRA fine-tuning as an optimization problem over the product of Stiefel manifolds and Euclidean spaces. Therefore, we propose an algorithmic framework called Manifold-LoRA to accelerate the fine-tuning process and enhance model performance. Moreover, by exploiting projected gradients and incorporating a parameter-free penalty, the overhead that our method incurs is relatively negligible. Our contributions are as follows:• We first prove the existence of explicit choice for the penalty parameter by establishing a strong convexity-like condition of the nonconvex penalty problem associated with the Stiefel manifold constraint. Furthermore, for the given penalty parameter, under mild conditions, we prove that the iterates of our proposed retraction-free gradient descent method eventually land on the Stiefel manifold and achieve the optimality of (1).• Building upon the established landing theory of retraction-free and penalty parameter-free method and the AdamW framework, we proposed a new method, Manifold-LoRA, which employs a carefully designed step size strategy to accelerate the training process of finetuning. Compared with the conventional AdamW method, we use the penalized gradient instead of the usual gradient, and the computational overhead is negligible.• Numerical experiments are conducted on a wide range of NLP tasks, demonstrating the efficiency of our algorithm. Specifically, compared to the vanilla LoRA, our Manifold-LoRA with half the trainable parameters not only delivers fast convergence but also yields improved generalization. In particular, Our method converges twice as fast as baseline methods on several typical datasets, including the SQuAD 2.0 dataset and the CoLA dataset.Optimization over the Stiefel manifold. Optimization over the Stiefel manifold has attracted lots of attention due to its broad applications. Through the use of retraction, known as the generalization of the exponential map, the Riemannian gradient descent is proposed [2, 6, 23] , where all iterates lie on the manifold. When such retraction is computationally costly, the authors [19] develop a retractionfree algorithm based on the augmented Lagrangian method. More recently, by defining the constraint dissolving operator and adding a sufficiently large penalty term, the authors [41] convert the manifold constrained problem (1) into an unconstrained problem and then apply unconstrained optimization algorithms. In [1] , motivated by the convergence of the Oja's flow, a landing flow, consisting of the projected gradient and the gradient of the penalty function, is developed to retraction-free method for the squared Stiefel manifold, i.e., d = r. All of these methods rely on an unknown penalty parameter to ensure the convergence. This motivates us to design penalty parameter-free algorithms, which could significantly reduce the need for tuning parameters in practical implementations.LoRA. There are numerous variants of LoRA aiming to improve performance or reduce memory usage. AdaLoRA [46] , a well-known successor, introduces the idea of adaptively adjusting the rank of different layers by incorporating an additional vector g to serve as the diagonal of a singular value matrix. This approach leverages a revised sensitivity-based importance measure to decide whether to disable entries in vector g and in matrices A and B. A similar work, SoRA [15] ,adopts the same model architecture as AdaLoRA, but proposes a different way to update vector g after training. This update rule is the proximal gradient of L 1 loss, acting as a post-pruning method. Additionally, a recently emerged method called VeRA [25] significantly reduces memory overhead while maintaining competitive performance. Based on the idea that networks with random initialization contain subnetworks that are near-optimal or optimal [17] , VeRA only uses two frozen low-rank matrices shared by all layers, training scaling vectors unique to each layer. Although LoRA has gained significant popularity and various variants have been developed, the potential for efficient training through leveraging the manifold geometry to reduce redundancy has not been well-explored.For a matrix X ∈ R d×r , we use ∥X∥ to denote its Frobenius norm. For a squared matrix A ∈ R d×d , we define sym(A) = A+A ⊤ 2 Retraction-free and penalty parameter-free optimization over the Stiefel manifoldIn this section, we focus on the design of retraction-free and penalty parameter-free algorithms for solving problem (1) . We will first present the retraction-free algorithm and then show how the penalty parameter can be explicitly determined by characterizing the landscape of the penalty function. 
retraction-free algorithms: 
 Inspired by the retraction-free algorithms [19, 41, 1] , we consider the following retraction-free gradient descent method for problem (1):EQUATIONwhere α, µ > 0 are step sizes and the projected gradient gradf (X k ) := ∇f (X k ) -X k sym(X ⊤ k ∇f (X k )). Note that the tangent space of St(d, r) is T X k St(d, r) := {ξ ∈ R d×r : X ⊤ k ξ + ξ ⊤ X k = 0}. Then, for X k ∈ St(d, r), gradf (X k ) is the projection of the Euclidean gra- dient ∇f (X k ) to the tangent space, i.e., gradf (X k ) = P T X k St(d,r) (∇f (X k )). Note that the term X k (X ⊤ k X k -I d )is exactly the gradient of the following quadratic penalty functionφ(X) := 1 4 ∥X ⊤ X -I∥ 2 .As will be shown in our theorem, the use of the projected gradient is essential for landing on the manifold. This differs with the usual penalty method, which optimizes f (X) + µφ(X) using the updateX k+1 = X k -α∇f (X k ) -µX k (X ⊤ k X k -I d ), needs µ → ∞ to guarantee the feasibility.It is known that a large penalty parameter yields better feasibility [29, Chapter 17] . To make the iterative scheme (2) be penalty parameter-free, we need a careful investigation on the landscape of the following optimization problem:EQUATIONIt can be easily verified that problem (3) is nonconvex and its the optimal solution set is St(d, r). The key of obtaining an explicit formula of µ is to establish certain strong convexity-type inequality and show the gradient descent method with step size µ has linear convergence.For any X ∈ St(d, r), let us denote X := P St(d,r) (X). Let X = U SV ⊤ be the singular value decomposition with orthogonal matrices U ∈ R d×r , V ∈ R d×d and diagonal matrix S ∈ R d×d , thenX = U V ⊤ .Building on these notations, we demonstrate that problem (3) satisfies the restrict secant inequality (RSI) [45] , which serves as an alternative to the strong convexity in the linear convergence analysis of gradient-type methods.Lemma 1. For any X ∈ R d×r with ∥X -X∥ ≤ 1 8 , we haveEQUATION)With the above RSI, we have the linear convergence of the gradient descent update for (3), i.e.,X k+1 = X k -µ∇φ(X k ).(5) Lemma 2. Let the sequence {X k } be generated by (5) with µ = 1 3 . Suppose that ∥X 0 -X0 ∥ ≤ 1 8 .We haveEQUATION)The proofs of Lemmas 1 and 2 can be found in Appendix B.Building on the established linear convergence of gradient descent for problem (3), we are now able to show that the iterates generated by (2) will land on the Stiefel manifold eventually, and the limiting point is a stationary point of (1), i.e., gradf (X ∞ ) = 0.Let us start with the Lipschitz continuity of gradf (X). For any X ∈ ŪSt(d,r) ( 1 8 ), we defineP T X St(d,r) (U ) = U -Xsym(X ⊤ U ) for U ∈ R d×r .We first have the following quadratic upper bound on f from its twice differentiability and the compactness of St(d, r).Lemma 3. There exists a constant L > 0 such that for any X, Y ∈ St(d, r), the following quadratic upper bound holds:EQUATION)In addition, there exists a constant L > 0 such that for anyEQUATIONBy the linear convergence result in Lemma 2, we have the following bound on the feasibility error.Lemma 4. Let {X k } be the sequence generated by (2) with µ = 1 3 and ∥X 0 -X0 ∥ ≤ 1 8 . We haveEQUATIONThe following one-step descent lemma on f is crucial in establishing the convergence.Lemma 5. Let {X k } be the sequence generated by (2) with µ = 1 3 and ∥X 0 -X0 ∥ ≤ 1 8 . We haveEQUATIONFrom the above lemma, the one-step descrease on f is related to both the gradient norm of f and the feasibility error. In terms of convergence, we need both gradf (X k ) and ∥X ⊤ k X k -I∥ converge to 0.The following theorem demonstrates that the retraction-free and penalty parameter-free update (2) converges.Theorem 1. Let {X k } be the sequence generated by (2) with µ = 1 3 and ∥X 0 -X0 ∥ ≤ 1 8 . If the step size α < 1 2c1 for some c 1 large enough, then we haveEQUATION)The proofs of the above lemmas and theorem are presented in Appendix B.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss our limitations in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
accelerate lora fine-tuning with landing: 
 In this section, we will first clarify where the Stiefel manifold constraint comes from in the LoRA fine-tuning. Then, we will apply the above developed retraction-free and penalty parameter-free method to enhance LoRA fine-tuning.In neural networks, the dense layers perform matrix multiplication, and the weight matrices in these layers usually have a full rank. However, when adapting to a specific task, pre-trained language models have been shown to have a low intrinsic dimension, allowing them to learn efficiently even with a random projection to a smaller subspace. One possible drawback in the current LoRA fine-tuning framework is that the low-rank decomposition ∆W into product BA is not unique. Specifically, for any invertible matrix C, it holds that BA = (BC)(C -1 A). Note that BC shares the same column space with B. This suggests us optimizing the subspace generated by B instead of B itself.Numerous studies in the field of low-rank optimization, e.g., [7, 13, 12] , investigate the manifold geometry of the low-rank decomposition and develop efficient algorithms. However, such geometry has not been explored in the LoRA fine-tuning.To address such redundancy (i.e., the non-uniqueness of BA representations), we regard B as the basis through the manifold constraint and A as the coordinate of ∆W under B. Hence, the optimization problem can be formulated asEQUATIONwhere Ob(d, r):= {B ∈ R d×r : diag(B ⊤ B) = 1}. Compared to the Stiefel manifold St(d, r),the oblique manifold Ob(d, r) necessitates that the matrix B has unit norms in its columns, without imposing requirements for orthogonality between the columns. Problem ( 12) is an optimization problem over the product of manifolds and Euclidean spaces.The retraction-free method is well-suited to address (12) , simultaneously minimizing the loss function L(BA) and constraint violation. To control the constraint violation, we use the quadratic penaltiesR s (B) := ∥B ⊤ B -I∥ 2 and R o (B) := ∥diag(B ⊤ B) -1∥ 2for the Stiefel manifold and oblique manifold, respectively. As shown in the landing theory in Section 2, we shall use the projected gradient of the loss part instead of the Euclidean gradient. For the Stiefel manifold and the oblique manifold, the respective projected gradients areEQUATIONandEQUATIONwhere sym(X) := (X + X ⊤ )/2. Thus, the gradients of our retraction-free method for A and B are∇ A L(BA) and grad B L(BA) + µ∇R s (B)( or ∇R o (B)).Note that B and A represent the basis and the coordinate of ∆W , respectively. This results in different magnitudes and different Lipschitz constants of their gradient function. In fact, let X = BA.∇ A L(BA) = B ⊤ ∇ X L(X), ∇ B L(BA) = ∇ X L(X)A ⊤ .Then,∥∇ A L(BA 1 ) -∇L(BA 2 )∥ ≤ ∥B∥ 2 L g ∥A 1 -A 2 ∥, ∥∇ B L(B 1 A) -∇L(B 2 A)∥ ≤ ∥A∥ 2 L g ∥B 1 -B 2 ∥,where L g is the Lipschitz constant of ∇ X L(X) and ∥ • ∥ 2 represent the matrix ℓ 2 norm (i.e., the largest singular value). Note that the step size generally should be propositional to the reciprocal of Lipschitz constant for the gradient type algorithms [29, 5] . Hence, we schedule the learning rates for the two matrices based on their respective ℓ 2 norms. Having prepared the above, we incorporate the AdamW optimizer [28] with our manifold-accelerated technique to enhance the LoRA fine-tuning, as presented in Algorithm 1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 In this section, we delve into the experimental results and their detailed analysis. This discussion is structured around two principal areas: (1) the performance gain compared to other mainstream finetuning methods and accelerated convergence achieved through our manifold-constrained optimization approach;(2) the convergence of matrix B onto the manifold, illustrated by the heat map of B ⊤ B.Baselines We compare our approach against several baseline methods, including full fine-tuning, Adapter [21] , BitFit [44] and LoRA [22] . The variants of the Adapter method are excluded from the baselines, as their performance are relatively similar.Implementation Details Our code is based on Pytorch [31] , Huggingface Transformers [40] and an open-source plug-and-play library for parameter-efficient fine-tuning opendelta [24] . The bottleneck dimension for the Adapter is set to 16 or 32, ensuring that the number of trainable parameters aligns Algorithm 1: Manifold-LoRAInput: Initial point A0, B0, µ ∈ R, β1 = 0.9, β2 = 0.999, upper bound ≥ lower bound > 0, ϵ = 10 -8 , γ > 0, λ ∈ R, and k = 0. while Stopping conditions not met do for C ∈ {A, B} do if C = B then Set g(C k ) according to (13) or ( 14) using the stochastic estimate of ∇BL(B k A k ) // Projected gradient for matrix B else Set g(C k ) to be the stochastic estimate of∇AL(B k A k ) end end m(C k ) ← β1m(C k ) + (1 -β1)g(C k ) v(C k ) ← β2v(C k ) + (1 -β2)g 2 t (C k ) m(C k ) ← m(C k ) 1-β t 1 v(C k ) ← v(C k ) 1-β t 2 η(C k ) ← clip(normC k , upper bound, lower bound)// Scheduling step size of matrix A and BC k ← C k-1 -ηt(C k ) mt(Ck)/ vt(Ck) + ϵ -λC k-1 if C = B then C k ← C k -µ∇Rs(C k )( or ∇Ro(C k )) // Apply penalty gradient for matrix B end end k ← k + 1 endclosely with that of the LoRA method and the new layers are inserted into the attention layer and feed-forward layer. The update of LoRA is scaled by a hyper-parameter α. This value is typically left unmodified, as it is usually set as 16 or 32 and never tuned [22, 43] . The exponential moving average parameters β 1 and β 2 of AdamW [27] are set to their default values of 0.9 and 0.999, respectively. All the experiments are conducted on NVIDIA A800 GPUs. More details are presented in Appendix C.We first evaluate our backbone model DeBERTaV3-base [20] on GLUE [37] benchmark containing nine sub datasets, including MNLI [39] , SST-2 [36] , CoLA [38] , QQP [37] , QNLI [35] , RTE [4] , MRPC [16] , and STS-B [37] .Experimental results of the GLUE dataset are recorded in Table 1 . It can be seen that our method is consistently superior to other baselines. Notably, for RTE and STS-B datasets, both sphereconstrained (i.e., oblique manifold-constrained) and Stiefel-constrained have an obvious performance gain even with only half the trainable parameters compared to the LoRA baseline, i.e., Sphere r=8 and Stiefel r=8 beat LoRA r=16 . In addition, with the help of manifold geometry, the fine-tuning process can be significantly accelerated compared to the vanilla AdamW optimizer, achieving a lower training loss, as shown in Figure 1 . Particularly on the CoLA dataset presented in Figure 1a , our approach achieves the same training loss as the standard Adam optimizer but requires nearly half the number of epochs.We conduct an evaluation on two question answering datasets: SQuAD v1.1 [35] and SQuADv2.0 [34] . Manifold-LoRA is used to fine-tune DeBERTaV3-base for these tasks, which are treated as sequence labeling problems predicting the probability of each token as the start or end of an answer span.The main experimental results are presented in Table 2 . For LoRA and our algorithms, new layersare inserted into W q , W k , W v , W o , F C 1 , F C 2 .Notably, both manifold-regularized LoRA variants consistently outperform all fine-tuning methods. Additionally, we plot the training loss, evaluation exact match, and evaluation F1 scores against epochs in Figure 2 . We conclude that the proposed Manifold-LoRA method achieves a 2x speed-up in training epochs compared to AdamW, while simultaneously improving model performance. We also illustrate the heat map of B ⊤ B in Figure 3 , which indicates that the matrix B lands on the manifold eventually. This supports our assertion that landing on manifold enhances the performance of LoRA.The E2E NLG Challenge [30] , as introduced by Novikova, provides a dataset for training end-to-end, data-driven natural language generation systems, widely used in data-to-text evaluations. test examples, all from the restaurant domain. We test our method on the E2E dataset using GPT-2Medium and Large models, following the experimental setup outlined by LoRA [22] . For LoRA, we set the hyperparameters to match those specified in the original paper.The results from the E2E dataset are recorded in Table 3 , where we focus on comparing LoRA and Manifold-LoRA. The results clearly indicate that our proposed algorithm outperforms the established baselines. Also, as shown in Figure 4 , the matrix B resides on the manifold even at the early training stage, validating the feasibility of our method.  
conclusion: 
 Optimization over the Stiefel manifold has been widely used in machine learning tasks. In this work, we develop a retraction-free and penalty parameter-free gradient method, and prove that the generated iterates eventually land on the manifold and achieve the optimality simultaneously. We then apply this landing theory to avoid the possible redundancy of LoRA fine-tuning in LLMs. Specifically, we reformulate the LoRA fine-tuning as an optimization problem over the Stiefel manifold, and propose a new algorithm, Manifold-LoRA, which incorporates a careful analysis of step sizes to enable fast training using the landing properties. Extensive experimental results demonstrate that our approach not only accelerates the training process but also yields significant performance improvements.Our study suggests several potential directions for future research. Although the established landing theory focuses on the Stiefel manifold, extending this theory to general manifolds is one potential direction. Additionally, evaluating the performance of Manifold-LoRA on LLMs with billions of parameters would be valuable. Due to the heterogeneity of different layers, incorporating adaptive ranks for ∆W across different layers is another possible direction. This may be achievable by adding sparsity regularization to the coordinate matrix A.The notion of proximal smoothness, as introduced by [11] , refers to the characteristic of a closed set whereby the nearest-point projection becomes a singleton when the point is in close enough to the set.This property facilitates algorithmic and theoretical advancements by endowing nonconvex sets with convex-like structural attributes. Specifically, for any positive real number γ, we define the γ-tube around M as U M (γ) := {x : dist(x, M) < γ}. We say a closed set M is γ-proximally smooth if the projection operator P M (x) := argmin y∈M ∥y -x∥ 2 is a singleton whenever x ∈ U M (γ).Obviously, any closed and convex set is proximally smooth for arbitrary γ ∈ (0, ∞). According to [11, Corollary 4.6] , a closed set M is convex if and only if it is proximally smooth with a radius of γ for every γ > 0. It is worth noting that the Stiefel manifold is 1-proximally smooth. By following the proof in [11, Theorem 4.8] ,EQUATIONwhere ŪSt(d,r) ( 12) := {x : dist(x, St(d, r)) ≤ 1 2 } is the closure of U St(d,r) ( 1 2). It is worth noting that for any closed convex set M ⊂ R d×r , the projection operator P M is 1-Lipschitz continuous over R d×r . 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We use the Hugging face and opendelta as our base code and make some modifications. We use GLUE, E2E, and Suqad three dataset.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We correctly cite the code and datasets we used in Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Proof of Lemma 1Proof. Denote the SVD of X by X = U SV ⊤ . Then, it holds that dist(X, St(d, r)) = ∥X -X∥ = ∥s -1∥ 2 , where s = diag(S). Furthermore, we have∇φ(X), X -X = U SV ⊤ (V S 2 V ⊤ -I), U SV ⊤ -U V ⊤ = U (S 3 -S)V ⊤ , U (S -I)V ⊤ = tr((S 3 -S)(S -I)) ≥ 3 2 ∥s -1∥ 2 2 = 3 2 ∥X -X∥ 2 ,where the last inequality is from min i s i (s i + 1) ≥ 105 64 ≥ 3 2 . This completes the proof.Proof. Assume that ∥X k -Xk ∥ ≤ 1 8 . Denote the SVD of X k by U SV ⊤ . Let s = diag(S). Then, we have 7 8 ≤ s i ≤ 9 8 for any i. This impliesEQUATIONHence, we have∥X k+1 -Xk+1 ∥ 2 ≤ ∥X k+1 -Xk ∥ 2 = ∥X k - 1 3 ∇φ(X k ) -Xk ∥ 2 = ∥X k -Xk ∥ 2 - 2 3 X k -Xk , ∇φ(X k ) + 1 9 ∥∇φ(X k )∥ 2 ≤ (1 -1 + 2 3 )∥X k -Xk ∥ 2 = 2 3 ∥X k -Xk ∥ 2 ,where the first inequality is from Xk+1 = argmin X∈St(d,r) ∥X -X k ∥ 2 and the second inequality is due to Lemma 1 and ( 16).Proof. Due to the twice differentiability of f and the compactness of St(d, r), the inequality (7) directly follows from [9, Lemma 2.4] and [14, Lemma 4.2] , where L := L f + D f with L f being the Lipschitz constant of ∇f (X) over St(d, r) and D f := max X∈St(d,r) ∥∇f (X)∥.For the second argument, we have∥gradf (X) -gradf (Y )∥ ≤∥P T X St(d,r) (∇f (X)) -P T X St(d,r) (∇f (Y ))∥ + ∥P T X St(d,r) (∇f (Y )) -gradf (Y )∥ ≤L f ∥X -Y ∥ + 1 2 ∥X(X ⊤ ∇f (Y ) + ∇f (Y ) ⊤ X) -Y (Y ⊤ ∇f (Y ) + ∇f (Y ) ⊤ Y )∥ ≤L f ∥X -Y ∥ + 1 2 ∥X((X -Y ) ⊤ ∇f (Y ) + ∇f (Y ) ⊤ (X -Y ))∥ + 1 2 ∥(X -Y )(Y ⊤ ∇f (Y ) + ∇f (Y ) ⊤ Y )∥ ≤L f ∥X -Y ∥ + 1 2 (2 Df + 3 Df )∥X -Y ∥ =(L f + 5 2 Df )∥X -Y ∥,where Df := max X∈ ŪSt(d,r) ( 1 8 ) ∥∇f (X)∥, the second inequality is due to the contractive property of P T X St(d,r) , and the last inequality is from the fact that ∥Y ∥ 2 ≤ 3 2 . By setting L = L f + 5 2 Df , we complete the proof.Proof. It follows that∥X k+1 -Xk+1 ∥ ≤ ∥X k+1 -Xk ∥ ≤ ∥X k -µφ(X k ) -Xk ∥ + α∥gradf (X k )∥ ≤ 2 3 ∥X k -Xk ∥ + α∥gradf (X k )∥.We complete the proof.Proof. It follows from (7) thatEQUATIONwhere the second inequality is from the 2-Lipschitz continuity of P St(d,r) over ŪSt(d,r) ( 1 8 ), the third inequality is due to the facts that X k -Xk ∈ N Xk St(d, r) and ⟨A, B⟩ ≤ 1 2 (∥A∥ 2 + ∥B∥ 2 ) for any A, B ∈ R n×d , and the last inequality comes from∥P T X k St(d,r) (∇φ(X k ))∥ = ∥X k (X ⊤ k X k -I) 2 ∥ ≤ 6∥X k -Xk ∥ 2 .Plugging µ = 1 3 into (17) gives (10) .Proof. Applying [42, Lemma 2] to (9) yieldsEQUATIONThen, summing (10) over k = 0, . . . , K givesEQUATIONDefine c 1 = 148 L2 + 148L + 36 Df + 28 and c 2 = (9 D2 f + 2)(4 Df + 16 L2 + 16L + 4). Then, we haveα(1 -c 1 α) K k=0 ∥gradf (X k )∥ 2 ≤ f ( X0 ) -f ( Xk+1 ) + c 2 .Therefore, for any α≤ 1 2c1 , taking K → ∞ gives ∞ k=0 ∥gradf (X k )∥ 2 < ∞.Then by (11) ,∞ k=0 ∥X k -Xk ∥ 2 < ∞.These lead to (11) . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our research is compatible with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of our work performed Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This meta review is based on the reviews, the authors responses and post-rebuttal discussions, and ultimately my own judgement on the paper. The reviewers are mostly positive about this submission and unanimously recommend it for borderline acceptance or weak acceptance. It was agreed in the reviews that the proposed retraction-free first-order algorithm is simple yet effective for solving a regularized variant of the Stiefel manifold optimization problem, and its application to LoRA for fine-tuning large language models sounds very interesting. In their initial review, the reviewers also expressed a number of concerns regarding the discrepancy between theory and application, setting of algorithm hyper-parameters, and explanation of empirical results as well, which appear to be resolved by the authors responses. However, during the reviewers discussion phase, a severe technical flaw in the proof of Theorem was identified as specified below: 

In the last but one inequality in (17), we believe the authors have used the self-adjointness $\langle grad f(X_k), \nabla \phi(X_k)\rangle = \langle \nabla f(X_k), P_{T_{X_k }St(d,r)}(\nabla \phi(X_k))\rangle$, of which the validness seems highly non-trivial, if not impossible, to show for the iterates $X_k\notin St(d,r)$ produced by the proposed retraction-free algorithm. Unfortunately, this passage was completely left unjustified in the block of comments below the proof. 

Since Theorem 1 lays the foundation on the landing convergence of algorithm, the above mathematical flaw could be fatal to substantially ruin the contribution of this work. In addtion, the entire convergence analysis part, Section 2.3, reads somewhat rough (see Lemma 4 and its proof arguments for another instance). 

Overall, I found that the weaknesses outweight the strengths of this paper. A major improvment for correcting the proof of Theorem 1 is mandatory should the paper be considered for publication, which however is estimated to be out of the scope of a camera-ready update. In view of this, it was eventually decided not to accept this paper in the current form. We do believe this paper, once carefully revised, will get published somewhere, and hope the authors would find the reviews helpful for future submission.

Reviews: review: paper_summary: paper_summary: This paper proposes a new algorithm, Manifold-LoRA, which incorporates the Stiefel manifold constraint to accelerate low-rank adaptation (LoRA) in fine-tuning LLMs. It also provides theoretical and experimental validation for the retraction-free and penalty parameter-free optimization methods.
strength_weakness: strength_weakness: W1: Some experimental results are unclear and not well defined.

W2: Lack of the discussion about limitations of your method.

W3: Some findings of the experiments are hard to understand.
questions: questions: Q1: In Table 1, could you please explain the meaning of m/mm? As I know, it's not an estimation of the performance of NLP models. You also didn't define Acc and Mcc. Also, why not use Acc to evaluate the performance of all the datasets?

Q2: The finding "It can be seen that our method 211 is consistently superior to other baselines." doesn't match the results in Table 1. What makes you conclude this?

Q3: Some findings are not clearly represented. For example, "We conclude that the proposed 228 Manifold-LoRA method achieves a 2x speed-up in training epochs compared to AdamW, while 229 simultaneously improving model performance". 

Q4: The two datasets in the QA task are wildly used, but also weak. Have you tried other datasets? For example, the HotpotQA dataset.
limitations: limitations: Limitation is not enough and is meaningless.
paper_summary: paper_summary: This paper considers solving optimization problems with constraints that have orthonormal columns (i.e. the matrix belongs to the Stiefel manifold). The leading method for solving such problems is the Riemannian optimization. However, Riemannian optimization requires a costly retraction operation. The authors propose to circumvent this by introducing an additional penalty terms that stirs the optimization towards respect the manifold constraints. Indeed, the authors show that with correcting setting of parameters, the optimum will be on the manifold, and the algorithm will find it. The authors advocate that an additional advantage of their algorithm is that we know how to set the parameters for the penalty term, and so their algorithm is parameter-free.

A significant part of the paper is devoted towards motivating the study in terms of low-rank adaption in LLMs, and showing experiments in that vain.
strength_weakness: strength_weakness: (The following were addressed by the authors in the rebuttal)

Major issues (affecting the recommendation):
1) Novelty: Citation [1] considers optimization with constraints on the orthogonal group (i.e. St(n,n)). It seems that the core idea on how to implement retraction free optimization already appears there. The authors mention this in "related work", and say that [1[ does not discuss the r<n case. Inspection of [1] reveals that this is not the entire story. In Sec 3.5 of [1] the case of r<n is discussed briefly, and it is said the results can be extended for that case. However, the authors of [1] are skeptical of the value of this, as they mention that there are fast retraction methods (i.e. Cayley) for the Stiefel manifold. 
-  Setting the penalty parameter: the authors advocate that they give an explicit value for the penalty parameter. And indeed Theorem 1 sets the parameter \mu to 1/3. However, I do not think the situation is so simple. The theorems have the additional assumption that the iterates starts close to the manifold (1/8). This is, of course, easy to achieve - just start on the manifold itself. However, for the proof to work shouldn't all iterates stay inside this bound? This necessitates for the other parameter (step size) to be small enough. And indeed, the theorem requires that the step size be small enough, and does not specify how small. Without looking in detail in the proofs, my guess is that changing the penalty step size (\mu)  affects how close you need to be to the manifold (the value 1/8), which affects how small the step size need to be. In other words, the authors load all the complexity of setting the parameters onto the step size of the main objective. Saying there is  a upper bound on its value , without specifying what that value is. You cannot call this parameter free. 

Another point is that the values of the parameter probably affect convergence rate, though the authors do not discuss this at all.

Minor comments (do not affect the recommendation):
- Line 117: If X is on the manifold, shouldn't \bar{X}, which is the projection of X on the manifold, be exactly X?
- Line 119: "satisfies the restricted secant"
- Line 131: What is U_St (1/8)? Not defined.  Ditto line 136.
- Line 133: If the condition of twice diff is assumed, then state it earlier. 
- Eq (10): \hat{D}_f is not defined. 
- Table 1: Why metrics are changing between columns?
questions: questions: - Please address more carefully why you think you are novel with respect to [1]?
- Why are fast retraction methods not sufficient?
- How setting the parameters affect convergence rate?
limitations: limitations: Nothing to add.
paper_summary: paper_summary: Retraction-free optimization algorithms on the Steifel manifold have been proposed in [1,18,19,41] etc. The motivation is that if the cost of the objective function/gradient evaluation is significantly larger than the evaluation of a retraction, then the retraction-free optimization algorithms show their advantages and efficiency. In particular, for the landing algorithm proposed in [1], the choice of the parameter in the penalty is important and may not be easy to choose. This paper gives an analysis that shows if the parameter is 1/3, the initial point is sufficiently close to the Stiefel manifold, and the step size is chosen sufficiently small, then the algorithm converges linearly to a stationary point. This result gives a concrete value of the parameter. Such a result is further merged into optimization on low-rank matrices and Manifold-LoRA is proposed. Numerical experiments show that the proposed method outperforms the baseline algorithms.
strength_weakness: strength_weakness: (1) Though the value of mu and an upper bound of \|x_0 - \bar{x}_0\| are given concretely, the choice of step size is unknown. Theoretically, the step size needs to be sufficiently small (See Theorem 1). Any theoretical suggestion for the choice of the step size?
(2) Numerical experiments report results of the comparisons. However, the definition of ``result'' is not given. Is the result computational time or classification accuracy or a notion of correctness or something else?
(3) Problem (12) does not remove all the ambiguity. Note that if B \in St(d, r), then B A = B O O^T A = \tilde{B} \tilde{A}, where O is an orthonormal matrix and \tilde{B} = B O is still in St(d, r). Likewise for B \in Ob(d, r). Is it possible to completely remove the ambiguity by considering the quotient manifold? 
(4) Why is the numerical performance of Manifold-LoRA for using Stiefel and Oblique manifold in (12) different? The optimization problem is equivalent in the sense that the local minimizer/stationary point does not change.
questions: questions: The questions are given in the weaknesses section. More questions are given below.
(1) L117, X \in St(d, r) or X \in \mathbb{R}^{d \times r}
(2) What is the percentage of the computational cost of the retraction in the overall computations?
limitations: limitations: The limitations of the paper are discussed in the conclusion section.
paper_summary: paper_summary: The authors propose a retraction-free Riemannian optimization scheme on Stiefel and oblique manifolds to perform parameter-efficient fine-tuning (PEFT) in LoRA style. The proposed approach exploits the theory of landing flows on Stiefel manifolds. Theoretical results demonstrating convergence of this iterative scheme are presented, and complemented by numerical experiments.
strength_weakness: strength_weakness: All the presented theory is developed on a function defined on $St(d,r)$, while LoRA fine-tuning gives rise to an objective function $f(B,A) = L(BA)$, where $B \in St(d,r)$ (or $Ob(d,r)$), and $A \in \mathbb{R}^{r \times m}$. This objective function has to be minimized over $St(d,r) \times \mathbb{R}^{r \times m}$, and the advantages of optimizing on a compact manifold are thus lost.

Unfortunately, this makes the presented theoretical results not directly useful for the practical case under consideration. 
To give a more precise statement, for example in Lemma 3, the constant $\widehat{L}$ would depend on $A$. By the mean value theorem, we would get a bound of the kind

$$
||grad_B f(A,B_1) - grad_B f(A,B_2)|| \leq C(A) ||B_1 - B_2||
$$
 (as noted in equation after line 183 for the Euclidean gradient).

 Since the space in which $A$ resides is not compact, one would need at least a uniform control on $||A_k||$ over the iterations to make the theory interesting for LoRA fine-tuning. 
It is interesting to note, however, that the authors observe exact numerical convergence to the constraint in all cases.
questions: questions: 1. **Concerning the comment made in the "weaknesses" section:** In all the proofs, the Lipschitz constants $L_f$ and $D_f$ depend on $A$, making the convergence analysis not directly applicable in the numerical case under investigation. I would appreciate the authors' comment on this, maybe I am missing something here?

2. I believe however that the observed convergence behavior is not a coincidence, even if the setting is different.
The issue can be addressed by assuming $L(X) \to +\infty$ as $||X|| \to +\infty$. With enough regularity assumed, this ensures that the sublevel sets of $L$ are compact. Given an initial condition $X_0 = B_0A_0$ in the compact set $L^{-1}([-\infty,M])$, the flow $\dot A = -\nabla_A L(BA)$ decreases $L$, thereby remaining within the initial compact set $L^{-1}([-\infty,M])$. This would provide an effective bound for the Lipschitz constant, without compactness of the original domain.

3. I don't see however a way to control $||A||$ uniformly in time for "classification-like" problems, where the global interpolating minima may be off at infinity. 

4. Line 93: the definition of $f$ has a typo. Also, "differentiable" is more common to use.

5. Line 131: Even if it's clear what you mean, $\bar{U}_{St(d,r)}(\frac18)$ was defined nowhere in the main manuscript.

6. The choice of notation is a bit unfortunate in some points. I would personally substitute the loss function $L$ with $\mathcal L$, just to avoid confusion with the Lipschitz constants.

I am inclined to raise my score, provided the authors clarify my doubts and address the mentioned issues (or at least discuss these points in the manuscript).
limitations: limitations: As noted in the "weaknesses" section, I believe there is a delicate point that is not addressed in the manuscript.

Comments: comment: Thank you for the answers. I will raise my score to 5.
Thank you for appreciating our work!
Thank you for the answers. I will raise my score to 6.
The primary concerns you raised focus on the experimental section of our paper. We have clarified the metrics used in our experiments, which align with the standards of the filed. In addition, We have conducted experiments on the dataset as you suggested, and the results suggest the effectiveness of our method. Given that we addressed your primary concerns raised in the review, we would kindly ask you to adjust your review score while taking the discussion into account.
Thank you for your valuable feedback. 

For the Table 1, we will revise it to make it more clear.

We use MCC for CoLA for two main reasons:

1. MCC is a widely recognized metric in the context of LoRA-fine-tuning. **It is considered a standard in LoRA fine-tuning due to the fact that the original LoRA paper[22](in their section 5.1 Table 2) reported MCC scores for the CoLA experiments in the GLUE benchmark**. This is the primary reason.

2. MCC is advantageous in handling imbalanced datasets because it evaluates all elements of the confusion matrix—true positives, true negatives, false positives, and false negatives—providing an unbiased assessment that accuracy (ACC) may not offer in skewed data situations. Moreover, MCC's scale from -1 to +1 provides detailed information about model performance, distinguishing between perfect predictions, random guesses, and complete inaccuracies, unlike ACC's more limited 0 to 1 range.

We are, of course, willing to accommodate your suggestion by including the ACC scores for the CoLA dataset as well.

We will also refine the statement "Our method is consistently superior to other baselines" to be more rigorous based on the results as you suggested.
Thanks for your reply.

1. For Table 1, the design should clearly differentiate between dataset pairs and evaluators. For example, `m/mm` represents two similar datasets and should not be listed on the same line as evaluation metrics like Acc, Mcc, etc. Although Pearson Correlation is commonly used for the STS-B dataset, it is more appropriate to use Accuracy for the CoLA dataset, along with Mcc. Given that Accuracy is used for most datasets, it would be reasonable to present the performance of all methods for CoLA using Acc.

2. In your conclusion, instead of stating, "It can be seen that our method is consistently superior to other baselines," it would be more accurate to acknowledge that while your method outperforms other methods on most datasets, it does not do so on all datasets. Therefore, a more rigorous statement would be to highlight that your method demonstrates superior performance in the majority of cases.
Thank you for your valuable feedback and quick response.

We chose the Stiefel manifold over the Grassmann manifold for two main reasons:

1. The similar geometry between the Stiefel and Grassmann manifolds.
2. The landing theory on the Stiefel manifold is more straightforward to establish. In contrast, the Grassmann manifold requires a more careful process to select representative elements and define the Riemannian metric (e.g., horizontally invariant metric) due to its quotient geometry.

In the future, we plan to explore the landing theory for optimization on the Grassmann manifold and consider its potential uses in the context of LoRA.
I am still concerned about the ambiguity. Using the Grassmann manifold would be more intuitive since it removes all the ambiguity. Moreover, the geometry structure of the Grassmann manifold has been well-explored, and therefore optimization on such a manifold may be accelerated. At this stage, I prefer to raise the score to 5.
Thank you for your valuable feedback. If you have any further comments, we would be happy to address them. Given that we addressed your primary concerns raised in the review, we would kindly ask you to adjust your review score while taking the discussion into account.
The different effectiveness in the numerical experiments in two manifold constraints owes to the following three aspects as we suggest.

- The distinct manifold constraints lead to variations in the search space and in the process of ambiguity elimination.
- Our algorithm employs an adaptive strategy to adjust the learning rate of matrix $A$, based on the observation of unbalanced Lipschitz constants. This approach aims to mitigate the asymmetry in the training process of matrices $A$ and $B$. However, the effectiveness of this adaptive estimation may vary between the two manifolds.
- From a theoretical standpoint, the Stiefel manifold is expected to have a higher ambiguity elimination than the oblique manifold due to its richer geometric structure. Nevertheless, optimization on the Stiefel manifold is inherently more complex, which may result in lower competitive accuracy during testing in some cases.
Thanks for the answer. I agree that manifold reformulation of an optimization problem can improve the numerical performance in the sense of convergence rate since a manifold reformulation removes ambiguity and reduces the dimension of the search space. However, the stationary points and minimizers usually remain the same as the original formulation. What I am curious about is that the effectiveness of the oblique manifold and stiefel manifold reformulations yield different effectiveness in the numerical experiments, which does not make sense to me. Is there any explanation? If yes, I think it is important to add to the paper.
Thank you for your valuable feedback. If you have any further comments, we would be happy to address them. Should there be no additional remarks, we would greatly appreciate it if you could consider raising your score.
1. Though many algorithms do not know the constant step size, a commonly-used practical approach is using a line search algorithm such as backtracking with an appropriate initial step size to guarantee convergence. Is it possible to give a computable initial step size with a backtracking algorithm such that the result of Thm 1 holds. If it is true, then the proposed algorithm is more practical.

    **Reply:** Note that Theorem 1 differs from the standard line search-based convergence in two key aspects:
    - The decrease in each step is applied not directly to the iterates, but to their projected versions.
    - Each step may not always decrease the objective function value, but the accumulated effect over several steps will reduce the objective value, assuming the step size is small enough.

    While this type of convergence analysis is common in optimization areas, such as, distributed optimization and min-max optimization, designing a line search method based on this approach is not straightforward. Exact convergence with a constant step size (even if the step size is unknown) is generally considered crucial, as opposed to convergence to a neighborhood or within a certain horizon.   
    However, it would be interesting to explore the development of a new Lyapunov function to establish a one-step decrease directly on the iterates, which could lead to a line search method suitable for this setting in future work.

2. I agree if only the B-subproblem is considered, then the stationary point of the subproblem is different for different constraints on B. However, Problem (12) considers $L(BA)$ with an arbitrary $A$. It follows that the stationary points of L should be the same. That is what I am concerned.

    **Reply:** Thank you for your further clarification. By treating $BA$ as a single entity, the two problems become equivalent. Alternatively, different manifold constraints on  $B$  correspond to different parameterizations of the low-rank matrix $W$.

    Although adding different constraints on $B$ (or $A$ ) may not change the stationary point to which the algorithm converges, it accelerates the convergence process by leveraging the manifold geometry, enabling more efficient movement along the manifold. This is why manifold optimization methods can be more powerful than traditional constrained optimization approaches. A similar acceleration is observed in low-rank matrix completion problems, where methods with a Grassmann manifold constraint converge faster, even though they ultimately reach the same solution with zero recovery error. Please refer to [MPC 2012, NeurIPS 2011, IEEE TIT 2012, IEEE TIT 2009] for the details. 

    - [MPC 2012] Wen Zaiwen, Yin Wotao, Zhang Yin; Solving A Low-Rank Factorization Model for Matrix Completion by A Nonlinear Successive Over-Relaxation Algorithm; Mathematical Programming Computation; 4 (2012), 333-361.
    - [NeurIPS 2011] Boumal, Nicolas, and Pierre-antoine Absil. "RTRMC: A Riemannian trust-region method for low-rank matrix completion." Advances in neural information processing systems 24 (2011).
    - [IEEE TIT 2012] Dai W, Kerman E, Milenkovic O. A geometric approach to low-rank matrix completion[J]. IEEE Transactions on Information Theory, 2012, 58(1): 237-247.
    - [IEEE TIT 2009] W. Dai and O. Milenkovic, "Subspace Pursuit for Compressive Sensing Signal Reconstruction," in IEEE Transactions on Information Theory, vol. 55, no. 5, pp. 2230-2249, May 2009, doi: 10.1109/TIT.2009.2016006.
Thank the authors for the replay. 
1) Though many algorithms do not know the constant step size, a commonly-used practical approach is using a line search algorithm such as backtracking with an appropriate initial step size to guarantee convergence. Is it possible to give a computable initial step size with a backtracking algorithm such that the result of Thm 1 holds. If it is true, then the proposed algorithm is more practical.
4) I agree if only the B-subproblem is considered, then the stationary point of the subproblem is different for different constraints on B. However, Problem (12) considers L(BA) with an arbitrary A. It follows that the stationary points of L should be the same. That is what I am concerned.
1, 2: I agree that this issue is not specific to your proposed algorithm; it was not my intention to make it sound like a peculiarity of your work. However, my impression of the manuscript is that there is a "distance" between the theory and the test cases under consideration. Of course, a minimal set of assumptions is always the best, but I was expecting at least a separate result that would, in some way, also cover the cases presented in the experimental section, even if this requires making additional assumptions.

**Reply:** The global Lipschitz continuity may not hold in certain instances. To enhance the adaptability of our convergence results for the applications detailed in the experimental section, we will add a paragraph to discuss more practical alternatives, such as coercivity, differentiability, and local gradient Lipschitz continuity [SIOPT 2023], aiming to clarify how these properties can be met in terms of the application presented in the experimental section.

[SIOPT 2023] Xiaoxi Jia, Christian Kanzow, and Patrick Mehlitz. "Convergence Analysis of the Proximal Gradient Method in the Presence of the Kurdyka–Łojasiewicz Property Without Global Lipschitz Assumptions." SIAM Journal on Optimization 33, no. 4 (2023): 3038-3056.

3: As mentioned before, my suggestion was to try to integrate your theoretical results more into the experimental setting. For example, in classification tasks (often of interest for your application), my concern was that the loss is of exponential type, and it is known that, for instance, on separable data, $||BA|| \rightarrow +\infty$ [1]. This is why I was skeptical about convergence guarantees in some cases of interest for your work (some of the GLUE Debertav3 tasks fit this setting for example).

[1] M.S. Nacson et al., "Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate", AISTATS 2019.

**Reply:** Thanks for your further clarification.

In [1], the authors comment that replacing the global Lipschitz smoothness requirement with local Lipschitz smoothness and an appropriately small step size could be sufficient to keep iterates within a region where the Lipschitz constants remain uniformly bounded for the exponential-type loss function. Specifically, the footnote in [JMLR 2018, Assumption 3], a companion paper to [1], states: "The exponential loss does not possess a global $\beta$ smoothness parameter. However, initialization with $\eta < \frac{1}{\mathcal{L}(w_0)}$ ensures that the gradient descent iterates maintain bounded local smoothness."

Furthermore, there is a growing body of research within the optimization community that focuses on algorithm convergence using local rather than global Lipschitz smoothness. This approach often depends on additional assumptions to ensure favorable properties of the local Lipschitz constant during iterations, such as Assumption 3.2 in [JOTA 2022], where the objective function is both bounded from below and lower-bounded by an affine function. For recent advancements, please refer to [SIOPT 2023]. 

[JOTA 2022] C. Kanzow and P. Mehlitz. Convergence properties of monotone and nonmonotone proximal gradient methods revisited. Journal of Optimization Theory and Applications, 195(2):624–646, 2022.

[JMLR 2018] Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S. Srebro, N. . The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70), 1-57, 2018.

In any case, I would like to increase my score, provided that you add a small paragraph discussing additional necessary assumptions for a broader class of problems (on the line of what you did in your answer).

**Reply:** Thank you for your insightful comment. We will definitely add a paragraph to discuss the potential lack of global Lipschitz smoothness and how to establish more practical assumptions for a broader class of problems, particularly in terms of the considered numerical applications.
I wish, first of all, to thank the authors for their rebuttal.
Regarding their answers:

**1, 2**: I agree that this issue is not specific to your proposed algorithm, it was not my intention to make it sound like a peculiarity of your work. 
However, my impression on the manuscript is that there is a "distance" between the theory and the test cases under consideration. Of course a minimal set of assumptions is always the best, but I was expecting at least a separate result that would, in some way, also cover the cases presented in the experimental section, even if this requires making additional assumptions.

**3**:  As mentioned before, my suggestion was to try to integrate your theoretical results more into the experimental setting. For example, in classification tasks (often of interest for your application), my concern was that the loss is of exponential type, and it is known that, for instance, on separable data, $||BA|| \rightarrow +\infty$ [1]. 
This is why I was skeptical about convergence guarantees in some cases of interest for your work (some of the GLUE Debertav3 tasks fit this setting for example).

[1] M.S. Nacson et al., "Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate", AISTATS 2019.

In any case, I would like to increase my score, provided that you add a small paragraph discussing additional necessary assumptions for a broader class of problems (on the line of what you did in your answer).