Title: Retraction-free optimization over the Stiefel manifold with application to the LoRA fine-tuning
Venue: NeurIPS 2024
Accepted: no
Keywords: Manfold

Landing

LoRA

fine-tuning
Abstract: Optimization over the Stiefel manifold has played a significant role in various machine learning tasks. Many existing algorithms either use the retraction operator to keep each iterate staying on the manifold, or solve an unconstrained quadratic penalized problem. The retraction operator in the former corresponds to orthonormalization of matrices and can be computationally costly for large-scale matrices. The latter approach usually equips with an unknown large penalty parameter. To address the above issues, we propose a retraction-free and penalty parameter-free algorithm, which lands on the manifold. A key component of the analysis is the convex-like property of the quadratic penalty of the Stiefel manifold, which enables us to explicitly characterize the penalty parameter. As an application, we introduce a new algorithm, Manifold-LoRA, which employs the landing technique and a carefully designed step size strategy to accelerate low-rank adaptation (LoRA) in fine-tuning large language models. Numerical experiments on the benchmark datasets demonstrate the efficiency of our proposed method.

1

Abstract

Optimization over the Stiefel manifold has played a significant role in various machine learning tasks. Many existing algorithms either use the retraction operator to keep each iterate staying on the manifold, or solve an unconstrained quadratic penalized problem. The retraction operator in the former corresponds to orthonormalization of matrices and can be computationally costly for large-scale matrices. The latter approach usually equips with an unknown large penalty parameter. To address the above issues, we propose a retraction-free and penalty parameter-free algorithm, which lands on the manifold. A key component of the analysis is the convex-like property of the quadratic penalty of the Stiefel manifold, which enables us to explicitly characterize the penalty parameter. As an application, we introduce a new algorithm, Manifold-LoRA, which employs the landing technique and a carefully designed step size strategy to accelerate low-rank adaptation (LoRA) in fine-tuning large language models. Numerical experiments on the benchmark datasets demonstrate the efficiency of our proposed method.

1 Introduction

Optimization over the Stiefel manifold has attracted considerable attention in the context of machine learning, e.g., RNN [3], batch normalization [10], and distributionally robust optimization [8]. The mathematical formulation of this class of problems is:

where r ≤ d and f : R d × r → R is a continuously differentiable function. The most popular methods for solving (1) are retraction-based algorithms, which have been extensively studied in the context of manifold optimization [2, 23, 6]. Recently, to alleviate the possible computational burden of the retraction operator, some retraction-free methods have been developed in [19, 18, 41, 1]. The ideas in these papers are based on a combination of the manifold geometry and a penalty function for the manifold constraint, which involves an unknown but sufficiently large penalty parameter. For largescale machine learning applications, retraction-free algorithms are preferred. However, designing retraction-free algorithms with a known penalty parameter for solving (1) remains a challenge.

Another motivation for studying retraction-free methods arises from its application in the fine-tuning of large language models (LLMs). Recently, LLMs have revolutionized the field of natural language processing (NLP), achieving unprecedented performance across various applications [33, 32]. To tailor pretrained LLMs for specific downstream tasks, the most common approach is full fine-tuning, which requires prohibitively large computational resources due to the need to adapt all model weights, hindering the deployment of large models. As a result, parameter-efficient fine-tuning (PEFT) has gained widespread attention for requiring few trainable parameters while delivering comparable

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

or even superior results to full fine-tuning. This paradigm involves inserting learnable modules or designating only a small portion of weights as trainable, keeping the main model frozen [21, 26, 44]. Among fine-tuning methods, low-rank adaptation (LoRA) [22] has become the de facto standard among parameter-efficient fine-tuning techniques. It assumes that the change in weights lies in a 'low intrinsic dimension', thereby modelling the update ∆ W ∈ R d × m by two low-rank (not greater than a small integer r ) matrices A ∈ R r × m and B ∈ R d × r , i.e., ∆ W = BA . Since r ≪ d , the requirements on both storage and computation are significantly reduced. Due to its decompositional nature, there is redundancy in the representation of ∆ W . Traditional optimization methods for LoRA do not exploit this redundancy, which consequently undermines model performance. Instead, we reformulate LoRA fine-tuning as an optimization problem over the product of Stiefel manifolds and Euclidean spaces. Therefore, we propose an algorithmic framework called Manifold-LoRA to accelerate the fine-tuning process and enhance model performance. Moreover, by exploiting projected gradients and incorporating a parameter-free penalty, the overhead that our method incurs is relatively negligible. Our contributions are as follows:

We first prove the existence of explicit choice for the penalty parameter by establishing a strong convexity-like condition of the nonconvex penalty problem associated with the Stiefel manifold constraint. Furthermore, for the given penalty parameter, under mild conditions, we prove that the iterates of our proposed retraction-free gradient descent method eventually land on the Stiefel manifold and achieve the optimality of (1).

Building upon the established landing theory of retraction-free and penalty parameter-free method and the AdamW framework, we proposed a new method, Manifold-LoRA, which employs a carefully designed step size strategy to accelerate the training process of finetuning. Compared with the conventional AdamW method, we use the penalized gradient instead of the usual gradient, and the computational overhead is negligible.

Numerical experiments are conducted on a wide range of NLP tasks, demonstrating the efficiency of our algorithm. Specifically, compared to the vanilla LoRA, our Manifold-LoRA with half the trainable parameters not only delivers fast convergence but also yields improved generalization. In particular, Our method converges twice as fast as baseline methods on several typical datasets, including the SQuAD 2.0 dataset and the CoLA dataset.

1.1 Related Work

Optimization over the Stiefel manifold. Optimization over the Stiefel manifold has attracted lots of attention due to its broad applications. Through the use of retraction, known as the generalization of the exponential map, the Riemannian gradient descent is proposed [2, 6, 23], where all iterates lie on the manifold. When such retraction is computationally costly, the authors [19] develop a retractionfree algorithm based on the augmented Lagrangian method. More recently, by defining the constraint dissolving operator and adding a sufficiently large penalty term, the authors [41] convert the manifold constrained problem (1) into an unconstrained problem and then apply unconstrained optimization algorithms. In [1], motivated by the convergence of the Oja's flow, a landing flow, consisting of the projected gradient and the gradient of the penalty function, is developed to retraction-free method for the squared Stiefel manifold, i.e., d = r . All of these methods rely on an unknown penalty parameter to ensure the convergence. This motivates us to design penalty parameter-free algorithms, which could significantly reduce the need for tuning parameters in practical implementations.

LoRA. There are numerous variants of LoRA aiming to improve performance or reduce memory usage. AdaLoRA [46], a well-known successor, introduces the idea of adaptively adjusting the rank of different layers by incorporating an additional vector g to serve as the diagonal of a singular value matrix. This approach leverages a revised sensitivity-based importance measure to decide whether to disable entries in vector g and in matrices A and B . A similar work, SoRA [15], adopts the same model architecture as AdaLoRA, but proposes a different way to update vector g after training. This update rule is the proximal gradient of L 1 loss, acting as a post-pruning method. Additionally, a recently emerged method called VeRA [25] significantly reduces memory overhead while maintaining competitive performance. Based on the idea that networks with random initialization contain subnetworks that are near-optimal or optimal [17], VeRA only uses two frozen low-rank matrices shared by all layers, training scaling vectors unique to each layer. Although LoRA has gained significant popularity and various variants have been developed, the potential for efficient training through leveraging the manifold geometry to reduce redundancy has not been well-explored.

1.2 Notation 89

For a matrix X ∈ R d × r , we use ∥ X ∥ to denote its Frobenius norm. For a squared matrix A ∈ R d × d , 90 we define sym( A ) = A + A ⊤ 2 and use diag( A ) ∈ R d to denote its diagonal part. For two matrices 91 X,Y ∈ R d × r , we use ⟨ X,Y ⟩ := ∑ d i =1 ∑ r j =1 X ij Y ij to denote their Euclidean inner product. For a 92 differential function f : R d × r → d , we use ∇ f ( X ) to denote its Euclidean gradient at X . 93

94

95

2 Retraction-free and penalty parameter-free optimization over the Stiefel manifold

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

We have

X

k

}

In this section, we focus on the design of retraction-free and penalty parameter-free algorithms for solving problem (1). We will first present the retraction-free algorithm and then show how the penalty parameter can be explicitly determined by characterizing the landscape of the penalty function.

2.1 Retraction-free algorithms

Inspired by the retraction-free algorithms [19, 41, 1], we consider the following retraction-free gradient descent method for problem (1):

where α, µ > 0 are step sizes and the projected gradient grad f ( X k ) := ∇ f ( X k ) -X k sym( X ⊤ k ∇ f ( X k )) . Note that the tangent space of St( d, r ) is T X k St( d, r ) := { ξ ∈ R d × r : X ⊤ k ξ + ξ ⊤ X k = 0 } . Then, for X k ∈ St( d, r ) , grad f ( X k ) is the projection of the Euclidean gradient ∇ f ( X k ) to the tangent space, i.e., grad f ( X k ) = P T Xk St( d,r ) ( ∇ f ( X k )) . Note that the term X k ( X ⊤ k X k -I d ) is exactly the gradient of the following quadratic penalty function

As will be shown in our theorem, the use of the projected gradient is essential for landing on the manifold. This differs with the usual penalty method, which optimizes f ( X ) + µφ ( X ) using the update X k +1 = X k -α ∇ f ( X k ) -µX k ( X ⊤ k X k -I d ) , needs µ →∞ to guarantee the feasibility.

2.2 Explicit choice for the penalty parameter

It is known that a large penalty parameter yields better feasibility [29, Chapter 17]. To make the iterative scheme (2) be penalty parameter-free, we need a careful investigation on the landscape of the following optimization problem:

It can be easily verified that problem (3) is nonconvex and its the optimal solution set is St( d, r ) . The key of obtaining an explicit formula of µ is to establish certain strong convexity-type inequality and show the gradient descent method with step size µ has linear convergence.

For any X ∈ St( d, r ) , let us denote ¯ X := P St( d,r ) ( X ) . Let X = USV ⊤ be the singular value decomposition with orthogonal matrices U ∈ R d × r , V ∈ R d × d and diagonal matrix S ∈ R d × d , then ¯ X = UV ⊤ . Building on these notations, we demonstrate that problem (3) satisfies the restrict secant inequality (RSI) [45], which serves as an alternative to the strong convexity in the linear convergence analysis of gradient-type methods.

With the above RSI, we have the linear convergence of the gradient descent update for (3), i.e.,

be generated by

(5)

with

µ

=

1

3

. Suppose that

∥

X

0

-

¯

X

0

1

8

.

The proofs of Lemmas 1 and 2 can be found in Appendix B. 126

Let the sequence

{

127

128

129

130

131

132

133

134

135

136

137

138

2.3 Landing on the Stiefel manifold

Building on the established linear convergence of gradient descent for problem (3), we are now able to show that the iterates generated by (2) will land on the Stiefel manifold eventually, and the limiting point is a stationary point of (1), i.e., grad f ( X ∞ ) = 0 .

Let us start with the Lipschitz continuity of grad f ( X ) . For any X ∈ ¯ U St( d,r ) ( 1 8 ) , we define P T X St( d,r ) ( U ) = U -X sym( X ⊤ U ) for U ∈ R d × r . We first have the following quadratic upper bound on f from its twice differentiability and the compactness of St( d, r ) .

Lemma 3. There exists a constant L > 0 such that for any X,Y ∈ St( d, r ) , the following quadratic upper bound holds:

In addition, there exists a constant ˆ L > 0 such that for any X ∈ St( d, r ) , Y ∈ U M ( 1 8 ) ,

By the linear convergence result in Lemma 2, we have the following bound on the feasibility error. Lemma 4. Let { X k } be the sequence generated by (2) with µ = 1 3 and ∥ X 0 -¯ X 0 ∥ ≤ 1 8 . We have

The following one-step descent lemma on f is crucial in establishing the convergence. 139

Lemma 5. Let { X k } be the sequence generated by (2) with µ = 1 3 and ∥ X 0 -¯ X 0 ∥ ≤ 1 8 . We have 140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

From the above lemma, the one-step descrease on f is related to both the gradient norm of f and the feasibility error. In terms of convergence, we need both grad f ( X k ) and ∥ X ⊤ k X k -I ∥ converge to 0. The following theorem demonstrates that the retraction-free and penalty parameter-free update (2) converges.

Theorem 1. Let { X k } be the sequence generated by (2) with µ = 1 3 and ∥ X 0 -¯ X 0 ∥ ≤ 1 8 . If the step size α < 1 2 c 1 for some c 1 large enough, then we have

The proofs of the above lemmas and theorem are presented in Appendix B.

3 Accelerate LoRA fine-tuning with landing

In this section, we will first clarify where the Stiefel manifold constraint comes from in the LoRA fine-tuning. Then, we will apply the above developed retraction-free and penalty parameter-free method to enhance LoRA fine-tuning.

3.1 Manifold optimization formulation of LoRA fine-tuning

In neural networks, the dense layers perform matrix multiplication, and the weight matrices in these layers usually have a full rank. However, when adapting to a specific task, pre-trained language models have been shown to have a low intrinsic dimension, allowing them to learn efficiently even with a random projection to a smaller subspace. One possible drawback in the current LoRA fine-tuning framework is that the low-rank decomposition ∆ W into product BA is not unique. Specifically, for any invertible matrix C , it holds that BA = ( BC )( C -1 A ) . Note that BC shares the same

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

column space with B . This suggests us optimizing the subspace generated by B instead of B itself. Numerous studies in the field of low-rank optimization, e.g., [7, 13, 12], investigate the manifold geometry of the low-rank decomposition and develop efficient algorithms. However, such geometry has not been explored in the LoRA fine-tuning.

To address such redundancy (i.e., the non-uniqueness of BA representations), we regard B as the basis through the manifold constraint and A as the coordinate of ∆ W under B . Hence, the optimization problem can be formulated as

where Ob( d, r ) := { B ∈ R d × r : diag( B ⊤ B ) = 1 } . Compared to the Stiefel manifold St( d, r ) , the oblique manifold Ob( d, r ) necessitates that the matrix B has unit norms in its columns, without imposing requirements for orthogonality between the columns. Problem (12) is an optimization problem over the product of manifolds and Euclidean spaces.

3.2 Manifold-LoRA

The retraction-free method is well-suited to address (12), simultaneously minimizing the loss function L ( BA ) and constraint violation. To control the constraint violation, we use the quadratic penalties R s ( B ) := ∥ B ⊤ B -I ∥ 2 and R o ( B ) := ∥ diag( B ⊤ B ) -1 ∥ 2 for the Stiefel manifold and oblique manifold, respectively. As shown in the landing theory in Section 2, we shall use the projected gradient of the loss part instead of the Euclidean gradient. For the Stiefel manifold and the oblique manifold, the respective projected gradients are

and

where sym( X ) := ( X + X ⊤ ) / 2 . Thus, the gradients of our retraction-free method for A and B are ∇ A L ( BA ) and grad B L ( BA ) + µ ∇ R s ( B )( or ∇ R o ( B )) .

Note that B and A represent the basis and the coordinate of ∆ W , respectively. This results in different magnitudes and different Lipschitz constants of their gradient function. In fact, let X = BA . It follows

Then,

where L g is the Lipschitz constant of ∇ X L ( X ) and ∥ · ∥ 2 represent the matrix ℓ 2 norm (i.e., the largest singular value). Note that the step size generally should be propositional to the reciprocal of Lipschitz constant for the gradient type algorithms [29, 5]. Hence, we schedule the learning rates for the two matrices based on their respective ℓ 2 norms. Having prepared the above, we incorporate the AdamW optimizer [28] with our manifold-accelerated technique to enhance the LoRA fine-tuning, as presented in Algorithm 1.

4 Experiments

In this section, we delve into the experimental results and their detailed analysis. This discussion is structured around two principal areas: (1) the performance gain compared to other mainstream finetuning methods and accelerated convergence achieved through our manifold-constrained optimization approach; (2) the convergence of matrix B onto the manifold, illustrated by the heat map of B ⊤ B .

Baselines We compare our approach against several baseline methods, including full fine-tuning, Adapter [21], BitFit [44] and LoRA [22]. The variants of the Adapter method are excluded from the baselines, as their performance are relatively similar.

Implementation Details Our code is based on Pytorch [31], Huggingface Transformers [40] and an open-source plug-and-play library for parameter-efficient fine-tuning opendelta [24]. The bottleneck dimension for the Adapter is set to 16 or 32, ensuring that the number of trainable parameters aligns

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

Algorithm 1: Manifold-LoRA

closely with that of the LoRA method and the new layers are inserted into the attention layer and feed-forward layer. The update of LoRA is scaled by a hyper-parameter α . This value is typically left unmodified, as it is usually set as 16 or 32 and never tuned [22, 43]. The exponential moving average parameters β 1 and β 2 of AdamW [27] are set to their default values of 0.9 and 0.999, respectively. All the experiments are conducted on NVIDIA A800 GPUs. More details are presented in Appendix C.

4.1 Natural language understanding

We first evaluate our backbone model DeBERTaV3-base [20] on GLUE [37] benchmark containing nine sub datasets, including MNLI [39], SST-2 [36], CoLA [38], QQP [37], QNLI [35], RTE [4], MRPC [16], and STS-B [37].

Experimental results of the GLUE dataset are recorded in Table 1. It can be seen that our method is consistently superior to other baselines. Notably, for RTE and STS-B datasets, both sphereconstrained (i.e., oblique manifold-constrained) and Stiefel-constrained have an obvious performance gain even with only half the trainable parameters compared to the LoRA baseline, i.e., Sphere r =8 and Stiefel r =8 beat LoRA r =16 . In addition, with the help of manifold geometry, the fine-tuning process can be significantly accelerated compared to the vanilla AdamW optimizer, achieving a lower training loss, as shown in Figure 1. Particularly on the CoLA dataset presented in Figure 1a, our approach achieves the same training loss as the standard Adam optimizer but requires nearly half the number of epochs.

4.2 Question Answering

We conduct an evaluation on two question answering datasets: SQuAD v1.1 [35] and SQuADv2.0 [34]. Manifold-LoRA is used to fine-tune DeBERTaV3-base for these tasks, which are treated as sequence labeling problems predicting the probability of each token as the start or end of an answer span.

The main experimental results are presented in Table 2. For LoRA and our algorithms, new layers are inserted into W q , W k , W v , W o , F C 1 , F C 2 . Notably, both manifold-regularized LoRA variants consistently outperform all fine-tuning methods. Additionally, we plot the training loss, evaluation

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

exact match, and evaluation F1 scores against epochs in Figure 2. We conclude that the proposed Manifold-LoRA method achieves a 2x speed-up in training epochs compared to AdamW, while simultaneously improving model performance. We also illustrate the heat map of B ⊤ B in Figure 3, which indicates that the matrix B lands on the manifold eventually. This supports our assertion that landing on manifold enhances the performance of LoRA.

4.3 Natural Language Generation

The E2E NLG Challenge[30], as introduced by Novikova, provides a dataset for training end-to-end, data-driven natural language generation systems, widely used in data-to-text evaluations. The E2E dataset comprises approximately 42,000 training examples, 4,600 validation examples, and 4,600 test examples, all from the restaurant domain. We test our method on the E2E dataset using GPT-2 Medium and Large models, following the experimental setup outlined by LoRA [22]. For LoRA, we set the hyperparameters to match those specified in the original paper.

The results from the E2E dataset are recorded in Table 3, where we focus on comparing LoRA and Manifold-LoRA. The results clearly indicate that our proposed algorithm outperforms the established baselines. Also, as shown in Figure 4, the matrix B resides on the manifold even at the early training stage, validating the feasibility of our method.

0

5

10

15

20

25

Epoch

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Loss

Sphere

Stefiel

(a) CoLA train loss

0

1

2

3

4

5

Epoch

0.150

0.175

0.200

0.225

0.250

0.275

0.300

Loss

Sphere

Stefiel

(b) QQP train loss

0

5

10

15

20

25

Epoch

0.0

0.1

0.2

0.3

0.4

0.5

Loss

Sphere

Stefiel

(c) STSB train loss

5

10

15

20

25

Epoch

0.60

0.62

0.64

0.66

0.68

0.70

0.72

Eval Matthews Correlation

Sphere

Stefiel

(d)

CoLA evaluation

matthews

0

1

2

3

4

5

Epoch

0.86

0.87

0.88

0.89

0.90

0.91

0.92

Sphere

Stefiel

(e) QQP evaluation accuracy

5

10

15

20

25

Epoch

0.87

0.88

0.89

0.90

0.91

0.92

Sphere

Stefiel

(f) STSB evaluation pearson

correlation

Figure 1: The figures illustrate that both sphere constrained and Stiefel constrained manifold-LoRA achieve a faster convergence rate and attain a lower training loss within same optimization steps compared to LoRA method on three distinct datasets CoLA, QQP, STSB.

1

2

3

4

Epoch

0.5

1.0

1.5

2.0

2.5

Train Loss

Sphere

Stefiel

(a) SQuADv2.0 Train Loss

1

2

3

4

Epoch

60

65

70

75

80

85

Exact Match

Sphere

Stefiel

(b) SQuADv2.0 Eval Exact Match

1

2

3

4

Epoch

65

70

75

80

85

90

Eval F1

Sphere

Stefiel

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.value_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.attention.output

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.intermediate.dense

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.output.dense

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.query_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.key_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.value_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.attention.output

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.value_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.attention.output

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.intermediate.dense

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.2.output.dense

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.query_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.key_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.value_proj

0

1

2

3

4

5

6

7

7 6 5 4 3 2 1 0

layer.3.attention.output

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

0

1

2

3

0

1

2

3

module.transformer.h.0.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

module.transformer.h.3.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

module.transformer.h.6.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

module.transformer.h.7.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

module.transformer.h.8.attn.c_attn.lora_B

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0

1

2

3

0

1

2

3

module.transformer.h.0.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

module.transformer.h.3.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

module.transformer.h.6.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

module.transformer.h.7.attn.c_attn.lora_B

0

1

2

3

0

1

2

3

module.transformer.h.8.attn.c_attn.lora_B

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

5 Conclusion

Optimization over the Stiefel manifold has been widely used in machine learning tasks. In this work, we develop a retraction-free and penalty parameter-free gradient method, and prove that the generated iterates eventually land on the manifold and achieve the optimality simultaneously. We then apply this landing theory to avoid the possible redundancy of LoRA fine-tuning in LLMs. Specifically, we reformulate the LoRA fine-tuning as an optimization problem over the Stiefel manifold, and propose a new algorithm, Manifold-LoRA, which incorporates a careful analysis of step sizes to enable fast training using the landing properties. Extensive experimental results demonstrate that our approach not only accelerates the training process but also yields significant performance improvements.

Our study suggests several potential directions for future research. Although the established landing theory focuses on the Stiefel manifold, extending this theory to general manifolds is one potential direction. Additionally, evaluating the performance of Manifold-LoRA on LLMs with billions of parameters would be valuable. Due to the heterogeneity of different layers, incorporating adaptive ranks for ∆ W across different layers is another possible direction. This may be achievable by adding sparsity regularization to the coordinate matrix A .

258

259

260

261

References

Pierre Ablin and Gabriel Peyr´ e. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics , pages 5636-5657. PMLR, 2022.

P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix 262 manifolds . Princeton University Press, 2008. 263

Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. 264 In International conference on machine learning , pages 1120-1128. PMLR, 2016. 265

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing 266 textual entailment challenge. TAC , 7(8):1, 2009. 267

268

269

L´ eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

270

271

272

273

Nicolas Boumal. An introduction to optimization on smooth manifolds . Cambridge University Press, 2023.

Nicolas Boumal and Pierre-antoine Absil. Rtrmc: A riemannian trust-region method for low-rank matrix completion. Advances in neural information processing systems , 24, 2011.

Robert S Chen, Brendan Lucier, Yaron Singer, and Vasilis Syrgkanis. Robust optimization for 274 non-convex objectives. Advances in Neural Information Processing Systems , 30, 2017. 275

Shixiang Chen, Alfredo Garcia, Mingyi Hong, and Shahin Shahrampour. Decentralized Rie276 mannian gradient descent on the Stiefel manifold. In International Conference on Machine 277 Learning , pages 1594-1605. PMLR, 2021. 278

Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. Advances in 279 Neural Information Processing Systems , 30, 2017. 280

Francis H Clarke, Ronald J Stern, and Peter R Wolenski. Proximal smoothness and the lower-C2 281 property. Journal of Convex Analysis , 2(1-2):117-144, 1995. 282

Wei Dai, Ely Kerman, and Olgica Milenkovic. A geometric approach to low-rank matrix 283 completion. IEEE Transactions on Information Theory , 58(1):237-247, 2012. 284

Wei Dai, Olgica Milenkovic, and Ely Kerman. Subspace evolution and transfer (set) for low-rank 285 matrix completion. IEEE Transactions on Signal Processing , 59(7):3120-3132, 2011. 286

Kangkang Deng and Jiang Hu. Decentralized projected riemannian gradient method for smooth 287 optimization on compact submanifolds. arXiv preprint arXiv:2304.08241 , 2023. 288

Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and 289 Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint 290 arXiv:2311.11696 , 2023. 291

Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. 292 In Third international workshop on paraphrasing (IWP2005) , 2005. 293

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable 294 neural networks. arXiv preprint arXiv:1803.03635 , 2018. 295

Bin Gao, Guanghui Hu, Yang Kuang, and Xin Liu. An orthogonalization-free parallelizable 296 framework for all-electron calculations in density functional theory. SIAM Journal on Scientific 297 Computing , 44(3):B723-B745, 2022. 298

Bin Gao, Xin Liu, Xiaojun Chen, and Ya-xiang Yuan. A new first-order algorithmic framework 299 for optimization problems with orthogonality constraints. SIAM Journal on Optimization , 300 28(1):302-332, 2018. 301

Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using 302 electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint 303 arXiv:2111.09543 , 2021. 304

305

306

307

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Jiang Hu, Xin Liu, Zai-Wen Wen, and Ya-Xiang Yuan. A brief introduction to manifold optimization. Journal of the Operations Research Society of China , 8:199-248, 2020.

Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, and Maosong Sun. Opendelta: A plug-and-play library for parameter-efficient adaptation of pre-trained models. arXiv preprint arXiv:2307.03084 , 2023.

Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454 , 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Jekaterina Novikova, Ondˇ rej Duˇ sek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.09254 , 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 , 2023.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 , 2018.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics.

Nachuan Xiao, Xin Liu, and Kim-Chuan Toh. Dissolving constraints for riemannian optimization. Mathematics of Operations Research , 49(1):366-397, 2024.

Jinming Xu, Shanying Zhu, Yeng Chai Soh, and Lihua Xie. Augmented distributed gradient methods for multi-agent optimization under uncoordinated constant stepsizes. In 2015 54th IEEE Conference on Decision and Control (CDC) , pages 2055-2060. IEEE, 2015.

Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 , 2020.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

Hui Zhang and Wotao Yin. Gradient methods for convex minimization: better rates under weaker conditions. arXiv preprint arXiv:1303.4645 , 2013.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations , 2023.

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

Hence, we have

where the first inequality is from ¯ X k +1 = argmin X ∈ St( d,r ) ∥ X -X k ∥ 2 and the second inequality is 396 due to Lemma 1 and (16). 397

398

Proof of Lemma 3

A Proximal smoothness

The notion of proximal smoothness, as introduced by [11], refers to the characteristic of a closed set whereby the nearest-point projection becomes a singleton when the point is in close enough to the set. This property facilitates algorithmic and theoretical advancements by endowing nonconvex sets with convex-like structural attributes. Specifically, for any positive real number γ , we define the γ -tube around M as U M ( γ ) := { x : dist( x, M ) < γ } . We say a closed set M is γ -proximally smooth if the projection operator P M ( x ) := argmin y ∈M ∥ y -x ∥ 2 is a singleton whenever x ∈ U M ( γ ) .

Obviously, any closed and convex set is proximally smooth for arbitrary γ ∈ (0 , ∞ ) . According to [11, Corollary 4.6], a closed set M is convex if and only if it is proximally smooth with a radius of γ for every γ > 0 . It is worth noting that the Stiefel manifold is 1 -proximally smooth. By following the proof in [11, Theorem 4.8],

where ¯ U St( d,r ) ( 1 2 ) := { x : dist( x, St( d, r )) ≤ 1 2 } is the closure of U St( d,r ) ( 1 2 ) . It is worth noting that for any closed convex set M⊂ R d × r , the projection operator P M is 1-Lipschitz continuous over R d × r .

B Proofs

Proof of Lemma 1

Proof. Denote the SVD of X by X = USV ⊤ . Then, it holds that dist( X, St( d, r )) = ∥ X -¯ X ∥ = ∥ s -1 ∥ 2 , where s = diag( S ) . Furthermore, we have

where the last inequality is from min i s i ( s i +1) ≥ 105 64 ≥ 3 2 . This completes the proof.

Proof of Lemma 2

Proof. Assume that ∥ X k -¯ X k ∥ ≤ 1 8 . Denote the SVD of X k by USV ⊤ . Let s = diag( S ) . Then, we have 7 8 ≤ s i ≤ 9 8 for any i . This implies

Proof. Due to the twice differentiability of f and the compactness of St( d, r ) , the inequality (7) 399 directly follows from [9, Lemma 2.4] and [14, Lemma 4.2], where L := L f + D f with L f being the 400 Lipschitz constant of ∇ f ( X ) over St( d, r ) and D f := max X ∈ St( d,r ) ∥∇ f ( X ) ∥ . 401

where ˆ D f := max X ∈ ¯ U St( d,r ) ( 1 8 ) ∥∇ f ( X ) ∥ , the second inequality is due to the contractive property 403 of P T X St( d,r ) , and the last inequality is from the fact that ∥ Y ∥ 2 ≤ 3 2 . By setting ˆ L = L f + 5 2 ˆ D f , 404 we complete the proof. 405

Proof of Lemma 4 406

Proof. It follows that 407

We complete the proof. 408

Proof of Lemma 5 409

Proof. It follows from (7) that 410

where the second inequality is from the 2-Lipschitz continuity of P St( d,r ) over ¯ U St( d,r ) ( 1 8 ) , the third 411 inequality is due to the facts that X k -¯ X k ∈ N ¯ X k St( d, r ) and ⟨ A,B ⟩ ≤ 1 2 ( ∥ A ∥ 2 + ∥ B ∥ 2 ) for any 412 A,B ∈ R n × d , and the last inequality comes from 413

Plugging µ = 3 into (17) gives (10). 414

Proof of Theorem. 415

Proof. Applying [42, Lemma 2] to (9) yields 416

Then, summing (10) over k = 0 , . . . , K gives 417

Define c 1 = 148 ˆ L 2 +148 L +36 ˆ D f +28 and c 2 = (9 ˆ D 2 f +2)(4 ˆ D f +16 ˆ L 2 +16 L +4) . Then, we 418

Therefore, for any α ≤ 1 2 c 1 , taking K → ∞ gives ∑ k =0 ∥ grad f ( X k ) ∥ 2 < ∞ . 420 ∑ ∞ k =0 ∥ X k -¯ X k ∥ 2 < ∞ . These lead to (11). 421

C Hyperparameters 422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our empirical results in Section 4 justify ours claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss our limitations in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide complete proofs in Appendix B and full set of assumptions in Section 2

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We specify the training details in Section 4 and Appendix C Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

Answer: [Yes]

Justification: We specify the code and dataset in Section 4.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify training details in Section 4 and hyperparameters in Appendix C. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We specify these in our Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use the Hugging face and opendelta as our base code and make some modifications. We use GLUE, E2E, and Suqad three dataset.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research is compatible with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of our work performed

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We correctly cite the code and datasets we used in Section 4.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our study does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a retraction-free Riemannian optimization scheme on Stiefel and oblique manifolds to perform parameter-efficient fine-tuning (PEFT) in LoRA style. The proposed approach exploits the theory of landing flows on Stiefel manifolds. Theoretical results demonstrating convergence of this iterative scheme are presented, and complemented by numerical experiments.\n\nSTRENGTHS:\nThe proposed method combines the advantages of Riemannian methods while avoiding the computational burden of retracting on the Stiefel manifold. The application in the context of parameter-efficient fine-tuning represents a novelty, and enhances the relevance of the numerical results. The work is well-presented, covering both algorithmic aspects and the experimental section effectively.\n\nWEAKNESSES:\nAll the presented theory is developed on a function defined on $St(d,r)$, while LoRA fine-tuning gives rise to an objective function $f(B,A) = L(BA)$, where $B \\in St(d,r)$ (or $Ob(d,r)$), and $A \\in \\mathbb{R}^{r \\times m}$. This objective function has to be minimized over $St(d,r) \\times \\mathbb{R}^{r \\times m}$, and the advantages of optimizing on a compact manifold are thus lost.\n\nUnfortunately, this makes the presented theoretical results not directly useful for the practical case under consideration. \nTo give a more precise statement, for example in Lemma 3, the constant $\\widehat{L}$ would depend on $A$. By the mean value theorem, we would get a bound of the kind\n\n$$\n||grad_B f(A,B_1) - grad_B f(A,B_2)|| \\leq C(A) ||B_1 - B_2||\n$$\n (as noted in equation after line 183 for the Euclidean gradient).\n\n Since the space in which $A$ resides is not compact, one would need at least a uniform control on $||A_k||$ over the iterations to make the theory interesting for LoRA fine-tuning. \nIt is interesting to note, however, that the authors observe exact numerical convergence to the constraint in all cases.\n\nQUESTIONS:\n1. **Concerning the comment made in the \"weaknesses\" section:** In all the proofs, the Lipschitz constants $L_f$ and $D_f$ depend on $A$, making the convergence analysis not directly applicable in the numerical case under investigation. I would appreciate the authors' comment on this, maybe I am missing something here?\n\n2. I believe however that the observed convergence behavior is not a coincidence, even if the setting is different.\nThe issue can be addressed by assuming $L(X) \\to +\\infty$ as $||X|| \\to +\\infty$. With enough regularity assumed, this ensures that the sublevel sets of $L$ are compact. Given an initial condition $X_0 = B_0A_0$ in the compact set $L^{-1}([-\\infty,M])$, the flow $\\dot A = -\\nabla_A L(BA)$ decreases $L$, thereby remaining within the initial compact set $L^{-1}([-\\infty,M])$. This would provide an effective bound for the Lipschitz constant, without compactness of the original domain.\n\n3. I don't see however a way to control $||A||$ uniformly in time for \"classification-like\" problems, where the global interpolating minima may be off at infinity. \n\n4. Line 93: the definition of $f$ has a typo. Also, \"differentiable\" is more common to use.\n\n5. Line 131: Even if it's clear what you mean, $\\bar{U}_{St(d,r)}(\\frac18)$ was defined nowhere in the main manuscript.\n\n6. The choice of notation is a bit unfortunate in some points. I would personally substitute the loss function $L$ with $\\mathcal L$, just to avoid confusion with the Lipschitz constants.\n\nI am inclined to raise my score, provided the authors clarify my doubts and address the mentioned issues (or at least discuss these points in the manuscript).\n\nLIMITATIONS:\nAs noted in the \"weaknesses\" section, I believe there is a delicate point that is not addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRetraction-free optimization algorithms on the Steifel manifold have been proposed in [1,18,19,41] etc. The motivation is that if the cost of the objective function/gradient evaluation is significantly larger than the evaluation of a retraction, then the retraction-free optimization algorithms show their advantages and efficiency. In particular, for the landing algorithm proposed in [1], the choice of the parameter in the penalty is important and may not be easy to choose. This paper gives an analysis that shows if the parameter is 1/3, the initial point is sufficiently close to the Stiefel manifold, and the step size is chosen sufficiently small, then the algorithm converges linearly to a stationary point. This result gives a concrete value of the parameter. Such a result is further merged into optimization on low-rank matrices and Manifold-LoRA is proposed. Numerical experiments show that the proposed method outperforms the baseline algorithms.\n\nSTRENGTHS:\nThis paper gives a concrete value of $\\mu$ and a gap between x_0 and \\bar{x}_0 such that the algorithm converges under reasonable assumptions. Numerical experiments show that the proposed method is more effective than the existing approach.\n\nWEAKNESSES:\n(1) Though the value of mu and an upper bound of \\|x_0 - \\bar{x}_0\\| are given concretely, the choice of step size is unknown. Theoretically, the step size needs to be sufficiently small (See Theorem 1). Any theoretical suggestion for the choice of the step size?\n(2) Numerical experiments report results of the comparisons. However, the definition of ``result'' is not given. Is the result computational time or classification accuracy or a notion of correctness or something else?\n(3) Problem (12) does not remove all the ambiguity. Note that if B \\in St(d, r), then B A = B O O^T A = \\tilde{B} \\tilde{A}, where O is an orthonormal matrix and \\tilde{B} = B O is still in St(d, r). Likewise for B \\in Ob(d, r). Is it possible to completely remove the ambiguity by considering the quotient manifold? \n(4) Why is the numerical performance of Manifold-LoRA for using Stiefel and Oblique manifold in (12) different? The optimization problem is equivalent in the sense that the local minimizer/stationary point does not change.\n\nQUESTIONS:\nThe questions are given in the weaknesses section. More questions are given below.\n(1) L117, X \\in St(d, r) or X \\in \\mathbb{R}^{d \\times r}\n(2) What is the percentage of the computational cost of the retraction in the overall computations?\n\nLIMITATIONS:\nThe limitations of the paper are discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers solving optimization problems with constraints that have orthonormal columns (i.e. the matrix belongs to the Stiefel manifold). The leading method for solving such problems is the Riemannian optimization. However, Riemannian optimization requires a costly retraction operation. The authors propose to circumvent this by introducing an additional penalty terms that stirs the optimization towards respect the manifold constraints. Indeed, the authors show that with correcting setting of parameters, the optimum will be on the manifold, and the algorithm will find it. The authors advocate that an additional advantage of their algorithm is that we know how to set the parameters for the penalty term, and so their algorithm is parameter-free.\n\nA significant part of the paper is devoted towards motivating the study in terms of low-rank adaption in LLMs, and showing experiments in that vain.\n\nSTRENGTHS:\n- A very elegant method for retraction free optimization on the Stiefel manifold.\n- Detailed theoretical analysis showing the algorithm converges to a critical point on the manifold.\n- The theoretical analysis gives explicit guidance on how to set the penalty parameter.\n- The LLM applicaiton and experiments appear impressive. However, I am not an expert on this subject, so it is hard for me to asses how significant the results and evaluations are.\n\nWEAKNESSES:\n(The following were addressed by the authors in the rebuttal)\n\nMajor issues (affecting the recommendation):\n1) Novelty: Citation [1] considers optimization with constraints on the orthogonal group (i.e. St(n,n)). It seems that the core idea on how to implement retraction free optimization already appears there. The authors mention this in \"related work\", and say that [1[ does not discuss the r<n case. Inspection of [1] reveals that this is not the entire story. In Sec 3.5 of [1] the case of r<n is discussed briefly, and it is said the results can be extended for that case. However, the authors of [1] are skeptical of the value of this, as they mention that there are fast retraction methods (i.e. Cayley) for the Stiefel manifold. \n-  Setting the penalty parameter: the authors advocate that they give an explicit value for the penalty parameter. And indeed Theorem 1 sets the parameter \\mu to 1/3. However, I do not think the situation is so simple. The theorems have the additional assumption that the iterates starts close to the manifold (1/8). This is, of course, easy to achieve - just start on the manifold itself. However, for the proof to work shouldn't all iterates stay inside this bound? This necessitates for the other parameter (step size) to be small enough. And indeed, the theorem requires that the step size be small enough, and does not specify how small. Without looking in detail in the proofs, my guess is that changing the penalty step size (\\mu)  affects how close you need to be to the manifold (the value 1/8), which affects how small the step size need to be. In other words, the authors load all the complexity of setting the parameters onto the step size of the main objective. Saying there is  a upper bound on its value , without specifying what that value is. You cannot call this parameter free. \n\nAnother point is that the values of the parameter probably affect convergence rate, though the authors do not discuss this at all.\n\nMinor comments (do not affect the recommendation):\n- Line 117: If X is on the manifold, shouldn't \\bar{X}, which is the projection of X on the manifold, be exactly X?\n- Line 119: \"satisfies the restricted secant\"\n- Line 131: What is U_St (1/8)? Not defined.  Ditto line 136.\n- Line 133: If the condition of twice diff is assumed, then state it earlier. \n- Eq (10): \\hat{D}_f is not defined. \n- Table 1: Why metrics are changing between columns?\n\nQUESTIONS:\n- Please address more carefully why you think you are novel with respect to [1]?\n- Why are fast retraction methods not sufficient?\n- How setting the parameters affect convergence rate?\n\nLIMITATIONS:\nNothing to add.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a new algorithm, Manifold-LoRA, which incorporates the Stiefel manifold constraint to accelerate low-rank adaptation (LoRA) in fine-tuning LLMs. It also provides theoretical and experimental validation for the retraction-free and penalty parameter-free optimization methods.

STRENGTHS:
This paper is highly technical and indicates a strong mathematical background in optimization and manifold theory. Manifold-LoRA leverages manifold geometry to reduce redundancy in LoRA fine-tuning, leading to enhanced performance and faster convergence. Furthermore, it has robust experimental validation across various datasets.

WEAKNESSES:
W1: Some experimental results are unclear and not well defined.

W2: Lack of the discussion about limitations of your method.

W3: Some findings of the experiments are hard to understand.

QUESTIONS:
Q1: In Table 1, could you please explain the meaning of m/mm? As I know, it's not an estimation of the performance of NLP models. You also didn't define Acc and Mcc. Also, why not use Acc to evaluate the performance of all the datasets?

Q2: The finding "It can be seen that our method 211 is consistently superior to other baselines." doesn't match the results in Table 1. What makes you conclude this?

Q3: Some findings are not clearly represented. For example, "We conclude that the proposed 228 Manifold-LoRA method achieves a 2x speed-up in training epochs compared to AdamW, while 229 simultaneously improving model performance". 

Q4: The two datasets in the QA task are wildly used, but also weak. Have you tried other datasets? For example, the HotpotQA dataset.

LIMITATIONS:
Limitation is not enough and is meaningless.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12