Title: Label Privacy in Split Learning for Large Models with Parameter-Efficient Training
Venue: NeurIPS 2024
Accepted: no
Keywords: Split Learning

Vertical Federated Learning

Federated Learning

Parameter Efficient Fine-tuning

Privacy

Large Language Models
Abstract: As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. 
These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model.
While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure.
This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model.
In this study, we systematically search for a way to fine-tune models over an API *while keeping the labels private*.
We analyze the privacy of LoRA, a popular approach for parameter-efficient fine-tuning when training over an API.
Using this analysis, we propose P$^3$EFT, a multi-party  split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead.
To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that P$^3$EFT is competitive with existing privacy-preserving methods in multi-party and  two-party setups while having higher accuracy.

Abstract

As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model. While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure. This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model. In this study, we systematically search for a way to fine-tune models over an API while keeping the labels private . We analyze the privacy of LoRA, a popular approach for parameterefficient fine-tuning when training over an API. Using this analysis, we propose P 3 EFT, a multi-party split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead. To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that P 3 EFT is competitive with existing privacy-preserving methods in multi-party and two-party setups while having higher accuracy.

1 Introduction

One of the main reasons behind deep learning success is its ability to transfer knowledge between tasks [34]. When training a model for any particular problem, it is common to reuse previously trained models from other, related problems. In the past, this was typically done by downloading pre-trained model weights from public hubs, then fine-tuning the said models on the downstream task. However, as models grow larger and more compute-intensive, fine-tuning them locally becomes an increasingly difficult task. Furthermore, many recent models are not released, but instead made available as proprietary services.

When a model cannot be fine-tuned locally, many practitioners opt instead for the so-called fine-tuning APIs [27, 16, 6, 26]. These APIs are web services that host one or several pre-trained models and allow clients to perform limited fine-tuning. More specifically, APIs usually allow their clients to run parameter-efficient fine-tuning (PEFT), such as LoRA [15] or Prefix-tuning [21]. These techniques allow adapting a model to a dataset while training a relatively small number of additional weights, which is particularly important for large language or image generation models that have billions of parameters.

Although the fine-tuning APIs can be convenient, they also introduce new risk in terms of data privacy. 32

When a client uses such API to train on sensitive data, they need to ensure that their data will stay 33

34

private [7].

This is particularly important when dealing with patient's medical records, personal

user data or trade secrets [24, 19]. The two main threats to data privacy are that the API provider 35

obtains the private data and that a third party intercepts data in transit. Therefore, data privacy is 36

not guaranteed even if the API provider is trusted. Several recent works propose LLM fine-tuning 37 protocols that establish a certain level of privacy for multi-party fine-tuning [42, 7, 22]. Unfortunately, 38 these algorithms work for a narrow class of fine-tuning algorithms or assume that a client can run 39 LLM training locally using an obfuscated version of the model, provided by a remote server [42]. 40 As a result, these algorithms are impractical for our use case of fine-tuning over an API. The few 41 algorithms that are suitable for API fine-tuning guarantee the privacy of input tokens [22], meaning 42 that the attacker can infer private training labels . 43

44

45

46

47

48

49

50

51

52

In this work, we seek to alleviate this problem by designing a two-party fine-tuning protocol that performs standard parameter-efficient fine-tuning with privacy guarantees. We formulate our protocol as a special case of split learning (or vertical federated learning), where one side (server) holds the pre-trained model and the other (client) has private training data. More specifically, we focus on the privacy of client's training labels . While input privacy is often crucial, there are scenarios where input data is publicly available, such as social media user pages. In these cases, labels could include ad clicks (known to the social network) or financial information (known to a bank that matches social profiles to its internal records). This example further justifies the use of LLMs, as social media pages often contain substantial amounts of text, and LLMs excel at processing long-context data.

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

Instead of developing a specific privacy-preserving architecture, we seek algorithms that can work with popular existing models and PEFT algorithms. Furthermore, our approach relies on the properties of parameter-efficient fine-tuning. Notably, since the adapters are compact, both parties can maintain multiple sets of adapters and swap between them with relative ease. This allows us to design a PEFT-specific algorithm that can solve its task more effectively than general split learning strategies [18].

We summarize our main contributions as follows:

We analyze Low-Rank Adaptation, a common parameter-efficient fine-tuning algorithm, from the perspective of label privacy in the split learning setup. We observe that, despite fine-tuning less than 0 . 1% of model parameters, PEFT algorithms leak client's training labels against simple attacks that work for modern pretrained transformers.

Based on our analysis, we formulate a framework for privacy-preserving parameter-efficient fine-tuning (P 3 EFT). This framework leverages the properties of PEFT to obfuscate the gradients and parameters communicated during fine-tuning with little impact on the finetuned model quality.

To verify the practical viability of P 3 EFT, we conduct experiments on popular real-world PEFT workloads 1 . Specifically, we fine-tune DeBERTa-v2-XXL [13], Flan-T5-Large [4] and LLaMA-2 7B [35] on a set of standard language understanding problems. We find that, compared to prior split learning algorithms, P 3 EFT can maintain label privacy throughout training with a significantly smaller accuracy drop.

2 Background

2.1 Federated learning and split learning

Privacy preservation in machine learning has been a subject of active study within several frameworks. An important branch of privacy-preserving learning methods is federated learning, or FL [24], which can be broadly described as an approach allowing several parties to train a model jointly without sharing their private data. In particular, vertical federated learning [12, 43] targets the scenario where different features (including the label) of each training instance are kept by different parties.

One of the most popular approaches to vertical FL for neural networks is split learning [10, 37], where each party stores its part of the overall model. To train the model in such an approach, it is only necessary to transfer the intermediate activations and the gradients between layers, while the data itself is stored at the premises of the participant hosting each layer. In this work, we focus on the two-party formulation of split learning, where one side stores the features for each example and another one stores the labels.

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

Recent works have investigated the setting of two-party split learning from the label leakage perspective [38, 28]: because the label party needs to pass the gradients of the loss function to the non-label party, it is possible for the latter party to deduce the labels by inspecting the gradients or activations or by hijacking the training procedure. Li et al. [18] provide a set of attack methods that allow recovering private labels and propose a defense mechanism that injects noise into the gradients; however, they test the approach on pretraining smaller models, and we study finetuning large models on private downstream data.

2.2 Parameter-efficient finetuning

The majority of large neural networks today are not trained with a specific task in mind: instead, they are pretrained on a general objective and then adapted for the downstream problem. Importantly, the growth in the size of foundation models has led to the increased popularity of parameter-efficient finetuning (PEFT) methods that adapt the model to a given task by training a small number of task-specific parameters. There are several prominent approaches to parameter-efficient finetuning, ranging from trainable prompts [21, 11], to residual adapters [14, 29]. We focus on Low-Rank Adaptation (or LoRA, 15), one of the most popular PEFT methods that adds extra parameters to each weight matrix in the form of a low-rank factorization (see Appendix C for a more detailed description). Such formulation allows LoRA adapters to be merged into the original weights after finetuning; this ability, combined with the simplicity of the method, has made LoRA a broadly popular approach in multiple domains. Still, the approach we propose can be applied to any PEFT method.

Several recent lines of work explore the problem of fine-tuning LLMs with privacy guarantees [44, 31]. Zhao et al. [46] analyze the viability of prompt tuning for federated learning, and Zhang et al. [45], Liu et al. [23] study PEFT algorithms in the setting of horizontal federated learning, that is, where multiple users train a shared model on their local private data. Another, more relevant research direction considers private fine-tuning in a vertical federated learning scenario, where participants hold different model layers [22, 40]. Most of these studies leverage the idea of differential privacy to prove an upper bound on how much information is leaked [8]. Unfortunately, these upper bounds are typically loose and do not match practical observations for real models. Furthermore, the majority of these studies only guarantees privacy of specific parts of the training procedure: for instance, Li et al. [22] only protects the input features, and not labels or model parameters. Finally, Xiao et al. [42] presents an alternative algorithm that protects client data by running the entire fine-tuning on client side by emulating the server-side model layers. While this approach is more holistic, it assumes that clients can run fine-tuning locally, which makes it impractical for many real-world users of LLM fine-tuning APIs. The primary distinction between our work and these studies is that we investigate parameter-efficient adaptation in the setting of split learning: we aim to finetune a model without disclosing the labels of examples to the model provider.

3 Privacy-preserving parameter-efficient fine-tuning

In this section, we analyze the privacy of parameter-efficient fine-tuning and propose a protocol for two-party parameter-efficient fine-tuning with the desired privacy guarantees. We begin by analyzing the privacy of API fine-tuning with popular PEFT algorithms in Sections 3.1 and 3.2. Then, in Section 3.3, we formulate a protocol for privately computing gradients over fine-tuning APIs. Finally, we formulate the full P 3 EFT protocol in Section 3.4.

3.1 Setup

To analyze the privacy of API fine-tuning, we first need to formulate a common framework for this type of APIs and develop private learning protocols. This step is important, because existing fine-tuning APIs greatly vary in what they offer to the client: from closed APIs that require users to submit their full training data [27] to more flexible APIs where clients can run individual training steps [20, 2, 30]. Similarly to most existing works on split learning, we focus on the latter type of APIs that allows clients to run individual forward and backward passes over a remote model. Thus, a client can use these APIs to obtain the training gradients for their PEFT adapters, then update adapters locally with any optimization method. In our work, we adopt this archetype of fine-tuning API as it offers sufficient flexibility to develop privacy-preserving algorithms.

We formulate fine-tuning over an API for two or more parties: a client, and one or several servers. 137 The client owns a training dataset with inputs X and labels Y . In turn, each server has the same 138 pre-trained model h ( x i , θ ) ∈ R d . Note that the parameters θ denote not the pre-trained model 139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

Step: 0

Step: 1000

Step: 4000

Step: 16000

Step: 0

Step: 1000

Step: 4000

Step: 16000

weights, but the trainable adapter weights for a certain PEFT algorithm. A model can encode an input x i ∈ X and produce a d -dimensional vector of activations that depend on the learned adapter weights θ .

To allow fine-tuning, a server offers two API methods:

forward ( x, θ ) → h ( x, θ ) that computes model activations on input x using adapter weights θ ;

backprop ( x, θ, g h ) → g θ that receives gradients of an arbitrary loss function w.r.t. model activations g h = ∂L ( h ( x,θ )) ∂h ( x,θ ) and returns the gradients w.r.t. adapter parameters, g θ = ∂L ( h ( x,θ )) ∂θ .

We further assume that both forward ( · ) and backprop ( · ) APIs are stateless and deterministic, i.e. calling the same API method multiple times (or on multiple servers) with the same inputs produces identical results. Thus, if the model uses dropout or any other form of non-determinism, we assume that clients provide the random seed as a part of x .

To fine-tune a model with this API, a client can initialize adapters locally, alongside with a small task-specific head 2 , then train both adapters and the head. For each training batch ( x, y ) ∈ D , a client calls forward ( x, θ ) to compute feature representations, then predicts with local 'head' and computes task-specific loss function L . After that, a client performs backward pass: first, it computes gradients w.r.t. local head inputs g h = ∂L ∂h , then passes those gradients to a remote server via backprop ( x, θ, g h ) API call to compute gradients w.r.t. ∂L ∂θ . Finally, a client updates both θ and local 'head' parameters using the optimizer of choice.

Before building more advanced algorithms, let us analyze the privacy of client's labels under standard fine-tuning. We consider an 'honest, but curious' attacker model. This means that the server will faithfully run the forward and backprop computations as requested by the client without changing the results. Furthermore, we assume that servers are independent and do not communicate client's data between each other. However, a server can recover client's labels by performing arbitrary computations using any information it receives from the client. When training in this way, a client does not directly communicate training labels to the server. However, it communicates inputs, adapter parameters, and gradients. Furthermore, the server communicates input representations that can be intercepted by a third party.

3.2 Label Leakage of Standard Split Learning

In Figure 1, we train a DeBERTa-v2-XXL model on the SST-2 [32] sentiment classification dataset. The top row depicts the gradients g h communicated by the client when calling backprop ( · ) at different training stages. In the bottom row, we similarly track activations h ( x, θ ) that server may compute based on the specified x, θ . We defer further additional figures and details to Section 4.1.

As we can see, both gradients and activations are arranged in such a way that simple k-means 174 clustering would reveal which objects have the same label. The training activations (bottom row) do 175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

SERVER

CLIENT

n model instances with unique LoRA weights

𝜃

activations

weights

weighted activations

model head

local layers

h 1

W1

h 2

h n

Wn

𝜃

1

→

→

→

⊙

=

𝜃

2

→

→

→

⊙

=

𝜃

n

→

→

→

⊙

𝚺

=

h

'

𝓛

…

…

…

…

…

…

forward

last layer

activations

from n models

backward

weighted

gradients

to n models

…

not reveal labels right away (at least not against this attack). However, they gradually 'leak' private label information during training. Informally, it appears that the training gradients gradually pull apart the feature representations for each label, until eventually they turn into separate clusters. From an information-theoretic perspective, knowing just one vector of gradients or trained activations allows the attacker to learn all but one bit 3 of information about client's private labels.

To summarize, leaving any one data source unprotected (gradients, activations or parameters) would already compromise label privacy. However, we found that gradients and activations require different means of protection.

3.3 Privacy-preserving backpropagation

In this section, we formulate an algorithm for 'anonymizing' the gradients communicated over a single training step with arbitrary PEFT type. Several prior works approach this by modifying the training objective or model architecture. However, when dealing with a real-world PEFT workload with optimized hyperparameters, changing the model or loss function often results in reduced model accuracy 4 . Thus, we seek an algorithm that preserves both model and training objective.

We design our algorithm based on an observation that backpropagation is conditionally linear in output gradients , even when the model itself is nonlinear. Formally, if we take a model h ( · , · ) , a fixed set of trainable parameters θ and input samples x , the backprop function 5 computes backprop ( x, θ, ∂L ∂h ( x,θ ) ) = ∂L ∂θ . For convenience, we shorten it to backprop ( x, θ, g h ) = g θ , where g h = ∂L ∂h ( x,θ ) represents the gradients of some objective function with respect to model activations (outputs), and g θ = ∂L ∂θ are gradients of the same objective function w.r.t. trainable parameters. In this notation, backprop is linear in terms of g h for any fixed x, θ .

This becomes self-evident if we view backprop as multiplying g h by the Jacobian of model outputs w.r.t. trainable parameters, ∂h ( x,θ ) ∂θ . If x, θ are constant, the Jacobian is also constant, and backprop is a linear operator:

This observation allows us to design a private backpropagation protocol. To illustrate this protocol, let us first consider a distributed API with two identical independent servers that offer backprop API. Then, for arbitrary vector z , we can rewrite backprop ( x, θ, g h ) as backprop ( x, θ, g h + z )+ backprop ( x, θ, g h -z ) .

During API fine-tuning, we obtain backprop ( x, θ, g h + z ) using an API call to server 1, whereas the second term backprop ( x, θ, g h -z ) translates to an API call to server 2. Note that neither of two servers has access to the true gradient g h : they only receive the sum [ g h + z ] . If we sample a large noise vector z (Var ( z ) ≫∥ g h ∥ 2 2 ), this sum also becomes dominated by noise. However, when both API calls finish, a client can sum the results to recover the true gradient of the loss with respect to parameters.

If both requests are processed by the same server, it can obviously recover g h by adding up gradients 210 from both calls, which leads us to the final step. Instead of generating a single noise vector, a client 211

needs to generate (privately) a set of m> 1 random vectors ˆ g 1 h , . . . , ˆ g m h and scalars α 1 , . . . , α m such 212 that 213

214

215

Then, for each ˆ g i h , client computes backprop ( x, θ, ˆ g i h ) as m parallel API calls. Once this is done, client recovers

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

Note that the client does not reveal α 1 , . . . , α m to anyone.

The resulting procedure is formulated in Algorithm 1. This algorithm is conceptually similar to the secure aggregation protocol for conventional (horizontal) federated learning [1]. This protocol allows clients to average their local vector with peers while keeping each individual vector provably private. Similarly to our scheme, clients perturb the vector in such a way that the average of perturbed vectors remains the same. Unlike Bonawitz et al. [1], our protocol privately backpropagates through a server-hosted model by leveraging the conditional linearity of the backpropagation operator.

The private backpropagation algorithm can allow client to safely compute gradients once , but, in practice, client usually needs to run many consecutive steps. This creates an additional vector of attack: if the same server receives two sets of parameters θ t , θ t +1 , they could potentially recover g θ by inverting the optimizer.

In the simplest case, if the server somehow knows that the client computes θ t +1 = θ t -η · g θ , then they can compute g θ = ( θ t -θ t +1 ) /η . While g θ does not necessarily leak private labels, a server could, in some cases, use g θ to recover g h , either fully (e.g. if Jacobian is invertible), or partially.

The client has two ways to prevent this attack. The first one is to ensure that no single server runs backprop on two consecutive steps. This is easy to do in decentralized systems where there are many potential servers. However, even when there is a single server, they could be required to set up multiple trusted execution environments [25]. A more risky alternative is to ensure that the gradients cannot be reversed from consecutive parameters: randomize initial optimizer statistics or add noise to parameters. This solution is easier, but it can slow down training in some cases.

To summarize, we formulated a procedure that allows a client to compute gradients privately for any given model and PEFT type. Furthermore, since Equation 3 recovers true gradients, this obfuscation method does not affect the training dynamics. However, as we have shown in Section 3.1, gradients are not the only source of privacy leakage.

3.4 Full fine-tuning

The other major attack vector are training activations. As the model fits to training data, it's intermediate activations h ( x, θ ) allow attackers to recover labels, e.g. by clustering (see Figure 1). To combat this issue, we take advantage of the fact that PEFT has few trainable parameters. Instead of learning just one set of trainable parameters, a client creates n independent adapter sets θ 1 , ..., θ n . Note that this does not require n unique servers: a single server can run multiple sets of adapters. Furthermore, a client can alternate between using different servers for the same adapters. During forward pass, the outputs of different adapters are mixed together using randomized mixing weights W ∈ R n,d :

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

Overall, we design this model in such a way the combined model h ′ can predict the labels, but the adapters h ( x, θ i ) do not allow predicting these labels without knowing the mixing weights W. The mixing weights are generated such that initial activations h ′ ( x, . . . ) are equal to mean h ( x, · ) for all x . To achieve this, we generate W as follows: first, we generate n · ( n -1) / 2 d-dimensional random vectors ξ i,j ∈ R d ∀ i ∈ [1 , n ] , j ∈ [ i +1 , n ] . Then, we add them up in the following way:

Here, e stands for a vector of all ones. The purpose of these mixing weights is to ensure that the gradients w.r.t. individual h ( x, θ i ) are obfuscated, but the averaged model behaves the same as regular PEFT adapter. To illustrate this, consider n =2 identical LoRA adapters θ 1 , θ 2 . During the first training step h ( x, θ 1 ) = h ( x, θ 2 ) . Therefore,

However, the two adapters will learn different functions as they receive different gradients. From the first update on, h ′ will be equal to an average of adapter predictions.

Finally, to ensure that individual adapters h ( x, θ ) do not accidentally 'learn to leak' labels, we maintain this over the course of training with a privacy regularizer inspired by [9]. This ensures that it is impossible to predict labels from individual adapters h ( x, θ i ) . Intuitively, on each training step, client fits n linear 'heads' that learn to predict labels y from h ( x, θ i ) , then performs an adversarial update of θ i to prevent the 'head' from predicting y . Formally, each of n 'heads' minimize the same objective function as the full model. For instance, if the full model solves multi-class classification, each head is trained to minimize cross-entropy:

where y is one-hot encoding of the correct class.

The whole adversarial update takes place locally on client's side, using the same h ( x, θ ) it uses for the main training objective. The resulting procedure appears complicated but it typically takes negligible time compared to running the large pre-trainied model h ( x, θ ) . Furthermore, since adversarial 'heads' are linear, minimizing the objective above is done with standard logistic regression solver.

To summarize, our approach combines the two proposed ideas: we use the private backpropagation algorithm from Section 3.3 to protect the gradients, then trains a mixture of adapters in such a way that obfuscates learned activatons leaking labels. The resulting procedure is described in Algorithm 2. In the next section, we will evaluate the efficacy of P 3 EFT on popular NLP benchmarks.

4 Experiments

The main goal of our study is to find a practical method of private fine-tuning that would scale to large models. Because our approach leverages parameter-efficient fine-tuning techniques, we evaluate P 3 EFT with fine-tuning Transformer models on popular NLP benchmarks that these techniques were designed for.

To that end, we chose three pre-trained models: DeBERTa-XXLarge [13], Flan-T5-Large [4] and LLaMA-2 7B [35]. We train these models on several datasets from the GLUE benchmark [39]: SST-2 [32], MNLI [41] and QNLI.

4.1 Privacy of gradients and activations

For this experiment, we train DeBERTa-XXLarge on SST-2 dataset using LoRA adapters with hyperparameters from [15]. First, we train the model locally and track model activations h and gradients w.r.t. those activations. We apply principal component analysis to them and plot the first

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

Step: 0

Step: 1000

Step: 4000

Step: 16000

Step: 0

Step: 1000

Step: 4000

Step: 16000

2 dimensions in Figure 1. Similarly, we visualize gradients of individual per-sample loss functions w.r.t. LoRA parameters θ in Figure 3 (top row). The results suggest that a hypothetical attacker could easily recover private labels by performing K-Means clustering over any data source: activations, gradients with respect to activations, or individual gradients with respect to parameters.

Next, we run the same experiment using privacy-preserving backpropagation as defined in Section 3.3. We use n = 2 with the noise variance set to 1000. As expected, we observed the same learning curve as with normal training. However, instead of sending gradients w.r.t. activations to the server, a client uses specially crafted random noise vectors that are not informative. In Figure 3 (bottom) we plot the same kind of individual gradients as in the top row, except that we visualize the gradients computed by the first of the two servers. Finally, we train XGBoost [3] with default hyperparameters to predict labels given the noisy gradients (pre-PCA): the resulting classifier is able to fit the training data perfectly, but has at most 50 . 4% accuracy on a balanced test set.

4.2 Main fine-tuning experiments

Next, we evaluated the entire P3EFT algorithm. To control tasks and model type, we examined DeBERTa and Flan-T5 across all four datasets mentioned above, in addition to evaluating LLaMA on SST2 and QNLI datasets. For each setup, we compare against three baselines:

Without LoRAs. In this baseline, the client gathers h activations at the beginning (with no adapters), then proceeds to train local 'head' layers using these activations. This method cannot leak information about training labels except for what is stored in X.

Regular fine-tuning (Regular FT) refers to training a single LoRA adapter normally. This baseline represents an upper bound on model accuracy, but lacks privacy.

Distance Correlation (DC). Our re-implementation of the distance correlation defense formulated in [33] for Transformer models.

For each algorithm, we evaluated a task-specific metric (accuracy or F1), as well as the privacy leakage value for the 3 following measures:

Spectral attack AUC -ameasure of vulnerability to an attack proposed in [33], measured as classifier ROC AUC: lower value corresponds to better privacy.

Norm attack AUC -vulnerability to a variant of attack proposed in [18], measured as classifier ROC AUC (lower is better). Despite the initial proposal of this approach for attacking gradients, we observed that it is also well-suited for attacking activations.

K-means accuracy -vulnerability to clusterization attack, measured in the percentage of correctly clustered activations, lower is better.

For all setups, we report the worst (least private) value among these metrics throughout the entire 322 training period as a measure of privacy leakage, because it is the worst possible scenario that matters 323 from the client's perspective. For DC and P 3 EFT, we specify the values for the best configuration in 324 terms of the utility-privacy trade-off. See details in Appendix A. We also report adjusted standard 325 deviations for the two privacy aware algorithms: P 3 EFT and DC. To do so, we run the full training 326 procedure from scratch with 3 random seeds. 327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

The results for DeBERTa are presented in Table 1. To improve reproducibility, we reuse the hyperparameters from original paper, with the exception of the LoRA dropout value. We disable dropout because it interferes with the mixing weights (5). In preliminary experiments, we observed that with dropout enabled, both our algorithm and DC begin to perform significantly worse.

We use n = 2 adapter sets for P 3 EFT for all datasets and adhered to the same approach for the other models as well. Overall, P 3 FT achieves nearly the same accuracy as traditional (non-private) fine-tuning, outperforming the DC-based algorithm in terms of accuracy given the same privacy level. On the MNLI dataset, we could not find the hyperparameters for DC that ensure stable training while maintaining privacy. Meanwhile, P 3 EFT maintains consistent performance on this task with a slight drop in quality.

Table 2 a reports evaluation for the Flan-T5 base model[4]. For this model, we adapt the exact same hyperparameters as in the previous evaluation with DeBERTa-XXLarge. Compared to DeBERTa, these results are more closely matched. Both both our algorothm and DC consistently solve all three tasks, but P 3 EFT slightly outperforms DC in terms of privacy.

To evaluate how our algorithm scales to larger models, we also fine-tune Llama-2 7B [35] on SST2 [32] and QNLI [39] datasets. For these evaluations, we use LoRA hyperparameters that Hu et al. [15] used when fine-tuning GPT-3, with several changes inspired by Dettmers et al. [5]. Namely, we use the NF4 weight format, apply LoRA to both attention and MLP layers with rank 16. We fine-tune both tasks with maximum context length of 512 and weight decay 0.01. Table 3 summarizes our results: for QNLI, P 3 EFT achieves somewhat better privacy-accuracy trade-off. On SST2, P 3 EFT shows similarly favorable trade-offs while DC struggles to preserve privacy.

5 Conclusion and Discussion

In this work, we analyze privacy-preserving fine-tuning of large neural networks in the context of parameter-efficient fine-tuning and the two-party split learning setting. We show that while standard fine-tuning suffers from label leakage even in the parameter-efficient case, it is possible to leverage the efficiency of PEFT to alter the procedure without any significant performance drawbacks. We test the resulting method, named P 3 EFT, on a range of pretrained language models and multiple datasets, showing that it is competitive with a strong baseline in terms of label privacy while having higher task performance.

In future work, it is natural to explore how this approach can be extended to establish holistic privacy in both labels and inputs. This problem can be approached from two directions: either adapt the ideas of P 3 EFT for input privacy, or combine it with an existing work like [22]. Another important direction for future research is exploring the privacy of the long-term client-provider interaction. In a typical real-world use case of API fine-tuning, a client performs multiple training runs on overlapping data and hyperparameters. This could open additional attacks vectors that combine information from multiple training runs.

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

References

Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , pages 1175-1191, 2017.

Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. arXiv preprint arXiv:2209.01188 , 2022. URL https://arxiv.org/abs/ 2209.01188 .

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, pages 785-794, New York, NY, USA, 2016. ACM. ISBN 978-1-4503-4232-2. doi: 10.1145/2939672.2939785. URL http://doi.acm.org/10.1145/2939672.2939785 .

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022. URL https://arxiv.org/abs/2210.11416 .

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 , 2023.

Dreambooth API. Dreambooth API - Easily finetune Stable Diffusion and generate customised AI images - dreamboothapi.ai. https://dreamboothapi.ai/ , 2023. [Accessed 28-092023].

Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. arXiv preprint arXiv:2305.15594 , 2023.

Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming , pages 1-12. Springer, 2006.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 11801189, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/ ganin15.html .

Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications , 116:1-8, 2018. ISSN 1084-8045. doi: https://doi.org/10.1016/j.jnca.2018.05.003. URL https://www.sciencedirect.com/ science/article/pii/S1084804518301590 .

Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4921-4933, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.381. URL https://aclanthology.org/2021.acl-long.381 .

Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption, 2017.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=XPZIaotutsD .

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr. press/v97/houlsby19a.html .

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9 .

Hugging Face. AutoTrain - huggingface.co. https://huggingface.co/autotrain , 2023. [Accessed 28-09-2023].

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. Label leakage and protection in two-party split learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=cOtBRgsf2fO .

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering , 2021.

Shen Li, Pritam Damania, Luca Wehrstedt, and Rohan Varma. PyTorch RPC: Distributed Deep Learning Built on Tensor-Optimized Remote Procedure Calls. In Proceedings of Machine Learning and Systems 5 (MLSys) , 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.353. URL https://aclanthology.org/2021.acl-long.353 .

Yansong Li, Zhixing Tan, and Yang Liu. Privacy-preserving prompt tuning for large language model services. ArXiv , abs/2305.06212, 2023. URL https://api.semanticscholar.org/ CorpusID:258588141 .

Xiao-Yang Liu, Rongyi Zhu, Daochen Zha, Jiechao Gao, Shan Zhong, and Meikang Qiu. Differentially private low-rank adaptation of large language model using federated learning. arXiv preprint arXiv:2312.17493 , 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 1273-1282. PMLR, 20-22 Apr 2017. URL https://proceedings.mlr.press/v54/ mcmahan17a.html .

Nvidia. Nvidia confidential computing. https://www.nvidia.com/en-us/data-center/ solutions/confidential-computing , 2023. [Accessed 28-09-2023].

OctoAI. Fine-tuning Stable Diffusion - docs.octoai.cloud. https://docs.octoai.cloud/ docs/fine-tuning-stable-diffusion , 2023. [Accessed 28-09-2023].

OpenAI. OpenAI Platform - platform.openai.com. https://platform.openai.com/ docs/guides/fine-tuning , 2023. [Accessed 28-09-2023].

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

Dario Pasquini, Giuseppe Ateniese, and Massimo Bernaschi. Unleashing the tiger: Inference attacks on split learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , CCS '21, page 2113-2129, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384544. doi: 10.1145/3460120.3485259. URL https://doi.org/10.1145/3460120.3485259 .

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning, 2021.

Yuma Rao, Jacob Steeves, Ala Shaabana, Daniel Attevelt, and Matthew McAteer. Bittensor: A peer-to-peer intelligence market, 2021.

Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204.07667 , 2022.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/D13-1170 .

Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451 , 2022.

Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Vˇ era K˚ urková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis, editors, Artificial Neural Networks and Machine Learning - ICANN 2018 , pages 270-279, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01424-7.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data, 2018.

Praneeth Vepakomma, Otkrist Gupta, Abhimanyu Dubey, and Ramesh Raskar. Reducing leakage in distributed deep learning for sensitive health data. 05 2019.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Privatelora for efficient privacy preserving llm. arXiv preprint arXiv:2311.14030 , 2023.

Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017.

Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.04870 , 2023.

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. , 10(2), jan 2019. ISSN 2157-6904. doi: 10.1145/3298981. URL https://doi.org/10.1145/3298981 .

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=Q42f0dfjECO .

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 9963-9977, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.632. URL https://aclanthology.org/ 2023.findings-acl.632 .

Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communicationefficient and privacy preserving prompt tuning in federated learning, 2023.

A Hyperparameters search

In P 3 EFT and Distance Correlation methods resulting loss L function can be viewed in the form

where L m - main task loss, L r - regularizer and α is a coefficient that controls the tradeoff between these two losses. The selection of this coefficient affects the final performance of the model. Therefore, to find the best configurations for both methods, we iterated through this hyperparameter using a grid search.

We started with α = 1 and then altered it with a multiplicative step of 10 1 2 . Values were discarded if the quality did not exceed that achieved by solely training the classifier without LoRA. This criterion was adopted because such outcomes would suggest the method's inability to outperform training scenarios in which the server does not engage with the labels whatsoever. Additionally, we excluded values that led to unstable training. By this, we mean instances where, although the model initially trained on the primary task, at some point, the regularizer began contributing significantly more, and the utility value dropped to the starting value. We observed this issue for the DC method with DeBERTa on the MNLI. From the remaining values, we aimed to choose the one that offered the lowest privacy leakage. The final hyperparameter values for P 3 EFT can be found in the Table 4 and for DC in the Table 5.

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

B Formal algorithm definition

Below, we define the full P 3 EFT algorithm. In Algorithm 2, main_loss is the task-specific objective e.g. cross-entropy; reg_loss is the adversarial regularizer described in Section 3.4. We denote client-side model "head" as f ( h, ψ t ) , where ψ represent trainable head parameters. Finally, opt_step function performs a single gradient descent step with a task-specific optimizer, typically Adam [17].

Algorithm 2 P 3 EFT - full training algorithm

C Informal description of LoRA fine-tuning

For convenience, we provide a brief summary of fine-tuning with LoRA [15]. This PEFT method was originally designed for fine-tuning large pre-trained language models on downstream NLP tasks. These language models are typically based on the Transformer architecture [36], where most trainable parameters are allocated to linear layers in multi-head self-attention and feedforward blocks.

In the first stage of LoRA fine-tuning, user augments the model with adapters. To do so, a user goes over linear layers in transformer blocks and adds two trainable matrices, A and B that affect this layer's forward pass. Let W i × x + b i be the original layer with n inputs and m hidden units. Here, W i ∈ R m × n is a pre-trained weight matrix, b i ∈ R m is a pre-trained intercept vector and x ∈ R n represents a vector of inputs to this particular layer. During the forward pass, a layer with LoRA adapter computes W i × x + b i + B i × A i × x , or equivalently, ( W i + B × A ) × x + b i . Here, A i and B i are two newly added matrices that constitute a LoRA adapter.

The adapter matrices A ∈ R r × n and B ∈ R m × r have a very small intermediate dimension r . For instance, when training GPT-3 with LoRA adapters, authors use 1 ≤ r ≤ 64 , whereas the main weight dimensions are m = n = 12288 . The first matrix A is initialized with small random normal values, and the second matrix B is initialized at zeros. That way, initial A and B do not affect the model predictions.

Once all adapters are initilized, the user trains all A i and B i matrices of the model, while keeping the rest of the weights frozen. This way, only a small faction (less than 1%) of model weights are updated. Once the training is over, the learned adapters A i and B i can be merged into the main weights ( W i := W i + A i × B i ) or used separately.

LoRA adapters are designed with two objectives in mind: i) to allow fine-tuning models in limited GPU memory and ii) to allow inferencing many fine-tuned models using one inference server. When fine-tuning, LoRA achieves small memory footprint due to the fact that user does not need to compute

568

569

570

571

572

573

gradients (or optimizer statistics) for billions of main model parameters. During inference, a server can keep a library of several adapters for different tasks and swap between them on demand.

D Informal description of LoRA fine-tuning

We used NVIDIA A100 GPUs for all the experiments. Experiments with DeBERTA [13] and Flan-T5 [4] on SST2 [32] were conducted on the single GPU, while experiments on MNLI [41] and QNLI require 4 A100. LLaMA-2 [35] expetiments were carried out on the node of 8 A100.

All the experiments last 12-24 hours. However, it is possible to speed up some of them using more 574

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See our main contributions in 1 and Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss potential limitations of the method in Section 1, Section 3 and Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We include no proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe all the technical details in the Section 4 and in the README of our attached code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide attached code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4, Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars in main results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We describe a potential social impact in Introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We do not describe the safeguards

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use open datasets from GLUE benchmark and open-sourced models and cite them in our work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any of the assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revisited the problem of label leakage in split learning in the context of fine-tuning large models with parameter-efficient training. Based on modern use cases, they proposed two privacy-preserving protections for gradients and activations during split learning. The proposed methods are evaluated on several large models including Llama2-7B, fine-tuned using LoRA and full model fine-tuning.\n\nSTRENGTHS:\n1. The paper moved forward a step on the urgent need to privacy-preserving split learning over large models and fine-tuning with LoRA.\n2. The writing is generally good despite some minor issues. The flow of ideas is clear.\n3. The proposed method is evaluated over different pre-trained large models, conforming to current real-world use cases of LLMs.\n\nWEAKNESSES:\n1. There exist several related works discussing the attacks and defense regarding the label leakage in split learning. The authors may need to compare the differences between the proposed methods and previous literature. The evaluation part lacks the comparison to existing privacy-preserving solutions over label leakage and some trivial solutions such as directly applying differential privacy, which is easy to implement.\n2. In modern use cases of API fine-tuning, apart from the applications of classification, text generation with LLMs and image generation with multimodal models and diffusion are more common cases. And it is very critical to protect labels in these applications. For example, labels in text generation can contain answers to private questions in the private dataset. However, the leakage study and proposed privacy-preserving methods do not apply to these applications.\n3. There are some minor writing issues that could be improved. For example, content introducing API fine-tuning and potential privacy concerns can be shortened in the introduction. The paragraph from line 53 to 58 can be reorganized so that it won't leave '[18]' for a whole line. Same thing for line 209. On line 28, write the full name of LoRA before using acronym. The authors are suggested to talk about split learning and no need to raise extra efforts for readers to understand what vertical federated learning is.\n\n[1] Wan, Xinwei, Jiankai Sun, Shengjie Wang, Lei Chen, Zhenzhe Zheng, Fan Wu, and Guihai Chen. \"PSLF: Defending Against Label Leakage in Split Learning.\" In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2492-2501. 2023.   \n[2] Kariyappa, Sanjay, and Moinuddin K. Qureshi. \"ExPLoit: Extracting private labels in split learning.\" In 2023 IEEE conference on secure and trustworthy machine learning (SaTML), pp. 165-175. IEEE, 2023.     \n[3] Erdoğan, Ege, Alptekin Küpçü, and A. Ercüment Çiçek. \"Unsplit: Data-oblivious model inversion, model stealing, and label inference attacks against split learning.\" In Proceedings of the 21st Workshop on Privacy in the Electronic Society, pp. 115-124. 2022.      \n[4] Xu, Hengyuan, Liyao Xiang, Hangyu Ye, Dixi Yao, Pengzhi Chu, and Baochun Li. \"Permutation Equivariance of Transformers and Its Applications.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5987-5996. 2024.\n\nQUESTIONS:\n1. From my understanding, split learning usually involves multiple clients and a server. But according to line 137, it is a client and several servers. Can the authors explain about this?\n\nLIMITATIONS:\nDiscussed in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study addresses the privacy concerns associated with the fine-tuning of Large Language Models (LLMs), focusing on SplitNN. It explores how gradients and activations can leak data, potentially allowing attackers to reconstruct original data sets. In experiments, the proposed method reduces label leakage while maintaining minimal utility loss.\n\nSTRENGTHS:\nS1. The manuscript highlights significant privacy issues in LLM fine-tuning, specifically the potential for data leakage through gradients and activations in SplitNN.\n\nS2. Experimental results show that the proposed method significantly mitigates label leakage with minimal impact on utility.\n\nWEAKNESSES:\nW1. The claim that backpropagation is \"conditionally linear\" is not sufficiently rigorous. The manuscript suggests that $\\text{backprop}(x, \\theta, g_h+z)+\\text{backprop}(x, \\theta, g_h−z) = \\text{backprop}(x, \\theta, g_h)$ under the assumption that $\\theta$ is constant. However, $\\theta$ updates during each backpropagation, invalidating this assumption. Moreover, swapping the order of $\\text{backprop}(x, \\theta, g_h+z)$ and $\\text{backprop}(x, \\theta, g_h−z)$ could lead to different outcomes. Formal proof and a clearer statement of assumptions are needed to substantiate this claim.\n\nW2. Section 3.4 describes a method to protect activations that resembles secure multi-party computation [1], lacking novelty. Its effectiveness is also questionable when only one adapter is present.\n\nW3. The proposed protection mainly focuses on labels. In practice, data such as personal identifiers may pose a greater risk than labels. For example, knowing (a) Alice's salary (label) is included in the database is considered a more serious leakage than knowing (b) someone earns a salary of 3.2k. The manuscript should explore if the proposed method can also protect other sensitive features.\n\n**References**\n\n[1] Du, Wenliang, and Mikhail J. Atallah. \"Secure multi-party computation problems and their applications: a review and open problems.\" Proceedings of the 2001 workshop on New security paradigms. 2001.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nThere elaboration on limitations is insufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper addresses privacy leakage during API-based Parameter Efficient Fine-Tuning (PEFT). Their designed P3EFT is a multi-party split learning algorithm that leverages PEFT adjustments to uphold privacy with minimal performance overhead. Their method proves competitive in both multi-party and two-party setups while achieving higher accuracy.

STRENGTHS:
Researching API-based fine-tuning for large models is an intriguing topic, especially considering that many clients face challenges loading such large models due to size and computational constraints. In this scenario, privacy concerns regarding client data become paramount. This paper aims to mitigate potential privacy leakage by obfuscating gradients and parameters communicated during transmissions.

WEAKNESSES:
Their approach shows limited privacy improvement compared to the scenario  Without LoRAs, as indicated in Tables 1, 2, and 3, thereby restricting the overall benefits.

QUESTIONS:
1. What is the justification for using n=2 in line 332 for all the datasets?
2. How does the server benefit from this Split learning? How does the inference work after fine-tuning?
3. Address weaknesses.

LIMITATIONS:
The paper addresses limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

comments: SUMMARY:
This paper proposes an algorithm to preserve the label privacy while achieve good accuracy in the split learning regime. The algorithm is used in parameter-efficient fine-tuning and empirically tested on some language models.

STRENGTHS:
The paper clearly presents its motivation and contribution. The modification on the back-propagation is reasonable and empirically effective across three models and different attacks that are tested

WEAKNESSES:
The main concern is the scalability of this method. For one iteration, the number of backpropagation is m (at least 2), which is too slow even for PEFT. The computation cost of PEFT is 2/3 of full training so if m=2, the total cost of this method is 4/3 of full training.

Looking at the code, there are 7 hyperparameters introduced by this method, which may be hard to use in practice. I would suggest the authors fix some hyperparamters that the algorithm is robust to, to reduce the number of tunable hyperparameters.

Also the experiment results on SST2 show a severe leakage around 10% compared to without LoRA (even though this is relatively weaker than other methods).

QUESTIONS:
Have the authors considered other PEFT like linear probing and BiTFiT?

LIMITATIONS:
As in its current presentation, the method is limited to language models, split learning (two parties), and label privacy. The empirical evidence is limited to text classification (specifically, this method does not apply to natural language generation where LLAMA is originally trained for) and LoRA. Each limitation can be relaxed, e.g. extending to vision models, data reconstruction, additional PEFT, etc.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14