Label Privacy in Split Learning for Large Models with Parameter-Efficient Training

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 09:47:30

Abstract: As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. 
These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model.
While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure.
This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model.
In this study, we systematically search for a way to fine-tune models over an API *while keeping the labels private*.
We analyze the privacy of LoRA, a popular approach for parameter-efficient fine-tuning when training over an API.
Using this analysis, we propose P$^3$EFT, a multi-party  split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead.
To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that P$^3$EFT is competitive with existing privacy-preserving methods in multi-party and  two-party setups while having higher accuracy.

Main paper: 
 introduction: 
 One of the main reasons behind deep learning success is its ability to transfer knowledge between tasks [34] . When training a model for any particular problem, it is common to reuse previously trained models from other, related problems. In the past, this was typically done by downloading pre-trained model weights from public hubs, then fine-tuning the said models on the downstream task. However, as models grow larger and more compute-intensive, fine-tuning them locally becomes an increasingly difficult task. Furthermore, many recent models are not released, but instead made available as proprietary services.When a model cannot be fine-tuned locally, many practitioners opt instead for the so-called fine-tuning APIs [27, 16, 6, 26] . These APIs are web services that host one or several pre-trained models and allow clients to perform limited fine-tuning. More specifically, APIs usually allow their clients to run parameter-efficient fine-tuning (PEFT), such as LoRA [15] or Prefix-tuning [21] . These techniques allow adapting a model to a dataset while training a relatively small number of additional weights, which is particularly important for large language or image generation models that have billions of parameters.Although the fine-tuning APIs can be convenient, they also introduce new risk in terms of data privacy.When a client uses such API to train on sensitive data, they need to ensure that their data will stay private [7] . This is particularly important when dealing with patient's medical records, personal user data or trade secrets [24, 19] . The two main threats to data privacy are that the API provider obtains the private data and that a third party intercepts data in transit. Therefore, data privacy is Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. not guaranteed even if the API provider is trusted. Several recent works propose LLM fine-tuning protocols that establish a certain level of privacy for multi-party fine-tuning [42, 7, 22] . Unfortunately, these algorithms work for a narrow class of fine-tuning algorithms or assume that a client can run LLM training locally using an obfuscated version of the model, provided by a remote server [42] .As a result, these algorithms are impractical for our use case of fine-tuning over an API. The few algorithms that are suitable for API fine-tuning guarantee the privacy of input tokens [22] , meaning that the attacker can infer private training labels.In this work, we seek to alleviate this problem by designing a two-party fine-tuning protocol that performs standard parameter-efficient fine-tuning with privacy guarantees. We formulate our protocol as a special case of split learning (or vertical federated learning), where one side (server) holds the pre-trained model and the other (client) has private training data. More specifically, we focus on the privacy of client's training labels. While input privacy is often crucial, there are scenarios where input data is publicly available, such as social media user pages. In these cases, labels could include ad clicks (known to the social network) or financial information (known to a bank that matches social profiles to its internal records). This example further justifies the use of LLMs, as social media pages often contain substantial amounts of text, and LLMs excel at processing long-context data.Instead of developing a specific privacy-preserving architecture, we seek algorithms that can work with popular existing models and PEFT algorithms. Furthermore, our approach relies on the properties of parameter-efficient fine-tuning. Notably, since the adapters are compact, both parties can maintain multiple sets of adapters and swap between them with relative ease. This allows us to design a PEFT-specific algorithm that can solve its task more effectively than general split learning strategies [18] .We summarize our main contributions as follows:• We analyze Low-Rank Adaptation, a common parameter-efficient fine-tuning algorithm, from the perspective of label privacy in the split learning setup. We observe that, despite fine-tuning less than 0.1% of model parameters, PEFT algorithms leak client's training labels against simple attacks that work for modern pretrained transformers.• Based on our analysis, we formulate a framework for privacy-preserving parameter-efficient fine-tuning (P 3 EFT). This framework leverages the properties of PEFT to obfuscate the gradients and parameters communicated during fine-tuning with little impact on the finetuned model quality.• To verify the practical viability of P 3 EFT, we conduct experiments on popular real-world PEFT workloads 1 . Specifically, we fine-tune DeBERTa-v2-XXL [13] , Flan-T5-Large [4] and LLaMA-2 7B [35] on a set of standard language understanding problems. We find that, compared to prior split learning algorithms, P 3 EFT can maintain label privacy throughout training with a significantly smaller accuracy drop.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See our main contributions in 1 and Section 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
federated learning and split learning: 
 Privacy preservation in machine learning has been a subject of active study within several frameworks.An important branch of privacy-preserving learning methods is federated learning, or FL [24] , which can be broadly described as an approach allowing several parties to train a model jointly without sharing their private data. In particular, vertical federated learning [12, 43] targets the scenario where different features (including the label) of each training instance are kept by different parties.One of the most popular approaches to vertical FL for neural networks is split learning [10, 37] ,where each party stores its part of the overall model. To train the model in such an approach, it is only necessary to transfer the intermediate activations and the gradients between layers, while the data itself is stored at the premises of the participant hosting each layer. In this work, we focus on the two-party formulation of split learning, where one side stores the features for each example and another one stores the labels.Recent works have investigated the setting of two-party split learning from the label leakage perspective [38, 28] : because the label party needs to pass the gradients of the loss function to the non-label party, it is possible for the latter party to deduce the labels by inspecting the gradients or activations or by hijacking the training procedure. Li et al. [18] provide a set of attack methods that allow recovering private labels and propose a defense mechanism that injects noise into the gradients; however, they test the approach on pretraining smaller models, and we study finetuning large models on private downstream data.The majority of large neural networks today are not trained with a specific task in mind: instead, they are pretrained on a general objective and then adapted for the downstream problem. Importantly, the growth in the size of foundation models has led to the increased popularity of parameter-efficient finetuning (PEFT) methods that adapt the model to a given task by training a small number of task-specific parameters. There are several prominent approaches to parameter-efficient finetuning, ranging from trainable prompts [21, 11] , to residual adapters [14, 29] . We focus on Low-Rank Adaptation (or LoRA, 15), one of the most popular PEFT methods that adds extra parameters to each weight matrix in the form of a low-rank factorization (see Appendix C for a more detailed description).Such formulation allows LoRA adapters to be merged into the original weights after finetuning; this ability, combined with the simplicity of the method, has made LoRA a broadly popular approach in multiple domains. Still, the approach we propose can be applied to any PEFT method.Several recent lines of work explore the problem of fine-tuning LLMs with privacy guarantees [44, 31] .Zhao et al. [46] analyze the viability of prompt tuning for federated learning, and Zhang et al. [45] , Liu et al. [23] study PEFT algorithms in the setting of horizontal federated learning, that is, where multiple users train a shared model on their local private data. Another, more relevant research direction considers private fine-tuning in a vertical federated learning scenario, where participants hold different model layers [22, 40] . Most of these studies leverage the idea of differential privacy to prove an upper bound on how much information is leaked [8] . Unfortunately, these upper bounds are typically loose and do not match practical observations for real models. Furthermore, the majority of these studies only guarantees privacy of specific parts of the training procedure: for instance, Li et al. [22] only protects the input features, and not labels or model parameters. Finally, Xiao et al. [42] presents an alternative algorithm that protects client data by running the entire fine-tuning on client side by emulating the server-side model layers. While this approach is more holistic, it assumes that clients can run fine-tuning locally, which makes it impractical for many real-world users of LLM fine-tuning APIs. The primary distinction between our work and these studies is that we investigate parameter-efficient adaptation in the setting of split learning: we aim to finetune a model without disclosing the labels of examples to the model provider.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss potential limitations of the method in Section 1, Section 3 and Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
privacy-preserving parameter-efficient fine-tuning: 
 In this section, we analyze the privacy of parameter-efficient fine-tuning and propose a protocol for two-party parameter-efficient fine-tuning with the desired privacy guarantees. We begin by analyzing the privacy of API fine-tuning with popular PEFT algorithms in Sections 3.1 and 3.2. Then, in Section 3.3, we formulate a protocol for privately computing gradients over fine-tuning APIs. Finally, we formulate the full P 3 EFT protocol in Section 3.4.To analyze the privacy of API fine-tuning, we first need to formulate a common framework for this type of APIs and develop private learning protocols. This step is important, because existing fine-tuning APIs greatly vary in what they offer to the client: from closed APIs that require users to submit their full training data [27] to more flexible APIs where clients can run individual training steps [20, 2, 30] . Similarly to most existing works on split learning, we focus on the latter type of APIs that allows clients to run individual forward and backward passes over a remote model. Thus, a client can use these APIs to obtain the training gradients for their PEFT adapters, then update adapters locally with any optimization method. In our work, we adopt this archetype of fine-tuning API as it offers sufficient flexibility to develop privacy-preserving algorithms.We formulate fine-tuning over an API for two or more parties: a client, and one or several servers.The client owns a training dataset with inputs X and labels Y . In turn, each server has the same pre-trained model h(x i , θ) ∈ R d . Note that the parameters θ denote not the pre-trained modelStep: 0Step: 1000Step: 4000Step: 16000Step: 0Step: 1000Step: 4000Step: 16000 weights, but the trainable adapter weights for a certain PEFT algorithm. A model can encode an inputx i ∈ X and produce a d-dimensional vector of activations that depend on the learned adapter weights θ.To allow fine-tuning, a server offers two API methods:1. forward(x, θ) → h(x, θ) that computes model activations on input x using adapter weights θ;2. backprop(x, θ, g h ) → g θ that receives gradients of an arbitrary loss function w.r.t. modelactivations g h = ∂L(h(x,θ)) ∂h(x,θ)and returns the gradients w.r.t. adapter parameters, g θ = ∂L(h(x,θ)) ∂θ.We further assume that both forward(•) and backprop(•) APIs are stateless and deterministic, i.e.calling the same API method multiple times (or on multiple servers) with the same inputs produces identical results. Thus, if the model uses dropout or any other form of non-determinism, we assume that clients provide the random seed as a part of x.To fine-tune a model with this API, a client can initialize adapters locally, alongside with a small task-specific head2 , then train both adapters and the head. For each training batch (x, y) ∈ D, a client calls forward(x, θ) to compute feature representations, then predicts with local "head" and computes task-specific loss function L. After that, a client performs backward pass: first, it computes gradients w.r.t. local head inputs g h = ∂L ∂h , then passes those gradients to a remote server via backprop(x, θ, g h ) API call to compute gradients w.r.t. ∂L ∂θ . Finally, a client updates both θ and local "head" parameters using the optimizer of choice.Before building more advanced algorithms, let us analyze the privacy of client's labels under standard fine-tuning. We consider an "honest, but curious" attacker model. This means that the server will faithfully run the forward and backprop computations as requested by the client without changing the results. Furthermore, we assume that servers are independent and do not communicate client's data between each other. However, a server can recover client's labels by performing arbitrary computations using any information it receives from the client. When training in this way, a client does not directly communicate training labels to the server. However, it communicates inputs, adapter parameters, and gradients. Furthermore, the server communicates input representations that can be intercepted by a third party.In Figure 1 , we train a DeBERTa-v2-XXL model on the SST-2 [32] sentiment classification dataset.The top row depicts the gradients g h communicated by the client when calling backprop(•) at different training stages. In the bottom row, we similarly track activations h(x, θ) that server may compute based on the specified x, θ. We defer further additional figures and details to Section 4.1.As not reveal labels right away (at least not against this attack). However, they gradually "leak" private label information during training. Informally, it appears that the training gradients gradually pull apart the feature representations for each label, until eventually they turn into separate clusters. From an information-theoretic perspective, knowing just one vector of gradients or trained activations allows the attacker to learn all but one bit3 of information about client's private labels.h 1 W 1 h 2 W 2 h n W n 𝜃 1 → → → ⊙ = 𝜃 2 → → → ⊙ = 𝜃 n → → → ⊙ 𝚺= h' 𝓛 … … … … … …To summarize, leaving any one data source unprotected (gradients, activations or parameters) would already compromise label privacy. However, we found that gradients and activations require different means of protection.In this section, we formulate an algorithm for "anonymizing" the gradients communicated over a single training step with arbitrary PEFT type. Several prior works approach this by modifying the training objective or model architecture. However, when dealing with a real-world PEFT workload with optimized hyperparameters, changing the model or loss function often results in reduced model accuracy 4 . Thus, we seek an algorithm that preserves both model and training objective.We design our algorithm based on an observation that backpropagation is conditionally linear in output gradients, even when the model itself is nonlinear. Formally, if we take a model h(•, •), a fixed set of trainable parameters θ and input samples x, the backprop function5 computes backprop(x, θ, ∂L ∂h(x,θ) ) = ∂L ∂θ . For convenience, we shorten it to backprop(x, θ, g h ) = g θ , whereg h = ∂L ∂h(x,θ)represents the gradients of some objective function with respect to model activations (outputs), and g θ = ∂L ∂θ are gradients of the same objective function w.r.t. trainable parameters. In this notation, backprop is linear in terms of g h for any fixed x, θ.This becomes self-evident if we view backprop as multiplying g h by the Jacobian of model outputs w.r.t. trainable parameters, ∂h(x,θ) ∂θ . If x, θ are constant, the Jacobian is also constant, and backprop is a linear operator: backprop(x, θ, ∂L ∂h(x, θ)EQUATIONThis observation allows us to design a private backpropagation protocol. To illustrate this protocol, let us first consider a distributed API with two identical independent servers that offer backprop API. Then, for arbitrary vector z, we can rewrite backprop(x, θ, g h ) as backprop(x, θ, g h +z)+backprop(x, θ, g h -z).During API fine-tuning, we obtain backprop(x, θ, g h + z) using an API call to server EQUATIONThen, for each ĝi h , client computes backprop(x, θ, ĝi h ) as m parallel API calls. Once this is done, client recoversEQUATIONNote that the client does not reveal α 1 , . . . , α m to anyone.The resulting procedure is formulated in Algorithm 1. This algorithm is conceptually similar to the secure aggregation protocol for conventional (horizontal) federated learning [1] . This protocol allows clients to average their local vector with peers while keeping each individual vector provably private. Similarly to our scheme, clients perturb the vector in such a way that the average of perturbed vectors remains the same. Unlike Bonawitz et al. [1] , our protocol privately backpropagates through a server-hosted model by leveraging the conditional linearity of the backpropagation operator.Algorithm 1 private_backprop -Privacy-Preserving Backpropagation (from the client's perspective) g θ = m j=1 α j • ĝj θ 7: Return: g θThe private backpropagation algorithm can allow client to safely compute gradients once, but, in practice, client usually needs to run many consecutive steps. This creates an additional vector of attack: if the same server receives two sets of parameters θ t , θ t+1 , they could potentially recover g θ by inverting the optimizer.In the simplest case, if the server somehow knows that the client computes θ t+1 = θ t -η • g θ , then they can compute g θ = (θ t -θ t+1 )/η. While g θ does not necessarily leak private labels, a server could, in some cases, use g θ to recover g h , either fully (e.g. if Jacobian is invertible), or partially.The client has two ways to prevent this attack. The first one is to ensure that no single server runs backprop on two consecutive steps. This is easy to do in decentralized systems where there are many potential servers. However, even when there is a single server, they could be required to set up multiple trusted execution environments [25] . A more risky alternative is to ensure that the gradients cannot be reversed from consecutive parameters: randomize initial optimizer statistics or add noise to parameters. This solution is easier, but it can slow down training in some cases.To summarize, we formulated a procedure that allows a client to compute gradients privately for any given model and PEFT type. Furthermore, since Equation 3 recovers true gradients, this obfuscation method does not affect the training dynamics. However, as we have shown in Section 3.1, gradients are not the only source of privacy leakage.The other major attack vector are training activations. As the model fits to training data, it's intermediate activations h(x, θ) allow attackers to recover labels, e.g. by clustering (see Figure 1 ).To combat this issue, we take advantage of the fact that PEFT has few trainable parameters. Instead of learning just one set of trainable parameters, a client creates n independent adapter sets θ 1 , ..., θ n .Note that this does not require n unique servers: a single server can run multiple sets of adapters. Furthermore, a client can alternate between using different servers for the same adapters. During forward pass, the outputs of different adapters are mixed together using randomized mixing weightsEQUATIONOverall, we design this model in such a way the combined model h ′ can predict the labels, but the adapters h(x, θ i ) do not allow predicting these labels without knowing the mixing weights W. The mixing weights are generated such that initial activations h ′ (x, . . . ) are equal to mean h(x, •) for allx. To achieve this, we generate W as follows: first, we generate n• (n -1)/2 d-dimensional random vectors ξ i,j ∈ R d ∀i ∈ [1, n], j ∈ [i + 1, n].Then, we add them up in the following way:EQUATIONHere, e stands for a vector of all ones. The purpose of these mixing weights is to ensure that the gradients w.r.t. individual h(x, θ i ) are obfuscated, but the averaged model behaves the same as regular PEFT adapter. To illustrate this, consider n=2 identical LoRA adapters θ 1 , θ 2 . During the first training step h(x, θ 1 ) = h(x, θ 2 ). Therefore,EQUATIONHowever, the two adapters will learn different functions as they receive different gradients. From the first update on, h ′ will be equal to an average of adapter predictions.Finally, to ensure that individual adapters h(x, θ) do not accidentally "learn to leak" labels, we maintain this over the course of training with a privacy regularizer inspired by [9] . This ensures that it is impossible to predict labels from individual adapters h(x, θ i ). Intuitively, on each training step, client fits n linear "heads" that learn to predict labels y from h(x, θ i ), then performs an adversarial update of θ i to prevent the "head" from predicting y. Formally, each of n "heads" minimize the same objective function as the full model. For instance, if the full model solves multi-class classification, each head is trained to minimize cross-entropy:EQUATIONwhere y is one-hot encoding of the correct class.The whole adversarial update takes place locally on client's side, using the same h(x, θ) it uses for the main training objective. The resulting procedure appears complicated but it typically takes negligible time compared to running the large pre-trainied model h(x, θ). Furthermore, since adversarial "heads" are linear, minimizing the objective above is done with standard logistic regression solver.To summarize, our approach combines the two proposed ideas: we use the private backpropagation algorithm from Section 3.3 to protect the gradients, then trains a mixture of adapters in such a way that obfuscates learned activatons leaking labels. The resulting procedure is described in Algorithm 2.In the next section, we will evaluate the efficacy of P 3 EFT on popular NLP benchmarks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We provide attached code.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 The main goal of our study is to find a practical method of private fine-tuning that would scale to large models. Because our approach leverages parameter-efficient fine-tuning techniques, we evaluate P 3 EFT with fine-tuning Transformer models on popular NLP benchmarks that these techniques were designed for.To that end, we chose three pre-trained models: DeBERTa-XXLarge [13] , Flan-T5-Large [4] and LLaMA-2 7B [35] . We train these models on several datasets from the GLUE benchmark [39] :SST-2 [32] , MNLI [41] and QNLI.For this experiment, we train DeBERTa-XXLarge on SST-2 dataset using LoRA adapters with hyperparameters from [15] . First, we train the model locally and track model activations h and gradients w.r.t. those activations. We apply principal component analysis to them and plot the firstStep: 0Step: 1000Step: 4000Step: 16000Step: 0Step: 1000Step: 4000Step: 16000 Next, we run the same experiment using privacy-preserving backpropagation as defined in Section 3.3.We use n = 2 with the noise variance set to 1000. As expected, we observed the same learning curve as with normal training. However, instead of sending gradients w.r.t. activations to the server, a client uses specially crafted random noise vectors that are not informative. In Figure 3 (bottom) we plot the same kind of individual gradients as in the top row, except that we visualize the gradients computed by the first of the two servers. Finally, we train XGBoost [3] with default hyperparameters to predict labels given the noisy gradients (pre-PCA): the resulting classifier is able to fit the training data perfectly, but has at most 50.4% accuracy on a balanced test set.Next, we evaluated the entire P3EFT algorithm. To control tasks and model type, we examined DeBERTa and Flan-T5 across all four datasets mentioned above, in addition to evaluating LLaMA on SST2 and QNLI datasets. For each setup, we compare against three baselines:• Without LoRAs. In this baseline, the client gathers h activations at the beginning (with no adapters), then proceeds to train local "head" layers using these activations. This method cannot leak information about training labels except for what is stored in X.• Regular fine-tuning (Regular FT) refers to training a single LoRA adapter normally. This baseline represents an upper bound on model accuracy, but lacks privacy.• Distance Correlation (DC). Our re-implementation of the distance correlation defense formulated in [33] for Transformer models.For each algorithm, we evaluated a task-specific metric (accuracy or F1), as well as the privacy leakage value for the 3 following measures:• Spectral attack AUC -a measure of vulnerability to an attack proposed in [33] , measured as classifier ROC AUC: lower value corresponds to better privacy.• Norm attack AUC -vulnerability to a variant of attack proposed in [18] , measured as classifier ROC AUC (lower is better). Despite the initial proposal of this approach for attacking gradients, we observed that it is also well-suited for attacking activations.• K-means accuracy -vulnerability to clusterization attack, measured in the percentage of correctly clustered activations, lower is better.For all setups, we report the worst (least private) value among these metrics throughout the entire training period as a measure of privacy leakage, because it is the worst possible scenario that matters from the client's perspective. For DC and P 3 EFT, we specify the values for the best configuration in terms of the utility-privacy trade-off. See details in Appendix A. We also report adjusted standard deviations for the two privacy aware algorithms: P 3 EFT and DC. To do so, we run the full training procedure from scratch with 3 random seeds. The results for DeBERTa are presented in Table 1 . To improve reproducibility, we reuse the hyperparameters from original paper, with the exception of the LoRA dropout value. We disable dropout because it interferes with the mixing weights (5) . In preliminary experiments, we observed that with dropout enabled, both our algorithm and DC begin to perform significantly worse.We use n = 2 adapter sets for P 3 EFT for all datasets and adhered to the same approach for the other models as well. Overall, P 3 FT achieves nearly the same accuracy as traditional (non-private) fine-tuning, outperforming the DC-based algorithm in terms of accuracy given the same privacy level.On the MNLI dataset, we could not find the hyperparameters for DC that ensure stable training while maintaining privacy. Meanwhile, P 3 EFT maintains consistent performance on this task with a slight drop in quality.Table 2 a reports evaluation for the Flan-T5 base model [4] . For this model, we adapt the exact same hyperparameters as in the previous evaluation with DeBERTa-XXLarge. Compared to DeBERTa, these results are more closely matched. Both both our algorothm and DC consistently solve all three tasks, but P 3 EFT slightly outperforms DC in terms of privacy. To evaluate how our algorithm scales to larger models, we also fine-tune Llama-2 7B [35] on SST2 [32] and QNLI [39] datasets. For these evaluations, we use LoRA hyperparameters that Hu et al. [15] used when fine-tuning GPT-3, with several changes inspired by Dettmers et al. [5] . Namely, we use the NF4 weight format, apply LoRA to both attention and MLP layers with rank 16. We fine-tune both tasks with maximum context length of 512 and weight decay 0.01. Table 3 summarizes our results: for QNLI, P 3 EFT achieves somewhat better privacy-accuracy trade-off. On SST2, P 3 EFT shows similarly favorable trade-offs while DC struggles to preserve privacy. 
conclusion and discussion: 
 In this work, we analyze privacy-preserving fine-tuning of large neural networks in the context of parameter-efficient fine-tuning and the two-party split learning setting. We show that while standard fine-tuning suffers from label leakage even in the parameter-efficient case, it is possible to leverage the efficiency of PEFT to alter the procedure without any significant performance drawbacks. We test the resulting method, named P 3 EFT, on a range of pretrained language models and multiple datasets,showing that it is competitive with a strong baseline in terms of label privacy while having higher task performance.In future work, it is natural to explore how this approach can be extended to establish holistic privacy in both labels and inputs. This problem can be approached from two directions: either adapt the ideas of P 3 EFT for input privacy, or combine it with an existing work like [22] . Another important direction for future research is exploring the privacy of the long-term client-provider interaction. In a typical real-world use case of API fine-tuning, a client performs multiple training runs on overlapping data and hyperparameters. This could open additional attacks vectors that combine information from multiple training runs. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 4, Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report error bars in main results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We describe a potential social impact in Introduction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: We do not describe the safeguards Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix: In P 3 EFT and Distance Correlation methods resulting loss L function can be viewed in the formL = L m + α • L r ,where L m -main task loss, L r -regularizer and α is a coefficient that controls the tradeoff between these two losses. The selection of this coefficient affects the final performance of the model. Therefore, to find the best configurations for both methods, we iterated through this hyperparameter using a grid search.We started with α = 1 and then altered it with a multiplicative step of 10 1 2 . Values were discarded if the quality did not exceed that achieved by solely training the classifier without LoRA. This criterion was adopted because such outcomes would suggest the method's inability to outperform training scenarios in which the server does not engage with the labels whatsoever. Additionally, we excluded values that led to unstable training. By this, we mean instances where, although the model initially trained on the primary task, at some point, the regularizer began contributing significantly more, and the utility value dropped to the starting value. We observed this issue for the DC method with DeBERTa on the MNLI. From the remaining values, we aimed to choose the one that offered the lowest privacy leakage. The final hyperparameter values for P 3 EFT can be found in the Table 4 and for DC in the Table 5 .Table 4 : Regularization parameter α for the P 3 EFT method. The values in the table represent powers of the 10 1 2 . SST2 QNLI MNLI DeBERTa XXLarge 1 1 1 Flan-T5-Large -1 1 1 LLaMA-2 7B 0 0 -Below, we define the full P 3 EFT algorithm. In Algorithm 2, main_loss is the task-specific objective e.g. cross-entropy; reg_loss is the adversarial regularizer described in Section 3.4. We denote client-side model "head" as f (h, ψ t ), where ψ represent trainable head parameters. Finally, opt_step function performs a single gradient descent step with a task-specific optimizer, typically Adam [17] .Algorithm 2 P 3 EFT -full training algorithm 1: Input: dataset D = {X, Y }, n > 1 number of adapters, α ≥ 0 -regularizing weight, m > 1 number of obfuscated gradients 2: Initialize head ψ 0 , mixing weights W i and adapters θ 0 i , i = 1, n 3: for t = 0, 1, . . . , T -1 do for i = 1, . . . , n do 6:h t i = h(x t , θ t i )▷ by server 7:l i = reg_loss(h t i , y t ) ▷ by client 8:end for 9:h ′ = n i=1 W i ⊙ h t i 10: l = main_loss(f (h ′ , ψ t ), y t ) 11: L = l + α • n i=1 l i 12:for i = 1, . . . , n do 13:g h = ∂L/∂h t i ▷ Client performs partial backprop 14:g t i = private_backprop(x, θ t i , g h , m)15:EQUATIONend for 17:ψ t+1 = opt_step(ψ t , ∂l/∂ψ t , t) 18: end for 19: Return: ψ T , θ T 1 , . . . , θ T MFor convenience, we provide a brief summary of fine-tuning with LoRA [15] . This PEFT method was originally designed for fine-tuning large pre-trained language models on downstream NLP tasks.These language models are typically based on the Transformer architecture [36] , where most trainable parameters are allocated to linear layers in multi-head self-attention and feedforward blocks.In the first stage of LoRA fine-tuning, user augments the model with adapters. To do so, a user goes over linear layers in transformer blocks and adds two trainable matrices, A and B that affect this layer's forward pass. Let W i × x + b i be the original layer with n inputs and m hidden units. Here,W i ∈ R m×n is a pre-trained weight matrix, b i ∈ R m is a pre-trained intercept vector and x ∈ R nrepresents a vector of inputs to this particular layer. During the forward pass, a layer with LoRA adapter computes Once all adapters are initilized, the user trains all A i and B i matrices of the model, while keeping the rest of the weights frozen. This way, only a small faction (less than 1%) of model weights are updated. Once the training is over, the learned adapters A i and B i can be merged into the main weights (W i := W i + A i × B i ) or used separately.W i × x + b i + B i × A i × x, or equivalently, (W i + B × A) × x + b i .LoRA adapters are designed with two objectives in mind: i) to allow fine-tuning models in limited GPU memory and ii) to allow inferencing many fine-tuned models using one inference server. When fine-tuning, LoRA achieves small memory footprint due to the fact that user does not need to computeNeurIPS Paper ChecklistThe code is available at github.com/anonymousauthor56/P3EFTA linear layer that predicts class logits or regression target.The missing bit corresponds to attacker not knowing which cluster corresponds to label "1".We validate this empirically in 4.2.This is the same as the backprop API defined in Section 3.1. 


Decision: This paper proposes an MPC algorithm for protecting the privacy of labels when using parameter efficient fine tuning techniques in a split learning setting.

The reviewers agree that this work is highly important and that its motivation is clear. Further, reviewers have agreed the empirical results show the promise and efficacy of the approach. Yet, there are some key concerns that need to be addressed: i) in some cases, it appears that there is severe leakage compared with the baseline (“Without Loras”), ii) the per-step computational cost exceeds that of full training despite using PEFTs, iii) there are lacking formal guarantees, like DP, and iv) the setting of label privacy is perhaps a bit limited. I recommend the authors consider addressing these weaknesses.

Reviews: review: paper_summary: paper_summary: This paper proposes an algorithm to preserve the label privacy while achieve good accuracy in the split learning regime. The algorithm is used in parameter-efficient fine-tuning and empirically tested on some language models.
strength_weakness: strength_weakness: The main concern is the scalability of this method. For one iteration, the number of backpropagation is m (at least 2), which is too slow even for PEFT. The computation cost of PEFT is 2/3 of full training so if m=2, the total cost of this method is 4/3 of full training.

Looking at the code, there are 7 hyperparameters introduced by this method, which may be hard to use in practice. I would suggest the authors fix some hyperparamters that the algorithm is robust to, to reduce the number of tunable hyperparameters.

Also the experiment results on SST2 show a severe leakage around 10% compared to without LoRA (even though this is relatively weaker than other methods).
questions: questions: Have the authors considered other PEFT like linear probing and BiTFiT?
limitations: limitations: As in its current presentation, the method is limited to language models, split learning (two parties), and label privacy. The empirical evidence is limited to text classification (specifically, this method does not apply to natural language generation where LLAMA is originally trained for) and LoRA. Each limitation can be relaxed, e.g. extending to vision models, data reconstruction, additional PEFT, etc.
paper_summary: paper_summary: This paper addresses privacy leakage during API-based Parameter Efficient Fine-Tuning (PEFT). Their designed P3EFT is a multi-party split learning algorithm that leverages PEFT adjustments to uphold privacy with minimal performance overhead. Their method proves competitive in both multi-party and two-party setups while achieving higher accuracy.
strength_weakness: strength_weakness: Their approach shows limited privacy improvement compared to the scenario  Without LoRAs, as indicated in Tables 1, 2, and 3, thereby restricting the overall benefits.
questions: questions: 1. What is the justification for using n=2 in line 332 for all the datasets?
2. How does the server benefit from this Split learning? How does the inference work after fine-tuning?
3. Address weaknesses.
limitations: limitations: The paper addresses limitations.
paper_summary: paper_summary: This study addresses the privacy concerns associated with the fine-tuning of Large Language Models (LLMs), focusing on SplitNN. It explores how gradients and activations can leak data, potentially allowing attackers to reconstruct original data sets. In experiments, the proposed method reduces label leakage while maintaining minimal utility loss.
strength_weakness: strength_weakness: W1. The claim that backpropagation is "conditionally linear" is not sufficiently rigorous. The manuscript suggests that $\text{backprop}(x, \theta, g_h+z)+\text{backprop}(x, \theta, g_h−z) = \text{backprop}(x, \theta, g_h)$ under the assumption that $\theta$ is constant. However, $\theta$ updates during each backpropagation, invalidating this assumption. Moreover, swapping the order of $\text{backprop}(x, \theta, g_h+z)$ and $\text{backprop}(x, \theta, g_h−z)$ could lead to different outcomes. Formal proof and a clearer statement of assumptions are needed to substantiate this claim.

W2. Section 3.4 describes a method to protect activations that resembles secure multi-party computation [1], lacking novelty. Its effectiveness is also questionable when only one adapter is present.

W3. The proposed protection mainly focuses on labels. In practice, data such as personal identifiers may pose a greater risk than labels. For example, knowing (a) Alice's salary (label) is included in the database is considered a more serious leakage than knowing (b) someone earns a salary of 3.2k. The manuscript should explore if the proposed method can also protect other sensitive features.

**References**

[1] Du, Wenliang, and Mikhail J. Atallah. "Secure multi-party computation problems and their applications: a review and open problems." Proceedings of the 2001 workshop on New security paradigms. 2001.
questions: questions: See Weakness
limitations: limitations: There elaboration on limitations is insufficient.
paper_summary: paper_summary: The paper revisited the problem of label leakage in split learning in the context of fine-tuning large models with parameter-efficient training. Based on modern use cases, they proposed two privacy-preserving protections for gradients and activations during split learning. The proposed methods are evaluated on several large models including Llama2-7B, fine-tuned using LoRA and full model fine-tuning.
strength_weakness: strength_weakness: 1. There exist several related works discussing the attacks and defense regarding the label leakage in split learning. The authors may need to compare the differences between the proposed methods and previous literature. The evaluation part lacks the comparison to existing privacy-preserving solutions over label leakage and some trivial solutions such as directly applying differential privacy, which is easy to implement.
2. In modern use cases of API fine-tuning, apart from the applications of classification, text generation with LLMs and image generation with multimodal models and diffusion are more common cases. And it is very critical to protect labels in these applications. For example, labels in text generation can contain answers to private questions in the private dataset. However, the leakage study and proposed privacy-preserving methods do not apply to these applications.
3. There are some minor writing issues that could be improved. For example, content introducing API fine-tuning and potential privacy concerns can be shortened in the introduction. The paragraph from line 53 to 58 can be reorganized so that it won't leave '[18]' for a whole line. Same thing for line 209. On line 28, write the full name of LoRA before using acronym. The authors are suggested to talk about split learning and no need to raise extra efforts for readers to understand what vertical federated learning is.

[1] Wan, Xinwei, Jiankai Sun, Shengjie Wang, Lei Chen, Zhenzhe Zheng, Fan Wu, and Guihai Chen. "PSLF: Defending Against Label Leakage in Split Learning." In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2492-2501. 2023.   
[2] Kariyappa, Sanjay, and Moinuddin K. Qureshi. "ExPLoit: Extracting private labels in split learning." In 2023 IEEE conference on secure and trustworthy machine learning (SaTML), pp. 165-175. IEEE, 2023.     
[3] Erdoğan, Ege, Alptekin Küpçü, and A. Ercüment Çiçek. "Unsplit: Data-oblivious model inversion, model stealing, and label inference attacks against split learning." In Proceedings of the 21st Workshop on Privacy in the Electronic Society, pp. 115-124. 2022.      
[4] Xu, Hengyuan, Liyao Xiang, Hangyu Ye, Dixi Yao, Pengzhi Chu, and Baochun Li. "Permutation Equivariance of Transformers and Its Applications." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5987-5996. 2024.
questions: questions: 1. From my understanding, split learning usually involves multiple clients and a server. But according to line 137, it is a client and several servers. Can the authors explain about this?
limitations: limitations: Discussed in the last section.

Comments: title: Response
Response to the Rebuttal
Response to Rebuttal
comment: Thank you for your thoughtful rebuttal. It addresses most of my concerns. I have read it carefully and decided to maintain my score.
Thank you for the explanation. I think it would be benefited to add this part related to DP into the future work, in the paper. As my primary concerns about comparison and DP are resolved, I raised the rate by 1, to 6.
I thank the authors for their thorough response. My concerns have generally been addressed. I have raise my rating to 5.

For the theoretical guarantee in the response, I would recommend authors to provide more rigorous and formal probablistic bound like the definition of differential privacy.

In addition, I suggest authors to check the privacy bound/proof of the existing MPC methods that are similar to their proposed one. This can improve the soundness of the proposed approach.
> Is this method specifically designed for LLM fine-tuning? In the original manuscript, it seems apply to all neural networks optimized by gradient descent.

In the original manuscript, we note that our method leverages existing PEFT properties (e.g., **lines 11-12, 53-58, and 243**) and is applicable to any model trained with PEFT. Consequently, the scope is not limited to LLMs (for instance, PEFT is also applied to ViT[2] and diffusion models) or fine-tuning (e.g., see [3]). However, LLM fine-tuning remains the most prevalent application.

Unfortunately, our method is not suitable for arbitrary neural networks. Specifically, the model must possess certain properties typical to PEFT — a large frozen model on the server side and a small number of trainable parameters. If you are interested in more detailed explanations, we would be happy to provide them.

>**Two-party case**:

We are unsure about your exact concern here, so we will try to respond to each point to the best of our understanding.

>the defense of the proposed approach is weakened as the number of parties decrease

We respectfully disagree with this statement. The experimental results with varying numbers of adapter sets ($n$) (see Table 2 in the attached PDF) **do not demonstrate this trend**. For instance, the results on SST2 with $n=4$ are inferior to those with $n=2$.

>The decreasing trend depends on the noise scale.

In Section 3.4, we employ noise solely in the coefficients $\xi$ in Eq. 5. Indeed, as the number of adapter sets ($n$) increases, the standard deviation of this noise increases as $\sqrt{n}$. However, the noise scale can also be directly amplified by increasing the standard deviation of the distribution from which $\xi$ is sampled. In our preliminary experiments, we attempted to vary the magnitude of the noise while keeping $n$ constant, but did not observe a clear correlation with the final results. If the reviewer is interested in this question, we can set up additional experiments to provide an answer.

>A theoretically analysis is expected to demonstrate what values $\mathcal{K}$ would likely invalidate the defense.

We did not use the notation $\mathcal{K}$ in our paper, but we assume this is a typo and the reviewer meant the number of adapter sets, $n$. We do not think that any particular value of $n$ will invalidate the defense. Even with $n=1$, the adversarial regularizer prevents the activations from pulling apart the feature representations for each label (unlike in Section 3.2). Our experiments conducted during the rebuttal phase with $n=1$ confirm this (for example, on SST2, the result with $n=1$ outperforms the DC baseline; see Table 2 for details).

We would also like to emphasize that the rationale for using $n>1$ is described **in line 250**. To elaborate, we aim to create a situation where the activation distribution for each individual adapter is difficult to cluster or attack, yet the weighted sum of activations from Eq. 4 yields a distribution that is "simple" for subsequent classification. Achieving such a scenario for a pre-trained LLM with $n=1$ seems challenging, as we would simultaneously be training the same activation distribution to be both "simple for classification" and "difficult for clustering", whereas during conventional training, these properties typically correlate (see Figure 1).

> Can authors provide specific applications where labels urgently needs privacy protection?

One such example is provided in **Section 1, lines 48-52**. Additionally, there are several canonical examples where label privacy is crucial:
1. Advertising[4]. Consider an advertising platform A and an advertiser company B. Party A can record viewing history, while B possesses the visitor's conversion rate. B's labels remain private as purchases occur exclusively on their site or application.
2. Finance[5, 6]. An invoice agency A contributes invoice-related features, while a bank B provides credit data and SME labels. They collaboratively construct a risk model.

[1] Bonawitz et al. Practical secure aggregation for privacy-preserving machine learning. ACM SIGSAC Conference on Computer and Communications Security 2017.

[2] Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.

[3] Lialin et al. Relora: High-rank training through low-rank updates. ICLR 2023.

[4] Li et al. Label leakage and protection in two-party split learning. ICLR 2022.

[5] Cheng et al. Federated learning for privacy-preserving AI. Communications of the ACM, 63(12):33–36, 2020.

[6] Hu et al. Is vertical logistic regression privacy-preserving? a comprehensive privacy analysis and beyond. arXiv preprint arXiv:2207.09087, 2022.
>highlight this staleness in Section 3.3

>modify Equation 1

We thank the reviewer for their recommendations and will add these clarifications to the next revision of the paper.

> developing theoretical guarantee

Following the reviewer's recommendation, we provide theoretical analysis for privacy guarantees of the `private_backprop` algorithm. We use notations $B$ for `batch_size` and $d$ for `hidden_size`; $h_b, g_b, l_b$ correspond to activation, gradient and loss of the $b$-th batch element; $g_h$ represents a vector of concatenated gradients of all batch elements. We consider binary classification as a task with the minimum number of possible label combinations - $2^B$.

We consider significantly stronger assumptions regarding the attacker's capabilities - namely, a white-box scenario. We assume the server knows the client-side model and, consequently, all possible $2^B$ vectors $g_h$ for different label sets. Thus, the server's task is to determine which of the $2^B$ label sets corresponds to the current batch based on the transmitted vectors.

We investigate the minimum $m$ required to ensure that all $2^B$ sets remain equally probable from the attacking server's perspective in several possible setups:

1. Two non-interacting servers. In this scenario, it suffices to set $m=2$ and send one vector $\xi_i$ to each server $i$. From each server's perspective, all $2^B$ variants have equal probability because for any $\xi_i$ and a given $g_h$, there exists a vector $\eta$ such that $g_h$ belongs to the span of $\xi_i$ and $\eta$.

2. Single server. In this scenario, it is sufficient to set $m=B$.
   
   To show this, we note that for the $b$-th element of the batch holds $\partial l_b/\partial h_b = \partial l_b/\partial p_b \times \partial p_b / \partial h_b$, where $p_b\in \mathbb{R}$ is the head's prediction for activations $h_b$ -  the probability of class 1. $\partial p_b/\partial h_b$ is a constant Jacobian of rank 1 and does not depend on the label value. Thus, both possible vectors $\partial l_b/\partial h_b$ lie in the Jacobian's image and belong to the same one-dimensional subspace.
   
   Therefore, it is sufficient for the client to send a basis vector of the corresponding one-dimensional subspace $\partial p_b/\partial h_b$ for each batch element $b$ (and zero vectors for the remaining batch elements) one by one. Knowing $\alpha_b$, the client can reconstruct the corresponding contribution of $b$-th element to the weight gradient $g_{\theta}$. The server, however, cannot determine which label generated the given gradient for each example, as both lie on the same line.

3. Single server, $m < B$. In this setup, the client is not able to protect all gradients of the batch. Indeed, for $B=3$ and $m=2$, the set of $2^3$ possible gradient combinations cannot be embedded in any $2$-dimensional plane that the client can construct from $2$ vectors (the linear span of these $2^3$ gradients is $3$-dimensional). At most, the client can fully protect $m-1$ labels, while the server will know the remaining $b - m + 1$ labels up to a flip of all labels.

We want to emphasize again that the above results obtained under a white-box assumption, which is significantly stronger than our practical setup. The general case is considerably more challenging for the attacking side, and developing a possible attack or determining the theoretical limits of the attacker's capabilities is a complex task. However, we believe that the theoretical analysis presented above may be a good first step in this direction.

>This paper appears to claim designing a new MPC approach because there is no citation or discussion of the used MPC mechanism.

Thank you for your comprehensive comment. We now have a better understanding of the reviewer's initial concern and will endeavor to provide a detailed response below.

However, we believe there may have been a misunderstanding. **We do not state** that our method for protecting activations, described in Section 3.4 and referenced by the reviewer in W2 of the official review, pertains to MPC.

Concurrently, we acknowledge that our method could be considered as belonging to MPC in a broader sense, as the `private_backprop` algorithm shares common ideas with this field. We **discuss this similarity in Section 3.3, L217-222, and cite prior work [1]** that leverages MPC.

We have also used the phrase "multi-party" in the paper in reference to our method (L11, 15, 38), but **we have not explicitly mentioned MPC**. We employed these terms to highlight the potential benefits of a multiple-server setup when using `private_backprop` with fine-tuning API, which we discuss in detail in L223-235. If the reviewer disagrees with the use of the phrase "multi-party split learning," we are open to considering an alternative taxonomy.

We hope our explanation has helped to resolve the misunderstanding. If the reviewer has any remaining questions regarding this concern, we would be glad to address them.
>can the authors explain why differential privacy is not compared or in other words, is there any particular reason that we do not have the necessity to compare with differential privacy?

Our decision not to include differential privacy (DP) comparisons into the original paper was based on two factors:

1. As noted in [1] (Section 2), differential privacy and its variants are not directly applicable metrics in the context of split learning.

2. We found a lack of existing literature applying label differential privacy **specifically to split learning**, which presented challenges in identifying appropriate differential privacy baselines for comparison.

However, during the review process, you brought to our attention the paper [2]. In this work, the authors developed a framework based on Randomized Response that satisfies Label Differential Privacy. We believe this method could indeed serve as a viable DP baseline that we had not previously identified. 

If the reviewer has any other suggestions for alternative DP baselines, we would be open to incorporating experiments with these in the final version of the paper.

[1] Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. Label leakage and protection in two-party split learning. ICLR 2022.

[2] Wan, Xinwei, Jiankai Sun, Shengjie Wang, Lei Chen, Zhenzhe Zheng, Fan Wu, and Guihai Chen. "PSLF: Defending Against Label Leakage in Split Learning." In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2492-2501. 2023.
Thank you for your thoughtful rebuttal. I appreciate the clarifications provided regarding the privacy trade-offs, particularly how the 'Without LoRAs' scenario represents the best-case privacy option and how your method balances this with significantly improved accuracy. The additional experiments on n=2 and your plan to include an ablation study in the final version address my concerns well. Your detailed explanation of the server’s benefits from split learning and the inference process post-fine-tuning also adds clarity. However, I will maintain my original score as the paper offers a valuable contribution, albeit with some trade-offs that merit further exploration.
Thank the authors for the rebuttal and complement experiments in tight time. My major concern about comparison with baselines  is complemented with experiments. Another question is can the authors explain why differential privacy is not compared or in other words, is there any particular reason that we do not have the necessity to compare with differential privacy?
I thank the authors for their thorough response. However, part of my concerns remain unaddressed.

W1. I now understand that the statement is **correct** because parameters are not updated immediately after `backprop`. Despite this, I still recommend to
- highlight this staleness in Section 3.3
- modify Equation 1 as it does not reflect a linearity property. I guess you may want to put something like $\text{backprop}(a, b, c1) + \text{backprop}(a, b, c2) = \text{backprop}(a, b, c1+c2)$
- developing theoretical guarantee for this mechanism, as the accumulation of Gaussian noise can lead to some probability bound that guide the selection of #steps $m$. (This could be a significant contribution)

W2. I agree that _applying secure multi-party computation (MPC) to split learning for label protection is novel_. However, applying an existing MPC approach is different from designing a new MPC approach. This paper appears to claim designing a new MPC approach because there is no citation or discussion of the used MPC mechanism. However, this proposed new mechanism must be compared with other MPC algorithms to support its novelty. The authors should clearly stated whether they are designing a new MPC method or using an existing one, followed by a comparison to the literature or clear citations.

> The core novelty of our work is an efficient protocol specifically for the task of LLM fine-tuning. 

A followup concern for this response: Is this method specifically designed for LLM fine-tuning? In the original manuscript, it seems apply to all neural networks optimized by gradient descent.

**Two-party case**:
> Originally, we only considered training with multiple adapters for our experiments

Two-party case is not an exceptional case of multi-party split learning. Instead, the defense of the proposed approach is weakened as the number of parties decrease.The decreasing trend depends on the noise scale. A theoretically analysis is expected to demonstrate **what values $\mathcal{K}$ would likely invalidate the defense**.

W3. I understand these are orthogonal topics, but my major concern is whether the focused scenario exists in practice. Can authors provide specific applications where labels urgently needs privacy protection?