Venue: NAACL 2025

# Soft Prompting for Unlearning in Large Language Models

Karuna Bhaila Minh-Hao Van Xintao Wu

University of Arkansas

{kbhaila, haovan, xintaowu}@uark.edu

###### Abstract

The widespread popularity of Large Language Models (LLMs), partly due to their emerging in-context learning ability, has highlighted the importance of ethical and safety considerations for deployment. Motivated by corresponding data protection guidelines, we investigate machine unlearning for LLMs. In contrast to the growing literature on fine-tuning methods to achieve unlearning, we focus on a comparatively lightweight alternative called soft prompting to realize unlearning in LLMs. With losses designed to enforce forgetting as well as utility preservation, our framework **S**oft **P**rompting for **U**nLearning (SPUL) learns prompt tokens that are prepended to a query to induce unlearning of specific training examples at inference time without updating LLM parameters. We conduct a rigorous evaluation of the proposed method, and results indicate that SPUL can significantly improve the trade-off between utility and forgetting for text classification and question-answering. We further validate our method with LLMs of varying parameter sizes to highlight its flexibility and provide detailed insights into the choice of hyperparameters and the influence of the size of unlearning data. Our implementation is available at [https://github.com/karuna-bhaila/llm_unlearning](https://github.com/karuna-bhaila/llm_unlearning).

## 1 Introduction

With evolving transformer models Vaswani et al. (2017) and the availability of massive text corpus, language models are progressing rapidly. The _pre-train and fine-tune_ pipeline has garnered wide popularity, especially since the release of LLMs such as GPT OpenAI (2024) and LLaMA Touvron et al. (2023). However, several ethical and security concerns have been raised due to the presence of private, sensitive, or harmful information in the training data. For example, LLMs can regurgitate personal information Nasr et al. (2023), or mimic harmful and/or hateful behavior as a consequence of such content being prevalent in the data Wen et al. (2023). The non-consented and unwarranted use of copyrighted content for LLM training has also raised significant concerns Eldan and Russinovich (2023); Grynbaum and Mac (2023).

Current policies governing the use and distribution of such models do not encompass all ethical avenues; nonetheless, certain regulations such as California Consumer Privacy Act (CCPA) and GDPR's Right to be forgotten (RTBF) serve as guidelines for organizations to ensure that their operations do not infringe upon user privacy. Specifically, these regulations stipulate that businesses and data collectors provide and exercise an _optout_ mechanism essentially allowing individuals to request the deletion of their data on reasonable grounds. In machine learning literature, these regulations have been conceptualized as machine unlearning Cao and Yang (2015); Bourtoule et al. (2021), which aims to eliminate the influence of unwanted data points on a model's behavior as if they had never been observed during training. Naturally, machine unlearning should be integrated into the LLM pipeline to address previously outlined issues resulting from the presence of sensitive or harmful data in pre-training. However, unlearning in LLMs faces unique challenges due to the inaccessibility of model and pre-training data, and the sheer size of the pre-trained LLMs making re-training practically infeasible. Much of the research in this direction therefore focuses on the fine-tuning approach which involves training all or a subset of LLM parameters to enforce unlearning Jang et al. (2023); Chen and Yang (2023); Yao et al. (2024b); Maini et al. (2024); Yao et al. (2024a).

In contrast, we present a resource-efficient approach to unlearning in LLMs via soft prompting. Soft prompting optimizes a set of task-specific token embeddings that learn contextual information from a corresponding dataset and instruct a frozen LLM during inference Lester et al. (2021); Li and

Liang, 2021). We leverage this ability to modulate LLM outputs using learnable prompts and formulate **S**oft **P**rompting for **U**nlearning (SPUL), a resource-efficient mechanism to achieve LLM unlearning in text classification and multiple-choice question answering (MCQA) tasks. SPUL optimizes soft prompts so that they learn to encode underlying information in the data relevant for unlearning. When prepended to the input tokens of an LLM during inference, the soft prompts guide the LLM towards a _generic response_. We implement a multi-objective loss aligned with specific unlearning goals to facilitate soft prompt learning. SPUL unlearns undesirable outcomes without updating large-scale LLM parameters and can fully capitalize on the language understanding capability offered by the pre-trained LLMs. Consequently, the same pre-trained LLM can be utilized for different unlearning tasks and datasets during inference.

We formulate LLM unlearning as the forgetting of a training data subset composed of benchmark NLP datasets for sentiment classification or MCQA Pawelczyk et al. (2024); Li et al. (2024); Chen and Yang (2023). The unlearning subset could contain data corresponding to entities or sensitive and potentially harmful information. To evaluate whether SPUL can achieve unlearning, we focus on quantifying the predictive performance on the unlearning subset as well as a general retain subset and consider the trade-off between them. In this setting, SPUL can effectively induce forgetting during inference while preserving the pre-trained utility with performance comparable to or significantly better than multiple baselines that implement fine-tuning. We conduct experiments to analyze the influence of SPUL hyperparameters including the contribution of loss components and the size of the soft prompts. We further validate SPUL on multiple pre-trained LLMs with different parameter sizes and using varied sizes of unlearning sets.

## 2 Related Work

### Soft prompting

Soft prompting or prompt tuning emerged as a lightweight alternative to fine-tuning while keeping pre-trained LLM parameters frozen. Motivated by discrete prompts that guide pre-trained LLMs via task-specific instructions or demonstration examples, soft prompting makes prompt design more efficient by employing trainable prompt parameters. Lester et al. (2021) added trainable embeddings to the encoder input sequence of an LLM and achieved performance comparable to fine-tuning on NLP classification tasks with models having over 10B parameters. Simultaneously, Li and Liang (2021) developed the notion of prefix tuning which prepends task-specific prefixes to the input embeddings along with the encoder and decoder inputs of an autoregressive LM and obtained comparable performance for text generation tasks. Liu et al. (2023) concatenated trainable continuous prompts with discrete prompts along with a prompt encoder that maps prompts to model inputs to improve performance on supervised and few-shot tasks. Subsequent research showed that deep prompt tuning achieves comparable performance to fine-tuning across several tasks on models of varying scales by inserting prompts at all layers Liu et al. (2022).

### Unlearning in LLMs

Machine unlearning addresses data protection guidelines by efficiently forgetting training samples corresponding to unlearning requests in place of costly retraining Bourtoule et al. (2021); Cao and Yang (2015); Guo et al. (2020); Sekhari et al. (2021) and is also gaining prominence in LLMs due to concerns regarding bias, toxicity, and privacy Si et al. (2023); Liu et al. (2024). Some works in this direction emphasize model parameter optimization via gradient ascent Jang et al. (2023); Chen and Yang (2023); Yao et al. (2024b); Maini et al. (2024); Yao et al. (2024a) to unlearn unwanted responses for specific examples or datasets. They also finetune the model with various knowledge alignment objectives to maintain model utility. Other works leverage parameter optimization via relabeling of unlearning data. For instance, Eldan and Russinovich (2023) unlearn Harry Potter content by finetuning the model via gradient descent to replace the model's response with outputs containing generic translations. Jia et al. (2024) utilize similar fine-tuning objectives with a focus on optimizer selection and propose a framework that performs influence-based model updates via second-order optimization. Additionally, some works propose localization-based objectives that aim to identify a subset of model units that represent unlearning data and effectively delete them Meng et al. (2022); Yu et al. (2023); Wu et al. (2023). A few works also focus on modifying LLM input sequences to promote unlearning for black-box LLMs but are limited in the size of unlearning data. For instance, Pawelczyk et al. (2024) formulate in-context unlearning 

by crafting inputs comprising unlearning samples paired with flipped labels. Thaker et al. (2024) investigate guardrail techniques for unlearning by instructing models to withhold unwanted knowledge or filtering undesirable LLM outputs. Unlike most fine-tuning-based approaches, our goal in this work is to develop a soft prompting strategy to facilitate unlearning in LLMs. We aim to modulate LLM behavior using prompts similar to input modification strategies. However, instead of specifying manual instructions or providing demonstration samples as context, we leverage soft prompting to automate prompt optimization while adhering to unlearning objectives through loss specifications.

## 3 Soft Prompting for Unlearning

### Soft Prompting

Let \(D=\{s_{i},y_{i}\}^{N}_{i=1}\) denote a dataset containing \(N\) input-output pairs where \(s_{i}\) is a text sequence containing \(n_{i}\) tokens and \(y_{i}\) is the corresponding output. Also, let \(h_{\theta}\) represent a pre-trained LLM with parameters \(\theta\); \(h_{\theta}\) can be prompted with \(s_{i}\) to obtain an output \(\hat{y}_{i}\). Assume \(\mathbf{x}_{i}\in\mathbb{R}^{n_{i}\times d}\) denotes the token embeddings obtained for an arbitrary text sample \(s_{i}\) from the embedding module of \(h_{\theta}\) where \(d\) is the dimension of the embedding space. We first define \(p\) prompt tokens as \(\mathbf{\phi}=\{\phi_{1},\cdots,\phi_{p}\}\) where \(\phi_{i}\in\mathbb{R}^{d}\). To adapt \(h_{\theta}\) over \(D\) using soft prompts, \(\mathbf{\phi}\) is appended to \(\mathbf{x}_{i}\) to form the sequence \(\{\mathbf{\phi},\mathbf{x}_{i}\}\in\mathbb{R}^{(p+n_{i})\times d}\) as input to the encoder or decoder in \(h_{\theta}\). During backpropagation, the pre-trained parameters \(\theta\) are frozen and gradient updates are applied only to \(\mathbf{\phi}\) when maximizing the likelihood of the output \(y_{i}\) as

\[\operatorname*{argmax}_{\mathbf{\phi}}\log h_{\theta}(\{\mathbf{\phi},\mathbf{x}_{i} \}). \tag{1}\]

The size of the learnable prompts \(\mathbf{\phi}\) is very small compared to that of the pre-trained parameters \(\theta\). Nonetheless, soft prompting has shown considerable performance over various language tasks with results comparable to fine-tuning. This motivates us to consider _whether we can achieve unlearning in LLMs by optimizing continuous prompt tokens._

### Problem Formulation

Given a training dataset \(D^{tr}\) that was observed during pre-training of \(h_{\theta}\), we assume a forget set, \(D^{tr}_{f}\subset D^{tr}\), as the data intended for forgetting/removal from \(h_{\theta}\). Simultaneously, we define a retain set \(D^{tr}_{r}=D^{tr}\setminus D^{tr}_{f}\) comprising the remaining samples. Then, the goal of unlearning is to forget the token sequences in \(D^{tr}_{f}\) while maintaining inference utility on \(D^{tr}_{r}\). For our work, we focus on the task of text classification and question answering and interpret unlearning as the forgetting of the predictive output token sequences \(y_{i}\in D^{tr}_{f}\). Essentially, we de-correlate text features and their corresponding labels for the relevant forget samples but preserve the predictive performance on the retain samples. To this end, we aim to design a soft prompting framework to obtain optimized prompt tokens that can guide the base model toward the forget and retain objectives. With our framework, we aim to address the following research questions.

**RQ1:** How can soft prompting be utilized to effectively unlearn subsets of training data in the text classification/QA domain?

**RQ2:** How can soft prompting be implemented to achieve utility preservation with forgetting?

**RQ3:** How efficient is soft prompting-based unlearning compared to fine-tuning?

### Method

As soft prompts can be trained to encode signals from a dataset with the purpose of adapting a pretrained LLM to a specific downstream task, we anticipate that the strategy can also be utilized to encode relevant information from an unlearning dataset containing forget and retain samples. Here, we propose the framework SPUL that leverages soft prompting to obtain effective prompt tokens \(\mathbf{\phi}\) from an unlearning dataset \(D^{tr}\). Since one of the unlearning objectives in our framework is to promote feature and text de-correlation for forget samples, we design a loss attuned to enforcing incorrect predictions for the respective text inputs. Specifically, we force the model to associate each input forget text sequence with a generic output token instead of its true label. We construct a generic label set \(\bar{Y}\) that is disjoint from the task labels and contains tokens such as _neutral_, _unknown_, or _none_ and define a loss over the forget inputs,

\[\mathcal{L}_{f}=\sum_{(\mathbf{x}_{i},\cdot)\in D^{tr}_{f}}l(\tilde{y}_{i}|\{ \mathbf{\phi},\mathbf{x}_{i}\},\bar{y}_{i}), \tag{2}\]

where \(\bar{y}_{i}\) denotes a uniform random sample drawn from the pre-defined generic label set \(\bar{Y}\), and \(l(\cdot)\) refers to the standard cross-entropy loss. Ideally, \(\mathcal{L}_{f}\) allows the prompt tokens \(\mathbf{\phi}\) to capture specific nuances from the samples in \(D^{tr}_{f}\) and consequently guide the LLM to change its predictive sequence for an arbitrary example containing the learned distinctions. Simultaneously, unlearning also aims 

to preserve the predictive performance for samples not included in the forget set. In SPUL, the prepended prompt tokens \({\mathbf{\phi}}\) should not change the predictive sequences for \({\bf x}_{j}\in D^{tr}_{r}\). Therefore, to preserve inference utility on the retain set, we define a loss using their true labels as

\[\mathcal{L}_{r}=\sum_{({\bf x}_{j},y_{j})\in D^{tr}_{r}}l(\hat{y}_{j}|\{{\mathbf{ \phi}},{\bf x}_{j}\},y_{j}), \tag{3}\]

where \(l(\cdot)\) again represents the cross-entropy loss. \(L_{r}\) ensures that the model's utility on the retain set does not degrade with the addition of prompt tokens. In addition to maintaining performance on the retain set, the model after unlearning should closely resemble the model before unlearning. In our framework, we constrain the predictive distribution of the model such that \(h_{\theta}(\{{\mathbf{\phi}},{\bf x}_{j}\})\) reflects \(h_{\theta}({\bf x}_{j})\) for any \({\bf x}_{j}\in D^{tr}_{r}\). We quantify this difference using KL divergence as

\[\mathcal{L}_{kl}=\sum_{({\bf x}_{j},\cdot)\in D^{tr}_{r}}\mathrm{KL}(h_{ \theta}(\{{\mathbf{\phi}},{\bf x}_{j}\})||h_{\theta}({\bf x}_{j})), \tag{4}\]

where \(\mathrm{KL}(\cdot)\) denotes the KL divergence term. \(h_{\theta}(\{{\mathbf{\phi}},{\bf x}_{j}\})\) represents the base model's predictive distribution conditioned on inputs prepended with the learnable prompt tokens and \(h_{\theta}({\bf x}_{j})\) refers to the output distribution conditioned only on the input text sequence. We utilize \(\mathcal{L}_{kl}\) in addition to \(\mathcal{L}_{r}\) to avoid large deviations in the base model's output due to the influence from \(L_{f}\). Finally, at each time step \(t\) during training, we update \({\mathbf{\phi}}\) by optimizing the overall loss obtained as

\[\mathcal{L}=\mathcal{L}_{f}+\alpha\cdot\mathcal{L}_{r}+\beta\cdot\mathcal{L}_ {kl}, \tag{5}\]

where \(\alpha\) and \(\beta\) are hyperparameters that specify the contribution of the respective loss components.

## 4 Experiments

### Experimental Setup

DatasetsWe evaluate SPUL on standard NLP datasets SST-2 Socher et al. (2013) and Yelp polarity Zhang et al. (2015) for sentiment classification task. SST-2 and Yelp contain reviews with each text sequence being labeled as a positive or negative sentiment. To build a realistic unlearning scenario where unlearning requests from each user would likely include multiple related training samples, we preprocess the classification datasets to construct the forget and retain sets such that the forget samples are semantically similar to each other (Yelp) or refer to common entities (SST-2). For SST-2, we perform Named Entity Recognition to identify named personalities, select a specific set of entities, and sample all related reviews to form the forget set \(D^{tr}_{f}\). The remaining reviews are consequently assigned to the retain set \(D^{tr}_{r}\). We perform a similar partitioning using the selected entities on the test set to obtain \(D^{te}_{f}\) and \(D^{te}_{r}\). For Yelp, we perform k-means clustering with cosine distance on the training data to divide the reviews into semantically similar groups. We randomly select a subset of the clusters and group them to form the \(D^{tr}_{f}\) and the rest as \(D^{tr}_{r}\). We utilize the same cluster centers to infer cluster identities for the test data and form the sets \(D^{te}_{f}\) and \(D^{te}_{r}\) accordingly.

We also evaluate SPUL for multiple-choice question answering using a combination of WMDP Li et al. (2024) and SciQ Welbl et al. (2017) datasets. SciQ consists of exam questions about Physics, Chemistry, and Biology, among others in a four-way multiple-choice format where each answer choice is associated with symbols such as "A", "B", etc. and WMDP contains questions about hazardous knowledge in biosecurity, cybersecurity, and chemical security in the same format. In this MCQA task, we construct forget sets to unlearn potentially harmful information while retaining general science knowledge, i.e.: we obtain the forget sets from WWDP containing questions about hazardous knowledge in biosecurity and the retain sets from SciQ with general science questions. We refer to this dataset as WWDP+SciQ. Table 1 includes the sizes of the constructed forget and retain sets.

BaselinesWe assess the effectiveness of SPUL by comparing its performance against multiple SOTA parameter-tuning baselines. Gradient Ascent (GA) optimizes pre-trained LLM parameters by maximizing the cross-entropy loss defined only on the forget set \(D^{tr}_{f}\) in place of standard minimization Jang et al. (2023). Fine-tuning with Random Labels (RL) similarly optimizes the LLM on \(D^{tr}_{f}\) but by enforcing convergence on random vocabulary terms Golatkar et al. (2020); Yao et al. (2024). We use the generic label set discussed in Section 3.3 as the random labels for RL. Gradient Ascent + KL Divergence (GA+KL) and Gradient

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Dataset & \(D^{tr}_{f}\) & \(D^{tr}_{r}\) & \(D^{te}_{f}\) & \(D^{te}_{r}\) \\ \hline SST-2 & 1425 & 46331 & 610 & 19855 \\ Yelp polarity & 5081 & 95012 & 885 & 18089 \\ WWDP+SciQ & 900 & 12679 & 373 & 1000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset Statistics 

Ascent + Descent (GA+GD) incorporate parameter optimization on the retain set \(D_{r}^{tr}\) in addition to the GA loss to balance forgetting effectiveness with utility Yao et al. (2024). GA+KL defines a KL-divergence constraint on the LLM's output distribution for \(D_{r}^{tr}\) and GA+GD minimizes the standard cross-entropy loss on \(D_{r}^{tr}\). Negative Preference Optimization (NPO) achieves unlearning via alignment by maximizing the difference in predictive probabilities for forget samples between the unlearned model and a reference model trained on the entire dataset Zhang et al. (2024). We implement the NPO+RT variant which additionally incorporates a loss on the retain set and was shown to achieve a better trade-off. We fully fine-tune the LLM for all baselines following prior works based on their publicly available implementations.

SettingsWe use LLaMA-2-7B Touvron et al. (2023) as the base LLM to evaluate SPUL and further validate its unlearning effectiveness with OPT-1.3B Zhang et al. (2022) and LLaMA-2-13B Touvron et al. (2023). To ensure familiarization with the unlearning dataset, we fine-tune the base LLMs on the full training dataset \(D^{tr}=D_{f}^{tr}\cup D_{r}^{tr}\) for 10 epochs on SST-2, 2 epochs on Yelp, and 5 epochs on WMDP+SciQ with a learning rate set to 0.0001 and context length set to 1024 using QLoRA Dettmers et al. (2023). We treat this fine-tuned version of the LLM as the base model for unlearning. For SPUL, we fix the learning rate at 0.0001 across all LLMs and datasets. We vary prompt token length \(p\) among {10, 20, 30, 40, 50} and the regularization parameters \(\alpha\) as {0.1, 0.5, 1.0} and \(\beta\) as {0.0, 0.1, 0.5, 1.0}. We train our unlearning framework for a total of 10 epochs. For baselines except NPO+RT, we follow earlier works and perform training over 1 epoch as it has been observed that training over multiple epochs quickly deteriorates model performance on the retain set Yao et al. (2024). For NPO+RT, we unlearn over 10 epochs and use the best-performing hyperparameters reported in its paper Zhang et al. (2024). We also conduct a parameter search for the best learning rates. All experiments are conducted on NVIDIA A100 GPUs with 40GB RAM and we report the evaluation metrics over a **single run** due to the resource-intensive nature of the experiments.

EvaluationWe demonstrate the efficacy of the unlearning methods by evaluating them based on the research questions posed in Section 3.2. To quantify how well SPUL addresses RQ1, we report the accuracy and weighted F1 on the forget set, \(D_{f}^{tr}\), which signifies whether the learned soft prompts can de-correlate the text features and labels. As \(D_{f}^{te}\) is composed of text sequences semantically or lexically similar to \(D_{f}^{tr}\), the prompt tokens should result in a comparable performance decline on \(D_{f}^{te}\). To evaluate SPUL on RQ2, we report performance on \(D_{r}^{tr}\) and consequently \(D_{r}^{te}\). We require the differences in the accuracy and F1 scores of the base model before and after unlearning to be minimal for utility preservation. On the whole, we consider the tradeoff between forget and retain metrics to evaluate unlearning. To answer RQ3, we compare the number of training parameters and required GPU hours. We conduct further experiments to evaluate the influence of different loss objectives and choice of hyperparameters.

### Experimental Results

Main ResultsWe include our main results with LLaMA-2-7B in Table 2. We report performance for the original pre-trained LLM as Vanilla and the fine-tuned base model as QLoRA. We notice that the Vanilla results are considerably poorer for SST-2 than Yelp. We attribute the difference in utility to the longer and more descriptive text sequences in Yelp that can provide more contextual information. Nonetheless, after fine-tuning with QLoRA, the LLM's performance increases to similar margins for both sentiment classification datasets. QLoRA fine-tuning similarly improves LLM's predictions for the WMDP+SciQ dataset. This indicates that the LLM has memorized relevant contextual information about the task and training data.

From Table 2, we observe for SST-2 that SPUL significantly reduces accuracy and F1 on \(D_{f}^{tr}\) compared to QLoRA demonstrating forget efficiency. Also, the performance gap for \(D_{r}^{tr}\) between them is minimal showing that SPUL can promote unlearning while preserving inference utility. Moreover, the metrics for \(D_{f}^{te}\) and \(D_{r}^{te}\) reflect those reported for \(D_{f}^{tr}\) and \(D_{r}^{tr}\) showing that soft prompts effectively impose unlearning constraints on unseen samples. We observe similar trends for Yelp. Although the performance drop for \(D_{f}^{tr}\) and \(D_{f}^{te}\) in Yelp are not equally as large as SST-2, the forget utility with the learned tokens is significantly lesser than that of the base model, QLoRA, and also lower than the Vanilla model which has not been fine-tuned on the dataset. We conjecture that the additional context provided by descriptive 

Yelp reviews restricts the forgetting capacity of the LLM. Nonetheless, the utility loss in retain sets is much smaller than in forget sets indicating effective unlearning. We also note that SPUL performs exceedingly well for the MCQA task on the WMDP+SciQ with the highest differences observed between the retain and forget metrics among the evaluated datasets. Therefore, SPUL can effectively unlearn unwanted or harmful training examples in sentiment classification and MCQA tasks.

Comparison with BaselinesSPUL outperforms most baselines by a large margin despite optimizing fewer parameters. Compared to GA and RL which utilize only \(D_{f}^{tr}\), SPUL consistently preserves retain utility with lower or comparable forget metrics. For WMDP+SciQ, both GA and RL underperform on forget sets. GA+KL and GA+GD optimize model parameters based on \(D_{f}^{tr}\) and \(D_{r}^{tr}\), however, GA+KL performs poorly on all datasets. GA+GD performs well on SST-2 and WMDP+SciQ but fails to enhance forget quality on Yelp which has more descriptive reviews than SST-2. In contrast, NPO+RT shows improved forget quality while maintaining model utility as it avoids catastrophic forgetting. For Yelp and WMDP+SciQ, SPUL obtains significantly better trade-offs than GA, RL, GA+KL, and GA+GD methods and approximates forget and retain metrics of NPO+RT. For SST-2 dataset, SPUL surpasses GA, RL, and GA+KL, and is comparable with GA+GD and NPO+RT. Note that all baselines require full fine-tuning whereas SPUL only updates the soft prompt parameters. Therefore, soft prompting can improve or approximate the unlearning performance of fine-tuning baselines while updating fewer parameters for contrasting performance degradation and utility preservation objectives.

General Downstream PerformanceWe additionally evaluate the unlearning methods for the generalization ability of LLM after unlearning by conducting inference with unlearned models on a downstream task. We utilize models unlearned on WMDP+SciQ and report performances on ARC 

Challenge (Clark et al., 2018) in Table 3. Our results indicate that SPUL achieves accuracy and f1 scores similar to the base model, QLoRA. Although GA, RL, and GA+KL also demonstrate similar downstream utility, their forget performances on WMDP+SciQ are less optimal than SPUL. GA+GD shows lower forget quality and downstream utility compared to SPUL. NPO+RT scores a better forget and retain trade-off than SPUL in the unlearning task but significantly downgrades model generalizability, possibly due to over-fitting the training data as made evident by its results on the train forget and retain sets. In conclusion, SPUL effectively unlearns target data without compromising the LLM's inference ability.

VisualizationWe also visualize model outputs to show the effectiveness of SPUL. We utilize outputs from the last embedding layer of the LLM and map them onto a t-SNE diagram as shown in Fig. 1. The plots represent 500 data points randomly sampled from the training dataset in SST-2 for each label. In the plots, we use colors to differentiate the retain and forget examples and shapes to differentiate the positive and negative examples. We visualize the embeddings from QLoRA, i.e., the base model before unlearning and we observe a clear divide between the positively and negatively labeled samples in the embedding space. The retain and forget samples are clustered within the regions defined by each label. For the t-SNE plot of SPUL, i.e., the embeddings obtained after prepending the learned soft prompts, we notice a clear separation between the retain and forget samples as indicated by the blue and orange regions in Fig. 1. This shows that the soft prompts truly capture the differences between the forget and retain sets. Moreover, the retain samples are further grouped into clusters per their labels whereas the forget samples are mixed. This shows that the soft prompt tokens learned by SPUL successfully guide the LLM to unlearn text and label correlation for the forget samples while preserving predictive utility on the retain set.

Referring back to Table 2, SPUL metrics on \(D_{f}^{tr}\) and \(D_{f}^{te}\) closely resemble each other. We make mostly similar observations for \(D_{r}^{tr}\) and \(D_{r}^{te}\). Our visualization results also show that the embeddings for forget samples are not distinguishable between labels. Compared to QLoRA visualization, outputs for positive and negative retain samples are closer in the embedding space. Consequently, in a black-box Membership Inference Attack (MIA) (Shokri et al., 2017) scenario, it would be challenging to infer whether a particular forget sample was observed during training based only on model outputs.

Ablation StudyWe investigate the influence of \(\mathcal{L}_{r}\) and \(\mathcal{L}_{kl}\) and report the results in Table 4 for the SST-2 dataset. The hyperparameters \(\alpha\) and \(\beta\) control the influence of the \(D_{r}^{tr}\) on the learned soft prompts via losses \(\mathcal{L}_{r}\) and \(\mathcal{L}_{kl}\) respectively. We fix the number of prompt tokens \(p\) at 30 and vary \(\alpha\) in {0.1, 0.5, 1.0} and \(\beta\) among {0.0, 0.1, 0.5, 1.0}. From Table 4, we observe that at a fixed \(\alpha\), uni

Figure 1: Embedding visualization results on SST-2 with QLoRA and SPUL



learning efficacy is fairly unaffected by the change in the value of \(\beta\). Model utility on the retain set, however, slightly increases as \(\beta\) increases from 0.0 to 0.5 as \({\cal L}_{kl}\) gets more significance in the overall loss. Generally, the best retain performance is at \(\beta\) = 0.5. \(\alpha\), however, influences both forget and retain performance; higher \(\alpha\) values benefit retain performance by prioritizing utility preservation whereas lower \(\alpha\) values improve unlearning efficacy.

Hyperparameter StudyWe study the effect of \(p\), the number of prompt tokens, on the effectiveness of SPUL. We fix both \(\alpha\) and \(\beta\) at 1 and report results for \(p\) ranging from 10 to 50 on SST-2 in Fig. 2. We find that inference utility on retain sets \(D^{tr}_{r}\) and \(D^{te}_{r}\) is largely unaffected by the choice of \(p\). However, we observe the most competitive forget performance at \(p\) = 30 with increasing accuracy and F1 as \(p\) increases/decreases. We speculate that the soft prompts mostly encode information from the forget set, for instance, the forget entities in SST-2, and ultimately instruct the LLM to misclassify examples with similar encodings. Accordingly, a larger \(p\) generally benefits SPUL as shown by the decline in forget metrics but may require longer training for optimal performance.

Forget Set SizeTo demonstrate SPUL's stability w.r.t. the size of forget data, we evaluate it on varying sizes of the train forget set \(D^{tr}_{f}\) by sub-sampling \(\tau\) = {25%, 50%, 100%} of the original forget set constructed for SST-2. For the remaining splits \(D^{ts}_{r}\), \(D^{te}_{f}\), and \(D^{te}_{r}\), we use the same sets from Section 4.1 for all three configurations of \(D^{tr}_{f}\) to facilitate comparison. The results presented in Table 5 indicate that SPUL can achieve utility preservation across differing numbers of forget samples with minimal loss as more forget samples are added to \(D^{tr}_{f}\). In contrast to the retain metrics, SPUL performs better for the forget metrics when more forget samples are present in the data for SST-2. Experimental results on Yelp presented in Table 2 also highlight the robustness of SPUL against large forget sets as we assign more than 5000 samples to \(D^{tr}_{f}\). As the training data contains fewer forget samples than retain samples, having a larger \(D^{tr}_{f}\) allows the framework to emphasize the forgetting objective thus improving the unlearning efficacy.

Results on LLaMA-2-13B and OPT-1.3BWe also evaluate SPUL on OPT-1.3B and LLaMA-2-13B, with respectively fewer and almost double the parameters than LLaMA-2-7B. We fix both \(\alpha\) and \(\beta\) at 1 and \(p\) at 30 and report the results for SST-2 in Table 6. We first observe that the Vanilla inference with OPT-1.3B model performs noticeably poorer than LLaMA-2-7B whereas LLaMA-2-13B improves over LLaMA-2-7B. This gap is attributed to the respective LLM's complexity which affects its generalization ability. For both OPT-1.3B and LLaMA-2-13B, SPUL can effectively achieve the forget and retain unlearning objectives as shown by the low forget accuracy and F1 compared to the retain metrics that closely resemble the base model's performance. The results also indicate larger LLMs better adapt to the unlearning task in the SPUL framework. With OPT-1.3B, SPUL notably outperforms most baseline methods and achieves a trade-off comparable to NPO+RT. We could not run experiments on baselines with LLaMA-2-13B due to limited GPU as they require full fine-tuning. This further highlights the advantage of SPUL over baselines for parameter efficiency.

EfficiencyRetraining LLMs from scratch is practically infeasible due to computational costs. Fine-tuning incurs fewer resources but is expensive nonetheless. For instance, the LLM architectures used in our experiments require gradient updates for 1.42B, 6.74B, and 13B parameters for OPT-1.3B, LLaMA-2-7B, and LLaMA-2-13B respec 

tively for fine-tuning. When \(p=30\), our SPUL reduces the computation cost by only optimizing 604K, 1.19M, and 1.49M parameters respectively while freezing LLM parameters. Further increasing \(p\) only linearly scales the number of training parameters. We also look at the running time of SPUL on the SST-2 compared against baseline methods and find the execution time required by each model of SPUL, GA+KL, GA+GD, and NPO+RT for one training epoch is fairly similar, around 1020 GPU seconds, as SPUL also accesses LLM parameters during backpropagation. GA and RL methods are much quicker with approximately 40 GPU seconds per epoch training time as these methods only consider the forget set. Nonetheless, SPUL avoids the overhead associated with updating LLM parameters, making it more resource-efficient.

## 5 Conclusion

We investigate unlearning in LLMs to remove the influence of unwanted/harmful training examples during text classification and MCQA. We present a soft prompting strategy to unlearn subsets of training data while keeping pre-trained LLM parameters frozen to maintain the model's generalizability. The proposed SPUL framework optimizes a small number of prompt tokens using a multi-objective loss function defined on disjoint training data subsets representing the forget data subjected to removal and the retain data that aims to preserve model utility. Experimental evaluation on sentiment classification and MCQA datasets demonstrates the efficiency of SPUL over fine-tuning-based baselines for trade-offs between forget quality, retain utility, and generalizability. We also empirically show that SPUL can adapt to multiple LLMs and is robust to large unlearning requests.

## Acknowledgements

This work was supported in part by NSF grants 1946391 and 2119691.

### Limitations

We address the limitations of this work in the following. Our experiments primarily focus on open-source LLMs as the soft prompting framework requires access to frozen pre-trained parameters to compute gradients for the soft prompts. Although, the framework avoids the overhead of updating LLM parameters. Furthermore, this work focuses on text classification and question-answering datasets for formulating and evaluating the unlearning framework. Future research could explore the efficiency of soft prompting to achieve unlearning in the context of other NLP tasks such as text generation and summarization. We further note the lack of an extensive evaluation pipeline for LLM unlearning in the current literature. Further research is needed to evaluate the robustness of LLM unlearning frameworks subject to model-stealing attacks, MIAs, and jailbreaking attempts.

### Broader Impacts

In this study, our focus is to achieve LLM unlearning in a resource-efficient manner. We aim to enable forgetting of unwanted or harmful knowledge in a pre-trained LLM while maintaining model efficiency to avoid exploitation of sensitive information. The datasets used for evaluation are publicly available and implemented within their intended use. Our usage of publicly available pre-trained LLMs also adheres to the associated licenses. We hope our study can further the research and literature on resource-efficient LLM unlearning.

 



## References

* Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In _42nd IEEE Symposium on Security and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2021_.
* Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In _2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015_.
* Chen and Yang (2023) Jiaoa Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_.
* Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Talfjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. _Preprint_, arXiv:1803.05457.
* 16, 2023_.
* Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023. Who's harry potter? approximate unlearning in llvm. _Preprint_, arXiv:2310.02238.
* Golatkar et al. (2020) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_.
* Grynbaum and Mac (2023) Micheal Grynbaum and Ryan Mac. 2023. The times sues openai and microsoft over a.i. use of copyrighted work. The New York Times.
* Guo et al. (2020) Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. 2020. Certified data removal from machine learning models. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, Proceedings of Machine Learning Research.
* Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (OPTvolume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_.
* Jia et al. (2021) Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. SOUL: unlocking the power of second-order optimization for LLM unlearning. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024_, pages 4276-4292. Association for Computational Linguistics.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_.
* Li et al. (2021) Nathaniel Li, Alexander Pan, Anjalil Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisna, Birugu Bharathi, Ariel Herbert Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Kamakar, Uday Kiran Tupakula, Vijay Varadharajan, Yan Shohitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. 2024. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_.
* Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (OPTvolume 1: Long Papers), Virtual Event, August 1-6, 2021_.
* Liu et al. (2024) Sijia Liu, Yuanshu Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sammi Koyejo, and Yang Liu. 2024. Rethinking machine unlearning for large language models. _Preprint_, arXiv:2402.08787.
* Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (OPTvolume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_.
* Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023. GPT understands, too. _AI Open_.
* Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. TOFU

A task of fictitious unlearning for llms. _Preprint_, arXiv:2401.06121.
* December 9, 2022_.
* Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ipolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramer, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. _Preprint_, arXiv:2311.17035.
* OpenAI (2024) OpenAI. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Pawelczyk et al. (2024) Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2024. In-context unlearning: Language models as few-shot unlearners. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_.
* Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_.
* Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In _2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017_.
* Si et al. (2023) Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. 2023. Knowledge unlearning for llms: Tasks, methods, and challenges. _Preprint_, arXiv:2311.15766.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_.
* Thaker et al. (2024) Pratisha Thaker, Yash Maurya, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. _Preprint_, arXiv:2403.03329.
* Tourron et al. (2022) Hugo Tourron, Thibaut Lavril, Gautier Lacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_.
* Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. _Preprint_, arXiv:1707.06209.
* Wen et al. (2023) Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jianfeng Bai, and Minlie Huang. 2023. Unveiling the implicit toxicity in large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_.
* Wu et al. (2023) Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuanghi Wu, Chao Bian, and Deyi Xiong. 2023. DEPN: detecting and editing privacy neurons in pre-trained language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_.
* Yao et al. (2024a) Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. 2024a. Machine unlearning of pre-trained large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Yao et al. (2024b) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024b. Large language model unlearning. _Preprint_, arXiv:2310.10683.
* Yu et al. (2023) Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. 2023. Unlearning bias in language models by partitioning gradients. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_.
* Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. _Preprint_, arXiv:2404.05868.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Miabylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Korna, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. _Preprint_, arXiv:2205.01068.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_.