Make Text Unlearnable: Exploiting Effective Patterns to Protect Personal Data
Year: 2023

Abstract:
This paper addresses the ethical concerns arising from the use of unauthorized public data in deep learning models and proposes a novel solution. Specifically, building on the work of Huang et al. (2021), we extend their bilevel optimization approach to generate unlearnable text using a gradient-based search technique. However, although effective, this approach faces practical limitations, including the requirement of batches of instances and model architecture knowledge that is not readily accessible to ordinary users with limited access to their own data. Furthermore, even with semantic-preserving constraints, unlearnable noise can alter the text’s semantics. To address these challenges, we extract simple patterns from unlearnable text produced by bilevel optimization and demonstrate that the data remains unlearnable for unknown models. Additionally, these patterns are not instanceor dataset-specific, allowing users to readily apply them to text classification and questionanswering tasks, even if only a small proportion of users implement them on their public content. We also open-source codes to generate unlearnable text and assess unlearnable noise to benefit the public and future studies.

heading: 1 Introduction
text: With the increase in the prevalence of deep learning, public data has become more frequently used for developing predictive models. However, the use of unauthorized public data, such as tweets, raises ethical concerns. Furthermore, it is considered even more unethical to charge the public for services based on these models. In addition to the ethical concerns, our research can help address privacy issues associated with the development of sensitive applications that impede public privacy. For instance, facial recognition systems can recognize individuals even when they are on the street (Hill, 2020). To prevent deep learning models from exploiting textual content and potentially predicting
private information such as sentiments on sensitive topics (Kouloumpis et al., 2021; Severyn and Moschitti, 2015), political affiliations (Conover et al., 2011), age, and gender of users (Farnadi et al., 2018; Suman et al., 2021), we propose making text unlearnable. While Huang et al. (2021) proposed a process to make images unlearnable, our work extends this idea to generate unlearnable text using a gradient-based search approach.
In our study, we investigate the performance of error-minimization modifications for text unlearning in three tasks: sentiment analysis, topic classification, and question answering. Sentiment analysis and topic classification can reveal users’ interests, such as political leaning, while question answering can extract information from users’ text. Due to data accessibility limitations and privacy concerns, we conduct our experiments on open data that is commonly used for academic purposes.
Our contributions include the adaptation of the bi-level optimization formulation from Huang et al. (2021) to text, and the development of a search procedure to modify text for (inner) error minimization. Our results show the efficacy of error-minimization modifications in making text unlearnable for all three tasks. However, the optimization process is impractical in real-world scenarios. Therefore, we extract two synthetic patterns from error-min modifications: label hints for text classification and an answer hint for question answering. These patterns can make text unlearnable and can be applied to any individual text without requiring a computationally expensive algorithm.
We also consider the effectiveness of these synthetic patterns in real-world scenarios. Our results show that they can be effective on models with different network architectures and training paradigms, including training from scratch and the pretrain-then-fine-tune paradigm. Importantly, we demonstrate that these patterns remain effective even when extracted during the training process of
249
simpler models such as LSTMs and BiDAF. Moreover, they remain effective even when only a portion of users use them, and can be applied to one of the classes, which can be helpful in making one specific sensitive class unlearnable.

heading: 2 Background
text: In this section, we will conduct an analysis of the existing privacy protection methods designed to safeguard against training deep learning models. We will then proceed to explicate the bi-level optimization approach adopted in this study to generate unlearnable images. In the subsequent section, we will demonstrate the generalizability of this method to generate unlearnable text

heading: 2.1 Privacy Protection
text: The development of deep learning models with public data has raised concerns about privacy leakage. Several research directions have been proposed to address this concern. Differentially-private techniques (Dwork et al., 2014; Chaudhuri and Monteleoni, 2009; Shokri and Shmatikov, 2015; McMahan et al., 2018; Abadi et al., 2016) have been suggested as a solution to prevent the memorization of user-specific information during the training process. However, the application of such techniques requires users to trust those who collect their data. Another proposed approach is machine unlearning (Cao and Yang, 2015), which aims to remove the training impact of specific samples provided by users after the models have successfully learned from the data.
Protection of textual messages against unauthorized neural natural language processing (NLP) models is critical. Especially, statistical features learned by these models can lead to the extraction of private informationextracted by hackers (Fredrikson et al., 2015; Carlini et al., 2020) since DNNs can memorize private information such as name and address in training data. This paper concentrates on user-end solutions for privacy protection, exploring noise-addition approaches against unauthorized NLP models. While several noise-addition approaches have been proposed by the computer vision community against facial recognition models (Shan et al., 2020; Cherepanova et al., 2021; Huang et al., 2021), to the best of our knowledge, no similar work has been conducted in the NLP community.

heading: 2.2 Formulating the Unlearnable Objective as a Bi-level Optimization Problem
text: Consider a training set D = (x, y)Ni=1, where the i-th instance consists of a text x and its true label y for classification. A DNN f(θ), where θ is parameters of the model f , maps the input space X to the output pace Y. The training objective is to minimize the loss function L:
argmin θ L(f(x), y)] (1)
Min-min Optimization by Huang et al. (2021). Huang et al. (2021) nested the unlearnable objective within the training objective (Equation 1) to formulate a bi-level optimization problem:
argmin θ E(x+η,y)∼D[argmin η L(f(x+ η), y)],
(2)
where a pixel-wise vector η ∈ RC×H×W is optimized to minimize L, , where C,H,W are the numbers of channels, height and weight of images respectively.
They solved the outer objective with the common training routine, i.e., the gradient descent algorithm to iteratively optimize the model parameters θ:
θt+1 = θt − γ∇θtL, (3)
where γ is the learning rate. For the inner objective, they nested another iterative process of projected gradient descent (PGD) (Madry et al., 2018) to optimize the noise η (errormin noise) for each training sample (sample-wise noise) or each class (class-wise noise), which is a common routine to solve bi-level optimizations (Finn et al., 2017; Huang et al., 2020). Equation 4 shows the one-step update:
ηt+1 = ηt − ε sgn∇ηtL(x̃t), (4)
where they obtained perturbed images via element-wise addition x̃ = x + η, and ε sgn performs a `∞ norm.
We detail the whole min-min optimization in Algorithm 1.
Unlike the original process, we add the exit condition when the evaluation metrics on test sets are unchanged for computational efficiency, which indicates the noise’s effectiveness. 1 To generate
1We would use accuracy for text classification tasks and F1 scores for question answering.
Algorithm 1 Generating Unlearnable Noise. Require: neural network f(θ), training set D, test
set Dtest, training loss L, initialized noise η, num of training steps per modification M 1: num_train_steps← 0 ; test_metric← null 2: for each batch Z ∈ D do 3: if num_train_steps (mod M) = 0 then 4: Evaluate the current checkpoint f(θ)
on Dtest to get new_metric 5: if test_metric=null ∨ new_metric >
test_metric then 6: test_metric = new_metric 7: else 8: return the noise η 9: end if
10: Update noise η via an error-min optimization (images: Equation 4) 11: end if 12: Apply current unlearnable noise for all x ∈
Z (images: x̃ = x+ η) 13: θ ← θ − γ∇θL(Z) 14: num_train_steps + = 1 15: end for
unlearnable text, we replace the step 10 with a loss approximation search procedure, as demonstrated in the next section.

heading: 3 Adaptation to Text
text: In this section, we first formulate noise as discrete text modifications in contrast to pixel-wise vectors for images. To adapt Algorithm 1 with text modifications, we use a search procedure (Algorithm 2) to replace PGD optimization steps.

heading: 3.1 Text Modifications
text: Unlike images, a textual input x consists of a sequence of words w1, w2, ..., wT , where T is the number of words. A vocabulary V consists of all the words. Therefore, we define noise as substituting the word wp ∈ x indexed by the position p with a word s ∈ V , denoting as η = (p, s).
However, there are two problems: 1) The discrete operation (p, s) is not differentiable: Since the noise η for images is continuous, it is differentiable and can be optimized via gradient descent. However, we cannot use gradient descent to optimize (p, s); 2) Modifying a single token may change the semantics of text (e.g., "I love you" to "I the you"), while a simple `∞ norm on noise for an image can make it imperceptible.

heading: 3.2 A Search Procedure
text: To solve the first problem, we approximate the loss change for all possible substitutions and search for a substitute word causing the lowest loss. Specifically, each word w can be transformed into a dense vector ew via a matrix E ∈ Rn×m parameterized in a DNN f(θ), where n is the size of a vocabulary V and m is the size of each embedding vector. We measure the loss change of substituting a word wp with another word s ∈ V by the inner product of es and the gradient of loss w.r.t. ew (∇ewL(x, y)).
argmin s
eTs∇ewL(x, y) (5)
The first-order approximation approach has been used for adversarial attacks (Wallace et al., 2019, 2020; Ebrahimi et al., 2018) with different implementations.
For semantic preservation, we select the modified word s from semantically similar words for each substitution. Following the setting of Alzantot et al. (2018) for generating adversarial candidates, we calculate the cosine similarity between w and s and select candidate words within the threshold. We discuss the setting of the hyperparameters in Appendix B.
Besides, we only consider one modification (p, s) for a text. For question answering, we exclude positions in answer spans.
Implementation. To search for a (p, s) to minimize the training loss, we acquire the gradients for all the positions of the original example by one forward and backward pass, i.e., ∇xL = ∇ew1L, ...,∇ewT L.
Instead of searching over the vocabulary for each wp, we efficiently approximate the loss changes for all the candidates (P, S) by one matrix multiplication as Equation 6. We discuss the approximation errors in Appendix C.
A = ∇xLTE, (6)
where∇xL ∈ RT×m, and embedding matrix E ∈ Rn×m,
We then rank all the candidates according to the approximation scores A ∈ T × n and select the one with the lowest score satisfying the constraints.
Algorithm 2 demonstrates the process of searching for a optimal (p∗, s∗) for an instance (x, y) at one iteration.
Algorithm 2 Error-min for Gradient-based Word Substitutions. Require: a neural network f with E, training loss L, and a sample (x, y)
1: Generate∇xL(f(x), y) 2: Generate approximation scores A for all the
candidates (P, S) according to Equation 6 3: Sort (P, S) in the ascending order of A 4: for each candidate modification (p, s) ∈
(P, S) do 5: if all the constraints are satisfied for (p, s)
then 6: return (p, s) 7: end if 8: end for

heading: 4 Experimental Settings
text: This section will first introduce all our experiment’s tasks, datasets, and models. We then demonstrate essential factors for generating unlearnable modifications.

heading: 4.1 Tasks and Datasets
text: Text classification. A neural network f(θ) takes a text x and outputs a probability distribution over the output space Pr(Ŷ |x) after normalizing by the Softmax function, i.e., Pr(Ŷ |x) = Softmax(f(x)). L is defined as a negative log likelihood of Pr(y|x, θ) or a cross entropy between Pr(Ŷ |x) and one-hot representation of the true label y.
We choose two datasets to simulate real-world scenarios to identify users’ sentiments and interests, each with training, validation, and test sets.
• SST2: It contains movie reviews from the Stanford Sentiment Treebank (SST) dataset. Each sentence is labelled as either positive or negative sentiment. (Socher et al., 2013)
• AG-News: This dataset divides news articles into four classes: world, sports, business, and sci/tech. It involves 10,800 training samples, 12,000 validation samples, and 7,600 test samples. It works as a proxy task to detect users’ interests.
Question answering. Given a passage of text p and a question q, models aim to extract a correct answer span a from p. Given x = (p, q), f(θ) will output probability distributions for both the beginning and ending positions of the answer span a,
denoting as Prstart and Prend. The loss L is calculated by adding negative log likelihoods of Prstart and Prend. We aim to prevent QA models from learning the passage when we maintain correct answers in the passage.
We use the Stanford Question Answering Dataset (SQuAD) v1.1 dataset (Rajpurkar et al., 2016), which contains more than 100,000 questionanswer pairs based on about 500 articles. Since the SQuAD test set is unavailable, we use the validation/test splits from Du et al. (2017) derived from the original validation set.

heading: 4.2 Models
text: To generate error-min modifications, we use LSTMs (Hochreiter and Schmidhuber, 1997) (∼ 3.8M parameters) for all the text classification tasks and Bidirectional Attention Flow (BiDAF) model (Seo et al., 2016) (∼ 2.5M parameters) for question answering. Specifically, BiDAF uses one bidirectional LSTM to represent each context and question respectively and applies an attention mechanism to generate two question-aware context representations with a dimension of H , where H is the hidden size. A linear layer parameterized by a matrix MH×2, followed by a softmax function, transforms them into the probability distributions Prstart and Prend respectively. We use the 300-dimensional GloVe word vectors (Pennington et al., 2014) for the above models.
To answer whether we can make text unlearnable when fine-tuning powerful pretrained language models, we evaluate BERTBASE with 110M parameters (Devlin et al., 2019) for text classification and RoBERTaBASE with 125M parameters (Liu et al., 2019) for question answering. In contrast to BiDAF, RoBERTa is pretrained to support a pair of sequences as inputs by concatenating them with a special token.

heading: 4.3 Computational Considerations
text: Generating modifications by the min-min optimization is computationally expensive. Due to limited computational resources, we down-sample the training set for AG-News and SQuAD to validate the min-min optimization, i.e., using the first 3,200 articles and their categories of AG-News and 1,000 question-answer pairs from the SQuAD training set. However, we construct the vocabulary on the whole training data to avoid out-of-vocabulary when evaluating test data. Note that such size of SQuAD examples is not large enough to train a good QA
model. However, we can evaluate the effectiveness of the min-min optimization by comparing model performance on clean and modified data.
Even so, we find that the algorithm 2 runs much slower on AG-News and SQuAD than SST2 since it is harder to find substitute words to satisfy the similarity constraint. We would not apply the constraint to AG-News and SQuAD. Since the text in these two datasets are much longer (19 for SST2, 43 for AG-News, and more than 100 for SQuAD), it is unlikely to change the semantics of a text by substituting one word. 2

heading: 5 Effectiveness of Min-min Optimization
text: In this section, we report the effectiveness of modifications generated via the min-min optimization and further analyze why min-min modifications are effective.

heading: 5.1 Experimental Results
text: The min-min optimization generates several sets of error-min modifications (S0, P0), ..., (Si, Pi), ..., (SN , PN ) at different training checkpoints (see step 10 in Algorithm 1). For example, Error-min-i = (Si, Pi) is generated by Algorithm 2 after M × i training steps, which would be applied on the next M training steps (see step 12 in Algorithm 1) until (Si+1, Pi+1) is generated. Error-min-N = (SN , PN ) is the final output from the min-min optimization.
We not only answer whether the final min-min modifications (Error-min-N) can make text unlearnable but also evaluate whether other sets of errormin modifications (e.g., Error-min-i) can be effective. Specifically, we apply each set of error-min modifications to the clean training data and optimize neural networks on the modified training data. We then follow the strategy from Huang et al. (2021) to measure metrics on test samples during different training epochs. The min-min optimization over LSTM on SST2 generates three sets of error-min modifications (i.e., N = 3), while two sets for SST2 and SQuAD.
All the results in Figure 1 demonstrate that the Error-min-0 modifications effectively make text unlearnable. They are even more effective than the last error-min modifications for SST2 and AGNews. With this, the bi-level optimization may
2Even so, running Algorithm 2 to generate one set of errormin modifications once costs around 4 hours for AG-News and more than 10 hours for SQuAD with RTX3080 (16GB).
be unnecessary to generate effective modifications, and one-step error minimization on randomly initialized DNNs can generate effective modifications.

heading: 5.2 Analysis
text: After exploring why Error-min-0 appears more effective in this section, we find that there exist simple, explicit patterns which correlate to the taskspecific outputs (i.e., labels for text classification or answers for QA) to make text unlearnable.
Specifically, we first investigate whether substitute words in each set of error-min modifications correlate with labels. We divide all the substitute words for each class into bags of words (labelwise BOWs) and calculate the average Jaccard similarity between each pair of BOWs as Equation 7. Table 1 shows that effective modifications (e.g., Error-min-0) present low similarity, which indicates that label-wise patterns may make text unlearnable.
Average Similarity = K∑
i=1
K∑
j=i+1
|BOWi ∩ BOWj | |BOWi ∪ BOWj |
(7)
where K is the number of classes/labels. We
also find little sample-wise feature in each labelwise BOW. Specifically, we calculate the probabilities over all the substitute words. For example, PrBOW0(w) denotes the probability that the word w appears amongst all the samples with the label indexed by 0. We then rank the probabilities in descending order and cumulate the probabilities for the top 5 words. Figure 2 shows that we only need five words to make most of the examples unlearnable.
We then investigate the distribution of positions P . We calculate the relative position prel for each sample by dividing each position p (the index of
the modified word) by the length of the sentence x. Extremely, prel = 0 when modifying the first word, while prel = 1 if the last word is modified. Figure 2 shows that text tends to be modified at the end.
We also find a simple pattern in the error-min modifications for SQuAD: 1) all the positions are identified within the one-word distance of the answers. 2) Similar to text classification, the top 5 substitute words modify 98% of 1000 samples.
Therefore, we can reasonably hypothesize that the min-min optimization would generate noise with task-specific patterns to make text unlearnable,
e.g., words correlating to labels for text classification or words to indicate the positions of answers for QA.

heading: 6 Manually Generating Simple Patterns
text: In this section, we test the effectiveness of synthetic patterns according to the previous findings since it is difficult to use the min-min optimization in reality. First, it assumes that users can access model architectures and the whole training data (or at least a batch of instances). In real life, users can only access their portion of data and publish one instance (e.g., a tweet) once at a time. Besides, generating modifications with the min-min optimization is very computationally expensive.
Hence we construct synthetic patterns, including class-wise symbols (label hints) for text classification and a symbol surrounding the answer spans (answer hints) for question answering. Another benefit is that inserting such symbols maintains semantics without complicated constraints.
To show that the patterns can be generalized to other network architectures, we evaluate them by fine-tuning two popular pretrained transformers: BERT for text classification and RoBERTa for question answering. Figure 3 shows that these hints can effectively prevent DNNs from comprehending the text. Surprisingly, class-wise symbols are effective at any position (the beginning/middle/end). Although we show experimental results with characters (e.g., "a", "b") as the hints, we can also achieve the same outcome by inserting an exclamation mark ("!") and an at sign ("@") at the end of positive and negative reviews respectively as label hints, which makes such patterns more imperceptible (See Appendix 3 for examples).
The patterns’ effectiveness when only partial training instances can be modified. Since it may not be possible to let all users add the patterns, we explore their effectiveness when applying
such patterns to partial training data. We randomly select a certain percent of training instances (Dpartial) and apply unlearnable patterns on them (Dunlearn). To show the effectiveness of unlearnable patterns, we calculate the change in the test accuracy after adding Dunlearn into the training process. For comparison, we report the result by adding Dpartial. As shown in Table 4, models rarely learn useful information from Dunlearn compared to Dpartial.
Can we only make one class of examples unlearnable? We select one class in AG-News (i.e., the "World" category) and insert a symbol ("a") only on instances belonging to the "World" class. A BERT model fine-tuned on such a dataset shows low accuracy on the test instances belonging to the "World" class (0.015) and high accuracy on others (0.93). Henceforth, users can make a sensitive class of data unlearnable by agreeing on a class-specific symbol.

heading: 6.1 Why Do Simple Patterns Make Text Unlearnable?
text: We consider simple patterns as biased features. Without any biased feature, the gradient descent
algorithm would optimize θ to approximate the conditional probability Pr(y|x) by minimizing empirical errors of any training instance. When we embed a simple biased feature b into x, the DNN would first learn Pr(y|b). Many previous works (He et al., 2019; Branco et al., 2021) have found that deep learning tends to learn superficial patterns. As shown in our experiments, once the model learns such Pr(y|b), models have difficulty exploiting the semantics of the input x during the latter training process since the performance on test data does not improve. This property coincides with shortcuts found in question answering Lai et al. (2021).
An unlearnable state. We assume that there exists an unlearnable state when models confidently correlate b with model outputs, i.e., Pr(y|b) ≈ 1, which would lead to L ≈ 0 for any input x with b. Correspondingly, the forward pass would generate zero gradients to update the model during the backward pass. Since the model has no update according to the data, we can ensure that there is no information leakage. We verify this by tracing gradient norms during fine-tuning BERT on synthetic patterns. Figure 4 shows that the unlearnable state appears at about 250 iterations, where the model stops updating parameters. The same phenomenon occurs during training LSTM on error-min modifications (see Appendix A).

heading: 7 Conclusion
text: By adapting min-min optimization, we develop an approach to expose vulnerabilities of deep learning to make text unlearnable. To overcome the limitation of requiring knowledge of models and training data, we extract simple patterns (e.g., label hints and answer hints) from the min-min optimization to make text unlearnable. Although our experiment explores patterns for text classification and question-answering tasks, the pipeline potentially
works for any NLP task.
Reproducibility. To ensure the effectiveness of unlearnable modifications, we slightly tuned the training hyperparameters to achieve well-trained models, such as setting maximum gradient norms and early stopping according to validation sets. We open-source codes with configuration files, which contain hyperparameters regarding model architectures (e.g., the number of layers), batching (e.g., data sampling), and training setups (e,g., learning rate). Since these files are configurable in JSON format, future works can easily reproduce and extend the experiments.

heading: 8 Limitations
text: The main concern is that debiased techniques may remove simple biased features. However, to our knowledge, most debiased techniques (Rathore et al., 2021) can only remove biases across a concept subspace (e.g., the bias direction for gender) in the embedding space. Another setup of data debiasing, e.g., He et al. (2019), requires hypothesized biases to train biased models and is limited to tasks with known hypothesized biases (e.g., lexical overlap for NLI). Also, they remove biased examples rather than identify biased symbols (e.g., label hints). However, we still expect future works to consider other complicated patterns beyond symbol insertions or word substitution.

heading: A The Change of Gradient Norms
text: Figure 5 shows gradient norms with error-min modifications and further proves the argument. The set of the Error-min-0 modifications with label-wise patterns (see Table 1) has almost zero gradients during training. It even has a small gradient update in the first few steps. It may be because the randomly initialized models can easily learn class-wise patterns, while BERT has to overcome its pretrained priors.

heading: B Hyperparameter Setting
text: The interval of optimizing the error-min noise M . If M is too small, the test accuracy after another M iterations easily plateaus due to insufficient model update, which causes the early stop of
the min-min process. On the other hand, a large interval will linearly increase the computational complexity. Specifically, since we use modifications for batches of instances in the nextM training iterations, error-min optimization needs to be run for M ×B instances, where B is the batch size.
Hence, we set M = 30 for text classification tasks and a smaller M (10) for SQuAD because of a larger batch size and longer sequence lengths to train SQuAD models.
The threshold of cosine similarity. We set the threshold to 0.5 to follow the work (Alzantot et al., 2018) for generating adversarial noise. The effect of the threshold: Increasing the threshold can help find more semantically similar words (even synonyms), as specified in Mrksic et al. (2016). For example, when we use this threshold, the word "award-winning" is identified to replace "charming". However, by increasing the threshold to 0.9, the substitute word becomes "lovely". However, Algorithm 1 runs much slower by denying most of the high-ranked candidates and leads to noise that is hard to make data unlearnable. Also, it stops us from deriving general unlearnable patterns via qualitative analysis of substitute words. For example, the cumulative probabilities in Table 2 would be smaller due to more varying substitution sets.

heading: C Errors of Approximating Loss
text: Changes
Generally, in our experiment, Equation 6 can always approximate the loss change in a correct direction, in our case, leading to the decrease of the actual loss. Specifically, the errors of the approximate loss change depend on the state of the models (the outcome of the outer minimization). For exam-
ple, the results (the loss on the original SST2 training instances/the loss on the modified instances/the approximate loss change) for a randomly initialized LSTM would be 0.6931/0.6833/-0.0004, while, at the other extreme, the results for the LSTM checkpoint which has converged on our label hint are 0.4457/0.0782/-0.0012 or 0.4905/0.0714/-0.0379.

heading: Future Work
text: However, we still expect future works to consider other complicated patterns beyond symbol insertions or word substitution