Year: 2021

Abstract:
Recently, increasing interest has been focused on prefix-tuning, an efficient finetuning method for large-scale pretrained language models that only needs to maintain the embedding parameters of prefix tokens for each downstream task. Despite being lightweight and modular, prefix-tuning is still lacking in robustness, as a variety of adversarial attack approaches have been proposed to fool a text classifier. However, most of the currently developed textual defense techniques necessitate auxiliary model update and storage, which will inevitably hamper the modularity and low storage of prefix-tuning. In this work, we propose a robust prefixtuning framework that preserves the efficiency and modularity of prefix-tuning. The core idea of our framework is leveraging the layerwise activations of the language model by correctly classified training data as the standard for additional prefix finetuning. During the test phase, an extra batch-level prefix is tuned for each batch and added to the original prefix for robustness enhancement. Extensive experiments on three text classification benchmarks show that our framework substantially improves robustness over several strong baselines against five textual attacks of different types while maintaining comparable accuracy on clean texts. We also interpret our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research.

heading: 1 INTRODUCTION
text: Large-scale pretrained language models (LMs) (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Yang et al., 2019; Raffel et al., 2020; Lewis et al., 2020; Brown et al., 2020; Xue et al., 2021) have proven effective for downstream NLP tasks. While finetuning a pretrained model for a specific task has been the common practice, it comes at the cost of maintaining a full copy of the LM with the parameters entirely modified. The prohibitively huge memory demand poses a severe challenge for the deployment of practical NLP systems, which motivates the development of low-storage adaptation methods (Houlsby et al., 2019; Li & Liang, 2021).
Recently, increasing interest has been focused on prompt-based tuning approaches for pretrained language models (Wallace et al., 2019; Puri & Catanzaro, 2019; Shin et al., 2020; Jiang et al., 2020b; Zhong et al., 2021; Gao et al., 2021; Hu et al., 2021; Liu et al., 2021). By prepending several elaborately-selected tokens to the given input sequences, the LM is triggered to respond with appropriate outputs without updating its parameters. Prefix-tuning (Li & Liang, 2021) introduces the idea of replacing the discrete prompt tokens at the input with the virtual ones at the start of each layer in the LM. By optimizing the layerwise continuous prefix embedding instead of selecting candidates in the vocabulary list, the expressive ability of prompts is further enhanced with a rather small amount of parameters to be updated. As a result, prefix-tuning requires near 1000× fewer task-specific parameters than finetuning the entire pretrained model (Bommasani et al., 2021).
Despite being lightweight and modular, prefix-tuning is still lacking in robustness. In the NLP community, a variety of techniques for generating adversarial examples have been proposed to attack a text classifier by perturbing inputs (Zhang et al., 2020). Conventional attack techniques include character-level (Eger et al., 2019; He et al., 2021), word-level (Alzantot et al., 2018; Ren et al., 2019; Garg & Ramakrishnan, 2020), sentence-level modification (Iyyer et al., 2018; Ribeiro et al., 2018; Xu et al., 2021), or a mixture of them (Ebrahimi et al., 2018; Li et al., 2019). Instead of perturbing each input sentence separately, recently, universal adversarial triggers (UAT) (Wallace
et al., 2019) becomes powerful by prepending the same adversarial tokens to all test inputs. UAT prompts the model to generate malicious outputs, which shares the same spirit with the promptbased tuning approaches. It remains a mystery whether prefix-tuning, a variant of prompt-based tuning techniques, can defend against UAT as well as other different kinds of attacking techniques.
In defense of adversarial attacks, different types of defense techniques are developed, including model functional improvement (Li & Sethy, 2019; Jones et al., 2020), certification (Jia et al., 2019; Huang et al., 2019; Shi et al., 2020; Xu et al., 2020; Ye et al., 2020), adversary detection (Pruthi et al., 2019; Zhou et al., 2019), and adversarial training (Miyato et al., 2017; 2019; Zhu et al., 2020; Jiang et al., 2020a; Liu et al., 2020; Wang et al., 2021; Dong et al., 2021; Zhou et al., 2021). While these approaches have enhanced model robustness, difficulties emerge when fitted to prefix-tuning. Most of the techniques require modification to the architecture and the parameters of the LM or additional maintenance of adversary detectors. Directly applying such techniques necessitates auxiliary model update and storage, which will inevitably hamper the modularity of prefix-tuning. Moreover, The excessively long time for adversarial training is also a hindrance to the efficient use of prefix-tuning. We ask the following question: Can we improve the robustness of prefix-tuning while preserving its efficiency and modularity, without modifying the pretrained model parameters?
In this work, we propose a robust prefix-tuning framework for text classification. The main idea of our framework is to add an extra batch-level prefix tuned for each batch to the original prefix embedding during test time for robustness enhancement. We first record the layerwise activations in the LM at the position of generating label prediction with correctly classified training data. We project the collected activation matrices of each layer onto low-level canonical manifolds as the charac-
terization of “correct” model behavior. In this way, the correctness of any layerwise activations at the position of prediction generation can be estimated by projecting to the canonical manifolds and measuring the distance between them. For each test batch during inference, the added extra prefix is tuned on the fly with the original prefix fixed to minimize the calculated distance. Triggered by the summed prefix, the LM is prone to generating correct label predictions. We conduct extensive experiments on three text classification benchmarks and show that the proposed framework substantially improves model robustness against five strong textual attack approaches including input perturbation attack of different levels as well as the UAT attack. To the best of our knowledge, we are the first to propose the defense approach for prefix-tuning while keeping its lightweightness and modularity. Moreover, we provide an interpretation of our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research.

heading: 2 PREFIX-TUNING FOR TEXT CLASSIFICATION
text: Prefix-tuning is a lightweight alternative to finetuning when using large-scale pretrained language models to solve downstream NLP tasks. The intuition of prefix-tuning follows prompt-based methods that a proper context prepended to input sentences triggers the desired response of the LM without changing the large amount of LM parameters. Instead of instantiating the prepended context with discrete tokens, prefix-tuning uses trainable prefix embeddings as a replacement, which is also known as soft prompts. The continuous prefix embeddings enable continuous optimization and are prepended to all Transformer layers to improve expressiveness. Following the notation of Li & Liang (2021), the activation at the i-th position of the j-th layer in an L-layer autoregressive Transformer LM is denoted as h(j)i . hi = [h (0) i ; · · · ;h (L−1) i ] represents the stacked activations:
hi = { Pθ[i, :], if i ∈ Pidx, LMϕ(zi, h<i), otherwise.
(1)
where Pidx is the sequence of prefix indices and zi is the i-th token in the input sequence. The activations of the first |Pidx| positions are directly calculated by Pθ. All of the activations at the following positions depend on the prefix as the autoregressive LM follows the left-to-right calculation process. To stabilize the optimization, the prefix embedding matrix Pθ is reparameterized as Pθ[i, :] = MLPθ(P̂θ[i, :]) by a feedforward network MLPθ with a smaller matrix P̂θ.
While prefix-tuning is proposed for conditional generation tasks, in this work, we use prefix-tuning for text classification. As shown in Figure 1, following the protocol of decaNLP (McCann et al., 2018), we frame the samples in classification tasks into a SQuAD-like scheme consisting of context, question, and label: the context and the label part refer to the text sequence to be classified and the ground-truth label, while the question part is a prescribed task description sentence fixed for all samples. We denote x = [context, question, [ANS]], where [ANS] is a special token that separates question and label. We let y = [label] and |y| = 1 as the label is one token. At the position that [ANS] is inputted, the LM generates the prediction of the next label token, and we denote this position as the output position o. While o can be different for different input x’s, in this paper, we omit the relation o = o(x) for simplicity. Prefix-tuning aims to steer the LM to maximize the probability of the label. We use all samples in the training setDtr to optimize the prefix Pθ[i, :]. The objective is
min θ E(x,y)∼DtrL(y|x; θ) = max θ
E(x,y)∼Dtr log [ W ( h(L)o )] y , (2)
where W in the LM transforms the top-layer output h(L)o to a probability vector over the vocabulary.
With continuous optimization on training samples, prefix-tuning is expected to steer the LM to generate correct label predictions for test data. With the large-scale LM parameters fixed, the obtained task-specific prefix is lightweight and modular. However, prefix-tuning is still vulnerable to text attacks. With the context part perturbed by text attack techniques, the LM can be fooled to generate erroneous label prediction at the output position. Figure 1 shows an example of perturbation: by modifying a single character m in the word remember with k, the prediction of the LM is shifted from positive to negative. Therefore, it remains under exploration how to robustify prefix-tuning without hampering its modularity or introducing additional large model updates and storage.

heading: 3 ROBUST PREFIX-TUNING
text: We propose a robust prefix-tuning framework for text classification. Our intuition follows prefixtuning that proper prefix embeddings prepended to inputs can steer a LM with correct responses.
When the inputs are adversarially perturbed, the LM activations at the output position fail to be steered in the correct way by the original prefix Pθ[i, :]. Inspired by Khoury & Hadfield-Menell (2018) that the perturbed data often deviates from the low-dimensional data manifold, our robust prefix-tuning framework uses the layerwise activations by correctly classified training data to construct canonical manifoldsM. When provided with perturbed inputs during inference, we add an extra prefix P ′ψ[i, :] tuned for each test batch to Pθ[i, :] that aims to rectify the erroneous activations at the output position so that they stay close to the canonical manifolds. In this way, we expect the summed prefix to steer the LM with correct label generation against input perturbations. As shown in Figure 1, our robust prefix-tuning framework consists of three steps. The first step is collecting correct LM activations at the output position o triggered by Pθ[i, :]. We denote SC as the set of correctly classified training examples. For the j-th layer, the collected activation matrix H(j)C stacks the j-th layer LM activation at the output position o with the input of all c ∈ SC :
H (j) C = [ h(j)o,c ] ∈ R|SC |×d. (3)
The d represents the dimension of the LM hidden state. In practice, we always have |SC | >> d. The second step is constructing canonical manifolds. We project the collected j-th layer activation matrixH(j)C onto a low-level manifoldM(j) as the characterization of the correct j-th layer behavior. We use PCA (Pearson, 1901) to get the projectionQ(j) onto the canonical manifold of the j-th layer:
H̃ (j) C = U (j)Σ(j)V (j) T , (4)
Q(j) = V (j)p T V (j)p , (5)
where H̃(j)C = H (j) C −11 TH (j) C /|SC | normalizes the rows ofH (j) C to mitigate the randomness among samples in SC before projection. V (j) p consists of the first p singular vectors and Q(j) ∈ Rd×d.
The third step is tuning P ′ψ[i, :] to robustify prefix-tuning during inference. Here the vector P ′ ψ[i, :] is not reparameterized by MLP. With the additional prefix P ′ψ[i, :], the token-wise activations become
hi =
{ Pθ[i, :] + P ′ ψ[i, :], if i ∈ Pidx,
LMϕ(zi, h<i), otherwise. (6)
For the j-th layer at the output position, the LM activation matrix triggered by Pθ[i, :]+P ′ψ[i, :] with the potentially perturbed test input batch ST is stacked as
H (j) T = [ h (j) o,t ] ∈ R|ST |×d (7)
for all t ∈ ST . We use the distance from H(j)T to the j-th canonical manifoldM(j) as the loss for the tuning of P ′ψ[i, :] for each batch. Projecting H (j) T toM(j) yields H (j) T Q (j), thus the objective is
min ψ N−1∑ j=0 L ( ψ(j), Q(j) ) = N−1∑ j=0 ∥∥∥H(j)T (I −Q(j))∥∥∥ 2 . (8)
We also replace the H(j)T in Eq. (8) with H (j) T − 11 TH (j) T /|ST | as normalization before projection to mitigate randomness among test samples when |ST | > 1. After tuning P ′ψ[i, :], the activated H (j) T is closer to M(j). As the manifold characterizes the correct behavior of the j-th layer activation, by regulating the layerwise activations at the output position, the summed prefix Pθ[i, :] +P ′ψ[i, :] is prone to steering the LM to generate correct label predictions. Our framework is also applicable to other soft prompt-based tuning methods (Qin & Eisner, 2021; Hambardzumyan et al., 2021; Lester et al., 2021; Cho et al., 2021; Tsimpoukelli et al., 2021) by recording the activations of correctly classified training data, constructing canonical manifolds for the soft prompts, and tuning additional soft prompts for robustness during inference. In this work, we conduct experiments on prefix-tuning.
Remark. From the optimal control (OC) perspective, prefix-tuning can be formalized as seeking the OC of the pretrained LM for downstream tasks, and our robust prefix-tuning can be interpreted as seeking the close-loop control for robust downstream tasks. We attach the details in Appendix G.

heading: 4 EXPERIMENTS

heading: 4.1 SETUP
text: While we have also provided the regular finetuning as baselines We consider three text classification benchmarks in our experiments: binary Stanford Sentiment Treebank (SST-2) (Socher et al., 2013), AG’s News (Zhang et al., 2015), and Stanford Natural Language Inference (SNLI) (Bowman et al., 2015). We evaluate our robust prefix-tuning with five text attacks: PWWS (Ren et al., 2019), VIPER (Eger et al., 2019), SCPN (Iyyer et al., 2018), TextBugger (“BUG” for short) (Li et al., 2019), and UAT (Wallace et al., 2019). We use the GPT2-medium (with L = 24 layers in total) as the largescale LM and set prefix length = 10 for all prefix-tuning experiments. We train 100 epochs for SST-2 and 25 epochs for AG’s News and SNLI. We setN = 3 and record the bottomN -layer activations of the LM at the output position for the additional tuning. Other experiment configurations and details can be found in Appendix B. We have also conducted both standard and adversarial full tuning experiments (Appendix A) to discuss the challenges and opportunities in robustifying prefix-tuning.

heading: 4.2 RESULTS ON ADVERSARIAL TRAINING
text: We first apply adversarial training to prefix-tuning as our baselines named adversarial prefix-tuning. Following Miyato et al. (2017), we inject perturbation restricted within the ℓ2 ball of the flattened word embedding of original sentences during training to adversarially optimize the prefix Pθ[i, :]. We have also attempted to use other types of adversarial training, (i.e., adding the KL-divergence term for regularization (Miyato et al., 2019; Zhang et al., 2019)) but obtained poor accuracy, suggesting the difficulty of optimizing the small amount of prefix embedding parameters. The experimental details can be found in Appendix C.1. Our framework is applicable to the adversarially-trained prefix embedding, as it keeps the acquired prefix Pθ[i, :] fixed and tunes extra P ′ψ[i, :] for each test batch. The experimental results are listed in Table 1. According to Table 1, our approach significantly improves the robustness of prefix-tuning over all baselines against each type of text attack. To the best of our knowledge, our framework is the first to defend against UAT on large-scale pretrained LMs (see Appendix A for details). Compared with the standard prefix-tuning baseline, the adversarial baselines achieve better robustness and clean accuracy. In fact, fewer training epochs are needed to achieve certain clean accuracy when applying adversarial training to prefix-tuning. Figure 2-(a) illustrates the loss on the training set and the clean accuracy on the validation set for each epoch during standard and adversarial prefix-tuning. From Figure 2-(a), it is clear that adversarial prefix-tuning outperforms standard prefix-tuning in both the convergence rate and generalization.
However, the adversarial prefix-tuning approaches suffer from the catastrophic drawback of taking far greater training time than the standard prefix-tuning baseline. Figure 2-(b) changes the horizontal axis in Figure 2-(a) from epoch to clock time. According to Figure 2-(b), given the same time budget, standard prefix-tuning finishes training for 100 epochs while adversarial prefix-tuning only manages to train for 20 epochs. The training losses of the two methods at the time are also roughly
the same (0.06). Earlystopping adversarial prefix-tuning at 20 epochs achieves slightly better robustness results than 100 epochs, but the clean accuracy becomes lower than standard prefix-tuning (results can be found in Appendix C.2). To conclude, the adversarial prefix-tuning approaches make regularization effect on prefix-tuning but lack competitiveness in practice.
In contrast with adversarial prefix-tuning, our approach is more practical as the duration of the inference-phase on-the-fly optimization of the additional prefix P ′ψ[i, :] is negligible compared with training-phase adversarial prefix-tuning. From Table 1, the robustness of the std. prefix-tuning applied with our approach has surpassed that of the adv. prefix-tuning without our approach. The adv. prefix-tuning with our framework achieves the strongest robustness against most attacks.

heading: 4.3 RESULTS ON ADVERSARIAL DATA AUGMENTATION
text: In this section, we conduct adversarial data augmentation during the training phase of standard prefix-tuning as our baseline methods. During training, we use a specified text attack method to perturb each batch and augment the perturbed data to the training set. Due to the computational inefficiency of discrete optimization for text attacks, the training of adversarial data augmentation is even far slower than that of adversarial prefix-tuning introduced in Section 4.2. We set milestones for the training of adversarial data augmentation at 20, 50, and 100 epochs for comparison. When applying our framework, we calculate the canonical manifolds M using the correctly classified samples from both the clean training set and the perturbed training set by the specified text attack.
Considering the training time budget, we select SST-2 as the benchmark. We use PWWS as the attack method to generate perturbed inputs with results listed in Table 2. On one hand, the clean accuracy of the PWWS adversarial data augmentation method is affected at the 20-epoch and the 50- epoch milestones. This might be attributed to the consideration of perturbed inputs during training. The robustness results, on the other hand, are improved compared with standard prefix-tuning. The robustness against PWWS obtains steady improvement thanks to the adversarial data augmentation
generated by the attack method of the same type. However, for other types of attack, the robustness improvement is decreasing with the training process, suggesting the possibility that overfitting to the specified type of adversarial data can be harmful to robustness against other types of attacks.
With our robust prefix-tuning framework applied, all milestones of the adversarial data augmentation baseline obtain substantial robustness improvement. While we calculate the layerwise canonical manifolds with the correctly classified samples from both the clean training set and the PWWSperturbed training set, it can be seen that the robustness against PWWS attack is still enhanced by several percent, while the clean accuracy is slightly affected. This shows that when calculating the layerwise manifolds, the used samples from the two types of training data, though are all correctly classified, trigger the LM with activations of different properties at the output position. It is left for future work about how to construct the dataset to calculate the canonical manifolds to achieve the best performance on both accuracy and robustness. We have also used VIPER and SCPN as the attack methods for adversarial data generation and draw similar conclusions to the PWWS adversarial data generation experiments, results of which can be found in Appendix D.

heading: 4.4 EFFECT OF TUNING DIFFERENT LAYERS
text: In this section, we study the effect of tuning our robust prefixes with different layers. While we tuned the robust prefixes of the bottom N = 3 layers for all previously reported experiments, different results are obtained when tuning the robust prefixes of alternative layers. We experiment on the SST-2 development set by tuning the bottom N -layer robust prefixes as well as the top ones with N enumerating from 0 to 24 with the step size = 3. N = 0 represents the original standard prefix-tuning method, and N = 24 means tuning all layers. The results are shown in Figure 3. According to the results, tuning the bottom N layers achieves better robustness improvement than
tuning the top ones. This can be attributed to layerwise bottom-up error accumulation of the output position activations triggered with perturbed inputs. For the in-sentence text attacks, tuning the bottom N ≤ 15 layers achieves comparable robustness improvement and slightly outperforms the larger N ’s. For the UAT attack, setting the bottom N = 3 is significantly better than the choices of larger N ’s. One possible reason for the exceptional performance is that the collected low-layer activations capture richer positional information (Voita et al., 2019), which helps to defend against UAT as UAT shifts right all inputs with the same trigger. We proceed to study how our method steers the LM with different N ’s for a better understanding of its robustness improvement in Section 5.

heading: 4.5 PERFORMANCE UNDER MORE REALISTIC THREAT MODELS
text: In the previous experiments, the robustness evaluation metric for each method is the accuracy under specific attack on the test set. While this has been adopted in previous work on adversarial defense in NLP (Dong et al., 2021; Si et al., 2021; Li et al., 2021), in this section, we consider more realistic threat models. We first study the effect of test batch size on the performance of our framework. We used an adaptive test batch size to make full use of GPU memory in previous experiments (detailed in Appendix B). However, the loss in Eq. (8) depends on the size of the inference batch, and models in deployment should respond with real-time data. Based on standard prefix-tuning on SST-2, we
additionally evaluate our framework with small test batch sizes: 1, 2, and 4. According to the results in Table 3, a fixed small test batch size achieves comparable or slightly better results under clean data and in-sentence attacks. Under UAT, however, there exists a performance gap of the robustness between our framework with test batch size of 1 and others. We discuss the phenomenon in depth and attach our attempt for improvement in Appendix E.1). Nevertheless, our framework with test batch size of 1 still substantially outperforms the baseline, and we also find the performance gap is smaller on other benchmarks with other prefix-tuning baselines (see Appendix E.1).
We continue to consider a more challenging yet more realistic setting where the test data is mixed with unperturbed and perturbed samples or perturbed samples under different attacks. We use test batch size of 1 in our framework based on adversarial prefix-tuning for SST-2. While it is unknown under which attack each test sample is perturbed (or whether it is perturbed), we find the results shown in Table 4 still impressive when using a fixed learning rate tuned on the development set under UAT for the additional tuning of P ′ψ . We attach the detailed analysis in Appendix E.2.

heading: 5 HOW DOES ROBUST PREFIX-TUNING STEER THE LM?
text: As the robust prefix-tuning framework with the proper choice of N substantially improves the robustness of prefix-tuning, it deserves to be explored how the robust prefix-tuning steers the LM compared with the original prefix. In this section, we leverage different proxies to study the behavior of our framework with the standard prefix-tuning baseline. We find that our robust prefixtuning framework steers different LM behavior between the in-sentence attacks and UAT. For the in-sentence attacks, the behavior of the final layer of the LM can be summarized as averaging the attention. For UAT, with the proper setting of N , the LM functions as ignoring the distraction.

heading: 5.1 IN-SENTENCE ATTACKS: AVERAGING THE ATTENTION
text: In this subsection, we leverage attention weights in the final layer of the LM as the proxy and use the adversarial inputs generated by the BUG attack on SST-2 development set for our following in-sentence case studies. We ignore the prefix embeddings and renormalize the attention weights of the rest of the tokens in visualization. Note that the observed behavior is of the final layer only: the attention is shown to be averaged only over the vectors inputted to the final layer, as attention is unreliable for indicating importance over input tokens (Serrano & Smith, 2019; Jain & Wallace,
2019; Wiegreffe & Pinter, 2019; Abnar & Zuidema, 2020). Table 5 shows an example of BUGperturbed input, and Figure 4 shows the attention map visualization. Similar behavior is observed as well by other in-sentence attacks; more visualization can be found in Appendix F.1.

heading: 5.2 UNIVERSAL ADVERSARIAL TRIGGERS: IGNORING THE DISTRACTION
text: In this subsection, we leverage an alternative proxy based on the product of gradient and attention as token importance rankings (detailed in Appendix F.2.1). While the proxy is not a gold-standard indicator of importance value either, it can provide more reliable predictions of token importance orderings (Serrano & Smith, 2019). We use UAT to attack SST-2 development set, and Table 5 shows an example of input with triggers “lifts mates who” with visualization in Figure 5. As a result, the
LM is steered to ignore the distraction from the trigger and assign the highest importance to the most essential token as the baseline does with clean inputs. The findings are also supported by quantitative studies. By designing metrics of corpus-level Degree of Distraction (cDoD) and Recognition of the Essential token (cRoE), we find that the scores of our framework with N = 3 are better than those of other methods with statistical significance (p < 0.001), which also explains the results in Figure 3. We attach more visualization and details of the quantitative studies in Appendices F.2.2 and F.2.3.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work, we propose robust prefix-tuning for text classification that maintains only one extra amount of embedding parameters of prefix-tuning while achieving substantial robustness improvement. We conduct both quantitative and qualitative studies to analyze the behavior of our framework. We also interpret prefix-tuning and our framework from the optimal control perspective. Future work includes constructing better canonical manifolds and extending our framework to text generation.

heading: ETHICS STATEMENT
text: Large-scaled pretrained language models have been criticized in terms of bias and stereotypes (Zhao et al., 2019; Bender et al., 2021; Field et al., 2021), and can be steered to generate malicious outputs with universal adversarial triggers (Wallace et al., 2019). The lack of robustness leads to more uncertain behavior of LM, especially under text attacks. In our work, we contribute to improving the robustness of LM with the robust prefix-tuning framework. Figure 11 illustrates an example of LM fooled to predict a gender-biased sentence as positive. With our framework, the prediction of the LM is rectified. The huge resource consumption of training large-scaled LMs has also led to sustainability concerns (Strubell et al., 2019). As specified in Appendix B and Table 11, our framework has preserved the efficiency and modularity of prefix-tuning by requiring significantly less time than all baseline methods and keeping the pretrained models unmodified.

heading: REPRODUCIBILITY STATEMENT
text: For all of our experiments, we have listed the detailed settings in Appendices B and C.1. For the theory part, we have provided our detailed formulation as well as a brief introduction to the optimal control theory in Appendix G. We have also provided the statistics of the datasets used, as well as the time and storage taken by all methods in Tables 10 and 11 and Appendix B, respectively.

heading: A COMPARISON WITH REGULAR FINETUNING
text: In this section, we provide comparison between our work with both standard and adversarial regular finetuning baselines on SST-2. For regular finetuning experiments, we use the 345M GPT2-medium LM, same as the LM adopted in the prefix-tuning experiments. We trained for 9 epochs for standard regular finetuning and 3 epochs for adversarial regular finetuning (earlystopping). The initial learning rate is also set as 5e-5 for both standard and adversarial regular finetuning. The settings of perturbation range ϵ, step size α and iteration of the inner maximization M is the same as the those adopted in adversarial prefix-tuning experiments. The results are listed in Table 6.
Due to time limit, we have only conducted regular finetuning experiments on SST-2 with one adversarial training baseline (Miyato et al., 2017) for comparison. The aim of the comparison, however, is neither to set a new SOTA nor to demonstrate that our robust prefix-tuning framework has beaten the robust finetuning techniques. On the contrary, we aim to demonstrate several special properties in the scenario of prefix-tuning, as well as challenges and opportunities in robustness:
• Difficulty in optimization. It can be seen that prefix-tuning requires more epochs to converge. A potential explanation is that the loss landscape of the prefix parameters is highly non-convex, as the downstream task should be learned with far fewer amount of free parameters. The difficulty in optimization might have also brought more challenges on robustness to prefix-tuning. According to Table 6, all prefix-tuning baselines underperform the corresponded regular finetuning approaches against almost all types of attacks.
• Trade-off between space and time. It is worthy to be mentioned that for each prefix-tuning experiment, the saved prefix parameters consume disk storage of only 2MB. In contrast, each regular finetuning experiment takes 725MB disk storage. However, prefix-tuning also takes more epochs (and thus longer time) to converge. This brings even more challenges in robustifying prefix-tuning, as an ideal solution should be neither harmful to the benefits of prefix-tuning (lightweightness, modularity, not modifying the LM parameters) nor too slow for fear that the time complexity of prefix-tuning further deteriorates. Our framework serves as a possible solution that keeps the strengths without weakening the weakness.
• Lack of robustness against UAT. According to Table 6, both adversarial regular finetuning and adversarial prefix-tuning fail to defend against UAT. A possible explanation is that the adversaries induced during adversarial training are more similar to in-sentence attacks, yet few inductive bias has been introduced against UAT. Existing defense approaches against UAT maintain additional adversary detectors. For example, DARCY (Le et al., 2021) searches for the potential triggers first, and then retrain a classifier using the exploited triggers as a UAT adversary detector; T-Miner (Azizi et al., 2021) leverages Seq2Seq model to probe the hidden representation of the suspicious classifier into a synthetic text sequence that is likely to contain adversarial triggers. In addition, neither of the methods have been tested on large-scale pretrained LM. To the best of our knowledge, our framework is the first solution to defend against UAT by tuning an additional prefix based on prefix-tuning for pretrained LM. Positioned before the adversarial triggers, the robust prefix regulates the activation at the output position so that “the distraction can be ignored”.

heading: B DETAILS OF EXPERIMENTAL SETTINGS
text: We use the SST-2, AG’s News, and SNLI benchmark for our text classification experiments. We use the 345M GPT2-medium (725MB storage) as the large-scale pretrained LM and implement prefixtuning with prefix length of 10. We initialize the prefix token embeddings with parameters from the LM word embedding matrix to ease the optimization. We train 100 epochs for SST-2 and 25 epochs for AG’s News and SNLI. We use the AdamW optimizer (Loshchilov & Hutter, 2019) provided by the HuggingFace Transformers library (Wolf et al., 2020) to optimize the prefix with initial learning rate as 5e-5 in all experiments. Other settings of prefix-tuning follows Li & Liang (2021). During inference in our framework, unless specified (Section 4.5 and Appendix E), we use an adaptive test batch size with dynamic padding to make full use of GPU memory. The total number of tokens in a test batch is determined by the setting of the ratio of GPU memory (0.12 for 24GB NVIDIA-3090 GPUs) allocated for loading data. In contrast, the number of sentences in a test batch is fixed in experiments in Section 4.5 and Appendix E.
The size of obtained prefix vectors are 1.9MB for all tasks. We also save the projection matrices of the layerwise canonical manifolds, but only the matrices of the bottom N = 3 canonical manifolds need to be saved as we tune the bottomN = 3 layers in our robust prefix-tuning. The saved matrices take 12MB storage and can be used for all types of text attacks. We use the Adam optimizer (Kingma & Ba, 2015) to tune the additional prefix P ′ψ[i, :]. The initial learning rate and the number of steps (5 or 10) for its optimization are tuned on the development set.
We use the following five text attack approaches in our experiments:
• PWWS (Ren et al., 2019), a score-based word-level text attack that fools a text classifier by applying greedy word substitution on inputs.
• VIPER (Eger et al., 2019), a blind character-level text attack that applies visually similar character substitution on inputs.
• SCPN (Iyyer et al., 2018), a blind sentence-level text attack that paraphrases inputs.
• TextBugger (Li et al., 2019), a gradient-based word- and character-level attack that applies greedy word substitution and character manipulation. We refer to it as “BUG” for short.
• UAT (Wallace et al., 2019), a gradient-based word-level text attack that fools a text classifier by prepending the same adversarial tokens to all test inputs.
We use the OpenAttack toolkit (Zeng et al., 2021) for the in-sentence attacks (PWWS, VIPER, SCPN, TextBugger) and implement UAT by ourselves. For the in-sentence attacks, we use the default hyperparameter settings provided by OpenAttack. Following prior arts (Ren et al., 2019; Jia et al., 2019; Dong et al., 2021), we do not attack the premise on SNLI. For UAT, following Wallace et al. (2019), we search for a 3-token trigger for inputs of each class in the test set. We set beam size as 5 and iterate over the test set for 5 epochs. We use NVIDIA-3090 GPUs for all of our experiments.

heading: C ADVERSARIAL TRAINING

heading: C.1 ADVERSARIAL TRAINING BASELINES
text: We start with adversarial training for text classification that restricts perturbation within ℓ2 balls of word or sentence level (Miyato et al., 2017). The objective is defined as
min θ E(x,y)∼Dtr [ max ∥r∥2≤ϵ L(y|emb(x) + r; θ) ] , (9)
where r is the perturbation injected to the word embeddings of x. The inner maximization of r can be implemented by projected gradient descent (Madry et al., 2018) with M iterative updates:
r ← Proj{∥r∥2≤ϵ} [ r + α
∇emb(x)L(y|emb(x) + r; θ) ∥∇emb(x)L(y|emb(x) + r; θ)∥2
] . (10)
In all of our adversarial prefix-tuning experiments, we set ϵ = 5, α = 1.25, and M = 10.
For texts, the ℓ2 ball can be defined for both word level and sentence level. For the word level, the ℓ2 balls are constructed for each token in the context part of the input by offsetting each word vector
within the range of radius of ϵ. For the sentence level, the single ℓ2 ball is constructed for the entire context part of the input by flattening the word embedding vectors and offsetting the flattened vector within the range of radius ϵ. The word-level and the sentence-level prefix-tuning take similar training time. As a consequence, adversarial prefix-tuning with both levels of ℓ2 balls are far slower than standard prefix-tuning. The adversarial training approach used in Table 1 is of word-level ℓ2 balls. The complete results of our approach applied with both word-level and sentence-level adversarial prefix tuning, as well as standard prefix-tuning, are shown in Table 7. It can be seen that our robust prefix-tuning framework substantially improves robustness of different baseline methods.
We have also experimented with other types of adversarial training for prefix-tuning, such as adding the KL-divergence term for regularization (Miyato et al., 2019; Zhang et al., 2019). The objective is
min θ E(x,y)∼Dtr
[ L(y|x; θ) + max
∥r∥2≤ϵ βKL [L(y|emb(x); θ) ∥ L(y|emb(x) + r; θ)]
] . (11)
We set β = 1 and β = 4 and train for 20 epochs for different attempts of KL-divergence regularized adversarial prefix-tuning with Eq. (11). The training loss and the clean accuracy of the development set on the SST-2 benchmark is shown in Figure 6. In this figure, we also visualize the word-level adversarial prefix-tuning method shown in Figure 2 for comparison. The KL-divergence regularized adversarial prefix-tuning of both settings start with very large training loss. For β = 1, the clean accuracy on the development set remains 0 for the first 15 epochs. The accuracy raises to around 27% at the 17-th epoch, but lacks stability (back to 0 again at the 19-th epoch). For β = 4, the clean accuracy remains 0 for all 20 epochs. In comparison, the word-level adversarial prefix-tuning used in Section 4.2 achieves low training loss and high clean accuracy on the development set. The training loss and the accuracy on the development set have suggested the difficulty in optimizing the prefix embedding, which might be attributed to its rather small amount of parameters compared with the whole language model. While there are other variants of adversarial training approaches for text classification, in this work, we did not use them for our baselines as most of them (Zhu et al., 2020; Jiang et al., 2020a; Liu et al., 2020; Dong et al., 2021) are based on the KL-divergence regularizer, and also require excessively long training time. It is also of future interest on how to empower our robust prefix-tuning framework with faster and better adversarial prefix-tuning approaches.

heading: C.2 ADVERSARIAL TRAINING WITH EARLYSTOPPING
text: We apply earlystopping to both word-level and sentence-level adversarial prefix-tuning approaches on the SST-2 benchmark at 20 epochs. The results are shown in Table 8. According to the results, the robustness results against different types of in-sentence attacks have been slightly improved. The robustness against UAT, in contrast, slightly drops. In terms of clean accuracy, the earlystopped adversarial prefix-tuning is even surpassed by the standard prefix-tuning baseline. Our robust prefixtuning framework still works on the prefixes obtained by the earlystopped adversarial prefix-tuning approaches, as the robustness results are consistently improved.

heading: D ADVERSARIAL DATA AUGMENTATION
text: In this section, we report additional experimental results of adversarial data augmentation using VIPER or SCPN to generate adversarial data for augmentation during training. We also set the milestones of 20, 50, and 100 epochs for training. According to Table 9, the robustness against the attack of the same type used for adversarial data augmentation during training has gained significant improvements compared with the standard prefix-tuning baseline. Robustness against other types of attacks, however, fails to improve with the process of training. For our approach, we use the correctly classified data from both the clean training set and the training data perturbed by the specific type of attack used for augmentation to construct the canonical manifolds. The experimental results indicate that our robust prefix-tuning framework still improves the robustness against all types of text attacks over all types of baselines (except for the VIPER attack in VIPER adversarial data augmentation that incurs slight drop at the 20-epoch and 50-epoch milestones).
We attach the clock time that each method takes on each benchmark for comparison in Table 11. The time differs for each benchmark because of different sizes (Table 10). The training-phase adversarial prefix-tuning takes far greater time than standard prefix-tuning. The adversarial data augmentation methods during training phase take the longest time due to the computationally-challenging discrete search/optimization of text attacks. For our framework, steps 1 and 2 that construct the canonical manifolds are viewed as preprocessing since it can be used against all attacks during inference. As ∥SC∥ >> d (mentioned in Section 3), we can further speed up step 1 by random sampling in SC . The additional prefix P ′ψ[i, :] is tuned on the fly during test phase in step 3 of our framework, which slackens the standard testing but is negligible compared with the training-phase baseline methods. In conclusion, our framework has preserved the efficiency and modularity of prefix-tuning.

heading: E MORE REALISTIC EVALUATION SETTINGS

heading: E.1 PERFORMANCE WITH SMALL TEST BATCH SIZES
text: In this section, we continue to study why our methods with test batch size of 2 or 4 achieve stronger robustness against UAT (shown in Table 3). We note that the j-th layer activation at the output position H(j)T is normalized by rows before being projected to the canonical manifoldM(j) when the test batch size is larger than > 1. According to Eq. (8) and the description in Section 3, when the size of a test batch is larger than 1, namely, when
H (j) T = [ h (j) o,t ] ∈ R|ST |×d (12)
has more than 1 row (|ST | > 1), the objective for tuning P ′ψ is
min ψ N−1∑ j=0 L ( ψ(j), Q(j) ) = N−1∑ j=0 ∥∥∥(H(j)T − 11TH(j)T /|ST |)(I −Q(j))∥∥∥ 2 . (13)
The normalization is performed to alleviate the randomness among the samples in the test batch. It is not applicable when |ST | = 1, as H(j)T − 11 TH (j) T /|ST | would be 0. Therefore, for test batch size = 1, the loss for tuning P ′ϕ is Eq. (8) without normalization before projection.
We conduct experiments with test batch sizes of 2 and 4 without normalizing the test batch before projection during inference for comparison. According to the results in Table 12, for test batch sizes of 2 and 4, robustness results against UAT have degenerated without the normalization, but they are still better than that with test batch size of 1 as mini-batch optimization may achieve better results than the fully online setting. The robust performance of our framework against other in-sentence attacks remains similar, with or without normalizing.
The results lead to a temporary conclusion that normalization before projection during inference can be important when using our framework to defend against UAT. However, the performance gap between the unnormalized test batch size = 1 setting and the normalized test batch size > 1 setting is much smaller on other benchmarks and other prefix-tuning baselines. We set test batch size as 1 and evaluate the performance of our framework with both standard and adversarial prefix-tuning on both SST-2 and AG’s News benchmarks. The results in Table 13 show that the robustness gap against UAT between the unnormalized test batch size = 1 setting and the normalized test batch size > 1 setting, though still exists, is relatively small compared with the results in Table 12.
As a result, for conventional evaluation where accuracy under specific attack is reported, we will use test batch size larger than 1 (2 or 4 should be satisfactory), as normalization before projection for additional tuning during inference can be beneficial. In contrast, we will set test batch size as 1 in more realistic settings, as it is able to cope with real-time or mixed test data.
Our another attempt for improvement is to set the normalization before projection during inference with pre-computed vectors beforehand. Currently the normalization 11TH(j)T /|ST | in Eq. (13)
depends on H(j)T , with limitation in the setting of test batch size = 1 and concern in reproducibility. In our additional experiments, we pre-compute the normalization of the j-th layer activation during inference using SC , the set of correctly classified training examples. Consequently, we can replace 11TH
(j) T /|ST | in Eq. (13) with 11 TH (j) C /|SC |, which can be obtained in the second step in Section
3. In this way, the normalization is static for different batches and applicable to test batch with the size of 1. The results are shown in Table 14.
On the one hand, it can be seen that our framework with static normalization outperforms the ones with dynamic normalization under UAT. The improvement is especially substantial for the setting with test batch size = 1, indicating that while the dynamic normalization 11TH(j)T /|ST | is not applicable for test batch size = 1, the static normalization 11TH(j)C /|SC | can be helpful in this setting. On the other hand, for in-sentence attacks, the robustness results are comparable between static and dynamic normalization, though there is a performance drop under VIPER. Due to time limit, we have only experimented on SST-2 with standard prefix-tuning as the baseline. We will conduct more experiments to further evaluate static normalization with other baselines on different benchmarks. In the following experiments under more realistic threat models, we adopt the setting of the test batch size = 1 without normalization before projection.

heading: E.2 PERFORMANCE UNDER MIXED TEST DATA
text: In this section, we provide more experimental results of our framework under mixed test data. We combine the clean test set with the test set under one attack, or perturb the test set with two different attacks separately and combine the two sets as mixed test data. The mixed test data can be from a more realistic scenario. To cope with it, we set the test batch size as 1 in order to avoid the situation where both perturbed and unperturbed test sentences or perturbed sentences under different attacks exist in a test batch. Besides, by setting the test batch size as 1 we can deal with real-time data, which is also a practical approach. We use adversarial prefix-tuning as the baseline to obtain a strong prefix Pθ[i, :]. While we are not able to tune the robust prefix under specific attack in this setting, we find it effective to tune the initial learning rate and the number of steps (5 or 10) for our inference-phase robust prefix on the UAT-attacked development set and fix them for every incoming test sentence. The results are listed in Table 15.
According to the results, the performance under mixed test data is consistently improved using our framework. We also provide the averaged results with the robust prefix in our framework separately tuned on the two test sets as an upper bound for comparison. It can be seen that small performance gap exists between the results under mixed data and the averaged results with separately tuning for specific test sets. A potential reason is that our framework can better adapt to the data under specific attack. To reduce the performance gap, we can construct more powerful canonical manifolds that capture richer data information of multiple granularity so that our framework also achieves optimal performance under mixed test data. We leave the exploration for future work.

heading: F MODEL BEHAVIOR ANALYSIS
text: F.1 IN-SENTENCE ATTACKS: AVERAGING THE ATTENTION
We provide the attention weights from the LM final layer over the entire x = [context, question, [ANS]] with PWWS as the text attack from the development set of SST-2. The perturbed inputs are shown in Table 16, and the visualizations are shown in Figure 7, Figure 8, and Figure 9. Similar behavior of averaging the attention of the final layer in the LM can be also observed as the BUG attack visualized in Figure 4. Caveat: the observed behavior is of the final layer only, as attention is unreliable for indicating importance over input tokens (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Abnar & Zuidema, 2020; Serrano & Smith, 2019).

heading: F.2 UNIVERSAL ADVERSARIAL TRIGGERS: IGNORING THE DISTRACTION

heading: F.2.1 THE ALTERNATIVE PROXY
text: In this section, we introduce the alternative proxy for the analysis of token importance as a replacement of attention. To interpret token importance, following Serrano & Smith (2019), we leverage an alternative gradient-based importance ranking proxy. As stated in Serrano & Smith (2019), while it does not provide a gold-standard indicator of importance value for each token to the model, the proxy is more reliable in reflecting the importance ordering for tokens in the sequence. In the proxy, the importance for each token is given by the multiplication of the gradient of the decision function with respect to each attention weight and the attention weight magnitude. As there are either 2, 3, or 4 classes in our text classification benchmarks, the decision function at the output position is given by
do
( W ( h(L)o )) = expmaxi
[ W ( h (L) o )] i∑
i exp [ W ( h (L) o )] i , (14)
where W in the LM transforms the top-layer output h(L)o to a probability vector over the dictionary, and i ∈ Slabel traverses all the class label tokens in the vocabulary (|Slabel| = the number of classes in the benchmark). The decision function is proposed in Serrano & Smith (2019) to analyze hierarchical attention network (Yang et al., 2016). We further augment the decision function for time steps < o to study the entire behavior of how the autoregressive LM assigns importance to each token visible at each time step. As introduced in Section 2, the autoregressive LM generates sequence [x, ỹ] with [[CLS], x] as the input sequence for text classification tasks (ỹ is the generated label token as prediction). For each time step < o, the output of the LM is a token from x. As a result, we define the decision function at the k-th position as
dk
( W ( h (L) k )) =
exp [ W ( h (L) k )] xk∑
j∈V exp [ W ( h (L) k )] j , (15)
with xk−1 (or [CLS] for k = 0) as input and xk as output. The complete decision function is then d ( W ( h (L) 0 ) , · · · ,W ( h (L) k ) , · · · ,W ( h(L)o )) = o∑ k=0 dk ( W ( h (L) k )) . (16)
In this way, at each time step i, the gradient taken from the decision function (Eq. (16)) with respect to each attention weight aij (j ≤ i) in the final layer of the LM can be obtained with automatic differentiation in PyTorch (Paszke et al., 2019). We multiply the gradient with the attention weight magnitude as the value of token importance, which ranks a token with both high attention weight and high calculated gradient to be most important. Formally, for the attention weight on the j-th token at the i-th time step in the h-th head of the final layer (namely a(h)ij ), its importance value is
I (h) ij =
∂d
∂a (h) ij
a (h) ij , (17)
where j ≤ i. The upper triangle part of the importance matrix is still masked, namely I(h)ij = 0 when j > i. We average the importance matrix over all heads in the final layer. Note that the averaged importance value can be negative, as each calculated gradient ∂d/∂a(h)ij in Eq. (17) can be negative. For better visualization, we further normalize all the token importance values at each time step i by subtracting the minimum value so that the negative values are eliminated. We finally normalize all the (now positive) token importance values at each time step i into a probability vector. Formally,
I ′ij = 1
H ∑ h I (h) ij , Îij = I ′ ij − min 0≤j≤i I ′ij , Iij = Îij∑i j=0 Îij , (18)
where H is the number of heads in the final layer. The latter two equations transform I ′[i, :] to I[i, :] as a probability vector. This is similar to an attention row, which is a better form for visualization. The importance matrix I is plotted for all the case studies under UAT attack (Figure 5 and Appendix F.2.2). Also note that the transformation from I ′ to I is order-preserving for each row, and we will only use the importance ordering information in quantitative studies F.2.3.

heading: F.2.2 QUALITATIVE ANALYSIS
text: We use UAT to attack inputs from the development set of SST-2 for visualization. The perturbed inputs are shown in Table 17, and importance visualizations are shown in Figures 10, 11, 12. Here we also plot the importance between each token pairs of the adversarial triggers. We can observe the behavior of ignoring the distraction for our robust prefix-tuning framework with N = 3.

heading: F.2.3 QUANTITATIVE ANALYSIS
text: In this section, we conduct quantitative analysis to show that for UAT, the robust prefix steers the LM to ignore the distraction of the adversarial triggers. We first investigate whether the trigger is assigned with less importance under the robust prefix. Based on the proxy, at each time step i, we get the rankings Kij for all visible tokens xj :
K[i, : i+ 1] = argsort(I[i, : i+ 1]) (19)
where the argsort operator, provided in frameworks like NumPy (Harris et al., 2020) and PyTorch (Harris et al., 2020), returns the ranking of each value in the vector by increasing order. Our quantitative analysis is based on token importance rankings at each time step, as the proxy I does not provide a gold indicator for importance value but rather an alternative ordering method. We investigate the importance of the adversarial trigger by calculating the relative ranking of the token with maximal importance in the trigger. We set the trigger length as 3 in our experiments, so for each time step i ≥ 3, the relative ranking is
max(K[i, : 3])/(i+ 1)× 100%. (20) The example-level Degree of Distraction (DoD) is defined as the importance of the adversarial trigger averaged at all time steps since the generation of the first token in context:
DoD(x) = 1
o− 3 o∑ i=3 max(K[i, : 3])/(i+ 1)× 100% (21)
where x represents the input example and K is the calculated importance ranking matrix. Corpuslevel Degree of Distraction (cDoD) is then defined as the example-level degree of distraction averaged over all UAT-attacked examples. We calculate the cDoD of the original prefix-tuning as well as our framework with N = 24 and N = 3 under UAT attack on the SST-2 development set. The results are listed in Table 18. Similar to a trustworthy evaluation for machine translation (Koehn,
2004; Marie et al., 2021), we also conduct statistical significance tests by bootstrapping in the sets of DoD of all test samples calculated for different methods. According to Table 18, the cDoD results of our framework (both N = 24 and N = 3) are lower than that of the original prefix-tuning with statistical significance (p < 0.001).
We proceed to study whether the robust prefix manages to steer the LM to assign the highest importance to the most essential token as the original prefix-tuning does with inputs without adversarial triggers. Let I0 be the importance matrix for the baseline method with clean input, and Iu be the importance matrix for some method with the same input but prepended with adversarial trigger tokens. Let the notation o indicate the output position for the UAT-attacked input, then the output position for the clean input is o− 3 (3 is the trigger length). At each time step i, We use the indicator
Eu(i) = 1 (argmax (I0[i, : i+ 1]) = argmax (Iu[i+ 3, : i+ 4])− 3) , 0 ≤ i ≤ o− 3 (22) to determine whether the method with UAT-attacked inputs recognize the essential token by assigning the highest importance to the same token (argmax (Iu[i+ 3, : i+ 4]) − 3, left-shifted by the trigger length 3) as the baseline with clean inputs does (argmax (I0[i, : i+ 1])). The subtraction of 3 in the RHS of Eq. (22) is due to the right shift of the original input by the trigger tokens. Similar to the calculation of DoD and cDoD, we compute the example-level recognition of the essential token (RoE) for different methods under UAT attack by averaging Eu(i) over all time steps:
RoE(x) = 1
o− 3 o−3∑ i=0 Eu(i). (23)
We then compute the corpus-level Recognition of the Essential token (cRoE) by averaging RoE over all test samples. We calculate the cRoE of the original prefix-tuning as well as our framework with N = 24 and N = 3 under UAT attack on the SST-2 development set.
The results are listed in Table 19. It can be seen that our framework withN = 3 achieves the highest cRoE score, which is also statistically significant compared with the standard prefix-tuning baseline and our framework with N = 24. We also find that the cRoE score of our framework with N = 24 is lower than that of the baseline. As a result, though the cDoD score shows that our framework with N = 24 ignores the distraction of the adversarial triggers to some extent, it seems that the robust prefix with N = 24 fails to steer the LM to assign the highest importance to the most essential token under UAT attack. In contrast, when the LM is attacked by UAT, the robust prefix-tuning with N = 3 steers the LM so that it not only ignores the distraction from adversarial triggers, but is also (very much likely to be) able to assign the highest importance to the most essential token as the baseline method does with the input without adversarial trigger tokens.
The results from Tables 18 and 19 provide an explanation to the performance shown in Figure 3-(a) that under UAT attack, our framework with N = 3 substantially outperforms that with N = 24, and both of them achieve higher robustness compared with the standard prefix-tuning baseline. We hope that our quantitative studies can better support the observed behavior and provide a deeper understanding of the robust prefix-tuning framework.
G INTERPRETATION FROM THE OPTIMAL CONTROL PERSPECTIVE
In this section, we provide an interpretation from the optimal control perspective for prefix-tuning as well as our robust prefix-tuning framework. Prefix-tuning seeks Pθ[i, :] with forward propagation of the j-th layer of the L-layer LM at the output position o as
h(j+1)o = LM (j) Φ ( h(j)o ∣∣∣ h(j)<o) (24) with h(j)i = Pθ(j) [i, :] for all j = 0 to L− 1, i ∈ Pidx and h (0) i = zi for i /∈ Pidx to optimize
min {θ(0),...,θ(L−1)} E(x,y)∼Dtr S (h(L)o , y)+ L−1∑ j=0 R ( θ(j) ) , (25) with S as the softmax scoring function, R as the regularizer, Φ as the LM parameters, zi as the i-th token in the input and y as the label. According to Li et al. (2017), the S and R can be viewed as the terminal and the running loss with Pθ as the control variables, and the forward and backward propagation of prefix-tuning are equivalent to the calculation of co-state process in Pontryagin’s Maximum Principle (Kopp, 1962). Therefore, prefix-tuning can be formalized as seeking the optimal control of the pretrained models for specific downstream tasks.
Similarly, Our robust prefix-tuning framework seeks P ′ψ[i, :] with h (j) i = Pθ(j) [i, :] + Pψ′(j) [i, :] for all j = 0 to L− 1, i ∈ Pidx in the forward propagation of Eq. (24). Our goal is given by
min {ψ(0),...,ψ(L−1)} E(x,y)∼Dte L−1∑ j=0 ℓ ( ψ(j), Q(j) ) , (26)
where the running loss ℓ is given by Eq. (8) for j ≤ N − 1 and set to 0 for j ≥ N . Compared with Eq. (25), there is no terminal loss in Eq. (26) as the label is unknown during test phase. As
suggested by Chen et al. (2021), our robust prefix-tuning can be formalized as seeking the close-loop control for robust downstream tasks.
Before using optimal control theory to prove the above formalizations, we first provide a brief introduction to Li et al. (2017) that reveal the relationship between the optimal control theory and deep learning. We borrow the theorems in Section 4 of Li et al. (2017) and directly follow their notations: Theorem G.1. (discrete-time PMP) Consider the discrete-time control problem
min {θ0,...,θT−1}∈ΘT Φ(xT ) + δ T−1∑ t=0 L(θt),
xt+1 = xt + δft(xt, θt), x0 = x, 0 ≤ t ≤ T − 1
(27)
where Φ is the termination loss and L is the running loss. Then there exists a co-process
x∗t+1 = gt(x ∗ t , θ ∗ t ), x ∗ 0 = x, (28)
p∗t = ∇xHt(x∗t , p∗t+1, θt), p∗T+1 = −∇xΦ(x∗T+1) (29) such that
Ht(x ∗ t , p ∗ t+1, θ ∗ t ) ≥ Ht(x∗t , p∗t+1, θ), θ ∈ Θ, 0 ≤ t ≤ T − 1. (30)
Here gt(xt, θt) := xt + δft(xt, θt) and
Ht(x, p, θ) = p · gt(x, θ)− δL(θ) (31) is the scaled discrete Hamiltonian. Theorem G.2. (discrete-time MSA) The co-process in Theorem G.1 can be determined by applying the discrete-time method of successive approximations (MSA). For each iteration k,
set xk0 = x, and xkt+1 = gt(x k t , θ k t ) (32) with t enumerating from 0 to T − 1; then set pkT = −∇xΦ(xkT ), and pkt = ∇xHt(xkt , pkt+1, θkt ) (33) with t enumerating from T − 1 to 0; finally, with t enumerating from 0 to T − 1, set
θk+1t = θ k t + η∇θHt(xkt , pkt+1, θkt ). (34)
Theorem G.3. (equivalence between MSA and back-propagation) The MSA in Theorem G.2 is equivalent to gradient descent with back-propagation.
The proofs of Theorems G.1, G.2 and G.3 are provided in Li et al. (2017).
We now investigate the forward propagation of prefix-tuning. In Eq. (24), LM(j)Φ , the j-th layer of the LM, can be decomposed into a self-attention layer (SAN(j)Φ ) and a FFN layer (FFN (j) Φ ). Formally,
h(j+1)o = h (j) o + SAN (j) Φ ( h(j)o , h (j) <o ) + FFN (j) Φ ( h(j)o + SAN (j) Φ ( h(j)o , h (j) <o )) (35)
with h(j)i = Pθ(j) [i, :] for i ∈ Pidx. As Eq. (35) is recursive and according to the fact that the pretrained LM is autoregressive, after unrolling the recursion for all h(j)<o in Eq. (35), h (j+1) o can be represented in the form of h(j+1)o = h (j) o + G (j) Φ ( h(j)o , θ (j) ) . (36)
Now we take Eq. (25), the objective of prefix-tuning, into consideration. The optimization problem is now formulated as
min {θ0,...,θL−1}∈ΘL E(x,y)∼Dtr S (h(L)o , y)+ L−1∑ j=0 R ( θ(j) ) h(j+1)o = h (j) o + G (j) Φ ( h(j)o , θ (j) ) , h(0)o = zo = [ANS], 0 ≤ j ≤ L− 1.
(37)
Using Theorem G.1, we know that the objective of prefix-tuning can be formulated as a discretetime control problem. We then use the MSA described in Theorem G.2 to determine the co-process that solves the control problem and realize that the MSA is equivalent to back-propagation with Theorem G.3, which recovers the training phase of prefix-tuning. As a result, we can conclude that prefix-tuning seeks the optimal control of the pretrained models for specific downstream tasks.
Our robust prefix-tuning framework can also be formalized as the close-loop control of the obtained prefix for robust downstream tasks. According to Chen et al. (2021), the close-loop control framework for robust neural networks is defined as
min {π0,...,πL−1} L−1∑ t=0 ℓ(xt, πt(xt)),
xt+1 = ft(xt, πt(xt)), x0 = x, 0 ≤ t ≤ L− 1.
(38)
where ℓ(xt, πt(xt)) is the running loss. According to Eq. (26), the goal of our robust prefix-tuning framework can be formulated into the form
min {ψ(0),...,ψ(L−1)} E(x,y)∼Dte L−1∑ j=0 R(j) Φ,θ(j) ( h(j)o , ψ (j) ( h(j)o )) ,
h(j+1)o = h (j) o + G̃
(j) Φ,θ(j) ( h(j)o , ψ (j) ( h(j)o )) , h(0)o = zo = [ANS], 0 ≤ j ≤ L− 1.
(39)
Since Theorems G.1, G.2 and G.3 can also be applied to Eq. (39), the optimization of our robust prefix is equivalent to seeking the close-loop control for robust downstream tasks.
In conclusion, recent years have witnessed the paradigm shift in NLP from adapting LMs to downstream tasks to adapting downstream tasks to LMs (Liu et al., 2021). In this work, we provide the interpretation of prefix-tuning as well as our robust prefix-tuning framework from the optimal control perspective. We hope that our work can bring insight to the future design of robust and efficient finetuning approaches for large-scale pretrained models.

Review summaries:
This paper focuses on improving the adversarial robustness of prefix tuning (Li et al. 2021), which is a recent parameter-efficient tuning method. Specifically, the paper proposes to add extra batch-level prefixes that are tuned for each test batch on the fly, to minimize the distance between hidden activations of the test samples and the canonical manifold obtained from the hidden activations from correctly classified training samples. The intuition is to optimize the added batch-level prefixes so that the geometry of hidden states from adversarial examples is closer to that of training examples. Experiments on three text classification benchmarks across several different adversarial attacks demonstrate the effectiveness of the method.

 This paper introduces a tweak to Prefix-Tuning to make it more resilient to adversarial perturbations of the input. The idea is to add a batch-level prefix at inference to the original one which enhances robustness. Critically, Robust Prefix-Tuning (RPT) does not require auxiliary model updates or storage, in contrast with other robustness methods.Thus, this approach makes prefix-tuning more robust while preserving its modularity and low storage requirements.  The authors conduct experiments on 3 text classification tasks, 5 textual attacks and different training regimes (normal training, adversarial training and adversarial data augmentation). In nearly all instances, their method improves robustness (sometimes considerably so) while preserving the accuracy on the original text.    The authors also present RPT from an optimal control perspective and conduct a qualitative study that shows how RPT impacts attention weights. 

 The paper is a focused contribution at the intersection of defending against text attacks and prompt tuning. The paper requires the reader to understand the context and motivation of several different things before understanding the contribution of the paper. First, adversarial examples can attack a text classifier, such as UAT. Second, various techniques defend against these attacks in different way. However, these techniques requiring modifying the parameters of the LM or other additional computational burdens. These techniques can be used with prompt tuning, but then the benefits of prompt tuning go away. Hence, there ought to be a technique that improves the robustness of prompt tuning without removing its benefits over regular finetuning. The paper proposes such a technique and do experiments for three text classification tasks and various adversarial attacks.