On Robust Prefix-Tuning for Text Classification
Authors: Zonghan Yang
Yang Liu
Venue: ICLR 2022 Poster (2022)
Keywords: prefix-tuning
pretrained language models
text classification
robustness in NLP
optimal control

Abstract: Recently, prefix-tuning has gained increasing attention as a parameter-efficient finetuning method for large-scale pretrained language models. The method keeps the pretrained models fixed and only updates the prefix token parameters for each downstream task. Despite being lightweight and modular, prefix-tuning still lacks robustness to textual adversarial attacks. However, most currently developed defense techniques necessitate auxiliary model update and storage, which inevitably hamper the modularity and low storage of prefix-tuning. In this work, we propose a robust prefix-tuning framework that preserves the efficiency and modularity of prefix-tuning. The core idea of our framework is leveraging the layerwise activations of the language model by correctly-classified training data as the standard for additional prefix finetuning. During the test phase, an extra batch-level prefix is tuned for each batch and added to the original prefix for robustness enhancement. Extensive experiments on three text classification benchmarks show that our framework substantially improves robustness over several strong baselines against five textual attacks of different types while maintaining comparable accuracy on clean texts. We also interpret our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research.

Published as a conference paper at ICLR 2022
ON ROBUST PREFIX-TUNING FOR TEXT CLASSIFICA- TION
Zonghan Yang, Yang Liu
Department of Computer Science and Technology, Institute for Al Industry Research Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China yangzh20 06 1s. tss Eisinghua... www.uaeeuuuu.......a.mmmmm cn, 1iuuangg0111111 @t wtt.innggaa....... edu. cn
ABSTRACT
parameter-efficient Recently, prefix-tuning has gained increasing attention finetuning method for large-scale pretrained language models. The method keeps the pretrained models fixed and only updates the prefix token parameters for each downstream task. Despite being lightweight and modular, prefix-tuning still lacks robustness to textual adversarial attacks. However, most currently developed de- fense techniques necessitate auxiliary model update and storage, which inevitably hamper the modularity and low storage of prefix-tuning. In this work, we propose a robust prefix-tuning framework that preserves the efficiency and modularity of prefix-tumingg The core idea of our framework is leveraging the layerwise activa- tions of the language model by correctly for additional prefix finetuning. During the test phase, an extra batch-level prefix is tuned for each batch and added to the original prefix for robustness enhance- ment. Extensive experiments on three text classification benchmarks show that our framework substantially improves robustness over several strong baselines against five textual attacks different types while maintaining comparable ac- curacy clean texts. We also interpret our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research
training data as the standard
INTRODUCTION
Large-scale pretrained language models (LMs) (Peters et al., 2018 Devlin et al., 2019 Radford al. 2019 Liu et al. 2019 Yang et al.) 2019 Raffel et al. [2200] Lewis 2020 Brown al. 2020 Xue et al.) 2021 1) have proven effective for downstream NLP tasks. While finetuning pre- trained model for specific task has been the common practice, comes at the cost of maintaining full copy of the LM with the parameters entirely modified. The prohibitively huge memory de- mand poses severe challenge for the deployment of practical NLP systems, which motivates the development of low-storage adaptation methods (Houlsby et al. 2019 Li & Liang 2021).
Recently, increasing interest has been focused on prompt-based tuning approaches for pretrained
language models (Wallace et al.] 2019 Puri & Catanzaro, 2019 (Shin al., 2020; Jiang et al.) 20206 Zhong et al. 2021 Gao et al., 2021; Hu et al.) 2021]]
Liu Liu et al.) 2021) By prepending tokens to the given input sequences, the LM triggered to respond with
several claborately-selece
appropriate outputs without updating its parameters. Prefix-tuning (Li & Liang 2021) introduces the idea of replacing the discrete prompt tokens the input with the virtual ones the start of each layer in the LM. By optimizing the layerwise continuous prefix embedding instead of selecting candidates in the vocabulary list, the expressive ability of prompts further enhanced with rather small amount of parameters to be updated. As result, prefix-tuning requires near 1000 fewer task-specific parameters than finetuning the entire pretrained model (Bommasani et al. 2021
Despite being lightweight and modular, prefix-tuning is still lacking in robustness. In the NLP community, a variety of techniques for generating adversarial examples have been proposed to attack a text classifier by perturbing inputs (Zhagg et al. 2020). Conventional attack techniques include
character-level (Eger et al. 2019 He et al. 2021 word-level (Alzantot et al. 2018 Ren et al.
We release the cade atthtps::/// thub. com wommmmniii chesh -ttp: runing

---

Published as a conference paper at ICLR 2022
Prefix-Tuning
Prefix Pa
Transformer (Pretrained)
Remember it
is this review negative or positive
[ANS] positive
context (original)
question (fixed for all Inputs)
label
Robust Prefix-Tuning
collect activations correctly classified data
construct canonical manifolds M by PCA
Prefix P
calcula ate the distance between the activation by Pe and the canonical manifolds M for additional tuning Pú
Prefix P
Transformer (Pretrained)
ke
mber It
this
positive
[ANS]
context (perturbed)
question
Figure 1: Overview of prefix-tuning as well as our robust prefix-tuning framework for text classifi- cation. We frame the samples into SQuAD-like scheme consisting of context, question, and label and optimize the original prefix Pa- For the robust prefix-tuning framework, we fix the obtained pre- fix Pe and tune an additional prefix P for each test batch. The additional tuning follows the three steps indicated in the figure, which aims to lead the summed prefix to steer correct activations the position of the [ANS] token with those activated by correctly classified training data as the standard.
Garg & Ramakrishnan, 2020). sentence-level modification [yyer et al., 2018 Ribeiro et al., 2018 Xu et al. (022), or mixture of them (Ebrahimi et al. 2018 Li et al. 2019). Instead of perturbing each input sentence separately, recently, universal adversarial triggers (UAT) (Wallace et al. 2019) becomes powerful by prepending the same adversarial tokens to all test inputs. UAT prompts the model to generate malicious outputs, which shares the same spirit with the prompt- based tuning approaches. It remains a mystery whether prefix-tuning, a variant of prompt-based tuning techniques, can defend against UAT as well other different kinds of attacking techniques.
2019
In defense of adversarial attacks, different types of defense techniques are developed, including Jones al. [2020), certification (Jia al. 2019
model functional improvement (Li & Sethy 2019
Huang et al. 2019 (hi et al. 2020 Xu et al. 2020 Ye et al. (0020), adversary detection (Pruthi et al.
2019 Zhou
2019), adversarial training MMyatto al. 2017 2019 Zhu al. 2020 Jiang
et al. 2020a Liu et al.) 2020 Wang et al. 2021 Dong et al. 2021] Zhou et al. (2021). While these
approaches have enhanced model robustness, difficulties emerge when fitted to
Most
of the techniques require modification to the architecture and the parameters of the LM additional maintenance of adversary detectors. Directly applying such techniques necessitates auxiliary model update and storage, which will inevitably hamper the modularity of prefix-tuning. Moreover, The excessively long time for adversarial training is also hindrance to the efficient use of teeiixtuunin.. We ask the following question: Can improve the robustness of prefix-tuning while preserving its efficiency and modularity, without modifying the pretrained model parameters?
In this work, propose robust prefix-tuning tuning framework for text classification. The main idea of our framework is to add an extra batch-leve prefix tuned for each batch to the original prefix embed- ding during test time for robustness enhancement. We first record the layerwise activations in the
2

---

Published as a conference paper at ICLR 2022
LM at the position of generating label prediction with correctly classified training data. We project the collected activation matrices of each layer onto low-level canonical manifolds as the charac- terization of "correct" model behavior. In this way, the correctness of any layerwise activations at the position of prediction generation can be estimated by projecting to the canonical manifolds and measuring the distance between them. For each test batch during inference, the added extra prefix is tuned on the fly with the original prefix fixed to minimize the calculated distance. Triggered by the summed prefix, the LM is prone to generating correct label predictions. We conduct extensive experiments on three text classification benchmarks and show that the proposed framework sub- stantially improves model robustness against five strong textual attack approaches including input perturbation attack of different levels as well as the UAT attack. To the best of our knowledge, we are the first to propose the defense approach for prefix-tuning while keeping its lightweightness and modularity. Moreover, we provide an interpretation of our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research.
PREFIX -UNNING FOR TEXT CLASSIFICATION
Prefix-tuning lightweight alternative to finetuning when using large-scale pretrained language models to solve downstream NLP tasks. The intuition of prefix-tuning follows prompt-based meth- ods that a proper context prepended to input sentences triggers the desired response of the LM without changing the large amount of LM parameters. Instead of instantiating the prepended con- text with discrete tokens, prefix-tuning uses trainable prefix embeddings as a replacement, which is also known soft prompts. The continuous prefix embeddings enable continuous optimization and are prepended to all Transformer layers to improve expressiveness. Following the notation of Li & Liang 2222T) , the activation at the i-th position of the j-th layer in an I--ayer autoregressive
Transformer LM is denoted as ()))) h4 = [h: (0) ... : (1-11) represents the stacked activations:
Po LMa(z,
if E Pidx: otherwise.
h4
where Pitx is the sequence of prefix indices and 2; is the i-th token in the input sequence. The activations of the first [Piax| positions are directly calculated by Po. All of the activations at the following positions depend on the prefix as the autoregressive LM follows the left-to-right calcula- tion process. To stabilize the optimization, the prefix embedding matrix Po is reparameterized as Poli. = MLP Po ji, :I) by a feedforward network MLP, & with a smaller matrix Pg.
While prefix-tuning is proposed for conditional generation tasks, in this work, we use prefix-tuning for text classification. As shown in Figure | following the protocol of decaNLP (McCann et al. 2018 we frame the samples in classification tasks into SQuAD-like scheme consisting of context, question, and label: the context and the label part refer to the text sequence to be classified and the ground-truth label, while the question part is a prescribed task description sentence fixed for all samples. We denote x = [context, question, [ANS]]. where [ANS] is special token that separates question and label. We let y = [label] and ly = 1 as the label is one token. At the position that [ANS] is inputted, the LM generates the prediction of the next label token, and denote this position as the output position o. While o can be different for different input x's, in this paper, omit the relation = o(x) for simplicity. Prefix-tuning aims steer the LM maximize the probability of the label. We use all samples in the training set Dr, to optimize the prefix Pa [.. The objective is
min (r.n) ((ylx;0) = max E B log [ (n15))
9
where Wi in the LM transforms the top-layer output (() to a probability vector over the vocabulary.
With continuous optimization on training samples, erate correct label predictions for test data. With the large-scale LM parameters fixed, the obtained task-specific prefix is lightweight and modular. However, prefix-tuning is still vulnerable to text attacks. With the context part perturbed by text attack techniques, the LM can be fooled to generate erroneous label prediction at the output position. Figure [ shows an example of perturbation: by modifying a single character m in the word remember with k, the prediction of the LM is shifted from positive to negative. Therefore, it remains under exploration how robustify prefix-tuning without hampering its modularity or introducing additional large model updates and storage.
is expected to steer the LM to gen-
3

---

Published as a conference paper at ICLR 2022
3 ROBUST PREFIX-TUNING
We propose robust prefix-tuning framework for text classification.. Our intuition follows prefix- tuning that proper prefix embeddings prepended to inputs can steer LM with correct responses. When the inputs are adversarially perturbed, the LM activations at the output position fail to be steered in the correct way by the original prefix Po Inspired by Khoury & WtadfieldMeenel (2018) that the perturbed data often deviates from the low-dimensional data manifol,, our robust prefix-tuning framework uses the layerwise activations by correctly classified training data to con- struct canonical manifolds M. When provided with perturbed inputs during inference, we add an extra prefix P P tuned for each test batch to Poli,: that aims to rectify the erroneous activations at the output position so that they stay close to the canonical manifolds. In this way, we expect the summed prefix to steer the LM with correct label generation against input perturbations, As shown in Figure T. our robust prefix- prefix-tuning framework consists of three steps. The first step is collecting correct LM activations the output position triggered by Poli. We denote Se the set of correctly classified training examples. For the j-th layer, the collected activation matrix H&) stacks the j-th layer LM activation the output position with the input of all Se:
( H
[o.e E Rl
(3)
The d represents the dimension of the LM hidden state. In practice, we always have |Sc| >> d.
The second step is constructing canonical manifolds. We project the collected j-th layer activation matrix H6) onto low-level manifold M as the characterization of the correct j-th layer behavior. We use PCA (Pearson 1901 to get the projection Q onto the canonical manifold of the j-th layer:
H z()=
(4)
Q VV,)
(5)
where n7 - -00 /|col normalizes the rows of H²" to mitigate the randomness among samples in Sc before projection. v, consists of the first singular vectors and QLi E Rddd
The third step is tuning Pi :] to robustify prefix-tuning during inference. Here the vector P [ is not reparameterized by MLP. With the additional prefix P. :]. the token-wise activations become
Poli. + P. if E Piax. (Z, hei).
h2
(6)
otherwise.
For the j-th layer at the output position, the LM activation matrix triggered by Poli.:] P( with the potentially perturbed test input batch Sr is stacked as
[n0]]] RRISiixd
for all t ST. We use the distance from 1 to the j-th canonical manifold M the loss for the tuning of :] for each batch. Projecting H.( to MG) yields H, QU), thus the objective is
N-1
--1
min
L (1) Qi)) Σ
[ (1 ((-Q
1=0
1=0
We also replace the H1'1 in Eq. 8 with H - 11TH H14') ( /|ST as normalization before projection
to mitigate randomness among test samples when |SSr| > 1. After tuning P' the activated H!(")
is closer to M(1) As the manifold characterizes the correct behavior of the j-th layer activation, by regulating the layerwise activations at the output position, the summed prefix [o P,(ú, is
prone to steering the LM to generate correct label predictions. Our framework is also applicable to
other soft cooppt-based tuning methods (Qin & Eisner 2021 Hammbadzumman et al.] [022]] Lester
al.] 202T: Cho et al. [22T]] Tsimpoukelli et al. (2221) by recording the activations of correctly classified training data, constructing canonical manifolds for the soft prompts, and tuning additional soft prompts for robustness during inference. In this work, we conduct experiments on -teiin.
Remark. From the optimal control (OC) perspective, prefix-tuning can be formalized as seeking the OC of the pretrained LM for downstream tasks, and our robust prefix-tunine can be interpreted as seeking the close-lopp control for robust downstream tasks. We attach the details in Appendix G
4

---

Published as a conference paper at ICLR 2022
Table 1: Results of different baselines with framework applied. Our framework substantially improves robustness of both standard and adversarial prefix-tuning against all types of attacks.
Benchmark
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
+ our framework
92.59
50.36
44.65
58.54
46.68
85.72
SST-2
adv. prefix-tuning 93.57
30.64
7.25
35,42
25,04
4.88
+ our framework
93.79
57.55
43.60
60.68
57.17
91.87
std. prefix-tuning 86.42
43.00
24.55
43.20
43.22
52.20
+ our framework
86.50
53.91
24.93
48.38
51.79
76.47
AG's News
adv. prefix-tuning 89.46
50.91
26.30
45.25
47.11
59.74
+ our framework 90.26
56.45
31.25
48.03
54.66
87.26
std. trefix-tuning 72.48
25.51
29.69
42,46
32,88
30.20
+ our framework
72.74
34.68
33.64
43.37
36.59
71.03
SNLI
adv. prefix-tuning
77.22
27.43
28.58
46.83
34.94
35.76
+ our framework
77.65
33.88
33.98
46.92
38.21
76.15
EXPERIMENTS
4.1 SETUP
While we have also provided the regular finetuning as baselines We consider three text classification benchmarks in our experiments: binary Stanford Sentiment Treehank (SST-2) (Socher et al.) 2013).
AG's News (Zhang al., 2015), and Stanford Natural Language Inference (SNLI) (Bowman al.
2013). We evaluate our robust prefix-tuning with five text attacks: PWWS Renn et al. 2019), VIPER (Eger et al. 2019), SCPN (yyer et al. 2018), TextBugger ("BUG" for short) (Li al. 2019), and
UAT (Wallace et al. 2019). We use the GPT22medium (with = 24 layers total) as the large- scale LM and set prefix length 10 for all prefix-tuning experiments. We train 100 epochs for SST-2 and 25 epochs for AG's News and SNLL We set N = 3 and record the bottom N-layer activations of the LM at the output position for the additional tuning. Other experiment configurations and details can be found in Appendix B We have also conducted both standard and adversarial full tuning experiments (Appendix A) to discuss the challenges and opportunities in robustifying prefix-tuning
4.2 RESULTS ON ADVERSARIAL. TRAINING
We first apply adversarial training to prefix-tuning as our baselines named adversarial Following Miyato et al. 2017). inject perturbation restricted within the l ball of the flattened word embedding of original sentences during training to adversarially optimize the prefix Po [.. We have also attempted use other types of adversarial training, (i.e., adding the KL-divergence term for regularization (Miyato et al. 2019 Zhang et al. (019)) but obtained poor accuracy, suggest- ing the difficulty of optimizing the small amount of prefix embedding parameters. The experimental details can be found in Appendix C.I] Our framework is applicable to the adversarially- trained prefix embedding, as keeps the acquired prefix Po fixed and tunes extra Po :] for each test batch. The experimental results are listed in Table [ According Table [ our approach signifi- cantly improves the robustness of prefix-tuning over all baselines against each type of text attack. To the best of our knowledge, our framework the first defend against UAT on large-scale pre- trained LMs (see Appendix A for details). Compared with the standard prefix-tuning baseline, the adversarial baselines achieve better robustness and clean accuracy. fact, fewer training epochs are needed achieve certain clean accuracy when applying adversarial training to prefix-tuning. Figure (a) illustrates the loss on the training set and the clean accuracy on the validation set for each epoch during standard and adversarial prefix- prefix-tuning. From Figure (a), it is clear that adversarial prefix-tuning outperforms standard prefix-tuning in both the convergence rate and generalization.
However, the adversarial prefix-tuning approaches suffer from the catastrophic drawback of taking far greater training time than the standard prefix prefix-tuning baseline. Figure (b) changes the horizon-
tal axis in Figure (a) from epoch to clock time. According to Figure 2 (b), given the same time budget, standard grefix-tuning finishes training for 100 epochs while adversarial prefix-tum tuning only manages to train for 20 epochs. The training losses of the two methods at the time also roughly
5

---

Published as a conference paper at ICLR 2022
2.0
Train Loss
adversorial, dev acc
adversanal, acc
standard,
standard, dev acc
standard, train loss
DBv
Train
standard, train loss
Dov
adversarial, train loss
adversarial. train loss
9.5
0.0
0.0
20
50
80
100
20000
40000
80000
90000
100000
Epoch
Time (s)
(b)
Figure 2: Comparison between standard and adversarial prefix-tuning for 100 epochs with respect to (a) epoch and (b) clock time. While adversarial prefix-tuning gains strengths in epoch-wise wise con- vergence rate and generalization, takes far greater training time than standard prefix-tuning.
the same (0.06). Earlystopping adversarial prefix-tuning at 20 epochs achieves slightly better ro- bustness results than 100 epochs, but the clean accuracy becomes lower than standard prefix-tuning (results can be found in Appendix C.2) To conclude, the adversarial prefix-tuning approaches make regularization effect on
but lack competitiveness practice.
In contrast with adversarial prefix-tuning, dnference-phase on-the-fly optimization the additional prefix P negligible compared with training-phant phase adversarial prefix-tuning. From Table [ the robustness of the std. prefix-tuning applied with our approach has surpassed that of the adv. prefix-tuning without our approach. The adv. pefix-tuning with our framework achieves the strongest robustness against most attacks.
approach is more practical as the duration of the
4.3 RESULTS ON ADVERSARIAL DATA AUGMENTATION
Table 2: Results of PWWS adversarial data augmentation baselines well as our methods on the SST-2 benchmark. Our methods consistently improve robustness over all baselines.
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
+ our framework
92.59
50.36
44.65
58.54
46.68
85.72
20-epoch adv. aug.
87.15
48.11
42.12
48.11
47.01
10.43
+ our framework
86.93
55.57
51.62
57.33
52.00
70.07
50-epoch adv. aug. + our framework
86.82
55.08
35.04
44.65
49.37
3.13
86.66
58.26
48.05
58.98
51.73
83.47
100-epoch adv. aug. + our framework
91.21
56.62
23.01
38.55
45.14
7.91
90.39
62.55
46.95
59.25
54.75
86.38
In this section, conduct adversarial data augmentation during the training phase of standard prefix-tuning as our baseline methods. During training, we use a specified text attack method to perturb each batch and augment the perturbed data to the training set. Due the computational inefficiency of discrete optimization for text attacks, the training of adversarial data augmentation is even far slower than that of adversarial prefix-tumin -tming introduced in Section 4.2 We set milestones for the training of adversarial data augmentation at 20, 50, and 100 epochs for comparison. When applying our framework, we calculate the canonical manifolds M using the correctly classified samples from both the clean training set and the perturbed training set by the specified text attack.
Considering the training time budget, we select SST-2 as the benchmark. We use PWWS as the attack method to generate perturbed inputs with results listed in Table On one hand, the clean accuracy of the PWWS adversarial data augmentation method is affected at the 20-epoch and the 50- epoch milestones. This might be attributed to the consideration of perturbed inputs during training. The robustness results, on the other hand, are improved compared with standard robustness against PWWS obtains steady improvement thanks to the adversarial data augmentation
The

---

Published as a conference paper at ICLR 2022
generated by the attack method of the same type. However, for other types of attack, the robustness improvement is decreasing with the training process, suggesting the possibility that overfitting to the specified type of adversarial data can be harmful to robustness against other types of attacks.
With our robust prefix ttnning framework applied, all milestones of the adversarial data au augmenta- tion baseline obtain substantial robustness improvement. While calculate the layerwise canonical manifolds with the correctly classified samples from both the clean training set and the PWWS- perturbed training set, it can be seen that the robustness against PWWS attack is still enhanced by several percent, while the clean accuracy is slightly affected. This shows that when calculating the layerwise manifolds, the used samples from the two types of training data, though are all correctly classified, trigger the LM with activations of different properties at the output position. It is left for future work about how construct the dataset calculate the canonical manifolds achieve the best performance on both accuracy and robustness. We have also used VIPER and SCPN as the at- tack methods for adversarial data generation and draw similar conclusions the PWWS adversarial data generation experiments, results of which can be found in Appendix D
4.4 EFFECT OF TUNING DIFFERENT LAYERS
In this section, study the effect of tuning robust prefixes with different layers. While we tuned the robust prefixes of bottom N layers for all previously reported experiments, different results are obtained when tuning the robust prefixes of alternative layers. We experiment SST-2 development set by tuning the bottom N -lleyeer robust prefixes as well as the top ones with N enumerating from to 24 with the step size 3. N prefix-tning method, and = 24 means tuning all layers. The results are shown in Figure 3 According to the results, tuning the bottom N layers achieves better robustness improvement than
the
represents the original standard
Dev Acc
40-
40
Clean
Gsan
PAWS
20
20
MPE
12
15
38
21
5
12
15
18
21
24
N
(a) Tuning bottom N layers
(b) Tuning top N layers
Figure 3: Results on the SST-2 development set when applying robust prefixes of different layers. Tuning bottom N layers outperforms tuning top N layers terms of robustness improvement.
tuning the top position activations triggered with perturbed inputs. For the in-sentence text attacks, tuning the bottom N < 15 layers achieves comparable robustness improvement and slightly outperforms the larger N's. For the UAT attack, setting the bottom N 3 is significantly better than the choices of larger N's. One possible reason for the exceptional performance is that the collected low-layer activations capture richer positional information (Voita et al. 2019), which helps to defend against UAT as UAT shifts right inputs with the same trigger. We proceed to study how our method steers the LM with different N's for better understanding of its robustness improvement in Section 5
This be attributed to layerwise bottom-up error accumulation of the output
4.5 PERFORMANCE UNDER MORE REALISTIC THREAT MODELS
In the previous experiments, the robustness evaluation metric for each method is the accuracy under specific attack on the test set. While this has been adopted in previous work on adversarial defense in NLP (Dong et al. 2021 Si et al. 2 Li al. 2021), }, in this section, consider more realistic threat models. We first study the effect of test batch size on the performance of our framework. We used adaptive test batch size to make full use of GPU memory in previous experiments (detailed in Appendix B). However, the loss in Eq. (8) depends on the size of the inference batch, and models in deployment should respond with real-time data. Based standard mefix-tuning SST-2,

---

Published as a conference paper at ICLR 2022
Table 3: Performance of our method with std. prefix-tuning with various test batch sizes SST-2.
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
ours, bsz adaptive
92.59
50.36
44.65
58.54
46.68
85.72
ours, bsz
92.64
51.46
43.99
58.32
46.73
73.09
ours, bsz 2
92.48
49.92
45.74
59.14
48.93
84.73
ours, bsz
92.64
50.36
45.85
58.54
49.31
86.66
additionally evaluate our framework with small test batch sizes: 1, 2, and According to the results in Table 3 fixed small test batch size achieves comparable slightly better results under clean data and in-sentence attacks. Under UAT, however, there exists a performance gap of the robustness between our framework with test batch size of and others. We discuss the phenomenon in depth and attach our attempt for improvement in Appendix E.I). Nevertheless, our framework with test batch size of still substantially outperforms the baseline, and we also find the performance gap is smaller on other benchmarks with other prefix-tuning baselines (see Appendix E.T).
challenging yet more realistic setting where the test data is mixed with unperturbed and perturbed samples or perturbed samples under different attacks. We use test batch size of in our framework based on adversarial prefix-tuning for SST-2. While it unknown under which attack each test sample is perturbed (or whether it is perturbed), we find the results shown in Table still impressive when using fixed learning rate tuned on the development set under UAT for the additional tuning of P We attach the detailed analysis in Appendix E.2
We continue to consider
Table 4: Performance of our framework under mixed test data SST-2. The '+' denotes combina- tion of test set clean or under attack, and 'C' short for "Clean", 'B' for "BUG", etc.
Method
+ B
C+D P
V+
V+P P
S S
S +
U B
+ P
adv. prefix-tuning
60.65
62.27
17.76
20.43
29.57
31.85 56.92
18.12
20.81
+ ours, bsz
69.71
71.11
46.02
47.23
55.63
70.04
71.67
How DOES ROBUST -UNNING STEER THE LM?
As the robust prefix-tuning framework with the proper choice of N substantially improves the ro- bustness of prefix-tuning. it deserves to be explored how the robust prefix-tuning steers the LM compared with the original prefix. In this section, we leverage different proxies to study the be- havior of framework with the standard prefix-tuning baseline. We find that robust prefix- tuning framework steers different LM behavior between the in-sentence attacks and UAT. For the in-sentence attacks, the behavior of the final layer of the LM can be summarized as averaging the attention. For UAT, with the proper setting of N, the LM functions as ignoring the distraction.
Table 5: Inputs from the SST-2 dev set and their perturbed versions by BUG/UAT attack under which the prediction is flipped. The two examples are used case studies in Sections and 5.2
Type
Input (the context part)
Predict
Label
Original
from the heart.
positive negative
positive
BUG-attack
One from te hart.
positive
Original
it 's just filler. lifts mates who it just filler.
negative
negative
UAT-attack
positive
negative
5.1 IN-SENTENCE ATTACKS: AVERAGING THE ATTENTION
In this subsection, we leverage attention weights in the final layer the LM as the proxy and use the adversarial inputs generated by the BUG attack SST-2 development set for our following in-sentence case studies. We ignore the prefix embeddings and renormalize the attention weights of the rest of the tokens in visualization. Note that the observed behavior is of the final layer only: the attention is shown to be averaged only over the vectors inputted to the final layer, as attention is unreliable for indicating importance over input tokens (Serrano & Smith 2019 Jain & Wallace
8

---

Published as a conference paper at ICLR 2022
one from the
heart
One
from
te
h art
One
from
te
h art
One
from te art
one
One
1.0
One
1.0
One 1.0
from
from
from
0.5
from
0.3 0.7
te
te
the
h-0.1
0.2
0.3
0.4
h
0.3 0.3 0.2
0.3
0-00
02
0.3
0.3
heart
0.0 0.1 0.1
0.8
art
0.1
0.2
0.2
0.2
0.2
art 0.2
0.2
0.2
0.2
art
0.2
0.2
0.2
0.2
0.4
0.
0.1
0.2
0.2
0.2
0.2
0.2 0.2 0.2
0.2 02
0.1 02
0.2 0.2
0.2
0.2
Predict:pssitive
(b) Predictrnegative
(c) Predict:positive
(d) Predict:positive
Figure 4: Visualized attention weight maps of the final layer in the LM for the case study with BUG attack. Each row represents the time step at which the token (labeled on the left) is inputted. Each column in the row illustrates the attention weight assigned the specific token (labeled on the top).
(a): the original input by the original prefix-tuning; (b): the BUG-perturbed input by the original prefix-tuning. Compared with (a), the attention weights are perplexed in (b) due to the perturbed input. (c) and (d): the BUG-pen -UG-perturbed input by our robust prefix-tuning with (c) N = 24 and (d) N Our robust peeii-tuning (with both N 24 and N 3) steers the behavior of the final layer to average the attention over the input vectors of the final layer (not the input tokens).
2019 Wiegreffe & Pinter 2019 Abnar & Zuidema, 2020). Table shows an example of BUG- perturbed input, and Figure 4 shows the attention map visualization. Similar behavior is observed as well by other in-sentence attacks; more visualization can be found in Appendix F.I
5.2 UNIVERSAL ADVERSARIAL TRIGGERS: IGNORING THE DISTRACTION
In this subsection, we leverage an alternative proxy based on the product of gradient and attention token importance rankings (detailed in Appendix F.2.1. While the proxy is not gold-standard indicator of importance value either, it can provide more reliable predictions of token importance orderings (Serrano & Smith 2019). We use UAT attack SST-2 development set, and Tsble an example of input with triggers "lifts mates who" with visualization in Figure As a result, the
Just
filler
Mate ét
jusller
Ripte Who it
JUSE
Filler
Ragte Who it
fustteer
3.0
1.0.6
0.000
0.0
02
5 0.2.0.2..0
just
just
02 02
0.0
just
0.2.0.2.0.
Just
filler
filler
0.1
filler 0.3.0.2.0.3 0.100.10:0
filler
02
0.1
0.100:1
0.0 0.000.000000000000000
(a) Predict:nogative
(b) Predict:positive
(c) Predict:positive
(d) Predictnegatiive
Figure 5: Importance visualization for the UAT case study. (a): the original input by the original prefix-tuning; (b); the UAT-attacked input by the original prefix-tuning. Compared with (a), the trigger tokens in (b) attract major importance. (c) and (d): the UAT-attacked input by our robust prefix-tuning with (c) N 24 and (d) N 3. For N 3, the LM is steered to ignore the distraction of the trigger tokens and assign high importance to "filler" at the time step token "."
LM is steered to ignore the distraction from the trigger and assign the highest importance to the most essential token as the baseline does with clean inputs. The findings are also supported by quantitative studies. By designing metrics of corpus-level Degree of Distraction (cDoD) and Recognition of the Essential token (cRoE), we find that the scores of our framework with N are better than those of other methods with statistical significance (p < 0.001), which also explains the results in Figure We attach more visualization and details the quantitative studies in Appendices F.2.2 and F2.3
CONCLUSION AND FUTURE WORK
In this work, we propose robust prefix-tuning for text classification that maintains only one extra amount of embedding parameters of prefix-tuning while achieving substantial robustness improve- ment. We conduct both quantitative and qualitative studies analyze the behavior of our framework. We also interpret
and our framework from the optimal control perspective. Future work includes constructing better canonical manifolds and extending our framework to text generation.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Large-scaled pretrained language models have been criticized in terms of bias and stereotypes (Zhao et al. 2019 Bender et al.) 2021; Field et al. (021). and can be steered to generate malicious outputs with universal adversarial triggers (Wallace al. 2019). The lack of robustness leads more uncertain behavior of LM, especially under text attacks. In our work, we contribute to improving the robustness of LM with the robust prefix-tuning framework. Figure [[] illustratees an example of LM fooled to predict gender-biased sentence as positive. With our framework, the prediction of the LM is rectified. The huge resource consumption of training large-scaled LMS has also led to sustainability concerns (Strubell et al. (219). As specified in Appendix [B] and Table [1] our framework has preserved the efficiency and modularity of prefix-tuning by requiring significantly less time than all baseline methods and keeping the pretrained models unmodified.
REPRODUCIBILITY STATEMENT
For all of our experiments, have listed the detailed settings in Appendices B and For the theory part, we have provided our detailed formulation as well as a brief introduction to the optimal control theory in Appendix G We have also provided the statistics of the datasets used, as well as the time and storage taken by all methods in Tables 10 and and Appendix B respectively.
ACKNOW
This work was supported by the National Key R&D Program of China (No. 2018YFB1005I 103), National Natural Science Foundation of China (No.61925601), and Huawei Noah's Ark Lab. We thank all anonymous reviewers for their valuable comments and suggestions on this work.
REFERENCES
Samira Abnar and Willem H. Zuidema. Quantifying Attention Flow in Transformers. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp- 4190-4197, Association for Computational Linguistics, 2020. doi: 10.18653/v 1/186663//2220..ae 1/2l20..cl------ main. 385. URL https: /ttps://doi..o
w1w.1/66111/00 acl ma main. 385
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei.
Chang. Generating Natural Language Adversarial Examples. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pp. 2890-2896, Brussels, Bel- gium, hallhttts///ccaattttt http::///ccllaccooooooooooor///.
2018. Association for Computational Linguistics. doi: 10.18653/v1/
D18-1316.
orq/
Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K. Reddy, and Bimal Viswanath. T-Miner: A Generative Approach to De- fend Against Trojan Attacks on DNN-based Text Classification. In Michael Bailey and Rachel Greenstadt (eds.), 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pp. 2255-2272. USENIX Association, 2021. URL https::////.. www. www.uu.ee...... org/ con
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. On the
Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 0--62,, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 450383097. doi: 10. 014534442188.3445000 URL /tRp::ttt//::: wwt.:ooc..org//... 1145/ 3442184. 3445922
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard,
10

---

Published as a conference paper at ICLR 2022
Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the Opportunities and Risks of Foundation Models. CoRR, abs/2108.07258,, 2021. URL httpss: /tss:///rxvv../ org/ abs 22//2/188/ 07758
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A Large An- notated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pp. 642-642,, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.15665/ URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhimi Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL bttps::/prooceedingg neur ips wttpc//ppppppppppeeepepp I.47c0d6bfcbb4B cb4.e C64967 www14996678888888888 8ofb8ac1 -at.aaaac14fffaaaaaaaaaaee pdf
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Towards Robust Neural Networks via Close-loop Control. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 1d=2AL06y9cDD
net/ /forrm?rm
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying Vision-and-Language Tasks via Text Generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1931-1942. PMLR, 18 24 Jul 2021. URL https: https:://proceeding. httppnn..............eeeeeeemeee pttss/v139/
cho2 html
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 4171-4186, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1421.33.9114....... URL Tttps://aclanthiee
wtwg///11/..0 1423
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards Robustness Against Natural Language Word Substitutions. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021. URL https://
openreview. net/ aerenreview.nne//ooummiiiii
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-Box Adversarial Examples for Text Classification. In Proceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 2: Short Papers), pp. 31 31-36, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 2006. URL
Steffen Eger, Gözde Gül Sahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. (634-1647, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1//11--1655 URL https:
//aclanthol wow..hoooo....ooooooooooo.oo... wow.lolyy..oogoggoooooooo 9-11119111155
Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. A Survey of Race, Racism, and Puti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Compu- fational I Linguistics and the I 1th International Joint Conference Natural Language Processing
11

---

Published as a conference paper at ICLR 2022
(Volume 1: Long Papers), pp. 1905-1925, Online, August 2021. Association for Computational Linguistics. doi: 10.18653//12222..allll 149. URL https://aclanthol..g .ogg
2021. ac2loonn 149.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/202 acl--ooogg.
URL https:
org.2021.
acl Long. 295
Siddhant Garg and Goutham Ramakrishnan. BAE: BERT-based Adversarial Examples for Text Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pp. 6174-6181, Online, November 2020. Association for Computational Linguistics. doi: 10. 18.18653/v 11018663/1/2000.mmlplpp- main. URL https: htttps://aclaatthoooogooooo org.
2020 www.pmmmppppm.mmm main. 498
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joini Conference on Natural Language Processing (Volume 1: Long Pupers), pp. Linguistics. doi: 10. 26653//1/222...cllloogg 2021.ac1-lom 381
Online, August 2021. Association for Computational org/
URL
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fernández del Río. Mark Wiebe, Pearu Peterson, Pierre Gérard- Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis Oliphant. Ar- ray programming with NumPy. Nature, 585(7825)3357-362 September 2020. doi: 10. 1038/ 020 2649-2
s4 841586-020-22 2649-2. URL htttps::/dci..or//111113333
Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model Extraction and Adversarial Trans- ferability, Your BERT is Vulnerable! In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2006-2012, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653v1//v maic--mnn-mm. 161. URL wwwpttttpss.. //acl httt:p:://ellanthoooooooooo org/2021 .nacc mmain.. m....... 161
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Effiiien Transfer Learning for NLP. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 2790-2799, PMLR, 09-15 Jun 2019. URL bttps::/ppoceeednngg/ Fress/v97/houlle
html
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. Knowledge- able Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification. CoRR, Abs025 2021. hotttt:ts///arra
wtgg/bs//1iss 02035
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Kr- ishnamurthy Dvijotham, and Pushmeet Kohli. Achieving Verified Robustness to Symbol Sub- stitutions via Interval Bound Propagation. In Proceedings of the 2019 Conference Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 4083-4093, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v//D19 1419. URL
https: h/t.aaaaaatahcccca/a wwannhhlogy..om ogy. wwt.oog...o/.11//////g 1419
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial Example Gener- ation with Syntactically Controlled Paraphrase Networks. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- Language Technologies, Volume (Long Papers), pp. 1875-1885, New Orleans, Louisiana,
12

---

Published as a conference paper at ICLR 2022
June 2018. Association for Computational Linguistics. doi: http::://aclamhoooooo.oor//////
1170. URL
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapoliss MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 3543-3556. Association for Computational Linguistics. 2019. doi: 10. 10.18653/11/119-1333 URL https:
//doi. org/10.0010000 0121110.886655 (11888653vv/// 9-135
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified Robustness to Adversarial Word Substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP?) pp. 4129-4142, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 1111663//1//11-144 URL
org/D19-144
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2177-2190, Online, July 2020a. Association for Computational Linguistics. doi: 10. 220/:33///2222.clmmmmma 2020 8020.ac1- main. 197.
URL https::/aclanttholoogg org/
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Lan- guage Models Know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020b. doi: 10. at02//talla_00322 URL https: //acl///cccco http:///acaaaattotaecee www.aaoooooooooo/aa0..
28
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. Robust Encodings: A Framework for Combating Adversarial Typos. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2762, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/020..com acl-00000.ccllaaaa 2020. acl in. 245.
main.245. mai.....222.. URL ht https://aclanthoollooo
org/
Mare Khoury and Dylan HedfieldMMenll.. On the Geometry of Adversarial Examples. CoRR, abs/1811.00522. 2018. URL (Rttpp http://awwxwvv..//b..sas/e ottg/bs//g/al..o.oe//pbb///// 1011b//55500
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
7/ 7/arxiv.co www.aii.org/aaaaaaaaaaaaaa abs 1411.11444 6980
Philipp Koehn. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of
the 2004 Conference on Empirical Methods in Natural Language Processing EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25- 26 July 2004, Barcelona, Spain, pp. 388-395. ACL, 2004. URL org/ 32500/
hollanthoooo
Richard E Kopp. Pontryagin maximum principle. In Mathematics in Science and Engineering, volume 5, pp. 255-279. Elsevier, 1962. doi:
Thai Le, Noseong Park, and Dongwon Lee. A Sweet Rabbit Hole by DARCY: Using Honeypots to Detect Universal Trigger's Adversarial Attacks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 3831-3844. Association for Computational Linguistics, 2021. doi:
11233//1/222.acllleee acll aacll long 296
URI.
https://doi. wwl.0.oorg/100 1tw.11.11113111111
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Cerrameter-fffcciin Prompt Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.),
Tuning. In
13

---

Published as a conference paper at ICLR 2022
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3045- 3059. Association for Computational Linguistics, 2021. URL httpa://aclantholo..
org/2021. emn Sonnlp-man pww ma 243
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising pepueenc-to-Sequeeee Pre- training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line. July 2020. Association for Computational Linguisties. doi: org/ 2020 acm.mminn....m..mmmmm.mm...mm..m
URL https: http://aaaaathhhh
Alexander Hanbo Li and Abhinav Sethy. Knowledge Enhanced Attention for Robust Natural Lan- guage Inference. CoRR, abs/1909.00102, 2019. URL http:///axx.v.org/abb///
00102
Jinfeng Li. Shouling Ji. Tianyu Du, Bo Li, and Ting Wang. TextBugger: Generating Adversar- ial Text Against Real-world Applications. In 26th Anmual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California. USA, February 24-27, 2019. The /wwp://wwwwww// ndss http:://www...noss.ssssspsss.sop wtt.:ppps.uuu..oo/rrdssepeeepepp
Internet Society, 2019. URL
Qianxiao Li, Long Chen, Cheng Tai, and Weinan E. Maximum Principle Based Algorithms for Deep Learning. J. Mach. Learn. Res., 18:165:1-165:29, 2017. WRR.http:///mmr..o
v18/17-66 html]
Xiang Lisa Li and Percy Liang. Prefix-Tuning Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582-4597, Online, August 2021, Association for Computational Linguistics. doi: 10.18653/ v1/2021 act-loog...- URL
-on-g....333
Zongyi Li, Jianhan Xu, Jichang Zeng, Linyang Li, Xiaoqing Zheng, Qi Zhang, Kai- Chang, and Cho-Jui Hsich. Searching for an Effective Defender: Benchmarking Defense against Ad- versarial Word Substitution. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3137-3147. Association for Computational Linguistics, 2021. URL
https: //tps///acl antho ogy. org/ 2021 .emmlppppp p-ma
ma eemnlpma 251
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. CORR, abs/2107. 13586, 2021. URL htt.:/ttt////axtttt http///aww.ooooaa/////////////at
22lpg//b///10..... 13586
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial Training for Large Neural Language Models. CoRR, abs/2004.08994, 2020. 2004. 08994
URL htttps:/aaxxvv.rggg
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. CoRR, abs/1907.1 11692 1692, 2019, URL httt:://aaxi..or//eeeee wwtp:////ggg/ba//g///////// 11692
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL/attss://ppeeeeeea
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: /sttps:/ccperrevicee
14

---

Published as a conference paper at ICLR 2022
Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific Credibility of Machine Transla- tion Research: A
of 769 Papers. In Chengqing Zong, Fei Xia. Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- fional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNIP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pp. 7297-7306. Association for Computational Linguistics, 2021. doi: 10.18653/v hwwp///2...llloonnn.... aaee-accl-20
URL
https://aoi.org/ 18..0.18663668
566
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The Natural Language Decathlon: Multitask Learning as Question Answering. CoRR, abs/1 0os.0800608730, 2018. URL.
http://arxiv. owg/o//gg///.
abs 2abs/1806. 1806 08730
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial Training Methods for Semi- Supervised Text Classification. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL opwnreview. net/ fett:/orrrm....r11122222222
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi- -emi-Supervised Learning. IEEE Trans. Pattern Anal. Mach. Intell., 41 (8):1999 1993, 2019. doi: 10. 1100///AMI.20111 2858881. 7/doi 10. 11 1109/ 2018. 2858821
URL https:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-PPeformance Deep Learning Library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. 8024 8035, 2019. URL http::sss/pppocceagg....ott neurips.. http:ppsc...pppe///////eeea SEA.:///EEE.EEEEEEEEETTTTO222222
Neural Information Pro-
Abst html
Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. doi: 00080//478644000945555
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18. 1202.
URL sttpss//aclantholoee hwtp:/hhoooooy...o/eeeeoeee owg/N18-//--....... 1202
Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. Combating Adversarial Misspellings with Robust Word Recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5582-55 5591, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.18653/11//11-15555 1561. URL https://aclantholl
org/
P19-1561
Raul Puri and Bryan Catanzaro. Zero-shot Text Classification With Generative Language Models. 101
CoRR, abs/1912. 10165, 2019. URI.
Guanghui Qin and Jason Eisner. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011
URL ptstps://aclannhoooote wrg. 2021 maacclmaile 410
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
Models are Unsupervised Multitask Learners. 2019. URL attps: : ht//pppnal. com/blog/
better-1anguagee motppaaaaemmdeels.//
15

---

Published as a conference paper at ICLR 2022
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text- to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URLhttp:
//jml r wwwpg//appr////a/aa wwwpppppppaaa-aaaaaappp - html
Shuhuai Ren, Yihe Deng. Kun He, and Wanxiang Che. Generating Natural Language Adver- sarial Examples through Probability Weighted Word Saliency. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pp. 1085-1097, Florence, Italy, URL
July 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/P19-111
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically Equivalent Adversarial
Rules for Debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume I: Long Papers). pp. 856 865, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1//65//v1/118/ 1079.
URL https:
Maclanthology wwwpolloooyyyoo/11o0///11
Sofia Serrano and Noah A. Smith. Is Attention Interpretable? In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.). Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp.
Association for Computational Linguistics, 2019. doi: 10. 10.18653/v1/p1..115 1282. URL http:////oaa.ooog111111666533///
Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. Robustness Verifi- cation for Transformers. In International Conference on Learning Representations, 2020.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pp. 0222-4235, Online, November 2020. Association for Computational Linguis- tics. doi: 1/06653//10222.mmmmlll main.346. URL htt.tpsss//cllamthoooo...o/ ozgg:///...ooy...2220..2.
lemn 1oc11p-mas ma in. 346
Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. Better Robustness by More Coverage: Adversarial and Mixup Data Augmentation for Ro- bust Finetuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Find- ings of the Association for Computational Linguisties: ACI/IJCNIP 2021, Online Event, Au- gust 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pp. 1569-1576. Association for Computational Linguistics, 2021. doi: 10.18653/v1202111 10.18663/1/2222.findnng.. findings-acl. URL https: www..11663vvv1
findings-aciiinn.c 137
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. (631-1642) Seattle, Washington, USA, October 2013. Association for Computa- org/ 1170
tional Linguistics. URL
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and Policy Considerations for Deep Learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650, Florence, Italy, July 2019. Association for Compu- tational Linguisties. doi: 10. 10.18863/111111113555 URL
orrg
P19-133 1355
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal Few-Shot Learning with Frozen Language Models. CoRR, 138844 2021. URL httpthtss///axxiiie
abs/ /21s2/166.. 13884
Elena Voita, Rico Sennrich, and Ivan Titov, The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 4396 4406, Hong Kong, China, November 2019. Association for Computational LirioonnnnLiiiiiiie Linguistics. doi: 10.18653/v1/D19. 1448. URL Utt.httts:///at h antho ogy. wwwp.orrgDD1--145
16

---

Published as a conference paper at ICLR 2022
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal Adversar- ial Triggers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNN.P-JCNLLP), pp. 2153-2162, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/11/D19.1221 URL.
atttps://acclantolooyy
1221
Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. InfoBERT:
Improving Robustness of Language Models from An Information Theoretic Perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, net/ Forum?id
May 3-7, 2021. OpenReview.net, 2021. URL https:://ppnnee
MBH998mK5Puk
Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP-IJCNLIP 2019, Hong Kong, China, November 3-7, 2019, pp- 11-20. Association for Computational Linguistics. 2019. doi: 10.18633/1/119-1002.. URL https:/ /ttps://doi..r www.ooo.or.......ooo 1ttp://...1155/111//
1002
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Ar Nat- ural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 45, Online, October 2020. As- sociation for Computational Linguistics. URL hothhttp:: /tww///wwwwwww. acwweb. ort.//hoooogggggy//// 2020. emn demoss..dees...
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlic Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic Perturbation Analysis for Scalable Certi- fied Robustness and Beyond. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1129-11411 Cur- ran Associates, Inc., 2020, URL http::::/pprocceeee ngs
Ying Xu, Xu Zhong, Antonio Jimeno Yepes, and Jey Han Lau. Grey-box Adversarial Attack And Defence For Sentiment Classification. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4078-40877 Online, June 2021. Association for Computational Linguistics. doi: 10. 1865vv1// 2021 1e2l1.aacl-mae m21...1. URL http:httt/:/aaaa anthology /ttrg///211.2 naacl manll-mainn... 321
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. Linguuttttioalllnnuuiissssss doi: 10.18653/v1/2021 www.:///a0222m2lnnmmm www.aaaaaaaaaal... URL https:
7/aclanthologys wwwplllyy...g/22500 http://22w..aaaaaaainn.nn.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
http://pwwceeddnno.e.eieieeeesss wwwpeeriissseeppppppeeeeeeeeeeee wttpa//e///p/////ieeeeee pdf
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander Smola, and Eduard H. Hovy. Hi- erarchical Attention Networks for Document Classification. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, San Diego Cal- iforriaa USA, June 12-17, 2016, pp. 1480-14488 1489. The Association for Computational Linguistics, 2016. doi: 10. 10..1865.00 1a.18853/v1ll 1/n 174. URL https:ttp///.t........ / .aciii.rgg/ii 100 /
17

---

Published as a conference paper at ICLR 2022
Mao Ye, Chengyue Gong. and Qiang Liu. SAFER: A Structure-fiee freee Approach for Certified Robustness to Adversarial Word Substitutions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3465-3475, Online, July 2020. Asso- URL https:
ciation for Computational Linguistics. doi: 10.18653/v /220://ww120222.mmammma..3 //aclanthollogy.ooo wtw.ogy.org/22000 a0l.:22000..ccmmmmmm acl-ma n. 317
Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Zixian Ma. Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. OpenAttack: An Open-souree Textual Adversarial Attack Toolkit. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 363-371, Online, August 2021. Association for Computational Linguis- acl--demo..3...
tics. doi: 10.18653/v1/2011 wtwp:////222..aamaooa.o...
URL
sol-demo 43
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learming Research, pp. 7472-7482, PMLR, 2019. httttttt///pceeeeeee w..x.peess/vv1 / zhangllp.... html
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial Attacks on Deep-Learning Models in Natural Language Processing: A Survey, ACM Truns. Intell. Syst. Technol. 11)3) April 2020. ISSN 2157-6904, doi: 10.1145/3374217. URL https: //doi.
0eg/10.1145/33777
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang- Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 629-634, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18158///1/N1911000 URL https:
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual Probing Is [MASK]: Learning vs. Learning to Recall. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 5017-5033, Online, June 2021. Association for Computational Linguistics. doi: 00118653/112222
URL
htwphhhoooogyorg/222222 org/ mww.c-mainnnna..mmmaammmm 398
Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, and Xuanjing Huang. Defense against Synonym Substitution-basse Adversarial Attacks via Dirichlet Neighborhood Ensemble. Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5492, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 www.llllogg...12 URL CtLLhhtps://// http:://caaaatthcoocooo wwtp:oyy.oo.o020o..o.o otg.a/aa....oo...oo/////...aaao.
Yichao Zhou, Jyun-Yu Yu Jiang, Kai-Wei Chang, and Wei Wang. Learning Discriminate Perturba- tions for Blocking Adversarial Attacks in Text Classification. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNI.--JCNIIP), pp. 4913,, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/vl/D19.14400 URL
1496
Chen Zhu, Yu Cheng. Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. FreeLB: Enhanced Adversarial Training for Natural Language Understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net, 2020. URL tttp::tts:////eeeeee
net. htt.://wwweeeeooooom.aaaaaaaoa/a
18

---

Published as a conference paper at ICLR 2022
A COMPARISON WITH REGULAR FINETUNING
In this section, we provide comparison between our work with both standard and adversarial regular finetuning baselines SST-2. For regular finetuning experiments, use the 345M GPT2-medium LM, same as the LM adopted in the prefix-tuning experiments. We trained for epochs for stan- dard regular finetuning and epochs for adversarial regular finetuning (earlystopping). The initial learning rate is also set as 5e-5 for both standard and adversarial regular finetuning. The settings of perturbation range e, step size a and iteration of the inner maximization M is the same the those adopted in adversarial prefix-tuming experiments. The results are listed in Table
Table 6: Performance of our robust prefix-tuning with regular finetuning baselines for comparison.
Method
#Epochs
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. regular finetuning
9
93.68
24.77
4.61
30.15
17.74
23,89
std. prefix-tuning
100
92.48
16.64
1.92
31.58
8.84
5.05
+ our framework
92.59
50.36
44.65
58.54
46.68
85.72
adv. regular finetuning
3
93.63
48.38
33.44
48.27
44.59
14.17
adv. prefix-tuning
20
89.51
32.02
17.35
43.33
27.38
8.57
+ our framework
89.57
53.93
48.38
57.88
49.70
73.97
adv. prefix-tuning + our framework
100
93.74
30.53
8.18
33.11
27.51
8.95
93.79
57.55
43.60
60.68
57.17
91.05
Due to time limit, we have only conducted regular finetuning experiments on SST-2 with one adver- sarial training baseline (Miyato et al.) (017)) for comparison. The aim of the comparison, however, neither to set a new SOTA nor to demonstrate that our robust prefix-tuning framework has beaten the robust finetuning techniques. On the contrary, we aim to demonstrate several special properties in the scenario of prefix-tuning, as well as challenges and opportunities in robustness:
Difficulty in optimization. It can be seen that prefix-tuning requires more epochs to con- verge. A potential explanation that the loss landscape of the prefix parameters is highly non-convex, as the downstream task should be learned with far fewer amount of free pa- rameters. The difficulty in optimization might have also brought more challenges on ro- bustness to prefix-tuning tuning. According to Table 6 all prefix-tuning baselines underperform the corresponded regular finetuning approaches against almost all types of attacks.
Trade-off between space and time. is worthy be mentioned that for each prefix-tuning experiment, the saved prefix parameters consume disk storage of only 2MB. In contrast, each regular finetuning experiment takes 725MB disk storage. However, prefix-tuning also takes more epochs (and thus longer time) to converge. This brings even more challenges in robustifying prefix -tnning as an ideal solution should neither harmful to the benefits of prefix-tuning (lightweightness, modularity, not modifying the LM parameters) nor too slow for fear that the time complexity of prefix-tuning further deteriorates. Our framework serves as a possible solution that keeps the strengths without weakening the weakness.
Lack of robustness against UAT. According to Table 6 both adversarial regular finetuning and adversarial refix-tuning fail defend against UAT. A possible explanation is that the adversaries induced during adversarial training are more similar to in-sentence attacks, yet few inductive bias has been introduced against UAT. Existing defense approaches against UAT maintain additional adversary detectors. For example, DARCY (Le et al.) (021) searches for the potential triggers first, and then retrain a classifier using the exploited trig- gers UAT adversary detector; T-Miner (Azizi et al. 2021] leverages Seq2Seq model to probe the hidden representation of the suspicious classifier into synthetic text sequence that is likely to contain adversarial triggers. In addition, neither of the methods have been tested on large-scale pretrained LM. To the best of our knowledge, our framework is the first solution defend against UAT by tuning for pretrained LMM Positioned before the adversarial triggers, the robust prefix regulates the activation the output position so that "the distraction can be ignored".
additional prefix based on prefix-tuning
19

---

Published as a conference paper at ICLR 2022
B DETAILS OF EXPERIMENTAL SETTINGS
We use the SST-2, AG's News, and SNLI benchmark for our text classification experiments. We use the 345M GPT2-medium (725MB storage) as the large-scale pretrained LM and implement prefix- tuning with prefix length of 10. We initialize the prefix token embeddings with parameters from the LM word embedding matrix to ease the optimization. We train 100 epochs for SST-2 and 25 epochs for AG's News and SNLI. We use the Adam W optimizer (Loshchilov & Hutter] 2019) provided by the HuggingFace Transformers library (Wolf et al. 2020) to optimize the prefix with initial learning rate Se-5 in all experiments. Other settings of prefix-tuning follows [i & Liang (2021) During inference our framework, unless specified (Section 4.5 and Appendix[E). we use an adaptive test batch size with dynamic padding to make full use of GPU memory. The total number of tokens in test batch is determined by the setting of the ratio of GPU memory (0.12 for 24GB NVIDIA-3090 GPUs) allocated for loading data. In contrast, the number of sentences in test batch is fixed in experiments in Section 4.3 and Appendix E
The size of obtained prefix vectors are 1.9MB for all tasks. We also save the projection matrices of the layerwise canonical manifolds, but only the matrices of the bottom N = =3 canonical manifolds need be saved
tune the bottom N layers in our robust prefix-tuning. The saved matrices take 12MB storage and can be used for all types of text attacks. We use the Adam optimizer (Kingma & Ba 2015) to tune the additional prefix [.. The initial learning rate and the number of steps (5 I0) for its optimization are tuned the development set.
We use the following five text attack approaches in our experiments:
PWWS (Ren et al. 2019), a score-based word-level text attack that fools a text classifier by applying greedy word substitution inputs.
VIPER (Eger et al. 2019). blind character-level text attack that applies visually similar
character substitution on inputs.
SCPN (yyyer et al. 2018), blind sentence-level text attack that paraphrases inputs.
TextBugger (Li et al. 2019). gradient- based word- and character-level attack that applies greedy word substitution and character manipulation. We refer to it as "BUG" for short.
UAT (Wallace et al. 2219). a gradicnt- based word-level text attack that fools a text classifier by prepending the same adversarial tokens to all test inputs.
We use the OpenAttack toolkit (Zeng et al., 2021) for the in-sentence attacks (PWWS, VIPER, SCPN, TextBugger) and implement UAT by ourselves. For the in-sentence attacks, we use the default hyperparameter settings provided by OpenAttack. Following prior arts (Ren et al. 2019 Jia cet al. 2019; Dong et al.) 2021), we do not attack the premise on SNLL For UAT, following Wallace lt al. 2019), we search for 3-token trigger for inputs of each class in the test set. We set beam size as and iterate over the test set for epochs. We use NVIDIA-3090 GPUs for all of our experiments.
C ADVERSARIAL TRAINING
C.I ADVERSARIAL TRAINING BASELINES
We start with adversarial training for text classification that restricts perturbation within & balls of word or sentence level (Miyato al., 2017). The objective is defined as
min E (e.y))
max C(ylemb(x)) + +r; ||rllae
where r the perturbation injected to the word embeddings of z. The inner maximization of can be implemented by projected gradient descent (Madry et al. 2018) with M iterative updates:
V Vemb(z) emb(x) + 0;
+ Proie|riase
(10)
| Elyb(t)[[yyembbb + +r;
In all of our adversarial prefix-tuning experiments, set 5, 1.25, and M 10.
For texts, the & ball can be defined for both word level and sentence level. For the word level, the & balls constructed for each token in the context, part of the input by offsetting each word vector
20

---

Published as a conference paper at ICLR 2022
Table 7: Results of different baselines with framework applied. This table covers the results of adversarial prefix-tuning with both word- and sentence-level Eg balls and serves as the supplementary for Table Our framework consistently improves robustness of all baselines on all benchmarks.
Benchmark
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
our framework
92.59
50.36
44.65
58.54
46.68
85.72
SST-2
word-level adv.
93.74
30.53
8.18
33.11
27.51
8.95
our framework
93.79
57.55
43.60
60.68
57.17
91.05
sent-level adv.
93.57
30.64
7.25
35,42
25.04
4.88
our framework
93.52
53.21
39.65
54.91
50.69
88.25
std. prefix-tuning
86.42
43.00
24.55
43.20
43.22
52.20
our framework
86.50
53.91
24.93
48.38
51.79
76.47
AG News
word-level adv. + our framework
89.46
50.91
26.30
45.25
47.11
59.74
90.26
56.45
31.25
48.03
54.66
87.26
sent-level adv.
90.18
49.97
24.87
44.54
46.41
38.11
our framework
90.37
57.75
26.50
48,43
54.39
80.63
std. prefix-tuning
72.48
25.51
29.69
42,46
32.88
30.20
framework
72.74
34.68
33.64
43.37
36.59
71.03
SNLI
word-level adv.
77.22
27.43
28.58
46.83
34.94
35.76
our framework
77.65
33.88
33.98
46.92
38.21
76.15
sent-level adv. 78.99
25.75
28.26
44.54
31.94
41.77
+ our framework
79.56
30.91
34.16
44.66
36.41
77.47
within the range of radius of c. For the sentence level, the single E ball is constructed for the entire context part of the input by flattening the word embedding vectors and offsetting the flattened vector within the range of radius The word-level and the sentence-level prefix-tuning take similar training time. As a consequence, adversarial prefix-tuning with both levels of & balls are far slower than standard prefix-tuning The adversarial training approach used in Table 1 is of word-level & balls. The complete results of our approach applied with both word-level and sentence- adversarial prefix tuning, as well as standard prefix-tuning, are shown Table It can be seen that our robust prefix-tuning framework substantially improves robustness of different baseline methods.
We have also experimented with other types of adversarial training for the KL-divergence term for regularization (Miyato al. 2019 Zhang et al. 2019). The objective
such as adding
min B (a..))>D.r C(yla;;
([y)mbb max 3K. (C(ylemb(z)) ((ylemb(x)) + r; 0)]
(11)
We set 3 = 1 and 3 4 and train for 20 epochs for different attempts of KL-divergence regularized adversarial prefix-tuning with Eq. (II). The training loss and the clean accuracy of the development set on the SST-2 benchmark shown in Figure 6 In this figure, we also visualize the word-level adversarial prefix-tuning method shown in Figure for comparison. The KI KL-divergence regularized adversarial prefix-tuning of both settings start with very large training loss. For 3 1, the clean accuracy on the development 27% at the 17-th epoch, but lacks stability (back to 0 again at the 19-th epoch). For 3 4, the clean accuracy remains 0 for all 20 epochs. comparison, the word-level adversarial prefix-tuning used in Section 4.2 achieves low training loss and high clean accuracy on the development set. The difficulty in optimizing the
remains 0 for the first 15 epochs. The accuracy raises around
training loss and the accuracy on the development set have suggested prefix embedding, which might be attributed to its rather small amount of parameters compared with the whole language model. While there are other variants of adversarial training approaches for text classification, in this work, did not them for baselines most of them (Zhu et al. 2020) Jiang et al.) 2020a| [iu et al. 2020) Dong et al. 2021 are based on the KL-divergence regularizer, and also require excessively long training time. It is also of future interest on how to empower robust prefix-tuning framework with faster and better adversarial prefix- feefiintuning approaches.
21

---

Published as a conference paper at ICLR 2022
400
100
80
300
dev adversarial
Train Loss
train loss, ==1
60
Dev ACC
train loss, B 8=4
dev acc, 8 1
dev acc, 8
40
train loss, adversarial
100
20
0
10
15
20
Epoch
Figure 6: Training loss and clean accuracy on the dev set of SST-2 with KL-divergence regularized adversarial prefix-tuning well as the word-level adversarial prefix-tuning used in Section 4.2
C.2 ADVERSARIAL TRAINING WITH EARLYSTOPPING
We apply earlystopping both word-level and sentence-level adversarial prefix-tuning approaches on the SST-2 benchmark at 20 epochs. The results are shown in Table 8 According to the results, the robustness results against different types of in-sentence attacks have been slightly improved. The robustness against UAT, in contrast, slightly drops. In terms of clean accuracy, the earlystopped adversarial prefix-tuiing is even surpassed by the standard prefix-tuning baseline. Our robust prefix- tuning framework still works the prefixes obtained by the earlystopped adversarial prefix-tuning approaches, as the robustness results are consistently improved.
Table 8: Results of earlystopping of adv. prefix-tuning at 20 epochs with our framework on SST-2,
Method Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
+ our framework
92.59
50.36
44.65
58.54
46.68
85.72
100-epoch word-level
93.57
30.64
7.25
35.42
25.04
4.88
our framework
93.52
53.21
39.65
54.91
50.69
88.25
100-epoch sent-level
93.74
30.53
8.18
33.11
27.51
8.95
our framework
93.79
57.55
43.60
60.68
57.17
91.05
20-epoch word-level
90.88
34.71
22.41
39.37
29.00
6.97
framework
90.88
51.18
46.46
58.26
50.74
84.95
20-epoch sent-level
89.51
32.02
17.35
43.33
27.38
8.57
our framework
89.57
53.93
48.38
57.88
49.70
73.97
D ADVERSARIAL DATA AUGMENTATION
In this section, we report additional experimental results of adversarial data augmentation using VIPER or SCPN generate adversarial data for augmentation during training. We also the milestones of 20, 50, and 100 epochs for training. According to Table the robustness against the attack of the same type used for adversarial data augmentation during training has gained significant improvements compared with the standard prefix-tuning baseline. Robustness against other types of attacks, however, fails to improve with the process of training. For our approach, we use the correctly classified data from both the clean training set and the training data perturbed by the specific type of attack used for augmentation to construct the canonical manifolds. The experimental results indicate that our robust prefix-tuning framework still improves the robustness against all types of text attacks over all types of baselines (except for the VIPER attack VIPER adversarial data augmentation that incurs slight drop the 20-epoch and 50-epoch milestones).
22

---

Published as a conference paper at ICLR 2022
Table 9: Results of VIPER and SCPN data augmentation baselines well as methods SST-2.
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
framework
92.59
50.36
44.65
58.54
46.68
85.72
20-epoch VIPER aug.
82.59
38.93
53.32
50.85
37.51
10.49
our framework
83.09
50.03
51.89
54.09
50.91
59.75
50-epoch VIPER aug.
90.06
36.02
63.64
45,47
30.97
21.03
our framework
89.73
51.89
63.21
58.15
47.83
81.49
100-epoch VIPER aug.
93.63
25.54
52.22
36.19
21.75
5.22
framework
92.97
51.18
52.61
59.75
49.53
84.57
20 20-epoch SCPN aug.
72.27
41.35
40.14
54.64
42.17
16.20
our framework
72.38
50.74
49.97
54.75
50.80
62.11
50-epoch SCPN aug.
87.64
34.32
43.33
62.93
35.04
10.38
our framework
87.64
48.82
48.76
63.98
49.82
82.43
100-epoch SCPN aug. our framework
89.13
28.45
39.32
59.91
23.39
2.69
89.24
51.67
48.60
62.11
48.33
79.35
We attach the clock time that each method takes on each benchmark for comparison in Table [1 The time differs for each benchmark because of different sizes TO). The training-phast adversarial prefix-tuning takes far greater time than standard prefix-tuning. TTee adversarial data augmentation methods during training phase take the longest time due to the cumpuuttionalll-ccalleengg search/optimization of text attacks. For our framework,, steps and 2 that construct the canonical manifolds are viewed as preprocessing since it can used against all attacks during inference. As further speed up step by random sampling in Sc- The additional prefix [ is tuned on the fly during test phase in step 3 of our framework, which slackens the standard testing but negligible compared with the training-phhi -haaiin baseline methods. In conclusion, framework has preserved the efficiency and modularity of prefix-tuning.
discrete
||Scll >> d (mentioned in Section 3).
Dataset statistics for each benchmark. We have also included the number of classes in each benchmark and the accuracy of random classifier in theory for better understanding.
Table 10:
Benchmark #Classes
Train
Dev
Test
Random Acc(%)
SST-2
6,920
872
1,821
50.00
AG's News SNLI
115,000
5,000
7,600
25,00
550.152
10,000
10,000
33.33
Table 11: Time used by different methods for all benchmarks. Compared with the time-consuming training- phase baseline methods, our test-phase robust prefix-tuning is significantly faster.
Time (GPU min)
Phase
Method
SST-2
AG's News
SNLI
std. prefix-tuning adv. prefix-tuning
3.7x 100
19.7x25 223.0x25
55,4x25
17.7x
641.1x25
Train
adv. aug. PWWS adv. aug. VIPER adv. aug. SCPN
82.5x 100
47.3x 100
285.3x 100
Preprocess
ours: steps 1 and 2 std. testing
1.0
69.9
721.5
0.3
1.2
0.7
Test
ours: step 3 (adaptive)
1.0
6.5
6.5
23

---

Published as a conference paper at ICLR 2022
E MORE REALISTIC EVALUATION SETTINGS
E. PERFORMANCE WITH SMALL TEST BATCH SIZES
In this section, we continue study why our methods with test batch size of 2 or 4 achieve stronger robustness against UAT (shown in Table 3 We note that the j-th layer activation at the output position F is normalized by rows before being projected the canonical manifold M when
the test batch size is larger than > I. According to Eq. (8) and the description in Section 3 when the size of test batch is larger than namely, when
u
(12)
has more than 1 row ((ST) 1), the objective for tuning Ph
N-1
N-1
min Σ L CL60
Σ j=0
(1
( (1-
(13)
The normalization is performed alleviate the randomness among the samples in the test batch. not applicable when [Sy] = 1, as H 11 2 (11HHH//11 /|Sy would be O. Therefore, for test batch size 1, the loss for tuning P Eq.
without normalization before projection.
We conduct experiments with test batch sizes of 2 and 4 without normalizing the test batch before projection during inference for comparison. According to the results in Table [2] for test batch sizes of 2 and 4, robustness results against UAT have degenerated without the normalization, but they still better than that with test batch size of 1 as mini batch optimization may achieve better results than the fully online setting. The robust performance of our framework against other in-sentence attacks remains similar, with or without normalizing.
Table 12: Performance of our method with std. prefix-tuning with various test batch sizes on SST-2. For test batch sizes of 2 and 4, we compare the robustness performance of our framework with and without normalization before projection during inference.
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
bsz adaptive
92.59
50.36
44.65
58.54
46.68
85.72
ours, bsz = (unnormalized)
92.64
51.46
43.99
58.32
46.73
73.09
ours, bsz 2 (normalized)
92.48
49.92
45.74
59.14
48.93
84.73
ours, bsz. = 4 (normalized)
92.64
50.36
45.85
58.54
49.31
86.66
ours, bsz 2, unnormalized
92.53
51.13
42.89
57.99
48,00
78.20
ours, bsz, 4, unnormalized
92.59
51.84
43.44
58.59
47.89
80.23
The results lead to a temporary conclusion that normalization before projection during inference can be important when using our framework to defend against UAT. However, the performance gap between the unnormalized test batch size setting and the normalized test batch size > 1 setting much smaller on other benchmarks and other prefix-tuning baselines. We set test batch size as 1 and evaluate the performance our framework with both standard and adversarial prefix-tuning on both SST-2 and AG's News benchmarks. The results in Table 13 show that the robustness gap against UAT between the unnormalized test batch size setting and the normalized test batch size setting, though still exists, is relatively small compared with the results Table 12
As result, for conventional evaluation where accuracy under specific attack is reported, we will use test batch size larger than (2 4 should be satisfactory). as normalization before projection for additional tuning during inference can be beneficial. In contrast, we will set test batch size as 1 in more realistic settings,
able to cope with real-time or mixed test data.
Our another attempt for improvement is to set the normalization before projection during inference with pre-computed vectors heforehand. Currently the normalization 11 TT (T() /\Sr| in Eq. (13)
24

---

Published as a conference paper at ICLR 2022
Table 13: Performance of our framework with both standard and adversarial prefix tunnnng on both SST-2 and AG's News benchmarks, with test batch size of 1 or adaptive.
Benchmark
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
+ ours, bsz adaptive
92.59
50.36
44.65
58.54
46.68
85.72
ours, bsz =
92.64
51.46
43.99
58.32
46.73
73.09
SST-2
adv. prefix-tuning
93.57
30.64
7.25
35.42
25.04
4.88
+ ours, bsz adaptive
93.79
57.55
43.60
60.68
57.17
91.87
ours, bsz
94.12
56.67
46.13
60.30
55.19
87.37
std. prefix-tuning
86.42
43.00
24.55
43.20
43.22
52.20
ours, bsz adaptive
86.50
53,91
24.93
48.38
51.79
76.47
+ ours, bsz =
86.75
46.47
35.82
45.61
49.00
73.08
AG's News
adv. prefix-tuning
89.46
50.91
26.30
45,25
47.11
59.74
ours, bsz adaptive
90.26
56.45
31.25
48.03
54.66
87.26
+ ours, bsz =
89.79
55.72
34.33
49.43
53.75
87.16
depends on H.) with limitation in the setting of test batch size 1 and concern in reproducibility. In our additional experiments, we pre-compute the normalization of the j-th layer activation during inference using Sc. the set of correctly classified training examples. Consequently, we can replace 11 HT /|Sr| in Eq. 13) with 11 (TTT /|Scl, which can be obtained the second step Section 3 In this way, the normalization is static for different batches and applicable to test batch with the size of 1. The results are shown in Table
Table 14: Performance our method with std. prefix-tuning with static or dynamic normalization before projection during inference SST-2.
Method
Clean
PWWS
VIPER
SCPN
BUG
UAT
std. prefix-tuning
92.48
16.64
1.92
31.58
8.84
5.05
ours, bsz adaptive
92.59
50.36
44.65
58.54
46.68
85.72
ours, bsz (unnormalized)
92.64
51.46
43.99
58.32
46.73
73.09
ours, bsz = 2 (dynamic norm.)
92.48
49.92
45.74
59.14
48.93
84.73
ours, bsz 4 (dynamic norm.)
92.64
50.36
45.85
58.54
49.31
86.66
ours, bsz = 1 static norm.
92.53
49.48
40.25
58.43
48.98
87.92
ours, bsz 2, static norm.
92.37
50.58
41.57
58.48
47.17
87.70
ours, bsz = 4, static norm.
92.48
49.70
40.69
58.15
48.22
87.64
On the one hand, can be seen that our framework with static normalization outperforms the ones with dynamic normalization under UAT. The improvement is especially substantial for the setting with test batch size 1, indicating that while the dynamic normalization 11 H /|Sr] is not applicable for test batch size 1, the static normalization 11 (1 /|Scl can be helpful in this setting. On the other hand, for in-sentence attacks, the robustness results are comparable between static and dynamic normalization, though there is performance drop under VIPER. Due to time limit, we have only experimented on SST-2 with standard prefix-tuming as the baseline. We will conduct more experiments to further evaluate static normalization with other baselines different benchmarks. In the following experiments under more realistic threat models, we adopt the setting of the test batch size without normalization before projection.
25

---

Published as a conference paper at ICLR 2022
E.2 PERFORMANCE UNDER MIXED TEST DATA
In this section, we provide more experimental results of our framework under mixed test data. We combine the clean test with the attacks separately and combine the two sets as mixed test data. The mixed test data can be from more realistic scenario. To cope with it, we set the test batch size as 1 in order avoid the situation where both perturbed and unperturbed test sentences or perturbed sentences under different attacks exist a test batch. Besides, by setting the test batch size as 1 can deal with real-time data, which is also a practical approach. We use adversarial wrefix-tuning as the baseline to obtain strong prefix Polú, While we are not able to tune the robust prefix under specific attack in this setting, we find it effective to tune the initial learning rate and the number of steps (5 or 10) for our inference-phase robust prefix on the UAT-attacked development set and fix them for every incoming test sentence. The results are listed in Table [5
under one attack, or perturb the test set with two different
Table 15: Performance of our framework under mixed test data on SST-2 and AG's News. The '+' denotes combination of test set clean under attack, and 'C' is short for "Clean", 'B' for "BUG", etc. The test batch size is set as for experiments with our framework. We also provide averaged results with the robust prefix in our framework separately tuned on the two test sets as an upper bound for comparison.
Benchmark
Method
C+ B
C+P
V+ B
V+P
S+B
S+P P
U+ B
U+D
adv. PT 60.65
62.27
17.76
20.43
29.57
31.85
18.12
20.81
SST-2
+ ours mixed
69.71
71.11
46.02
47.23
55.63
56.92
70.04
71.67
ours, separate
74.66
75.40
50.66
51,40
57.75
58.49
71.28
72.02
adv. PT
52.38
53.41
26.54
27.14
34.86
35.36
40.97
42.16
AG's News
+ mixed
71.36
71.39
42.84
43.99
49.04
50.12
69.88
71.30
ours, separate
71.77
72.76
44.04
45.03
51.59
52.58
70.46
71.44
According to the results, the performance under mixed test data is consistently improved using our framework. We also provide the averaged results with the robust prefix our framework separately tuned on the two test sets an upper bound for comparison. It can be seen that small performance gap exists between the results under mixed data and the averaged results with separately tuning for specific test sets. A potential reason is that our framework attack. To reduce the performance gap, we can construct more powerful canonical manifolds that capture richer data information of multiple granularity that our framework also achieves optimal performance under mixed test data. We leave the exploration for future work.
better adapt to the data under specific
26

---

Published as a conference paper at ICLR 2022
F MODEL BEHAVIOR ANALYSIS
F.1 IN-SENTENCE ATTACKS: AVERAGING THE ATTENTION
We provide the attention weights from the LM final layer over the entire [context, question, [ANS]] with PWWS as the text attack from the development set of SST-2. The perturbed inputs are shown in Table and the visualizations are shown Figure 7 Figure g and Figure / Similar behavior of averaging the attention of the final layer in the LM can be also ob- served as the BUG attack visualized in Figure Caveat: the observed behavior is of the final layer only, as attention is unreliable for indicating importance over input tokens (Jain & Wallace 2019
Wiegreffe & Pinter 2019: Abnar& & Zuidema. 2020: Serrano & Smith 2019).
Table 16: Original inputs from the development set of SST-2 with perturbation by PWWS.
Type
Input (the context part)
Predict
Label
Original
giggle a minute.
positive negative
positive
Perturbed
A giggle are minute.
positive
Original
marvel like none you' ve seen. A wonder I none you've get.
positive
positive
Perturbed
negative
positive
Original
rarely has so much money delivered so little entertainment.
negative
negative negative
Perturbed
Rarely has so much money delivered so picayune entertainment.
positive
minute
revies
negative
FANS]
mirrlt
negative
(ANS)
or
or
0.3
0.4
5gge
0.0 0.2
0.5 0.3
minute-
0.1 0.1
D:2
0.2 0.5
0.1
A1
07
A1
0.0
0.1
0.1
0.1
0.3 0.4
minute
0.
0.
0
0.
0
DJ
0.0
0.1
0.1
0.1
0.2
0.2 0.3
00.0
0.000
+0.0
0.1
0.1
0.1
0.7 0.7
0.00
0.0
0.2
0.2
0.1
0.1
review
negative
0.0
0.0
0.0
0.0 0.0
0.7
negative
00.0
or-00
positive
0.0
0.0
positiow
-0.0
6d
an
0.0
0.0
7-00
5.
0.
[ANS] 0.0 0.0
0.0
[ANS] 000
(a) Predict:positive
(b) Predict:negative
minute
megaliove
postive
[ANS]
minute
regatiive
poxbiive
[ANS]
3.0
0.5
0.E
fggle
lpgle
minute
minute
15- 1.1
9-1
this
-81
this 8.1 01
511
review
-81
03
5.5
0.1
0.1
review
0.1 03
51
0.1
negetive
8:1
negetive
9.1
0:3
811
or
Cr 0r-
iositvve
positive
[ANS]
81 01
(c) Predict:positive
(d) Prediet:positive
Figure 7: Visualized attention weights from the final layer in the LM for the first example in Table
16 (a): Original input with original -tafing (b): PWWS perturbed input with original prefix-
tuning: (c) and (d): PWWS- -errurrrt input with robust
of (c) N 24 and (d) N 3.
27

---

Published as a conference paper at ICLR 2022
none you marve Rice
seen
begetive
positive
TANS]
wonder
nond You
get
negative
posative
TANSI
wonder
0.2
0.8
0.4.0.4
-01 0.7
0.7
-01 c.
10.5
0.3
0.0
0.2
0.2
0.2
0.0
0.1
0.3
0.00
0.3
0.3
0.1
0.1
0.2
0.000.1
0.1
0.0
-0.0
0.0
0.2
0.1
0.2
0.0
0.1
DUI
0.0 0.3
0.1
0.0
0.0
0.0
0.1
0.2
01
0.1
0.0
0.1
3.1
0.000.0
0.0
0.0
0.2
+0.0
0.1
0.0
0.3
this
-.00 0.1
0.1 0.0
0.0
0.0
D.1
this
00.0
0.1
0.2
0.1
0.1
0.0
0.1
0.2
C.3
review
-..00.01
0.1
0.0
0.0
0.0
0
0.2
D.1 0.
review
+0.0
0.1
0.0
0.
0.1
0.0
0.0
3.1
0.2
0.2 (0.3
negative
0.
6.0
0.0 R.0
8.0
B.P
DV1
OLD
negative
-0.0
0.1
0.0
A.C
0.0
0.0
000
3.
O.1
C.D
OLO
or-
0.000.11
0.1
0.1
0.0
0.0
0.0
0.1
0.1
0.1
0.1
00.00.0
0.1
0.
0.0
0.1
0.0
0.1
311
0.2
C-3
O.3
10.3
pasitive-o
2.000.0
0.0
0.0
0.0 0.1
011
CLU
0.0 0L3
pasitive
-0.0
0.1
0.0
0.0
0.0
0.0
0.0
3.0
0.0
0.0
? 7-0.0
0.0
0.0
7-000
0.1
0.0
0.0
0.0
3.0
0.0
[ANS] 0.0
0.0
0.0
0.0
0.0
0.101
[ANS] 0.0
0.
0.0
0.0
0.0
0.0
0.0
0.0
Predict:positive
(b) Predictneggative
Wonder
Pone You
Ang Det
negative reV
Dositvv
[ANS]
wander
Honn you
ki Dot
negarive reV
Dostiie
[ANS]
0.
wonder
0.4
0..3
1.3.3
0.2 0.3
0.2 0.3
002
0.2 0.022
0.2 0.2022 0.2
0.2
0.2 0.2
0.2
0.1 0.2 0.2
0.3
0.3
0.1
0.1
0.2
0.1
0.1
0.3
0.3
0.1
-0.
0.1
0.1
0.1
0.1
0.1
0.1
0.1
-0.1
0.1
0.
0.3
0.1
0.1 0.1
-0.3
0.3
0.5
0.1
0.1
0.1
0.1
0.1 0.1
-0.3
0.3
6.
0.1 0.1
-0.3
0.2
0.2
0.5
01
0.5
0.1
0.3
-0.1
0.3
6.
0.
0.2
this
-0.3
0.2
0.5
0.1
0.3
this -0.1
0.2
0.1
0.1
review
-0.1
0.1
0.1
0.1
0.1
0.1
0.3
review -0.1
0.1
0.
negative
00.1
0.1
0
0.3
0.0 0.2
negative -0.1
0.1
0.2
0.3
0.3
Qr 00.1
0.1
0.1
0.3
0.0
or -0.1
0.1
0.1
positive
0..0
0.0
0.0
0.0
0.0
0.1
positive -0.1
0.1
70000
0.0
0.0
3.0
0.3
0.3
0.2
0.1
01
7-0.1
0.1
[ANS] -0.0
6.0
0.3
0.1
0.1
61
[AN5] -0.1
0.3
0.3
0.3
(c) Predict:positive
(d) Predictpositiive
Figure 8: Visualized attention weights from the final layer in the LM for the second example in Table 16 (a): Original input with original prefix-tuning: (b): PWWS-perturbed input with original prefix-tunings (c) and (d): PWWS- WWWS-perturbed input with robust prefix-tuning of (c) N = 24 and (d)
N 3.
28

---

Published as a conference paper at ICLR 2022
entertaion
Farely Agg
delluered
vve
Posetive [ANS)
Rore 0
Thee (ANS)
rarely
Rare
has
0500
ly
50
0.304.03
2.203.0....
has
-03 02
money
money
delivered
0.100/010000300
delivered
+53 6a 3177/1
-B8 a1
little
pic 00011/11000 F3
entertainment
8.00000..00.000000
0.000030
une -8x oz 00/00/ 22 (2000
ay 11
0.3
entertainment 00
22
1s
0.3
this
is 5 11
review
00.0000000.001000000000
this -83 pejonenpppp 1
negative
0.000.0 nd
review
00/00000
positive
or
2.0000..00.00
0.30000.00030000.... 00000.000000000000.0000000
000.000 0..0
010.000.000.0000.
negative
positive
or- -83-3300 60 00:0000000 20 13 14
p...0......000. 0.000000000000...00.0.0 0 0. 0
01000011000000000000000101001100 0
[ANS]
p.000......00.. 0.00.00
3.0.000 0.1.000
[ANS]
-Fa patoroopoopaaa aa
1000000 8
(a) Predicttnngative
(b) Predict:positive
entertaim
entertainnen
Poniive Wive
TANS!
Rare
Post [ASS)
Rare
Rare
has
has
50
50
much
87 p7
-11 17
money
money
delivered
delivered
-01
50
-1 010010111111000000000
50 -00
-01 ju
ay
ay
une
une -0 [3/
entertainment
entertainment |3
Is 23
this
+1
this [
01
review -00 (c/ (2 C2
negetive
negative
or
Ca
Or 53 ca La 53 22
positive
13
positive -nnon E3 pe 1n
-00000 E3
[ANS]
[ANS]
21 00000000 col
(c) Predict:negative
(d) Predictnegativv
Figure 9: Visualized attention weights from the final layer in the LM for the third example in Table
16
(a): Original input with original prefix-tuming (b): PWWS-perturdss perturbed input with original prefix-
tuning: (c) and (d): PWWS-perturbed input with robust peeix-tuunng of (c) N 24 and (d) N = 3.
In this example, robust prefix-tuning with N = 24 steers the final layer in the LM with the absolute behavior of averaging the attention, while that with N 3 steers to assign higher attention weights to the input vector at the position of the input token "entertainment" as well as that of each token in the question part. We use this visualization to show the slightly different behavior of our framework against in- in-sentence attacks when tuning different bottom N layers.
29

---

Published as a conference paper at ICLR 2022
F.2 UNIVERSAL ADVERSARIAL TRIGGERS: IGNORING THE DISTRACTION
F.2.1 THE ALTERNATIVE PROXY
In this section, we introduce the alternative proxy for the analysis of token importance replace- ment of attention. To interpret token importance, following [Serrano Smith (2019), we leverage an alternative gradient-based importance ranking proxy. As stated in Serrano & Smith (2019), while does not provide a gold-standard indicator of importance value for each token to the model, the proxy is more reliable in reflecting the importance ordering for tokens in the sequence. In the proxy, the importance for each token is given by the multiplication of the gradient of the decision function with respect to each attention weight and the attention weight magnitude. As there either 2, 3, or 4 classes in our text classification benchmarks, the decision function at the output position is given by
[W ( (n5-
]
exp max;
do W (N(41)
(14)
E. exp W (19
where W in the LM transforms the top-layer output h probability vector over the dictionary, and i E Slabel traverses all the class label tokens in the vocabulary (S)abell = the number of classes the benchmark). The decision function is proposed in [Serrano & Smith (2019) analyze hierarchical attention network (Yang et al.) 2016). We further augment the decision function for time steps < o to study the entire behavior of how the autoregressive LM assigns importance to each token visible at each time step. As introduced in Section [ the autoregressive LM generates sequence [xr ū) with [[CLS), xr] as the input sequence for text classífication tasks (ÿ is the generated label token as prediction). For each time step < o, the output of the LM is a token from x. As result, we define the decision function at the k-th position as
exp W (11²
dk (w (16 ))
(15)
Ejev exp W
(1e (1)) I,
with 1k-1 (or (CLS) for k- = 0) as input and I& as output. The complete decision function is then
d (w (1151) W (1(
W (1
(n(h))=
Σ
dx
W (nn
(14))
(16)
In this way, at each time step i, the gradient taken from the decision function (Eq. (6)) with respect to each attention weight a. (j S i) in the final layer of the LM can be obtained with automatic differentiation in PyTorch (Paszke et al. 2019). We multiply the gradient with the attention weight magnitude as the value of token importance, which ranks token with both high attention weight and high calculated gradient to be most important. Formally, for the attention weight on the j-th token the i-th time step the h-th head of the final layer (namely a
(f)), ), its importance value is
(h)
Od
(h) (h)
(17)
Da
where j < í. The upper triangle part of the importance matrix is still masked, namely T(hh) 0 when j > i. We average the importance matrix over all heads in the final layer. Note that the averaged importance value can be negative, as each calculated gradient Od Od/ (h) in Eq. (17) can be negative. For better visualization, we further normalize all the token importance values al each time step i by subtracting the minimum value so that the negative values are eliminated. We finally normalize all the (now positive) token importance values at each time step into probability vector. Formally,
13
1 H
i,
min Ty Lnj
(18)
E
where H. is the number of heads in the final layer. The latter two equations transform i' to [i. as a probability vector. This is similar to an attention row, which is a better form for visualization. The importance matrix I is plotted for all the case studies under UAT attack (Figure and Appendix F.2.2.. Also note that the transformation from I' to is prder-preserving for each row, and we will only use the importance ordering information in quantitative studies F2.3
30

---

Published as a conference paper at ICLR 2022
F.2.2 QUALITATIVE ANALYSIS
We use UAT attack inputs from the development set of SST-2 for visualization. The perturbed inputs are shown in Table 17 and importance visualizations are shown in Figures To Here we also plot the importance between each token pairs of the adversarial triggers. We can observe the behavior of ignoring the distraction for our robust prefix-tuning framework with N = 3.
Table 17; Original inputs from the development set of SST-2 with perturbation by UAT.
Type
Input (the context part)
Predict
Label
Original
this is so bad.
negative positive
negative negative
Perturbed
lifts mates who this so bad.
Original
treats women like idiots.
negative positive
negative
Perturbed
lifts mates who it treats women like idiots.
negative
beautifully observed, miraculously unsentimental comedy-drama
Original
positive
positive
grades lame Invalid beautifully observed, miraculously unsentimental comedy-dramas
Perturbed
negative
positive
this
bad
rey tnis
negative or
positive
[ANS]
Ange is ag0d
positive (ANS)
this
1.0
lifts 1.
mates
who 0.60.60.0
50
0.4
0.1
0.0
this
0.30.40.00.3
0.00
0.0
is -5.00.0.50.0..2.
0.2
0.0
0.1
0.2
0.5
so 9.20 222
0.00
0.0
0.3
bad 0110:10.0011:220111
5,20.00.0020000000000
this
0.1
0.0
0.1
0.2
is 0-10..0.0011011000010...00
review
0.1
0.1
0.1
0.0
0.1
0.0
0.1
this 0.20022..00.22::0:0000.......
negative
0.0
0.1
review
4.20:30.000110:03.00..00.
or
0.1
0.1
0.1
0.1
0.2
0.2
0.1
0.0 0.1
negative 3.00.00..0.....111
positive
0.0
0.0
0.0
01.000..10.0000p..o
10.100000011111100000000...
Dositive-0.00.110. 11100010000000000000000000000000 20
0.0
0.0
0.1
0.1
0.3
0.3
0.0
0.0
0.1
0.1
0.0
0.1
? 2-0.00.10.00.000333
1110000011111111111111111 1000
-0.0
0.0
0.0
0.2
[ANS)
b.20100.0..10330. 0tt.:/00000000........... htt.::pp.ppp.0011111111110000010
(a) Predict:negative
(b) Predict:oositive
Mat Rn ag0
positive
[ANS]
My 2 god
positive [ANS]
0.70.3
mates 050.5
who 0.70.30.0
who 0.30.30.4
this 0.30.30.20.2
this 0.30.20.20.4
is 0.22020.200.2.2
0.220.10.22..3.
so -.220.2..20.0..107
so -0..1..10..0..0.1.0
bad -.0000.0000110022220000
bad 1...0....p.........0
-0100:10.10010.p00000.00
-...0..0p..ppppoooo.....
is 0010000011010011000000::000
is 001.0.....0000001111.0..000
this 01101001001011000011111101
this 010.0000000000001100000000000000
020010.10110100111000.0000100000
0110:00110100::11000000.00000000
negative a1000000101100000000000000000
negative 4100000..0.000000000000111111100
or or.2ala.h......100000000.0......
or -r......pp.10..
positive -0.10.10.10.....01..11
10.00000.00000011111111
positive wtw..............0..... 01..0000000....00000000000000000
7-9.30.00.00.00. 10.11111111110000000
[ANS] (MS))00.00000.00.0.o 11110000111111000000000000000000
[WS]]-0.00.0p.pp.op.... 1010000000000110...000.000
(c) Predict:positive
(d) Predict:negative
Figure 10: Visualized importance matrices for the first example in Table 17 with original prefix-tuning: (b): UAT-attacked input with original attacked input with robust prefix-tuning of (c) N 24 and (d) N 3.
(a): Original input
(c) and (d): UAT-
31

---

Published as a conference paper at ICLR 2022
treats
women like
idiots
negative viee or this
positive
[ANS]
Magee hoo
P p
Regative
positive TAYS)
it 1.0
lifts
treats
0.1 0.9
mates 055.
who 0.40.0.0
women
it 0..00.
like
0.1 0.3
0.6 0.1
treats 0.00330.0.000.3
idiots
women 2.20020..00.0.300.
0.000.2 0.11 0.1 0.1
0.5
like 0200110.00000.044000
idiots 4.10:.0..001110.0.10...00
Is 0.1
0220000.0000002000000000000
this 0.1
0.1
0.1
0.1
0.0
0.1
0.1
0.3
-0110022.000.00220001:::00:00000
review
0.1
0.0
this cc11020.0..003200000011..o0
negative
0.1
0.1
0.1
0.1
0.1
0.0 0.4
review 422p:///uu....uuuu.......0011110
or 0.1
0.1
negative
positive
0.0 0.0
0.1
0.0
0.0
0.1
0.0
0.0
0.3
0.1
0.2
positive cttiiiee-..c......0000
or 011.0:0000....000000000000111110 0t.00.0..0..00..0........
7.0.0
? wttppppo..oo.............
[ANS] 0.1 0.1
0.0
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
[ANS] ott.po.....p...0.....0..........
10.10.10100000000000
Predict:negative
(b) Predict:positive
Magee tho
positive
(ANS)
Mage ho 7 Higts
positive (ANS) OF
lifts 1.0
lifts 1.0
mates
0.3
mates
0.5
who
0.7 0.30..0.0
who
0.3.0.30.4
it 2.330.00.2202
it 6.2.3.220333.3
treats
0.00.10..0..3..
treats
0.200.0.200027
women
8.40.0010000.10..1
women
8.20020.20.20.1101.
like 4.200.10.10.100100..1
6.100.1..j0100000223
idiots
0:10:10:10110:101100000
idiots
2.00000.000103:1110000
0:10010:1|011011000000000
0.100:10:10.1000000111 00000000100100100000.00000
[=1101001001000000000101111
is
0:1001001111010110000000
this
10100.
this
|00000000..011100000000000000000 |0100000000000000000000011011
review
0::0:10:|0110.pp100::01 1..00000000000000000
review
6:10:10.j0110:]000:::00000
1010...1
negative
0.200100100.10 1010 0.10 0.000
0.000
negative
2.00000.000.1011000000
0.00
or 0:10110110011
or 0.000000000110111110111
positive
Dositive-E.10.10.1100 6.10.10..00.10
1.000
positive-
0.10 00.0000.100 3..0...00.
w.0...0.00.00..coo 0.000
0.00
10101011
0.0.0.00..0000..20000 2.11000......0000000 012100.1..2000000000000000000000 011.0:00000001110000000000000000
[ANS]
w.d.a....oo..p..oo...
0.00
0.000. 0.10
0.00 0.00
[ANS] 0=100.100:10.000.0.01111 ap.p..0...000................. 0..../.....................
Predict:positive
(d) Prodictneeative
Figure 11: Visualized importance matrices for the second example in Table 17 (a): Original input with original prefix-tuning: (b): UAT-attacked input with original prefix- (c) and (d): UAT- attacked input with robust prefix-tuning of (c) N 24 and (d) N = 3.
32

---

Published as a conference paper at ICLR 2022
heautierrve
ulousty uns mirac
imental ent
comedy
rama
grad
momeady
beautifully
1.0
grades
observed
lame
0.400.6
Invalid 0.3.0.33...4
0.2 0.4 0.4
beautifully
0.2.0.4...1...3
mirac
0.0 0.2 0.5
0.2
observed
0..2:0.2/0.330000000
ulously
0..10.3.0.. 0.000.00....00000...
0.3
0.3
0.2
0.0
0.0
mirac
ulously
00:10..200.1000000000 0.000.1 0.200.0..
ent
0.1
0.3
0.2
0.1
0.2
0.1
0.0
uns 00.000.200.10.1100000 0.100000000000000000000000 01110000000000000000000000000000
imental
0.1
0.1
0.2
0.0
ent
00.1000100000000200000 0:000000000000000011100000000001
comedy
imental
00.10.100. 011010000.0000000000000000000000
0.1
0.1
0.1
0.1
0.1
0.0
0.1
comedy
00110010000000000100000110 00110010022000000000000000001011 00000000000000000100000000000000
00.10..200010000 00000000000000000000000000000000
d
0.1
0.1
0.1
0.1
0.1
0.0
0.0
0.1
0.1
0.1
d
0.100.00..000 01100000000000100000000000000000
01110000000000000000000000000000
rama
rama
0.1
0.1
0.0
0.1
0.1
0.1
1.11 0.1 000.000000.000000000000000000000
(a) Predict.positive
(b) Predictinegative
grad
aama
Mousiin
ragna
grades
grades
lame
lame
Invalid
Invalid
beautifully
beautifully
observed
0.2002002002
observed 0110000000004
0220222222222222
0010::00000022
mirac
0.161.0..1110
mirac 0.00:]0:122010222
ulously
0.111.1...1111
ulously 00.000000000000000000000
uns
0.100.1..111111
0.100.000
uns 00100:00:10010011110000
ent
0.11000
ent emtt=000c:]03]000000000007
imental 0.11
imental 00000000000000000000000000000000
comedy
0.100:000..
comedy -0000000000000000000000
0.33...
0.1
0.1
-0:10:1001
0.1
0.110.10
001
0.1 10
0.000.0
rama 001
0
rama
0.1
0.100.000
0.000.00 0.000
1100001000 0.000.000
0
-0.00.00000
0.0 0.0
0.1
0.1
0.100.10.
(c) Predict:negative
(d) Predict:positive
Figure 12:
Visualized importance matrices for the third example in Table 17 (a): Original input
with original prefix-tuning: (b): UAT-attacked input with original prefix-tuning: (c) and (d): UAT- attacked input with robust prefix-tuming of (c) N = 24 and (d) N = 3. We omit the visualization of the question part for this example due to its context length.
33

---

Published as a conference paper at ICLR 2022
F.2.3 QUANTITATIVE ANALYSIS
In this section, we conduct quantitative analysis show that for UAT, the robust prefix steers the LM to ignore the distraction of the adversarial triggers. We first investigate whether the trigger is assigned with less importance under the robust prefix. Based on the proxy, at each time step i, we get the rankings Kty for all visible tokens T.;:
(19) where the argsort operator, provided in frameworks like NumPy (Harris et al. 2020) and PyTorch (Harris al., 2020), returns the ranking of each value in the vector by increasing order. Our quan- titative analysis is based on token importance rankings at each time step, as the proxy I does not provide a gold indicator for importance value but rather an alternative ordering method. We inves- tigate the importance of the adversarial trigger by calculating the relative ranking of the token with maximal importance the trigger. We set the trigger length as in our experiments, so for each time step 2 3. the relative ranking is
Kii.: i 1] = argsort(((i;; : 1])
max K[i : 3)) 1) x 100%.
(20)
The example-level Degree of Distraction (DoD) is defined as the importance of the adversarial trigger averaged at all time steps since the generation of the first token in context:
DoD(x)
max (K(;: 3)/(1 + 1) x 100%
(21)
1
where x represents the input example and K the calculated importance ranking matrix. Corpus- level Degree of Distraction (cDoD) is then defined as the example-level degree of distraction aver- aged over all UAT-attacked examples. We calculate the eDoD of the original prefix-tuning as well as our framework with N = 24 and N = 3 under UAT attack on the SST-2 development set. The results are listed in Table 18 Similar to a trustworthy evaluation for machine translation (Koehn)
Table 18: Calculated corpus-level Degree of Distraction of the original framework under UAT attack. Results with & are statistically significant over the that of the standard prefix-tuning baseline with p < 0.001.
as well as our
Method
cDoD + (%)
std. prefix-tuning
67.67
ours, N = 24
62.06t
ours, N = 3 61.68t
Marie et al. 2021), we also conduct statistical significance tests by bootstrapping the sets of DoD of all test samples calculated for different methods. According to Table [8] the cDoD) results of our framework (both N = 24 and N 3) are lower than that of the original prefix-tuning with statistical significance (p 0.001).
2004
We proceed to study whether the robust prefix manages to steer the LM to assign the highest impor- tance to the most essential token as the original prefix-tuning does with inputs without adversarial triggers. Let Lo be the importance matrix for the baseline method with clean input, and I, be the im- portance matrix for some method with the same input but prepended with adversarial trigger tokens. Let the notation o indicate the output position for the UAT-attacked input, then the output position for the clean input is 3 (3 is the trigger length). At each time step i, We use the indicator
(1) = (argmax (Tost; i 1)) = argmax (Tuli 3, i + 4]) - 3), 0 O
(22)
to determine whether the method with UAT-attacked inputs recognize the essential token by assign- ing the highest importance to the same token (argmax (T. + 3, i + 4)) 3. left-shifted by the trigger length 3) the baseline with clean inputs does (argmax (To
+ 1J)) The subtraction of 3 in the RHS of Eq. (22) is due to the right shift of the original input by the trigger tokens. Similar to the calculation of DoD and cDoD, we compute the example-Ievel recognition of the essential token
(RoE) for different methods under UAT attack by averaging E. (i) over all time steps:
RoE(x) = o - 3
Σ (t).
(23)
1=0
34

---

Published as a conference paper at ICLR 2022
Table 19: Calculated corpus-level Recognition of the Essential token of the original prefix-tuning as well as our framework under UAT attack. Result with is statistically significant over that of the standard prefix-tuning baseline with p < 0.001. Result with & is statistically significant over that of our framework with N = 24 with 0.001.
Method
cRoF 1 (%)
std. prefix-tuning
48.05
ours, 24
34.60
ours, N = 3 71.63t1
We then compute the corpus-level Recognition of the Essential token (cRoE) by averaging RoE over all test samples. We calculate the cRoE of the original prefix-tuning as well as our framework with N 24 and N under UAT attack on the SST-2 development set.
The results are listed in Table 19 It can be seen that our framework with N = 3 achieves the highest cRoE score, which is also statistically significant compared with the standard and our framework with N 24. We also find that the cRoF score of our framework with N = 24 lower than that of the baseline. As result, though the cDoD score shows that our framework with N 24 ignores the distraction of the adversarial triggers to some extent, it seems that the robust prefix with = 24 fails to steer the LM to assign the highest importance to the most essential token under UAT attack. In contrast, when the LM is attacked by UAT, the robust prefix-tuning with N = steers the LM so that it not only ignores the distraction from adversarial triggers, but is also (very much likely to be) able assign the highest importance the most essential token as the baseline method does with the input without adversarial trigger tokens.
baseline
The results from Tables T8 [9 provide explanation the performance shown in Figure 3 (a) that under UAT attack, our framework with N = 3 substantially outperforms that with N 24, and both of them achieve higher robustness compared with the standard prefix-tuning baseline. We hope that our quantitative studies can better support the observed behavior and provide a deeper understanding of the robust prefix-tuning framework.
G INTERPRETAIION FROM THE OPTIMAL CONTROL PERSPECTIVE
In this section, provide interpretation from the optimal control perspective for prefix-tuning as well as our robust prefix-tuning framework. Prefix-tuning seeks Po (i, with forward propagation of the j-th layer of the I.-layer LM the output position o as
((+t) = LM (
(,99 11
(24)
with h1 Pocss :] for all j t0 1, E P idx and h:t))
2; for Pidx to optimize
+ Σ R (0o0) 3=0
L-1
(ayy))DDrr S (1(l). vy)
min ACL 1)
(25)
(8
with S as the softmax scoring function, R as the regularizer, + as the LM parameters, 2, as the i-th token in the input and y as the label. According to Li al. (2017). the S and I can be viewed as the terminal and the running loss with Pa the control variables, and the forward and backward propa- gation of prefix-tuning are equivalent to the calculation of co-state process in Pontryagin's Maximum Principle (Kopp) 1962). Therefore, prefix-tuminn -tfiig can be formalized seeking the optimal control of the pretrained models for specific downstream tasks.
Similarly, Our robust prefix-tuning framework seeks Pd [i, with n1(5) Paus [i.:] + Paran [i:]] for all j = to L 1, Piax in the forward propagation of Eq. (24). Our goal is given by
1-1
min
E(x.y))D Dea Σ (1))), Q61)
(26)
(1
1=0
where the running loss l is given by Eq. (8) for j S N and set to 0 for j 2 N. Compared with Eq. (25). there is terminal loss in Eq. (66) the label is unknown during test phase. As
35

---

Published as a conference paper at ICLR 2022
suggested by Chen al. (2021), our robust prefix-tuning can be formalized as seeking the close-loop control for robust downstream tasks.
Before using optimal control theory prove the above formalizations, first provide a brief intro- duction to Li et al. (017)) that reveal the relationship between the optimal control theory and deep learning. We borrow the theorems in Section 4 of of et al. (2017) and directly follow their notations:
Theorem G.1. (discrete-time PMP) Consider the discrete-time control problem
min 100.....r--1)8
(xT)) - 8 _ ((0),,
(27)
E++1 It. + (x 02), 4o 0SIST-1
where & is the termination loss and L is the running loss. Then there exists co-process
_i-1 gt (0( 0,);
Fo x;
(28)
Př = V2Ho(ci.Piiu
PT+1 -V.D(1T+)
(29)
such that
Here He(xt,pi+11 01) 2 (z 01) and
0) 0 100,05115 -
(30)
Hz(x,p,0) = (2. 5L(()
(31)
the scaled discrete Hamiltonian.
Theorem G.2. (discrete-time MSA) The co-process in Theorem G.1 can be determined by applying the discrete-time method of successive approximations (MSA). For each iteration
set z = x, and
x = ge(xx; (e(xt.00 07)
(32)
with enumerating from 0 to 1;
then set p = -V (( and
p = V (el
(33)
with enumerating from T - 0:
finally, with t enumerating from 0 to T - 1, set
0.+1 = = ² + 17 VV pi+ 07).
(34)
Theorem G.3. (equivalence between MSA and back-propagation) The MSA in Theorem G.2
equivalent to gradient descent with pac-prroaggation
The proofs of Theorems G. and provided in Li et al. (201))
We now investigate the forward propagation of prefix-tuning. In Eq. 24 LMQ), the j-th layer of the LM, be decomposed into self-attention layer (SAN!)) and FFN layer (FFNC)). Formally,
h(+T) = ((.9) + SAN!
()) (hO), nn) + +FFN! (A!) + SAN! (AO) 2
(35)
with 1160) Pais [t:) for Pidx- As Eq. 335) is recursive and according to the fact that the pretrained LM is autoregressive, after unrolling the recursion for all (.) in Eq. 35 h (+1)
can be
represented in the form of
kg+1) = + g6 (10),06))
(36)
Now we take Eq. (2 the objective of now formulated as
into consideration. The optimization problem
L-1
S (1(),,))
+ Σ R (0L.)) -=
min (G 1)
B(
(37)
(++1) = 1( + c(1)) 001))00))) 11000 = 2. = [ANS], 0 L-1
36

---

Published as a conference paper at ICLR 2022
know that the objective of prefix-tuning be formulated discrete- time control problem. We then use the MSA described in Theorem G.2 to determine the co-process that solves the control problem and realize that the MSA is equivalent Theorem G.3 which recovers the training phase of prefix-tuning. As a result, we can conclude that prefix-tuning seeks the optimal control of the pretrained models for specifie downstream tasks.
Using Theorem G.1
pack-propggttion with
Our robust prefix-tuning framework can also be formalized as the close-loop control the obtained prefix for robust downstream tasks. According to Chen et al. 2021) the close-loop control frame- work for robust neural networks is defined as
L- Σ (ar (e(2)))
min (A
(38)
F++1 fa (x)))) x. 00,00. 0515
where (12,, ((re)) is the running loss. According to Eq. (6). the goal of our robust prefix-tuning framework can be formulated into the form
Σ R Rog (M²), 2/6 (100,)0000 (M!))),
E (a Die 1000 1111-111
min
(39)
h(++1) -100 + 4.05))
(16), (ili) (n!")) 10ah = 20 [ANS], 0 O<i
Since Theorems G.I G.2 and G.3 can also be applied to Eq.
39 the optimization of our robust prefix is equivalent to seeking the close-loop control for robust downstream tasks.
In conclusion, recent years have witnessed the paradigm shift in NLP from adapting LMs to down- stream tasks to adapting downstream tasks LMs (Liu et al. 2021). In this work, we provide the interpretation of prefix-tuning as well as our robust prefix-tuning framework from the optimal con- trol perspective. We hope that work bring insight to the future design of robust and efficient finetuning approaches for large-scale pretrained models.
37


Reviews:
[{"review_id": "rgWJqmWP_pt", "reviewer": "Reviewer_GPh4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The problem being addressed is cutting edge (we are barely understanding prompt tuning, and this paper already jumps ahead to adversarial defenses!)\n- The approach seems novel, though I haven't searched for related literature carefully. \n- Clear logic motivating the problem and what constraints need to be accounted for while solving it.\n- Clear research question.\n\nWeaknesses\n- The experiments are OK but could be a bit more streamlined. I guess two things came up when I was looking at them. \n\t- It may be good to expand the scope of the paper to generation tasks, as these are likely more suceptible to adversarial attacks. What I mean by this is that the worse case scenario is much worse for generation tasks: while in binary classification the worse case is bad accuracy, for adversarial attacks on generation tasks, potentially very harmful text could be generated, which is much worse than simply getting the answer wrong.\n\t- The other thing is that it is hard to contextualize how good the numbers in your framework are. For example, I have no idea how good 52% on VIPR for SST2 is. If previously proposed defenses are able to get 90% on the same task, then it is unlikely that the proposed method will gain traction, even if it is more computationally efficient with prefix tuning. So it would strengthen this paper a lot to have this comparison.\n\nOther comments\n- I think the paper may benefit from spending just a little more time describing how susceptible prefix tuning is to adversarial attacks compared with regular finetuning. It seems like it is very susceptible, especially for easy tasks like text classification.\n- Appreciate the candidness in Figure 2 in showing that adversarial training takes longer. \n- It will be good to see if the proposed method not only improves performance against adversarial attacks, but also if it improves performance against different paraphrases of expressing the context and question, for example.\n\nMinor points\n- The grammar of the paper will benefit from having a native english speaker proof-read it. E.g., \"as well as remaining the pretrained models unmodified\" can be re-written as \"without modifying the pretrained model parameters\".\n- Since a lot of work was put into the paper, I will pay my respect and make a picky point: the bibliography could be cleaned up a bit. E.g., capitalize RoBERTa correctly, add URLs consistently, capitalize all conference names (e.g., IEEE transactions).", "summary_of_the_paper": "\nThe paper is a focused contribution at the intersection of defending against text attacks and prompt tuning.\nThe paper requires the reader to understand the context and motivation of several different things before understanding the contribution of the paper. First, adversarial examples can attack a text classifier, such as UAT. Second, various techniques defend against these attacks in different way. However, these techniques requiring modifying the parameters of the LM or other additional computational burdens. These techniques can be used with prompt tuning, but then the benefits of prompt tuning go away. Hence, there ought to be a technique that improves the robustness of prompt tuning without removing its benefits over regular finetuning.\nThe paper proposes such a technique and do experiments for three text classification tasks and various adversarial attacks.", "summary_of_the_review": "\nThe paper studies the very timely topic of adversarial attacks against prefix tuning. The paper proposes a method that maintains the advantages of prefix-tuning against finetuning, a formidible problem. The experiments are OK as of now, though expanding the scope to generation tasks could make the impact substantially larger. The characterization of the method could also be further improved by providing finetuning (with and without defenses) as baselines. Overall, I lean towards acceptance, though I am not an expert in adversarial attacks/defenses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OKSiUC7jBSY", "reviewer": "Reviewer_iz2s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors study a novel problem in lightweight fine-tuning methods. Most studies aim to match the performance of full model tuning via updating a subset of parameters but rarely study the robustness of lightweight fine-tuning methods. \n\n\nStrengths:\n1. The authors study an important and novel problem.\n2. The authors provide a simple yet effective method with motivations. The proposed method is shown to be effective even combining with adversarial methods. \n\nWeakness:\n1. The authors argue that robustness is important for lightweight tuning methods. But I still think it is better to provide a comparison between lightweight tuning methods will full tuning methods. Provide a basic starting observation about whether lightweight methods bring more challenges on robustness or not/\n\n", "summary_of_the_paper": "The paper investigates the robustness of prefix-tuning methods and proposes a simple yet effective method to improve the robustness. The experiments show that the proposed method can largely improve the performance in adversarial settings and slightly improve the performance in clean settings.  ", "summary_of_the_review": "The studied problem is important and novel. The proposed method is simple and clear. The experiments justify the effectiveness of the proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LyvgWTLHL_e", "reviewer": "Reviewer_rkLx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Review note: I am not familiar enough with Optimal Control Theory to evaluate the soundness of Section 6 / Appendix E. I will leave it outside the scope of my review and hope other reviewers can fill in the gap.\n\nPros:\n- The results are quite convincing in nearly all settings (aside from important caveats in cons). While some scores are still quite low after RPT, they are consistently better than the baselines. Notably, the method can provide additional gains when used with other defense methods, such as adversarial data augmentation and adversarial training.\n- The need for this method is well-motivated\n- The paper is good at emphasizing different priorities rather than only the end accuracy. For instance, time to accuracy in Fig. 2\n- Although there are some questions about the validity of using attention weights (see https://arxiv.org/abs/1902.10186 which likely should be mentioned as a caveat), I found Section 5 insightful.\n\nCons:\n- Inference Batch, batch size and its importance:\nThe batch size at inference seems like an important variable. Indeed, my understanding of Section 3 and Eq. 8 is that the specific batches used at inference will play an important role as they impact P_psi. This is my **key concern with this paper** as it brings up quite a few issues that should be mentioned.\n  - Inference for a datapoint depends also on other datapoints in the batch. This is different enough from other ML setups that it should be highlighted. It also causes reproducibility issues. For instance, batch norm avoids this by fixing batch statistics at inference.\n  - It is not clear that the method works for low inference batch size (opt of Eq. 8). The inference batch size used is not mentioned anywhere. If the method requires batch size > some N (or if performance varies widely with batch size), this seems a strong assumption that should be made clear. You do not always get several samples of the attack on your system.\n  - It is not clear how well the method works when the inference batches are a mix of unperturbed samples and perturbed ones. This seems like a more realistic attack scenario. \n  - It is not clear how well the method works if there are different attacks in the inference batch, which also seems like a more realistic threat model.  \n\nOverall, I feel like some answers to the above would diminish my concerns, most notably:\n  - What is the inference batch size used in the experiments?\n  - How does performance vary with inference batch size (one or two settings should be enough)\n  - Does the method work when only x of N samples in the test batch are adversarial? Does the example work when they are two types of attack \n\nDataset statistics and mode prediction\n\nIt would be helpful to remind the reader for each dataset how many classes they are / what the mode/random pred accuracy is to help interpret results. For instance, on Table 1 under PWWS improves from 16.64 to 50 for SST-2 and 25 to 34 on SNLI but that is the same as the accuracy of a random predictor.\n\nWriting:\n\nThe writing could be improved substantially. Some examples: p2 “remaining the pretrained models unmodified” -> “keeping”. Also on p3 “remaining its lightweightness”. I suggest doing another pass as it does not align with the quality of the rest of the paper.\n\nRelated work:\n\nI feel like other methods for Parameter efficient transfer learning, such as Adapters, normal prompting should be quickly mentioned. Same goes for comparable approaches such as P-tuning. \nAs mentioned earlier, I would caveat Section 5 with discussions of the validity of using attention for explanation, such as the Attention is Not Explanation paper.\n\nEdit:\n\nGiven the author's response, I am raising my score slightly. There are still some concerns over the threat model but some of my questions on the impact of test batch size have been answered.\n\n", "summary_of_the_paper": "\nThis paper introduces a tweak to Prefix-Tuning to make it more resilient to adversarial perturbations of the input. The idea is to add a batch-level prefix at inference to the original one which enhances robustness. Critically, Robust Prefix-Tuning (RPT) does not require auxiliary model updates or storage, in contrast with other robustness methods.Thus, this approach makes prefix-tuning more robust while preserving its modularity and low storage requirements.\n\nThe authors conduct experiments on 3 text classification tasks, 5 textual attacks and different training regimes (normal training, adversarial training and adversarial data augmentation). In nearly all instances, their method improves robustness (sometimes considerably so) while preserving the accuracy on the original text.  \n\nThe authors also present RPT from an optimal control perspective and conduct a qualitative study that shows how RPT impacts attention weights.\n", "summary_of_the_review": "The motivation, experimental settings and results look good overall.\n\nOne major caveat however is that it is not clear how flexible the inference setup is. This is critical as the predictions for a datapoint at inference depend on the other datapoints in the batch. Currently, it seems like the inference is being done with a batch of all the same attack. These assumptions are simply not realistic as a threat model. Without understanding how this performs under a more realistic threat model, I cannot score this paper higher. I have highlighted experiments that would provide more realistic results. \nDespite my \"marginally below\" score, I do not think the paper can be accepted without an answer on this point. Reject is too harsh for a paper that is otherwise promising.  \n\nThe paper would also benefit from another writing pass, mentioning missing related work and clarifying some properties of the dataset. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hnj0gLq1HeX", "reviewer": "Reviewer_yKwP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below are the detailed strengths and weaknesses:\n\n*Strengths:*\n\n1. Adversarial robustness is an important problem and has not been explored much for relatively new prefix/prompt tuning approaches. Thus the topic of this paper can be of interest to a general audience.  Also, this paper is timely given the recent attention on prompts.  \n2. The idea of optimizing the geometry similarity to defend against attacks is interesting and novel from my perspective. Particularly I like test-time tuning which could adapt to different types of attacks on the fly.\n3. The experimental results are strong. \n\n\n\n~~~~\nUpdates after Rebuttal: Most of the following concerns have been addressed in the revision, and I have increased my score\n~~~~\n\n*Weaknesses*:\n\n1. What is the batch size at test time tuning? Is the added robust prefix the same for the entire test set, or the same within a batch but different across batches, or unique for every test example? This is an important point to assess whether the experiments are in an online setting where test data arrives in-stream or not.   \n2. Section 5 is not very convincing to me:   \n(1) there are only several case studies without any quantitative results; I think it may appear in the appendix only or just uses a short paragraph in the main body (because the statements in Section 5 can only be good hypotheses which are not well-supported by quantitative evidence, also, attention is not a convincing proxy for the explanation, see point (2)), yet it takes more than one page in the current version;  \n(2) the interpretation from the perspective of attention weights bothers me a bit, attention as an explanatory tool is known to not be faithful [1] – a larger attention weight does not necessarily mean the final prediction depends on it more than others, and vice versa, thus maybe not overread it too much. As said above, it is ok to do it this way in the appendix, but using over one page of the main content for this is not convincing to me.  \n3. In the attention visualization figures (e.g. Figure 4), why does a word attend itself as well instead of only attending to the contexts? For LM, I think that the diagonals in the attention figure should be zero, or am I missing something? Also, the presentation could be improved here, in the figure caption you can explain what the rows and columns mean in the visualization to make it easier to read.   \n4. Section 6 is difficult to follow without reading the appendix, and it is disconnected from the rest of the paper in terms of the paper structure – having a theoretical interpretation section after experiments at the end of the paper is not a good presentation structure in my view. If you think Section 6 is important to have, move it before the experiment section and clarify more details to make it more self-contained; if it is not very important, you can just put it into the appendix while briefly mentioning it in the main body.   \n5. Besides the presentation issues above, there are some other minor presentation places which could be improved, e.g.  \n(1) Eq 4, an undefined variable X_C suddenly comes in without explanation  \n(2) Better to explain Eq (8) with more text given that this is an important equation for the proposed method\n\n[1] Serrano et al. Is Attention Interpretable? ACL 2019\n\n\n", "summary_of_the_paper": "This paper focuses on improving the adversarial robustness of prefix tuning (Li et al. 2021), which is a recent parameter-efficient tuning method. Specifically, the paper proposes to add extra batch-level prefixes that are tuned for each test batch on the fly, to minimize the distance between hidden activations of the test samples and the canonical manifold obtained from the hidden activations from correctly classified training samples. The intuition is to optimize the added batch-level prefixes so that the geometry of hidden states from adversarial examples is closer to that of training examples. Experiments on three text classification benchmarks across several different adversarial attacks demonstrate the effectiveness of the method.", "summary_of_the_review": "There are novel contributions of the paper both technically and empirically, however some major analysis sections are not convincing and a significant portion of the presentation needs to be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles a relatively novel problem that is the result of recent work on prefix tuning - specifically the need to be robust to adversarial perturbation in the context of prefix tuning and they show a method for achieving this without requiring more storage and obtain good results.

There were some clarity issues that were addressed by the reviewers during the rebuttal. The main issue that was pointed out was the effect of batch size on the success of the model. The authors gave experiments with batch size 1 where results are less impressive but still outperform the baseline. Also the authors say that for now they are not considering the case where only some of the elements in the batch are adversarial, which I think is ok for a research paper on such a cutting-edge topic. 

Thus, the result of the discussion is to lean to accept this paper given that it is now more clear, has experiments that make it clear what the benefits are in realistic settings and obtains improvements.
Decision: Accept (Poster)