PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer
Authors: Lichang Chen
Jiuhai Chen
Heng Huang
Minhao Cheng
Venue: EMNLP 2023 Main (2023)
Keywords: prompt tuning; training with regularizers
TL;DR: A method to stabilize and improve the prompt tuning.

Abstract: Recent studies show that prompt tuning can better leverage the power of large language models than fine-tuning on downstream natural language understanding tasks. However, the existing prompt tuning methods have training instability issues, as the variance of scores under different random seeds is quite large. To address this critical problem, we first investigate and find that the loss landscape of vanilla prompt tuning is precipitous when it is visualized, where a slight change of input data can cause a big fluctuation in the loss landscape. This is an essential factor that leads to the instability of prompt tuning. Based on this observation, we introduce perturbation-based regularizers, which can smooth the loss landscape, into prompt tuning. We propose a new algorithm, called Prompt Tuning with Perturbation-based regularizer~(PTP), which can not only alleviate training instability dramatically but also boost the performance of prompt tuning. We design two kinds of perturbation-based regularizers, including random-noise-based and adversarial-based. In particular, our proposed perturbations are flexible on both text space and embedding space. Extensive experiments show the effectiveness of our proposed methods in stabilizing the training. Our new algorithms improve the state-of-the-art prompt tuning methods by 1.94\% and 2.34\% on SuperGLUE and FewGLUE benchmarks, respectively.

РTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer
Lichang Chen
Jiuhai Chen
University of Maryland
University of Maryland
bobchenêcs. umd. edu
jchen 69@umd. edu
Heng Huang
Minhao Cheng HKUST minhaocheng@ust.c edu, hk
University of Maryland heng@umd. edu
Abstract
Diserete Space Perturb to
Recent studies show that prompt tuning bet- ter leverage the power of large language mod- els than fine-tuning on downstream natural lan- guage understanding tasks. Nonetheless, cur- rent prompt tuning methods encounter instabil- ity during training. marked by high variance in scores given different random seeds. In ad- dressing this crucial issue, we uncover that the loss landscape of standard prompt tuning, when visualized, is remarkably steep, i.e., minor al- terations in the imput data can trigger substan- tial fluctuations in the loss landscape, which is an essential factor that leads to the training instability. In light of this finding, we incorpo- rate perturbation- based regularizers to temper the loss landscape within the prompt tuning process. We thus present novel algorithm, called Prompt Tuning with Perturbation-baat regularizer (PTP), that can significantly reduce training instability and concurrently enhance the performance of prompt tuning. Specifically, we design two variants of perturbation- based regularizers: one that employs random noise, and another that uses an adversarial approach. Importantly, our proposed perturbations dis- play flexibility in both the text and embedding spaces. Extensive experiments show the ef- fectiveness of our proposed methods in stabi- lizing the training. Our new algorithms im- prove the statc-of-the-ar prompt tuning meth- ods by 1.94% and 2.34% on SUP and FewGLUE benchmarks, respectively.
[5'1.8"2 s'x))
Input sequence
Continuous Space Perturb to E
Continuous Prompt Embedding Ey
Input embeddings E,
Pre-trained Language Model
Gradient Vg
Figure I: A simple pipeline of our Propmt Tuning with Perturbation- based regularizor (PTP) algorithm.
pre-trained LM and wraps up the input sequence with natural language templates. However, natu- ral language prompts are handcrafted by experts and the performance is not comparable with fine- tuning methods. To tackle this issue, Lester et al. (2021); Li and Liang (2021) proposed prompt tun- ing, which prepends the input sequence with contin- uous embeddings and only tunes these embeddings during training. Liu et al. (2021b, 2022) verified the effectiveness of prompt tuning on natural language understanding (NLU) tasks under both few-shot learning and supervised learning settings, which is comparable the fine-tuning methods but with much fewer (1000x less) lask-specific tunable pa- rameters. However, under different random seeds, we observe that the current prompt tuning methods suffer from high variance of scores, which indi- cates they suffer from training instability issues.
1 Introduction
Releasing the burden of training models from scratch while keeping the outstanding perfor- mance on downstream tasks, pretrained Language (Raf-
To investigate the factor that causes the instabil- ity of prompt tuning, we visualize the loss land- scape of the vanilla prompt tuning and observe that there exist many sharp crests, shown in Figure 2(a), which harms the training stability. Mo- tivated by the recent study (Chen and Hsieh, 2020) which shows that perturbation based regularizers are powerful tools to smooth the loss landscape
Model (LM) brought NLP to fel al., 2020; He al., 2021; Shoeybi al., 2019). Recently, inspired by the success of GPT- 3 (Brown et al., 2020) on few-shot and zero-shot learning with manually created prompts, there has been a surging interest in prompting that freezes

---

200
(4) Original
(a) PTP-RN
(b) PTP-ADV
Figure 2: The loss landscapes on different continuous prompt tuning procedures. The X-axis and Y-axis denote the magnitude of perturbations (gradient direction) and another random orthogonal direction. Z-axis represents the cross-entropy loss of the different training methods.
and stabilize the training of machine learning sys- tems, we introduce them into prompt tuning to ad- dress the lack of stability and generalization issues. To be specific, we propose Prompt Tuning with Perturbation- based regularizer (PTP) algorithm to make the training stable and boost the performance of prompt tuning.
the training instability problem in prompt tun- ing, which can also boost performance. To- gether with the PTP algorithm, we design two types of perturbations as our implicit regu- larizers, which are Random- Noise-bassee per- turbation (PTP-RN) and ADVersarial-basee perturbation (PTP-ADV).
Specifically, we consider two kinds of pertur- bations in РTP. Random-Noise-basee (PTP-RN) perturbation and ADVersarial-based (PTP-ADV) perturbation. PTP-RN is motivated by randomized smoothing (Cohen et al., 2019), which applies the neighborhood averaging and makes the neural net- work smoother while PTP-ADV motivated by adversarial training. a method proposed to make the predictor capable resisting the adversarial ex- amples (Goodfellow et al., 2015) as well as boost- ing the clean accuracy of the predictors (Xie et al., 2020; Zhu et al., 2020). Moreover, in order to bring more flexibility and make our exploration more comprehensive, both text (discrete) and embedding (continuous) space, depicted in Figure 1.
Moreover, depicted in Figure 1, pro- posed PTP-ADV and PTP-RN can be ap- plied to both text space and embedding space, which makes our perturbations more flexible.
We conduct extensive experiments to evaluate the effectiveness of our algorithms on Super- GLUE and FewGLUE benchmarks. The ex- perimental results demonstrate our proposed PTP algorithm can boost the standard perfor- mance of prompt tuning FewGLUE and Su- perGLUE by 2.34% and 1.94%, respectively. It also shows the great power of our algorithm improving training stability.
apply perturbations to
Preliminaries and Related Work
In the experiments, we conduct extensive exper- iments to evaluate our proposed algorithms, PTP- RN and PTP-ADV, on SuperGLUE (Wang et al., 2019) and FewGLUE (Schick and Schütze, 2021b) benchmark. By applying the PTP algorithm text embedding space to the existing prompt tuning methods, we can boost the performance of prompt tuning on SuperGLUU and FewGLUE benchmarks by 1.94% and 2.34% as well as make prompt tun- ing more stable. We also present a comprehensive ablation study and analysis of our algorithms with different perturbations.
Prompt Tuning
Discrete Prompt. Discrete prompt, also known as hard prompt (Liu et al., 2021a; Davison et al., 2019; Jiang et al., 2020; Haviv et al., 2021; Chen et al., 2023; Bao et al., 2023; Liu et al., 2023), is typically template composed of task descriptions and original input texts. Brown et al. (2020) created templates for GPT-3 based on their introspection and make it suitable for various downstream tasks, such machine translation, QA, etc. Utilizing discrete prompts, they can achieve stunning results on NLU tasks under few-shot learning settings. By employing the predefined templates and converting the tasks into cloze questions, Schick and Schütze
Our contributions can be summarized as:
We propose a new PTP algorithm to tackle

---

(2021a,b) showed that even with 'greener' back- bone (Lan et al., 2020), which has 100x fewer parameters than GPT-3, they can also reach pre- vailing results on the few-shot version of Super- GLUE benchmark (Wang et al., 2019) (also known as FewGLUE). Liu et al. (2021b) utilized the exist- ing discrete templates and tuned embeddings of the selected tokens, which achieves SOTA results on FewGLUE. As the few-shot scenario common and useful, in this puper, we also test the few-shot learning ability of our PTP in our experiments.
RTE
BoolQ
WIC
Acc.
87.7 :: 1.81
83.9 + 0.92
72.0 +: 1.38
Var(FT)
0.12
0.09
0.11
Var(PT) 1.45
0.74
1.16
Table 1: Re-imppementation results PT2 (Liu et al.. 2022) with RoBERTa-large backbone SuperGL.UE benchmark. Acc.: mean accuracy. Var: variance score computed by runs with different random seeds. FT: fine-tuning. PT: prompt-tuning.
Besides enhancing the robustness against ad- versarial examples, adversarial training has been shown great power in boosting the standard perfor- mance of image classifier (Xie et al., 2020), visual language representation learning (Gan et al., 2020), and GNN (Kong et al., 2022). In this paper, we also apply similar idea prompt tuning and focus on boosting its performance rather than its adversarial robustness.
Continuous Prompt Tuning. As prompts aim to boost LM's performance, it is not necessary to make tokens of prompts interpretable. Without the limit of tokens being natural words, Li and Liang (2021) proposed to prepend series of tunable embeddings Fr to the input embedding sequence prompt and optimize them with training data, which provides in-context information for LMs to condition on. Lester ct al. (2021) prepended se- quence of special tokens T to the input sequence, and similarly, they tune the embeddings ET (em- beddings of special tokens) on the downstream tasks. Moreover, to further leverage the power of prompt embeddings, Liu et al. (2022) presented PT2, method that adds the trainable prompt em- beddings to every layer pretrained LMs as prefix embeddings. To keep the consistency of the nota- tion, we also apply the same prompt embedding representation Ey to represent trainable prompts in every layer of LMs.
Discrete Space AT. Different from adversarial attacks on images, in NLP attack, due to the dis- crete nature of text space, it typically formu- lated as a combinatorial optimization problem to create adversarial input sequence s', which is clas- sically solved by heuristic search while maintain- ing the semantic similarity of the original input sequence 8 (Li et al., 2020; Ren et al., 2019; Morris et al., 2020). However, the searching algorithms of adversarial attacks, such beam search (Lbrrahimi et al., 2018), greedy search based on word impor- tance (Ren et al.. 2019), and deletion-based search- ing (Jin et al., 2019), are usually slow because of the high computation cost on sentence encoding and large search space (Yoo et al., 2020). Yoo and Qi (2021) proposed A2T algorithm to accel- erate the heuristic search process, which replaces the slow USE encoder (Li al., 2020) with Dis- tIBBERT (Sanh et al., 2019) to calculate the cosine similarity between original input text and perturbed text, and they obtain significant speedup compar- ing to the textfooler (Jin et al., 2019). Thus, in this paper, we adapt the attacking algorithm in (Yoo and Qi, 2021) to generate noises in text space.
2.2 Adversarial Training
Continuous Space AT. Over the past few years, Adversarial Training (AT) has demonstrated im- pressive results in improving model robust- ness (Goodfellow et al., 2015; Tramèr et al., 2018; Athalye et al., 2018). AT can be formulated min-max optimization problem:
min E (ZD [161l
max C(0, 0.
(1)
where the loss function, || llp represents tp- norm distance and e denotes the perturbation bud- get. Madry et al. (2018) proposed PGD algorithm to compute an adversarial example (inner maxi- mization problem) iteratively as:
Proposed Framework
3.1 Preliminary
gt I1 11 Se (6' sign (V,L(0,
Before introducing the proposed algorithms, we first briefly describe the word to embedding process of LMs as well as the final embedding input of the continuous prompt tuning. Given the n word input
(2)
where is the iteration step; II 18 projects per- turbation into e-ball.

---

Algorithm 1: PGD on prompt tuning
Algorithm 2: PTP
Require: Perturbation iteration n and size The bound of perturbation e;
Require: Prompt embeddings Er; input embeddings Es; trainable parameter 0 for prompt encoder P and E,y; Training data D: Pre-trained LM M,; Loss function L;
for epoch , do Egruuree grad - True;
Vi f arg maxy (Pr M (I))):
2 Initialize parameters 0:
5
C(ú, y) backward();
3 for epoch 1, ......... K do
F'g <- Fg a** F.. Fg.ggrd.ssgn())
4
/* standard prompt tuning */
= 118l1 sc(E" Es);
5
Sample minibatch data (s, v) from D;
E f Eg + +d; M.zero gr;
6
e.equires Lerquireesgraddee <- True;
I + [r;
ÿ argmaxy (Pr [y]
10 return Eyg
C(ū, y))bbkwwardd) and update ET
9
1e training with perturhed data */
sequence 8 = [$1:.... sn] and word to embedding function fv, where V denotes embedding matrix, the input embedding E, = [e (s1),...,e (sn)] E Raxd can be obtained by E. fv(s), where d denotes the dimension of the word embedding. Let continuous prompts be ET [e].... e7" Rmxdd the update of Ey be either directly, through reparameterizatioon encoder P like MLP/LSTM, P(Er) [ho...., hmm] E med For simplicity, we still use Er to represent the output of P(ET), and the final input embedding sequence is written as [Er; E.] E IR k)
11
r.qquiress grad <- False;
12
Apply PTP-RN or PTP-ADV to s or Eg;
I' - E3];
13
erequires geauuurresgrddd_de
grad + True;
Us <- arg maxy (Pr [ul (T"));
[( y) backward() and update ET;
smooth. Specifically, we introduce two versions of perturbation based regularizers that can be formu- lated as follows:
The training objective of continuous prompt tun- ing can be formulated as:
ming (s,)) [M + + s.t. PTP-RN: 8 ~
(4)
PTP-ADV: max 8 C 6,
min (( [4 (M (0, 8,y))l
(3)
8
where M denotes LM; 0 represents the trainable parameters of Ey and prompt encoder P while ID is the underlying data distribution.
where N denotes Gaussian distribution. For PTP- RN, minimize 0 under small random perturba- tion, aiming to force the model focus per- turbed pair (s - 8, y) and have a robust prediction within the neighborhood of s. It is related the idea of randomized smoothing (Cohen et al., 2019) 9), which obtains a smoother predictor via randomly averaging the neighborhood of the given function. For PTP-ADV, the perturbation 8 is generated by adversarial attack algorithms such PGD (Madry et al., 2018), A2T (Yoo and Qi. 2021), and the worst-case training loss is minimized under small perturbation bounded by e. The idea motivated by adversarial training, which usually applied as a form of adversarial defense. (Goodfellow et al., 2015; Cheng et al., 2021)
3.2 Proposed Formulation
Although continuous prompt tuning (Liu et al., 2022) could achieve comparable performance with the fine-tuning method by only using 0.1 % to 3% trainable parameters, suffers from unstable train- ing issues: as shown in Table even only changing the random seed in different runs, the final perfor- mance is very unstable. To investigate the issue, we plot the loss landscape of the vanilla prompt tuming as Figure 2 and observe sharp crests in a small local region with small noise, which means small perturbation embedding space would cause significant reduction in the final accuracy. It also known as the training instability prob- lem (Chen and Hsieh, 2020). To address this chal- lenge in prompt tuning, we propose perturbation- based regularizers to force the loss landscape to be
Since it is still unknown which space to inject perturbation ő is better stabilize the training, we apply on both text and embedding space to bring more flexibility and make our exploration more comprehensive. The perturbed sequence s' or the

---

perturbed embedding E' can be obtained as
of [MASK], our model will not predict anything on the tokens we mask and we just use [MASK] token as a perturbation on discrete space. PTP+RM is used to denote PTP-RN on text embedding space.
s' = 8 8, E - E. + 6,
(5)
where 8, Eg denote the input sequence and the input embedding, respectively. It is worth noticing that if the perturbation is on (text space), through fv. the perturbed s' will be converted into input em- bedding. which also denoted as Ey,
3.4 PTP-ADV
3.4.1 Embedding Space (PGD Perturbation)
Different from previous PGD training methods which focus on improving the models' robust- ness, wc aim to smooth the loss landscapes and boost the performance of prompt tuning by adding adversarial- based regularization. Given the em- bedding sequence F.g, PTP-ADV adopts multi-step PGD to generate perturbations on embedding space. The perturbation is computed iteratively as:
The main idea of our proposed formulation is that we force our algorithm to not only learn from the clean data pair (s, y) but also perturbed data pair (s + 6, y) to make the training more smooth.
3.3 PTP-RN
3.3.1
Embedding Space (RG Perturbation)
E E, + 8 S.t. S.t. 8 15 (5t + vVol)
(8)
In embedding space, how create perturbed ex- amples is still an unsolved problem. But since the ultimate effects of PTP-RN are the only thing we care about, not the interpretability, it is feasi- ble for us to add random-noise-based perturbation on word embeddings. Given the embeddings of the input sequence E, [e(so), e(s)),.... e(s,,), }], where e(s;) E Rdd, PTP-RN in embedding space samples ó from Gaussian distribution and randomly selects some embeddings perturb, which make sure. The perturbation can be formulated as:
where d' denotes the t-th iterations of PGD pertur- bation and it will be added to the input embedding sequence E, after all the iterations are finished. Algorithm shows the implementation details of our PGD attack on prompt tuning. PTP+PGD is applied to denote our PTP-ADV algorithm with perturbation on embedding space.
3.4.2 Text Space (A2T Perturbation)
Furthermore, to enhance the flexibility of PTP- ADV and boost model generalization ability, we ap- ply it to the text space: PTP-ADV adopts the attack algorithm in A2T (Yoo and Qi, 2021) to generate its perturbation 6, which is an algorithm composed of NLP attack and adversarial training. Given the input sequence s, the perturbed sequence s', with A2T perturbation, can be represented as:
- E. + 8,t. =
(6)
0}.
where 5 E Raad has the same length as the in- put embeddings; i denotes the number of embed- dings being perturbed and d_=1,..., ~ (0, oll,), with d denoting the dimension of the word embed- ding and controlling the magnitude of perturba- tion. We represent PTP-RN on embedding space as PTP+RG.
s' = {s0, si.....,. - 8n))
(9)
where s denotes the perturbed word. For simplic- ity, also call it PTP+A2T.
3.3.2 Text Space (RM Perturbation)
Algorithm 2 provides the details about the whole training process of PTP algorithm. the standard prompt tuning. the input of LM is composed of prompt embedding Fr and input embedding Eg, which is denoted as / = [Er: E.]. After LM gives prediction of 1, we backpropagate the loss to up- date Fy. In training with perturbed data part (Line 10-16, Algorithm 2), the discrete continuous space perturbations of input data are generated by PTP-RN PTP-ADV firstly. Then the perturbed input is employed to conduct training with original label y, which also plays a data data-augmaan aattaa-atmmttitonn role to boost the performance of the prompt tuning.
In text space, similarly, our goal is to create label- preserving and perturbed input data to augment the training data and make the training stable. Given input sequence s, PTP-RN randomly selects some tokens and converts them into [MASK]. The perturbed sequence s' can be formulated as:
= (Su: [MASK]..... .... [MASK]. },
(7)
where we perturb i tokens. It should be noticed that unlike BERT pretraining process (Vaswani
et al., 2017), where the model predicts the label

---

Method
BoolQ
CB
WiC
RTE RoBERTa
BERT
RoBERTa
BERT
RoBERTa
BERT
RoBERTa
BHRT
FT
77.7
86,9
94.6
98.2
74.9
75.6
70.4
86.6
PT2
75.8
84.8
94.6
100
75.1
73.4
78.3
89.5
PT2+Span-Off Off
77.2
85,9
95.3
100
75.6
74.3
79.0
90.9
PTP+A2T
76.4
85.7
94.5
99.6
75.8
72.9
78.6
89.7
PTP+RG
77.3
86.2
95.8
99.8
76.711.6
75.5
79.9
90.6
PTP+RM
77.4
85,9
95.7
100
76.4
75.2
79,6
91.6
PTP+PGD
78.3
86.711.
96.111.5
10010
76.6
75.7
80,3
92.012.5
Method
MultRCCF1a)
ReCoRD
WSC
BERT
RoBERTa
BERT
RoBERTa
BERT
RoBERTa
BERT
RoBERTa
FT
69.0
94.0
70.5
85.7
70.6
89.0
68.3
63.5
PT2
73.0
93.0
70.6
82.5
72.8
89.3
68.3
63.5
PT2+Span-Orf
73.8
93.6
71.7
82.9
73.5
90.2
69.0
63.9
PTP+A2T
73.3
93.2
71.4
82.6
73.6
89.7
68.5
63.8
PTP+RG
75.1
93.9
72.6
84,9
74.9
90.5
69.4
65.0
PTP+RM
74.6
93.8
72.9
84.4
74.8
90.6
69.2
64.8
PTP+PGD
74.7
94.101.1
73.4
84.6
75.1
91.9116
69.7+1.4
65.0
Table 2: Results of proposed PTP algorithm in fully-supervieed learning settings. We employ the large-size version of BERT and RoBERTa models (BERT-Larg size: 335M and RoBERTa-larg size: 355M, respectively), We use bold font to mark the best and red suhseript to mark the improvement compared to PT2. PT2+Span-Off: strong data-an tata-augmentation method proposed by Shen et al. (2020).
(Dev 32)
BoolQ (Acc.)
RTE
MultiRC
WSC
COPA
Method
(F1)
(Acc.)
(Acc.)
(EM)
(Fla)
(Acc.)
(Acc.)
PET Best
75.1
52.6
65.7
35.2
75.0
80.4
83.3
PT
77.8
92.3
56.3
76.5
36.1
75.0
84.6
87.0
PT+Span-Off
79.0
92.5
57.1
77.8
36.7
76.9
85.2
87.8
PTP+RM
79.9
93.2
58.1
78.612.1
36.2
78.3
85.9
88.6
PTP+RG
79.5
93.711.4
58.0
77.7
36.6
78.1
85.4
88.3
PTP+A2T
78.6
92.6
56.6
77.4
36.5
76.0
84.7
87.7
PTP+PGD
80.2 12.4
93.5
58.5 12.2
78.5
37.4+1.
78.9
86.011.4
88.9119
PET(Full Dev)
79.4
59.4
52.4
69.8
37.9
77.3
80.1
95.0
iPET(Full Dev)
80.6
92.4
52.2
74.0
33.0
74.0
Table Results our PTP algorithm Few-shot learning(32 training examples) settings. PT: P-tuning (Liuu et al., 2021b) and the backbone LM is alberta-xxl-v2. We use bold font mark the best. The red subscript denotes the increase of our method compared with the baseline method PT. Dev 32: development set contains 32 unused examples from the training set, same (Liu al., 2021b). Full Dev: original development set.
4 Experiments
guage understanding (NLU) tasks. We also include the few-shot version of SuperGLUE, FewGLUE benchmark (Schick and Schütze, 2021b) to test the ability of our algorithm, which consists of 32 training samples in each dataset on SuperGLUE. Following (Schick and Schüize, 2021b; Liu et al., 2021b), we report results on of 8 NLU tasks few-shot settings.
We conducted empirical studies on two popu- lar natural language understanding (NLU) bench- marks: SuperGLUE benchmark (Wang et al., 2019) and FewGLUE benchmark (Schick and Schütze, 2021b). We tested the proposed framework in both fully-supervised and few-shot settings to verify the effectiveness of our proposed PTP-RN and PTP- ADV algorithm with perturbations on both and embedding space. To further verify the effective- ness of our algorithm, we also conduct experiments on text summarization tasks (Narayan et al., 2018)) (See Appendix $A.)
Fully & Few-shot Setting. In fully-supervised setting, the full training set of each task in Super- GLUE (Wang et al., 2019) is employed during the prompt tuning process. Besides, in the model lection part, we adopt the whole validation set. As few-shot learning ability of prompt tuning can duce the cost of annotations in real-world applica- tions, following (Schick and Schütze, 2021b; Liu et al., 202 we also test our algorithm under few-
4.1 Experimental Setup
NLU Dataset. SuperGLUE benchmark (Wang et al., 2019) contains 8 challenging natural lan-

---

shot settings. To be specific, we use the training set provided by FewGLUE (Schick and Schütze, 2021b), the few-shot version of SuperGLUE, con- taining 32 training pairs in each task. Besides, we use the same version of the development set as (Liu et al., 2021b) to select models, which are created by randomly choosing 32 unused training pairs.
and A2T, which indicates perturbing on contin- uous space is more effective than perturbing on discrete space fully-supervised settings. As for pretrained LMs (BERT-large and RoBERTa-large), results show the superb learning ability of our PTP algorithm regardless of which backbone.
4.3 Results on Few- Setting
Baseline Methods. We incorporate two popular prompt tuning methods as baselines: P-tuning (Liu et al., 2021b) (PT) and P-tuning-v2 (Liu et al.. 2022) (PT2), where PT is the state-of-the-an method in FewGLUE benchmark while PT2 also achieves excellent performance in SuperGLUUE benchmark. Moreover, we also include the strong data augmentation method proposed by Shen et al. (2020) and select the Span-Off, which is the best approach among the multiple augmentation ap- proaches in their paper, for reproducing their per- formance on the PT and PT2. The details of hy- perparameters. such as learning rate and prompt length, are deferred to Appendix B.
In few-shot learning settings, we employ FewGLUE, also known as the few-shot version of SuperGLUE. PET (Schick and Schütze, 2021b) and iPET (Schick and Schütze, 2021a) are the methods using discrete prompts. We test the previous SOTA method on FewGLUE, PT (Liu et al., 2021b) as our baseline method and validate on the same development set (Dev 32). As illustrated in (Liu et al., 2021b), for a fair comparison, the results of PET Besst (Dev 32) are reported removing all the additional tricks like ensemble, distillation, etc. PET (Full Dev) and iPET (Full Dev) denote the methods with the original validation set.
Our main results are shown in Table PTP achieves better results than the previous state-of- the-art method PT in all 7 tasks, which verifies the effectiveness of our algorithms in few-shot NLU tasks. Especially, PTP+PGD outperforms the pre- vious PT by 2.34% average. Comparing the P'T++GGD (Dev 32) to the methods with the orig- inal dev set, it still has an advantage on most of the tasks (5 of 7) while the results are similar in BoolQ (better than PET with full dev set but worse than iPET). The PTP with RM and RG perturba- tion method also achieves remarkable improvement when compared to the baseline method PT. More- over, PTP with A2T perturbation can also boost the performance of the baseline by small margin.
Following the settings in (Liu et al., 2022, 2021b), we include BERT-large (De- vlin et al., 2019) and RoBERTa-large (Liu et al., 2019) for sully-supervised settings and ALBERTA- xxlarge-v2 v2 (Lan al., 2020) for few-shot settings. To have a fair comparison with the baseline meth- ods, for fully-supervised settings, all backhone LMs are frozen, except in fine-tuning, same as (Liu et al., 2022). For fhw-shot learning settings, back- bone LMs are tuned with trainable prompt embed- dings, same as (Liu et al., 2021b).
Pretrained LMs.
4.2 Results on Fully-supervised Setting
In fully-supervised settings, Table demon- strates the results of our proposed PTP algo- rithm with 4 different perturbations on all tasks of SuperGLUE benchmark. It is worth noticing that PTP+PGD achieves the best perfor- mance in almost all datasets except WiC (BERT), COPA(BERT), and MultiRC (RoBERTa). Over- all, the best method PTP+PGD outperforms the baseline method PT2 by 1.94% (with BaRT-large backbone) and 1.63% (with RoBERTa-lagge back- bone) on average. Compared to the Span-Off (Shen et al., 2020), a strong data augmentation method, our algorithm also exhibits its advantage.
4.4 Results on Improving Training Stability In addition improved generalization perfor- mance, the proposed method could stabilize the training process. Figure 2 provides strong evidence that our proposed PTP-RN and PTP-ADV train- ing methods have a much smoother loss landscape compared to the vanilla prompt tuning. For the few-shot learning setting, we demonstrate the re- sults in Table 7. It could be seen that all our PTP methods have smaller variances than the baseline method. Specifically, PTP+PGD has the smallest variance in runs, which indicates its training and generalization stability. Compared with the PTP- RN methods (RG, RM), PTP-ADV methods (A2T,
Text vs. Embedding Perturbation. PTP with PGD and RG perturbation on continuous space (em- bedding space) perform better than PTP with RM

---

5. Under fully-supervised learning settings, we present the the results of COPA dataset in Table 9. shows that large a PGD will detrimental to the performance. Under few-shot learning settings, Figure demonstrates the results of different a and different iterations of PTP+PGD on few-shot set- tings. In all 3 datasets, when a is 10-3, not too small nor too large, and PGD iters is 4, PTP+PGD can achieve outstanding performance.
8000
CDP4
MSC
W:C
CB
ReCoRo
1.4
P12
1.2
+RM
-.00
+PDD
0.
RM Perturbation. We investigate how different numbers of random [MASK] inserted affect the РTP. Formally, the number of [MASK] inserted is defined as i in Eq. (7). We conduct experiments with i from 1 to 10. Under few-shot learning set- tings, Figure 6 presents the results of PTP with RM perturbation on FewGLUE (MultiRC and RTE dataset). We observe that RM perturbation can boost the performance substantially settings and the best choice of is 8. Under fully- supervised settings, Table 10 presents the ablation of RM perturbation. It also shows that large num- ber of [MASK] inserted harms the performance, especially when the backbone is RoBERTa.
Figure 3: The variance of the scores on the dev sets from SuperGLUE Benchmark. We compute on runs with different random seeds report. The reported experi- ments are all using BERT-large models as backbones.
PGD) achieve smaller variance. We also conduct the experiments under fully-supervised learning set- tings in Figure 3. It shows that in all 8 tasks from SuperGLUU benchmark, our proposed method can still reduce the training variance of difffrent runs with the same hyperparameter except the seeds.
few-shot
5 Ablation Study
A2T Perturbation. We investigate how the mini- mum cosine similarity between normal input and perturbed input s' of A2T perturbation affects the results. We run experiments with minimum cosine similarity from (0.2, 0.4, 0.6, 0.8) ] and show results in Table 11. It indicates small similarity may cause damage to the standard performance because the perturbation can be too large in this case.
In this section, we conduct comprehensive ablation studies of our perturbation methods under both sully-supervised and few-shot learning settings.
RG Perturbation. We investigate the strength of Gaussian noise 5 and number of perturbed em- beddings i affects the performance (see Eq. (6)). Specifically, we run experiments with o which is the variance of the added Gaussian noise, from (10-4, 10-3, 10-2) and the number of word em- beddings perturbed, which is denoted as i in Eq. (6), from (1, 5, 10, 20). Under the sully-supervised learning setting, report the results COPA, RTE, and WiC tasks in Figure 4. The results show the appropriate number of perturbed embeddings to large a and large i, PTP+RG more likely to fail in comparison to the baseline method. Under few- shot learning settings, we select results in MultiRC task to report, as shown in Table 8. The encour- aging result also demonstrates that the best choice of and number of embeddings perturbed is 10 and 5, respectively.
Conclusion
In this paper, we first investigated the training insta- bility issues on prompt tuning, which has a precipi- tous loss landscape in its visualization. To tackle the problem, we proposed РTP-RN and PTP-ADV algorithms, which include four different perturba- tions (RG, RM, ADV, A2T) on both discrete and continuous spaces, to smooth the loss landscape and make the training stable. Furthermore, algorithms are also capable of boosting the per- formance of prompt tuning. The extensive experi- ments validate the effectiveness of our proposed al- gorithms on NLU tasks under both fully-supervised and few-shot settings.
supposed be 10-2 and the With
PGD Perturbation. We investigate how differ- ent a (See Eq. (8)) and PGD iterations affect the performance. We run experiments with a from (10 4 10-3 10 2) and PGD iterations from 1 to
7 Limitation
In this paper, we introduce the PTP algorithm, which enhances the stability and performance of

---

prompt tuning. It should be noted, however, that our approach exclusively relevant to continuous prompt tuning, and cannot be used to improve other models like GPT-3 or ChatGPT, which are only ac- cessible via their APIs. Another limitation of our algorithm is that we do not provide theoretical analysis.
of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, FAAT 2021, Virtual Event, February 2-9, 2021. pages 7107-7115. AAAT Press.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044.
8 Acknowledgement
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning. volume of Pro- ceedings of Machine Learning Research, pages 1310- 1320. PMLR.
LC Chen and H Huang were partially supported by NSF IIS 1838627, 1837956, 1956002, 2292, CNS 2213701, CCF 2217003, DBI 2225775.
References
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge In Machine learning challenges workshop, pages 177-190. Springer.
Anish Athalye, Nicholas Carlini, and David A. Wag- ner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Muchine Learning, ICMI. 2018, Stock- holmsmässann, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pages 274-283. PMLR.
Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, E.MNIP-IJCNI.P 2019, Hong Kong, China, November 3-7, 2079, pages 1173-1178. Association for Computational Linguis- tics.
Wentao Bao, Lichang Chen, Heng Huang. and Kong. language-informed informed distribution for
2023. Prompting language- compositional zero-shot learning. arXiv preprint arXiv:2305. 14428.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Hoss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box box adversarial exam- ples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2023. Instructzero: Ef- ficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.0m 03082.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjin Liu. 2020. Large-scale adver- sarial training for vision-and- -annguua- representation learning. In Advances in Neural Information Process- ing Systems, volume 33, pages 661 6-6628. Curran Associates, Inc.
Xiangning Chen and Cho-Jui Hsieh. 2020. Stabilizing differentiable architecture search via perturbation- based regularization. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1554-1565. PMLR.
Ian J. Goodfellow. Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adver- sarial examples. In 3rd International Conference on Learning Representations, ICLR 2015.
Minhao Cheng. Pin-Ye Yu Chen, Sijia Liu, Shiyu Chang. Cho-Jui Hsieh, and Payel Das. 2021. Self- progressing robust training. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirly-Third Conference on Innovative Applications
Andrew Gordon, Zomitsa Kozareva, and Melissa Roem- mele. 2012. Semeval-2012 task Choice of plau- sible alternatives: An evaluation of commonsense

---

causal reasoning. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evalua- fion (SemEval 2012), pages 394-398.
Hector Levesque, Ernest Davis, and Leora Morgenstem. 2012. The winograd schema challenge. In Thir- teenth international conference the principles of knowledge representation and reasoning.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising seneence-t--eqqeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.1344
Adi Haviv, Jonathan Berant, and Amir Globerson. 2021. Bertese: Learning to speak to BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, pages 3618-3623, Association for Computational Linguistics.
Linyang Li, Ruotian Ma. Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversar- ial attack against BERT using BERT. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193 6202, Online. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhannce bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACI/JJNNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, pages 4582- 4597. Association for Computational Linguistics.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423 438.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is BERT really robust? natural language attack text classification and entailment. CoRR, abs/1907.1192.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang. Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hal- lucination large instruction tuning.
models via robust
Daniel Khashabi, Snigdha Chaturvedi. Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pa- pers). pages 252-262.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107. 13.666.6...
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61 61-68. Dublin, Ireland. Association for Computational Lin- guistics.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. 2022. Robust optimization as data aug- mentation for large-scale graphs. In IEEF/CVF Con- ference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 60-69. IEEE
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30. 2020. OpenReview.net.
Xiao Liu, Yanan Zheng, Zhengxia Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 202 lb. GPT understands, too. CoRR, abs/2103.10333
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, 1tt/19977.1192
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net,

---

John Morris, Jin Yong Yoo, and Yanjun Qi. 2020. Tex- tAttack: Lessons learned in designing python frame- works for NLP. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 126-131. Association for Computational Linguistics.
Florian Tramer, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick D. McDaniel. 2018. Ensemble adversarial training: Attacks and de- fenses. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. ArXiv, abs/1808 888.08748..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CORR, abs/1706.03762.
Mohammad Taher Pilehvar and Jose Camacho- Cooaho--o 2018. Wic: the word-in-con in--cotteetttt dataset for evaluat- ing context-sensitive meaning representations. arXiv preprint 1rXXvv:800.001222
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general penerae-purpose language understand- ing systems. In NeurlPS 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text- to-text transformer. Journal of Machine Learning Research([MLR]
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoe V. Le. 2020. Adversar- ial examples improve image recognition. JEEF/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 816-825. Computer Vision Foundation / IEEE.
2020
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial exam- ples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 1085- 1097, Florence, Italy. Association for Computational Linguistics.
Jin Yong Yoo, John Morris, Eli Lifland, and Yanjun Qi. 2020. Searching for search method: Benchmarking search algorithms for generating NLP adversarial ex- amples. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Net- works for NI.P, pages 323-332, Online. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Jin Yong Yoo and Yanjun Qi. 2021. Towards improving adversarial training of NLP models. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 16- November, 2021, pages 945-956. 956. Association for Computational Linguistics.
Timo Schick and Hinrich Schüitze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter the As- sociation for Computational Linguistics: Main Vol- ume, EACL 2021, pages 255-269, Association for Computational Linguistics.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Gold- stein. and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understand- ing. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Technologies, NAACL-HHI 2021, Online, June 6-1 1, 2021. pages 2339-2352. Association for Computa- tional Linguistics.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language un- derstanding and generation. CoRR, abx/2009.1338 13818. 18.
Mohammad Shoeybi, Mostofa Patwary Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion parameter language models using model parallelism. CoRR, abss/199.00053.

---

APPENDIX
Tasks
Learning Rate
BoolQ
5c-5
A PTP on Seq2seq tasks
RTE
5e-5
In this section, demonstrate the versatility of our method by applying it to text summarization. We utilize different language model backbone, BART- large (Lewis et al., 2019), which is an encoder- decoder model. Our experiment setting follows Li and Liang (2021), including the hyperparameter details and dataset split. We also use their results our baseline.
WiC
le-5
WSC
5e-5
COPA
lc-5
MultiRC
le-4
CB
le-5
Table 6: Prompts' Learning Rate details for 7 tasks in FewGLUE.
Method
R-1
R-2
R-L
the same as the PT (Liu et al., 2021b), thus we ig- nore it here. The hyperparameters of Span-Off are followed as the original paper (Shen et al., 2020). To ensure fair comparison, we make the training epochs of Span-Off the same as our algorithm.
Prefix-tuning
43.80
20.93
37.25
PTP A2T
44.19
21.06
37.28
PTP + RG
44.21
21.15
37.84
PTP + RM
44.12
21.31
37.91
PTP + PGD
44.35
21.42
38.23
C Supplement for Experiment
Table 4: Text summarization XSUM. Prefix-tuning is the baseline method.
This section includes the tables and figures as a supplement our experiment and ablations. Ta- ble 7 demonstrates the comparison of the variance of different training methods FewGLUE bench- mark. Table presents the ablation of our algo- rithm with RG perturbation on MultiRC (Khashabi et al., 2018) task. We show the ablation of PGD perturbation in Table 9. The ablation RM per- turbation is presented as Table 10. Table 11 shows the ablation of our proposed PTP+A2T training algorithm.
As for the number of parameters, we use the same as the prefix tuning, with only 2% of the fine- tuning required. As Tabel 4 shows, Our algorithm can also achieve better performance on all three scores, ROUGE-1, ROUGE-2, and ROUGE-L.
Details of learning rate and prompt length
Figure 4 shows the ablation of RG perturbations on WiC (Pilchvar and Camacho-Collades 2018), RTE (Dagan al., 2005), and COPA (Gordon et al., 2012) datasets under fully-supervised learn- ing setting while Figure 5 presents the ablation our PTP+PGD training method on WiC (Pilehvar and Hamacho-Colladoo 2018) 8), BoolQ (Clark et al., 2019) and WSC (Levesque et al., 2012) datasets under few-shot learning settings. We show the ablation of our proposed PTP+RM algorithm Figure 6 on FewGLUE benchmark.
Tasks
LRI
LR2
PL1
PL2
BoolQ
le-3
5e-3
40
16
COPA
5e-3
7e-3
16
16
RTE
le-2
5e-3
20
128
WSC
3e-3
7c-3
16
8
7e-3
9e-3
16
MultiCC
lc-4
3c-3
40
20
ReCoRD
3e-4
5e-3
16
40
WiC
le-4
5e-3
20
16
All experiments are conducted on servers with RTX A6000 GPUs, each having 48GB of memory.
Table 5: Prompt Length and Learning Rate details for 8 tasks SuperGLUE. LRI, PL.:: learning and prompt length for continuous prompts with BERT-large backbone. LR2 and PL2: learning rate and prompt length for prompts with RoBERTa-large backbone.
Under sully-sueer settings, the prompt length and learning rate details are presented as Table 5. Under few-shot learning settings, we re- port as Table 6. The prompt length exactly

---

COPA
RTB
WVC
702
76
14.3
dara-1tr-4
patab
panaab
Acaumey
Actunay
160
22.5
110
to Patatied Enleadiins
140
Petucied Euibeldings
Patatied Eutseskings
Figure 4: Performance of PTP+RG on SuperGLUE (WiC, RTE, COPA datasets) with from (10- 10-3 10 10-2) and perturbed embeddings from (1, 5, 10, 20). The dashed red line represents the performance of baseline method PT2 with BaRT-large backbone LM.
BoolQ
390
38.2
Accuaay
74-
57A
64.5
368
254
350
POD Nee.
PGD Bers
PGD Ben
Figure Performance PTP+PGD on FewGLUE (WiC, BoolQ. WSC datasets) with different a and PGD iterations. The dashed red line represents the performance of the baseline method PT (Liu et al., 10). It shows the best a and PGD iterations are 10-3 and 4, respectively.
Tasks
RTE
WSC
WiC
BoolQ
PT
1.89
1.68
1.77
1.45
+RG
0.81
0.78
0.91
0.56
+22T
0.45
0.43
g1.991.38
0.59
+RM
0.68
0.95
0.87
0.65
+PGD
0.35
0.31
0.47
0.43
Table The variance of the scores on the dev sets of RTE, COPA and BoolQ from the FewGLUE benchmark. We compute runs with different random seeds (other hyper-parametts the same). We employ bold font denote the smallest deviation in each task and blue font to denote the decrease when compared to PT.
MultiRC
E=1
E=5
E=10
E=20
0=le-2
75.2
74.3
75.8
74.7
o=le-3
77.3
78.1
77.1
76.6
o=le-4
76.9
77.2
76.8
75.5
Table Results different a in RG perturbation (MultiRC task). The baseline method PT2 and its performance is 75.0. E denotes number of embeddings that perturbed. We mark the best and the worst.
COPA
t=
3
4
a=le-2
70.3
72.1
72.0
71.0
69.014.0
le-3
73.4
70.8
72.9
73.5
73.2
le-4
73.1
73.4
74.7
73.8
72.5
Table 9: Results of PTP+PGD fully-supervised COPA dataset. The baseline method is PT2 (Liu et al., 2022). whose accuracy is 73.0 The backbone LM employed is large, is the perturbation size while is PGD iterations (see Eq. (8)).

---

Dataset
LM
9
10
BoolQ
BERT
-0.12
+0.50
+1.57
+0.65
+1.35
+1.14
+1.12
+0.24
+0.25
-0.38
(Full)
RoBERTa
+0.31
+0.76
+1.10
+0.88
+0.69
40.36
+0.19
-0.43
-0.67
-0.36
Table 10: The results of different i in Eq. (7) (number of [MASK] randomly inserted into input sequence perturbation). We select BoolQ with fully-supervised settings to report. The reported increase or decrease is compared to the baseline method PT2.
vs PT+RM on MubiR
us PT+AM on
PAAT wooe +a.s ENTT waas
of (M4n)) Insertes
Number [Maad) Insettd
Figure 6: The results of different i in Eq. (7) (number of [MASK] randomly inserted into input sequence perturbation). We select MultiCC and RTE tasks Few-shot setting to report.
Cosine Sim
0.2
0.4
0.6
0.8
BoolQ(Full)
-0.83
-0.46
+0.61
+0.52
BoolQ(Few)
-0.96
-0.13
+0.78
+0.84
C(FuiR
-0.77
-0.35
+0.25
+0.71
MultiRC(Few)
-0.59
-0.54
+1.03
+0.68
Table 11: The results of different minimum cosine similarity in A2T perturbation. Full: fully-supervised learning setting. Few: few-shot learning setting.


Reviews:
[{"review_id": "xZmAjkNSwd", "reviewer": "Reviewer_RZJH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The writing is super clear. I can easily follow the contents, even for relatively complex concepts. Can see that the authors put efforts on revising. \n* The proposed algorithm is simple yet effective, and allows for flexibility for future studies (e.g., where to inject noises, which kinds of adversarial noises to inject, etc.).\n* I like the reasoning from sharp crests to regularization — and the performance is convincing.", "reasons_to_reject": "* This paper didn’t discuss the research line of the stability of vanilla fine-tuning or generally neural models, which I believe is important and relevant (relevant papers are mentioned in missing references). \n* The authors motivate the work with stability; however, stability is a rather small theme in experiments (Table 7 is in Appendix C). I would sugest the authors to better test the motivating theory (e.g., how is the loss landscape under PTP? Do we observe better stability after maunally injecting noise during inference?), and measure instability (e.g. Churn, Bhijanapalli et al., 2021; Prediction and representation instability, Du and Nguyen, 2023; both papers see missing references). I am happy to raise my soundness to 4 or 5 if the authors can put more effort here.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studied the problem of the instability of prompt-tuning, i.e., different fine-tuning runs with only random seed difference result in very different performance. The author observed that there were many sharp crest in the loss landscape of prompt-tuning, and suspected this phenomenon to be the origin. To mitigate this instability, this paper proposed to add perturbation-based regularizers for prompt-tuning (PTP). Concretely, two different types of perturbations on both input and embedding spaces were considered: random noises and adversarial-based perturbation. Experiments on FewGLUE and SuperGLUE showed that PTP could both stabilize fine-tuning and enhance fine-tuned model’s performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 309-310: which makes sure... (the sentence is not finished)"}, "extra_text": {"Missing_References": "Papers on neural model/vanilla fine-tuning instability, e.g., \n\n* Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping, Dodge et al., 2020\n* On the Reproducibility of Neural Network Predictions, Bhojanapalli et al., 2021\n* Similarity and Matching of Neural Network Representations, Csiszárik et al., 2021\n* Noise Stability Regularization for Improving BERT Fine-tuning, Hua et al., 2021\n* On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines, Mosbach et al., 2021\n* Revisiting Few-sample BERT Fine-tuning, Zhang et al., 2021\n* Measuring the Instability of Fine-Tuning, Du and Nguyen, 2023"}}, {"review_id": "qjyS6KruYB", "reviewer": "Reviewer_uB8g", "summary": "", "questions": "A) Did the different approaches used (roughly) the same amount of compute?   If not, how much would the results differ if the different approaches were provided with the same amount of computation?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Prompt tuning is widely used and notoriously unstable, hence the paper addresses a timely problem.", "reasons_to_reject": "The paper misframes its contributions as it is not the first to identify and try to mitigate the instability of prompt tuning.  \nAs data augmentation and prompt tuning with the augmented data arguably comes with additional computational burden, it is not not clear if simply training longer without data augmentation can achieve similar results.  \nThis question is especially important as the suggestion from prior work for making prompt tuning more stable is to reduce the number of tunable parameters and to train longer.", "questions_to_authors": "A) Did the different approaches used (roughly) the same amount of compute?   If not, how much would the results differ if the different approaches were provided with the same amount of computation?", "paper_topic_and_main_contributions": "The paper deals with improving the stability of prompt tuning.  \nThe idea is to apply data augmentation (both in the continuous and the discrete space) via either adding random perturbations to the input or employing adversarial training for the real data points and optimize towards the proper prompts over the augmented set of data points.\n\nThe paper gives the impression that the instability of prompt tuning was not investigated earlier (e.g. line 281 says 'Since we are the first to investigate the training stability issue of the prompt tuning'), however, this statement is not true as [Chen et al. (2022)](https://aclanthology.org/2022.emnlp-main.168) have already tackled the problem and also identified potential causes (and remedies).  \n\nThe data augmentation comes with reasonable computational burden (especially the one incorporating adversarial training).  \nIf the less resource intensive prompt tuning (without augmentation) or traditional fine-tuning was performed longer and with the same computational budget as the proposed approach enjoying the additional benefit of using augmented data points, their results might as well improve and become more stable.  \nIt is not clear if the different methods were ensured to have (roughly) the same computational budget.\n  \nEven though the main motivation of the paper is to reduce the variance of prompt-tuning methods, that aspect is not well displayed in the paper among the experimental results.  \nAs for the few shot evaluation, the variance scores are reported only in the appendix, and the variance scores are only reported for 4 of the 7 investigated tasks.  \nThese results are either incomplete at best, or cherry-picked at worst.  \nFor the fully-supervised setting, variances for all the tasks are reported, but only in the form of a heatmap, which is [hard to interpret and can be misleading](https://link.springer.com/chapter/10.1007/978-3-642-02574-7_4) without the actual figures being reported.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* [Chen et al., 2022](https://aclanthology.org/2022.emnlp-main.168): Revisiting Parameter-Efficient Tuning: Are We Really There Yet?  \nThis paper investigates the instability of prompt tuning and offers potential causes and also remedies to mitigate it.  \nCertain findings of that paper contradict the ones in this paper (e.g. Chen et al. found that prompt-tuning lags far behind fine-tuning), which needs to be discussed in the paper.", "Typos_Grammar_Style_and_Presentation_Improvements": "Adding the actual variance figures to the heatmap in Figure 3 (or simply using a table) would help assessing the variability of the different approaches.\n\n### Typos\nline 434: ALBERTA -> ALBERT  \nline 458: are perform better -> perform better  \nline 472: mind thte extra whitespace before the coma in ' , as '  \nline 475: PET Besst -> PET Best\nline 497: stable -> stabilize"}}, {"review_id": "0sLr6rsXvJ", "reviewer": "Reviewer_uMXZ", "summary": "", "questions": "\nDetailed comments and questions:\n1.\tAny detailed comparison of prefix-tuning with PTP? Where prefix-tuning appends past-key-value information by virtual tokens to the middle layers, different from current embedding space and text space. Or, any idea of combination them?\n2.\tCan you show several examples of perturbed datasets? Just wonder what are exactly perturbed when used in lines 10 to 16 in Algorithm 2.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong:\n1.\tPTP idea with two types of implicit regularizers are impressive;\n2.\tSignificantly better results are achieved.\n", "reasons_to_reject": "Weak:\n1.\tThe approach is mainly constrained by the quality of “perturbed data” and further detained methods of construction these datasets together with the consequent performance change are required, in multilingual and multi-task scenarios.\n2.\tNot quite clear about why PTP are not applied to the middle transformer layers in LLMs, but only at embedding space and text space?\n", "questions_to_authors": "\nDetailed comments and questions:\n1.\tAny detailed comparison of prefix-tuning with PTP? Where prefix-tuning appends past-key-value information by virtual tokens to the middle layers, different from current embedding space and text space. Or, any idea of combination them?\n2.\tCan you show several examples of perturbed datasets? Just wonder what are exactly perturbed when used in lines 10 to 16 in Algorithm 2.\n", "paper_topic_and_main_contributions": "This paper proposes “Prompt Tuning with Perturbation-based regularizer” (PTP), that can significantly reduce training instability and concurrently enhances the performance of prompt tuning. Two types of perturbations are designed as implicit regularizers, random-noise-based perturbation (PTP-RN) and adversarial-based perturbation (PTP-ADV). These two regularizers can be applied to both text space and embedding space. Experiments under super-glue and few-glue benchmarks show that PTP can significantly improve the performance of standard prompt tuning.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper address the problem of instability of prompt tuning approaches. The paper addresses an important problem as prompt-tuning is a very popular choice to fine-tune LLMs for specific tasks without incurring in prohibitive costs. The author propose an approach to introduce perturbations in the input or embedding spaces in order to stabilize the model. The reviewers agree that the idea of the paper is sound and that the contribution is relevant to the problem. The experiments proposed by the authors are comprehensive and demonstrate that the approach is working to reduce the instability of prompt tuning training.
Decision: Accept-Main