Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
Authors: Ningyu Zhang
Luoqiu Li
Xiang Chen
Shumin Deng
Zhen Bi
Chuanqi Tan
Fei Huang
Huajun Chen
Venue: ICLR 2022 Poster (2022)
Keywords: prompt-tuning
pre-trained language model
few-shot learning

Abstract: Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners. The main principle behind this approach involves reformulating potential natural language processing tasks into the task of a pre-trained language model and differentially optimizing the prompt template as well as the target label with backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any pre-trained language models; (ii) Extended to widespread classification tasks. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves a better few-shot performance.

Published as a conference paper at ICLR 2022
DIFFERENTIABLE PROMPT MAKES PRE-TRAINED LANGUAGE MODELS BETTER FEW-SHOT LEARNERS
Ningyu Zhang!22 1.2,3x Luoqiu Lit.3+ Xiang Chen 1.3
Shumin Deng!.
Zhen Bi2.3
Chuanqi Tan Fei Huang Huajun Chen 13,44
'College of Computer Science and Technology, Zhejiang University
Schooo of Software Technology. Zhejiang University
3 "Alibaba-Zhejie ZZeaiian University Joint Research Institute of Frontier Technologies
"Hangzhou Innovation Center, Zhejiang University
Allibba Group
zhhnggingyu, 3160102409, xiang_chen, 231 sm,m, bizhen.2 jitteennnjuj@22 ju. edu. cn, cchaanqi teq, f huang) 0 www.bbaaaiinn...o
ABSTRACT
Large-scale pre-trained language models have contributed significantly natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes novel pluggable, extensible, and efficient approach named DifferentiAble pRomp'T' (DART), which can convert small language models into better few-shot learners. The main principle behind this approach involves refor- mulating potential natural language processing tasks into the task of a pre-trained language model and differentially optimizing the prompt template as well as the target label with backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any pre-trained language models; (ii) Extended to widespread classifi- cation tasks. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves better few-shot performancd
INTRODUCTION
The pre-train-fine-ture fine-tune paradigm has become the de facto standard for natural language processing (NLP), and has achieved excellent results in several benchmarks (Devlin et al. 2019 Liu et al. 2019; Lewis al., 2020 Dong al. 2019 Bao et al., 2020a). The success of these pioneers seems to suggest that large-scale pre-trained models are always nothing short of panacea for boosting machine intelligence. However, supervised fine-tuning still prone to labeled data in practice and faces unignorable challenges owing to the variations of domains, language, and tasks. These drawbacks lead to the research of an important technique, fen- few-shot learning. which can significantly improve the learning capabilities of machine intelligence and practical adaptive applications by accessing only small number of labeled examples.
The GPT-3 model, introduced by Brown et al. (2020), exhibits impressive few-shot learning capabili- ties. Given a natural language prompt and 16 labeled samples as demonstrations in the contextual input, GPT-3 achieves 80% of the SOTA results. However, GPT-3 is fully dense transformer model with 175B parameters, which makes it challenging to deploy in most real-world applications.
Recently, emerging fine-tuning methodology has arisen equip smaller language models (LMs) with few-shot capabilities: adapting the pre-trained IM directly as predictor through completion of cloze task (Schick & Schütze (202T (2020); Gao et (2020); Liu et al.) (2021c)). which treats the downstream task as a (masked) language modeling problem. These prompts can be used in fine- tuning to provide the classifier with additional task information, especially in the low-data regime.
"Equal contribution and shared co-first authorship.
I *Coresponning author.
Code is available in Inttps://ot http://tthhuu.ccmm//////////////

---

Published as a conference paper at ICLR 2022
Vocabulary V
Label y
MLM Head
discloses
CLS Head
label: positive
lahel: negative
[CI.S] The drama [MASK] norking [SEP)
(CLS] The drama [MASK] porking.
[SEP)
MLM Pre-training
Conventional Fine-tuning
Class Disertmination Object
Fluency Comstrint Object
drama
report
Output Lugits
X
Vy
Pre-trained I. Language Model
Token Emheddings
Token Embeddings
e(discloses) elnothing) e(SEP) Mi bi(T, ba(T,) D) e[MASK].
e(The) e(JMASK) e(discloses) einothing) <(SEP)
MIT, h(TT b(]T,)) e(Y)..
Differentiable
Template Tokens
The drama discloses nothing [T,][T,] [T,) [MASK]
Differentiable Prompt Fine-tuning (Our Approach DART)
The architecture of DifferentiAble pRompTT (DART) model comparing with MLM pre- training and conventional fine- fine-tuning, where 7; and Y; are unused or special tokens in the vocabulary. We leverage few parameters within the language model the template and label tokens and optimize them via backpropagation without introducing additional parameters apart from the model.
Figure 1:
Notably, Scao & Rush (2021) | observe that prompting can often compensate for hundreds of data points on average across multiple classification tasks. However, determining the appropriate prompts requires domain expertise, and handcrafting a nigh-performing prompt often requires impractically large validation sets (Perez et al. (202T). Recent studies (Lu et al. (2021), Zhao et al. (2021) have reported that the manual prompt format can sub-optimal, which would result the accuracy varying from random guess performance to near the state-of-the-art. Therefore, previous approaches have attempted to search for discrete prompt tokens automatically. However, is non-trivial for widespread classification tasks to obtain optimized prompt template and target label token. For example, specific classification tasks such as relation extraction with the label of alternate name and country of birth cannot specify single label token in the vocabulary.
In this paper, we propose a novel DifferentiAble pRompT (DART) fine-tuning approach, which model-agnostic, parameter-efficient. As illustrated in Figure [ the key idea is to leverage few parameters (unused tokens) in the language model, which serve as the template and label tokens, and to optimize them in the continuous space using backpropagation. Subsequently, introduce differentiable prompt learning to obtain optimized prompt templates as well as labels. Since fine-tuning with limited samples can be affected by instability (Dodge et al. (2020); Zhang al.) (2021)), we propose an optimization algorithm to jointly learning templates as well as labels. We further introduce an auxiliary fluency constraint object to ensure association among the prompt embeddings.
We conduct extensive experiments on NLP datasets. With only a few training samples across all the tasks, our approach (DART) can obtain a better performance. Notably, absolute performance improvement of up to 23.28%, over the conventional fine-tuning, is obtained on average in the setting of K = 8 (and 1.55% for fully supervised settings) relation extraction datasets with complex label semantics. Our approach can be applied to real-world classification tasks without the high cost of collecting and annotating a large amount of data. The main contributions of this study
follows:
We propose new simple framework for few-shot learning, which is pluggable, extensible, and efficient. To the best of our knowledge, optimizing label tokens in continuous space is also new branch of research that has not been explored in language model prompting.
2

---

Published as a conference paper at ICLR 2022
A systematic evaluation of 15 NLP tasks shows that the simple-yet-effectiv method con- tributes towards improvements across all these tasks. Remarkably, given only 8 labeled samples per class, proposed approach can achieve 90% performance of the SOTA results (full dataset).
2 RELATED WORK
Language Model Prompting. The language model prompting has emerged with the introduction of GPT-3 (Brown et al. (2020), which demonstrates excellent few-shot performance (Liu et al.
(2021b)). However, GPT is not designed for fine-tuning; it mainly relies the handcraft prompt
(in-context learning (Liu ct al. 2021a): Zhao et al. (2021): Ding et al. [202 Min et al. (202I)). Thus,
recent studies (Qin & Eisner 2021); Hambardzumyan et al. (2021 Chen al. (2021) conducted in this field have been focused on automatically searching the prompts. |Sciick & Schütze (2021)
2020) propose the PET, which reformulates the NLP tasks as cloze-style questions and performs gradient-based fine-tuning. [Tam et al. (2021) improve the PET with a denser supervision object during fine-tuning. |Shin et al. (2020) propose the AUTOPROMPT to create prompts for diverse set of tasks based gradient-guide search. Han et al. (202T) propose an approach called PTR, which leverages logic rules to construct prompts with sub-prompt for many-class text classification, Wang et al. (202T) reformulate potential NLP task into entailment one, and then fine-tune the model with few-shot samples. Hu et al. (2021) propose an approach to incorporate external knowledge graph into the verbalizer with calibration. Additionally, Gao al. (2020) present
BFF-BFFFFtttt few-shot fine-tuning of language models, which leverages T5 (Raffel et al. (2020) to generate templates and search label tokens in the vocabulary. However, the utilization of the generative model and the label search with validation is computation-intensiiee Moreover, the prompt search over discrete space is sub-optimal due to the continuous nature of neural networks.
To overcome these limitations, Liu et al. (2021c) propose P-tuming, which employs trainable continu- ous prompt embeddings learned by an LSTM. Zhong et al. (202T) propose an effective continuous method called OPTIPROMPT to optimize prompts for factual probing. [Lu et al. (202cc) propose prefix-tuning which keeps language model parameters frozen but optimizes a small continuous task- specific vector for natural language generation tasks. Lester et al. (222l) propose mechanism for learning "soft prompts" to condition frozen language models to perform downstream tasks. However, these approaches still have to optimize the external parameters (e.g., LSTM in P-tuning) and are prone to complex label space.
Conversely, this study aims to develop novel few-shot learning framework based pre-trained language models which can reduce the prompt engincering (including templates and labels) and external parameter optimization. Furthermore, the proposed approach only leverages the noninvasive modification of the model, which can be plugged into any pre-trained language model and extended the widespread classification task.
Few-shot Learning. Few-shot learning can significantly improve the learning capabilities for machine intelligence and practical adaptive applications by accessing only a small number of labeled examples (Zhang al. (2020). The proposed approach corresponds
the other few-shot NLP methods, including: (1) Meta-learning Yu et al. (2018); Bao et al. (2020b): Bansal et al. 2020);
Deng al, (2020b a): eu (2020). which the quantities of the auxiliary tasks are optimized.
(2) Intermediate training (Phang et al. (2018); Yin et al. (200)), which supplements the tre-trained LMs with further training on the data-rich supervised tasks. (3) Semi-supervised learning (Miyato) et al. (2017); Xie et al. (2020)), which leverages unlabeled samples. The proposed approach focuses on a more realistic few-shot setting (the number of labeled instances per class can be any variable).
3 BACKGROUND
Let X_, -x Xi. be sentence, where is the b token in the input sentence and L is the number of tokens. Specifically, X is converted to a fixed token sequence Xn, and then mapped to sequence of hidden vectors (h+ Rd). Given the input sequence, &n, [CLS] Xm [SEP], the conventional fine-tuning approaches leverage a generic head layer over [CLS] embeddings (c.g.. an MI.P layer) to predict output class. For the prompt-based method, task-specific pattern string
3

---

Published as a conference paper at ICLR 2022
(template ' is designed to coax the model into producing textual output corresponding to given class (label token M (Y)) we refer to these two things together as prompt. Specifically, Xprompt containing one [MASK] token is directly tasked with the MLM input as:
Xprompt [CLS] J Xin [SEP]T [SEP]
When the prompt is fed into the MLM, the model obtain the probability distribution p( [MASK) (Xprompi) of the candidate class, E Y as:
p(yl Xprompt)
Σ WEN
p( [MMAS] w/ Xpromps)
where represents the whth label token of class y.
4
OUR APPROACH
4.1 MOTIVATION
It can be observed from the previous empirical findings (Gao et al. (2020); Scao & Rush (2021) that an optimal prompt is necessary for the improvement the pre-trained language models for the few-shot learners. Since templates with discrete tokens may be sub-optimal and are insufficient to represent specific class this study proposes DifferentiAble pRompTT, referred to as DART, which can reduce the requirement of prompt engincering in order to improve the applicability of the proposed method in various domains.
4.2 DIFFERENTIBBLE TEMPLATE OPTIMIZATION
Since the language tokens are discrete variables, finding the optimal prompts with token searching is non-trivial and may easily fall into the local minima. To overcome these limitations, we utilize pseudo tokens to construct templates and then optimize them with backpropagation. Specifically, given the template, T = {[Toa], [MASK], [Ta+::]]] which varies from the traditional discrete prompts, satisfying [Ty] E V and map T into:
(w([Toz]) w( (MMASK)) w([T++1mm))
(3)
DART considers [Ty] as pseudo tokens and maps the template as follows:
(ho, hi w( (,
(4)
where h2 < i are trainable parameters. Differentiable template optimization can obtain expressive templates beyond the original vocabulary V. Lastly, the templates, h, differentially optimized by:
hrom arg min L (X y)
(5)
h
Note that the values of the prompt embeddings, h, must be co-dependent with each other rather than independent. Unlike P-tuning (Liu et al. (202Tc), which utilizes bidirectional L.TM, DART leverages an auxiliary fluency constraint objective to associate the prompt embeddings with each other, thus stimulating the model to focus on context representation learning.
4.3 DIFFERENTIABLE LABEL OPTIMIZATION
Prompt-based fine-tuning requires filling in one word. and the masked word prediction is mapped to verbalizer, which produces a class (i.c., "Yes": True. "No": False). For each class c e Y, the
PI is non-trivial evaluate all options of templates and label tokens.
4

---

Published as a conference paper at ICLR 2022
previous approaches such as LM-BFF (Gao et al. (2020)) estimate the conditional likelihood of the initial L on pruned set 1/" C V of the top k vocabulary words.
However, the brute-foreing label searching: (1) is computationally intensive and tedious because the Pevv is generally very large, requiring multiple rounds of evaluation. (2) has poor scalability with increase in the class numbers (many classification datasets have more than 100 classes), the number of searches may be gf (C represents the total number of classes), which is exponential and thus intractable. Additionally, the labels of classes contain rich, complex semantic knowledge, and one discrete token may be insufficient to represent this information.
Specifically, with the labels, Y (Yr (y- different from the previous approach which converts the class type Y into a variable number of label tokens (....')...Ne...... DART maps the Y, to continuous vocabulary space follows:
M (Y,) = (hm=
(6)
where m is the number of trainable embedding in template. To avoid optimizing any external parameters, ( Mmy.., is replaced with unused tokens (e.g.. [unused]] or special tokens in vocabulary) in V to generate V., shown in Figure I
4.4 TRAINING OBJECTIVES
Since the pseudo tokens in the prompt template must be co-dependent with each other, introduce an auxiliary fluency constraint training without optimizing any other parameters inspired by [iu et al.] (2021c); Tam et al. (2021). Overall, there are two objectives: the class discrimination objective Lc and the fluency constraint objective Lp.
Class Discrimination Object
The class discrimination objective is the main objective that aims to classify the sentences. As shown in Figure | given T)), we can generate Xprompt as:
Lc = CC(g@y/Xgpoommem Xpromp)))
where CE is the cross-entropy loss function, Le represents the class discrimination loss.
Fluency Constraint Object To ensure the association among the template tokens and to maintain the ability of language understanding inherited from the PLMs, we leverage a fluency constraint object with the MLM. As shown in Figure | one token in the input sentence is randomly masked, and the masked language prediction conducted. x and are the original and masked sequences, respectively. Let x" be the target token that has been masked out in x', and g(x" as F
"x'","" y) is maximized
(x Σ (x"(( NEE
(8)
LF = Σ (CE(M(("
(9)
By optimizing Lr, the language model can obtain better contextual representation with rich association among the template tokens. We have the following training object:
L L-
(10)
where 2 is the typer-parameter Lastly, we introduce the overall optimization procedure of DART. To mitigate the instability of the few-shot fine- tne-tuning, we jointly optimize templates and labels. Note that our approach can reuse the same transformer architecture (rather than additional LSTM) so that it enjoys the beauty of simplicity for prompt-tuning.
We use the golden label y rather than the [MASK] in the input of the fluency constraint object.
5

---

Published as a conference paper at ICLR 2022
Model
SST-2 (acc)
MR (acc)
CR (acc)
Subj (acc)
TREC (ace)
Majority?
50.9
50.0
50.0
50.0
18.8
Prompt-based zero-shot
83.6
80.8
79.5
51.4
32.0
"GPT-3" in-context learning
84.8 (1.3)
80.5 (1.7)
87.4 (0.8)
(1.6
26.2 (2.4)
Fine-tuning
81.4 (3.8)
(5.9)
75.8 (3.2)
90.8 (1.8)
88.8 (2.1)
LM-BFF
92.3 (1.0)
85.5 (2.8)
89.0 (1.4)
91.2
88.2 (2.0)
P-Tuning
92.2 (0.4)
86.7 (1.2)
91.8 (1.1)
90.3 (2.2)
86.3 (4.5)
DART
93.5 (0.5)
88.2 (1.0)
91.8 (0.5)
90.7 (1.4)
87.1(38)
Fine-tuning (full))
95.0
90.8
89.4
97.0
97.4
Model
MNLI (acc)
SNLI (acc)
QNLI (acc)
MRPC (F1)
QQP (F1)
Majority
32.7
33.8
49.5
81.2
0.0
Prompt-based zero-shot
50.8
49.5
50.8
61.9
49.7
"GPT-3" in-context learning
52.0 (0.7)
47.1 (0.6)
53.8 (0.4)
45.7 (6.0)
36.1 (5.2)
Fine-tuning
45.8 (6.4)
48.4 (4.8)
60.2 (6.5)
76.6)
60.7 (4.3)
LM-BFF
68.3 (2.5)
77.1 (2.1)
68.3 (7.4)
76.2 (2.3)
67.0 (3.0)
P-Tuming
61.5 (2.1)
72.3
64.3 (2.8)
74.5 (7.6)
65.6 (3.0)
DART
67.5 (2.6)
75.8 (1.6)
66.7 (3.7)
78.3 (4.5)
67.8 (3.2)
Fine-tuning (full)²
89.8
92.6
93.3
91.4
81.7
Table I: Our main results with RoBERTa-large. t: the full training set is used. t: training examples are used. Otherwise, we use K 16 (# examples per class). We report mean (and standard deviation) performance over 5 different splits. Majority: majority class "GPT-3" in-context learning: using the in-context learning proposed in with RoBERTa-large (no parameter updates); LM-BFF: we report the performance in Gao al. (2020). full: fine-tuning using full training set.
5 EXPERIMENTS
In this section, detail the comprehensive experimental results conducted classification tasks. The promising results demonstrate that our proposed DART substantially outperforms the conven- tional fine-tuning method, thus, making pre-trained language models better few-shot learners.
5.1 DATASET STATISTICS
We conduct comprehensive study across 15 NLP tasks, which covers sentiment analysis, natural language inference, paraphrases, sentence similarity, relation extraction, and event extraction (We only report event argument extraction performance). The evaluation consisted of 10 popular sentence classification datasets (SST-2, MR, CR, Subj. TREC, MNLI, SNLI, QNLI, MRPC, QQP).To further evaluate the effectiveness of the proposed approach with complex label space, we conduct experiments on the relation extraction and event extraction datasets, including SemEval-2010 Task (Hendriekx al.
2010) MACRED-Revisg (Alt al. (2020)), Wiki8d (Han et al. 2019). ChemProt Kringelum
ctt al. 2016). and ACE-2005
5.2 SETTINGS
The proposed model is implemented using Pytorch (Paszke al. (2019)) Our experiments conducted with the same setting following LM-BFF (aao et al. (2020)) which measures the average performance with fixed set of seeds, Sseed, across five different sampled Drain for each task. We utilize a grid search over multiple hyperparameters and select the best result as measured on Daev for each set {2> (20miin ) ) S We employ AdamW the optimizer. We conduct experiments with a RoBERTa-large (Liu et al. (2019)) on classification tasks for a fair comparison with LM-BFF. We leverage an uncased BERT- large (Devlin et al. (2019)) for relation extraction datasets, except that use SCIBERT (Beltagy et al. (019)) for the ChemProt dataset. We follow Soares et al. (2019) and use special entity markers uniformly highlight the entity mentions for relation extraction.
https: 1dc. www.nppennn.mmmmm

---

Published as a conference paper at ICLR 2022
Dataset
Model
==8
==
K 32
Full
Fine-tuning
26.3
43.8
64.2
87.8
SemEval
LM-BFF
43.2
62.0
72.9
88.0
DART
51.8 (+25.5)
67.2 (+234))
77.3 (+13.))
89.1 (+1.3)
Fine-tuning
7.4
15.5
25.8
75.0
MICRED-Revise
LM-BFF
21.0
23.7
27.1
76.4
DART
25.8 (+18.4)
30.1 (+14.))
31.8 (+6.0)
77.8 (+2.8)
Fine-tuning
46.3
60.3
70.0
87.5
WiKi80
LM-BFF
66.5
73.5
78.1
86.2
DART
68.5 (+22.2)
75.2 (+14.)
79.4 (+9.4)
88.1 (+0.6)
Fine-tuning
30.2
41.5
52.5
79.5
ChemProt
LM-BFF
55.0
56.1
60.0
79.1
DART
57.2 (220.0
60.8 (+19.3)
63.1 (x10.6)
81.0 (+1.5)
Table 2: Results on RE dataset WiKi80 (accuracy), while other datasets (micro Fj). We use 8, 16, 32 examples per class). Full represents the full training set used.
Method
K=8
K=16
K=32
Full
Conventional FT
26.3
43.8
64.2
87.8
DART
51.8
67.2
77.3
89.1
fluency constraint object
50.3 (-1.5)
66.1 (411)
76.0 (-1.)
88.2 (09)
differentiable template
49.8 (:2.0)
66.3 (09)
76.2 (411)
88.4 (0.7)
differentiable label
47.5 (4.3)
62.5 (4:2)
73.7 (0.6)
87.8 (-1.3)
Table 3: Ablation of DART with different components on SemEval. (FT= Fine tuning)
5.3
MAIN RESULTS
As shown in Table we observe that our approach obtains better performance than conventional fine-tuning and achieves comparable results with LM-BFF Note that DART can reduce the prompt engineering without external models (c.g., T5 in LM-BFF) to generate templates that are readily easy to adapt to other datasets. DART can obtain 11.3% improvement with only 16 training samples class on the MR dataset, comparable with LM-BFF, which leverages T5 to generate appropriate prompts. These results indicate that DART can better stimulate potential ability and makes the pre- trained language model better few-shot learner. We also notice that DART yields better performance than P-tuning, which indicates that label optimization is beneficial.
For the classification tasks with the complex label space, as shown in Sableeaan Figure 2(a) we observe that DART outperforms the conventional fine-tuning approach as well as LM-BFF with large margin relation extraction and event extraction datasets in both the few-shot and fully supervised settings. The proposed approach achieves an improvement of 2.8% of the absolute performance the TAviED-Revisit dataset with full supervision and yields 18.4% gains with only 8 training samples per class. These findings also indicate that more relevant templates and labels can be determined without expert intervention, making possible generalize the proposed approach other domains. We attribute the significant improvements to the fact that, unlike the GLUE datasets containing small categories, in relation extraction and event extraction tasks, the datasets consist of large number of classes with complex label space, making it more challenging to obtain suitable label tokens. Furthermore, we notice that the improvement decays slowly when K becomes larger (i.e., from 8 to 32). Our approach is a simple yet effective fine-tuning paradigm that can reduce prompt engineering within the complex label space, thus, making possible to be an appropriate plug-in for some SOTA models.
5.4 ABLATION STUDY
We conduct an ablation study to validate the effectiveness of the components in the proposed approach. We observe that DART exhibits performance decay in the absence of any one of the modules. i.e.,
7

---

Published as a conference paper at ICLR 2022
Micro (%)
Micro F1 (%)
DART(BERT)
40
DARI
30
LM-RFF
P1:0F1-3-madim
BERT
F7:BERT)
full
(a) Event extraction results on ACE-2005.
(b) BERT-large & GPT-2-mediin results on Se- mEval.
Figure 2: (a) Few-shot results using the ACE-2005. We used 4, 8, 16, and 32 (# examples per class) with BERT. (FT= Fine-tuning) (b) BERT-large vs. GPT-2-medium results for the SemEval. Moreover, for lower K, our method consistently outperforms conventional fine-tuning.
fluency constraint object, differentiable template, differentiable label, demonstrating that all the modules are advantageous. Furthermore, we notice that differeeniable label optimization more sensitive to performance and is highly beneficial for DART, especially for low-resource settings. Since the proposed approach is the first approach that utilizes the differentiable label optimization, these findings illustrate that suitable label token is important.
5.5 ANALYSIS AND DISCUSSION
CAN DART BE APPLIED TO OTHER PRE-TRAINED LMs?
To evaluate whether the proposed approach can be applied to other LMs, we conduct experiments using GPT-2-mediun From Figure 2(b) observe that DART with GPT-2-medium yields better performance than the conventional tunntuning approach. Furthermore, we notice that DART with GPT-2-medium achieve performance par with BERT- observed by Liu et al. (2021c). indicating that the potential of GPT-style architectures for natural language understanding has been underestimated.
WIYY DO DIFFERENTIABLE PROMPTS YIELD BETTER PERFORMANCE?
To further analyze why our differentiable prompts method yields better performance compared with prompts with fixed templates and label tokens, we visualize the representation of masked tokens in the CR dataset during different training steps (from left right) as shown in Figure 3 (fixed) and 4 (differentiable), respectively. While both methods learn separable hidden states, differentiable prompts' representation is relatively more compact while the representation generated from fixed prompts is more scattered. This observation of differentiable prompts generating more discriminative representations than the fixed prompts method supported by an indicator Rp, the ratio between average intra-class and average inter-class distance. We believe the main reason behind its better performance lies the more discriminative representation of the differentiable method. More details can be found in Appendix A.6
WHAT EXACTLY IS OPTIMIZED PROMPT?
Since prompt templates and label tokens in the proposed approach are mapped as (h......m.-......... hm we further analyze what exactly optimized label learned. We conduct nearest-neighhor vocabulary embedding search to project the Top-3 optimized pseudo-label tokens z/ readable natural
We do not utilize the fluency constraint object in GPT-2-medium since the model is not pre-trained with ML. objective.
8

---

Published as a conference paper at ICLR 2022
75
-22
-23
-50
150
-55
100
100
Figure 3: Visualization of masked tokens' representation in different training steps (with training 10,
30, 50, 70 steps from left to right) with fixed prompts.
40
40
-50
-70
-42
-20-
-20
-80
-40
-20
60
100
-50
100
Figure 4: Visualization of masked tokens' representation in different training steps (with training 10, 30, 50, 70 steps from left to right) with differentiable prompts.
language. We use i-SNE (ann der Maaten & Hinton 2008 with normalization to visualize labels on Wiki80 dataset. For example, "military branch" refers to as red in rrereenss the relation type, which is learned by optimizing the pseudo label in the continuous space, and the "volunteered", "corporal" and "buddies", refers to as are the tokens closest to the label. This finding indicates that the differentiable method generates better semantic representation.
DART V.S. CONVENTIONAL FINE-TUNING
sibling
The ability of DART perform few-shot learning can be attributed to the label and being a true language under- standing task, that once the model is capable of perform- ing it correctly, it can easily apply this knowledge other tasks that are framed such. Superficially. (i) DART does not optimize any new parameters; however, conventional fine-tuning should learn an explicit classifier head over [CLS] embeddings, which may fail in the low-data regime. (ii) DART has the same task setting as large-scale language model pre-trainin mre-training.
brothers
iilling
sisters
intern
worker
empooye
member
buddics
voluntecred
comp branch
corpore
CONCLUSION AND FUTURE WORK
Figure 5: A 3D visualization of several label representations learned in DART.
This paper presents DART, a simple yet effective fine- tuning approach that improves the fast-shot learning pre- trained language model. The proposed approach can pro- duce satisfactory improvements in the few-shot scenarios when compared to the conventional fine- tuning approaches. The proposed method is also pluggable for other language models (e.g., BART) and can be extended to other tasks, such as intent detection and sentiment analysis. Intuitively, the results obtained in this study can be used to stimulate future research directions the few-shot or lifelong learning for NI.P.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We want to express gratitude to the anonymous reviewers for their hard work and kind comments. This work is funded by National Key R&D Program of China (Funding No.SQ2018YFC0000044 NSFCUJ9B2007/NSSFC) 846204, Zhejiang Provincial Natural Science Foundation of China (No. LGG22F03001 1), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-6)
REPRODUCIBILITY STATEMENT
Our code is available in https: / com/zjunip/DAR) for reproducibility. Hyper- parameters are provided the Appendix A.1
REFERENCES
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig, TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 1558-1569, Association for Computational Linguistics, 2020. doi: maa..mmmann..... 142
142. URL https://doi
orq/10.18653/
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. Learning to few-shot learn across diverse natural language classification tasks. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 5108-5123. International Committee on Computational Linguistics, 2020. doi: 10.
448 URL https://s:://
cxg//0.18653/ www.:..w....c.o.oooo.......... main. 448
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and SsiiooWuenn Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 642-652, PMLR, 2020a. URL /tRL:tt:ppppppceiiiiiie .mmlr
press/v119/baa20a..ro
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. Few-shot text classification with distributional signatures. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020b. URL https:// openreview net/ forum Fid= emfllffff4we
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019. Hong Kong. China, November 3-7, 2019, pp. 3613-3618. Association for Computational I. Linguistics, 2019. doi: 10. 10.18653/11/D11-1133 URL https::/tt///t org /ttp://............6663///////// wtt.1665//ww/11-11111
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vool Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien-iienn Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https:tppp.... htRl:ttttt////pcccdiineo 7457c0dde 6b cb 49741 8b http://ooceeinnn.e.eee.eeeee newrips ctt.r/oa.cc///ppp/////0000///
Abst html
10

---

Published as a conference paper at ICLR 2022
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajum Chen. Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. arXiv preprint arXfn:2104.07650, 2021.
Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen. Meta- learning with Synamic-memorry-basee prototypical network for few-shot event detection. In James Caverlee, Xia (Ben) Hu. Mounia Lalmas, and Wei Wang (eds.), WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020. pp. 151-159. ACM, 2020a. doi: 10. 10.1145/3336111.337733 URL https: //oi org/10 1145 0445/333619 3371796
Shumin Deng, Ningyu Zhang, Zhanlin Sun, Jiaoyan Chen, and Huajun Chen. When low resource NLP meets unsupervised language model: Meta- prettrann then meta-learning for few-shot text classification (student abstract). In The Shirry-Fouutth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 13773-13774. AAAI Press, 2020b. URL https://aaae
pttp://AA//rttcle//ieeee 1158
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- doi: 10.18653/v1/n19 1423. URL
4186. Association for Computational Linguisties, 2019.
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. Prompt-learning for fine-grained entity typing. arXiv preprint arXiv:2108. 10604, 2021.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06605.. 2020. URL htt.h:tts:// http://axvvvvv. org/ abs 200200000222 06305
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuer Hon. Unified language model pre-training for natural language understanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042-13054, 2019. URL
www.roooceecinnn.........eep.pp neurips pc/papex//211/haae =2052208500acc111111133333333333
Abst html
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. CoRR, abs/2002..15722 2020. URL httpttttt///axaiii wwtp:/awwwwwwr.rb.aa/aaaaaaeae// 22022 15723
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: word-level adversarial reprogramming. CoRR, Abs/2100.00122, 2021. URL
00121
Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun. Opennre: An open and extensible toolkit for neural relation extraction. In Sebastian Padó and Ruihong Huang (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNI 2019, Hong Kong, China, November 3-7, 2019 System Demonstrations, pp. 169-174. Association for Computational Linguistics, 2019. doi: 10. (0.18653/11/D19-3000 URL
18653/v1/D19 3029
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. PTR: prompt tuning with rules for text classification. CoRR, abs/2105.1 1259, 2021. httttts::://aat www.arwwww....raaaaaaaaa abs/22 abs////s5/ 2205 11259
11

---

Published as a conference paper at ICLR 2022
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In Katrin Erk and Carlo Strappa- rava (eds.), Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval @ ACI. 2010, Uppsala University, Uppsala, Sweden, July 15-16, 2010, pp. 33-38. The Association for Com- puter Linguistics, 2010. URL URLhttps:: http::/:///wwwwwwwebeeee www.ww..wwbb.... org http://www..//aatooooog/////////
1006/
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. Knowledge- able prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. CoRR, bbos2100.020033 2021. URL https:
Jens Kringelum, Sonny Kim Kjærulff. Søren Brunak, Ole Lund, Tudor Oprea, and Olivier Taboureau. Chemprot-3.0: global chemical biology diseases mapping. Database J. Biol. Databases Curation, 2016, 2016. doi: 10. 10.1093/database/bar URL https:://oi......
10. wwtt://aaaaae//////1a
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691, 2021. URL
08691
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sqquence-t-sequeene pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020. doi: 10. 1/2/6///111022.a1lllll m033. URL acl main. 703
pettp::/ddi..rgge wtt../og//...163/111111
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What
makes good in-context examples for gpt-3? CoRR, 0bs/2101.06804 2021a. URL https
//axxiv.org/abs/222
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. CoRR, abs/2107. 13586, 2021b. httlhtttss///axiiv.trt wat.rrgg//bb//aaa abs/ 13586
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, abs/2103.10385, 2021c URL
10385
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CORR, abs/1907..1169. 2019. (ttp:tt:://arrt otg. 1197.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. CoRR, abs/2104.08786, 2021.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. CoRR, abs/2108.04106. 2021. URL https
//arxiv. wwwpiiiorrg/bb... / 2108 04106
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial training methods for semi- supervised text classification. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htt.httts///openeeeeeeeee net. feeet/formmidddr113330
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perffrmerreeee deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Gareett (eds.), Advances in
12

---

Published as a conference paper at ICLR 2022
Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019. Vancouver, BC, Canada. pp. 8024-8035 2019. URL httpa:// mew.seeii..innn..niip........... wttpppppe///.///aaalll. bdbca288 fee F Abstttaat-bb.atttt.......... htmml
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. arXiv preprint arXiv:2105.1144: 2021.
Jason Phang. Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088, 2018. URL httpp://rxxiy
org/abs/ 111/111111 11111111 01008
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. CoRR, abs/2104.06599 2021. URL
0104 06599
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020, http:://mmmm....
paaers/v21/20-077 07t.mmmmmm
Teven Le Scao and Alexander M. Rush. How many data points is prompt worth? CoRR, abs/2103.08493, 2021.
URL
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. CoRR, abs/2009.071 18, 2020. URL httpss///ariiv.orr//aaaa
07118
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 255 269. Association for Computational www.bb..ggattooooooooooooo
Linguistics, 2021. URL. https: www. ac web. or org/ ant htgp///nooooooloo//oo////// eac -
207
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pp. emnlp-main.344. /tthtttss:///o...oo
Association for Computational Linguistics, 2020. doi: 10.18653/v1/0200.
1e..0.8866665 B63.66631vv// emm.20000eeeemmmmmmmmmmmm p-mnnlpmmaaa iop-main.33 34
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. Association for Computational Linguistics, 2019. doi: 10. 1211863//1//11912799 URL. https: //doi .or.orggg. /10 /113/61llvv// 9-11199-99999
Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. CoRR, abs/2103. 1955, 2021. URL wttph::///// //arxiv//xivv//
erq//bs// 2103.110110 11955
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment few-shot learner. 14690
CoRR, abs/2104. 1144.11690 14690, 2021. URL
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consistency training. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Coria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual, 2020. URL URL
iet.rssneeiis.. cc/pper// / htm
4
13

---

Published as a conference paper at ICLR 2022
Wenpeng Yin, Nazneen Fatema Rajani, Dragomir R. Radev, Richard Socher, and Caiming Xiong.
Universal natural language processing with limited annotations: Try few-shot textual entailment as start. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 8229-82282 0229-2239. Association for Computational Linguistics, 2020. doi: 10. 11..8863//1/2222.mmmi/// www.:0mmmnllmmmmmmmmm..mm 660. /tL.ttts:://..oor/////////// 1wt.:///....1//1///0000000
emnip main. 660
Haiyang Yu, Ningyu Zhang. Shumin Deng, Hongbin Ye, Wei Zhang, and Huajun Chen. Bridging text and knowledge with multi-pre prototype embedding for few-shot relational triple extraction. In Donia Scott, Núria Bel, and Chengging Zong ). Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spuin (Online), December 8-13, 2020, pp. -499-6410. International Committee on Computational Linguistics, 2020. doi: 1166653/1/22200 coling- m63..563. URL http::///oob..or/11118886/////22 ct1.200..oling main. 563
10 18w.38863333
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse few-shot text classification with multiple metrics. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pp. 1206-1215, Association for Computational Linguistics, 2018. doi: 10.18653/v https///nn//1--111 URL ht.ts: . .rgg / 10. 11653/666vll.
1109
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Jiaoyan Chen, Wei Zhang, and Huajun Chen, Relation adversarial network for low resource knowledge graph completion. In Yennun Huang, Irwin King, Tie-Yan Yan Liu, and Maarten van Steen (eds.), WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pp. 1-12. ACM/ / IW3C2, 2020. doi: 10. 10.1145/33642333380099 URL
11t.:///.....11133333333 338008
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinherger, and Yoav Artzi. Revisiting few- sample (bert] fine-tuning. In International Conference on Learning Representations, 2021. URL
Setts:://ppeneeviee net wwtpeneereee.eeetroouuleooeoe-e.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving sho-shot performance of language models. CoRR, abs/2102.09690. 2021. URL https://
arxiv.org/abs//200 09690
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is[mask]: Learning vs. learning to recall. In North American Association for Computational Linguistics (NAACL), 2021.
A APPENDIX
Our code is available in the supplementary materials for reproducibility. This section contains details about the training procedures and hyperparameters for each of the datasets. We utilize Pytorch (Paszke jet a 2019) to conduct experiments with 1 Nvidia 3090 GPUs. All optimizations are performed with the AdamW optimizer with linear warmup of learning rate over the first 10% of gradient updates to a maximum value, then linear decay over the remainder of the training. Gradients are clipped their norm exceeds 1.0, and weight decay all non-biar parameters is set to 0.01. Early stopping is adopted to reduce over- fittinn on the training set.
We follow LM-BFF (Gao et al. 2020) to measure the average performance of models trained on 5 different randomly sampled Dirain and Dey splits, and perform grid search for optimal parem-par combinations each split, including learning-rate, weight decay, and batch size.
For P-tuning (Lu et al. 2021c). due to the limit of search space, we do not set anchor tokens in prompt tokens.
For DART, we adopt joint optimization to acquire optimal prompts and fine-tune over global parame- ters. Note that use base prompts templates of pseudo tokens accelerate convergence.
14

---

Published as a conference paper at ICLR 2022
To compare fairly, use RoBERTa-large (Liu et 2019)
pre- treitrained model for both DART and P-tuning framework, following LM-BFF (Gao et al. 2020). We also adopt the best discrete prompts
together with label words in LM-BFF as base prompt settings for each framework, stated below.
A.1 HYPER-PARAMETER SEARCH SPACE OF OUR METHOD IN GRID SEARCH
SST-2, MR, CR, Subj, TREC, QNLI, MRPC, QQP
The hyper-paramette search space (the optimal set of parameters may vary across different tasks and data splits):
learning rate [lc-5, 5e-5, lc-4, 2c-4]
weight decay [0.0, 0.01, 0.05, 0.10]
number epochs [20,30]
batch size: [4, 8, 16, 24, 32]
max seq length: 128
gradient accumulation steps: [1, 2]
MNLI, SNLI
The hyper-paramene parameter search space (the optimal set of parameters may vary across different tasks and data splits):
learning rate [le-5, 5e-5, le-4, 2e-4]
weight decay [0.0, 0.01, 0.05, 0.10]
number epochs [30,40]
batch size: 8, 16]
max seq length: 256
gradient accumulation steps: [1, 2]
TACRED-Revisit, WiKi80, SemEval
The hyper-paramen search space are:
learning rate (3e-5,5e-5,1ee5555e
number epochs [20,30]
batch size: 48
seq length: 128
gradient accumulation steps: 2
ChemProt
The hyper-paramete parameter search space are:
learning rate [3e-5,5e-5, 1e-5,5e66
number epochs [20,30]
batch size: 48
max seq length: 256
gradient accumulation steps: 4
DialogRE
The hyper-parameter search space is (the optimal set of parameters may vary across different tasks and data splits):
learning rate [le-5, 5e-5, le-4, 2e-4]
15

---

Published as a conference paper at ICLR 2022
weight decay [0.0, 0.10]
number epochs [20,30,40]
batch size: [4, 8]
seq length: 256
gradient accumulation steps: [1, 2]
A.2
BASE PROMPT AND LABEL WORDS
SST-2, MR, CR
prompt template(length 3) ["tex"", "it", "was", "<mask>", "."
label words ("0": "terrible", "1"; "great")
Subj
prompt template(length= = 3) ["text", "This", "is", "<mask>",""
label words ("0": "incorreet"" "I": "correct")
TREC
prompt template(length= = ["<mask>", ":", "text"]
label words ("0": "Description", "I"""Entit"","" "Expression","""" "Human"""4"" "Loca- tion", "S""""ubb""""""
MNLI, SNLI
prompt template(length 2) ["texta", "?", "<mask>", ",", "rext""]
label words ("contradiction": "No","entailment" "Yes", "neutral": "Maybe"]
QNLI
prompt template(lengthh = ["texta", "?", "<mask>>,") "". "texty"
label words ("mottentailment:: "o't."entallmenn". "Yes")
MRPC, QQP
prompt template(length- = 2) ["rexta", "?", """ "<mask>", ".. "rext,")
label words ("0": "No". "1": "Yes")
RAviii- WiKi80, SemEval,
prompt template(lengthh ["text", Entityl, "is", "the", "<mask>>", "of", Entity2]
label words ("country of origin", "participating team", "participant parrtcippnnt.off.... of......
A.3 TEMPLATE LENGTH ANALYSIS
Model
Accuracy
DART (length 2)
92.6 (0.6)
DART(length = 3)
93.5 (0.5)
DART (length 5)
91.2 (1.1)
DART (length = 10)
90.6 (0.5)
Fine-tuning
81.4 (3.8)
Table 4: Few-shot performance on SST-2 task using templates with different length.
16

---

Published as a conference paper at ICLR 2022
We define the length of a template as the number of tokens except for input sentence and <mask>> token, and apply DART on templates with different length. The performance of a specific template length / is derived by summarizing the averaging accuracy each few-shot data splits, using template y'= /1.42,2.. /1.24.... From the Table 4 we observe that for the SST-2 task, the model whose template length is three yield best performance; however, the overall impact of template length is rather insignificant as models with different template length obtain relatively similar performance.
A.4 PERFORMANCE ON FULL TRAINING SET
Model
SST-2 (acc)
MR (acc)
CR (acc)
Subj (acc)
TREC (acc)
Fine-tuning
95.0
90.8
89.4
97.0
97.4
LM-BFF
94.9
91.9
92.4
96.9
97.3
DART
94.6
91.3
93.8
96.6
95.6
Model
MNLI (acc)
SNLI (acc)
QNLI (acc)
MRPC (F1)
QQP (FI)
Fine-tuning 89.8
92.6
93.3
91.4
81.7
LM-BFF
89.6
90.3
92.8
91.7
86.4
DART 87.3
89.5
92.3
90.4
89.5
Table 5: Full training set results with RoBERTa-large. Fine-tuning: we reported same results as Gao al. 2220). LM-BFF: we trained LM-BFF model (without demonstration) on full-training set.
conduct experiments and report the performance of DART with full-sized training data of GLUE tasks. From Table 5 we notice that DART obtain better or comparable results compared with the standard fine-tuning and LM-BFF, indicating that prompt-based tuning methods benefit less from full-sized data.
A.5 PERFORMANCE WITH CONSTRAINED LABEL TOKENS
We conduct a nearest neighbor vocabulary embedding search to project the best optimized differen- tialble label token to readable natural token. Those tokens are chosen based on cosine-similarity between tokens' embedding and the optimized differentialble label token of DART. We list them in descending order with similarity scores (i.e., the token 'great" is chosen its cosine-similarity score with trained positive label embedding "terrible" the most similar token with the trained negative label embedding: the other tokens selected and listed in descending order with similarity scores). From Table [] we observe that the performance of fixed prompt models is related the similarity score of the chosen label token and that the DART model learns more semantic representation for label tokens, thus, yield best performance.
DART is the highest among all tokens, and the token
Label tokens
Accuracy
differentiable token (DART)
91.8 (0.5)
great/terrible
91.5 (0.3)
fantastic/aw/uu
91.0 (0.6)
amazing/horrible
90.2 (0.8)
good/bad
89.6 (0.5)
Table Few-shot performance on CR task using constrained label tokens with DART.
17

---

Published as a conference paper at ICLR 2022
Intra- Intra-class class distance inter- saar-class distance
Fixed
Differentiable
0.8
inter-class
0.7
0.6
0.5
0
10
20
30
40
50
60
70
Training steps
Figure 6: The RD ratio curve dev set of CR task of fixed prompt and differentiable prompt during training.
A.6 MORE EXPERIMENTS
numeralize our observation on representation of masked token with a ratio between the average
intra-class distance and average inter-class distance of hidden state vectors RD
Damm Daww
where:
1
1 C
N:
D Dintra
Σ D intraae) c=l
distance (H.(i.
No
c=1
(11)
Ny Nog distance 1,,,H,,,,,
Σ Σ D (iterrer.ss) Dinter C(C-1) cc-1e
C(C- Σ
Σ EE
where distance the euclidean metric between two vectors, and H. means the hidden state representation of masked token of i-th sample from class For discriminative representation, its average intra-class distance is low as data points within the same class tend to gather together, and its average inter-class distance is high as data points from different classes are separated, so its RD ratio should be close to
As is shown in Figure 6 the RD ratio of the differentiable method grows lower than that of the fixed label method, which shows the hidden state representation trained in the differentiable method has better linear separability.
Note that in a masked language model, a linear transformation is performed on the hidden state representations, with linear decoder sharing weights with the model's word embeddings serving as the final token classifier. Hence it is evident that better linear separability of the representations leads to better performance, In our case, the differentiable method yields better performance due its better linear separability.
A.7 LIMITATIONS
Our work may fail when the distribution the task corpus varies from that the pre-training corpus. For example, a general pre-trained language model may be fine-tuned with more training instances in specific domain (e.g., medical domain). This issue can be addressed by intermediate training (Phang et al.) 2018 Yin al. 2020 Zhao et al. 2021), and will be analyzed in the future work. Besides, our work also shows an instability associated with saper-pprameters which is also observed by Dodge al. (2020): /hang et al. (202T); Perez al. (2021) volatility of few-shot learning
18

---

Published as a conference paper at ICLR 2022
in NLP. Overall, however, believe our work will inspire future work to few-shot settings with more practical applications to low-data settings, e.g., that involve low-resource languages or expert annotation.
A.8 BROADER IMPACT
The pre-train-fing fine-tune approach has become the standard for natural language processing (NLP). However, supervised fine-tuning is still practically affected by labeled data. This study proposes novel pluggable, extensible, and efficient approach named Differnti Able pRomp'T (DART), which can convert small language models into better few-shot learners. We believe that our study makes significant contribution to the literature because determining the appropriate prompts requires domain expertise, and handcrafting a high-performing prompt often requires impractically large validation sets, and these issues have been overcome with the use of the proposed method, which model--ggostic, parameter- efficiennt. We experimentally verified our proposed approach on 13 standard NLP tasks, and it was seen to outperform several standard NLP platforms.
19


Reviews:
[{"review_id": "zMCCAwyozPJ", "reviewer": "Reviewer_LUWM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) The paper proposes an interesting framework for few-shot fine-tuning without any prompt engineering. This approach can be easily used with other pre-trained models, which makes it a more generalizable framework.\n\n2) The problem space is well defined (and also well contrasted with previous work), and the paper is clearly written and easy to follow. \n\nWeaknesses: \n\n1) Even though the results in Table 3 for relation extraction and event extraction datasets suggest that the proposed methods perform significantly better than LM-BFF, the results on the 10 popular datasets seems to be very close to LM-BFF. From the results, only two datasets (SST-2 and MR) seem to have significant improvement over LM-BFF. It would be great to further discuss these results in detail. \n", "summary_of_the_paper": "The paper proposes a new approach called DART (Differentiable Prompt) which can perform few-shot fine-tuning without any prompt engineering (main difference w.r.t. previous works). This is achieved by optimizing the prompt template and the target label with backpropagation. Since the proposed approach doesn’t use any extra parameters, it can be easily used for any pre-trained language models. Further, the empirical evaluations suggest that it does better few-shot learning than previous works. ", "summary_of_the_review": "Overall, the approach is technically sound and is also very easy to apply to other pre-trained language models. The empirical results on the 10 popular datasets seem closer to LM-BFF and it’s hard to interpret if the approach empirically performs better than static/fixed prompts.  On the other hand, the proposed method performs much better on relation extraction and event extraction datasets, and it is not clearly discussed in the paper on these variations in the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ls2airr7Xoj", "reviewer": "Reviewer_pzjK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The paper is generally well-written, with excellent motivation and empirical setup/analysis\n - The overall strategy of differentiable prompt optimized to maintain fluency is reasonable and novel.\n - The ablation experiments and optimized prompt analysis are insightful.\n\nWeaknesses:\n - The notation/description of section 4 is not immediately intuitive. I had to read it a couple of times before I could fully follow the method. The authors could add more content to figure 1, which may resolve this issue.\n - The choice in deciding how many template tokens are used is unclear. An additional ablation experiment trying different number of tokens to optimize could be illuminating (even for just 1 dataset).\n - The ablation study would be better if specific numbers were provided.\n - I know GPT3 access is hard to get, but I wish experiments with k=8 prompts could be compared against\n\nPost rebuttal: I think another pass for clarity over the paper would be good for the final version, but otherwise I'm happy with the paper updates and I'm happy to see the ablation numbers aren't too sensitive to token count. I have updated my score to reflect this.", "summary_of_the_paper": "This paper proposes a new few-shot learning method for NLP problems by incorporating a simple,effective framework. This method is extensively validation and shows compelling performance.", "summary_of_the_review": "This paper provides a simple, yet effective approach to the few-shot learning problem. While it has a couple minor issues, I think the broader community would find this paper interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jiUwp09W1Ep", "reviewer": "Reviewer_H1jf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n- Clearly written and easy to follow.\n- A simple, scalable, and effective approach of prompt tuning has been proposed.\n- Importance of differentiable label has been well demonstrated.\n\nWeaknesses of the paper:\n- Missing information: Because the proposed Fluency Constraint Object is based on MLM, GPT-2-medium (in section 5.5 and figure 2-b) cannot use this objective.\n\nComments:\n- In section 4.5, the authors said that their method requires no external parameters. But this claim could be controversial since adding new tokens (for prompts and labels) on the pre-trained language models' token embedding seems more efficient than pre-training language models that already have unused or special tokens in their token embedding. I know many PLMs already have unused tokens in practice, but I would like to treat this claim theoretically.\n- For GLUE tasks, I'd like to know the performance of DART with full training set.\n- Comparing with Prefix-tuning and WARP in experiments would be great. But this is not mandatory.\n- In figure 3 and 4, training iteration or epoch should be presented.", "summary_of_the_paper": "This paper proposes a new soft prompt tuning method called DART. The authors claim that training label representations for prompt tuning is important. With trainable label representations as well as prompt ones, auxiliary training task called fluency constraint object. Without the addition of a task-specific architecture, DART outperforms previous state-of-the-art prompt tuning in few-shot performance on 9/14 NLP tasks.", "summary_of_the_review": "Recently, there has been heavy interests in prompt tuning such as P-tuning, Prefix-tuning, WARP due to its efficiency in use of large pre-trained language models for downstream tasks. The proposed DART has similar motivation. However, DART is more efficient than previous prompt tuning methods and is on par with them. Especially, focusing on differentiable label would be important observation for the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YyFIWwaDP-2", "reviewer": "Reviewer_EFFN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Very clearly written, the key ideas were well explained and simple.\n- The results seemed convincing.\n- As far as I can tell, all related works have been covered.\n\nWeakness\n- I am bit unconvinced about claiming that this work requires no external architecture in Table 1. There are new parameters that are getting trained/fine-tuned before we can do inference. The main difference I see between this and another approach like p-tuning is that the most of the existing architecture is getting reused vs in p-tuning there is explicitly an external LSTM.\n- It would've been nice to see what happens if we constrain the additional tokens to be a part of the same vocabulary. This is more or less the approach taken in AutoPrompt, except with the current training methodology.\n\nQuestions\n- I don't think I quite understood Sec 4.3, based on this section, it looks like there is an explicit embedding of output classes into an embedding? How are these learnt without fine-tuning? If they are learnt from scratch, then how is this different from regular fine-tuning?\n\n- While the work claims that it reduces the need to fine-tune, I am seeing this work as a point on the spectrum between effort-to-tune vs zero-shot. Engineering prompts is truly zero-shot, P-tuning is somewhere in middle that it requires an external LSTM, whereas this approach requires retraining the big model (modulo all parameters except T1, T2.. held constant). Is this fair to say? How computationally easy is it to actually fine-tune these unused embeddings vs learning an external model\n\n- How do we decide the number of unused tokens T1, T2..  to use? \n\n- How much does this work translate to a completely novel task beyond classification? e.g. let's say constituency parsing or dependency parsing task, for which it is possible to engineer a novel prompt.\n\n\n\n\n\n", "summary_of_the_paper": "This work reduces the need for prompt engineering for few-shot tasks by optimizing only the word-embeddings of unused tokens in the LLM. Results on multiple datasets showed reasonable results which IMO seems worth sharing to the broader audience. The exploration on what exactly are learned in the prompt was also interesting.", "summary_of_the_review": "Clearly written paper with key insights nicely presented. I had some comments/questions on the key insights, but I don't think it should affect acceptance, unless I misunderstood.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a prompt learning method for few-shot learning in NLP.  In particular, they proposed DART, a new soft prompt tuning method, to optimize the label representations and template. 

Overall, the paper is well-written and well-motivated. The proposed approach is interesting. The experiments were well justified and sufficient experimental analyses are provided. All reviewers support the paper. 

There are a few remaining critics of the papers. 

- The major one is the positioning of the paper raised by the reviewers. I agree with the reviewers that it is a bit misleading to emphasize the approach requires no external architecture. Although the approach can reuse the same transformer architecture (rather than additional LSTM) so that it enjoys the beauty of simplicity, it is still required additional parameters. I would suggest better clarifying this point in the final version. 

- There is also a critic that the paper is related to ADAPET. However, the key ideas in this paper are sufficiently different from ADAPET. Also, ADAPET is published at EMNLP 21 after the paper is submitted, although it was in Arxiv earlier. It's fair to say this work is concurrent with ADAPET.
Decision: Accept (Poster)