Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners

1 INTRODUCTION . The pre-train—fine-tune paradigm has become the de facto standard for natural language processing ( NLP ) , and has achieved excellent results in several benchmarks ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Lewis et al. , 2020 ; Dong et al. , 2019 ; Bao et al. , 2020a ) . The success of these pioneers seems to suggest that large-scale pre-trained models are always nothing short of a panacea for boosting machine intelligence . However , supervised fine-tuning is still prone to labeled data in practice and faces unignorable challenges owing to the variations of domains , language , and tasks . These drawbacks lead to the research of an important technique , few-shot learning , which can significantly improve the learning capabilities of machine intelligence and practical adaptive applications by accessing only a small number of labeled examples . The GPT-3 model , introduced by Brown et al . ( 2020 ) , exhibits impressive few-shot learning capabilities . Given a natural language prompt and 16 labeled samples as demonstrations in the contextual input , GPT-3 achieves 80 % of the SOTA results . However , GPT-3 is a fully dense transformer model with 175B parameters , which makes it challenging to deploy in most real-world applications . Recently , an emerging fine-tuning methodology has arisen to equip smaller language models ( LMs ) with few-shot capabilities : adapting the pre-trained LM directly as a predictor through completion of a cloze task ( Schick & Schütze ( 2021 ; 2020 ) ; Gao et al . ( 2020 ) ; Liu et al . ( 2021c ) ) , which treats the downstream task as a ( masked ) language modeling problem . These prompts can be used in finetuning to provide the classifier with additional task information , especially in the low-data regime . ∗Equal contribution and shared co-first authorship . †Corresponding author . 1Code is available in https : //github.com/zjunlp/DART . Notably , Scao & Rush ( 2021 ) observe that prompting can often compensate for hundreds of data points on average across multiple classification tasks . However , determining the appropriate prompts requires domain expertise , and handcrafting a high-performing prompt often requires impractically large validation sets ( Perez et al . ( 2021 ) ) . Recent studies ( Lu et al . ( 2021 ) ; Zhao et al . ( 2021 ) ) have reported that the manual prompt format can be sub-optimal , which would result in the accuracy varying from random guess performance to near the state-of-the-art . Therefore , previous approaches have attempted to search for discrete prompt tokens automatically . However , it is non-trivial for widespread classification tasks to obtain an optimized prompt template and target label token . For example , specific classification tasks such as relation extraction with the label of alternate name and country o f birth can not specify a single label token in the vocabulary . In this paper , we propose a novel DifferentiAble pRompT ( DART ) fine-tuning approach , which is model-agnostic , parameter-efficient . As illustrated in Figure 1 , the key idea is to leverage a few parameters ( unused tokens ) in the language model , which serve as the template and label tokens , and to optimize them in the continuous space using backpropagation . Subsequently , we introduce differentiable prompt learning to obtain optimized prompt templates as well as labels . Since fine-tuning with limited samples can be affected by instability ( Dodge et al . ( 2020 ) ; Zhang et al . ( 2021 ) ) , we propose an optimization algorithm to jointly learning templates as well as labels . We further introduce an auxiliary fluency constraint object to ensure the association among the prompt embeddings . We conduct extensive experiments on 15 NLP datasets . With only a few training samples across all the tasks , our approach ( DART ) can obtain a better performance . Notably , absolute performance improvement of up to 23.28 % , over the conventional fine-tuning , is obtained on average in the setting of K = 8 ( and 1.55 % for fully supervised settings ) on relation extraction datasets with complex label semantics . Our approach can be applied to real-world classification tasks without the high cost of collecting and annotating a large amount of data . The main contributions of this study are as follows : • We propose a new simple framework for few-shot learning , which is pluggable , extensible , and efficient . To the best of our knowledge , optimizing label tokens in continuous space is also a new branch of research that has not been explored in language model prompting . • A systematic evaluation of 15 NLP tasks shows that the simple-yet-effective method contributes towards improvements across all these tasks . Remarkably , given only 8 labeled samples per class , our proposed approach can achieve 90 % performance of the SOTA results ( full dataset ) . 2 RELATED WORK . Language Model Prompting . The language model prompting has emerged with the introduction of GPT-3 ( Brown et al . ( 2020 ) ) , which demonstrates excellent few-shot performance ( Liu et al . ( 2021b ) ) . However , GPT-3 is not designed for fine-tuning ; it mainly relies on the handcraft prompt ( in-context learning ( Liu et al . ( 2021a ) ; Zhao et al . ( 2021 ) ; Ding et al . ( 2021 ) ; Min et al . ( 2021 ) ) ) . Thus , recent studies ( Qin & Eisner ( 2021 ) ; Hambardzumyan et al . ( 2021 ) ; Chen et al . ( 2021 ) ) conducted in this field have been focused on automatically searching the prompts . Schick & Schütze ( 2021 ; 2020 ) propose the PET , which reformulates the NLP tasks as cloze-style questions and performs gradient-based fine-tuning . Tam et al . ( 2021 ) improve the PET with a denser supervision object during fine-tuning . Shin et al . ( 2020 ) propose the AUTOPROMPT to create prompts for a diverse set of tasks based on a gradient-guided search . Han et al . ( 2021 ) propose an approach called PTR , which leverages logic rules to construct prompts with sub-prompts for many-class text classification . Wang et al . ( 2021 ) reformulate potential NLP task into an entailment one , and then fine-tune the model with few-shot samples . Hu et al . ( 2021 ) propose an approach to incorporate external knowledge graph into the verbalizer with calibration . Additionally , Gao et al . ( 2020 ) present LM-BFF—better few-shot fine-tuning of language models , which leverages T5 ( Raffel et al . ( 2020 ) ) to generate templates and search label tokens in the vocabulary . However , the utilization of the generative model and the label search with validation is computation-intensive . Moreover , the prompt search over discrete space is sub-optimal due to the continuous nature of neural networks . To overcome these limitations , Liu et al . ( 2021c ) propose P-tuning , which employs trainable continuous prompt embeddings learned by an LSTM . Zhong et al . ( 2021 ) propose an effective continuous method called OPTIPROMPT to optimize prompts for factual probing . Liu et al . ( 2021c ) propose prefix-tuning , which keeps language model parameters frozen but optimizes a small continuous taskspecific vector for natural language generation tasks . Lester et al . ( 2021 ) propose a mechanism for learning “ soft prompts ” to condition frozen language models to perform downstream tasks . However , these approaches still have to optimize the external parameters ( e.g. , LSTM in P-tuning ) and are prone to complex label space . Conversely , this study aims to develop a novel few-shot learning framework based on pre-trained language models which can reduce the prompt engineering ( including templates and labels ) and external parameter optimization . Furthermore , the proposed approach only leverages the noninvasive modification of the model , which can be plugged into any pre-trained language model and extended to the widespread classification task . Few-shot Learning . Few-shot learning can significantly improve the learning capabilities for machine intelligence and practical adaptive applications by accessing only a small number of labeled examples ( Zhang et al . ( 2020 ) ) . The proposed approach corresponds to the other few-shot NLP methods , including : ( 1 ) Meta-learning ( Yu et al . ( 2018 ) ; Bao et al . ( 2020b ) ; Bansal et al . ( 2020 ) ; Deng et al . ( 2020b ; a ) ; Yu et al . ( 2020 ) ) , in which the quantities of the auxiliary tasks are optimized . ( 2 ) Intermediate training ( Phang et al . ( 2018 ) ; Yin et al . ( 2020 ) ) , which supplements the pre-trained LMs with further training on the data-rich supervised tasks . ( 3 ) Semi-supervised learning ( Miyato et al . ( 2017 ) ; Xie et al . ( 2020 ) ) , which leverages unlabeled samples . The proposed approach focuses on a more realistic few-shot setting ( the number of labeled instances per class can be any variable ) . 3 BACKGROUND . Let Xin = { x1 , x2 , ... , xL } be a sentence , where xi is the ith token in the input sentence and L is the number of tokens . Specifically , Xin is converted to a fixed token sequence X̃in and then mapped to a sequence of hidden vectors { hk ∈ Rd } . Given the input sequence , X̃in = [ CLS ] Xin [ SEP ] , the conventional fine-tuning approaches leverage a generic head layer over [ CLS ] embeddings ( e.g. , an MLP layer ) to predict an output class . For the prompt-based method , a task-specific pattern string ( template T ) is designed to coax the model into producing a textual output corresponding to a given class ( label token M ( Y ) ) —we refer to these two things together as a prompt . Specifically , Xprompt containing one [ MASK ] token is directly tasked with the MLM input as : Xprompt = [ CLS ] Xin [ SEP ] T [ SEP ] ( 1 ) When the prompt is fed into the MLM , the model can obtain the probability distribution p ( [ MASK ] | ( Xprompt ) of the candidate class , y ∈ Y as : p ( y|Xprompt ) = ∑ w∈Vy p ( [ MASK ] = w|Xprompt ) ( 2 ) where w represents the wth label token of class y . 4 OUR APPROACH . 4.1 MOTIVATION . It can be observed from the previous empirical findings ( Gao et al . ( 2020 ) ; Scao & Rush ( 2021 ) ) that an optimal prompt is necessary for the improvement of the pre-trained language models for the few-shot learners . Since templates with discrete tokens may be sub-optimal and are insufficient to represent a specific class2 , this study proposes DifferentiAble pRompT , referred to as DART , which can reduce the requirement of prompt engineering in order to improve the applicability of the proposed method in various domains . 4.2 DIFFERENTIABLE TEMPLATE OPTIMIZATION . Since the language tokens are discrete variables , finding the optimal prompts with token searching is non-trivial and may easily fall into the local minima . To overcome these limitations , we utilize pseudo tokens to construct templates and then optimize them with backpropagation . Specifically , given the template , T = { [ T0 : i ] , [ MASK ] , [ Ti+1 : j ] } , which varies from the traditional discrete prompts , satisfying [ Ti ] ∈V and map T into : { w ( [ T0 : i ] ) , w ( [ MASK ] ) , w ( [ Ti+1 : m ] ) } ( 3 ) DART considers [ Ti ] as pseudo tokens and maps the template as follows : { h0 , ... , hi , w ( [ MASK ] ) , hi+1 , ... , hm } ( 4 ) where hi ( 0 ≤ i ≤ j ) are trainable parameters . Differentiable template optimization can obtain expressive templates beyond the original vocabulary V . Lastly , the templates , hi , are differentially optimized by : ĥ0 : m = argmin h L ( Xprompt , y ) ( 5 ) Note that the values of the prompt embeddings , hi , must be co-dependent with each other rather than independent . Unlike P-tuning ( Liu et al . ( 2021c ) ) , which utilizes a bidirectional LSTM , DART leverages an auxiliary fluency constraint objective to associate the prompt embeddings with each other , thus stimulating the model to focus on context representation learning .

Summary: This paper proposes a new soft prompt tuning method called DART. The authors claim that training label representations for prompt tuning is important. With trainable label representations as well as prompt ones, auxiliary training task called fluency constraint object. Without the addition of a task-specific architecture, DART outperforms previous state-of-the-art prompt tuning in few-shot performance on 9/14 NLP tasks.