Venue: NAACL 2022

# IDPG: An Instance-Dependent Prompt Generation Method

Zhuofeng Wu1 &Sinong Wang2 &Jiatao Gu2 &Rui Hou2 \\ Yuxiao Dong2,3\({}^{\dagger}\) &V.G.Vinod Vydiswaran4,1 &Hao Ma2 \\  &\({}^{1}\)School of Information, University of Michigan &\({}^{2}\)Meta AI   \({}^{3}\)Tsinghua University &\({}^{4}\)Department of Learning Health Sciences, University of Michigan &\(\{\)zhuofeng, vgvinodv\(\}\)@umich.edu &\(\{\)sinongwang, jgu, rayhou, haom\(\}\)@fb.com yuxiaod@tsinghua.edu.cn

Work partially done while interning at Meta AI.Work done when at Meta AI.

Footnote 1: Our code is publicly available at [https://github.com/CSerxy/IDPG](https://github.com/CSerxy/IDPG).

###### Abstract

Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. It freezes the pre-trained language model and only optimizes a few task-specific prompts. In this paper, we propose a conditional prompt generation method to generate prompts for each input instance, referred to as the Instance-Dependent Prompt Generation (IDPG). Unlike traditional prompt tuning methods that use a fixed prompt, IDPG introduces a lightweight and trainable component to generate prompts based on each input sentence. Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.1

Footnote 1: Our code is publicly available at [https://github.com/CSerxy/IDPG](https://github.com/CSerxy/IDPG).

+
Footnote †: Work done when at Meta AI.

## 1 Introduction

In recent years, pre-training a transformer model on a large corpus with language modeling tasks and fine-tuning it on different downstream tasks has become the primary transfer learning paradigm in natural language processing Devlin et al. (2019). Notably, this paradigm requires updating and storing all the model parameters for each downstream task. As the model size proliferates (e.g., 330M parameters for BERT Devlin et al. (2019) and 175B for GPT-3 Brown et al. (2020)), it becomes computationally expensive and challenging to fine-tune the entire pre-trained language model (LM). Thus, it is natural to ask whether we can transfer the knowledge of a pre-trained LM to downstream tasks by keeping most of the parameters fixed and tuning only a small fraction of them.

Previous studies have attempted to address this question from different perspectives. One line of research Li and Liang (2021) suggests augmenting the model with smaller, trainable modules and freezing the original transformer weights. Adapters Houlsby et al. (2019); Pfeiffer et al. (2021, 2020), for example, insert a small set of additional modules between each transformer layer. Only these additional and task-specific modules are trained during fine-tuning, reducing the number of trainable parameters to \(\sim\) 1-3% of the original transformer model per task. Compact Mahabadi et al. (2021) optimizes the training parameters further by designing a lightweight module to replace the bottleneck architecture in Adapters.

Another line of work focuses on prompting. The GPT-3 models Brown et al. (2020); Schick and Schutze (2021) find that, with proper manual prompts, a pre-trained LM can successfully match the fine-tuning performance of BERT models. LMBFF Gao et al. (2021), EFL Wang et al. (2021), and AutoPrompt Shin et al. (2020) extend this direction by inserting prompts in the input embedding

Figure 1: Overall evaluation of competing approaches on ten NLU tasks, with parameters from classification heads excluded. Our method approaches RoBERTaFT’s performance and uses fewer parameters than Adapter-based methods.



layer. However, these methods rely on grid-search for a natural language-based prompt from an ample search space, leading to optimization challenges.

To tackle this issue, prompt tuning (Lester et al., 2021), prefix tuning (Li and Liang, 2021), and P-tuning (Liu et al., 2021, 2021) approaches propose to prepend trainable prefix tokens to the input layer and train these soft prompts only during the fine-tuning stage. In doing so, the problem of searching discrete prompts is converted to a continuous optimization task, which can be solved by a variety of optimization techniques such as SGD. This significantly reduced the number of trainable parameters to just a few thousand. However, all existing prompt-tuning methods have thus far focused on task-specific prompts, which are inadequate to address the gap between pre-training and fine-tuning objectives. Specifically, it is unlikely to see many different sentences with the same prefix in the pre-training corpus. Thus, a unified prompt may disturb the prediction and lead to a performance drop. In light of these limitations, we instead ask the following question: _Can we generate input-dependent prompts to smooth the domain difference?_

This paper presents the instance-dependent prompt generation (IDPG) strategy for efficiently tuning large-scale LMs. Unlike traditional prompt-tuning methods that rely on a fixed prompt for each task, IDPG instead develops a conditional prompt generation model to generate prompts for each instance. Formally, the IDPG generator can be denoted as \(f(x;{\bf W})\), where \(x\) is the instance representation and \({\bf W}\) represents the trainable parameters. Note that by setting \({\bf W}\) to a zero matrix and only training the bias, IDPG would degenerate into the traditional prompt tuning process (Lester et al., 2021). To further reduce the number of parameters in the generator \(f(x;{\bf W})\), we propose to apply a lightweight bottleneck architecture (i.e., a two-layer perceptron) and then decompose it by a parameterized hypercomplex multiplication (PHM) layer (Zhang et al., 2021). To summarize, this work makes the following contributions:

* We introduce an input-dependent prompt generation method--IDPG--that only requires training 134K parameters per task, corresponding to \(\sim\)0.04% of a pre-trained LM such as RoBERTa-Large (Liu et al., 2019).
* Extensive evaluations on ten natural language understanding (NLU) tasks show that IDPG consistently outperforms task-specific prompt tuning methods by 1.6-3.1 points. Additionally, it offers comparable performance to Adapter-based methods while using fewer parameters.
* We conduct substantial intrinsic studies, revealing how and why each component of the proposed model and the generated prompts could help the downstream tasks.

## 2 Preliminary

### Manual Prompt

Manual prompt learning (Brown et al., 2020; Schick and Schutze, 2021) inserts a pre-defined label words in each input sentence. For example, it reformulates a sentence sentiment classification task with an input sentence \(S_{1}\) as

\[x_{in}=\texttt{[CLS]}P\texttt{[SEP]}S_{1}\texttt{[EOS]},\]

where \(P\) is the prompt such as "indicating the positive user sentiment". Using the pre-trained language model \({\bf M}\), we can obtain the sentence representation \({\bf h}_{\texttt{[CLS]}}={\bf M}(x_{in})\), and train a task-specific head softmax(\({\bf Wh}_{\texttt{[CLS]}}\)) to maximize the log-probability of the correct label. LM-BFF Gao et al. (2021) shows that adding a specifically designed prompt during fine-tuning can benefit the few-shot scenario. EFL Wang et al. (2021) further suggests that reformulating the task as entailment can further improve the performance in both low-resource and high-resource scenarios.

### Prompt Tuning

Prompt tuning (Lester et al., 2021), prefix tuning (Li and Liang, 2021), and P-tuning Liu et al. (2021, 2021) methods propose to insert a trainable prefix in front of the input sequence. Specifically, they reformulate the input for single sentence tasks as

\[x_{in}=\texttt{concat}[{\bf W}_{p},{\bf E}(\texttt{[SEP]}S_{2}\texttt{[EOS]})]\]

and for sentence pair tasks as

\[x_{in}=\texttt{concat}[{\bf W}_{p},{\bf E}(\texttt{[SEP]}S_{2}\texttt{[SEP]} S_{3}\texttt{[EOS]})],\]

where \({\bf W}_{p}\) is the embedding table of the inserted prompt, \(S_{2}\) and \(S_{3}\) are input sentences, and \({\bf E}\) denotes the operation of tokenization and extraction of embeddings. Apart from LM-BFF and EFL, there is no corresponding real text for the prompt as \({\bf W}_{p}\) is a set of random-initialized tensors to represent the soft prompt 



## 3 Instance-Dependent Prompt Generation (IDPG)

We now introduce our proposed method, IDPG, along with various model optimizations. The main procedure is illustrated in Figure 2.

### Instance-Dependent Generation

Let us assume a task \(T\) with training data \(D_{train}=\{(x_{i},y_{i})\}^{K}_{i=1}\). Following prompt tuning, we define the input \(x_{i}=\mathbf{E}([SEP]S_{1}[SEP]S_{2}[EOS])\) for sentence-pair task or \(x_{i}=\mathbf{E}([SEP]S_{1}[EOS])\) for single-sentence task, where \(\mathbf{E}(\cdot)\) is the token embedding for input sentences. Different from all previous works that only define a task-specific prompt \(\mathbf{W}_{p}(T)\in\mathbb{R}^{d\times t}\), where \(t\) is the number of tokens in prompt representation and \(d\) is the hidden dimension, we propose a instance-dependent prompt generation method. Specifically, we suppose that the generation of prompt should not only depend on the task \(T\), but also be affected by input sequence \(x_{i}\). If \(\mathbf{M}(x_{i})\in\mathbb{R}^{d}\) is a representation of the input sequence \(x_{i}\) from same pre-trained LM \(\mathbf{M}\), we design a lightweight model \(\mathbf{G}\) to generate the prompt,

\[\mathbf{W}_{p}(T,x_{i})=\mathbf{G}(\mathbf{M}(x_{i}),T),~{}x_{i}\in D_{train} \tag{1}\]

Then, we insert a prompt \(\mathbf{W}_{p}(T)\) together with input sequence \(x_{i}\) to infer \(y_{i}\) during fine-tuning. In this way, we have a unified template

\[\texttt{softmax}(\textbf{Wh}_{[CLS]}) \tag{2}\]

\[\textbf{h}_{[CLS]}=\textbf{M}(\texttt{concat}[x_{i},\textbf{W}_{p}(T,x_{i})]) \tag{3}\]

where \(\mathbf{W}\) is the trainable LM classification head.

To reduce the number of trainable parameters in \(\mathbf{G}\), we apply a lightweight bottleneck architecture (i.e., a two-layer perceptron) for generation. As illustrated in Figure 2 (c), the generator \(\mathbf{G}\) first projects the original \(d\)-dimensional sentence representation \(\textbf{h}_{i}\) into \(m\) dimensions. After passing through a nonlinear function, generator \(\mathbf{G}\) projects the hidden representation back to a \(d\) dimensions

Figure 2: An illustration of (a) manual prompt; (b) prompt-tuning method; (c) our proposed method. The red block refers to the trainable module, while the blue block refers to the frozen module.



with \(t\) timestamps. The total number of parameters for generator \({\bf G}\) is \(m(d+1)+td(m+1)\) (bias term included). This model can be regarded as the general version of prompt tuning: in the second layer of \({\bf G}\), the bias term \(td\) is a task-specific prompt, with preceding parts \(td\times m\) generating an instance-dependent prompt. The final prompt our method generated is a combination of both. In short, what we discussed here is to generate a \(t\)-length prompt for one Transformer layer. An optimization of multi-layer prompt generation will be introduced in Section 3.2.2.

We can control the added number of trainable parameters by setting \(m\ll d\), but it is still expensive since hidden dimension \(d\) is usually large (1024 in BERT/RoBERTa-Large). In the sequel, we will introduce a parameter squeezing method to further reduce trainable parameters without sacrificing performance.

Note that our proposed method relies on the input sentence representation \({\bf M}(x_{i})\) to generate prompts. One caveat is that this method will have two forward passes of the pre-trained LM during inference time - first to generate \({\bf M}(x_{i})\) and then to generate classification results. However, the sentence representation \({\bf M}(x_{i})\) used in our method is task-agnostic. In practice, we can cache the prediction \({\bf M}(x_{i})\) and use it in various downstream tasks or rely on a lightweight sentence representation such as GloVe Pennington et al. (2014) (Cf. Section 4.5.1).

### Optimization

We propose two optimization techniques to further improve our proposed method.

#### 3.2.1 Parameterized Hypercomplex Multiplication (PHM) Layers

Inspired by the recent application of parameterized hypercomplex multiplication (PHM) layers Zhang et al. (2021) in Compacter Mahabadi et al. (2021), we leverage PHM layers to optimize our prompt generator, \({\bf G}\). Generally, the PHM layer is a fully-connected layer with form \(y={\bf W}x+b\), where \(x\in\mathbb{R}^{d}\) is the input feature, \(y\in\mathbb{R}^{m}\) is the output feature, and \({\bf W}\in\mathbb{R}^{m\times d}\) and \(b\in\mathbb{R}^{m}\) are the trainable parameters. When \(m\) and \(d\) are large, the cost of learning \({\bf W}\) becomes the main bottleneck. PHM replaces the matrix \({\bf W}\) by a sum of Kronecker products of several small matrices. Given a user-defined hyperparameter \(n\in\mathbb{Z}^{+}\) that divides \(m\) and \(d\), \({\bf W}\) can be calculated as follows:

\[{\bf W}=\sum_{i=1}^{n}{\bf A}_{i}\bigotimes{\bf B}_{i} \tag{4}\]

where \({\bf A}_{i}\in\mathbb{R}^{n\times n}\), \({\bf B}_{i}\in\mathbb{R}^{\frac{m}{n}\times\frac{d}{n}}\), and \(\bigotimes\) is Kronecker product. In this way, the number of trainable parameters is reduced to \(n\times(n\times n+\frac{m}{n}\times\frac{d}{n})=n^{3}+\frac{m\times d}{n}\). As \(n\) is usually much smaller than \(m\) and \(d\), PHM reduces the amount of parameters by a factor of \(n\).

Suppose that we have a two layer perceptron with down-sample projection \({\bf W}_{1}\in\mathbb{R}^{m\times d}\) and up-sample projection \({\bf W}_{2}\in\mathbb{R}^{l\times d\times m}\), where \(d\) is the input embedding dimension, \(m\) is the hidden layer dimension, and \(t\) is the number of tokens we generate. For example, we use RoBERTa-Large with hidden size \(d=1024\), generator hidden size \(m=256\), \(n=16\), prompt length \(t=5\). By substituting the \({\bf W}_{1}\) and \({\bf W}_{2}\) by two PHM layers and letting \(A_{i}\) shared by both layers, we can reduce the number of parameters from 1.5M to 105K.

#### 3.2.2 Multi-layer Prompt Tuning

Prompt tuning Lester et al. (2021) and P-tuning Liu et al. (2021) both insert continuous prompts into the first transformer layer (cf. Figure 2(b)). While proven efficient in some specific settings, single layer prompt tuning has two main limitations: (i) Capturing deep contextual information: the impact of the first-layer prompts on final prediction is low when transformer goes deeper. (ii) Generalizing to long sequence tasks: it is unclear that prompt tuning can perform well in tasks with long input when only a limited number of parameters can be inserted in single layer.

Following Prefix tuning Li and Liang (2021) and P-tuning v2 Liu et al. (2021), we prepend our generated prompts at each transformer layer to address the above issues. However, simply generalizing our model (IDPG) to a multi-layer version (M-IDPG), will significantly increase the number of training parameters, since each layer requires an independent generator \({\bf G}\). Instead, we explore different architectures in Section 4.5.3 to balance the number of tuned parameters against model performance. In short, assuming each layer generator \(G_{i}\) has form \(y={\bf W}x+b_{i}\), we share the weight matrix \({\bf W}\) across generators and set the bias term \(b_{i}\in\mathbb{R}^{m}\) to be layer-specific, where \(i=1,\ldots,N\) is the layer index and \(N\) is the number of transformer layers 



## 4 Experiment Results

### Experimental Setup

We evaluate on ten standard natural language understanding (NLU) datasets - MPQA [23], Subj [17], CR [50], MR [16], and six tasks from GLUE [18], viz. SST-2, QNLI, RTE, MRPC, STS-B [9] and QQP. We compare our proposed method with a wide range of methods, as follows:

* a vanilla transformer fine-tuning [50] and the entailment-based fine-tuning [18].

* standard prompt tuning [9] and multi-layer prompt tuning [9; 50].

**Adapter-based fine-tuning:** This efficient transfer learning method inserts an adaptation module inside each transformer layer including Compactor [15] and Adapter [14].

We compare these against two versions of single-layer instance-dependent generation methods: S-IDPG-DNN and S-IDPG-PHM. The first version is based on a 2-layer perceptron generator, which contains 1.5M parameters. The second one uses the PHM layer and only contains 10SK parameters.

We also explore three versions of multilayer instance-dependent generation methods: M-IDPG-DNN, M-IDPG-PHM, M-IDPG-PHM-GloVe. Again, the difference between the first two is in the prompt generator, while M-IDPG-PHM-GloVe uses GloVe to encode input sequences.

For a fair comparison, all the pre-trained LMs are 24-layer 16-head RoBERTa-Large models [50]. Additional training details can be found in Appendix A.1. Notably, Prompt-tuning-134 uses 134 prompt lengths in Table 1, and it is set so to match the training parameters of the proposed method, M-IDPG-PHM.

### Performance in high-resource scenario

Table 1 shows the results of all the methods on full datasets across 10 NLU tasks. We observe that: (i) Our proposed method M-IDPG-PHM consistently outperforms the prompt tuning method and Ptuning v2 by average 3.1pt and 1.6pt, respectively (except on the RTE dataset). (ii) Compared with other efficient transfer learning methods, IDPG performs slightly worse than the Compacter [15] and Adapter [14], across the ten tasks. However, the gap is mostly from RTE and QQP. Note that IDPG uses 15K fewer parameters than the Compacter. M-IDPG-PHM is better than Compacter on four tasks and has the same performance on three tasks. (iii) The improvement of our method is more prominent in the single-sentence classification task. The four best results (MPQA, Subj, CR, MR) among all competing methods in single-sentence classification tasks are made by IDPG models. Specifically, M-IDPG-PHM performs 0.84pt and 0.36pt better than RoBERTa and EFL, respectively. (iv) PHM

\begin{table}
\begin{tabular}{l c c c c c c c c|c} \hline \hline
**Method** & **MPQA** & **Subj** & **CR** & **MR** & **SST-2** & **QNLI** & **RTE** & **MRPC** & **STS-B** & **QQP** & Avg \\ \hline _Transferer Fine-tuning_ & & & & & & & & & & \\ \hline RoBERTa & 90.4\({}_{+0.2}\) & 97.1\({}_{+0.1}\) & 90.7\({}_{+0.7}\) & 91.7\({}_{+0.2}\) & 96.4\({}_{+0.2}\) & **94.7\({}_{+0.1}\)** & 85.7\({}_{+0.2}\) & 91.8\({}_{+0.4}\) & 92.2\({}_{+0.2}\) & **91.0\({}_{+0.1}\)** & 92.2 \\ EFL & 90.3\({}_{+0.2}\) & 97.2\({}_{+0.1}\) & 93.0\({}_{+0.7}\) & 91.7\({}_{+0.2}\) & **96.5\({}_{+0.1}\)** & 94.4\({}_{+0.1}\) & 85.6\({}_{+1.2}\) & 91.2\({}_{+0.4}\) & **92.5\({}_{+0.1}\)** & **91.0\({}_{+0.2}\)** & 92.3 \\ \hline _Adapter_ & & & & & & & & & \\ \hline Computer & 91.1\({}_{+0.1}\) & 97.5\({}_{+0.1}\) & 92.7\({}_{+0.1}\) & **92.6\({}_{+0.2}\)** & 96.0\({}_{+0.2}\) & 94.3\({}_{+0.2}\) & 87.1\({}_{+1.4}\) & 91.6\({}_{+0.6}\) & 91.6\({}_{+0.1}\) & 87.1\({}_{+0.2}\) & 92.2 \\ Adapter & 90.8\({}_{+0.2}\) & 97.5\({}_{+0.1}\) & 92.8\({}_{+0.3}\) & 92.5\({}_{+0.1}\) & 96.1\({}_{+0.1}\) & 94.8\({}_{+0.2}\) & 88.1\({}_{+0.4}\) & 91.8\({}_{+0.6}\) & 92.1\({}_{+0.1}\) & 89.9\({}_{+0.1}\) & **92.6** \\ \hline _Prompting_ & & & & & & & & & \\ \hline Prompt-tuning & 90.3\({}_{+0.2}\) & 95.5\({}_{+0.4}\) & 91.2\({}_{+1.1}\) & 91.0\({}_{+0.2}\) & 94.2\({}_{+0.3}\) & 86.0\({}_{+0.3}\) & 87.0\({}_{+0.4}\) & 84.3\({}_{+0.3}\) & 87.2\({}_{+0.2}\) & 81.6\({}_{+0.1}\) & 88.8 \\ Prompt-tuning-134 & 65.7\({}_{+1.9}\) & 95.6\({}_{+0.2}\) & 86.7\({}_{+1.8}\) & 89.7\({}_{+0.5}\) & 92.0\({}_{+0.05}\) & 83.0\({}_{+1.1}\) & 87.4\({}_{+0.5}\) & 84.1\({}_{+0.5}\) & 87.6\({}_{+0.5}\) & 82.4\({}_{+0.3}\) & 85.4 \\ Ptuiningv2 & 90.4\({}_{+0.3}\) & 96.5\({}_{+0.3}\) & 92.7\({}_{+0.3}\) & 91.6\({}_{+0.1}\) & 94.4\({}_{+0.2}\) & 92.9\({}_{+0.1}\) & 78.4\({}_{+0.4}\) & 91.4\({}_{+0.4}\) & 89.9\({}_{+0.2}\) & 84.4\({}_{+0.4}\) & 90.3 \\ S-IDPG-PHM & 89.6\({}_{+0.3}\) & 94.4\({}_{+0.3}\) & 90.3\({}_{+0.2}\) & 89.3\({}_{+0.4}\) & 94.7\({}_{+0.2}\) & 90.7\({}_{+0.3}\) & 89.2\({}_{+0.2}\) & 84.3\({}_{+0.3}\) & 84.7\({}_{+0.8}\) & 82.5\({}_{+0.2}\) & 89.0 \\ S-IDPG-DNN & 89.5\({}_{+0.7}\) & 94.9\({}_{+0.4}\) & 89.9\({}_{+0.5}\) & 90.2\({}_{+0.6}\) & 95.1\({}_{+0.2}\) & 90.5\({}_{+0.5}\) & 89.8\({}_{+0.4}\) & 83.0\({}_{+0.4}\) & 83.0\({}_{+0.5}\) & 82.7\({}_{+0.3}\) & 89.1 \\ M-IDPG-PHM-GloVe & 90.9\({}_{+0.2}\) & 97.4\({}_{+0.1}\) & 93.3\({}_{+0.1}\) & **92.6\({}_{+0.3}\)** & 95.4\({}_{+0.2}\) & 94.4\({}_{+0.2}\) & 82.1\({}_{+0.6}\) & 92.1\({}_{+0.4}\) & 91.0\({}_{+0.4}\) & 86.3\({}_{+0.2}\) & 91.6 \\ M-IDPG-PHM & **91.2\({}_{+0.2}\)** & 97.5\({}_{+0.1}\) & 93.2\({}_{+0.3}\) & **92.6\({}_{+0.3}\)** & 96.0\({}_{+0.3}\) & 94.5\({}_{+0.1}\) & 83.5\({}_{+0.3}\) & **92.3\({}_{+0.2}\)** & 91.4\({}_{+0.4}\) & 86.2\({}_{+0.1}\) & 91.9 \\ M-IDPG-DNN & **91.2\({}_{+0.3}\)** & **97.6\({}_{+0.2}\)** & **93.5\({}_{+0.3}\)** & **92.6\({}_{+0.1}\)** & 95.9\({}_{+0.1}\) & 94.5\({}_{+0.2}\) & 85.5\({}_{+0.6}\) &

based generator performs on par with the DNN-based generator while having a significantly lower number of trainable parameters. (v) GloVe-based sentence encoder also performs similar to LM-based sentence encoder, indicating the advancement of instance-dependent prompt generation does not rely on a robust contextual sentence encoder. (vi) When we fix the training parameters to be the same, the comparison between Prompt-tuning-134 and M-IDPG-PHM illustrates that our approach works better than prompt tuning not just because of using more parameters.

### Efficiency

Table 2 lists the number of trainable parameters for different methods excluding the classification head. The general goal for efficient transfer learning is to train models with fewer parameters while achieving better performance. Traditional prompt-tuning method only requires training a token embedding table with a few thousand parameters. However, its performance is worse than a lightweight adapter model (e.g., Compacter with 149K parameters). Our proposed method, especially the M-IDPG-PHM, falls in the gap between prompt-tuning and adapter model, since it only requires training 134K parameters and performs on par with Compacter.

### Performance in low-resource scenario

We further evaluate our proposed method in the low-resource scenario. Following the existing evaluation protocols in the few-shot setting He et al. (2021), we sample a subset of the training data for each task with size \(K\in\{100,500,1000\}\) as our training data and another subset with size 1000 as a development set. We compare our proposed methods with all prompt tuning methods, one fine-tuning model (EFL), and one adapter tuning model (Compacter).

In the extreme low-resource case when \(K\)=100, M-IDPG-PHM performs 2.5pt better than the traditional prompt tuning method and 0.5pt better than the multi-layer P-Tuning v2 method. This improvement illustrates that our method has better generalization in few-shot settings. When \(K\) becomes larger, IDPG-PHM still maintains good results with 1.9pt and 0.2pt improvement (\(K\)=500); and 2.0pt and 0.2pt improvement (\(K\)=1000) in accuracy with traditional prompt tuning and P-tuning v2 approaches, respectively. We also observe that sometimes when \(K\) is small, our method results have high variance (e.g., 4.6 on MPQA, when \(K=100\)). We suspect that this may be due to poor initialization leading the model to non-optimal parameters.

We also note that other state-of-the-art models, such as LM-BFF Gao et al. (2021), attempt to address the few-shot learning problem from a different perspective. We want to highlight that we are exploring a solution by training as few parameters as possible while maintaining good performance. Testing the limitation of our model without freezing any parameters would be an interesting investigation, but is not the main focus of this paper.

### Intrinsic Study

We conduct several ablation studies including exploration of different generator architectures and impact of selecting different prompt positions.

#### 4.5.1 Sentence Encoder: GloVe or LMs?

The proposed IDPG method relies on pre-trained LM to extract sentence representation, i.e., [CLS] token embedding. Obtaining contextualized transformer sentence embedding is often expensive if it is not pre-computed. One open question is to explore reliability on lightweight sentence representations such as GloVe embedding Pennington et al. (2014) or token embedding of pre-trained language models.

To answer this question, we apply the pre-trained GloVe word vectors2 to extract the sentence representation. Specifically, we take the average of word vectors as the sentence embeddings:

Footnote 2: Obtained from [https://nlp.stanford.edu/data/glove](https://nlp.stanford.edu/data/glove) ̇.GB.zip version glove.6b.300d.txt

\[\mathbf{M}(x_{i})=\frac{1}{k}\sum_{j=1}^{k}\texttt{GloVe}(t_{j}),\ x_{i}\in D _{train} \tag{5}\]

\begin{table}
\begin{tabular}{l|c} \hline \hline
**Method** & **\# Parameters** \\ \hline Transformer Fine-tune Liu et al. (2019) & 355M \\ Adapter Houlsby et al. (2019) & 1.55M \\ Compact Mahabadi et al. (2021) & 149K \\ Prompt-tuning Lester et al. (2021) & 5K \\ Prompt-tuning-134 Lester et al. (2021) & 134K \\ P-TuningsyZ Liu et al. (2021) & 120K \\ S-IDPG-PHM & 105K \\ S-IDPG-DNN & 1.5M \\ M-IDPG-HMM-GloVe & 141K \\ M-IDPG-PHM & 134K \\ M-IDPG-DNN & 216K \\ \hline \hline  & \\ \end{tabular}
\end{table}
Table 2: Number of trainable parameters of different methods. Note that we did not include the parameters from classification heads.

 

where \(x_{i}\) is the input sequence with \(k\) tokens \(t_{1},\ldots,t_{k}\). According to Table 1, using GloVe as sentence encoder to generate prompts doesn't sacrifice much performance over the ten tasks and outperforms prompt tuning and P-tuning v2. It indicates that our model does not benefit a lot from a strong contextual pre-trained LM. Instead, a light sentence encoder such as GloVe can also help the tasks. Also, instance-dependent prompt tuning shows promising improvement over non-instance-dependent prompt tuning models. One of the drawbacks of our method is that it is twice as expensive to run compared to Compacter, even though it uses slightly fewer parameters. Adopting GloVe as sentence encoder would avoid going through the LM twice, thus effectively reducing IDPG's run-time complexity by half.

#### 4.5.2 Prompt Generator: PHM or DNN?

To reduce the tuning parameters, we substitute the DNN layers with PHM layers. An open question we seek to answer is what is the best generation model for prompt regardless of training parameters. Hence, we compare the PHM-based prompt generator with the DNN-based prompt generator, as shown in Table 1. We observe that including DNN as a generator doesn't improve performance significantly, with +0.1pt gain on average, while adding 87K parameters (with hidden size m=16). On the other hand, this ablation study further verifies PHM layers' efficiency in the generation model.

#### 4.5.3 Multi-layer Architecture Exploration

When applying the instance-dependent generation model **G** into a multi-layer case, the first challenge we face is the considerable increase in training parameters. If each transformer layer requires an independent generator **G\({}_{i}\)**, the number of training

Figure 3: Comparison between three different multilayer generator models (S, M, L versions), and comparison between taking layer 0’s output or previous layer’s output as input.

 

parameters increases \(N\) times, where \(N\) is the number of transformer layers (24 in RoBERTa-Large). Assuming **G** has the form \(y={\textbf{W}x+b}\), there are three alternatives: (i) Smallest version (S version): sharing both \(W\) and \(b\); (ii) Middle version (M version): sharing \(W\) and making \(b\) layer-specific; and (iii) Largest version (L version): making both \(W\) and \(b\) layer-specific.

Another way to reduce the training parameters is by adjusting the hidden size \(m\) of the generator. We compare two models with \(m=16\) and \(m=256\). Surprisingly, we find that generator with a hidden size 16 is not far from the large model (92.0 vs. 92.1, respectively, in M version). We hypothesize that the smaller hidden size of 16 is already enough to store useful instance information, and setting \(m\) too large may be less efficient.

Besides, in single-layer prompt generation model, the input to **G** is \(M(x_{i})\) - the representation of input sequence \(x_{i}\). In a multi-layer case, the input to each layer generator has another option, i.e., the previous layer's output. However, as shown in Figure 3, the experiment results suggest no significant difference between the two input ways. As for the generator selection, the three models perform as expected (S version < M version < L version). In Table 1, M-IDPG-PHM uses the previous layer's output as input, M version as the generator, and 16 as the generator hidden size. Detailed information for all models' performance on each task can be found in Appendix A.3.

#### 4.5.4 Prompt Insertion: Single-layer or Multi-layer?

P-tuning v2 Liu et al. (2021) conducted substantial ablation studies on the influence of inserting prompt into different transformer layers. To boost single-layer IDPG performance, we add supplementary training (cf. Appendix A.4) and conduct ablation studies in Appendix A.5. We come to a similar conclusion that multi-layer instance-dependent prompt tuning model (M-IDPG) is significantly better than the single-layer method (S-IDPG) in both evaluation settings. An interesting finding is that the impact of supplementary training on S-IDPG is high while it is limited for M-IDPG.

#### 4.5.5 How Prompts Help?

Given two sentences, we encode each of them by one of the comparison models and compute the cosine similarity. We sort all sentence pairs in STS-B dev set in descending order by the cosine similarity scores and get a distribution for number of pairs in each group that is included in Top-k ranking. We compare a vanilla model without any prompts with M-IDPG-PHM. Both models are fine-tuned on STS-B training set. As shown in Figure 4, prompts bring the similar sentences closer while pushing the dissimilar ones apart.

#### 4.5.6 IDPG Scalability

We study our proposed model's scalability in this section. In general, the performance of IDPG in downstream tasks improves gradually when using a larger prompt length (Cf. Appendix A.7).

## 5 Related Work

**Supplementary Training:** Existing works Phang et al. (2018); Liu et al. (2019) have observed that starting from the fine-tuned MNLI model results in a better performance than directly from the vanilla pre-trained models for RTE, STS, and MRPC tasks. A series of work SentenceBERT Reimers and Gurevych (2019), BERT-flow Li et al. (2020), SimCSE Gao et al. (2021) explored intermediate training to improve STS tasks. All of them applied pre-fine tuning on NLI datasets. More recently, EFL Wang et al. (2021) proposed a task transformation paradigm, improving single sentence tasks with less labels using rich sentence-pair datasets.

* small bottleneck layers
- are inserted and trained between frozen pre-trained transformer layers. On the GLUE benchmark, adapters attain within 0.4% of the performance of full fine-tuning by only training 3.6% parameters per task. Compactor Mahabadi

Figure 4: The number of pairs of each group in Top-200 cosine similarity ranking. More results can be found in Appendix A.6.

 

et al., 2021) substitutes the down-projector and up-project matrices by a sum of Kronecker products, reducing the parameters by a large margin while maintaining the overall performance.

**Prompting:** Hand-crafted prompts were shown to be helpful to adapt generation in GPT-3 Brown et al. (2020). Existing works including LM-BFF Gao et al. (2021a); Wang et al. (2021) explored the prompt searching in a few-shot setting.

Recently, several researchers have proposed continuous prompts training to overcome the challenges in discrete prompt searching. Prefix tuning Li and Liang (2021) and P-tuningv2 Liu et al. (2021) prepend a sequence of trainable embeddings at each transformer layer and optimizes them. Two contemporaneous works - prompt tuning Lester et al. (2021) and P-tuning Liu et al. (2021b), interleave the training parameters in the input embedding layer instead of each transformer layer. All these methods focus on task-specific prompt optimization. Our proposed method, IDPG, is the first prompt generator that is not only task-specific but also instance-specific.

## 6 Conclusion and Discussion

We have introduced IDPG, an instance-dependent prompt generation model that generalizes better than the existing prompt tuning methods. Our method first factors in an instance-dependent prompt, which is robust to data variance. Parameterized Hypercomplex Multiplication (PHM) is applied to shrink the training parameters in our prompt generator, which helps us build an extreme lightweight generation model. Despite adding fewer parameters than prompt tuning, IDPG shows consistent improvement. It is also on par with the lightweight adapter tuning methods such as Compacter while using a similar amount of trainable parameters. This work provided a new research angle for prompt-tuning of a pre-trained language model.

## Acknowledgment

We thank Brian Lester, Jonas Pfeiffer, Tianyu Gao, Qinyuan Ye, Zhenhao Zhang, Fei Sun, members of the University of Michigan's NLP4Health research group, and all anonymous reviewers for helpful discussion and valuable feedback.

## References

* B. Ba, J. R. Kiros, and G. E. Hinton (2016)Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: SS2.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, External Links: Link Cited by: SS2.
* D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia (2017)SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 1-14. Note: Vancouver, Canada, Association for Computational Linguistics External Links: Link Cited by: SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS2.
* T. Gao, A. Fisch, and D. Chen (2021)Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 3816-3830. External Links: Link Cited by: SS2.
* T. Gao, X. Yao, and D. Chen (2021)SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 6894-6910. External Links: Link Cited by: SS2.
* K. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng, Jiawei Low, Lidong Bing, and 

Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 2208-2222. External Links: Link Cited by: SS4.
* N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Vol. 97, pp. 2790-2799. External Links: Link Cited by: SS4.
* M. Hu and B. Liu (2004)Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177. Cited by: SS4.
* B. Lester, R. Al-Rfou, and N. Constant (2021)The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 3045-3059. External Links: Link Cited by: SS4.
* B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li (2020)On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 9119-9130. External Links: Link Cited by: SS4.
* X. Lisa Li and P. Liang (2021)Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 4582-4597. External Links: Link Cited by: SS4.
* X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang (2021)P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602. Cited by: SS4.
* X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang (2021)Gpt understands, too.. arXiv preprint arXiv:2103.10385. Cited by: SS4.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)RoBERTA: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS4.
* R. Karimi Mahabadi, J. Henderson, and S. Ruder (2021)Compacter: efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2106.04647. Cited by: SS4.
* M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019)fairseq: a fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, Minnesota, pp. 48-53. External Links: Link Cited by: SS4.
* Bo Pang and L. Lee (2004)A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain, pp. 271-278. External Links: Link Cited by: SS4.
* B. Pang and L. Lee (2005)Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), Ann Arbor, Michigan, pp. 115-124. External Links: Link Cited by: SS4.
* J. Pennington, R. Socher, and C. Manning (2014)GloVe: global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, pp. 1532-1543. External Links: Link Cited by: SS4.
* J. Pfeiffer, A. Kamath, A. Ruckle, K. Cho, and I. Gruevych (2021)AdapterFusion: non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 487-503. External Links: Link Cited by: SS4.
* J. Pfeiffer, I. Vulic, I. Gurevych, and S. Ruder (2020)MAD-X: an adapter-Based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 7654-7673. External Links: Link Cited by: SS4.
* J. Phang, T. Fevry, and S. R. Bowman (2018)Sentence encoders on silts: supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. Cited by: SS4.
* N. Reimers and I. Gruevych (2019)Sentence-BERT: Sentence embeddings using Siamese BERT networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982-3992. External Links: Link Cited by: SS4.

 

Timo Schick and Hinrich Schutze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 255-269. External Links: Link Cited by: SS4.
* T. Shin, Y. Razeghi, R. L. IV, E. Wallace, and S. Singh (2020)AutoPrompt: eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4222-4235. External Links: Link, Document Cited by: SS4.
* A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2019)GLUE: a multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, Cited by: SS4.
* S. Wang, H. Fang, M. Khabsa, H. Mao, and H. Ma (2021)Entailment as few-shot learner. arXiv preprint arXiv:2104.14690. Cited by: SS4.
* J. Wiebe, Theresa Wilson, and Claire Cardie (2005)Annotating expressions of opinions and emotions in language. Language resources and evaluation39 (2), pp. 165-210. External Links: Link, Document Cited by: SS4.
* A. Zhang, Y. Tay, S. Zhang, A. Chan, A. T. Luu, Si. Cheung Hui, and J. Fu (2021)Beyond fully-connected layers with quaternions: parameterization of hypercomplex multiplications with \(1/n\) parameters. arXiv preprint arXiv:2102.08597.

 

Appendix

### Experimental Settings

#### a.1.1 Training hyperparameters

We use RoBERTa-Large Liu et al. (2019) model implemented by Fairseq Ott et al. (2019) as our basic model. The detailed model hyperparameters are listed in Table 4.

Note that for both transformer fine-tuning methods, including RoBERTa Liu et al. (2019) and EFL Wang et al. (2021), we follow their official training instructions, i.e., using a polynomial learning rate scheduler with 6% steps to warm up and tuning for ten epochs. For all adapter-based and prompt-based methods, we train them more sufficiently (with fifty epochs) on small datasets (i.e., MPQA, Subj, CR, MR, RTE, MRPC, STS-B).

#### a.1.2 Model hyperparameters

We report the detailed model hyperparameters for each method in Table 1 and illustrate how numbers in Table 2 are computed.

**Compacter:** hidden size \(d=1024\), adapter hidden size \(m=16\), user defined \(n=4\), each transformer layer inserts 2 compactors. Down-project \(s_{i}\) matrix takes \(1024/4\times 4\times 24\times 2=48K\), down-project \(t_{i}\) matrix takes \(16/4\times 4\times 24\times 2=0.75K\), hidden bias takes \(16\times 24\times 2=0.75K\), up-project \(s_{i}\) and \(t_{i}\) matrix takes the same number of parameters as down-projector, the output bias takes \(1024\times 24\times 2=48K\), the shared matrix \(A_{i}\) takes \(4^{3}\times 24\times 2=3K\). **Total parameters**: \(48+0.75+0.75+48+0.75+48+3=149.25K\).

**Adapter:** hidden size \(d=1024\), adapter hidden size \(m=16\). **Total parameters**: \((1024\times 16+16+16\times 1024+1024)\times 24\times 2=1.55M\).

**Prompt-tuning:** prompt length \(t=5\). **Total parameters**: \(5\times 1024=5K\).

**Prompt-tuning-134:** prompt length \(t=134\). **Total parameters**: \(134\times 1024=134K\).

**P-tuning v2:** prompt length \(t=5\), inserted layers 24. **Total parameters**: \(5\times 24\times 1024=120K\).

**S-IDPG-PHM:** hidden size \(d=1024\), generator hidden size \(m=256\), prompt length \(t=5\), user defined \(n=16\) (Cf. Equation 4). First PHM layer \(W_{1}\) takes \(1024/16\times 256/16\times 16+256=16.25K\) parameters, second PHM layer \(W_{2}\) takes \(256/16\times 5\times 1024/16\times 16+5\times 1024=85K\) parameters, the shared matrix \(A_{i}\) takes \(16^{3}=4K\) (Note we use one shared matrix in single version IDPG). **Total parameters**: \(105K\).

**S-IDPG-DNN:** hidden size \(d=1024\), generator hidden size \(m=256\), prompt length \(t=5\). **Total parameters**: \(1024\times 256+256+256\times 5\times 1024+5\times 1024=1.5M\).

**M-IDPG-PHM-GloVe:** input vector size 300, generator hidden size \(m=16\), prompt length \(t=5\), user defined \(n=4\) (Cf. Equation 4). First PHM layer \(W_{1}\) takes \(300/4\times 16/4\times 4+16=1216\) parameters, second PHM layer \(W_{2}\) takes \(16/4\times 5\times 1024/4\times 4+5\times 1024\times 24=140K\) parameters, the shared matrix \(A_{i}\) takes \(4^{3}\times 2=128\). **Total parameters**: \(141K\).

**M-IDPG-PHM:** hidden size \(d=1024\), generator hidden size \(m=16\), prompt length \(t=5\), user defined \(n=16\) (Cf. Equation 4). First PHM layer \(W_{1}\) takes \(1024/16\times 16/16\times 16+16=1K\) parameters, second PHM layer \(W_{2}\) takes \(16/16\times 5\times 1024/16\times 16+5\times 1024\times 24=125K\) parameters, the shared matrix \(A_{i}\) takes \(16^{3}16\times 2=8K\). **Total parameters**: \(134K\).

**M-IDPG-DNN:** hidden size \(d=1024\), generator hidden size \(m=16\), prompt length \(t=5\). **Total parameters**: \(1024\times 16+16+16\times 5\times 1024+5\times 1024\times 24=216K\).

### Datasets

We provide a detailed information in Table 5 for 10 NLU datasets we used.

### Detailed results for Multi-layer Architecture Exploration

We provide a detailed result table for all compared methods in Section 4.5.3. Note that the M version model with \(m=16\) and previous layer as input one is slightly higher than the results shown in Table 1(Cf. M-IDPG-PHM), this is because we tune the learning rate more carefully in Table 6 (\(lr\in\{1e^{-2},7e^{-3},5e^{-3},3e^{-3},1e^{-3},7e^{-4},5e^{-4},3e^{-4},\\ 1e^{-4}\}\)) to seek the best performance each model can reach. While in Table 1, we tune the learning rate from \(\{5e^{-3},1e^{-3},5e^{-4},1e^{-4}\}\) to make the fair comparison with other models.

### Supplementary Training for Single-layer IDPG

According to previous works Phang et al. (2018); Wang et al. (2021), supplementing pre-trained LMs with rich data helps tasks with limited labels and stabilizes downstream fine-tuning. Following this idea, we also conduct intermediate training for single-layer IDPG 

However, a drawback of supplementary training is that if the data distribution of the downstream tasks is quite different from the supplementary training task, i.e., MRPC vs. MNLI (Wang et al., 2019), it may harm the downstream performance. Figure 5 provides a comprehensive statistic among all sentence pair tasks in GLUE benchmark. For example, the length of the first sentence in MNLI is \(9.8\) longer than the second sentence on average, while this length difference in MRPC is only \(0.6\). One natural solution to smooth the length distribution difference between tasks is to insert prompt in both supplementary training and downstream fine-tuning stage. For example, assuming that we are adding a prompt with a length \(t=5\) after the second sentence in the supplementary training stage on MNLI. Then, when fine-tuning downstream tasks

Figure 5: Length difference of GLUE sentence pair datasets.

 

such as MRPC, we concatenate the prompt after the first sentence. In this way, the length difference in MNLI and MRPC becomes more balanced: 4.8 vs. \(0.6+5=5.6\). As shown in Figure 6, we test five different insertion positions (Pos 0-4) for sentence pair tasks and three different positions (Pos 0, 1, 4) for single sentence tasks. We further reduce the distribution difference by reconstructing the supplementary training data. We double the MNLI dataset by reordering the two sentences on one shard, and use the doubled dataset during intermediate training.

### Ablation study for single-layer IDPG

#### a.5.1 Generator Architecture Exploration

We explore three different architectures for the proposed PHM-based generator: (i) Residual: a residual structure He et al. (2016) is applied to add the sentence representation to each generated tokens; (ii) LayerNorm: layer normalization Ba et al. (2016) is also added to normalize the generated token embedding; (iii) residual + layerNorm: a mixed model that uses both the residual component and LayerNorm. Note that, to balance the token embedding and sentence embedding, we apply LayerNorm to each embedding first, then after the add-up, use LayerNorm again to control the generated tokens. We observe that adding LayerNorm slightly improves the voting results, while residual performs slightly worse. One surprising result is that the mixed model of Residual and LayerNorm has significantly poorer performance.

#### a.5.2 Prompt Position

As we discussed in Section A.4, the prompt position has a direct impact on the prediction results. We conduct a comprehensive study of the prompt position for our proposed method in both supplementary training and downstream fine-tuning phases.

Looking at the prompt position in downstream tasks first, Figure 7(a) shows that for both standard prompt tuning and our proposed method, the best position is 0 for single-sentence tasks and 1 for sentence-pair tasks. This result is intuitive for single-sentence tasks since prompt in position 0 can be regarded as the premise and original input sentence as the hypothesis. For sentence-pair tasks, we hypothesize that inserting prompt into position 1 can better align the two input sentences. Figure 7(b) illustrates the effect of prompt position on the supplementary training phase. It is interesting that IDPG achieves best results in position 0 while the standard prompt-tuning achieves the best results in position 4 for both single-sentence and sentence-pair tasks.

Figure 6: Insertion positions for sentence-pair tasks.

\begin{table}
\begin{tabular}{l l c c c c c c c c c|c c c} \hline \hline
**Method** & **m** & **MPQA** & **Subj** & **CR** & **MR** & **SST-2** & **QNLI** & **RTE** & **MRPC** & **STS-B** & **QQP** & Avg \\ \hline _Input: Layer 0_ & & & & & & & & & & & & \\ \hline S version & 256 & 91.2\({}_{<0.2}\) & 97.6\({}_{>0.1}\) & 93.8\({}_{<0.3}\) & 92.6\({}_{<0.2}\) & 95.9\({}_{<0.1}\) & 93.8\({}_{<0.1}\) & 79.9\({}_{<0.0}\) & 90.8\({}_{<0.5}\) & 90.9\({}_{<0.4}\) & 85.9\({}_{<0.4}\) & 91.2 \\ M version & 256 & 91.2\({}_{<0.3}\) & 97.5\({}_{<0.1}\) & 93.6\({}_{<0.3}\) & 92.7\({}_{<0.2}\) & 95.7\({}_{<0.2}\) & 94.3\({}_{<0.1}\) & 85.5\({}_{<0.1}\) & 91.8\({}_{<0.3}\) & 91.4\({}_{<0.2}\) & 87.0\({}_{<0.4}\) & 92.1 \\ L version & 256 & 91.3\({}_{<0.1}\) & 97.6\({}_{>0.2}\) & 93.8\({}_{<0.3}\) & 92.6\({}_{<0.1}\) & 95.5\({}_{<0.2}\) & 94.5\({}_{<0.2}\) & 86.5\({}_{<0.5}\) & 92.5\({}_{<0.8}\) & 91.6\({}_{<0.1}\) & 87.3\({}_{<0.3}\) & 92.3 \\ \hline \multicolumn{10}{l}{_Input: Previous Layer_} \\ \hline S version & 256 & 91.2\({}_{<0.2}\) & 97.5\({}_{<0.1}\) & 93.5\({}_{<0.3}\) & 92.6\({}_{<0.1}\) & 95.8\({}_{<0.3}\) & 94.0\({}_{<0.1}\) & 83.4\({}_{<0.3}\) & 91.9\({}_{<0.3}\) & 86.9\({}_{<0.2}\) & 91.8 \\ M version & 256 & 91.0\({}_{<0.2}\) & 97.5\({}_{<0.1}\) & 93.4\({}_{<0.4}\) & 92.6\({}_{<0.2}\) & 96.0\({}_{<0.1}\) & 94.4\({}_{<0.2}\) & 86.6\({}_{<1.2}\) & 91.5\({}_{<0.4}\) & 91.4\({}_{<0.2}\) & 86.3\({}_{<0.1}\) & 92.1 \\ L version & 256 & 91.3\({}_{<0.2}\) & 97.4\({}_{<0.0}\) & 93.3\({}_{<0.3}\) & 92.5\({}_{<0.2}\) & 95.8\({}_{<0.1}\) & 94.5\({}_{<0.3}\) & 86.9\({}_{<0.8}\) & 92.1\({}_{<0.4}\) & 91.7\({}_{<0.2}\) & 87.1\({}_{<0.2}\) & 92.3 \\ \hline \multicolumn{10}{l}{_Input: Previous Layer_} \\ \hline S version & 16 & 91.4\({}_{<0.2}\) & 97.5\({}_{<0.1}\) & 93.6\({}_{<0.2}\) & 92.5\({}_{<0.2}\) & 95.7\({}_{<0.2}\) & 93.9\({}_{<0.0}\) & 83.6\({}_{<0.3}\) & 91.9\({}_{<0.4}\) & 90.9\({}_{<0.3}\) & 85.5\({}_{<0.4}\) & 91.6 \\ M version & 16 & 91.2\({}_{<0.2}\) & 97.5\({}_{<0.1}\) & 93.4\({}_{<0.3}\) & 92.6\({}_{<0.3}\) & 96.0\({}_{<0.3}\) & 94.5\({}_{<0.1}\) & 83.5\({}_{<0.7}\) & 92.3\({}_{<0.2}\) & 91.4\({}_{<0.4}\) & 87.1\({}_{<0.1}\) & 92.0 \\ \hline \end{tabular}
\end{table}
Table 6: Main results of different transfer learning method. Each methods are evaluated on full test sets (dev sets for GLUE tasks). We report average results across 5 runs with different initialization. We report the average of accuracy and F1 for both MRPC and QQP, and average of Pearson and Spearman correlation coefficients for STS-B. For all the other tasks, we report accuracy.

 



### Cosine Similarity Distributions in STS-B

We present the cosine similarity distributions when \(k=100\) and \(k=300\) in Figure 7(a) and in Figure 7(b), respectively.

### Ablation Study on Prompt Length

We present the impact of prompt length among several prompt tuning methods in Figure 9. IDPG shows its stability when scaling to larger models with longer prompts.

### Potential Risks

Our proposed model IDPG is a novel efficient transfer learning method. It tunes small portion parameters while directly employs backbone model parameters without any changing. However, if the backbone model stored online is attacked, whether IDPG could still work well remains unknown. One should be careful to apply our proposed model and all other prompt tuning methods in high-stakes areas without a comprehensive test.

Figure 8: The number of pairs of each group in Top-k cosine similarity ranking.

Figure 7: Impact of prompt position on (a) downstream tasks; (b) supplementary training phase.

Figure 9: Impact of prompt length.