SlowGuess/NAACL_2022_OCR huggingface dataset:
Yaqing Wang\*, Subhabrata Mukherjee†, Xiaodong Liu†, Jing Gao§, Ahmed Hassan Awadallah†, Jianfeng Gao†
$^{\S}$ Purdue University, †Microsoft Research
{wang5075, jinggao}@purdue.edu,
{submukhe, xiaodl, hassanam, jfgao}@microsoft.com
We present a new method  $\mathsf{LiST}^1$  for parameter-efficient fine-tuning of large pre-trained language models (PLMs) for few-shot learning. LiST improves over recent methods that adopt prompt-based fine-tuning (FN) using two key techniques. The first is the use of self-training to leverage large amounts of unlabeled data for prompt-based FN in few-shot settings. We use self-training in conjunction with meta-learning for re-weighting noisy pseudo-prompt labels. Self-training is expensive as it requires updating all the model parameters repetitively. Therefore, we use a second technique for lightweight fine-tuning where we introduce a small number of task-specific parameters that are finetuned during self-training while keeping the PLM encoder frozen. Our experiments show that LiST can effectively leverage unlabeled data to improve the model performance for few-shot learning. Additionally, the fine-tuning is efficient as it only updates a small percentage of parameters and the overall model footprint is reduced since several tasks can share a common PLM encoder as backbone. A comprehensive study on six NLU tasks demonstrate LiST to improve by  $35\%$  over classic fine-tuning and  $6\%$  over prompt-based FN with  $96\%$  reduction in number of trainable parameters when finetuned with no more than 30 labeled examples from each task. With only  $14M$  tunable parameters, LiST outperforms GPT-3 in-context learning by  $33\%$  on few-shot NLU tasks $^2$ .
Large pre-trained language models (PLMs) have obtained state-of-the-art performance in several natural language understanding tasks (Devlin et al., 2019a; Clark et al., 2020; Liu et al., 2019a). Despite their remarkable success in fully supervised
(a) Performance Comparison
(b) Tunable Parameters
Figure 1: LiST leverages prompt-based fine-tuning (FN) with unlabeled data for label-efficiency and adapters for reducing tunable parameters. (a) shows classic tuning, prompt-based FN and LiST using RoBERTa-large as backbone on MNLI task for a comparison. The red dash line depicts ceiling performance with full supervision with RoBERTa-large. (b) shows the number of tunable parameters for each method. Table 2 shows a comparison with GPT-3 in-context learning.
settings, their performance is still not satisfactory when fine-tuning with only a handful of labeled examples. While models like GPT-3 (Brown et al., 2020) have obtained impressive few-shot performance with in-context task adaptation, they have a significant performance gap relative to fully supervised SoTA models. For instance, few-shot GPT-3 performance is 20 points worse than fully-tuned DeBERTa (He et al., 2021) on SuperGLUE. This
*Most of the work was conducted while the first author was interning at Microsoft.
LiST is short for Lite Prompted Self-Training.
$^{2}$ Our code is publicly available at https://github.com/microsoft/LiST
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2262 - 2281
poses significant challenges for many real-world tasks where large labeled data is difficult to obtain.
In this work, we present a new fine-tuning method LiST that aims to improve few-shot learning ability over existing fine-tuning strategies using two techniques as follows.
The first one is to leverage self-training with large amounts of unlabeled data from the target domain to improve model adaptation in few-shot settings. Prompt-based fine-tuning (Gao et al., 2021) have recently shown significant improvements over classic fine-tuning in the few-shot learning setting. In this paper, we demonstrate that self-training with unlabeled data is able to significantly improve prompt-based fine-tuning (Gao et al., 2021) where we iteratively update a pair of teacher and student models given natural language prompts and very few labeled examples for the task. Since the uncertain teacher in few-shot setting produces noisy pseudo-labels, we further use meta-learning to reweight the pseudo-prompt labels.
Traditional self-training can be expensive if we have to update all model parameters iteratively. To improve the efficiency of self-training, the second key technique introduces a small number of task-specific adapter parameters in the PLM that are updated with the above technique, while keeping the large PLM encoder fixed. We demonstrate such light-weight tuning with self-training to match the model performance where all parameters are tuned. This enables parameter-efficient use of self-training and reduces the storage cost of the fine-tuned model since multiple fine-tuned models can now share the same PLM as backbone during inference. Note that the computational cost of inference is out of the scope of this paper and previous work has studied several ways to address it including techniques like model distillation (Hinton et al., 2015) and pruning (LeCun et al., 1990).
We perform extensive experiments in six natural language understanding tasks to demonstrate the effectiveness of LiST. We devise a comprehensive evaluation framework considering the variance in few-shot performance of PLMs with different shots, random seeds and splits<sup>3</sup>. Results show that LiST improves over traditional and more recent prompt-based FN methods by  $35\%$  and  $6\%$ , respectively, with  $96\%$  reduction in number of trainable parameters given only 30 labeled examples for
each downstream task. Figure 1 shows the results on MNLI (Williams et al., 2018a) as an example. We compare LiST with GPT-3 in-context learning outperforming it by  $33\%$  as well as several SoTA few-shot semi-supervised learning approaches with LiST outperforming the strongest baseline by  $6\%$  given only 30 labeled examples for each task.
Problem statement. Each downstream task in our framework consists of very few labeled training examples  $\mathcal{D}_K^{Train}$  for different shots  $K\in$ $\{10,20,30\}$  where  $|\mathcal{D}_K^{train}| = K$  , unlabeled data  $\mathcal{D}^U$  where  $\mathcal{D}^U\gg \mathcal{D}_K^{train}$  , and a test set  $\mathcal{D}^{Test}$
Given above dataset  $\mathcal{D}_K = \mathcal{D}_K^{Train} \cup \mathcal{D}^U$  for a task with shots  $K$ , a PLM with parameters  $\Theta_{\mathrm{PLM}}$  and loss function  $\mathcal{L}$ , we want to adapt the model for the few-shot learning task by introducing a small number of tunable model parameters  $\psi \ll \Theta_{\mathrm{PLM}}$ .
2 Background on Model Fine-tuning
Given a text sequence  $x$  or a pair of sequences  $\{x_1, x_2\}$  separated by special operators (e.g., [CLS] and [SEP]) and a language model encoder  $enc(\theta)$  parameterized by  $\theta$  - classic fine-tuning popularized by (Devlin et al., 2019b) leverages hidden state representation  $h_{[CLS]}$  of the sequence(s) obtained from  $enc([CLS] x_1 [SEP] x_2 [SEP])$  as input to a task-specific head  $softmax(W^T \cdot h_{[CLS]})$  for classification, where  $W \in \mathbb{R}^{d \times L}$  with  $d$  and  $L$  representing the hidden state dimension and number of classes, are randomly initialized tunable parameters. In the process it updates both task-specific head  $W$  and encoder  $\theta$  parameters jointly.
However, this introduces a gap between pretraining and fine-tuning objective with disparate label spaces and additional randomly initiated parameters  $W$  introduced for task-specific fine-tuning. This is particularly challenging for few-shot classic fine-tuning, where the limited labeled data is inadequate for adapting the task-specific head and PLM weights effectively. Prompt-based FN (Schick and Schütze, 2021b; Gao et al., 2021) addresses this gap, by re-formulating the objective as a clozestyle auto-complete task. This is done by adding a phrase (also called prompt) to a sentence like  $x_{1} =$  "contains no wit, only labored gags" in the form of  $\tilde{x} = x_{1} \oplus$  "It was [MASK]", where  $\oplus$  denotes concatenation of two strings; and output mappings (also called verbalizers) from vocabulary  $\mathcal{V}$  to the label space  $\mathcal{V}$  like " $\{\text{great}, \text{terrible}\}"$  corresponding to positive and negative classes (refer to Figure 3 for an example). The probability of
<sup>3</sup>LiST repository contains the dataset partitions for different shots, seeds and splits for every task for reproducibility and benchmarking of efficient few-shot language models.
predicting class  $y \in \mathcal{V}$  is equal to calculating the probability of corresponding label word  $v \in \mathcal{V}$ :
$$
p ([ \mathrm {M A S K} ] = v | \tilde {x}) = \frac {\exp \left(W _ {v} ^ {T} \cdot h _ {[ \mathrm {M A S K} ]}\right)}{\sum_ {v ^ {\prime} \in V} \exp \left(W _ {v ^ {\prime}} ^ {T} \cdot h _ {[ \mathrm {M A S K} ]}\right)} \tag {1}
$$
where  $W_{v}$  indicates the tunable parameters. Since it is identical to masked language modeling (MLM),  $W_{v}$  is initialized by pre-trained weights of PLMs.
In this work, we demonstrate lite self-training with unlabeled data to significantly improve prompt fine-tuning of PLMs in few-shot settings.
Few-shot and Semi-supervised Learning. Recent works have explored semi-supervised methods for few-shot learning with task-specific unlabeled data, including data augmentation (Xie et al., 2019; Vu et al., 2021), self-training (He et al., 2019; Mukherjee and Awadallah, 2020; Wang et al., 2021c) and contrastive learning (Gunel et al., 2020). GPT-3 (Brown et al., 2020) leverages massive scale with 175 billion parameters to obtain remarkable few-shot performance on several NLU tasks given natural language prompts and a few demonstrations for the task. Recent works (Schick and Schütze, 2021a; Gao et al., 2021) extend this idea of prompting to language models like BERT (Devlin et al., 2019b) and RoBERTa (Liu et al., 2019b). The most related work to ours is iPET (Schick and Schütze, 2021a), which combines prompt-based FN with semi-supervised learning. While iPET ensembles multiple fully-tuned models, we develop a lightweight prompted self-training framework to achieve both data and parameter efficiency.
Few-shot adaptation works (Finn et al., 2017; Li et al., 2019; Zhong et al., 2021; Wang et al., 2021b,a; Beck et al., 2021) train models on massive labeled data on source tasks and develop techniques to adapt them to a target task with few-shot labels. For instance, (Beck et al., 2021) first trains on miniImagenet with 38,400 labeled examples (64 classes and 600 samples per class). Similarly, (Zhong et al., 2021; Beck et al., 2021) first train their models on thousands of labels from the source task to study few-shot target adaptation. In contrast, we focus on single-task true few-shot learning with only 10 to 30 labeled examples available overall and no auxiliary supervision labels. Refer to (Perez et al., 2021) for an overview of true few-shot learning for NLU. The objective of few-shot adaptation and true few-shot learning is also quite different. The objective of true few-shot learning
is to learn a new task with limited labeled data while the objective of few-shot adaptation is to efficiently transfer to a new task/domain with limited labeled data. Thus, few-shot adaptation leverages multi-task setting with auxiliary labeled data from the source tasks that are not available in our setting. The most relevant works to our setup (Gao et al., 2021; Wang et al., 2021c; Schick and Schütze, 2021a; Mukherjee and Awadallah, 2020) are used as baselines in our paper.
Light-weight tuning. Standard fine-tuning methods tune all trainable model parameters for every task. Recent efforts have focused on lightweight tuning of large PLMs by updating a small set of parameters while keeping most of parameters in PLMs frozen, including prefix tuning (Li and Liang, 2021), prompt token tuning (Lester et al., 2021a) and Adapter tuning (Houlsby et al., 2019; Pfeiffer et al., 2020). All of the above works focus on fully supervised settings with thousands of labeled examples using classic fine-tuning. In contrast, we focus on few-shot learning settings leveraging prompts for model tuning, where we make several observations regarding the design and placement of adapters in few-shot settings in contrast to its resource-rich counterpart. Some recent works (Beck et al., 2021; Zhong et al., 2021) pretrain adapters with full supervision with thousands of labeled examples from source tasks for few-shot target adaptation. Different from this, we explore adapter tuning for single-task few-shot learning without any auxiliary supervision labels.
We adopt a PLM (e.g., RoBERTa (Liu et al., 2019b)) as the shared encoder for both the student and teacher for self-training. The shared PLM encoder is frozen and not updated during training. We introduce tunable adapter parameters in both teacher and student (discussed in Section 4.2) that are iteratively tuned during self-training. Refer to Figure 2 for steps in the following discussion.
We first use prompt-based fine-tuning to update the teacher adapter (Step 1) with few-shot labeled examples and leverage the teacher model to assign pseudo-prompt labels (Step 2) on unlabeled data  $\mathcal{D}^u$ . The teacher is often uncertain in few-shot learning and produces noisy pseudo-labels. Therefore, we adopt meta-learning (discussed in Section 4.3) to re-weight the noisy pseudo-labeled
samples (Step 3). The re-weighted data is used to train the student adapter (Step 4). Since adapter training with noisy pseudo labels is quite unstable, we introduce knowledge distillation warmup (discussed in Section 4.3.1). Finally, we assign the trained student adapter to be the new teacher adapter (Step 5). Following true few-shot learning settings, we do not use any held-out development or validation set. Therefore, we repeat the above steps for a pre-defined number of times  $(M = 6)$ . The overall training procedure is summarized in Algorithm 1 (Appendix B). Throughout the training, we keep the shared student and teacher encoder parameters frozen and update the corresponding adapter parameters along with their language model heads.
Figure 2: Lite prompted self-training on unlabeled data with prompts and adapters make parameter-efficient few-shot learners with LiST.
4.2 Lightweight Prompt Adapter Tuning
The predominant methodology for task adaptation is to tune all of the trainable parameters of the PLMs for every task. This raises significant resource challenges both during training and deployment. A recent study (Aghajanyan et al., 2021) show that PLMs have a low intrinsic dimension that can match the performance of the full parameter space. To adapt PLMs for downstream tasks with a small number of parameters, adapters (Houlsby et al., 2019) have recently been introduced as an alternative approach for lightweight tuning. Consider the following scenario for demonstration, where we want to use RoBERTa-large with  $\mathcal{M} = 355M$  parameters as the PLM for  $\mathcal{T} = 100$  tasks. Full fine-tuning for this scenario requires updating and storing  $\mathcal{M}\times \mathcal{T} = 35.5B$  parameters. Now, consider
fine-tuning with LiST that requires  $\mathcal{A} = 14M$  (tunable) adapter parameters for every task while keeping the PLM fixed. This results in overall  $\mathcal{M} + \mathcal{A}\times \mathcal{T} = 1.8B$  parameters, thereby, reducing the overall storage cost by  $20\mathrm{x}$ . Adapters have been shown to match the PLM performance in fully supervised settings with thousands of training labels in classic fine-tuning. In contrast, this is the first work to study the role of adapters in few-shot prompt-based FN. We explore different design and placement choices of adapters in few-shot settings and investigate the performance gap with fully supervised as well as fully tunable parameter space.
Figure 3: The underlined text depicts task prompt to transform classification into Fill-in-MASK task. Label words are used as proxy for original task labels.
The adapter tuning strategy judiciously introduces new parameters into the original PLMs. In contrast to standard prompt-based FN that updates all the PLM parameters  $\Theta_{\mathrm{PLM}}$ , prompt-adapter tuning only updates the newly introduced adapter parameters as well as the (masked) language model head of the PLM (jointly denoted as  $\psi$ ), while keeping the remaining parameters of the original network frozen. The adapter used in LiST consists of two fully connected layers as shown in Figure 4, where a feedforward layer down projects input representations to a low dimensional space  $d$  (referred as the bottleneck dimension), and another feedforward layer up projects the low-dimensional features back to the original dimension. However, these newly-inserted parameters can cause divergence resulting in up to  $20\%$  performance degradation in few-shot settings (discussed in Section 5.3). To handle this issue, we adopt a skip-connection design where the adapter parameters are initialized with zero-mean small Gaussian noise.
Adapter placement. Prior works on lightweight adaptation tune bias (Cai et al., 2020b) or embeddings (Lester et al., 2021a) of Transformers in
Figure 4: List explores several adapter placement choices (numbered positions in left) in standard Transformer architecture, with adapter design shown in right.
fully-supervised settings for improving parameter-efficiency with minimal performance loss. However, for few-shot settings, we note that adapter placement is critical to bridge the performance gap with that of a fully tunable model and the choices of tuning bias or embedding can result in upto  $10\%$  performance degradation (discussed in Section 5.3). To this end, we explore several choices of adapter placement (refer to Figure 4) corresponding to the most important transformer modules, namely, embedding, intermediate feedforward, output feedforward and attention module in every layer of the Transformer. Based on empirical experiments (refer to Section 5.3) across six diverse NLU tasks, we observe the feedforward output and attention modules to be the most important components for parameter-efficient adaption in few-shot settings.
Formally, consider  $\widetilde{\mathcal{D}}_K^{Train} = \{\widetilde{x}^l,\widetilde{y}^l\}$  to be the few-shot labeled data and  $\widetilde{\mathcal{D}}^U = \{\widetilde{x}_u\}$  to be the unlabeled data, where we transform the input sequences  $x$  to cloze-style input  $\widetilde{x}$  containing a single mask following the prompting strategy outlined in Section 2. We use the same pattern templates and verbalizers (output mapping from the task-specific labels  $\mathcal{V}$  to single tokens in the vocabulary  $\nu$  ) from traditional prompt-based FN works (Gao et al., 2021). Given the above adapter design and placement of choice with parameters  $\psi$  , a dataset  $\widetilde{\mathcal{D}}_K^{Train}$  with shots  $K$  , a PLM encoder enc with parameters  $\Theta_{\mathrm{PLM}}$  , where  $\Theta_{\mathrm{PLM}}\gg \psi$  , we want to perform the following optimization for efficient model adaptation:
$$
\psi \leftarrow \underset {\psi} {\arg \min } \mathcal {L} \left(\widetilde {\mathcal {D}} _ {K} ^ {T r a i n}; \Theta_ {\mathrm {P L M}}, \psi\right) \tag {2}
$$
4.3 Re-weighting Noisy Prompt Labels
Consider  $\{\hat{y}_n^{(t)}\}_{n = 1}^N$  to be the pseudo prompt-labels (for the masked tokens in  $\widetilde{x}_n^u\in \widetilde{X}$ ) from the
teacher  $(\Theta_{\mathrm{PLM}},\hat{\psi}_{\mathrm{tea}})$  in the  $t$  -th iteration where  $N$  is the number of unlabeled instances and  $\hat{\psi}_{\mathrm{tea}}$  represent the teacher adapter parameters. In selftraining, the student model is trained to mimic the teacher predictions on the transfer set. Consider  $\mathcal{L}(\hat{y}_n^{(t)},\mathrm{enc}(\widetilde{x}_n^u;\Theta_{\mathrm{PLM}},\psi_{\mathrm{stu}}^{(t)}))$  to be the loss of the student model with parameters  $(\Theta_{\mathrm{PLM}},\psi_{\mathrm{stu}}^{(t)})$  on the pseudo-labeled data in the  $t$  -th iteration, where  $\Theta_{\mathrm{PLM}}$  and  $\psi_{\mathrm{stu}}$  represent the PLM and the student adapter parameters respectively. In order to reduce error propagation from noisy pseudo-labels, we leverage meta-learning to re-weight them based on the student model loss on the validation set as our meta-objective. The intuition of meta re-weighting is to measure the impact or weight of a pseudo-labeled example given by its performance on the validation set. Since we do not have access to a separate validation set in the spirit of true few-shot learning, we leverage the labeled training set  $\widetilde{\mathcal{D}}_K^{Ttrain}$  judiciously for re-weighting. To this end, we leverage the idea of weight perturbation (Ren et al., 2018) to set the weight of pseudo-labeled example  $(\widetilde{x}_i^u,\hat{y}_i^{(t)})$  to  $\epsilon_i^{(t)}$  at iteration  $t$  as:
$$
\mathcal {L} _ {r} ^ {(t)} (\epsilon , \psi) = \frac {\sum_ {i = 1} ^ {N} \left[ \epsilon_ {i} ^ {(t)} \cdot \mathcal {L} \left(\hat {y} _ {i} ^ {(t)}, \operatorname {e n c} \left(\tilde {x} _ {i} ^ {u}; \Theta_ {\mathrm {P L M}} , \hat {\psi} _ {\mathrm {s t u}} ^ {(t - 1)}\right)\right) \right]}{N}. \tag {3}
$$
$$
\hat {\psi} _ {\mathrm {s t u}} ^ {(t)} (\epsilon) = \hat {\psi} _ {\mathrm {s t u}} ^ {(t - 1)} - \alpha \nabla \mathcal {L} _ {r} ^ {(t)} (\epsilon , \psi). \tag {4}
$$
where  $\alpha$  is the step size. Weight perturbation is used to discover data points that are most important to improve performance on the validation set. Optimal value for the perturbation  $\epsilon_i^{(t)*}$  can be obtained via minimizing student model loss on the validation set at iteration  $t$  as:
$$
\epsilon_ {i} ^ {(t) *} = \underset {\epsilon_ {i}} {\arg \min } \frac {\sum_ {i = 1} ^ {| \widetilde {\mathcal {D}} _ {K} ^ {T r a i n} |} \mathcal {L} \left(y _ {i} , \operatorname {e n c} \left(x _ {i} ; \Theta_ {\mathrm {P L M}} , \hat {\psi} _ {s t u} ^ {(t)} \left(\epsilon_ {i}\right)\right) \right.}{| \widetilde {\mathcal {D}} _ {K} ^ {T r a i n} |} \tag {5}
$$
To obtain a cheap estimate of the meta-weight at step  $t$ , we take a single gradient descent step on a mini-batch  $\widetilde{\mathcal{D}}^{(t)} \in \widetilde{\mathcal{D}}_K^{Train}$  as:
$$
u _ {i} ^ {(t)} = - \frac {\partial}{\partial \epsilon_ {i}} \left(\frac {\sum_ {i = 1} ^ {| \widetilde {\mathcal {D}} ^ {(t)} |} \mathcal {L} \left(y _ {i} , \operatorname {e n c} \left(\widetilde {x} _ {i} ; \Theta_ {\mathrm {P L M}} , \hat {\psi} _ {s t u} ^ {(t)} (\epsilon)\right)\right)}{| \widetilde {\mathcal {D}} ^ {(t)} |}\right) \tag {6}
$$
The weight  $w_{i}^{(t)}$  of  $(\widetilde{x}_i^u, \hat{y}_i^{(t)})$  at iteration  $t$  is set to be proportional to the negative gradient  $u_{i}^{(t)}$  to reflect the importance of pseudo-labeled samples. Samples with negative weights are filtered out since they could potentially degrade the student performance. Finally, we update student adapter parameters  $\psi_{\mathrm{stu}}$  while accounting for re-weighting as:
$$
\mathcal {L} ^ {(t)} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ w _ {i} ^ {(t)} \cdot \mathcal {L} \left(\hat {y} _ {i} ^ {(t)}, \operatorname {e n c} \left(\widetilde {x} _ {i} ^ {u}; \Theta_ {\mathrm {P L M}}, \hat {\psi} _ {\mathrm {s t u}} ^ {(t - 1)}\right)\right) \right]. \tag {7}
$$
4.3.1 Knowledge Distillation For Student Warmup