SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
Zhengxiang Shi
zhengxiang.shi.19@ucl.ac.uk
Aldo Lipani
aldo.lipani@ucl.ac.uk
Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to  $20.1\%$  absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabeled examples. Additionally, prompt-based FT with the PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets. Code is available at https://github.com/ZhengxiangShi/PowerfulPromptFT.
Pre-training language models (LMs) [25, 53, 69] over massive unlabelled data and then fine-tuning on task-specific labelled data for the specific downstream task offer large performance gains across NLP tasks. In this study, we re-visit the commonly held belief in NLP [39, 82, 35] that continued pre-training LMs on either task-specific data [1, 56] or in-domain data [54, 97] is generally beneficial for improving the performance of fine-tuning (FT) on downstream tasks. As shown in Figure 1, our experiments on eight single-sentence tasks
and eight sentence-pair tasks in both semi- and fully-supervised settings reveal that conventional continued pre-training on task-specific data [35], known as task adaptive pre-training (TAPT) (see
Figure 1: Mean performance of CLS- and prompt-based FT across 16 NLP tasks when trained by themselves or in combination with either TAPT or our proposed PCP in the semi-supervised setting. Please refer to Table 1 for details.
Figure 2: The overview of Prompt-based Continued Pre-training (g), in comparison to conventional continued pre-training (e) and instruction tuning (f), along with fine-tuning methods (a,b) and continued pre-training techniques (c,d). The verbalizer functions as a mapping from the task label space to individual words. We use masked language modelling for illustrative purposes, where <mask> represents a masked token in the LM vocabulary.
Figure 2e): (1) can lead to a substantial drop in performance of CLS-based FT (see Figure 2a) on sentence-pair tasks; and (2) may perform unstably across different tasks for prompt-based FT (see Figure 2b), which is typically considered a better alternative to CLS-based FT by previous studies [73, 45] ( $\S 4.2$ ). These findings suggest that exclusively presenting task-related texts to LMs through continued pre-training may not be the most effective approach for improving the performance of FT in the aforementioned situations.
Recent research [42, 3, 65, 91, 61, 72, 89, 59] on cross-task generalization has demonstrated the impressive improvement on zero-shot or few-shot learning capabilities of LMs (see Figure 2f). These studies suggest that presenting appropriate instructions/prompt templates to LMs through training on a range of NLP tasks improves their downstream performance on held-out tasks. Although these works train LMs with different objectives from pre-training phases, we interpret "fine-tuning LMs on a range of NLP tasks" as a special type of continued pre-training. Therefore, we hypothesize that presenting both task-related texts and instructions/prompt templates to LMs can relieve the above-mentioned issues for conventional continued pre-training and be beneficial for the target task performance. Rather than improve the generalizability of the LMs with supervised objectives, our work places a greater emphasis on enhancing specific target task performance with unsupervised pre-training objectives.
In this work, we propose Prompt-based Continued Pre-training (PCP) ( $\S 3$ ), which integrates instructions/prompt templates into task-related texts with golden or pseudo labels (see Figure 2g). Our experiments demonstrate that PCP consistently improves the performance of state-of-the-art prompt-based FT approaches [28, 100] in both semi- and fully-supervised settings, covering both single sentence tasks and sentence pair tasks, and that the performance gains from PCP exceed those from conventional continued pre-training (TAPT) by a substantial margin ( $\S 4.2$ ). In the most favourable case, PCP boosts the performance of prompt-based FT by more than  $20\%$  absolute while TAPT results in a  $9.2\%$  performance decline. Furthermore, our results show that PCP outperforms state-of-the-art semi-supervised approaches [80, 94, 96, 99, 12] with greater simplicity, eliminating the need for an iterative process and extra data augmentation ( $\S 4.3$ ). Additionally, our analysis suggests that the PCP can efficiently improve the performance of prompt-based FT with only hundreds of unlabelled examples. Meanwhile, our analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets ( $\S 4.4$ ). Finally, we outline the limitations of our study and suggest avenues for future research ( $\S 6$ ).
In summary, the main contributions of this paper are as follows:
- Our study empirically demonstrates that conventional continued pre-training might not be as effective as initially thought and can even negatively impact fine-tuning performance, particularly in sentence pair tasks or when utilising prompt-based FT;
- Our evaluation on 21 classification and regression NLP tasks shows that our proposed method PCP provides a superior option to conventional continue pre-training for prompt-
based FT. This approach consistently yields performance improvements in diverse model and dataset settings, even with only a few hundred unlabelled examples. Moreover, it can outperform state-of-the-art semi-supervised approaches with greater simplification;
- Our result shows the effectiveness of presenting both task-related texts and templates/instructions to the LMs through unsupervised pre-training objectives on improving the performance of prompt-based FT on downstream tasks. To the best of our knowledge, this is the first work to perform instruction tuning via unsupervised objectives.
Suppose that we focus on the LMs trained with the masked language modelling (MLM) objective [25, 53]. Let  $X = \{x_{1}, x_{2}, \dots, x_{N}\}$  be a sequence of tokens, where  $N$  represents the total number of tokens. LMs are designed to encode the input text  $X$  into a corresponding sequence of hidden vectors  $\{\mathbf{h}_i \in \mathbb{R}^d\}$ . As shown in Figure 2a, the conventional CLS-based FT [25, 35, 76] trains the output vector  $\mathbf{h}$  corresponding to the [CLS] token with an additional head layer (e.g., an MLP layer). However, there is a discrepancy between the pre-training objective (see Figure 2c) and the CLS-based FT objective, which has led to research on prompt-based techniques for better LM performance.
The prompt-based FT is formulated as a MLM problem where the objective is to predict masked tokens [73, 74]. Specifically, the input text  $X$  is conditioned with a specific prompt template  $\tilde{X} = \mathcal{T}(X)$ , which includes one special token [MASK]. The prompt-based FT then maps the output vector associated with the [MASK] token to a label word. The probability of predicting class  $y \in \mathcal{V}$  is computed as:
$$
p (y | X) = p \left(\left[ \operatorname {M A S K} \right] = \mathcal {M} (y) | \tilde {X}\right), \tag {1}
$$
where the verbalizer  $\mathcal{M}:\mathcal{V}\to \mathcal{V}$  is a mapping from the task label space to individual words in the vocabulary  $\mathcal{V}$ .
Prompt-based FT can use either hard or soft prompt templates  $\mathcal{T}$ , with label words potentially being a part of the prompt templates as well [36, 100]. Hard prompt template [73, 28, 78] requires careful designs of prompts and label words for each task. The use of hard prompts, however, was found to be sub-optimal and sensitive to the choice of the prompt [102, 52]. Soft prompt [52, 100] was then proposed to use unused tokens from the vocabulary  $\nu$  or additional tokens as tuneable embeddings for prompt template and can be directly trained with the task-specific supervision. This design allows the token embeddings in the prompt template to be updated independently of specific word embeddings after initialization, thus reducing the effort of searching for prompt templates and label words.
3 Our Approach: Prompt-based Continued Pre-training (PCP)
In this section, we introduce the proposed method, Prompt-based Continued Pre-training (PCP), which aims to improve the performance of LMs on downstream tasks through continued pre-training with prompt templates, as shown in Figure 2g. Let  $L \triangleq \{(X_1, y_1), \ldots, (X_n, y_n)\}$  denote  $n$  labelled examples and  $U \triangleq \{X_1', \ldots, X_m'\}$  denote  $m$  unlabelled examples. Our approach consists of two main steps, as described below.
Step 1: Construct Continued Pre-training Corpus. Initially, we select a model  $F$ , pre-trained with the MLM objective and parameterized by  $\Theta$ . We then train this model using the prompt-based FT, minimizing the target loss function  $\ell$  on the labelled examples  $L$ , as illustrated in Figure 2b:
$$
\mathcal {L} (L) = \sum_ {X _ {i}, y _ {i} \in L} \ell \left(y _ {i}, F \left(\mathcal {T} \left(X _ {i}\right), \Theta\right)\right), \tag {2}
$$
Next, we use the trained model  $F$  with the learned parameters  $\Theta'$  to generate predictions (termed "pseudo-labels") on the unlabelled samples  $U$ :
$$
y _ {i} ^ {\prime} = F \left(\mathcal {T} \left(X _ {i} ^ {\prime}\right), \Theta^ {\prime}\right), \tag {3}
$$
For each text example  $X$  and its associated (golden or pseudo) label  $y$ , we create an example for our proposed PCP as  $X^{pcp} = \mathcal{T}(X, \mathcal{M}(y))$ , where the original [MASK] position is substituted with  $\mathcal{M}(y)$ . This results in a new corpus,  $\mathcal{C} = \{X_{i}^{pcp}\}_{i=1}^{n+m}$ . In the fully-supervised setting,  $m = 0$  and all examples use the golden labels.
Step 2: Perform continued pre-training and prompt-based FT. We then proceed to further pre-train another model  $G$ , parameterized by  $\Theta$ , using the MLM objective on the newly generated corpus  $\mathcal{C}$ , to obtain the PCP checkpoint  $\Phi$  (see Figure 2d). Finally, we train model  $G$ , initialised by  $\Phi$ , using Equation 2 with prompt-based FT for downstream tasks.
In comparison to conventional continued pre-training, PCP does not require any modification for the model architecture or training process. The sole difference is the addition of a few extra tokens to the input text during continued pre-training. This modification does not hinder the efficiency of the method, i.e., both conventional continued pre-training and PCP maintain equal levels of efficiency. In this study, we primarily focus on LMs pre-trained with the MLM objective [53]. It is noteworthy to mention that comprehensive exploration of other architectures [25, 70, 14, 65] remains an avenue for future research. Nonetheless, considering prompt-based fine-tuning approaches [52, 47, 51] have already been adapted for different model architectures and pre-training objectives [25, 70, 14, 65]. This implies that extending our method to alternative architectures should be a feasible undertaking.
In this section, we evaluate the proposed method PCP by comparing it with conventional continued pre-training and four state-of-the-art semi-supervised approaches. We assess their relative performance across 21 different classification and regression NLP tasks, including single-sentence and sentence-pair tasks. We conduct additional analysis concerning the performance lower bound of PCP and the effectiveness of the PCP across varying datasets and model sizes.
Datasets. Our study conducts a comprehensive analysis of 21 NLP datasets, including classification and regression tasks. Following previous studies [28, 36, 100] on prompt-based FT, we derive 8 single-sentence tasks and 8 sentence-pair English tasks from the GLUE benchmark [87], SNLI [13], and 6 other widely used sentence classification tasks (i.e., SST-5, MR, CR, MPQA, Subj, TREC). Additionally, we use 5 popular benchmarks for semi-supervised learning from previous research [34, 21, 94, 48, 29, 77], including IMDB [55], AG NEWS [101], YELP REVIEW $^{1}$ , YAHOO! ANSWER [18], and AMAZON REVIEW [57]. See dataset details in Appendix §A. We train the model with two different settings: (1) fully-supervised setting, where we train the model with the full training set; and (2) semi-supervised setting, where we sample the same amount of labelled data per class from the full training set. We re-sample the labelled data using the same five seeds for all comparison approaches and report the average performance with an error bar.
All Comparison Approaches. In our study, we mainly experiment using the ROBERTA-BASE (125M) and the ROBERTA-LARGE (355M) models. We utilise the conventional CLS-based FT and two state-of-the-art prompt-based FT approaches: (1) "CLS-based FT": fine-tuning with the [CLS] token representation with an extra MLP layer; (2) "Prompt-based FT (hard): fine-tuning with high-quality manual or auto-generated prompts and label words [73, 28]; and (3) "Prompt-based FT (soft): fine-tuning with soft prompts using additional tokens for both templates and label words [100]. Since the objective of soft prompt FT is to minimize the reliance on human-designed templates, we unify the template for all tasks here. See the template specifics used for each dataset in Appendix §B. We train these three types of FT approaches from three different types of checkpoints to evaluate their relative effectiveness: (i) the off-the-shelf ROBERTA-LARGE checkpoint; (ii) the task-adaptive pre-training (TAPT) checkpoint [35] (represents the conventional continued pre-training). For sentence pair tasks, we concatenate the two sentences as an input example; and (iii) the proposed PCP checkpoint, obtained in §3. For both (ii) and (iii), we perform MLM on all full training sets except MNLI, MNLI-mm, SNLI, QNLI, and QQP, where we select up to 10k unlabelled examples from the full training sets (see supplementary experiments on the full training sets in Appendix §D). Additionally, we compare the proposed PCP with four state-of-the-art semi-supervised approaches, including FixMatch [80], Dash [96], FlexMatch [99], and AdaMatch [12] (see descriptions of these approaches in Appendix §C), where back-translation [64] is used for data augmentation as previous works [94, 77] and prompt-based FT (hard) is used as the backbone. See hyperparameter and implementation details in Appendix §E.
1https://www.yelp.com/dataset
Table 1: Comparison between the PCP and conventional continued pre-training (TAPT) using ROBERTA-LARGE. The summary highlights the percentage of positive impact brought by the PCP and TAPT. The mean and standard deviation on test sets are reported over 5 different seeds. In semi-supervised learning, 16 examples per class are used for training, in line with previous studies [28, 52, 100]. Green and red arrows indicate changes with respect to the FT baselines that do not use TAPT or PCP.  $\dagger$  represents that no training examples are used. Three extra baselines sourced from [28] are included, where "Majority" refers to the majority class, and "in-context learning" indicates the usage of in-context learning [14] with ROBERTA-LARGE, without updating any parameters.
4.2 Comparison of the PCP and conventional continued pre-training
Table 1 presents and summarises our experimental results on 8 single-sentence tasks and 8 sentence-pair tasks. Below we delve deeper into our two major findings.
#1. TAPT is not consistently beneficial for sentence pair tasks, nor when prompt-based FT is employed. Initially, we re-visit the impact of TAPT (representing the conventional continued pre-training) on the CLS-based FT, as shown in Table 1. Our experimental results align with earlier
Method	IMDB	AG NEWS	YELP REVIEW	YAHOO! ANSWER	AMAZON REVIEW	Mean
20	100	40	200	40	200	40	200	40	200
DASH [96]	93.340.7	93.300.6	85.002.9	87.900.3	47.442.2	58.851.0	60.074.7	66.460.9	44.092.6	53.951.0	69.04
FIXMATCH [80]	95.260.4	94.280.5	85.441.1	88.210.4	47.261.2	58.510.3	61.566.9	68.370.7	44.262.0	52.331.7	69.55
FLEXMATCH [99]	95.220.3	94.840.4	85.331.4	88.570.6	50.602.5	58.341.9	58.093.7	66.431.3	45.483.1	54.191.1	69.71
ADAMATCH [12]	95.200.5	94.940.1	85.792.1	88.720.8	50.422.7	58.951.9	63.680.7	68.090.8	44.663.4	53.050.6	70.35
Prompt-based FT (hard)	86.782.1	89.521.6	84.871.1	86.990.3	46.694.2	58.270.7	60.631.5	66.941.1	44.341.5	57.010.4	68.20
+ PCP (ours)	92.491.2	94.240.9	87.061.0	88.940.4	52.924.5	63.151.3	65.581.8	70.220.9	53.443.0	59.641.6	72.77
Prompt-based FT (soft)	88.142.9	90.801.5	85.651.3	87.660.3	45.433.4	57.121.0	61.181.5	67.850.8	44.523.6	55.031.5	68.34
+ PCP (ours)	93.531.5	94.360.7	87.260.8	88.960.6	50.663.3	62.921.0	65.261.5	70.030.9	52.783.2	59.160.8	72.49
Prompt-based FT (hard)†	95.60	91.06	68.71	74.30	63.85	78.70
Prompt-based FT (soft)†	95.50	91.10	69.63	75.66	63.32	79.04
Table 2: Comparison between the PCP and four semi-supervised approaches using ROBERTA-LARGE. Each dataset is evaluated with two different labelled data sizes and full training set is used as unlabelled data.  $\dagger$  indicates that full training set is used as the labelled data. We report the average Macro- $F_{1}$  score on the test set across five seeds, with standard deviations as subscripts. For each column, blue represents the best performance and orange stands for the second-best performance.
studies [39, 35, 77], showing that TAPT generally improves the performance of the CLS-based FT on 7 out of 8 single sentence tasks in both semi-supervised and full-supervised setting. However, intriguingly, we observe that TAPT negatively affects the performance of CLS-based FT on 6 out of 8 sentence pair tasks, as summarised in Figure 1 and the at the bottom of Table 1. This finding implies that conventional continued pre-training (TAPT) may not be beneficial for sentence pair tasks.
Moreover, our investigation reveals that TAPT may negatively affect prompt-based FT. Specifically, in the fully supervised setting, TAPT results in reduced performance on 11 out of 16 tasks for prompt-based FT (hard) and on 9 out of 16 tasks for prompt-based FT (soft). In the most favourable scenario, TAPT enhances the performance of prompt-based FT (soft) from  $73.9\%$  to  $79.9\%$  on the QQP dataset. Conversely, in the least favourable situation, TAPT diminishes the performance of prompt-based FT (hard) from  $54.7\%$  to  $44.0\%$  on the CoLA dataset. In the semi-supervised setting, TAPT leads to a decline in performance on 12 out of 16 tasks for both prompt-based FT (hard) and prompt-based FT (soft) (see the summary of results in Figure 1 and the at the bottom of Table 1). Particularly, for sentence pair tasks, TAPT results in an average absolute decrease of  $9.5\%$  in performance for prompt-based FT. These results suggest that the effectiveness of TAPT varies across different tasks and cannot be universally applied. We conduct additional experiments to confirm the limitations of TAPT persist across different sizes of the pre-training corpus in Appendix D.
2. PCP offers consistent and substantial improvements in both semi- and fully-supervised settings. As depicted in Table 1, our experiments covering 16 datasets in both semi- and fully-supervised settings, including single sentence tasks and sentence pair tasks, reveal that (1) PCP consistently boosts the performance of prompt-based FT; and that (2) the performance gains achieved by PCP consistently exceed those obtained by TAPT by a substantial margin. Specifically, compared to prompt-based FT, PCP leads to more than a  $1.0\%$  average absolute improvement in the fully-supervised setting and contributes to an average absolute performance boost of  $6.8\%$  in the semi-supervised setting across 16 tasks. Compared to TAPT, PCP yields over a  $1.8\%$  average absolute improvement in the fully-supervised setting and contributes to an average absolute performance increase of  $11.2\%$  in the semi-supervised setting across 16 tasks. Notably, PCP can produce considerable gains in certain datasets. For instance, it elevates the performance of prompt-based FT (hard) from  $7.2\%$  (Matthews Correlation Coefficient) to  $25.0\%$ , while TAPT even reduces the performance of the prompt-based FT. Additionally, PCP improves the performance of prompt-based FT (soft) on the QNLI dataset from  $64.2\%$  to  $84.3\%$  with  $31\%$  improvement, while TAPT leads to a  $9.2\%$  absolute performance decline. We attribute the improvements to presenting the prompt template to the LMs through the further pre-training phrase, which implies that merely showing task-related texts to the LMs may not be the optimal approach for prompt-based FT.
4.3 Comparison of the PCP and state-of-the-art semi-supervised approaches
Table 2 presents our experimental results on five datasets, comparing the proposed PCP with state-of-the-art semi-supervised approaches. Below we delve deeper into our main finding with a discussion.
The proposed PCP outperforms state-of-the-art semi-supervised approaches on 4 out of 5 tasks. As shown in Table 2, our proposed PCP approach with either hard or soft variants of prompt-based
Figure 3: The performance lower bound of the PCP, where "wrong labels" indicates that all labels in the PCP are incorrect and "random labels" indicates that all labels in the PCP are randomly selected. For each dataset, 16 examples per class are used as labelled data and the full training set is used as unlabelled data. The mean performance on test sets is reported over 5 different seeds.
FT outperforms the best-performing semi-supervised approaches on 4 out of 5 datasets. Notably, the prompt-based FT (hard) with the PCP outperforms the best performing semi-supervised approaches (FLEXMATCH) with an absolute  $5.5\%$  Macro- $F_{1}$  score on the AMAZON REVIEW dataset when 200 labelled training examples are used. While the best performing semi-supervised approach, FIXMATCH, outperforms PCP by  $1.7\%$  in absolute value on the IMDB dataset using 20 labelled examples, the performance discrepancy narrows as the number of labelled training examples increases. Overall, the prompt-based FT (hard) and (soft) with the PCP outperform all these semi-supervised approaches with an average absolute performance improvement of more than  $2\%$  across various datasets and labelled dataset sizes, demonstrating the effectiveness of our proposed approach.