SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Discussion. State-of-the-art semi-supervised approaches typically rely on generating pseudo labels for unlabelled examples in order to train student and teacher models iteratively [4, 15, 95, 27, 94, 29]. However, this iterative process is prone to confirmation bias [83, 2, 31], which can result in error accumulation if the pseudo label is incorrect at any iterative step [49, 88, 31, 20]. Various efforts have been made to mitigate confirmation bias, such as using only high-confidence pseudo labels [80, 99, 12] or relying heavily on data augmentation [94, 21, 11]. While these efforts make the training process more sophisticated, the issue remains difficult to fully address [20, 77]. Our proposed method offers an alternative way to utilise pseudo labels different from previous semi-supervised approaches [98, 58]. Instead of relying on an iteration process with direct supervision signals from pseudo labels, we incorporate pseudo labels through continued pre-training with an unsupervised objective (i.e., MLM). While our proposed approach may not always outperform semi-supervised approaches across all benchmarks, it delivers highly competitive performance while significantly streamlining the process by removing the necessity for iteration and additional data augmentation. We will discuss the efficiency of the proposed PCP later (\$4.4). Additionally, PCP is orthogonal to these semi-supervised approaches and can be combined easily by initialising their backbone from the PCP checkpoint. In future work, we plan to investigate the more specific use cases where our proposed PCP may be preferred over these semi-supervised approaches.
4.4 Further Analysis
1. What is the lower bound of the model performance using the PCP? To understand the lower bound of PCP performance, we conduct additional analysis with two different configurations of pseudo labels in PCP: (1) all pseudo labels are incorrect; and (2) all labels are randomly selected. Figure 3 depicts the performance using different types of pseudo labels. We use the prompt-based FT without PCP (shown in yellow) and with PCP (shown in red) as baselines. Experimental results indicate that using incorrect pseudo labels (shown in blue) typically leads to inferior performance. In experiments using two prompt-based FT on 16 datasets, we find that using random labels leads to improved outcomes in 19 out of 32 scenarios. This suggests that PCP with random labels has over a  $50\%$  chance of improving the performance of prompt-based FT, indicating that the performance lower
Figure 4: (Left) The effect of different unlabelled data sizes using ROBERTA-LARGE. (Right) The effect of Scaling Laws, where ROBERTA-BASE (123M) and ROBERTA-LARGE (354M). All comparison approaches are trained with 16 examples per class for each dataset.
bound is satisfactory. Additionally, PCP with random labels improves the performance on sentence pair tasks in 8 out of 16 cases, while TAPT leads to poorer results in 15 of 16 cases (refer to Table 1). This suggests that PCP can be advantageous even when using random labels, providing benefits in scenarios where TAPT falls short. Interestingly, unlike prior study [60] on in-context learning [14], where LMs using random labels in demonstrations perform close to those using ground-truth labels, our results show that using pseudo labels assigned by a trained model (shown in red) consistently leads to the better performance, highlighting the importance of accurate pseudo labels.
2. What are the requirements of data size and computational resources for the PCP? To gain a deeper understanding of the efficacy of our proposed PCP method, we conduct additional analysis to determine the number of data points necessary for the PCP. Figure 4 (left) presents the performance of prompt-based FT methods, including both hard and soft variants, across four datasets. The prompt-based FT performance generally improves when the PCP is implemented with more than 1000 unlabelled examples, and some enhancements can be observed even with just 100 unlabelled examples. This indicates that contin
ued pre-training (both TAPT and PCP) is not necessarily computationally demanding and can be used efficiently even with only hundreds of training examples. In our experiments, performing the PCP on 1k unlabelled example takes less than 10 minutes using two 24GB NVIDIA 3090 GPUs, and all PCP performance achieved in §4.2 use fewer than 10k unlabelled examples. This is a stark contrast to the previous work [33] that pursued similar objectives (for parameter-efficient fine-tuning) to ours but utilised 10GB of English text data.
Dataset	Size	FT	+TAPT	+PCP
IMDB	23K	87.31.2	88.91.3	91.40.5
AG NEWS	100K	86.40.9	87.61.1	88.00.4
YELP REVIEW	250K	52.42.5	60.31.9	61.442.0
AMAZON REVIEW	250K	51.21.8	56.81.2	57.01.5
YAHOO! ANSWER	500K	64.90.8	64.91.1	69.01.4
Table 3: Test Results for prompt-based FT (soft) using ROBERTA-BASE with varying continued pre-training corpus sizes. Average Macro- $F_{1}$  with standard deviations are reported across five seeds. The model is trained on the IMDB dataset using 100 labelled examples and uses 200 labelled examples for other datasets. The best performance for each dataset is highlighted in blue.
#3. Power of scale. Our empirical analysis investigates the impact of increasing the backbone LM size on the model performance using the PCP. Figure 4 (right) shows the results of prompt-based FT methods, including hard and soft variants, trained using either TAPT or PCP, on four datasets. The performance of the PCP method largely improves as the backbone LM size expands, which aligns with the scaling laws observed in LMs [41, 37]. Furthermore, the PCP method consistently surpasses other baseline approaches, highlighting the advantages of the PCP across different model sizes.
SST-2	SST-5	MR	CR	MPQA	Subj	TREC	CoLA	Mean
Prompt FT	92.5	48.0	86.8	90.8	81.2	90.3	83.0	4.9	72.2
Prompt FT +PCP	93.9	50.7	89.8	92.0	88.3	94.9	88.6	21.5	77.5
Prompt FT +PCP (Labels Only)	93.7	50.8	87.7	91.3	85.1	94.3	85.7	-0.7	73.5
Prompt FT +PCP (Template Only)	90.7	43.5	88.6	92.6	82.0	95.1	84.1	0.7	72.2
Table 4: Ablation study on the inclusion of the template and labels in our proposed PCP. The test Results using soft prompt FT and ROBERTA-LARGE are reported. The best performance for each dataset is highlighted in blue.
SST-2	SST-5	MR	CR	MPQA	Subj	TREC	CoLA	Mean
CLS-based FT (1k steps) + TAPT	88.2	43.4	86.1	86.2	73.7	94.2	80.4	1.9	69.3
CLS-based FT (5k steps) + TAPT	89.6	43.4	86.7	87.0	72.9	94.6	79.0	1.7	69.4
Prompt FT (1k steps) + PCP	93.9	50.7	89.8	92.0	88.3	94.9	88.6	21.5	77.5
Table 5: Ablation study on the prolonged fine-tuning, where ROBERTA-LARGE is used as the backbone model. The test Results using CLS-based FT and soft prompt FT are reported. The best performance for each dataset is highlighted in blue.
#4. The impact of a larger continued pre-training corpus on the model performance using PCP and TAPT. Here we expand our investigation to whether the advantage of the proposed PCP approach persists as the size of the continued pre-training corpus increases. Table 3 presents the performance of prompt-based FT (soft), trained using either TAPT or PCP, across five datasets with varying sizes of unlabelled training examples. These experimental results are consistent with our findings in §4.2 and §4.3, showing that the proposed PCP approach consistently outperforms the model performance using the TAPT even when the larger corpus for continued pre-training is used.
#5. Ablation study on the label and template inclusion in PCP. To gain a deeper understanding of the individual contributions of pseudo labels and templates in our proposed PCP method, we conduct an additional ablation study, where we solely utilize pseudo labels or templates. This ablation study is carried out using soft prompt-based fine-tuning. As shown in Table 4, the experimental results reveals that using either labels or templates exclusively will hurt the model's performance compared to our proposed PCP method, highlighting the vital importance of integrating both templates and pseudo labels.
#6. The impact of prolonged fine-tuning on the model performance. To ascertain that the effectiveness of our proposed method is not simply due to an extended fine-tuning duration, we conduct additional experiments. We train CLS-based FT 5 times more steps (5k steps in total) from the TAPT checkpoint. As shown in Table 5, our results reveal that prolonged fine-tuning only brings about a marginal improvement of only  $0.1\%$  across the eight tasks. Notably, this still falls significantly short of our proposed method ( $8.1\%$  in absolute).
Prompt-based Approaches. In recent years, researchers have been exploring prompt-based approaches to improve the performance of fine-tuning. These approaches can be broadly divided into two research directions. The first direction, known as prompt-based FT, optimizes all parameters in LMs for better performance [73, 28, 52, 100], as discussed in §2. Adaprompt [22] improved the performance of hard prompt-based FT [73, 28] on single sentence tasks through conventional continued pre-training, which is generally consistent with our experimental results. The second direction is parameter-efficient fine-tuning (PEFT) [51, 68, 47, 81, 86], which aims to achieve competitive results while maintaining low computational costs. PPT [33] strives to improve the performance of PEFT [47] by further pre-training the T5 model [70], which pursues a similar idea as ours. However, this method relies on a series of hand-crafted and task-dependent designs for further pre-training, making it less adaptable to novel downstream tasks [86]. Furthermore, it demands a much larger training corpus, as discussed in §4.4. In contrast, our work offers a uniform design across all tasks and focuses on prompt-based FT. In future work, we plan to explore the compatibility of continued pre-training (including both TAPT and our proposed PCP) and PEFT methods.
Train LMs with Instructions/Templates. Our work is related to training LMs with templates. Recent studies [42, 3, 65, 91, 61, 72, 89, 59] have explored the idea of LMs training on a variety of NLP tasks with natural language instructions/templates, with the goal of generalizing to unseen tasks. Similar ideas, prompt transfer, have also been explored in the context of PEFT [33, 81, 86, 75], which seeks to learn an effective representation of the soft prompt for the target task by training on other tasks. In our approach, we transfer knowledge from task-related texts with prompt templates that are tailored to a single target task to LMs.
Semi-supervised Learning. Our work is related to semi-supervised learning [32, 19, 43], with the goal of utilising unlabelled data effectively. Continued pre-training followed by fine-tuning [39, 82, 35] is one type of semi-supervised approaches. While the benefits of continued pre-training are well acknowledged [6, 1, 56], it is commonly assumed that large amounts of data are necessary for continued pre-training [e.g., 50, 38, 33]. Contrarily, our research demonstrates that continued pre-training can improve performance using only a few hundred unlabelled samples. Self-training [98, 58] is another powerful semi-supervised approach, which typically uses student-teacher models to assign pseudo-labels to the unlabelled data [46, 44, 83, 62, 4, 15, 27, 94, 80, 29]. Our work offers an alternative way to use pseudo-labels without resorting to an iterative process, as discussed in §4.3.
6 Epilogue
Conclusion. This study challenges the widely accepted notion in NLP, showing that conventional continued pre-training can be detrimental to model performance, especially for sentence pair tasks and prompt-based FT. As an alternative, we propose Prompt-based Continued Pre-training (PCP), which consistently improves the performance of state-of-the-art prompt-based FT approaches over conventional continued pre-training. Additionally, our proposed PCP outperforms state-of-the-art semi-supervised approaches with a more streamlined process. Further analysis reveals that the advantages of PCP remain consistent across different sizes of models and datasets. This study emphasizes the importance of presenting both task-related texts and templates/instructions to LMs during pre-training for better fine-tuning performance on downstream tasks, contributing to the growing body of research on the optimisation of pre-training and fine-tuning strategies in NLP.
Limitations and Broader Impact. We outline several limitations inherent to our research:
- The scale of language models. Our experiments utilise relatively modestly-sized language models [53]. The implications of scaling up to more advanced language models, such as the Llama-2 [84] or the mixture-of-experts approach like GPT-4 [63], remains an open question. In the context of large language models, applying PCP with a full set of parameter updates for a specific task may not be justifiable in terms of computational costs. Future research could explore multi-task learning strategies or parameter-efficient continued pretraining.
- The architecture of language models. Our work is limited to encoder-only models [25, 53]. To generalize our findings, future research should investigate the effects of our method PCP on encoder-decoder [70] and decoder-only architectures [14].
- The diversity of tasks. Our evaluation is confined to text classification and regression tasks. Future research should investigate generative or multi-modal tasks, which may offer more comprehensive insights into the applicability of our methods PCP.
In addition, our work is based on pre-training and prompting methods for LMs. Previous works [8, 14, 7] have extensively discussed the risks and potential harms associated with LMs, including the amplification of undesirable biases learned from unlabelled training data [8, 5, 16]. The energy cost and carbon footprint for our work were approximately  $125\mathrm{kWh}$  and  $70\mathrm{kgCO_2e}$ , which are comparatively smaller than LM pre-training [25, 53, 14, 23].
The authors express their gratitude to the NeurIPS reviewers and area chairs for their insightful discussions. The authors are grateful to Xin Zhao for her contributions to proofreading. Zhengxiang Shi is funded by the Research Studentship from University College London (UCL).
[97] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics.
[98] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics.
[99] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, 2021.
[100] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations, 2022.
[101] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, page 649-657, Cambridge, MA, USA, 2015. MIT Press.
[102] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 18-24 Jul 2021.
The appendix is structured as follows:
Appendix §A provides a brief description for each dataset.
Appendix  $\S \mathbf{B}$  provides details of templates and label words used for each dataset.
Appendix §C presents a brief description of state-of-the-art four semi-supervised (self-training) approaches.
Appendix §D provides the supplementary experimental results to investigate the potential reasons for the ineffectiveness of CLS-based fine-tuning on the sentence pair tasks.
Appendix  $\S \mathbf{E}$  provides implementation details and hyperparameters for all comparison methods used in our experiments.
A Dataset
In this work, we use 21 popular datasets from previous few-shot learning and semi-supervised learning research.
For experiments in §4.2, we adhere to the approach in [28] and utilise 16 different datasets², including SST-2 [79], SST-5 [79], MR [67], CR [40], MPQA [92], Subj [66], TREC [85], CoLA [90], MNLI [93], SNLI [13], QNLI [71], RTE [24, 30, 10], MRPC [26], QQP³, and STS-B [17]. Consistent with prior research [28], our validation set comprises 16 examples per class from the aforementioned datasets. Additionally, we use 16 examples per class for the training set and the entire training set as the unlabeled set in the semi-supervised setting. We also utilise the full training set for training purposes in the fully supervised setting. For sentence pair tasks, we select at most 10k examples for continued pre-training to reduce the computational costs.
For experiments in §4.3, we follow the setup in [77] and utilise 5 different datasets, including IMDB [55], AG NEWS [101], YELP REVIEW $^{4}$ , YAHOO! ANSWER [18], and AMAZON REVIEW [57]. Refer to the dataset statistics in Table 6. Our validation set comprises 1,000 examples for each dataset.
B Templates for Prompt-based FT
Here we introduce the templates used in two state-of-the-art prompt-based FT approaches for each dataset. For "Prompt-based FT (hard)", we use high-quality manual or auto-generated prompts and label words for each task from previous works [73, 28]. For "Prompt-based FT (soft)", we use the STS-2 template for all single sentence tasks and STS-B template for all sentence pair tasks, while the label words for each task follow the description in Table 7.
C ST Frameworks
FIXMATCH. FIXMATCH [80] generates artificial labels using both consistency regularization and pseudo-labelling, where the artificial labels are produced based on weakly-augmented unlabelled data. These artificial labels are then used as targets to train the model on strongly-augmented unlabelled data. FIXMATCH only retains an artificial label if the model assigns a high probability to one of the possible classes.
DASH. DASH [96] extends FIXMATCH by introducing a mechanism with a dynamically adjusted threshold of loss to select a subset of training examples from the unlabelled data for performing SSL.
$^{2}$ https://github.com/princeton-nlp/LM-BFF/blob/main/data/download_dataset.sh
<sup>3</sup>https://www.quora.com/q/quoradata/
4 https://www.yelp.com/dataset
Single Sentence Tasks
Dataset	|y|	L	#Train	#Test	Type	Labels (classification tasks)
SST-2	2	19	6,920	872	Sentiment	positive, negative
SST-5	5	18	8,544	2,210	Sentiment	v. pos., positive, neutral, negative, v. neg.
MR	2	20	8,662	2,000	Sentiment	positive, negative
CR	2	19	1,775	2,000	Sentiment	positive, negative
MPQA	2	3	8,606	2,000	Opinion Polarity	positive, negative
Subj	2	23	8,000	2,000	Subjectivity	subjective, objective
TREC	6	10	5,452	500	Question cls.	abbr., entity, description, human, loc., num.
CoLA	2	8	8,551	1,042	Acceptability	grammatical, not_grammatical
IMDB	2	149	8,000	1,000	Movie Review	positive, negative
AG NEWS	2	37	8,000	1,000	News Topic	world, sports, business, sci/tech
YELP REVIEW	2	134	8,000	1,000	Review Sentiment	1, 2, 3, 4, 5
AMAZON REVIEW	2	79	8,000	1,000	Review Sentiment	1, 2, 3, 4, 5
YAHOO! ANSWER	2	32	8,000	1,000	Topic Classification	culture, science, health, education, computer, sports, business, music, family, politics
Sentence Pair Tasks
Dataset	|Y|	L	#Train	#Test	Type	Labels (classification tasks)
MNLI	3	22/11	392,702	9,815	NLI	entailment, neutral, contradiction
SNLI	3	14/8	549,367	9,842	NLI	entailment, neutral, contradiction
QNLI	2	11/30	104,743	5,463	NLI	entailment, not_ entailment
RTE	2	49/10	2,490	277	NLI	entailment, not_ entailment
MRPC	2	22/21	3,668	408	Paraphrase	equivalent, not_equivalent
QQP	2	12/12	363,846	40,431	Paraphrase	equivalent, not_equivalent
STS-B	R	11/11	5,749	1,500	Sent. Similarity	-
Table 6: The datasets evaluated in this work.  $|\mathcal{Y}|$ : # of classes for classification tasks (with one exception: STS-B is a real-valued regression task over the interval [0, 5]). L: average # of words in input sentence(s). Note that we only sample examples from the original training set in our few-shot experiments.
Single Sentence Tasks
Task	Template	Label words
SST-2	<S1> It was [MASK] .	positive: great, negative: terrible
SST-5	<S1> It was [MASK] .	v-positive: great, positive: good, neutral: okay, negative: bad, v.negative: terrible
MR	<S1> It was [MASK] .	positive: great, negative: terrible
CR	<S1> It was [MASK] .	positive: great, negative: terrible
MPQA	<S1> is [MASK] .	positive: positive, negative: negative
Subj	<S1> This is [MASK] .	subjective: subjective, objective: objective
TREC	[MASK] : <S1>	abbreviation: Expression, entity: Entity, description: Description human: Human, location: Location, numeric: Number
COLA	<S1> This is [MASK] .	grammatical: correct, not_grammatical: incorrect
IMDB	<S1> It was [MASK] .	positive: great, negative: terrible
AG NEWS	<S1> It was [MASK] .	World: world, Sports:sports, Business: business, Sci/Tech: tech
YELP REVIEW	<S1> It was [MASK] .	0: 0, 1: 1, 2: 2, 3: 3, 4: 4
AMAZON REVIEW	<S1> It was [MASK] .	0: 0, 1: 1, 2: 2, 3: 3, 4: 4
YAHOO! ANSWER	<S1> It was [MASK] .	culture: culture, science: science, health: health, education: education computer: computer, sports: sports, business: business music: music, family: family, politics: politics
Sentence Pair Tasks
Task	Template	Label words
MNLI	<S1> ? [MASK] , <S2>	entailment: Yes, neutral: Maybe, contradiction: No
SNLI	<S1> ? [MASK] , in this case <S2>	entailment: Yes, neutral: Maybe, contradiction: No
QNLI	<S1> ? [MASK] , <S2>	entailment: Yes, not_ entailment: No
RTE	<S1> ? [MASK] , I think that <S2>	entailment: Clearly, not_ entailment: Yet
MRPC	<S1> [MASK] , <S2>	equivalent: Yes, not_equivalent: No
QQP	<S1> [MASK] , <S2>	equivalent: Yes, not_equivalent: No
STS-B	<S1> [MASK] , <S2>	yu: Yes, yi: No
Table 7: Templates and label words used for "Prompt-based FT (hard)". We use the STS-2 and STS-B template for all single sentence tasks and sentence pair tasks using "Prompt-based FT (soft)", respectively.
FLEXMATCH. FLEXMATCH [99] also extends FIXMATCH by introducing the concept of curriculum learning [9] to flexibly adjust thresholds for different classes at each time step and select unlabelled data and their pseudo labels that are more likely to be informative.
ADAMATCH. ADAMATCH [12] aims to solve domain adaptation problems in SSL and build a high-accuracy model that trains on and tests on different data distributions. ADAMATCH builds on FIXMATCH and introduces a relative confidence threshold and a modified distribution alignment from [11].
D Supplementary Experiment
In this section, we investigate why TAPT does not work on sentence pair tasks. We have evaluated three possible explanations for TAPT's ineffectiveness on sentence pair tasks: (1) dataset size for continued pre-training, (2) sentence pairs with higher similarity than what was observed in pre-training data, and (3) lack of separation within sentence pairs. Our experimental results suggest that the ineffectiveness of TAPT on the sentence pair tasks is not an isolated incident but a recurring issue. Below we discuss each setting in detail.