Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
Authors: Zhengxiang Shi
Aldo Lipani
Venue: NeurIPS 2023 poster (2023)
Keywords: Continued Pre-training
Prompt-based Fine-tuning
Language Models
TL;DR: We challenge "continued pre-training typically improves fine-tuning performance" and propose a new approach. It consistently improves the performance of prompt-based fine-tuning (up to 20.1% absolute) and outperforms SOTA semi-supervised methods.

Abstract: Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to 20.1% absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabelled examples. Additionally, prompt-based FT with PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets.

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
Zhengxiang Shi
Aldo Lipani
University College London
University College London London, United Kingdom aldo. lipaniiiiii @ucl. ac uk
London, United Kingdom zhengxiang shi. 19@ucl....m ac uk
Abstract
Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pale tasks in both semi-supervised and supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks when prompt-based FT used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task- related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations 21 benchmarks demonstrate that the PCP consistently improves the performance state-of oftte-of-the-att prompt-based FT approaches (up to 20.1% absolute) in both semi- supervised and bully-superviee settings, even with only hundreds of unlabelled examples. Additionally, prompt- based FT with the PCP outperforms state-of-the- art semi-supervised approaches with greater simplicity. eliminating the need for iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages PCP persist across different sizes of models and datasets. Code is available https:/ / thub. cww.com///////// ZZengx www.annggooooeeaa wwwaa//ffoooroottrPPPPPe PromptFT.
1 Introduction
Signie Sentence
Pre-training language models (LMs) [25, 53, 69] over massive unlabelled data and then fine- tuning on task-specific labelled data for the specific downstream task offer large perfor- mance gains across NLP tasks. In this study, we re-visit the commonly held belief in NLP [39, 82, 35] that continued pre- pre-training LMs on either aask-specific data [1, 56] or in-domain data [54, 97] is generally beneficial for im- proving the performance of fine-tuning (FT) on downstream tasks. As shown in Figure 1, our experiments on eight single-sentence tasks and eight sentence-pait tasks in both semi- and fully-supervised settings reveal that conventional continued pre-training on task-specific data [35], known as task adaptive pre-training (TAPT) (see
Peir Tasks
60
Average Performance (%)
CLS based
Prompt-based
CL5-based
Prompt based FT+ TAPT
Prompt- Prompt-based FT+PCP (ours)
Figure I: Mean performance CLS- and prompt- based FT across 16 NLP tasks when trained by themselves or combination with either TAPT or proposed PCP in the semi-supervised setting. Please refer to Table for details.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Verbo bor: Roativ
[e] Conventional Continued Pro Pre-training (e.g. TAPT)
Encoder
CLS Head
Encoder
Train Lts taxt related to Task A using the pin- trahing objocsive
CLS based
LMS
Fine-tunin Tosk
Ireelly like this movie.
really like this movie. It was <marks
(9) Instruction Turing (n.g.
(x) CLS-based Fine-tuning
(b) Prompt-baed Fine-tuning
realt,
LMs
intructions/teeee on
Inferncc on
Enooder
LM
Encoder
(s) Prompt-based Continued Pre-training (Ouns)
Lemasky like this emasios
<masics like this <masks. If was positive
Train LMs text relatoci Trok templates using the tra nrng
Prompph-bowwe the turning on Task A
LMs
(s) Masked Language Modelling
[d] Promptod Masked Language Modelling
objectio
Midel Layers
Prompt Tokens
Go Pooudoo Labele
Mode Traning
Frozen
Temp 2905
Texts
The overview of Prompt-based Continued Pre-training (g), in comparison conven- tional continued pre-training (e) and instruction tuning (f), along with fine-tuning methods (a.b) and continued pre-training techniques (c,d). The verbalizer functions as a mapping from the task label space to individual words. We use masked language modelling for illustrative purposes, where <mask> represents a masked token in the LM vocabulary.
Figure 2:
Figure 2e): (1) can lead to a substantial drop in performance of CLS-based FT (see Figure 2a) on centence-pair tasks; and (2) may perform unstably across different tasks for prompt-based FT (see Figure 2b), which is typically considered a better alternative to CLS-based FT by previous studies [73, 45] (64.2). These findings suggest that exclusively presenting task-related texts to LMs through continued pre-training may not be the most effective approach for improving the performance of FT in the aforementioned situations.
Recent research [42, 3, 65, 91, 61, 72, 89, 59] on cross-task generalization has demonstrated the impressive improvement on zero-shot few-shot learning capabilities of LMs (see Figure 20). These studies suggest that presenting appropriate instructions/prompt templates to LMs through training on range of NLP tasks improves their downstream performance on held-out tasks. Although these works train LMs with different objectives from pre-training phases, we interpret "fine-tuning LMs on a range of NLP tasks" as a special type of continued pre-traing pre-training Therefore, we hypothesize templates to LMs can relieve
that presenting both task-related texts and instructionsprrome the mentione issues for conventional continued pre-training and be beneficial for the target task performance. Rather than improve the generalizability of the LMs with supervised objectives. our work places a greater emphasis on enhancing specific target task performance with unsupervised pre-training objectives.
In this work, we propose Prompt-based Continued Pre-training (PCP) ($3), which integrates in- structions/prompt templates into task-related texts with golden or pseudo labels (see Figure 2g). Our experiments demonstrate that PCP consistently improves the performance of state-of-the-art boospt-based FT approaches [28, 100] in both semi- and fully-supervised settings, covering both single sentence tasks and sentence pair tasks, and that the performance gains from PCP exceed those from conventional continued pre-training (TAPT) by a substantial margin ($4.2). In the most favourable case, PCP boosts the performance prompt-based FT by more than 20% absolute while TAPT results in a 9.2% performance decline. Furthermore, our results show that PCP outperforms ofte--of-the-ar
semi-supervised approaches [80, 94, 96, 99, 12] with greater simplicity, eliminating the need for an iterative process and extra data augmentation (54.3). Additionally, our analysis sug- gests that the PCP can efficiently improve the performance of prompt-based FT with only hundreds of unlabelled examples. Meanwhile, our analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets ($4.4). Finally, we outline the limitations of our study and suggest avenues for future research ($6).
In summary, the main contributions of this paper are as follows:
Our study empirically demonstrates that conventional continued pre-training might not be effective as initially thought and can even negatively impact fine-tuning performance, particularly in sentence pair tasks or when utilising prompt-based FT;
Our evaluation on 21 classification and regression NI.P tasks shows that our proposed method PCP provides superior option to conventional continue pre-training for prompt-

---

based FT. This approach consistently yields performance improvements in diverse model and dataset settings, even with only a few hundred unlabelled examples. Moreover, it can outperform state-of-the-ar semi-supervised approaches with greater simplification;
Our result shows the effectiveness of presenting both task-related texts and templates/instrue- tions to the LMs through unsupervised pre-training objectives improving the performance of prompt-based FT on downstream tasks. To the best of our knowledge, this is the first work perform instruction tuning via unsupervised objectives.
2 Background
Suppose that we focus on the LMs trained with the masked language modelling (MLM) objective [25, 53]. Let X f 2., be sequence of tokens, where N represents the total number 32
of tokens. LMs are designed to encode the input text into a corresponding sequence of hidden
vectors (h, Rd As shown in Figure 2a, the conventional CLS-based FT [25, 35, 76] trains the output vector h corresponding to the [CLS] token with an additional head layer (e.g., an MLP layer). However, there a discrepancy between the pre-training objective (see Figure 2c) and the CLS-based FT objective, which has led to research on prompt-based techniques for better LM performance.
The prompt- based FT is formulated as MLM problem where the objective is to predict masked tokens [73, 74]. Specifically, the input text X is conditioned with a specific prompt template X = T(x). which includes one special token [MASK]. The prompt-based FT then maps the output vector associated with the [MASK] token to label word. The probability of predicting class y E y is computed as:
x(y[X) = p( [MASK] = (x)
(1)
where the verbalizer : - V is mapping from the task label space individual words in the vocabulary V.
Prompt-based FT can use either hard or soft prompt templates T, with label words potentially being a part of the prompt templates as well [36, 100J. Hard prompt template [73, 28, 78] requires careful designs of prompts and label words for each task. The use of hard prompts, however, was found to be sub-optimal and sensitive the choice of the prompt [102, 52]. Soft prompt [52, 100] was then proposed to use unused tokens from the vocabulary V or additional tokens tuneable embeddings for prompt template and can be directly trained with the task-specific supervision. This design allows the token embeddings in the prompt template to be updated independently of specific word embeddings after initialization, thus reducing the effort of searching for prompt templates and label words.
3 Our Approach: Prompt-based Continued Pre-training (PCP)
In this section, we introduce the proposed method, Prompt-based Continued Pre-training (PCP), which aims to improve the performance of LMs on downstream tasks through continued pre-training with prompt templates, as shown Figure 2g. Let I, A ((X1,9),..... (Xn> yn)} denote labelled examples and U - {X!...., X,,, denote m unlabelled examples. Our approach consists of two main steps, as described below.
Step 1: Construct Continued Pre-training Corpus. Initially, we select model F, pre-trained with the MIM objective and parameterized by ④ We then train this model using the prompt-based FT, minimizing the target loss function E on the labelled examples L, as illustrated in Figure 2b:
L(L)
Σ E(,, O)), X.BIEL
Next, the trained model F with the learned parameters O' to generate predictions (termed "pseudo-labels") on the unlabelled samples U:
Jo = F(T(X))
(3)
For each text example X and its associated (golden or pseudo) label y, we create an example for our proposed PCP as XPP = T(X, M(v),, where the original [MASK] position is substituted with M(y). This results in new corpus, C = (XPT) n+m In the fully-supervised setting, m and all examples use the golden labels.
) i=1
3

---

Step 2: Perform continued pre-training and prompt-based FT. We then proceed to further pre-train another model G. parameterized by θ, using the MLM objective on the newly generated corpus C. to obtain the PCP checkpoint Φ (see Figure 2d). Finally, we train model G, initialised by Φ, using Equation? 2 with prompt-based FT for downstream tasks.
In comparison to conventional continued pre-training, PCP does not require any modification for the model architecture or training process. The sole difference is the addition of a few extra tokens to the input text during continued pre-training. This modification does not hinder the efficiency of the method, i.e., both conventional continued pre-training and PCP maintain equal levels of efficiency. In this study, we primarily focus on LMs pre-trained with the MLM objective [53]. It is noteworthy to mention that comprehensive exploration of other architectures [25, 70, 14, 65] remains an avenue for future research. Nonetheless, considering prompt-based based fine-tuning approaches [52, 47, 51] have already been adapted for different model architectures and pre-training objectives [25, 70, 14, 65]. This implies that extending our method to alternative architectures should be feasible undertaking.
4 Experiments and Results
In this section, we evaluate the proposed method PCP by comparing it with conventional continued pre-training and four state-of-the-an art semi-supervised approaches. We assess their relative perfor- mance across 21 different classification and regression NLP tasks, including single-sentence and sentence-pair tasks. We conduct additional analysis concerning the performance lower bound of PCP and the effectiveness of the PCP across varying datasets and model sizes.
4.1 Experimental Setup
Datasets. Our study conducts a comprehensive analysis of 21 NLP datasets, including classification
and regression tasks. Following previous studies [28, 36, 100] prompt-based FT, we derive 8 single-sentence tasks and 8 sentence-pair English tasks from the GLUE benchmark [87]. SNLI [13], and 6 other widely used sentence classification tasks (i.e., SST-5, MR, CR, MPQA, Subj, TREC). Additionally, we use 5 popular benchmarks for semi-supervised learning from previous research [34,21, 94, 48, 29, 77], including IMDB [55], AG NEWS [101]. YELP REVIEW YAHOO! ANSWER [18], and AMAZON REVIEW [57]. See dataset details in Appendix SA. We train the model with two different settings: (1) fully-superviser
setting, where we train the model with the full training set; and (2) semi-supervised setting, where we sample the same amount of labelled data per class from the full training set. We re-sample the labelled data using the same five seeds for all comparison approaches and report the average performance with an error bar.
All Comparison Approaches. In our study, we mainly experiment using the ROBERTA-BAA BASE (125M) and the ROBERTA-LARGE (355M) models. We utilise the conventional CLS-based FT and two state-of-the-arr prompt-based FT approaches: (1) "CLS-based FT"" fine-tuning with the [CLS] token representation with an extra MLP layer; (2) basee FT (hard)": fine- fine-tuning with high-quality manual or auto generated prompts and label words [73, 28]; and (3) "Prompt-basee FT (soft)": fine-tuning with soft prompts using additional tokens for both templates and label words [100]. Since the objective of soft prompt FT is to minimize the reliance on human-designed templates, we unify the template for all tasks here. See the template specifics used for each dataset in Appendix BB. We train these three types of FT approaches from three different types of checkpoints to evaluate their relative effectiveness: (i) the off-the-shelf ROBERTA-LARGE checkpoint; (ii) the task-adaptive pre-training (TAPT) checkpoint [35] (represents the conventional continued pre-training). For sentence pair tasks, we concatenate the two sentences as an input example; and (iii) the proposed PCP checkpoint, obtained in $3. For both (ii) and (iii), we perform MLM on all full training sets except MNLI, MNLI-mm, SNLI, QNLI, and QQP, where we select up to 10k unlabelled examples from the full training sets (see supplementary experiments on the full training sets in Appendix $D). Additionally, we compare the proposed PCP with four state-of- ohetthe-rt semi-supervised approaches, including FixMatch [80], Dash [96], Flex [99], and AdaMatch [12] (see descriptions of these approaches in Appendix $C), where back-translation [64] is used for data augmentation as previous works [94, 77] and prompt-based FT (hard) is used as the backbone. See hyperparameter and implementation details in Appendix $E.
'https://www yelp. wwwc/oo///ttaeeeeeeoo dataset

---

Single Sontence Tasks
SST-2
SST-5
MR
CR
MPQA
Subj
TREC
CoLA
(acc)
(acc)
(acc)
(acc)
(acc)
(ace)
(acc)
(Mai))
Majority (full)
50.9
23.1
50.0
50.0
50.0
50.0
18.8
0.0
Prompt-based zero-shor!
83.6
35.0
80.8
79.5
67.6
51,4
32.0
2.0
io-contest learning
8
30.6y,
81.51
87.46.
63.82
26.22.
-1.52.4
Fully Supervised Learning
CLS-based FT
95.1
59.1
90.8
90.8
89.1
96.9
96.8 8
54.33
TAPT
96.0 109
60.6
91.4
91.0
80.9 10u
96.9
97.6
43.6 (3.66.967
Prongt based ITT (hurd)
95.2
60.0
90.8
92.4
89.4
95.9
97.8
54.7
TAPT
93.5 11.
60.4 704
90.3 Jos
90.8 416
89.5 701
96.9 10.0
97.6 La
44.0 196.7
PCP (ours)
05.5 feus
60.5
91.7 feu
92.8
89.6 fuu
97.8
56.0 11.5
Prongt-bused FT (seft)
94.2
59.8
90.4
92.7
87.8
96.4
97.4
61.3
TTPPT
92.7 jas
50.5
91.8
92.5 Joz
80.5 110
96.8
97.8
62.6.
PCP (ours)
94.3 fe.1
60.7
91.8 1144
92.8 qu.1
90.4 f26
97.1 fur
98.0 (Tue
62.0 toc
Semi Supervised Learning
CLS-based FT
81.22.7
11.T1.3
3
79.5x.
65. 65.1
F1.
80.3s,
26.77.
TAPT
88/2..5 17.0
43.4g.6
86. 1o 198
86,20.4 160
73.744 186
94.2..
80. 80.4o.1 10.1
1.99 1348
Prompt-bused FT (hard)
92.75.3
46.71.5
86.21
90.7a.s
80.86
91.0..1
81.7.a
7.26.5
-TPPT
92.9.a 103
48.91.3 122
88 A8.Aus 12.2
89.82, Loo
84. 138
93.5,, 125
85 22. 105
1. 1.4g. 158
PCP (ours)
93.6kaa
50.9
89.06
92.50.4
87.965
86.7o.
90. 90.6g.5 159
25.03.a
Prompt-based FT (soft)
92.5r.2
18.0
86.8
90.8
81 81.2sa
90.32a
03.0
1.9k,7
TAPT
93-4c.6
47.0 L10
88 Soa 117
89.0g.4 L1n
83 83.46.1 tea
93.3,. 130
84. 12.4 11.5
2. 2.1 Laa
PCP (ours)
93.9pg
50.71.3 132
89.8 13.0
92.0g. 112
88.305 194
91.9cM 145
88.65.4 15.6
21.5g.5 1166
Sentence Pair Tasks
MNLI
MNLI-mm
SNLI (acc)
QNLI
RTE
MRPC
QQP
STS-B
(a0c)
(acc)
(acc)
(acc)
(F1)
(FI)
(Pear.)
Majority (full)
32.7
49.5
$2.7
0.0
Prompt-based zero-shor!
50.8
51.7
49.5
50.8
$1.3
61.9
49.7
-3.2
in-context learning
52.0gg
58.44..
4
53.8p.1
60..4.
45.7g.0
36.15,
14.30.
Fully Supervised Learning
CLS-based FT
82.1
82.7
88.1
90.2
83.4
91.9
91.2
TAPT
$1.0 Ja.1
$2.0 .0T
86.7 [14
85.6 Jan
83.4 tno
91.6 Jo.
80.2 tus
90.4 Jos
Prompt- based ITT (hurd)
85.4
85.8
89.0
89.6
88.1
83.1
73.8
91.5
-TAPT
82.8 126
88.3 [a7
90.9 f1.
83.8 14.3
92.7 Jo.
78.2 T4.
91.2 Jais
PCP (ours)
86.5 fu
96.2
80.5 fuu
91.5
88.5 11u
93.3 h02
70.6 155
91.9 ta-
Prompt-based FT (soft)
84.6
85.4
89.5
84.5
92.4
91.6
TAPT
83.5 Jui
84.1 J13
88.3 [07
90.9 (1.4
82.7 L1N
92.6 fe
79.9 two
90.9 (az
PCP (ours)
85.7 fuu
86.0 hus
89.5
91.0
85.5 f1.0
92.6
79.6
91.7 tou
Semi Supervised Learning
CL8-based FT
16.2..6
18.5..0
45.6s
61.4a.2
51.2g
73.2a.7
58. 55.5..
16.0nk.
36.0 o Jums
36.3 Jaza
45.7% 1u
55.6g.7 Jas
51.41. Jox
67.7a.5 Laa
55.0.. Jas
48.1 f1.1
Prompt-bused FT (hard)
67.3..3
68.9h.a
76.7
66.5..3
68.35.1
75.9.
66.81s
67.7s.1
'APPT
50.7a.0 1166
52.24.6
1167
71..5. 12a
55.3L. Lina
59:9a/
63.26.8 Lib7
2a.6 Jan
63.1a.0
PCP (ours)
75.6 7663
76.8u. 139
82.41
157
85.1gs fins
70.22. 119
80.7a.a 14.
71.8 1sm
71.5x. fiss
Prompt-based FT (5oft)
62.72
65.9h.2
75.4a
64.277
68.2s7
73.0
66. 66.51a
63.7a.8
TAPT
46.6a 1161
49.ba.s
1964
72.120 J93
55.0n. 192
58.42.4 Ja8
63.35. 1977
58.3 9 Jan
65.3a. T16
PCP (ours)
75.4e. 1127
76.8u.
1000
82.6
122
8433a 1200
70.45
122
80.0,. tha
72. 72.3 158
71.47 (10
Summary of results: the probubility of improving the performance for TAPT PCP
Single Sentence Tasks
Sentence Pair Tasks
Checkpoint
TAPT
PCPes
TAPT
PCP
TAPT
PCP
TAPTomi
PCP
CLS-based FT
87,5 (7/8)
87.5 (7/8)
25.0(2/8) (2/8)
25.0 (2/8)
Prompt-bused FT (hard)
37.5
100
(6/8)
(8/8)
25.0
(8/8)
(8/8)
Prompt-based FT (soft)
50.0 (4/8)
100 (8/8)
62.5 (578)
100 (8/8)
37.5 (318)
100 (8/8)
12/8
100 (8/8)
Table 1: Comparison between the PCP and conventional continued pre-training (TAPT) using ROBERTA-LARGE.. The summary highlights the percentage of positive impact brought by the PCP and TAPT. The mean and standard deviation semi-supervised learning, 16 examples per class are used for training, in line with previous studies [28, 52, 100). Green and red arrows indicate changes with respect to the FT baselines that do not use TAPT or PCP. represents that no training examples are used. Three extra baselines sourced from [28] are included, where "Majority" refers to the majority class, and "in-context learning" indicates the usage of in-context learning [14] with ROBERTA-LARGE without updating any parameters.
test sets reported over 5 different seeds. In
4.2 Comparison of the PCP and conventional continued pre-training
Table I presents and summarises our experimental results 8 single-sentence tasks and 8 sentence- pair tasks. Below we delve deeper into our two major findings.
#1. TAPT is not consistently beneficial for sentence pair tasks, nor when prompt-based FT employed. Initially, we ve-visit the impact TAPT (representing the conventional continued pre-training) on the CLS-based FT, as shown in Table 1. Our experimental results align with carlier
5

---

Method
IMDB
AG NEWS
REVIEW
YAHOO! ANSWER
AMAZON REVIEW
Mean
100
200
200
200
40
200
DASH [96]
93.3
9
85.00g.9
$7.900.s
47.4
58.85
60.07a.7
06. 16.9
11.09g.
58.95
69.04
FIXMATCH 180]
95.25u.
94.2%u.
85.44
88.2104
47 47,26
38.51a.3
61 61.566.9
08.37
44 44.261..
52.33,,7
FLEXMATCH [99]
95.220.1
91.84.4
85,331,4
58.57o.5
50.602.s
58.341.5
58.09%.7
08.43..
45.48%.1
51.19
69.71
ADAMATCH [12]
95.200.5
91.91a
85.79
88.726.
50.122.7
58.951,
63.680.7
08. 08.00
11.081..
58.05p.6
70.35
Prompt- hased FT (harc)
86,782.
80.52,
84.87
86.95
46.89aa
58.27,.7
60.63,
66.94..
44.34,a
57.01o.4
68.20
+ PCP (ours)
92..49/.2
94.24u.
87.00
88.4444
52,8924.1
63.115,3
65.68L.
70:222..
63.445.
50.644..
72.77
Promgt-based FT (soft)
88.1
90.80
85.651.3
87.060.3
45. 13.4
57.12L.
61.18
(7. 85..8
11.52%.
55.08,.s
68.31
+ PCP (ours)
95, .5
91.36
87,256.
88 88.96ua
30.661.3
02.822..
05,26,,5
70.005..
52 78.
59, 16g,
72.49
Prompt-hased (hardif
95
91 01.66
68.71
74.30
78.70
Prompe-hased FT (soft)t
U6. 5U
91.10
69:63
75.66
63.92
79.04
Table 2: Comparison between the PCP and four semi- sumirvsper approaches using ROBERTA- LARGE. Each dataset is evaluated with two different labelled data sizes and full training set is used as unlabelled data. t indicates that full training set is used as the labelled data. We report the average Macro-Fi score on the test set across five seeds, with standard deviations subscripts. For each column, blue represents the best performance and orange stands for the second-best performance.
studies [39, 35, 77], showing that TAPT generally improves the performance of the CLS-based FT on 7 out of 8 single sentence tasks in both semi-supervised and full-supervised setting. However, intriguingly, we observe that TAPT negatively affects the performance of CLS-based FT on 6 out of sentence pair tasks, as summarised in Figure 1 and the at the bottom of Table 1. This finding implies that conventional continued pre-training (TAPT) may not be beneficial for sentence pair tasks.
Moreover, our investigation reveals that TAPT may negatively affect prompt-based FT. Specifically, in the fully supervised setting. TAPT results in reduced performance on 11 out of 16 tasks for prompt- based FT (hard) and on 9 out of 16 tasks for prompt-based FT (soft). In the most favourable scenario. TAPT enhances the performance of prompt-based FT (soft) from 73.9% to 79.9% on the QQP dataset. Conversely, in the least favourable situation, TAPT diminishes the performance of prompt-based FT (hard) from 54.7% to 44.0% on the CoLA dataset. In the semi-supervised setting, TAPT leads to a decline in performance on 12 out of 16 tasks for both prompt-baad based FT (hard) and prompt-based FT (soft) (see the summary of results in Figure and the at the bottom of Table 1). Particularly, for sentence pair tasks, TAPT results in an average absolute decrease of 9.5% in performance for prompt-basee FT. These results suggest that the effectiveness of TAPT varies across different tasks and cannot be universally applied. We conduct additional experiments to confirm the limitations of TAPT persist across different sizes of the pre-training corpus in Appendix D.
#2. PCP offers consistent and substantial improvements in both semi- and fully-supervised settings. As depicted in Table 1, our experiments covering 16 datasets both semi- and fully- supervised settings, including single sentence tasks and sentence pair tasks, reveal that (1) PCP consistently boosts the performance of prompt-based FT; and that (2) the performance gains achieved by PCP consistently exceed those obtained by TAPT by substantial margin. Specifically, compared to prompt-basee based FT, PCP leads to more than a 1.0% average absolute improvement in the fully- supervised setting and contributes to average absolute performance boost of 6.8% in the semi- supervised setting across 16 tasks. Compared to TAPT, PCP yields over a 1.8% average absolute improvement in the fully-supervised setting and contributes to an average absolute performance increase of 11.29 11.2% in the semi-supervised setting across 16 tasks. Notably, PCP can produce considerable gains in certain datasets. For instance, it elevates the performance of prompt- based FT (hard) from 7.2% (Matthews Correlation Coefficient) to 25.0%, while TAPT even reduces the performance of the prompt-based FT. Additionally, PCP improves the performance of prompt-based FT (soft) on the QNLI dataset from 64.2% to 84.3% with 31 % improvement, while TAPT leads to a 9.2% absolute performance decline. We attribute the improvements to presenting the prompt template to the LMs through the further pre-training phrase, which implies that merely showing task-related texts the LMs may not be the optimal approach for prompt-based FT.
4.3 Comparison of the PCP and state-of-the-art semi-supervised approaches
Table 2 presents our experimental results on five datasets, comparing the proposed PCP with otate-of- the-art sumi-supervised approaches. Below delve deeper into our main finding with discussion.
The proposed PCP outperforms state-of-the-art semi-supervised approaches on 4 out of tasks. As shown in Table 2, our proposed PCP approach with either hard soft variants of prompt-based

---

(5) Single Sentence Thdes using prompt-based IT (hard)
(b) Seigle Sentence Thikk using promph ed FT (sof)
120
Porformanc
551/2
$5..5
NPFAA
VOLA
551/2
$3.55
VPSIA
IREL
COLA
(<] Santence Pair Tasks using prompt-based FT thardl
(6) Sentence Pair Tasks using prempt-based FT (soft)
018
RFC
MBPC
3GP
5754
94L
671
MBDC
00P
575-
Wrong Labels+FT
FT Only
Random Labels+F1
Pesude Labels#FT (PCP)
Figure 3: The performance lower bound of the PCP, where "wrong labels" indicates that all labels in the PCP are incorrect and "random labels" indicates that all labels the PCP are randomly selected. For each dataset, 16 examples per class are used as labelled data and the full training set is used as unlabelled data. The mean performance on test sets reported over 5 different seeds.
FT outperforms the best-performing semi-supervised approaches 4 out of datasets. Notably, the prompt-based FT (hard) with the PCP outperforms the best performing semi-supervised approaches (FLEXMATCH) with absolute 5.5% Macro-Fi score the AMAZON REVIEW dataset when 200 labelled training examples are used. While the best performing semi-supervised approach, FIXMATCH, outperforms PCP by 1.7% in absolute value the IMDB dataset using 20 labelled examples, the performance discrepancy narrows as the number of labelled training examples increases. Overall, the prompt-based FT (hard) and (soft) with the PCP outperform all these semi-supervised approaches with an average absolute performance improvement of more than 2% across various datasets and labelled dataset sizes, demonstrating the effectiveness of our proposed approach.
Discussion. State-of-the-art semi-supervised approaches typically rely on generating pseudo labels for unlabelled examples in order to train student and teacher models iteratively [4, 15, 95, 27, 94, 29]. However, this iterative process is prone confirmation bias [83, 2, 31], which can result in error accumulation if the pseudo label is incorrect at any iterative step [49, 88, 31, 20]. Various efforts have been made to mitigate confirmation bias, such as using only high-confidence pseudo labels [80, 99, 12] or relying heavily on data augmentation [94, 21, ]. While these efforts make the training process more sophisticated, the issue remains difficult to fully address [20, 77]. Our proposed method offers an alternative way to utilise pseudo labels different from previous semi-supervised approaches [98, 58]. Instead relying on an iteration process with direct supervision signals from pseudo labels, we incorporate pseudo labels through continued pre-training with unsupervised objective (i.e., MLM). While our proposed approach may not always outperform semi-supervised approaches across all benchmarks, it delivers highly competitive performance while significantly streamlining the process by removing the necessity for iteration and additional data augmentation. We will discuss the efficiency of the proposed PCP later ($4.4). Additionally, PCP is orthogonal to these semi-supervised approaches and can be combined easily by initialising their backbone from the PCP checkpoint. In future work, plan to investigate the more specific cases where proposed PCP may be preferred over these superviidd approaches.
4.4 Further Analysis
#1. What is the lower bound of the model performance using the PCP? To understand the lower bound of PCP performance, conduct additional analysis with two different configurations of pseudo labels in PCP: (1) all pseudo labels are incorrect; and (2) all labels are randomly selected. Figure depicts the performance using different types of pseudo labels. We use the prompt- based FT without PCP (shown in yellow) and with PCP (shown in red) as baselines. Experimental results indicate that using incorrect pseudo labels (shown in blue) typically leads to inferior performance. In experiments using two prompt-based FT on 16 datasets, we find that using random labels leads to improved outcomes out of 32 scenarios. This suggests that PCP with random labels has over 50% chance of improving the performance of prompt-based FT, indicating that the performance lower
7

---

(a) SST-2
(B) MR
(e) SST-2
(D MR
94.0
99.0
93.5
Test Acc (96)
89
Test Aec
9330
88.0
42.5
18
65.0
92.0
1000
ROBERD
ROBERTO
RoBERN
(c) MPQA
(d) TREC
MPOA
(h) TREC
40.0
08.0
90.0
Test Acc (9%)
86.0
86
94.0
64.0
78
80.0
8606
Unlabelled Size
Unlabelled Size
Model
Model
Prompt- based FT (Hard)
Prompt: based FT (Hard) TAPT
Prompt-based FT (Hard) PCP Prompt-based (Soft)
Prompt-based (Soft)
Prompt-based (SoR)
TAFT
Figure 4: (Left) The effect of different unlabelled data sizes using ROBERTA-LARGE. (Right)
The effect of Scaling Laws, where ROBERTA- -EAE (123M) and ROBERTA- (354M). All comparison approaches are trained with 16 examples per class for each dataset.
bound is satisfactory. Additionally, PCP with random labels improves the performance on sentence pair tasks in 8 out of 16 cases, while TAPT leads to poorer results in 15 of 16 cases (refer to Table 1).
This suggests that PCP can be advantageous even when using random labels, providing benefits in scenarios where TAPT falls short. Interestingly, unlike prior study [60] on in-context learning [14]. where LMS using random labels in demonstrations perform close to those using ground-tre labels, our results show that using pseudo labels assigned by trained model (shown in red) consistently leads to the better performance, highlighting the importance of accurate pseudo labels.
#2. What are the requirements of data size and computational resources for the PCP? To gain a deeper understanding of the efficacy of our proposed PCP method, we conduct additional analysis to deter- mine the number of data points necessary for the PCP. Figure (left) presents the per- formance of prompt-based FT methods, in- cluding both hard and soft variants, across four datasets. The prompt-based FT perfor- mance generally improves when the PCP implemented with more than 1000 unla- belled examples, and some enhancements can be observed even with just 100 unla- belled examples. This indicates that contin- ued pre-training (both TAPT and PCP) is not necessarily computationally demanding and can be with only hundreds of training examples. In experiments, performing the PCP on 1k unlabelled example takes less than 10 minutes using two 24GB NVIDIA 3090 GPUs, and all PCP performance achieved in $4.2 use fewer than 10k unlabelled examples. This is stark contrast to the previous work [33] that pursued similar objectives (for parameter-efficient efficient ane-tuning) ours but utilised 10GB of English text data.
Dataset
Size
FT
+TAPT
+PCP
IMDB
23K
87.31.2
88.91.3
91.40.5
AG NEWS
100K
86.44.9
87.6
88.00.4
YELP REVIEW
250K
52.42.5
60.31.0
61.442.0
AMAZON REVIEW
250K
51.21.
56.8
57.01.5
YAHOO! ANSWER
500K
64.90.8
64.91.1
69.01.4
Table 3: Test Results for prompt- FT (soft) using ROBERTA-BAA -BAS with varying continued pre-training corpus sizes. Average Macro- F, with standard devia- tions are reported across five seeds. The model is trained on the IMDB dataset using 100 labelled examples and uses 200 labelled examples for other datasets. The best performance for each dataset is highlighted in blue.
used efficiently
#3. Power of scale. Our empirical analysis investigates the impact of increasing the backbone LM size on the model performance using the PCP. Figure 4 (right) shows the results of prompt-based FT methods, including hard and soft variants, trained using either TAPT or PCP, on four datasets. The performance of the PCP method largely improves as the backbone LM size expands, which aligns with the scaling laws observed in LMS [41, 37]. Furthermore, the PCP method consistently surpasses other bascline approaches, highlighting the advantages of the PCP across different model sizes.

---

SST-2
SST-5
MR
CR
MPQA
Subj
TREC
CoLA
Mean
Prompt FT
92.5
48.0
86.8
90.8
81.2
90.3
83.0
4.9
72.2
Prompt FT +PCP
93.9
50.7
89.8
92.0
88.3
94.9
88.6
21.5
77.5
Prompt FT +PCP (Labels Only)
93.7
50.8
87.7
91.3
85.1
94.3
85.7
-0.7
73.5
Prompt FT +PCP (Template Only)
90.7
43,5
88.6
92.6
82.0
95.1
84.1
0.7
72.2
Table 4: Ablation study the inclusion of the template and labels in proposed PCP. The test
Results using soft prompt FT and ROBERTA-LARGE are reported. The best performance for each dataset is highlighted in blue.
SST-2
SST-5
MR
CR
MPQA
Subj
TREC
CoLAA
Mean
CLs-based FT (Ik steps) + TAPT CLs-based FT (5k steps) + TAPT
88.2
43.4
86.1
86.2
73.7
94.2
80.4
1.9
69.3
89.6
43.4
86.7
87.0
72.9
94.6
79.0
1.7
69.4
Prompt FT (Ik steps) PCP
93.9
50.7
89.8
92.0
88.3
94.9
88.6
21.5
77.5
Table 5: Ablation study on the prolonged fine-tuning, where ROBERTA-LARGE used as the backbone model. The test Results using Cl.s-based FT and soft prompt FT are reported. The best performance for each dataset is highlighted in blue.
#4. The impact of larger continued pre-training corpus the model performance using PCP and TAPT. Here we expand our investigation to whether the advantage of the proposed PCP approach persists the size of the continued pre-trainin pre-training corpus increases. Table 3 presents the performance of prompt-based FT (soft), trained using either TAPT or PCP, across five datasets with varying sizes of unlabelled training examples. These experimental results are consistent with our findings in $4.2 and $4.3, showing that the proposed PCP approach consistently outperforms the model performance using the TAPT even when the larger corpus for continued pre-training is used.
#5. Ablation study on the label and template inclusion in PCP. of the individual contributions of pseudo labels and templates in our proposed PCP method, we conduct an additional ablation study, where we solely utilize pseudo labels or templates. This ablation study is carried out using soft prompt-based fine-tuning. As shown in Table 4, the experimental results reveals that using either labels templates exclusively will hurt the model's performance compared to our proposed PCP method, highlighting the vital importance of integrating both templates and pseudo labels.
To gain deeper understanding
#6. The impact of prolonged fine-tuning on the model performance. To ascertain that the effectiveness of our proposed method not simply due to an extended fine-tuning duration, we conduct additional experiments. We train CLs-based FT 5 times more steps (5k steps in total) from the TAPT checkpoint. As shown Table our results reveal that prolonged fine-tuning only brings about a marginal improvement of only 0.1 % across the eight tasks. Notably, this still falls significantly short of our proposed method (8. 1% in absolute).
Related Work
Prompt-based Approaches. In recent years, researchers have been exploring prompt-based ap- proaches to improve the performance of fine-tuning. These approaches can be broadly divided into two research directions. The first direction, known as prompt-based FT, optimizes all parameters in LMs for better performance [73, 28, 52, 100]. discussed in $2. Adaprompt [22] improved the performance of hard prompt-based FT [73, 28] on single sentence tasks through conventional contin- ued pre-training, which is generally consistent with our experimental results. The second direction is trrametee--fficieen fine-tuning (PEFT) [51, 68, 47, 81, 86), which aims to achieve competitive results while maintaining low computational costs. PPT [33] strives to improve the performance of PEFT [47] by further pre-training the T5 model [70], which pursues a similar idea as ours. However, this method relies series of hand-crafted and task-dependent designs for further pre-training, making it less adaptable to novel downstream tasks [86]. Furthermore, it demands a much larger training corpus, as discussed in $4.4. In contrast, our work offers uniform design across all tasks and focuses on prompt-based FT. In future work, we plan to explore the compatibility of continued pre-training (including both TAPT and our proposed PCP) and PEFT methods.

---

Our work is related to training LMs with templates. Recent studies [42, 3, 65, 91, 61, 72, 89, 59] have explored the idea of LMs training on a variety of NLP tasks with natural language instractions/emmlaaes, with the goal of generalizing to unseen tasks. Similar ideas, prompt transfer, have also been explored in the context of PEFT [33, 81, 86, 75], which seeks to learn an effective representation of the soft prompt for the target task by training on other tasks. In our approach, we transfer knowledge from task-related texts with prompt templates that are tailored to single target task to LMs.
Train LMs with InstructionsTemmllaess
Semi-supervised Learning. Our work is related to semi-supervised learning [32, 19, 43], with the goal of utiliising unlabelled data effectively. Continued pre-training followed by fine-tuning [39, 82, 35] is one type of semi- sumi-supeeeieed approaches. While the benefits of continued pre- mre-training well acknowledged [6, 1, 56], it is commonly assumed that large amounts of data are necessary for continued pre- pre-training [e.g.. 50, 38, 33]. Contrarily, our research demonstrates that continued pre-training can improve performance using only few hundred unlabelled samples. Self- teff-training [98, 58] is another powerful semi-supervised approach, which typically uses student-teacher models to assign pseudo-lahels to the unlabelled data [46, 44, 83, 62, 15, 27, 94, 80, 29]. Our work offers an alternative way to use pseudo-labels without resorting to an iterative process, as discussed in $4.3.
Epilogue
Conclusion. This study challenges the widely accepted notion in NLP, showing that conventional continued pre-training be detrimental model performance, especially for sentence pair tasks and prompt-based FT. As an alternative, we propose Prompt-based Continued Pre-training (PCP), which consistently improves the performance of state-of-the-an art prompt- based FT approaches over conventional continued pre-training, Additionally, our proposed PCP outperforms state-of-the-a semi-supervised approaches with more streamlined process. Further analysis reveals that the advantages of PCP remain consistent across different sizes of models and datasets. This study emphasizes the importance of presenting both task raskerelated texts and templates/instructions to LMs during pre- vre-training for better fine-tuning performance on downstream tasks, contributing to the growing body of research on the optimisation of pre-training and fine-tuning strategies in NLP.
Limitations and Broader Impact. We outline several limitations inherent to our research:
The scale of language models. Our experiments utilise relatively modestly-sized language models [53]. The implications of scaling up more advanced language models, such as the Llama-2 [84] or the mixture-of-experts approach like GPT-4 [63], remains an open question. In the context of large language models, applying PCP with full set of parameter updates for a specific task may not be justifiable in terms of computational costs. Future research could explore multi-task learning strategies or parameter-efffcient continued pretraining.
The architecture of language models. Our work is limited to encoder-only models [25, 53). To generalize our findings, future research should investigate the effects of our method PCP encoder-decoder [70] and decoder-only architectures [14].
The diversity of tasks. Our evaluation is confined to text classification and regression tasks. Future research should investigate generative or multi-modal tasks, which may offer more
comprehensive insights into the applicability of our methods PCP.
In addition, our work is based on pre-training and prompting methods for LMs. Previous works [8, 14, 7] have extensively discussed the risks and potential harms associated with LMs, including the amplification of undesirable biases learned from unlabelled training data [8, 5, 16]. The energy cost and carbon footprint for our work were approximately 125 kWh and 70 kg COze, which are comparatively smaller than LM pre-training [25, 53, 14, 23].
Acknowledgments and Disclosure of Funding
The authors express their gratitude to the NeurIPS reviewers and area chairs for their insightful discussions. The authors are grateful to Xin Zhao for her contributions to proofreading. Zhengxiang Shi is funded by the Research Studentship from University College London (UCL).
10

---

References
[1] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72- 78, Minneapolis, Minnesota, USA, June 2019, Association for Computational Linguistics.
[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8, IEFE, 2020.
[3] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng. Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022.
[4] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789-798, Mclbourne, Australia, July 2018. Association for Computational Linguistics.
[5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
[6] Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP), pages 3615-3620, Hong Kong, China, November 2019. Association for Computational Linguistics.
[7] Emily M. Bender, Timnit Gebru, Angelina McMillan-Majos and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[8] Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Anmual Meeting of the Associ- ation for Computational Linguistics, pages 5185-5198, Online, July 2020. Association for Computational Linguistics.
[9] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 41-48, New York, NY, USA, 2009. Association for Computing Machinery.
[10] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. TAC, 2009.
[11] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. International Conference on Learning Representations, 2020.
[12] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In Interna- tional Conference on Learning Representations, 2022.
[13] Samuel R. Bowman, Gabor Angeli, Christopher Pous, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods Natural I Language Processing, pages 632-642, Lishon, Portugal, September 2015. Association for Computational Linguistics.
11

---

[14] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger: Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[15] Rui Cai and Mirella Lapata. Semi-supervised semantic role labeling with cross-view train- ing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1018- 1027, Hong Kong, China, November 2019. Association for Computational Linguistics.
[16] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Kather- ine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
[17] Daniel Cer, Mona Diab, Eneko Agire,, lnigo Lopez- SazezoGazpio,, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In the 11th International Workshop on Semantic Evaluation SemmEval-2017) 2017.
[18] Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic representation: dataless classification. Proceedings of the 23rd national conference on Artificial cealliieenne--oluume 2, pages 830-835, 2008
[19] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b06) reviews]. IEEE Transactions on Neural Networks, 20(3):542--442 2009.
[20] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang. and Mingsheng Long. Debiased self-training for semi-supervised learning. In Advances in Neural Information Processing Systems, NIPS'22, 2022.
[21] Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguisticall--inoomed interpolation hidden space for supervise text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online, July 2020. Association for Computational Linguistics.
[22] Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang. AdaPrompt: Adaptive model training for prompt-based NLP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6057-6068, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[23] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2001:22204002311,2022
[24] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual en- tailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, 2005.
[25] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
[26] William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In the Third International Workshop on Paraphrasing (IWP2005), 2005.
12

---

[27] Xin Dong and Gerard de Melo. A robust self- self-learning framework for cross-lingual text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage P Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIP)) pages 6306-6310, Hong Kong, China, November 2019. Association for Computational Linguistics.
[28] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few- shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 6-330, Online, August 2021. Association for Computational Linguistics.
[29] Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Liat Ein-Dor, and Noam Slonim. Zero- shot text classification with self-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022.
[30] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, 2007.
[31] Arushi Goel, Yunlong Jiao, and Jordan Massiah. Pars: Pseudo-label aware robust sample selection for learning with noisy labels. arXiv preprint arXiv:2201.10836 2022.
[32] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[33] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 8410-8423, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[34] Suchin Gururangan, Tam Dang, Dallas Card, and Noah Smith. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880-5894, Florence, Italy, July 2019. Association for Computational Linguistics.
[35] Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online, July 2020. Association for Computational Linguistics.
[36] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adver- sarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 1921-4933 Online, August 2021. Association for Computational Linguistics.
[37] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[38] Zejiang Hou, Julian Salazar, and George Polovets. Meta-Learning the Difference: Prepar- ing Large Language Models for Efficient Adaptation. Transactions of the Association for Computational Linguistics, 10: 10:1249-1266 1265, 11 2022.
[39] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classi- fication. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339 Melbourne, Australia, July 2018. Asso- ciation for Computational Linguistics.
[40] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. ACM SIGKDD international conference on Knowledge discovery and data mining, 2004.
13

---

[41] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint #rXiv:2001.88361 2020.
[42] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online, November 2020. Association for Computational Linguistics.
[43] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[44] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[45] Teven Le Scao and Alexander Rush. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636, Online, June 2021. Association for Computational Linguistics.
[46] Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, page 896, 2013.
[47] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[48] Changchun Li, Ximing Li, and Jihong Ouyang. Semi-supervised text classification with balanced deep representation distributions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5044-5053, Online, August 2021. Association for Computational Linguistics.
[49] Junnan Li, Richard Socher, and Steven C H Hoi. DIVIDEMIX: LEARNING WITH NOISY LABELS AS SEMI-SUPERVISED LEARNING. In ICLR 2020, page 14. ICLR, 2020.
[50] Shiyang Li, Semih Yavuz, Wenhu Chen, and Xifeng Yan. Task-adaptive pre-training and self- training are complementary for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1006-1015, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[51] Xiang Lisa Li and Percy Liang. Prefix- tunin: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582 4597, Online, August 2021. Association for Computational Linguisties.
[52] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. 1rXiv:2103.10338 2021.
[53] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:
2019.
[54] Lajanugen Logeswaran, Ming-We Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. shoo entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 3449-3460, Florence, Italy, July 2019. Association for Computational Linguistics.
14

---

[55] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christo- pher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Anmual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, USA, June 2011, Association for Computational Linguistics.
[56] Katerina Margatina, Loic Barrault, and Nikolaos Aletras. On the importance of effectively adapting pretrained language models for active learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925-836, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[57] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, page 165-172, New York, NY, USA, 2013, Association for Computing Machinery.
Understanding rating
[58] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing, In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152 159,159, New York City, USA, June 2006. Association for Computational Linguistics.
[59] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCI.: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human I Language Technologies, pages 2791-2809, Seattle, United States, July 2022. Association for Computational Linguistics.
[60] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods Natural Language Pro- cessing, pages 1 11048-1 1104, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[61] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[62] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and superviid learning. IEEE transactions pattern analysis and machine intelligence, 41:1979-1999, 2018.
[63] OpenAL. Gpt-4 technical report. arXiv, 2023.
[64] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[65] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[66] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Association for Computational Linguistics (ACL), 2004.
[67] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment catego- rization with respect to rating scales. In Association for Computational Linguistics (ACL), 2005.
[68] Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chupler of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online, June 2021. Association for Computational Linguisties.
15

---

[69] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8), 2019.
[70] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res.. 21)1), jan 2020.
[71] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016.
[72] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training en- ables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[73] Timo Schick and Hinrich Schütze. Exploiting floze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255-269, Online, April 2021. Association for Computational Linguistics.
[74] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1-399-2352 Online, June 2021. Association for Computational Linguistics.
[75] Zhengxaing Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning, arXiv preprint, 2023.
[76] Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning to execute actions or ask clarification questions. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2060-2070, Seattle, United States, July 2022. Association for Computational Linguistics.
[77] Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, and Yunlong Jiao. Rethinking semi-supervised learning with language models. Findings of ACL 2023, Toronto, Canada, 2023. Association for Computational Linguistics.
[78] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4222-4235, Online, November 2020. Association for Computational Linguistics.
[79] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013.
[80] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[81]
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, July 2022.
[82] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classifi- cation? In China national conference on Chinese computational linguistics, pages 194-206. Springer, 2019.
16

---

[83] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- -egeh-- consistency targets improve semi-supervised deep learning results. In Proceedings of the 31 st International Conference on Neural Information Processing Systems, NIPS' 17, page 199411204, Red Hook, NY, USA, 2017. Curran Associates Inc.
[84] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (XXi:23377009288, 2023.
[85] Ellen M Voorhees and Dawn M Tice. Building a question answering test collection. In the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 2000.
[86] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5039-5059, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[87] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355 Brussels, Belgium, November 2018. Association for Computational Linguistics.
[88] Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for data-efficient deep learning. In International Conference on Machine Learning (ICML), 2021.
[89] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Morad- shahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-Naturallnsstruccee Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[90] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association of Computational Linguistics (TACL). 7, 2019.
[91] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[92] Janyce Wiebe, Theresa Wilson, and Claire Cardie.
Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3), 2005.
[93] Adina Williams, Nikita Nangia, and Samuel Bowman. A croer-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018
[94] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[95] Qizhe Xic, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[96] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervied learning with dynamic thresholding. In International Conference on Machine Learning, pages 11525-11536. PMLR, 2021.
17

---

[97] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics.
[98] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics.
[99] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 34, 2021.
[100] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference Learning Representations, 2022.
[101] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 849-657, Cambridge, MA, USA, 2015. MIT Press.
[102] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 18-24 Jul 2021.
18

---

Appendix Overview
The appendix is structured as follows:
Appendix $A
provides brief description for each dataset.
Appendix $B
provides details of templates and label words used for each dataset.
Appendix $C approaches.
presents brief description of state-of-the-arr four semi-supervised (self-training)
Appendix $D provides the supplementary experimental results to investigate the potential reasons for the ineffectivness of CL.S-based fine-tuning on the sentence pair tasks.
Appendix $E provides implementation details and byperparameters for all comparison methods used our experiments.
A Dataset
In this work, we use 21 popular datasets from previous few-shot learning and semi-supervised learning research.
For experiments in $4.2, adhere to the approach in [28] and utilise 16 different datasets? includ- ing SST-2 [79]. SST-5 [79], MR [67], CR [40], MPQA [92]. Subj [66]. TREC [85]. CoLAA [90], MNLI [93], SNLI [13], QNLI [71], RTB [24, 10], MRPC [26], QQP3, and STS-B [17]. Consistent with prior research [28], our validation set comprises 16 examples per class from the aforementioned datasets. Additionally, we use 16 examples per class for the training and the entire training set as the unlabeled set in the semi-supervised setting. We also utilise the full training set for training purposes in the fully supervised setting. For sentence pair tasks, we select at most 10k examples for continued pre-training to reduce the computational costs.
For experiments in $4.3, we follow the setup in [77] and utilise 5 different datasets, including IMDB [55], AG NEWS [101], YELP REVIEW YAHOO! A ANSWER [18], and AMAZON REVIEW [57]. Refer to the dataset statistics in Table 6. Our validation set comprises 1,000 examples for each dataset.
B Templates for Prompt-based FT
Here introduce the templates used in two -tate-of-the-art prompt-based FT approaches for each dataset. For "Prompt- FT (hard)", we use high-quality manual or auto-generated prompts and label words for each task from previous works [73, 28]. For "Prompt-based FT (soft)"", the STS-2 template for all single sentence tasks and STS-B template for all sentence pair tasks, while the label words for each task follow the description in Table 7.
C ST Frameworks
FIXMATCH. FIXMATCH [80] generates artificial labels using both consistency regularization and pseudo-labelling, where the artificial labels are produced based on weakly-augmented unlahelled data. These artificial labels are then used as targets to train the model on auggentd unlabelled data. FIXMATCH only retains an artificial label if the model assigns a high probability to one of the possible classes.
DASH. DASH [96] extends FIXMATCH by introducing a mechanism with dynamically adjusted threshold of loss select subset of training examples from the unlabelled data for performing Sst.,
/nttps://github.. wttpaa/cccoomprnttoonnllilll Bttpae/nnnnnne//-/B////oooo////
BFF / dataset..ttttt httppp:://wwuuuuaaaaaaa quorq.uu.coooo q/quuuadataa///////aaaaaaaa
'httpa://ww
"nttps://www yelp. cow/co////a//o dataset
19

---

Single Sentence Tosks
Dataset
L
#Train
#Test
Type
Labels (classification tasks)
SST-2
19
6,920
872
Sentiment
positive, negutive
SST-5
8,544
2,210
Sentiment
v. pos., positive, neutral, negative, v. neg.
MR
8.662
2,000
Sentiment
positive, negative
CR
19
1,775
2,000
Sentiment
positive, negative
MPQA
3
8,606
2,000
Opinion Polarity
positive, negative
Subj
8,000
2,000
Subjectivity
subjective, objective
TREC
5,452
500
Question cls.
abbr. entity, description, human, loc., num.
CoLA
8
8,551
1,042
Acceptability
grammatical, not grammatical
IMDB
149
8,000
1,000
Movie Review
positive, negative world, sports, business, sci/tech
AG NEwS
8,000
1,000
News Topic
YELP REVIEW
134
8,000
1,000
Review Sentiment
1,2.3,4.5 I. 1,2,3,4,5
AMAZON REVIEW
8,000
1,000
Review Sentiment
science. health, education, computer. sports, business, music, family, polities
YAHOO! ANSWER
32
8,000
1,000
Topic Clussification
Sentence Pair Tasks
Dataset
[
L
#Train
#Test
Type
Labels (classification tasks)
MNLI
22/11
392,702
9,815
NLI
entailment, neutral, contradiction entailment, neutral, contradiction entailment, entailment entailment, not__oottttttt entailment equivalent, not equivalent equivalent, not, not__ equivalent
SNLI
14/8
549,367
9,842
QNLI
11/30
104.743
5,463
NLI
RTE
49/10
2,490
277
NLI
MRPC
22/21
3,668
408
Paraphrase
QQP
12/12
363,846
40,431
Paruphrase
STS-B
R
11/11
5,749
1,500
Sent. Similarity
Table 6: The datasets evaluated in this work. | # of classes for classification tasks (with one exception: STS-B real-valued regression task over the interval [0, 5). I, average # of words in input sentence(s). Note that only sample examples from the original training set in our few-shot experiments.
Single Sentence Tasks
Task
Template
Label words
SST-2
<S,> It was [MASK]
positive: great, negutive: terrible
SST-S
<S,> [MASK]
v.positive: great, positive: good, neutral: okay.
negative: bad, v.negative: terrible
MR
<S1> was [MASK]
positive: great, negative: temblle
<8,> [MASK]
positive: great, negutive: terrible
MPQA
<S,> is [MASK]
positive: positive, negative: negative
Subj
<S1> [MASK]
subjective: subjective, objective: objective
TREC
[MASK]
<<
abbreviation: Expression, entity: Entity, description: Description
human: Human, location: Location, numerie: Number
COLA
<S,> This [MASK]
grammatical: correct, not grammatical:
IMDB
<S,> It was [MASK]
positive: great, negutive: terrible
NEWS
<5,> was [MASK]
World: world, Sports:sports, Business: business, Sci/ Tech: tech
YELP REVIEW
<5,>
[MASK]
3,4;
AMAZON REVIEW
<S,> It was [MASK]
0: 0, 1: 2. 2, 3:
YAНоо!! ANSWER
<S,> was [MASK]
culture: culture, science: science, health: health, education: education computer.
computer, sports: sports, business: business
music: music, family: family, politics: politics
Sentence Pair Tasks
Task
Template
Label words
MNLI
<S,> [MASK] <<,>>
entailment: Yes, neutral: Maybe, contradiction: No
SNLI
<S,>? [MASK] in this case <S>>
entailment: Yes, neutral: Maybe, contradiction: No
QNLI
<S,> [MASK]
<Sg>
entailment: Yes, pos_entaimment No
RTE
<8,> [MASK] think that <S
eotailment: Clearly, entailment:
MRPC
<S,> [MASK] <S2>
equivalent: Yes, not eeu.valeuittt: No
QQP
<S,> [MASK] <S,>
equivalent: Yes, not_equivalent: Yes. gr: No
STS-B
<5,> [MASK] <SS>>
Table 7: Templates and label words used for "Prompt-based FT (hard)". We use the STS-2 and STS-B template for all single sentence tasks and sentence pair tasks using "Prompt-based FT (soft)", respectively.
FLEXMATCH. FLEXMATCH [99] also extends FIXMATCH by introducing the concept of cur- riculum learning [9] to flexibly adjust thresholds for different classes each time step and select unlabelled data and their pseudo labels that are more likely to be informative.
20

---

ADAMATCH. ADAMATCH [12] aims to solve domain adaptation problems in SsL and build high-accuracy model that trains on and tests on different data distributions. ADAMATCH builds on FIXMATCH and introduces a relative confidence threshold and a modified distribution alignment from [1]].
D Supplementary Experiment
In this section, we investigate why TAPT does not work on sentence pair tasks. We have evaluated three possible explanations for TAPT's ineffectiveness on sentence pair tasks: (1) dataset size for continued pre-training, (2) sentence pairs with higher similarity than what was observed in pre-training data, and (3) lack of separation within sentence pairs. Our experimental results suggest that the ineffectiveness of TAPT on the sentence pair tasks is not isolated incident but recurring issue. Below we discuss each setting in detail.
#1. The impact continued pre-training (TAPT) with a larger pre-training corpus on the performance of the prompt-based FT sentence pair tasks. In Section 4.2, we randomly selected at most 10k unlabeled examples from the full training sets of MNLI, MNLI-mm, SNLI, QNLI, and QQP, as corpus for continued pre-training due to our limited academic computational resources. For all other tasks, we use the full training set for continued pre-training because there are fewer than 10k examples in their training sets. To verify our findings that "TAPT is not consistently beneficial for sentence pair tasks, nor when prompt- based FT is employed" holds true when utilising larger continued pre-training corpus, perform conventional continued pre-training (TAPT) on the full training set on MNLI, MNL.-mm, SNLI, QNLI, and QQP.
Dataset Corpus Size
MNLI 393k
MNLI-mm 393k
SNLI 549k
QNLI 104k
QQP
364k
CLS-based FT TAPT
46.2o.6
48.51.0
45.65.4
61.4s.2
58.5g.8
34.70.44
35.1g.64
41.82.7
54.82.04
62.62.9
Prompt-based FT (hard)
67.31.3
68.91.2
76.71.6
66.54.3
66.81.9
TAPT
47.8..64
47.96.2 4
47.59.44
53.5u.84
53.5a.84
Prompt-base based FT (soft)
62.71.2
65.91.2
75.40.s
64.24.7
66.51.
TAPT
45.4g.74
45.84.1 +
50.2a.9
53.8p.94
53.8a.94
Table 8: Test Results using ROBERTA-LARGE. with corresponding continued pre-training corpus sizes for each task. The mean performance with standard deviations reported across five seeds.
Table 8 presents the performance of the CLS- CLs-based FT, prompt-basee FT (hard), and prompt- based FT (soft) using the TAPT. The experimental results reveal that the TAPT generally results in poorer performance, even when a larger continued pre-training corpus is used. Notably, the performance of these fine-tuning approaches could be even worse than those achieved using smaller continued pre-training corpus (refer to results in Table 1), suggesting that training with a larger corpus is not an effective solution the issues of conventional continued pre- trainninn (TAPT).
MNLI
MNLI-mm
SNLI
QNLI
RTE
MRPC
QQP
STS-B
Mean
CLS-based
46.2
48.5
45.6
61.4
54.2
73.2
58.5
46.0
54.2
+TAPT
36.0
45.7
55.6
53.4
67.7
55.0
48.1
49.7
+TAPT (Tokenizer Sep)
36.4
37.5
50.5
58.8
50.8
63.5
59.2
48.8
50.7
+TAPT (PCP Sep)
36.3
36.7
64.6
58.3
51.2
65.3
57.4
44.5
51.8
+TAPT (random pair)
34.8
35.4
37.7
52.2
51.2
64.8
56.9
23.8
44.6
+TAPT (first sent only)
35.6
35.9
42.7
52.2
52.6
62.5
53.6
16.7
44.0
Table 9: Ablation study on the performance of CL.s-based fine-tuning with different settings of conventional continued pre-training, where ROBERTA--LAREE is used as the backbone model.
#2. High similarity within sentence pairs. We consider that the high similarity between the sentence pairs might conflict with the word distribution that the model has observed during model pre-training. For instance, in the MNLI task, two sentences are Salt kept the town fed and Salt kept the
21

---

town thriving. To explore this, we perform TAPT on two different settings, one where we continually pre-train TAPT on randomly paired sentences within the dataset and another where we continually pre-train TAPT using just the first sentence of each pair. As shown in Table 9, the experimental results show that training TAPT with either case leads even worse performance.
#3. Token-based separation of sentence pairs. In an attempt to mitigate the effect above, we also consider that distinguishing two sentences using distinct tokens might make a difference. To test this, we perform TAPT with two types of separate tokens, the special token from the tokenizer and the template used in PCP (without labels). As shown Table 9, training TAPT with separate tokens somewhat mitigate the performance drop for CLS-based fine-tuning
between two sentences the sentence pair tasks. However, the results remain inferior compared to CLS-based fine-tuning without the use of TAPT.
In conclusion, our investigations highlight the difficulties that TAPT faces on sentence pair tasks, while our proposed method PCP provides simple yet effective solution. We hypothesize that TAPT's ineffectiveness for CLS-based fine-tuning on sentence pair tasks might be due various factors, which we leave for a more comprehensive investigation in future work.
E Implementation Details
Our code implemented using Pytorch² and Huggingface". The suniisupereised approaches are implemented upon the repository". Below, we provide a comprehensive list of the hyperparameters used in our code. For fine-tuning, as shown in Table 10, we conduct a grid search for learning rates within the set (1e-5, 2e-5, 5e-5), and choose batch size of 8. In each trial, train the model for 1,000 steps, evaluate performance every 100 steps, and select the best checkpoint based on optimal performance the evaluation set. The best performance is determined by the relevant evaluation metric. For continued pre-training, we utilise the same set of hyperparameters for both TAPT and PCP, shown in Table 11. The learning rate and unlabeled data size are closely linked and need to be adjusted simultancously. As a general guideline, we suggest decreasing the learning rate as the unlabeled data size decreases. In contrast its predecessor, BERT [25], which uses the next sentence prediction objective, ROBERTA [53] is trained solely with the masked language model (MLM) objective, specifically the cross-entropy loss on predicting randomly masked tokens. RoBERTa dynamically alters the masking pattern applied to training examples, typically employing a masking probability of 15. Additionally, Table 12 lists the hyperparameters for self-training approaches, where grid search for learning rates within the set (1e5, 2e-5, 5e-5) is conducted.
"nttps://pytorch.org
"uttpe://huggingttoe.coo
httpp////gttab....o/mam/////pppp -rr-tt--rrannngg training
22

---

Hyperparameter
Assignment
number of steps
1000 steps (evaluate every 100 steps)
batch size
maximum learning rate
le-05, 2e-5,
maximum sequence length
128, 256
learning rate optimizer
AdamW
Adam epsilon
le-6
Adam weights
0.9, 0.98
learning rate scheduler
Warmup linear
Weight decay
0.01
Warmup proportion
0.06
Table 10: Hyperparameters for hard and soft prompt-based fine-tuning.
Hyperparameter
Assignment
number of steps
100 epochs
batch size
256
maximum learning rate
1c-05, lc-4
learning rate optimizer
Adam
Adam epsilon
lc-6
Adam beta weights
0.9, 0.98
learning rate scheduler
Warmup linear
Weight decay
0.01
Warmup proportion
0.06
Masking Probability
0.15
Table 11: Hyperparameters for both conventional continued pre-training (TAPT) and prompt-based based conventional fine-tuning (PCP).
Hyperparameter
Assignment
number of steps
12 800 or 25 600 steps
batch size
16
learning rate
le-05, 2e-05, Se-05
learning rate optimizer
AdamW
maximum sequence length
learning rate scheduler
Warmup linear
Warmup proportion
0.05
learning decay
linear
Table 12: Hyperparameters for self training. Algorithm-specific hyperparameters will be released in configuration files with the code.
23


Reviews:
[{"review_id": "NNRMow1x6F", "reviewer": "Reviewer_CwVD", "summary": "The proposed method (CPC) is built on top of pseudo-labeling and continued pre-training via masked language modeling. This method provides an alternative way to use pseudo-labeled data before fine-tuning the model on downstream tasks. It improves the TAPT method and other semi-supervised methods for text classification.", "questions": "1.\tWhat masking strategy is used in Step 2? Are you using the random masking strategy as used in Robert?\n2.\tIf you have pseudo labels in CPC, is random masking in Step 2 the optimal strategy? How about some selective masking strategies (e.g., PMI-masking [1])?\n3.\tThe paper is missing a self-training baseline, where you only mask the pseudo labels for MLM pre-training in Step 2.\n4.\tIt’d be good to include the scores of TAPT in Fig 3, representing “no label + FT”. It will be helpful to compare “wrong label + FT” and “no label + FT”. Because the method is sensitive to the pseudo-labeling performance, it'd be good to add a few-shot setting where the base model is fine-tuned on few-shot examples in Step 1. And compare CPC with TAPT in the few-shot setup.\n5.\tTo better compare TAPT (“no label +FT”) with CPC (“pseudo-label +FT”), we should have a better understanding of the impact of the “pseudo-label” in the MLM process. Does the addition of “pseudo-label” in the input sentence improve the MLM prediction accuracy?\n\n[1] PMI-Masking: Principled masking of correlated spans\n", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The novelty of this paper lies in using pseudo-labeled data for masked language modeling. This idea is simple, yet effective. Even though pseudo-labeling and continued MLM training are not new, the way to combine these two is new.\n- Extensive experiments on a large number of text classification tasks show the effectiveness of the method.\n", "weaknesses": "- One of the motivations (mentioned in the introduction) is that TAPT does not work well on sentence pair classification. However, it’s still unclear why TAPT does not work well on sentence pair classification while CPC can address this issue.\n- Although CPC works empirically better than TAPT, the intuition of CPC is still unclear. Why should we continue pre-training on pseudo-labeled data instead of unlabeled data in TAPT?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort and time by the reviewer (CwVD). We are thrilled to receive positive feedback on the novelty and simplicity of our method, along with the extensive experiments that underscores the effectiveness of our method. We would like to address the reviewer's valuable feedback as follows:\n\n***it’s still unclear why TAPT does not work well on sentence pair classification while CPC can address this issue.***\n\nWe acknowledge the reviewer's valid question regarding the performance of TAPT on sentence pair tasks. In response, we have performed additional experiments to answer this question. We would invite the reviewer to refer to our general response for details.\n\n***Although CPC works empirically better than TAPT, the intuition of CPC is still unclear. Why should we continue pre-training on pseudo-labeled data instead of unlabeled data in TAPT?***\n\nWe appreciate the reviewer's question. As mentioned in our paper, our intuition for PCP is that `language model will benefit from seeing the template/prompt before the target task`, as the importance of presenting the templates to the language models have been shown in various instruction tuning works. We find that TAPT only tells the model about how the text for the target task looks like (let us call it in-domain knowledge). Based on this finding and intuition, we propose PCP, which not only tells the model the in-domain knowledge, but also the prompt information that will be used for fine-tuning on the target task. This also makes the objective of continued pre-training closer to the objective of fine-tuning. We will make it clear in our revised version. \n\n***What masking strategy is used in Step 2? Are you using the random masking strategy as used in Robert?***\n\nYes, we use the same masking strategy as used in RoBERTa. We dynamically mask 15% tokens. We will make this clear in our revised version.\n\n***If you have pseudo labels in CPC, is random masking in Step 2 the optimal strategy? How about some selective masking strategies (e.g., PMI-masking [1])***\n\nWe appreciate the reviewer's suggestion about PMI-masking, which is indeed highly relevant to our work. We will discuss this work in our revised version. We agree with the reviewer that there exists some potential variants, such as using PMI-Masking. However, it's essential to highlight that such a method could be applicable to both TAPT and our proposed method PCP. Our research is primarily centered on contrasting TAPT and PCP, rather than exploring strategies that could be equally beneficial for both. Moreover, assessing these variants across a broad range of datasets imposes a large resource burden on us. Consequently, we plan to leave the evaluation of these variants for future work.\n\n***The paper is missing a self-training baseline, where you only mask the pseudo labels for MLM pre-training in Step 2***\n\nWe thank the reviewer for their constructive feedback. To clarify, when we exclusively mask the pseudo labels, our task reverts to prompt-based fine-tuning. We compare our proposed method with 4 state-of-the-art self-training models in Table 2 of our paper, where all these self-training baselines use the prompt-based fine-tuning as the backbone. Results show that PCP outperforms these state-of-the-art self-training models. We will clarify this in our revised version.\n\n***It’d be good to include the scores of TAPT in Fig 3, representing “no label + FT”. It will be helpful to compare “wrong label + FT” and “no label + FT”.***\n\nWe thank the reviewer for this valuable suggestion. We will definitely do this.\n\n***Because the method is sensitive to the pseudo-labeling performance, it'd be good to add a few-shot setting where the base model is fine-tuned on few-shot examples in Step 1. And compare CPC with TAPT in the few-shot setup.***\n\nWe thank the reviewer for this valuable suggestion. Actually, we exactly follow the few-shot learning setting in the prior work [1], where the only difference is that we use additional unlabelled data. The pseudo-labels for this unlabelled data in PCP are assigned by the model trained in the few-shot learning setting. As shown in Figure 1 of our paper, PCP brings substantial improvement when using a base model trained on few-shot examples to produce pseudo-label. We will clarify this in our revised version.\n\n[1] Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. ACL 2021.\n\n***To better compare TAPT (“no label +FT”) with CPC (“pseudo-label +FT”), we should have a better understanding of the impact of the “pseudo-label” in the MLM process. Does the addition of “pseudo-label” in the input sentence improve the MLM prediction accuracy?***\n\nWe thank the reviewer for the insightful suggestion. In response, we have performed additional experiments, as shown in Table below. Our results indicate that PCP indeed improves the accuracy of MLM. In terms of average accuracy across 8 single-sentence tasks, TAPT attains a score of 0.6713, while PCP obtains a higher average accuracy of 0.7225. We also find that the accuracy of the pseudo label does not appear to significantly influence the MLM accuracy. To be specific, PCP with correct labels yields an average accuracy of 0.7188, while PCP with wrong labels records an average accuracy of 0.7238. Similar results can be observed for sentence pair tasks. The reason for improved accuracy might benefit from the usage of the same template in all sentences, which is easier to predict. We will include this in our revised version. We hope that our response has adequately addressed the reviewer’s concerns and would be most grateful if the reviewer could consider a score increase accordingly.\n\n\n|          |    TAPT   | PCP     | PCP (correct-label) | PCP (wrong-label) |\n| ----- | -----   | ---- |---               | ----   |\n| Single Sentence Tasks |  0.6713|   0.7225  |    0.7188 |  0.7238 |\n| Sentence Pair Tasks | 0.7586 | 0.7700  |  0.7800 |  0.7728 | \n"}, {"review_id": "IL9WpF5cDd", "reviewer": "Reviewer_TYSB", "summary": "This paper makes a contribution by studying how to adapt pre-trained models to downstream tasks. The authors identify the limitations of TAPTs and show when they do not work well. They then propose PCP, a better algorithm that can adapt a pre-trained model to a target task. PCP is shown to be more effective than TAPTs on a variety of tasks.\n", "questions": "Cloud the authors provide more intuitions or analysis why PCP works better? Based on the intuitions, what kind of additional analysis can be done here? \n", "limitations": "The paper only applies on text classifications tasks, ignoring many text generation tasks.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides an intriguing analysis of when TAPT style is effective for fine-tuning. TAPT is not very effective on sentence-pair tasks or prompt-based fine-tuning, particularly.", "weaknesses": "The paper does a good job of pointing out the weaknesses of TAPT, but it is less clear why PCP works better. The authors do not need to know exactly why it works, but some hypotheses would be helpful. For example, why does PCP-style pretraining work better for prompt-based fine-tuning? Is it because the \"pretraining\" is more similar to \"fine-tuning\" in PCP? If so, how can we understand why PCP also works for sentence pair tasks?\n\nThe presentation of the paper could also be improved. Figure 2 is not very easy to understand, and the overall flow of the paper is not as clear as it could be.\n\nOverall, the paper could be improved by providing more clarity and explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer (TYSB)’s thoughtful and thorough evaluation of our paper. We sincerely appreciate the positive feedback regarding our intriguing analysis and contribution to identify the limitations of the previous approach. We would like to address the reviewer's valuable feedback as follows:\n\n***The paper does a good job of pointing out the weaknesses of TAPT, but it is less clear why PCP works better. The authors do not need to know exactly why it works, but some hypotheses would be helpful. For example, why does PCP-style pretraining work better for prompt-based fine-tuning? Is it because the \"pretraining\" is more similar to \"fine-tuning\" in PCP? If so, how can we understand why PCP also works for sentence pair tasks?***\n\nWe acknowledge the reviewer's inquiry regarding the intuition of our method. The intuition of our method is that `language model will benefit from seeing the template/prompt before the task, no matter using supervised learning or unsupervised learning objectives`:\n- As we mentioned in the paper, the importance of presenting the templates to the language models have been also shown in various instruction tuning works. Our work differs from these works in two main aspects: (1) they aim to improve the zero-shot learning performance, while our work aims to improve the fine-tuning performance; and (2) they learn the template knowledge through supervised learning objectives, while we use unsupervised learning. However, the intuition (why PCP works better) is similar: showing the templates/prompt to the language models can improve the performance on the downstream tasks. \n- To explain why PCP works, we attribute the reason that PCP works better than TAPT to *presenting the prompt template to the model* as we mentioned in the paper, because this is the only difference between our method and TAPT. We find that TAPT only tells the model about how the text for the target task looks like (let us call it in-domain knowledge). In contrast, our proposed method PCP tells the model not only the in-domain knowledge but also the prompt information that will be used for fine-tuning on the target task. We also agree with the reviewer that the PCP brings the continued pre-training closer to the prompt-based fine-tuning. We ablate the importance of templates and labels in the PCP in our subsequent response to the reviewer.\nWe will clarify this in the revised version of our paper. \n\n***Cloud the authors provide more intuitions or analysis why PCP works better? Based on these intuitions, what kind of additional analysis can be done here?***\n \nThe main intuition behind our PCP is that it is important to show the model with the template/prompt that will be used in the target task. We perform an ablation study to emphasize the importance of including both the template/prompt and label in PCP. As presented in Table below, the experimental results suggest that relying solely on either labels or templates will hurt the model's performance. This highlights the importance of integrating both templates and pseudo labels into our proposed method PCP.\n\n|          | SST-2 | SST-5 | MR | CR | MPQA | Subj | TREC | CoLA | Mean |\n|----------|-------|-------|----|----|------|------|------|------|------|\n|Prompt FT | 92.5 | 48.0 | 86.8 | 90.8 | 81.2 | 90.3 | 83.0 | 4.9 | 72.2 |\n|Prompt FT+PCP| 93.9 | 50.7 | 89.8 | 92.0 | 88.3 | 94.9 | 88.6 | 21.5 | 77.5 |\n|Prompt FT+PCP(Pseudo Labels Only)| 93.7 |50.8 | 87.7 | 91.3 | 85.1 | 94.3 | 85.7 | -0.7 | 73.5 |\n|Prompt FT+PCP(Template Only)| 90.7 | 43.5 | 88.6 | 92.6 | 82.0 | 95.1 | 84.1 | 0.7 | 72.2 |\n\n***The presentation of the paper could also be improved. Figure 2 is not very easy to understand, and the overall flow of the paper is not as clear as it could be. Overall, the paper could be improved by providing more clarity and explanation.***\n\nWe express our gratitude to the reviewer for their insightful feedback on the presentation of our paper. We are committed to improving this aspect in our revised version. The presentation of this paper will be improved as follows:\n- We will merge Figure 2 with the aforementioned intuition.\n- To further clarify our work with more explanation, all five additional experiments mentioned in our rebuttal will be incorporated into the revised version of our paper.\n\n"}, {"review_id": "uBbO6atKoZ", "reviewer": "Reviewer_pe8v", "summary": "This paper explores the problem of continued pre-training on task-related text. The authors discovered that conventional continued pre-training methods may not be very effective and can even have a negative impact on fine-tuning performance. To address this, they introduce prompt-based continued pre-training. The approach involves generating pseudo labels on unlabeled data using a fine-tuned model and constructing a prompt-based pre-training corpus by applying templates to the pseudo labeled dataset. The researchers then utilize this corpus, which incorporates prompt information, for continued pre-training and prompt-based fine-tuning. Experimental evaluations conducted on various datasets validate the effectiveness of the proposed approach, as it achieves significant improvements over the baseline methods.", "questions": "Is it possible to extend this approach to the task of language modeling fine-tuning?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The researchers identify the limitations of conventional continued pre-training on task-related text and propose an innovative solution.\n* The experimental results provide strong evidence for the effectiveness of the proposed approach.", "weaknesses": "No major weaknesses have been identified in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort and time by the reviewer (pe8v). We are heartened by their positive appraisal of our work and their recognition that no major weaknesses have been identified in the paper. We would like to respond to their invaluable feedback as follows:\n\n***Is it possible to extend this approach to the task of language modeling fine-tuning?***\n\nWe greatly appreciate the reviewer's thoughtful comments and suggestions. If we understand correctly, the idea of incorporating prompt-based language modeling (prompt-based continued pre-training) as an auxiliary loss during fine-tuning is indeed a compelling one. We will definitely consider this perspective, viewing it as a potential route for exploration in our upcoming research.\n"}, {"review_id": "kM9AvUzZth", "reviewer": "Reviewer_fyTk", "summary": "\nThis work re-examines a well-known technique in the NLP literature, called continued pre-training, that can be utilized to enhance the performance of language models on downstream tasks (in this case, for classification and regression tasks).\nThe authors have revealed that the conventional approach to continued pre-training is ineffective when applied to sentence-pair tasks and prompt-based fine-tuning settings.\nBased on their findings, the authors propose a simple method that involves training on the masking language modeling objective while augmenting the text input with prompts.\nThis proposed approach has demonstrated its effectiveness across a range of tasks and configurations, suggesting that it can be an effective choice for natural language understanding tasks when using Transformer encoder-based models.", "questions": "\nTo the best of my understanding, it appears that your method incorporates prompts and their corresponding verbalized labels into the process of continued pre-training. If this is the case, there is a possibility that the tokens for verbalized labels are chosen as the target for masked language modeling. This implies that the method might be performing a similar task during both continued pre-training and fine-tuning. Consequently, there is a potential risk that the effectiveness of the proposed method is primarily attributed to unintentionally prolonged fine-tuning. If this turns out to be true, it could diminish the significance of this work.\n\nTo alleviate any ambiguity regarding this point, it would be helpful if you could provide concrete examples demonstrating how your method is applied to real data instances. This would offer a clearer understanding of the methodology and help evaluate its effectiveness more accurately.", "limitations": "The evaluation primarily focuses on the utilization of a single model, specifically RoBERTa-large, for relatively straightforward downstream tasks such as classification. These tasks are considered comparatively easier to solve.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method is technically simple and effective.\n- The work begins with a compelling finding that challenges the effectiveness of the conventional approach, which was previously regarded as effective.\n- The authors made efforts to provide analysis from multiple perspectives, aiming to convince readers of the effectiveness of the proposed approach.\n\n", "weaknesses": "- Although the paper offers a detailed explanation and analysis, there are still ambiguous points that need to be clarified for a better understanding by the readers. Let me ask about this point in the following Questions section.\n- There is room for improving the grammar and fluency of the paper's writing.\n- I would like to see a thorough analysis of the factors that potentially contribute to the success of the proposed method. For instance, it would be valuable to explore the significance of including labels in continued fine-tuning (although partially considered in Section 4.4) and investigate why sentence-pair tasks do not benefit from the conventional TAPT approach.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer (fyTk) for the insightful and comprehensive assessment. It is heartening to receive positive feedback on the simplicity and effectiveness of our approach, as well as the quality of our analysis. Furthermore, we appreciate the acknowledgement of our contribution in identifying the limitations of the previous approach. We would like to respond to the reviewer's valuable feedback as follows:\n\n***I would like to see a thorough analysis of the factors that potentially contribute to the success of the proposed method. For instance, it would be valuable to explore the significance of including labels in continued fine-tuning (although partially considered in Section 4.4)***\n\nWe are sincerely thankful for valuable comments and suggestions. As suggested by the reviewer, we carry out an additional experiment which involves the inclusion of labels in the Task-Adaptive Pre-training (TAPT) before performing CLS-based fine-tuning. The results of this experiment are presented in Table below. Our findings suggest that the inclusion of labels in the TAPT phase does not notably improve the model's performance, and there remains a considerable performance gap with our proposed method. We hope that our response has adequately addressed your initial concerns and would be most grateful if the reviewer could consider a score increase accordingly.\n\n|          | SST-2 | SST-5 | MR | CR | MPQA | Subj | TREC | CoLA | Mean |\n|----------|-------|-------|----|----|------|------|------|------|------|\n|CLS FT    | 81.2  | 41.7  | 76.3 | 79.5 | 65.1 | 91.7 | 80.3 | 26.7 | 67.1 |\n|CLS FT+TAPT| 88.2 | 43.4 | 86.1 | 86.2 | 73.7 | 94.2 | 80.4 | 1.9 | 69.3 |\n|CLS FT+TAPT with Labels| 89.2 | 42.3 | 84.2 | 85.7 | 75.8 | 93.8 | 84.5 | 0.7 | 69.5 |\n|Prompt FT+PCP| 93.9 | 50.7 | 89.8 | 92.0 | 88.3 | 94.9 | 88.6 | 21.5 | 77.5 |\n\n\n***investigate why sentence-pair tasks do not benefit from the conventional TAPT approach.***\n\nWe express our gratitude to the reviewer for their constructive suggestion. In response to this query, we have undertaken further experiments. For detailed information, please refer to our general response to all reviewers.\n\n***To the best of my understanding, it appears that your method incorporates prompts and their corresponding verbalized labels into the process of continued pre-training. If this is the case, there is a possibility that the tokens for verbalized labels are chosen as the target for masked language modeling. This implies that the method might be performing a similar task during both continued pre-training and fine-tuning. Consequently, there is a potential risk that the effectiveness of the proposed method is primarily attributed to unintentionally prolonged fine-tuning. If this turns out to be true, it could diminish the significance of this work.***\n\nWe appreciate the insightful comments and suggestions. To answer the reviewer’s question, we conduct additional experiments. We train cls-based fine-tuning 5 times more steps (5k steps in total) from the TAPT checkpoint. As shown in Table below, our results reveal that prolonged fine-tuning only brings about a marginal improvement of only 0.1% across the eight tasks. Notably, this still falls significantly short of our proposed method (8.1% in absolute). We will include this in the revised version of our paper.\n\n|          | SST-2 | SST-5 | MR | CR | MPQA | Subj | TREC | CoLA | Mean |\n|----------|-------|-------|----|----|------|------|------|------|------|\n|CLS FT(1k steps)+TAPT| 88.2 | 43.4 | 86.1 | 86.2 | 73.7 | 94.2 | 80.4 | 1.9 | 69.3 |\n|CLS FT(5k steps)+TAPT| 89.6 | 43.4 | 86.7 | 87.0 | 72.9 | 94.6 | 79.0 | 1.7 | 69.4 |\n|Prompt FT(1k steps)+PCP| 93.9 | 50.7 | 89.8 | 92.0 | 88.3 | 94.9 | 88.6 | 21.5 | 77.5 |\n\n\n***The evaluation primarily focuses on the utilization of a single model, specifically RoBERTa-large, for relatively straightforward downstream tasks such as classification. These tasks are considered comparatively easier to solve.***\n\nWe appreciate the insightful comments and suggestions. To facilitate a direct comparison, we conducted experiments utilizing benchmarks that are widely recognized and employed in previous research [1,2]. This approach aligns with previous studies that focus on prompt-based fine-tuning, a benchmarking methodology that is widely accepted within the field.\n\n[1] Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. ACL 2021.\n\n[2] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. ICLR 2022.\n\n***There is room for improving the grammar and fluency of the paper's writing.***\n\nWe are thankful to the reviewer for their constructive critique. We will conduct a thorough review and improve the writing in our revised version."}, {"review_id": "TSpMPg0n83", "reviewer": "Reviewer_bM1j", "summary": "The paper proposes a new approach to pre-training language models called Prompt-based Continued Pre-training (PCP). PCP combines the idea of instruction tuning with conventional continued pre-training. The authors argue that PCP can improve the performance of prompt-based fine-tuning on a variety of natural language processing tasks.", "questions": "1. More ablations and analysis are needed to understand why TAPT is not working for sentence-pair tasks.\n2. More experiments to ablate the effect of template alignment vs data augmentation via pseudo-labels.\n", "limitations": "Generalization of the approach to other pretraining objectives is unclear.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a clear comparison of the proposed PCP with the conventional continued pre-training. The evidence on benchmark datasets is strong to support the claim that PCP performs better on these tasks.\n2. The paper is well-written and easy to understand. The authors do a good job of explaining the technical details of the proposed approach, and they provide clear and concise summaries of the experimental results.\n3. The approach is simple and easy to adopt.\n", "weaknesses": "1. It is not well understood why conventional continued pretraining (TAPT) is doing so poorly on sentence-pair tasks.\n2. Because PCP makes use of templated prompts to align continued pretraining and fine-tuning, with additional pseudo-labels. It is unclear whether the gain is due to prompt alignment or the additional data augmentation from pseudo-labels. Further ablation is necessary.\n3. It is unclear how well the proposed approach generalizes to other types of pretraining objectives such as language modeling.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer (bM1j)’s thoughtful and thorough evaluation of our paper. We are genuinely appreciative of the positive feedback concerning simplicity and effectiveness of our approach, solid experimental evidence, and the quality of our presentation. We would like to respond to the reviewer's valuable feedback as follows:\n\n***Because PCP makes use of templated prompts to align continued pretraining and fine-tuning, with additional pseudo-labels. It is unclear whether the gain is due to prompt alignment or the additional data augmentation from pseudo-labels. Further ablation is necessary.***\n\nThank you for your insightful comments and suggestions. To address this question, we conduct an additional ablation study, where we solely utilize pseudo labels or templates in our proposed method PCP. This ablation study is carried out using soft prompt-based fine-tuning. As shown in Table below, the experimental results indicate that using either labels or templates exclusively will hurt the model's performance. This highlights the vital importance of integrating both templates and pseudo labels into our proposed method.\n\n|          | SST-2 | SST-5 | MR | CR | MPQA | Subj | TREC | CoLA | Mean |\n|----------|-------|-------|----|----|------|------|------|------|------|\n|Prompt FT | 92.5 | 48.0 | 86.8 | 90.8 | 81.2 | 90.3 | 83.0 | 4.9 | 72.2 |\n|Prompt FT+PCP| 93.9 | 50.7 | 89.8 | 92.0 | 88.3 | 94.9 | 88.6 | 21.5 | 77.5 |\n|Prompt FT+PCP(Pseudo Labels Only)| 93.7 |50.8 | 87.7 | 91.3 | 85.1 | 94.3 | 85.7 | -0.7 | 73.5 |\n|Prompt FT+PCP(Template Only)| 90.7 | 43.5 | 88.6 | 92.6 | 82.0 | 95.1 | 84.1 | 0.7 | 72.2 |\n\n***It is not well understood why conventional continued pretraining (TAPT) is doing so poorly on sentence-pair tasks?***\n\nWe express our gratitude to the reviewer for their constructive suggestion. In response to this query, we have undertaken further experiments. For detailed information, please refer to our general response to all reviewers.\n\n***It is unclear how well the proposed approach generalizes to other types of pretraining objectives such as language modeling.***\n\nWe appreciate your insightful comments and questions regarding the generalizability of our proposed approach to other pre-training objectives. We recognize the value of exploring this avenue. We will do our best to get the necessary resources to investigate this important direction, and we leave this to future work.\n"}]

Author rebuttal:
We appreicate all the reviewers for dedicating their time and effort to evaluate our work.  We are thrilled to receive positive feedback on **the novelty of our approach** (CwVD,pe8v), **the simplicity and effectiveness of our approach** (CwVD,fyTk,bM1j), **solid experimental evidence** (CwVD,pe8v,bM1j), **intriguing/thorough analysis** (TYSB,fyTk), and **the quality of the presentation** (bM1j). We also thank the reviewer (TYSB,pe8v,fyTk) for acknowledging our **contribution to identify the limitations of the previous approach**.

To answer the reviewers’s questions, we conduct 5 additional experiments, regarding **the potential reason for why TAPT does not work on sentence pair tasks**, **the impact of PCP on the MLM accuracy**, **an ablation study on template only and label only**, **the impact of adding label to TAPT**, **the impact of training CLS-based fine-tuning longer**. We hope that our response, paired with these additional experiments, will address the reviewers' concerns.

One common question is about why TAPT does not work on sentence pair tasks. We delve into this particular issue below. For the remaining concerns, we will respond to each reviewer individually. We have evaluated three possible explanations for TAPT's ineffectiveness on sentence pair tasks: **dataset size**, **sentence pairs with higher similarity than what was observed in pre-training data**, and **lack of separation within sentence pairs**. Our experimental results suggest that the ineffectiveness of TAPT on the sentence pair tasks is not an isolated incident but a recurring issue. Below we discuss each setting in detail.
- **Training data size**. We wonder if the size of training data could be a limiting factor. To test this, we perform TAPT on MNLI, MNLI-mm, SNLI, QNLI, QQP datasets, with up to 360k training data. Our experimental results, detailed in our paper's Appendix, reveal that training TAPT with a large corpus still undermines the performance of cls-based fine-tuning on sentence pair tasks.
- **High similarity within sentence pairs**. We consider that the high similarity between the sentence pairs might conflict with the word distribution that the model has observed during model pre-training. For instance, in the MNLI task, two sentences are `Salt kept the town fed` and `Salt kept the town thriving`. To explore this, we perform TAPT on two different settings, one where we continually pre-train TAPT on randomly paired sentences within the dataset and another where we continually pre-train TAPT using just the first sentence of each pair. As shown in the Table below, the experimental results show that training TAPT with either case leads to even worse performance. 
- **Token-based separation of sentence pairs**. In an attempt to mitigate the effect above, we also consider that distinguishing two sentences using distinct tokens might make a difference. To test this, we perform TAPT with two types of separate tokens, the special token from the tokenizer and the template used in PCP (without labels). As shown in the Table below, training TAPT with separate tokens between two sentences can somewhat mitigate the performance drop for cls-based fine-tuning on the sentence pair tasks. However, the results remain inferior compared to cls-based fine-tuning without the use of TAPT.

In conclusion, our investigations highlight the difficulties that TAPT faces on sentence pair tasks, while our proposed method PCP provides a simple yet effective solution. We hypothesize that TAPT's ineffectiveness for CLS-based fine-tuning on sentence pair tasks might be due to various factors, which we leave for a more comprehensive investigation in future work.
 

|           | MNLI      |MNLI-mm|SNLI |QNLI | RTE  | MRPC | QQP  |STS-B | Mean |
|---------- |-------    |-------|-----|-----|------|------|------|------|------|
|CLS FT     | 46.2      | 48.5  | 45.6| 61.4| 54.2 | 73.2 | 58.5 | 46.0 | 54.2 |
| +TAPT| 36.0      | 36.3  | 45.7| 55.6| 53.4 | 67.7 | 55.0 | 48.1 | 49.7 |
|+TAPT (Tokenizer Sep)| 36.4  | 37.5  | 50.5| 58.8| 50.8 | 63.5 | 59.2 | 48.8 | 50.7  |
|+TAPT (PCP Sep)| 36.3  | 36.7  | 64.6| 58.3| 51.2 | 65.3 | 57.4 | 44.5 | 51.8  |
|+TAPT(random sent pair)| 34.8 | 35.4  | 37.7| 52.2| 51.2 | 64.8 | 56.9 | 23.8  | 44.6  |
|+TAPT(first sent only)| 35.6 | 35.9  | 42.7| 52.2| 52.6 | 62.5 | 53.6 | 16.7 | 44.0 |


Meta-review: This paper argues that the conventional continued pre-training method may not always bring improvements for downstream tasks under certain scenarios. The authors then propose a novel method called "Prompt-based Continued Pre-training" which is shown to be able to improve the performance on downstream tasks. All the reviewers are largely positive on this work, with some main concerns such as the reasons the approach works better than previous approaches, and why the previous approach (TAPT) performs poorly on the sentence-pair classification task. Though the latter remains largely unanswered in this work, the current paper contains sufficient new knowledge and methodologies that are worth sharing with the community.
Decision: Accept (poster)