SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. The impact of continued pre-training (TAPT) with a larger pre-training corpus on the performance of the prompt-based FT on sentence pair tasks. In Section 4.2, we randomly selected at most  $10\mathrm{k}$  unlabeled examples from the full training sets of MNLI, MNLI-mm, SNLI, QNLI, and QQP, as corpus for continued pre-training due to our limited academic computational resources. For all other tasks, we use the full training set for continued pre-training because there are fewer than  $10\mathrm{k}$  examples in their training sets. To verify our findings that "TAPT is not consistently beneficial for sentence pair tasks, nor when prompt-based FT is employed" holds true when utilising larger continued pre-training corpus, we perform conventional continued pre-training (TAPT) on the full training set on MNLI, MNLI-mm, SNLI, QNLI, and QQP.
Dataset	MNLI	MNLI-mm	SNLI	QNLI	QQP
Corpus Size	393k	393k	549k	104k	364k
CLS-based FT	46.20.6	48.51.0	45.65.4	61.48.2	58.53.8
+ TAPT	34.70.4↓	35.10.6↓	41.82.7↓	54.82.0↓	62.62.9↑
Prompt-based FT (hard)	67.31.3	68.91.2	76.71.6	66.54.3	66.81.9
+ TAPT	47.85.6↓	47.95.2↓	47.59.4↓	53.50.8↓	53.50.8↓
Prompt-based FT (soft)	62.72.2	65.91.2	75.40.8	64.24.7	66.51.8
+ TAPT	45.43.7↓	45.84.1↓	50.23.9↓	53.80.9↓	53.80.9↓
Table 8 presents the performance of the CLS-based FT, prompt-based FT (hard), and prompt-based FT (soft) using the TAPT. The experimental results reveal that the TAPT generally results in poorer performance, even when a larger continued pre-training corpus is used. Notably, the performance of these fine-tuning approaches could be even worse than those achieved using a smaller continued pre-training corpus (refer to results in Table 1), suggesting that training with a larger corpus is not an effective solution to the issues of conventional continued pre-training (TAPT).
Table 8: Test Results using ROBERTA-LARGE, with corresponding continued pre-training corpus sizes for each task. The mean performance with standard deviations are reported across five seeds.
MNLI	MNLI-mm	SNLI	QNLI	RTE	MRPC	QQP	STS-B	Mean
CLS-based FT	46.2	48.5	45.6	61.4	54.2	73.2	58.5	46.0	54.2
+TAPT	36.0	36.3	45.7	55.6	53.4	67.7	55.0	48.1	49.7
+TAPT (Tokenizer Sep)	36.4	37.5	50.5	58.8	50.8	63.5	59.2	48.8	50.7
+TAPT (PCP Sep)	36.3	36.7	64.6	58.3	51.2	65.3	57.4	44.5	51.8
+TAPT (random sent pair)	34.8	35.4	37.7	52.2	51.2	64.8	56.9	23.8	44.6
+TAPT (first sent only)	35.6	35.9	42.7	52.2	52.6	62.5	53.6	16.7	44.0
Table 9: Ablation study on the performance of CLS-based fine-tuning with different settings of conventional continued pre-training, where ROBERTA-LARGE is used as the backbone model.
#2. High similarity within sentence pairs. We consider that the high similarity between the sentence pairs might conflict with the word distribution that the model has observed during model pre-training. For instance, in the MNLI task, two sentences are Salt kept the town fed and Salt kept the
town thriving. To explore this, we perform TAPT on two different settings, one where we continually pre-train TAPT on randomly paired sentences within the dataset and another where we continually pre-train TAPT using just the first sentence of each pair. As shown in Table 9, the experimental results show that training TAPT with either case leads to even worse performance.
3. Token-based separation of sentence pairs. In an attempt to mitigate the effect above, we also consider that distinguishing two sentences using distinct tokens might make a difference. To test this, we perform TAPT with two types of separate tokens, the special token from the tokenizer and the template used in PCP (without labels). As shown in Table 9, training TAPT with separate tokens between two sentences can somewhat mitigate the performance drop for CLS-based fine-tuning on the sentence pair tasks. However, the results remain inferior compared to CLS-based fine-tuning without the use of TAPT.
In conclusion, our investigations highlight the difficulties that TAPT faces on sentence pair tasks, while our proposed method PCP provides a simple yet effective solution. We hypothesize that TAPT's ineffectiveness for CLS-based fine-tuning on sentence pair tasks might be due to various factors, which we leave for a more comprehensive investigation in future work.
Our code is implemented using Pytorch $^5$  and Huggingface $^6$ . The semi-supervised approaches are implemented upon the repository $^7$ . Below, we provide a comprehensive list of the hyperparameters used in our code. For fine-tuning, as shown in Table 10, we conduct a grid search for learning rates within the set  $\{1e-5, 2e-5, 5e-5\}$ , and choose a batch size of 8. In each trial, we train the model for 1,000 steps, evaluate performance every 100 steps, and select the best checkpoint based on optimal performance on the evaluation set. The best performance is determined by the relevant evaluation metric. For continued pre-training, we utilise the same set of hyperparameters for both TAPT and PCP, as shown in Table 11. The learning rate and unlabeled data size are closely linked and need to be adjusted simultaneously. As a general guideline, we suggest decreasing the learning rate as the unlabeled data size decreases. In contrast to its predecessor, BERT [25], which uses the next sentence prediction objective, ROBERTA [53] is trained solely with the masked language model (MLM) objective, specifically the cross-entropy loss on predicting randomly masked tokens. RoBERTa dynamically alters the masking pattern applied to training examples, typically employing a masking probability of 0.15. Additionally, Table 12 lists the hyperparameters for self-training approaches, where a grid search for learning rates within the set  $\{1e5, 2e-5, 5e-5\}$  is conducted.
$^{5}$ https://pytorch.org/  
 $^{6}$ https://huggingface.co/  
 $^{7}$ https://github.com/amzn/pretraining-or-self-training
Hyperparameter	Assignment
number of steps	1000 steps (evaluate every 100 steps)
batch size	8
maximum learning rate	1e-05, 2e-5, 5e-5
maximum sequence length	128, 256
learning rate optimizer	AdamW
Adam epsilon	1e-6
Adam beta weights	0.9, 0.98
learning rate scheduler	Warmup linear
Weight decay	0.01
Warmup proportion	0.06
Table 10: Hyperparameters for hard and soft prompt-based fine-tuning.
Hyperparameter	Assignment
number of steps	100 epochs
batch size	256
maximum learning rate	1e-05, 1e-4
learning rate optimizer	AdamW
Adam epsilon	1e-6
Adam beta weights	0.9, 0.98
learning rate scheduler	Warmup linear
Weight decay	0.01
Warmup proportion	0.06
Masking Probability	0.15
Table 11: Hyperparameters for both conventional continued pre-training (TAPT) and prompt-based conventional fine-tuning (PCP).
Hyperparameter	Assignment
number of steps	12800 or 25600 steps
batch size	16
learning rate	1e-05, 2e-05, 5e-05
learning rate optimizer	AdamW
maximum sequence length	256
learning rate scheduler	Warmup linear
Warmup proportion	0.05
learning rate decay	linear
Table 12: Hyperparameters for self training. Algorithm-specific hyperparameters will be released in configuration files with the code.