SlowGuess/NAACL_2022_OCR huggingface dataset:
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. AEDA: an easier data augmentation technique for text classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 2748-2754. Association for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and Wlliam Campbell. 2019. A closer look at feature space data augmentation for few-shot intent classification. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 1-10, Hong Kong, China. Association for Computational Linguistics.
Judith Yue Li and Jiong Zhang. 2021. Semi-supervised meta-learning for cross-domain few-shot intent classification. MetaNLP 2021, page 67.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565-6576. PMLR.
Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. 2021. Bootstrapping semantic segmentation with regional contrast. arXiv preprint arXiv:2104.04465.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL.
Amr Sharaf, Hany Hassan, and Hal Daumé III. 2020. Meta-learning for few-shot nmt adaptation. ACL, page 43.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. In Empirical Methods in Natural Language Processing (EMNLP).
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020a. Contrastive multiview coding. In Computer Vision - ECCV 2020, pages 776-794, Cham. Springer International Publishing.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020b. Contrastive representation distillation. In International Conference on Learning Representations.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020c. What makes for good views for contrastive learning. In NeurIPS.
Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. 2021. Few-shot text classification with triplet networks, data augmentation, and curriculum learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5493-5500, Online. Association for Computational Linguistics.
Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6383-6389, Hong Kong, China. Association for Computational Linguistics.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. 2020. Loco: Local contrastive representation learning. In Advances in Neural Information Processing Systems, volume 33, pages 11142-11153. Curran Associates, Inc.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In International Conference on Machine Learning (ICML).
A Batch size and learning details
We use the same learning rate of  $1e^{-5}$  for MLM loss as LM-BFF. To take full advantage of SupCon, we apply large batch sizes (16, 32, 40). We show the batch size and learning rate for SupCon in Table A.1. Note that for results of LM-BFF shown in the main paper, we use the same large batch size of our method to allow for fair comparisons.
We set the batch size to be dividable by the total number of examples in the task and small enough to fit into the GPU memory. The experiments with RoBERTa-base are carried out on one NVIDIA RTX-A6000 with 48 GB of memory. Experiments with RoBERTa-large require 4x NVIDIA RTX-8000 (or RTX-A6000) with 192 (4x 48) GB of memory.
Following LM-BFF, our fine-tuning runs a maximum of 1000 steps.
Task	Batch	LR
SST-2	16	1e-6
Subj	16	1e-5
SST-5	40	1e-5
CoLA	16	1e-5
TREC	32	1e-5
MNLI	24	1e-5
MNLI-mm	24	1e-5
SNLI	32	1e-5
QNLI	16	1e-5
QQP	32	1e-5
RTE	32	1e-6
MRPC	16	1e-5
MR	16	1e-6
MPQA	16	1e-5
CR	32	1e-5
Table A.1: Batch size and learning rate (LR) for SupCon loss used for each task.
B Fine-tuning with prompts and demonstrations
We also consider LM-BFF as our baseline method due to its state-of-the-art performance in a wide range of few-shot tasks. The given masked language model  $\mathcal{M}$  first encodes the input sentence  $x_{\mathrm{in}}$  into a sequence of tokens  $\tilde{x}_{\mathrm{in}}$  and maps  $\tilde{x}_{\mathrm{in}}$  to a sequence of hidden states  $\{\mathbf{h}_1,\mathbf{h}_2,\dots \mathbf{h}_L\}$ , where  $L$  is the length of the sequence and  $\mathbf{h}\in \mathbb{R}^d$ , where  $d$  is the dimension of the hidden states. For example, in prompt-base fine-tuning, for single sentence text
$x_{\mathrm{in}}$  e.g., "The story is not worth reading.", the input with the prompt (e.g., "a truly [MASK] one .") takes the form of
$$
\begin{array}{l} x _ {\text {p r o m p t}} = [ \mathrm {C L S} ] x _ {\text {i n}}, \text {a t r u l y [ M A S K ] o n e . [ S E P ]} \\ \equiv \mathcal {T} (x _ {\mathrm {i n}}) \\ \end{array}
$$
Then, the model decides whether it is more likely to put the label word "great" or "terrible" at the [MASK] position. Fine-tuning with this fill-in-the-blank framework has been shown to be superior to standard fine-tuning (Schick and Schütze, 2021). By mapping the label space  $\mathcal{V}$  to the label words where  $\mathcal{V}(y)$  denotes the label word for class  $y$ , the prediction of the model  $\mathcal{M}$  for class  $y \in \mathcal{V}$  can be written as
$$
\begin{array}{l} p (y | x _ {\text {i n}}) = p \left(\left[ \text {M A S K} \right] = \mathcal {V} (y) \mid x _ {\text {p r o m p t}}\right) (4) \\ = \frac {\exp \left(\mathbf {w} _ {\mathcal {V} (y)} \cdot \mathbf {h} _ {[ \text {M A S K} ]}\right)}{\sum_ {y ^ {\prime} \in \mathcal {Y}} \exp \left(\mathbf {w} _ {\mathcal {V} (y ^ {\prime})} \cdot \mathbf {h} _ {[ \text {M A S K} ]}\right)} (5) \\ \end{array}
$$
where  $\mathbf{w}$  is the weight vector of MLM head.
In LM-BFF, the authors further append demonstrations to the input  $x_{prompt}$  to help the model better understand what is "great" and "terrible". Formally,  $x_{prompt} \equiv \mathcal{T}(x_{\mathrm{in}})$  and  $\tilde{\mathcal{T}}(x_{\mathrm{in}}^c, y^c)$  denote  $\mathcal{T}(x_{\mathrm{in}}^c)$  with [MASK] replaced by the label word  $\mathcal{V}(y^c)$ . Then, the input to LM-BFF takes the form of
$$
\mathcal {T} \left(x _ {\mathrm {i n}}\right) \oplus \tilde {\mathcal {T}} \left(x _ {\mathrm {i n}} ^ {1}, y ^ {1}\right) \oplus \dots \oplus \tilde {\mathcal {T}} \left(x _ {\mathrm {i n}} ^ {| \mathcal {Y} |}, y ^ {| \mathcal {Y} |}\right) \tag {6}
$$
In this paper, we use random sampling for the demonstrations, i.e.,  $x_{\mathrm{in}}^c$  is randomly chosen from the training set. The masked language modeling loss is then
$$
\mathcal {L} _ {\mathrm {M L M}} = \sum_ {(x _ {\mathrm {i n}}, y) \in \mathcal {D} _ {\mathrm {t r a i n}}} - \log p (y | x _ {\mathrm {i n}}) \tag {7}
$$
C Language-based Supervised Contrastive Loss
Our method extends the loss  $\mathcal{L}_{\mathrm{MLM}}$  with an additional Supervised Contrastive Loss (SupCon). For applying SupCon on multi-views of an input text, we need to first obtain a second view of a text:
$$
\tilde {x} _ {2 k} = \operatorname {A u g} \left(\tilde {x} _ {2 k - 1}\right) \tag {8}
$$
As we show in ablations, traditional data augmentation for text does not work well in the contrastive framework. Thus, we propose obtaining a second
Task	Template	Label words
SST-2	<S1> It was [MASK] .	positive: great, negative: terrible
SST-5	<S1> It was [MASK] .	vpositive: great, positive: good, neutral: okay, negative: bad, v.negative: terrible
MR	<S1> It was [MASK] .	positive: great, negative: terrible
CR	<S1> It was [MASK] .	positive: great, negative: terrible
MPQA	<S1> It was [MASK] .	positive: great, negative: terrible
Subj	<S1> This is [MASK] .	subjective: subjective, objective: objective
TREC	[MASK] : <S1>	abbreviation: Expression, entity: Entity, description: Description
human: Human, location: Location, numeric: Number
CoLA	<S1> This is [MASK] .	grammatical: correct, not_grammatical: incorrect
MNLI	<S1> ? [MASK] , <S2>	entailment: Yes, netural: Maybe, contradiction: No
SNLI	<S1> ? [MASK] , <S2>	entailment: Yes, netural: Maybe, contradiction: No
QNLI	<S1> ? [MASK] , <S2>	entailment: Yes, not_ entailment: No
RTE	<S1> ? [MASK] , <S2>	entailment: Yes, not_ entailment: No
MRPC	<S1> [MASK] , <S2>	entailment: Yes, not_ entailment: No
QQP	<S1> [MASK] , <S2>	entailment: Yes, not_ entailment: No
view by randomly changing the templates and/or demonstrations:
$$
\tilde {x} _ {2 k - 1} = \mathcal {T} _ {t _ {0}} \left(x _ {\text {i n}}\right) \oplus \dots \oplus \tilde {\mathcal {T}} _ {t _ {0}} \left(x _ {\text {i n}} ^ {c}, y ^ {c}\right) \oplus \dots \tag {9}
$$
$$
\tilde {x} _ {2 k} = \mathcal {T} _ {t _ {j}} \left(x _ {\text {i n}}\right) \oplus \dots \oplus \tilde {\mathcal {T}} _ {t _ {j}} \left(\hat {x} _ {\text {i n}} ^ {c}, y ^ {c}\right) \oplus \dots \tag {10}
$$
where  $T$  denotes a set of pre-defined templates,  $t_j \in T$  and  $t_j \neq t_0$ .  $\hat{x}_{\mathrm{in}}^c$  is another randomly sampled example as the demonstration text and  $\hat{x}_{\mathrm{in}}^c \neq x_{\mathrm{in}}$ . This strategy serves as a perfect form of augmentation for our purpose as it does not generate incomplete or inconsistent sentences, and since we do not edit the main input, the label for that input stays the same. Furthermore,  $\tilde{x}_{2k}$  has a substantial variation from  $\tilde{x}_{2k-1}$ , which allows for effective contrastive learning.
D Manual primary prompts
Table D.1 shows the primary prompts we used for each task. Those prompts are manually chosen by LM-BFF (Gao et al., 2021).
E Experiments with RoBERTa-large
While we use RoBERTa-base to conduct extensive experiments in our main study and ablations, here we compare our framework to LM-BFF using RoBERTa-large. We also include results directly reported from LM-BFF (Gao et al., 2021) (henceforth referred to as LM-BFF†), though the comparison between them could be unfair since the results reported in the original LM-BFF paper are:
- obtained with an additional sampling strategy to select similar demonstrations (section 6.2 of their paper), which put our results at a disadvantage.
- obtained from a set of batch sizes (2,4,8) and learning rates  $(1e^{-5}, 2e^{-5}, 5e^{-5})$  and the best models are selected from the validation set. Whereas we show experimental results with models trained with a fixed batch size and a learning rate of  $1e^{-5}$ .
Nevertheless, we show the state-of-the-art results we achieved in Table E.1. We only marginally under-perform LM-BFF† in 2 tasks, possibly due to the reasons listed above.
Table D.1: Primary templates and label words used in our experiments.
Task	LM-BFF†	LM-BFF‡	ours
SST-2 (acc)	92.6 (0.5)	91.9 (1.3)	94.2 (0.7)
Subj (acc)	92.3 (0.8)	91.0 (2.3)	92.4 (0.6)
SST-5 (acc)	50.6 (1.4)	51.5 (1.4)	54.0 (0.8)
CoLA (Matt.)	18.7 (8.8)	11.6 (5.4)	18.1 (10.1)
TREC (acc)	87.5 (3.2)	85.4 (2.6)	89.8 (1.8)
MNLI (acc)	70.7 (1.3)	70.0 (2.2)	72.4 (2.0)
MNLI-mm (acc)	72.0 (1.2)	71.8 (2.0)	74.2 (1.9)
SNLI (acc)	79.7 (1.5)	79.5 (1.7)	79.6 (2.6)
QNLI (acc)	69.2 (1.9)	67.9 (3.4)	71.1 (6.8)
QQP (acc)	N/A	71.1 (2.3)	74.0 (2.5)
RTE (acc)	68.7 (2.3)	69.3 (1.1)	71.8 (1.1)
MRPC (F1)	77.8 (2.0)	77.0 (5.7)	77.8 (4.6)
MR (acc)	86.6 (2.2)	85.6 (3.4)	89.6 (0.8)
MPQA (acc)	87.0 (1.1)	86.9 (1.3)	86.9 (1.1)
CR (acc)	90.2 (1.2)	90.4 (1.2)	91.0 (1.4)
Table E.1: Comparison of our method using RoBERT-large to two versions of LM-BFF: (1)  $\dagger$  the results reported in LM-BFF (Gao et al., 2021). Note that we do not have their strategy of sampling similar demonstrations, which may put us at a disadvantage. The training details, hyper-parameters and number of GPUs are also different (2)  $\ddagger$  The results of LM-BFF using the same batch size, learning rate, training steps, and number of GPUs. These results make the fairest comparison to ours.
Task	SR	RI	RS	RD	EDA	ours
SST-2 (acc)	90.6 (0.5)	90.8 (0.4)	90.8 (0.4)	90.8 (0.4)	90.7 (0.6)	90.6 (0.1)
Subj (acc)	90.4 (1.4)	90.4 (1.3)	90.4 (2.5)	90.3 (1.4)	90.4 (1.1)	90.4 (1.1)
SST-5 (acc)	47.6 (1.4)	47.0 (1.8)	46.5 (1.7)	46.9 (1.9)	45.2 (2.1)	49.5 (1.1)
CoLA (Matt.)	6.0 (5.3)	6.0 (5.4)	7.2 (3.8)	5.6 (3.0)	5.6 (2.9)	10.2 (5.8)
TREC (acc)	80.7 (2.2)	79.1 (3.8)	81.2 (2.2)	82.8 (2.1)	81.1 (4.3)	83.3 (1.5)
MNLI (acc)	60.3 (2.2)	61.2 (2.1)	60.7 (2.6)	60.2 (2.1)	58.3 (2.5)	64.0 (2.0)
MNLI-mm (acc)	62.2 (1.5)	63.3 (1.3)	63.0 (1.7)	62.9 (1.2)	60.2 (2.1)	65.5 (2.7)
SNLI (acc)	63.4 (4.1)	63.4 (3.8)	62.3 (3.6)	62.0 (4.3)	63.0 (4.2)	69.9 (2.4)
QNLI (acc)	63.2 (3.3)	64.5 (4.6)	64.0 (4.3)	64.8 (4.5)	60.8 (3.6)	66.4 (3.5)
QQP (acc)	64.8 (2.8)	62.9 (2.2)	62.0 (3.5)	63.9 (2.1)	60.4 (5.8)	68.8 (3.8)
RTE (acc)	61.2 (3.0)	62.2 (3.7)	47.9 (0.8)	47.9 (0.8)	64.3 (2.7)	65.1 (3.5)
MRPC (F1)	77.0 (3.8)	79.2 (4.6)	78.5 (2.1)	77.4 (3.2)	76.2 (6.0)	78.2 (3.1)
MR (acc)	83.3 (1.4)	85.5 (0.5)	85.6 (0.6)	85.2 (0.3)	85.7 (0.7)	85.8 (0.6)
MPQA (acc)	82.6 (2.8)	82.7 (2.4)	83.4 (2.4)	84.3 (2.0)	83.1 (2.9)	84.6 (1.5)
CR (acc)	87.8 (0.9)	88.1 (0.3)	87.5 (1.0)	88.5 (1.0)	87.9 (0.5)	89.4 (1.0)
F Augmentation with  $20\%$  of input tokens
In the main paper, we compare our augmentation strategy (random templates and random demonstrations) to standard augmentation techniques on  $10\%$  of input tokens for creating multi-view of inputs to apply the SupCon loss. Here, we show additional experimental results with synonym replacement (SR), random insertion (RI), random swapping (RS), random deletion (RD) and EDA (Wei and Zou, 2019) (with SR, RI, RS and RD all together) at  $20\%$  of input tokens. Same as before, the model under-performs when using standard augmentations. Results are shown in Table F.1.
G SimCLR vs. SupCon
Here, we compare our choice of contrastive loss SupCon (Khosla et al., 2020) to an unsupervised version SimCLR (Chen et al., 2020a). Unsupervised contrastive loss clusters examples at the instance level, i.e., it only pulls the same instance under different views close to each other and push away all the others in a mini-batch. Whereas SupCon clusters examples at the class level.
As shown in Table G.1. SupCon is better than SimCLR in all tasks. In many cases, SimCLR even underperforms the baselines by large margins (see LM-BFF in Table 1), indicating that learning
Table F.1: Comparing our random templates/demonstrations as data augmentation to synonym replacement (SR), random insertion (RI), random swapping (RS), random deletion (RD) and EDA (Wei and Zou, 2019) (with SR, RI, RS and RD all together) at  $20\%$  of input tokens. The results are means of 5 runs with different train-test splits.
Task	SimCLR	SupCon
SST-2 (acc)	89.7 (0.8)	90.6 (0.1)
Subj (acc)	86.4 (1.2)	90.4 (1.1)
SST-5 (acc)	42.1 (0.8)	49.5 (1.1)
CoLA (Matt.)	1.8 (3.6)	10.2 (5.8)
TREC (acc)	60.0 (4.4)	83.3 (1.5)
MNLI (acc)	52.5 (1.2)	64.0 (2.0)
MNLI-mm (acc)	53.0 (1.2)	65.5 (2.7)
SNLI (acc)	60.0 (4.5)	69.9 (2.4)
QNLI (acc)	53.5 (0.6)	66.4 (3.5)
QQP (acc)	56.4 (2.0)	68.8 (3.8)
RTE (acc)	58.6 (2.8)	65.1 (3.5)
MRPC (F1)	68.2 (6.3)	78.2 (3.1)
MR (acc)	84.7 (1.1)	85.8 (0.6)
MPQA (acc)	82.2 (1.6)	84.6 (1.5)
CR (acc)	88.1 (2.1)	89.4 (1.0)
Table G.1: Comparing SimCLR (Chen et al., 2020a) and SupCon (Khosla et al., 2020) as different forms of contrastive loss.
discriminative features at instance level only can hurt the fine-tuning process.
H [CLS] vs. [MASK]
SupCon takes the representations of inputs to perform contrastive learning. We use the hidden states at [MASK] tokens as the representations of sentences in the main experiments. Another common choice is to take the hidden states at [CLS] tokens.
Task	[CLS]	[MASK]
SST-2 (acc)	90.4 (0.5)	90.6 (0.1)
Subj (acc)	90.0 (1.5)	90.4 (1.1)
SST-5 (acc)	48.2 (0.6)	49.5 (1.1)
CoLA (Matt.)	10.7 (5.7)	10.2 (5.8)
TREC (acc)	81.1 (1.4)	83.3 (1.5)
MNLI (acc)	59.9 (3.7)	64.0 (2.0)
MNLI-mm (acc)	61.4 (4.0)	65.5 (2.7)
SNLI (acc)	66.9 (2.5)	69.9 (2.4)
QNLI (acc)	65.1 (3.1)	66.4 (3.5)
QQP (acc)	66.0 (3.0)	68.8 (3.8)
RTE (acc)	63.4 (3.1)	65.1 (3.5)
MRPC (F1)	77.7 (1.8)	78.2 (3.1)
MR (acc)	85.5 (0.8)	85.8 (0.6)
MPQA (acc)	84.2 (1.6)	84.6 (1.5)
CR (acc)	88.4 (1.2)	89.4 (1.0)
Table H.1: Using hidden states at [CLS] tokens or [MASK] tokens as the representations of sentences to perform contrastive learning.
For example, in standard fine-tuning, the algorithm takes the representation of a sentence at [CLS] token and attaches a linear classifier on top of it.
Based on Table H.1, applying the contrastive loss at [MASK] tokens is generally better than applying it at [CLS]. This is fairly intuitive, as the final classifications are performed at [MASK] tokens and enforcing class-level discriminative representations explicitly at [MASK] tokens helps models generalize better after fine-tuning.
I Adapting ADAPET
For results in Table 2, we adapt the open-source code from ADAPET to our codebase. In original ADAPET, there are multiple label words corresponding to a label class (e.g positive class: "great", "good", "nice"). To make a fair comparison to LM-BFF and ours, we only apply one label word corresponding to a label class (e.g positive class: "great"). The original Label Condition Loss implemented in ADAPET has a hyper-parameter  $\alpha$  to control the percentage of input tokens used for masked language modeling (see ADAPET (Tam et al., 2021) for more details). To match the training objective in LM-BFF with only one [MASK] token, we also set the Label Condition Loss to apply to one random token of inputs, i.e.,  $\alpha = \frac{1}{\text{len(input)}}$ .