SlowGuess/NAACL_2022_OCR huggingface dataset:
Han Wang $^{1*}$ , Canwen Xu $^{2*†}$ , Julian McAuley $^{2}$ $^{1}$ New York University,  $^{2}$ University of California, San Diego  
 $^{1}$ hwang@nyu.edu,  $^{2}$ $\{\text{c} \text{x} \text{u}, \text{jmcauley}\} @$ ucsd.edu
Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
Since the release of GPT-3 (Brown et al., 2020), several studies have focused on exploiting pretrained language models with only a few training examples (Brown et al., 2020; Gao et al., 2021; Shin et al., 2020). These works demonstrate the potential of using natural language prompts to encourage the model to recall similar patterns in its training corpus and thus make accurate predictions. This setting of few-shot learning is closer to how humans learn to solve a task, often without many examples as in a traditional deep learning paradigm. The use of prompts can strengthen the explicit connection between input and output, helping the model exploit the knowledge learned from pretraining in a better way. Furthermore, recent works (Schick and Schütze, 2021a,b; Gao et al., 2021) show that prompts can also help the model generalize better in fine-tuning.
Prompt-based learning (i.e., prompting) aims to use a template to convert the original input into a prompt-based input with some unfilled masked
tokens, and then use the pretrained language model to fill these masked tokens, and finally the tokens filled into these slots are mapped to the corresponding labels as the final output. In prompting, the design of prompts often plays an important role. Many attempts have been made in this emerging direction of prompt engineering (Shin et al., 2020; Gao et al., 2021). Meanwhile, finding a good mapping from the original task labels to tokens (i.e., label engineering) is also critical to few-shot performance, as found in Schick et al. (2020); Gao et al. (2021). However, manually assigning the label mapping requires human expertise with trial and error. One may argue that the same effort can be used to label more supervised data for a conventional deep learning pipeline. Thus, an efficient automatic label mapping method is desirable.
In this paper, we aim to design a method that can automatically find a good label mapping to save human effort from label engineering. We propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to tackle the label selection problem for few-shot classification. AMuLaP is a parameter-free statistical technique that can identify the label patterns from a few-shot training set given a prompt template. AMuLaP exploits multiple labels to suppress the noise and inherently extend the training set for prompt-based fine-tuning. Compared with a hand-crafted label mapping and previous works on automatic label mapping (Schick et al., 2020; Gao et al., 2021), AMuLaP achieves competitive performance despite being simpler and does not require access to the weights of the backbone model, or finetune an external pretrained language model for searching label mapping. We conduct extensive experiments and demonstrate the effectiveness of our method under multiple settings. Moreover, we attempt to scale AMuLaP with different sizes of the training set and find AMuLaP to work surprisingly well even with one or two shots. We further analyze
To whom correspondence should be addressed.
1The code is available at https://github.com/HanNight/AMuLaP.
Human Language Technologies, pages 5483 - 5492
why does AMuLaP work and discuss the pros and cons of prompting as a new paradigm.
Discrete Prompts The release of GPT-3 (Brown et al., 2020) has led to interest in prompting, a new way to leverage pretrained language models (PLM). Brown et al. (2020) proposes an intuitive in-context learning paradigm by concatenating a few input and output examples and feeding them to the language model and let the model autoregressively generate answers for new examples. Recent works (Petroni et al., 2019; Davison et al., 2019; Jiang et al., 2020) design prompts to probe the factual and commonsense knowledge encoded within a PLM. Recent works (Schick and Schütze, 2021a,b; Gao et al., 2021) demonstrate that even smaller PLMs have similar few-shot learning capacity. Le Scao and Rush (2021) analyzes the effect of prompting and concludes that a single prompt may be worth 100 training examples in fine-tuning.
Instead of manually designing prompts (i.e., prompt engineering), some recent studies also explore automatic prompt generation. PETAL (Schick et al., 2020) augments Pattern Exploiting Training (PET, Schick and Schütze, 2021a,b) with automatically identified label words; Gao et al. (2021) uses re-ranking to find the best label words by fine-tuning a RoBERTa model on the candidates searched by RoBERTa, and using an external generation model for data augmentation of prompt templates; AutoPrompt (Shin et al., 2020) uses a gradient-based search to determine both prompts and label words. However, these methods require parameter updates with gradient descent, which is infeasible without access to the model weights (e.g., GPT-3). PET and its variants also require a large unlabeled set and need to be fine-tuned multiple times. AutoPrompt uses discretization techniques to approximately map a continuous vector back to tokens in the vocabulary (i.e., "vocabulary"). These searched prompts and labels are often uninterpretable by humans. Different from these prior studies, our proposed AMuLaP is a simple and interpretable method for few-shot prompting that can work well with and without access to model weights. Concurrently to our work, Hu et al. (2021) propose a method that exploits an external knowledge base to find label mapping. T0 (Sanh et al., 2022; Bach et al., 2022) constructs a dataset of different NLP tasks
by manually writing prompt templates and shows that a large language model with multitask training can generalize to unseen tasks.
Continuous Prompts In parallel with text-based discrete prompts, there is also a line of work focused on tuning only a fraction of parameters of an LM with the help of continuous prompts (i.e., soft prompts). Zhong et al. (2021) and Qin and Eisner (2021) propose continuous prompts for knowledge probing by tuning some trainable vectors in the input sequence while fixing the rest of the input. Li and Liang (2021) applies a similar method for natural language generation and achieves comparable performance to fine-tuning while updating only  $0.1\%$  of model parameters. Lester et al. (2021) reveals that prompt tuning is more competitive when scaled up and can achieve identical performance to conventional fine-tuning when the model is large enough. Guo et al. (2021) introduces Q-Learning to optimize the soft prompt. Notably, different from discrete prompting, these works often use all training data to update model weights. Different from these works, AMuLaP is a discrete prompting method that has better interpretability and works well in the few-shot setting.
3 Prompting for Few-Shot Classification
We follow the setup in LM-BFF (Gao et al., 2021) for few-shot text classification. Given a pretrained language model  $\mathcal{L}$ , a task  $\mathcal{D}$  and its defined label space  $\mathcal{V}$ , we have  $n$  training examples per class for the training set  $\mathcal{D}_{train}$ . As pointed out in Perez et al. (2021), using the full development set may be misleading to claim a few-shot setting. Thus, we use a few-shot development set with the same size as the training set (i.e.,  $|\mathcal{D}_{train}| = |\mathcal{D}_{dev}|$ ), to be consistent with Gao et al. (2021) and constitute a "true few-shot" setting (Perez et al., 2021).
For an input example  $x$  (a single sentence or a sentence pair), we first use a task-specific template  $\mathcal{T}$  to convert it to  $x'$ , a token sequence with a [MASK] token. We then map the original label space to a set of selected words from the vocabulary, denoted as  $\mathcal{M}: \mathcal{V} \rightarrow \mathcal{V}'$ . Some examples of  $\mathcal{T}$  and  $\mathcal{M}$  are shown in Table 1. Note that since we focus on automatically finding the label mapping  $\mathcal{M}$ , we use the manual templates  $\mathcal{T}$  from Gao et al. (2021) throughout this paper. Since  $\mathcal{L}$  is trained to complete the [MASK] token in an input sequence, we can directly make zero-shot prediction of the probability of class  $y \in \mathcal{V}$  by the masked language
Task	Template	Class	Manual (2021)	Labels selected by AMuLaP
MNLI	<S1>? [MASK], <S2>	entailmentneutralcontradiction	YesMaybeNo	Yes, Indeed, Also, CurrentlyHistorically, Suddenly, Apparently, AndNo, However, Instead, Unfortunately
SST-2	<S1> It was [MASK].	positivenegative	greatterrible	great, perfect, fun, brilliant terrible, awful, disappointing, not
QNLI	<S1>? [MASK], <S2>	entailmentnot_ entailment	YesNo	Yes, Historically, Overall, IndeedWell, First, However, Unfortunately
RTE	<S1>? [MASK], <S2>	entailmentnot_ entailment	YesNo	Yes, Today, Specifically, AdditionallyHowever, Ironically, Also, Indeed
MRPC	<S1> [MASK], <S2>	equivalentnot_equivalent	YesNo	
, Currently, Additionally, TodayHowever, Meanwhile, Overall, Finally
QQP	<S1> [MASK], <S2>	equivalentnot_equivalent	YesNo	Or, So, Specifically, ActuallyAlso, And, Finally, Well
CoLA	<S1> This is [MASK].	grammaticalnot_grammatical	correctincorrect	why, true, her, amazingit, ridiculous, interesting, sad
Table 1: The manual and automatically selected labels by AMuLaP. The templates used for prompting are from Gao et al. (2021).
modeling:
$$
p \left(y | x\right) = p \left(\left[ \mathrm {M A S K} \right] = \mathcal {M} (y) \mid x ^ {\prime}\right). \tag {1}
$$
Alternatively, one can further fine-tune  $\mathcal{L}$  with supervised pairs  $\{x',\mathcal{M}(y)\}$  to achieve even better performance.
4 Automatic Multi-Label Prompting
4.1 Exploiting Multiple Labels
Selecting one label word can be insufficient for some complicated tasks, as mentioned in Schick et al. (2020). We also argue that selecting only one label (especially automatically) may bring noise. This can be resolved by introducing multiple label words. Schick et al. (2020) use multiple label combinations for PET (Schick and Schütze, 2021a) and ensemble them afterwards. We instead use a straightforward sum to consider multiple label words when making predictions. This design has a similar advantage of exploiting multiple labels without training and assembling multiple models.
Instead of a one-to-one mapping from the original label space  $\mathcal{V}$  to  $\mathcal{V}$ , we map each  $y\in \mathcal{V}$  to its label word set  $S(y)$  of  $k$  words. We denote the mapping function as  $\mathcal{M}':\mathcal{V}\to \mathcal{V}^k$ . For class  $y\in \mathcal{V}$ , the predicted probability is calculated as:
$$
p (y | x) = \sum_ {v \in \mathcal {S} (y)} p ([ \text {M A S K} ] = v \mid x ^ {\prime}) \tag {2}
$$
Then, we can simply make predictions by selecting the label with the largest likelihood.
Similarly, if we need to fine-tune  $\mathcal{L}$  with supervised pairs, instead of optimizing the cross-entropy loss between the gold label and a single token, we optimize the loss between the sum of the output probabilities of  $S(y)$  and the gold label with a cross-entropy loss:
$$
l = - \sum_ {x \in \mathcal {D} _ {\text {t r a i n}}} \sum_ {y \in \mathcal {Y}} [ \mathbb {1} [ y = \hat {y} ] \cdot \log p (y | x) ] \tag {3}
$$
where  $\hat{y}$  is the ground truth label for the input  $x$  and  $p(y|x)$  is defined in Equation 2.
4.2 Automatic Label Selection
Finding a good label mapping  $\mathcal{M}$  is non-trivial, especially when  $\mathcal{M}'$  maps an original label to a set of label words instead of one. Selecting a good label mapping often requires significant human effort, including domain knowledge and trial-and-error. Previously, Schick and Schütze (2021a,b) both use hand-crafted label mappings while Schick et al. (2020) explores automatic label mapping searching but it still requires manual pre-filtering and significantly underperforms the manual mapping. (Gao et al., 2021) exploits a large pretrained masked language model (RoBERTa, Liu et al., 2019) to construct a pruned set of label words and then determine the final mapping by fine-tuning on all of them and selecting the best one with  $\mathcal{D}_{dev}$ . We introduce a new selection algorithm for label mapping that achieves competitive results compared to previous efforts.
Figure 1: The illustration of implementing AMuLaP on a binary sentiment classification task (SST-2). Each training sample with the task-specific template (the underlined text) is fed into a pretrained language model  $\mathcal{L}$  to get its own probability distribution over the vocabulary  $\mathcal{V}$ . All the obtained probability distributions are summed by class and normalized to get the probability distribution of each class. Then each token in  $\mathcal{V}$  is assigned to the class with the highest probability (e.g., the token terrible is assigned to the class negative, the token great is assigned to the class positive). Finally, for each class, we choose the top- $k$  tokens as label words.
We aim to achieve two goals: (1) Selecting the most likely label mapping based on the training set. For example, in a sentiment classification task, we would like to see positive words in the label set of the "positive" class while negative words in the label set of the "negative" class. A simple solution is to select the  $k$  most likely tokens predicted for the [MASK] token in the training examples of each class  $y$ . However, in practice, we would find common words in more than one label set. For example, if we simply take the 10 most likely tokens for the SST-2 dataset (Socher et al., 2013), we would find "good" in both positive and negative label sets, although it is ranked second place in the positive set and ninth in the negative set. Thus, we want to make sure that (2) Each token only belongs to at most one label set where it has the highest probability. To ensure this, we have to iterate over the vocabulary and check that for every token. Then, we can truncate the candidate sets of each class and select the  $k$  most likely tokens from each set. The time complexity of this algorithm is  $O(k \cdot |\mathcal{V}| \cdot |\mathcal{Y}|)$ .
Formally, we select  $\mathcal{M}':\mathcal{V}\to \mathcal{V}^k$  by the following steps:
1. For each  $y_{i}\in \mathcal{V}$ , we iterate through all training samples  $x_{j}\in \mathcal{D}_{train}$  whose ground truth label  $\hat{y}_j = y_i$ . We use  $\mathcal{L}$  to predict the token probability of the [MASK] token and take the
average of the predicted probabilities of the  $n$  examples to be  $\mathbf{z}_i$ , where  $\mathbf{z}_i$  is a vector over the whole vocabulary.
2. For each  $y_{i}\in \mathcal{V}$ , initialize an empty candidate token set  $\tilde{S} (y_i)$
3. For each  $v \in \mathcal{V}$  where  $\mathcal{V}$  is the vocabulary of the model  $\mathcal{L}$ , we retrieve  $v$ 's probability value  $z_{i}^{v}$  from  $\mathbf{z}_i$  of each class.
4. We assign  $v$  to the most likely token set of the  $m$ -th class  $\tilde{S}(y_m)$  where  $m = \operatorname{argmax}_i z_i^v$ .
5. For  $y_{i}\in \mathcal{V}$ , we choose the top- $k$  tokens from  $\tilde{S} (y_i)$  with the largest probability  $z_{i}^{v}$  and obtain the truncated word set  $S(y_{i})$ .
The entire workflow is illustrated in Figure 1.
Datasets We evaluate seven classification tasks of the GLUE benchmark (Wang et al., 2019). Specifically, we test on Microsoft Research Paraphrase Matching (MRPC) (Dolan and Brockett, 2005), Quora Question Pairs (QQP) for Paraphrase Similarity Matching; Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) for Sentiment Classification; Multi-Genre Natural Language Inference Matched (MNLI-m), Multi-Genre
MNLI (acc)	MNLI-mm (acc)	SST-2 (acc)	QNLI (acc)	RTE (acc)	MRPC (F1)	QQP (F1)	CoLA (Matt.)	Avg.
Baselines
Majority	32.7	33.0	50.9	49.5	52.7	81.2	0.0	0.0	37.5
Manual Label 0-shot (2021)	50.8	51.7	83.6	50.8	51.3	61.9	49.7	2.0	50.2
Full Fine-tuning	89.8	89.5	95.0	93.3	80.9	91.4	81.7	62.6	85.5
Setting 1: Dtrain only; No parameter update.
In-context learning (2020)	52.0 (0.7)	53.4 (0.6)	84.8 (1.3)	53.8 (0.4)	60.4 (1.4)	45.7 (6.0)	36.1 (5.2)	-1.5 (2.4)	48.1 (2.3)
AMuLaP (ours)	50.8 (2.1)	52.3 (1.8)	86.9 (1.6)	53.1 (2.8)	58.9 (7.9)	56.3 (5.0)	60.2 (2.7)	2.3 (1.4)	52.6 (3.2)
Setting 2: Dtrain + Ddev; No parameter update.
PETAL-CE (2020)	48.8 (2.6)	49.7 (2.3)	75.6 (7.2)	49.5 (0.0)	63.5 (3.3)	28.9 (39.6)	59.2 (0.0)	1.3 (3.0)	47.1 (7.3)
PETAL-LR (2020)	38.6 (2.0)	38.4 (2.1)	85.3 (3.3)	53.3 (3.6)	54.7 (6.4)	28.0 (38.5)	55.6 (2.8)	1.5 (3.4)	44.4 (7.8)
Auto-L (2021)	41.6 (5.4)	42.3 (6.2)	84.3 (3.3)	57.9 (3.9)	61.9 (7.5)	67.7 (7.9)	55.5 (5.0)	1.2 (4.8)	51.6 (5.5)
AMuLaP (ours)	50.8 (2.1)	52.2 (1.9)	87.0 (1.5)	53.5 (2.3)	59.1 (7.4)	56.7 (5.7)	61.5 (1.7)	2.6 (1.8)	52.9 (3.1)
Auto-L + AMuLaP (ours)	52.9 (3.0)	54.2 (2.7)	90.1 (0.4)	57.9 (2.6)	59.9 (5.2)	66.0 (3.0)	59.4 (2.3)	2.7 (5.7)	55.4 (3.1)
Setting 3: Dtrain + Ddev; Prompt-based fine-tuning.
Fine-tuning	45.8 (6.4)	47.8 (6.8)	81.4 (3.8)	60.2 (6.5)	54.4 (3.9)	76.6 (2.5)	60.7 (4.3)	33.9 (14.3)	57.6 (6.1)
Manual Label FT (2021)	68.3 (2.3)	70.5 (1.9)	92.7 (0.9)	64.5 (4.2)	69.1 (3.6)	74.5 (5.3)	65.5 (5.3)	9.3 (7.3)	64.3 (3.9)
PETAL-CE FT (2020)	57.5 (3.2)	57.7 (2.6)	92.6 (1.0)	50.5 (0.0)	68.6 (6.5)	32.1 (42.5)	66.7 (3.2)	3.8 (8.4)	53.7 (8.4)
PETAL-LR FT (2020)	64.0 (6.5)	65.9 (6.4)	92.9 (1.7)	65.5 (6.8)	63.3 (7.7)	77.7 (3.9)	65.7 (4.2)	11.9 (7.5)	63.4 (5.6)
Auto-L FT (2021)	64.8 (4.7)	67.3 (4.3)	93.5 (0.5)	69.8 (3.0)	67.4 (3.9)	76.2 (4.8)	66.4 (4.5)	23.2 (17.1)	66.1 (5.4)
AMuLaP FT (ours)	70.6 (2.7)	72.5 (2.4)	93.2 (0.7)	65.1 (5.9)	65.9 (6.3)	79.3 (4.0)	69.1 (2.5)	18.3 (9.4)	66.8 (4.2)
Auto-L + AMuLaP FT (ours)	68.5 (2.2)	71.1 (2.3)	93.4 (1.0)	69.6 (1.1)	69.4 (4.0)	75.5 (5.6)	66.4 (3.0)	14.2 (14.0)	66.0 (4.2)
Table 2: Experimental results under three settings with RoBERTa-large as  $\mathcal{L}$ . For few-shot settings,  $n$  is set to 16 per class. We report the average of 5 runs along with their standard deviation in the parentheses.
Natural Language Inference Mismatched (MNLLmm) (Williams et al., 2018), Question Natural Language Inference (QNLI) (Rajpurkar et al., 2016) and Recognizing Textual Entailment (RTE) (Wang et al., 2019) for the Natural Language Inference (NLI) task; The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) for Linguistic Acceptability. We use the manual templates in Gao et al. (2021), as listed in Table 1. The metrics for each dataset are indicated in Table 2.
Baselines We compare our method to various baselines:
- Majority: always predict the majority class in the test set.
- GPT-3-style in-context learning (Brown et al., 2020): present a few examples to the language model and make it directly predict the next token as the prediction.
- Manual prompts: we use the human-designed prompts in Gao et al. (2021).
- PETAL-CE (Schick et al., 2020): the variant of PETAL using the cross-entropy metric.
- PETAL-LR (Schick et al., 2020): the variant of PETAL using the likelihood ratio metric.
- Auto-L (Gao et al., 2021): the automatic label searching method with an external pretrained language model, RoBERTa-large (Liu et al., 2019). The detailed description can be found in Appendix A. Note that the results of this baseline is different from those reported in Table 3 of Gao et al. (2021) since they search for both templates and label mapping whereas we fix the templates and search for the label mapping alone, for the sake of fair comparison. We use the officially released code and same hyperparameters for this baseline.
Task Setup We closely follow the setup in Gao et al. (2021). We sample  $n$  training examples and  $n$  development examples per class. We set  $k = 16$  throughout all experiments. We use RoBERTa-large (Liu et al., 2019) as the backbone LM  $\mathcal{L}$ . For each reported result, we measure average performance across 5 different randomly sampled  $D_{train}$  and  $D_{dev}$  splits. Following Gao et al. (2021), the original development split of each dataset is used as the test set in our experiments. We also report the standard deviation for each result. To fairly compare with different baselines, we consider the following three settings:
- Setting 1: We only use  $\mathcal{D}_{train}$  alone for both label selection and tuning  $k$ . The parameters of  $\mathcal{L}$  are not updated.  $\mathcal{D}_{dev}$  is not used. This
Class	PETAL-CE (Schick et al., 2020)	PETAL-LR (Schick et al., 2020)
positive	amazing, great, brilliant, perfect, fun, wonderful, beautiful, fantastic, awesome, not	superb, fearless, acclaimed, addictive, visionary, immersive, irresistible, timely, unforgettable, gripping
negative	not, awful, fun, funny, terrible, great, amazing, hilarious, awesome, good	annoying, insulting, meaningless, lame, shitty, humiliating, childish, stupid, embarrassing, irritating
Class	Auto-L (Gao et al., 2021)	AMuLaP (ours)
positive	exquisite, perfection, effective, fabulous, intense inspiring, spectacular, sublime, astounding, thrilling	great, perfect, fun, brilliant, amazing, good, wonderful, beautiful, excellent, fantastic
negative	embarrassing, boring, frustrating, ridiculous, awkward silly, nothing, disgusting, ugly, confusing	terrible, awful, disappointing, not, horrible, obvious, funny, inevitable, bad, boring
Table 3: Most likely label mapping for the SST-2 dataset obtained by PETAL (Schick et al., 2020), Auto-L (Gao et al., 2021) and our AMuLaP. Suitable labels annotated by the human annotator are underlined.
setting is for fair comparison with In-context learning.
- Setting 2: We use  $\mathcal{D}_{train}$  for label selection and an additional  $\mathcal{D}_{dev}$  for  $k$  tuning. The parameters of  $\mathcal{L}$  are not updated. This setting is for fair comparison with Auto-L (Gao et al., 2021) and PETAL (Schick et al., 2020).
- Setting 3: We use  $\mathcal{D}_{train}$  and  $\mathcal{D}_{dev}$  in the same way as Setting 2 but fine-tune the parameters of the language model  $\mathcal{L}$ . This setting is for fair comparison with conventional fine-tuning, prompt-based fine-tuning with manual prompts, Auto-L (Gao et al., 2021) and PETAL (Schick et al., 2020).
Implementation Details We implement AMuLaP based on Hugging Face Transformers (Wolf et al., 2020). When selecting  $k$ , if there are multiple  $k$  with identical performance (which happens occasionally given there are only 16 examples for each class in  $\mathcal{D}_{dev}$ ), we always choose the largest  $k$ . For Settings 1 and 2, we search  $k$  over  $\{1,2,4,\dots,1024\}$ . Note that for settings that do not update the parameters of  $\mathcal{L}$ , search over  $k$  is fast, as we only need to run the model once and cache the distribution of the [MASK] token. For prompt-based fine-tuning (Setting 3), where we fine-tune the model  $\mathcal{L}$ , we search  $k$  in a smaller space  $\{1,2,4,8,16\}$  due to the increased computational overhead. Following (Gao et al., 2021), we grid search the learning rate from  $\{1e-5,2e-5,5e-5\}$  and batch size from  $\{2,4,8\}$ .
We demonstrate experimental results under three settings in Table 2. Under Setting 1, AMuLaP