Pre-trained Language Models can be Fully Zero-Shot Learners
Authors: Xuandong Zhao
Siqi Ouyang
Zhiguo Yu
Ming Wu
Lei Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: pre-trained language models
zero-shot learning
prompt

Abstract: How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, and paraphrasing. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 18.9% on the GLUE benchmark.

Under review as a conference paper at ICLR 2023
PRE-TRAINED LANGUAGE MODELS CAN BE FULLY ZERO-SHOT LEARNERS
Anonymous authors Paper under double-blind review
ABSTRACT
How extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or man- ually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Un- like previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text en- tailment, similar text retrieval, and paraphrasing. Experimental results demon- strate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy text classifica- tion and 15.6% on the GLUE benchmark. Our source code is available at https://anonymon 4open. /cPrrot/r///PPrropt.
INTRODUCTION
Natural language understanding (NLU) has been important in many applications such as intelligent dialog assistants, online search, and social media analysis. Recent advancement of NLU has been driven by emergent pre-trained language models (PLMs) including BERT (Devlin et al., 2019; Liu et al., 2019b), GPT (Radford et al., 2018; 2019; Brown et al., 2020). BART (Lewis et al., 2020), and T5 (Raffel et al., 2020). Prior studies show that PLMs obtain substantial knowledge during pre-training on raw text corpus (Petroni et al., 2019; Feldman et al., 2019). By fine-tuning task- specific labeled data, PLMS exploit such knowledge and gain impressive accuracy on wide range of NLP tasks, such text classification (Kowsari al., 2019), question answering (Rajpurkar al., 2016), machine reading comprehension (Campos et al., 2016), etc.
However, fine-tuning approaches are expensive. It requires labeled datasets, which are rarely avail- able for many tasks. Significant computational efforts are needed to update PLMs' parameters for multiple tasks. In addition, fine-tuning results in one distinct model for each task maintain.
How can we generalize a pre-trained model to many NLP tasks, without labeled or additional un- labeled data? Existing few-shot and zero-shot approaches propose to construct prompts elicit desired predictions from PLMs (Brown et al., 2020). The main idea of prompting PLMs is to con- vert an input utterance to one with masked templates. For example, in text classification an input can be "The Warriors won the NBA championship 2022" and it is instead converted to "A [MASK] news: The Warriors won the NBA championship 2022". PI. (e.g. BERT) takes the converted text and produces predictions for the masked token, along with the probability. Ideally, PLM will generate a higher probability for the word "sports" than "politics" on the [MASK] token.
Although these prompting-based methods are effective, they require unlabeled data for training or huge human efforts to construct prompts and to choose designated tokens to represent class labels (Schick & Schütze, 202 la;b; Gao et al., 2021). In addition, these manually constructed verbalizers, ie. mapping from words (e.g. "basketball") to class labels (e.g. SPORTS), do not extend to new emerging categories after PL.Ms deployed.

---

Under review as a conference paper at ICLR 2023
In this paper, we investigate the fully zero-shot learning problem for NLU where only the target label names are available but not the extra raw text. We propose nonparametric prompting PLM (NPPrompt), novel method to generate predictions for semantic labels without any fine-tuning. NPPrompt uses PLM's own embeddings to automatically find relevant words to labels (e.g. "bas- ketball" and "NBA" for SPORTS), therefore it does not need humans to construct verbalizers. Our key idea is to search for the top k nearest neighbors to a label name in the embedding manifold and then generate and aggregate PLM's predicted logits from masked prompts. In the above case, both predicted values for "basketball" and "NBA" contribute to the final prediction for the SPORTS cate- gory. In this way, NPPrompt can be easily generalized to any new categories as long as the category names are semantically meaningful.
The contributions of this paper are as follows. a) We develop NPPrompt, novel method for fully zero-shot learning with PLMs. b) We conduct extensive experiments on diverse language under- standing tasks including text classification, text entailment, similar text retrieval, and paraphrasing- Experimental results show that NPPrompt outperforms the previous zero-shot methods by abso- lute 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Surprisingly, NPPrompt is par with the best prior method that trained with manual verbalizers, an additional knowledge base, and extra unlabeled data.
RELATED WORK
Prompting The success of GPT-3 (Brown al., 2020) has attracted much attention to prompting engineering, a new way to leverage pre-trained language models. Brown et al. (2020) concatenate a few input and output pairs and feed them to the large-scale GPT-3 language model, which is an intuitive in in-context learning paradigm, allowing the model to generate answers for additional cases autoregressively. Recent works (Schick & Schütze, 2021a;b) show that small-scale pre- preetrained lan- guage models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b) and ALBERT (Lan et al., 2019) can also achieve decent performance using prompt-tuning. Prompting has been applied to large variety of tasks such as Text Classification (Schick & Schütze, 2021a), Natural Language Understanding (Schick & Schütze, 2021b), Knowledge Probing (Petroni et al., 2019), and Relation Extraction (Han et al., 2021). Typically, a piece of prompt contains template and a verbalizer. The language model predicts a probability distribution over vocabulary given the template and the verbalizer transforms it into a prediction over class labels. In this work, we focus designing the verbalizers automatically.
Verbalizer Design The verbalizer is an important component in prompting which bridges model outputs and labels and greatly impacts the performance. Schick & Schütze (2021a) la) design human written verbalizers for prompting, however, they are highly biased towards personal vocabulary with inadequate coverage. Apart from manually designed verbalizers, some recent studies explore automatic verbalizer construction. Auto-L (Gao et al., 2021) uses re-ranking to find the label words set by fine-tuning the model on the candidates searched by RoBERTa; AutoPrompt (Shin et al., 2020) applies gradient-based search to create both prompts and label words automatically with a few trigger examples. But these approaches need to update parameters with gradient descent, which turns out to be infeasible without access to the model weights (c.g., GPT-3). KPT (Han et al., 2021) incorporates external knowledge into the verbalizer in which the unlabeled dataset is needed to refine the label words and thus is not applicable to scenarios where only label names are known. In contrast, our approach NPPrompt directly finds, without any gradient update, relevant words to label names with only PLM's initial word embedding.
Zero-shot Text Classification The general zero-shot text classification usually focuses on clas- sifying texts into classes that are unseen during the training process. Transferring knowledge from seen classes to unseen ones requires accurate and discriminative descriptions of all classes (Liu et al., 2019a; Xia et al., 2018) joint embeddings of categories and documents (Nam et al., 2016) or seman- tic correlations among classes (Rios & Kavuluru, 2018: Zhang et al., 2019). However, these methods require supervised data for the known label set and thus cannot be extended to settings where no la- beled pairs for any category is available. (Meng et al., 2020) propose the LOTClass model that uses label names with self-training to do zero-shot classification. But LOTClass still requires unlabeled corpus for extracting the tepic-related words and performing self self-training. In this work, NPPrompt achieves competitive and even better performance without using any unlabeled dataset.
2

---

Under review as a conference paper at ICLR 2023
Closest Words "SPORTS"
Initial word embedding
sports, Sports, sport, sparting, athletic, athletics, SPORTS, football, soccer, basketball, tennis NBA,
Logits over Vocabulary
Aggregation
NBA
basketball
Verbalizer
sports
PLM Head
Predict
SCIENCE SPORTS POLITICS
physics
SPORTS
athletics
football
technology
[MASK]
news
The Warriors won NBA championship 2022.
Template
Original input
The illustration of NPPrompt. We generate the label words by searching the related words from the initial word embedding of the pre-trained language model. By aggregating logits from the label words, we predict the category with the largest score (SPORTS).
Figure I:
BACKGROUND. PROMPT-BASED TUNING FOR PLMS
We first provide a standard paradigms, prompt-based tuning, that perform well in few-shot scenar- ios, before introducing our approach for the zero-shot case. Take N way text classification as an example. We aim predict the label y y for each sentence, where y the label set with N distinct classes.
Prompt-based tuning tunes PLM using customized prompts (Brown al., 2020). The regu- lar prompt-based tuning converts a specific task to a cloze-style mask language modeling prob- lem. For each input example x (single sentence or sentence pair), we first apply task tem- plate T on it, converting original input x to Zprompt- For instance, we concatenate the template "T(.) A [MASK] news :" with the original input "The Warriors won the NBA championship 2022" and wrap it into:
Proomp = T(x) = A [MASK] news T
(1)
The verbalizer f in vanilla prompt engineering map set of selected words V from the vocabulary the original label space }/, i.e., f V + y. Inversely, we use Myy,) to denote the label words in y that are mapped into specific label yj. Un, V. Then we calculate the probability of label yj:
P x) 9 (P([MASK]
xprompt) 0, M(y))),
(2)
where g(.) is for aggregating the probability label words into the probability of the label. Then PLMs can be fine-tuned by minimizing the cross-entropy loss with supervised examples.
4 PROPOSED METHOD: NPPROMPT
We inherit PLM with verbalizers framework but keep PLM's parameters frozen (Gao et al., 2021). The key idea of NPPrompt is using PLM's word embeddings to automatically construct verbalizers mapping from words to labels in a fully zero-shot way. It does not need any additional raw text corpus for fine-tuning. NPPrompt consists nonparametric form (Figure 1). 1) We search for all label words closely related to each class yj PLM's s token embedding manifold. 2) Then we use the PLM to predict values for [MASK], filter them using each class's 'ss set of label words, and aggregate the properly weighed outputs to produce the final prediction. In the following, we describe NPPrompt for text classification but generalizes to other language understanding tasks.
two steps to compute predictions for any labels in
k-Nearest- -eearest-Neighbor Verbalizer Construction For each class label (e.g. "SPORTS"), we search over the whole vocabulary V for the top-k words nearest to the label name in the PLM'' embedding
3

---

Under review as a conference paper at ICLR 2023
space. Here, the distance between words and label names is measured using the cosine similarity score. Other distance metrics work as well and are examined in Section 5. We denote as the neighborhood number. Assuming the embeddings of word 2 and label name Y/ are emb(e:) and emb(y;) respectively, the label words of the verbalizer for are selected by top-k ranking:
M(y;) Top-k (S(emb(u:) emb(y)))
where S(-) is the cosine similarity function: S (cmb(e;), emb(y;)) = emb(w.) emb(ay)
[mmb |emb(1y))
Since the PLM is already pre-trained raw text
Word
Sim
Word
Sim
corpus, it acquires sensible semantic knowledge and relatedness of words in the vocabulary. We use PLM's embedding to search for label words semantically relevant given label names, For illustration, we show the found label words of two categories in the AG News dataset (Zhang et al., 2015) and the corresponding similarity scores in Table We also extend our verbalizer to support label names with longer expression in Appendix A.I.
sports"
1.00
"uusiness"
1.00
Sports"
0.77
"usiness"
0.78
sport"
0.75
businesses"
0.74
sporting"
0.68
"business"
0,72
athletics"
0.65
"Business"
0.67
"sports"
0.65
businessman"
0.59
"Sports"
0.65
corporate"
0.58
Sport"
0.62
company"
0.56
athletic"
0.61
enterprise" 0,55
athletes"
0.61
businessmen" 0.55
Table 1: The top 10 similar words of the RoBERTa-large model for the AG News dataset categories SPORTS and BUSINESS. Sim: cosine similarity scores.
Nonparametric Aggregation of Prompted Pre- dictions For each input text z, we construct a prompt-augmented sequence Trommpt with a [MASK] token. We use the PLM to predict tokens for [MASK]. In contrast to previous prompting methods which directly calculate the proba- bility over the surface labels, we use the nearest label words from above to compute the probability for each output label. Only the words in label's top-k neighborhood will contribute to the class prediction. The contribution from each label word is non-equal.
T(x)
To be specific, with T(x), PLM produces the logit vector [ for all possible words at the [MASK] token. Notice that if the whole vocabulary is V, ( E BR( Then we compute the class probability for a label Yj by aggregating the logits filtered by the verbalizer's 's label words. We use kernel smoothing to aggregate follows:
Quilæ) = Σ
w(vi, vi) ([MASKK = Ug Dp/opp = T(x))
(4)
MEE((
Where the weight between label word and class name Vj are defined as:
exp (emb(((),) emb(y;)) vy M(ss) exp (m) emb(y;)))
( y3)
(5)
Finally, the best class prediction selected from the maximum of labels:
argmax Q (y) x)..
(6)
Øi
Notice since we use kernel smoothing on logits instead of probability, Q is also unnormalized prob- ability.
For example, AG News has two classes the verbalizer for SPORTS M(y1) includes label words "sports", "athletics". etc, and the ver- balizer for BUSINESS M(zz) includes label words "business", "corporate", etc. Given an in- put text x "The Warriors won the NBA championship 2022", the prompt-augmentted sequence Tprompt will be "A [MASK] news The Warriors won the NBA championship 2022". The PLM computes logits for every word ([[MASK] = v)zprompt). NPPrompt computes the un- normalized probabilities for SPORTS and BUSINESS, Q(SPORTS]z) ([MMASK] = "sports") "rpotts"""propp""" + w""athletics", SPORtS) [[MMASK] = "athletics"" (prompp)) + w""business". BUSINESS) [[[MASK]
(SCIENCE), = {SPORTS). From Table 1,
w""sports", SPORTS)
Q ((UUSINES| w(" "oorprrrt"""""" BUSINESS) ([MAASK] = "corporate"| pxpompp)) + .... If the aggregated predic- tion for SPORTS is larger than BUSINESS, NPPrompt outputs SPORTS.
"business" [nssiness"/<pcooppd +
4

---

Under review as a conference paper at ICLR 2023
There certain conditions where class has label names containing little semantic meaning where several keywords are needed to define a label. For instance, in the DBPedia dataset (Lehmann et al., 2015), one class is related to NATURALPLACE, then can use the keywords {"river", "lake", "mountain") to represent this class. In this setting, we pick out the keyword with the maximum score calculated by Equation 4 to represent each label first. Then choose the label with the largest score. We use 2 (j²z to denote all keywords in class Vj, and the final prediction is:
y arg max arg max Q (v/ 1z))
(1E((1)))
EXPERIMENT
We conduct extensive zero-shot learning experiments to demonstrate the effectiveness of our method. We present our implementation details together with the main results and address several research questions pertaining to NPPrompt in this section.
5.1 DATASETS, PROMPT TEMPLATES, AND EXPERIMENTAL SETUP
We adopt sentiment classification tasks on two datasets, IMDB (Maas et al., 2011) and Amazon (McAuley & Leskovec, 2013), and topic classification tasks on another two datasets, AG News (Zhang et al, 2015) and DBPedia (Lehmann et al., 2015). All datasets are in the English lan- guage. For each task, we directly use the test set to assess model performances, without incorporating validation or training sets for post- tuning or cherry-pickingg hand-crafted prompts. The statistics of each dataset are shown in Table
Dataset AG News
Classification Type
# Classes
# Test
News Topic
7,600
DBPedia
Wikipedia Topic
14
70,000
IMDB
Movie Review Sentiment
2
25,000
Amazon
Product Review Sentiment
400.000
Table Dataset statistics.
To concentrate on the verbalizer and reduce the influence of templates, we adopt multi- ple fixed manual templates following Hu et al. (2022). We report the best template used for the RoBERTa-large model in Table 3.
Dataset
Template
AG News
A [MASK] news : S.
DBPedia
21 22 In this sentence, æn is [MASK]
IMDB
All in all, it was [MASK] All in all,
Amazon
[MASK].
We implement our experiments based on an open-source toolkit OpenPrompt (Ding et al., 2021), which aims to conduct prompt learning easily. We choose RoBERTa-large (Liu et al., 2019b) pre-trained language model. We report the best accuracy of classification results for all ex- periments using different neighborhood numbers. Since we directly use the pre-trained models for testing, there randomness (random seed) in this process. All experiments are conducted Nvidia A6000 GPUs and more details can be found in Appendix A.
Table 3: Prompt templates for NPPrompt.
5.2 BASELINES
We evaluate the following baseline methods.
Semantic Retrieval We utilize sentence embedding models (Reimers & Gurevych, 2019) to ob- tain the embedding for each sentence and descriptions for each class. Then we calculate the cosine similarity between sentences and label deseriptions. We assign the most similar class labels to the sentence. Particularly, we use model, and the descriptions for each class can be found in Appendix A.I.
from Hugging Faceas the sentence embedding
NSP-BERT Sun et al. (2021) propose text entailment tasks to replace text classification tasks and then use the Next Sentence Prediction (NSP) head to predict the results. We show the template we use in Appendix A.I.
ManualVerb Manual verbalizers are defined by human experts with domain knowledge and we simply use the label words provided by OpenPrompt (Ding al., 2021).
LOTClass Meng al. (2020) employ pre-trained neural language models with unlabeled data for category understanding. i.e., finding words similar to label names. They then introduce a self- training approach to the entire unlabeled corpus to generalize the model.
5

---

Under review as a conference paper at ICLR 2023
GPT-3 with descriptions Following Brown et al. (2020), manually write the descriptions for each class and query GPT-3 where the predicted token serves as the prediction. We show the de- scriptions in Appendix A.1.
KPT
Hu et al. (2022) propose knowledgeable prompt-tuning. which expands the label words space using external knowledge bases (KB). KPT also refines the expanded label words based the unlabeled data. We show the best results of KPT in the zero-shot setting.
Null Prompt IV et al. (2022) insert a token at the end of the text (i.e. using the prompt template [=][MASK]") and then use the prediction of the [MASK] token to perform zero-shot classification.
Multi-Null prompting Wang et al. (2021) find that simply introducing a few prompt [MASK]s can improve the performance and robustness of the Null Prompt in the zero-shot settings.
Method
AG News
DBPedia
IMDB
Amazon
Avg.
Human/KB Unlabeled
ManualVerh
79.60.6
71.71.1
92.0o.7
87.30.4
82.7
Semantic Retrieval
73.11.2
78.6g.8
64.81.3
59.40.7
69.0
X
NSP-BERT
77.40.8
64.75.3
72.81.1
72.73.9
71.9
GPT-3 w. descriptions
83.4
82.5
88.8
89.4
86.0
LOTClass w/o. self train
82.2
86.0
80.2
85.3
83.4
LOTClass
86.4
91.1
86.5
91.6
88.9
KPT
86.7
87.4
94.0
94.6
90.7
Null Prompt
67.92.0
56.83.9
82.51.5
89.41.o
74.2
Multi-Null Prompt
68.21.a
67 07.61.8
86.60.e
86.22.7
77.2
X
X
NPPrompt
85.20.o
86,8p.1
94.2g.2
93.9u.u
90.0
X
X
Table 4: Classification performance on four datasets with average results and standard error. Human: with human efforts write deceptions design label words. KB: with external knowledge base; Unlabeled: with unlabeled corpus. Notice that our method achieves the best performance in a fully zero-shot setting, with absolute improvement of 12.8%. Surprisingly, it even approaches the best result with human effort/know ledge base and extra raw data.
MNLI MNLI-mm SST-2
QNLI
RTE (acc)
MRPC
QQP (F1)
CoLA (Matt.)
Avg.
(acc)
(acc)
(acc)
(acc)
(F1)
With human designed prompts
few-shot data
Manual Label
50.8
51.7
83.6
50.8
51.3
61.9
49.7
2.0
50.2
In-context lcarning
52.0p.r
53.40.6
84.81.a
53.80.4
60.41.4
45.76.0
36.1s.2
-1.52.4
Auto-L
41.65.4
42.36.2
84.3s.3
57.95.9
61.97.5
67.77.a
55.55.0
1.24.8
51.6
AMuLaP
50.82.1
52.31.s
86.91.a
53.12.8
58.97.a
56.35.0
60.22.7
2.31.4
52.6
Few-shot fine-tuning
45.86.4
47.86.
81.4g.s
60.2g.5
54.4g.9
76.62.5
60.74.3
33.9 14.3
57.6
Fully zero-shot
Majority
32.7
33.0
50.9
49.5
52.7
81.2
0.0
0.0
37.5
Null Prompt 38.0s.s
33.10.4
33.80.5
79.14.0
50.7o.1
47.2o.8
12.97.0
1.31.0
-1.12.0
32.1
Multi-Null Prompt
38.54.1
70.27.7
52.21.7
53.02.2
19.9s.7
25.513.4
6.22.0
37.9
NPPrompt
45.70.6
45.9a.5
86.31.2
57.60.7
55.03.4
79.81.6
52.4..
4.94.
53.5
Table 5: The performance of NPPrompt with RoBERTa-large on GLUE benchmark against other methods, including few-shot learning methods. Manual Label: using the buman-designe prompts in Gao et (2021): In-context learning: using the in-context learning proposed in Brown et al. (2020) with RoBERTa-large; Auto-L: method in Gao et al. (2021); AMuLaP: method in Wang et al. (2022); Majority: majority class.
5.3 MAIN RESULTS
We demonstrate experimental results in Table 4. Overall NPPrompt outperforms Null Prompt and Multi-Null Prompt remarkably by over 10 percent in a fully zero-shot setting. NPPrompt achieves accuracy of over 85% AG News and DBPedia and over 90% on IMDB and Amazon. We conjecture that topic classifications in AG News and DBPedia are more complicated than binary sentiment classifications in IMDB and Amazon, hence the higher accuracy the latter.

---

Under review as a conference paper at ICLR 2023
NPPrompt is only slightly worse than KPT but outperforms most baseline methods in which human efforts/cxternal knowledge or unlabeled data are strictly required. It's worth noting that NPPrompt performs much better than ManualVerb suggesting that the label words generated by method are more comprehensive and unbiased than human-designed ones. Besides, NPPrompt can beat GPT-3 by 4% in terms of average accuracy, strong sign of the great potential for RoBERTa-large with 355M parameters compared to 175B parameters giant GPT-3.
To explore how our method NPPrompt performs on different kinds of tasks, we also conduct ex- periments on GLUE benchmark (Wang et al., 2018). Specifically, we test on Multi-Genre Nat- ural Language Inference Matched (MNLI), Multi-Genre Natural Language Inference Mismatched (MNLI-mm)o WWilliams (WVllias et al., 2018), Question Natural Language Inference (QNLI) (Rajpurkar et al., 2016) and Recognizing Textual Entailment (RTE) (Bentivogli al., 2009) for Natural Language ference (NL.); Microsoft Research Paraphrase Matching (MRPC) (Dolan & Brockett, 2005) and Quora Question Pairs (QQP) (Chen et al., 2018) for Paraphrase Similarity Matching: Stanford Sen- timent Treebank (SST-2) (Socher al., 2013) for Sentiment Classification; The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) for Linguistic Acceptability.
As shown in Table 5, NPPrompt outperforms all other methods in fully zero-shot setting. Auto- (Gao et al., 2021) and AMuLaP (Wang et al., 2022) are both automatic label words searching methods utilizing few-shot examples. Our method NPPrompt can even outperform them without any unlaheled data or few-shot training examples.
5.4 EFFECTS OF SIMILARITY FUNCTIONS IN NONPARAMETRIC AGGREGATION
Both weight and similarity functions play a critical role in the design of NPPrompt and we test how NPPrompt performs on AG News with different con- figurations. The "Default" setting is stated in
0.84
Test Accuracy
Equation 3 and 5. We fix the similarity function S (emb(rs (o:), emb(y;)) set wv,, Vj) = 1 for the "Same weight" setting and
0.80
emb(v.) emb(y,)
0.78
[emb(v.]]] [[emb(,)]]
Same weight
.embbyy for the "Average weight" setting. Besides cosine similarity, the Euclidean distance and the dot product are also common similarity measures for embeddings. Con- sequently, we fix the weight w(e;, Yj) = 1, choose (emb(e;) emb(y,) S (emb(r:) emb(y;)
w(,, w3) E,,,,iy,,)
(eeemb(;)..mb(00)
0.74
Average weight
Bucidean distance
Dot product
10
11
12
Neighborhood Number:
Figure 2: Effects of different aggregation.
emb(y,)) for the "Euclidean distance" setting and emb(r:) emb(y;) for the "Dot product" setting. It can be informed from Figure 2 that with a fixed similarity function, different weight calculations yield comparable results, but with fixed weight, cosine similarity is the optimal similarity measure.
- -Ieemb(e;
5.5 CAN WE SUM OVER PROBABILITIES?
NPPrompt sums up all logits for a label word set as shown in Equation& 4. Another possible approach to sum up the probabilities from PLM's prediction for the label words and choose the argmax for all different labels as the prediction: P(y, prompp))
0.84
Test Accuracy
0.82
0.80
Evee((ss) w(v, y P(MMASK] = is prompp))
0.78
arg max P (V; pprompt) We conduct experi- M)
0.76
ments on AG News to compare the above two ap- proaches, one that sums up logits ("sum logit") and one that sums up probabilities ("sum prob"). Figure 3 presents the results and we find that "sum logit" performs better at small & but "sum prob" delivers better results when & exceeds 30. "sum logit" achieves the best result at k = 12 among all experi- ments.
prob
20
60
100
Neighborhood Number: k
Figure 3: Test results AG News.
7

---

Under review as a conference paper at ICLR 2023
5.6
How MANY LABEL WORDS SHOULD WE CHOOSE?
DBPedia
Test Accuracy
Test Accuracy
0.000
0.730
0.75
0.725
0.675
1b
11
12
13
14
15
16
10
11
IMDB
Amazon
0.9
0.9
Accuracy
Test Accuracy
0.0
0.8
0.7
0.7
100
200
300
420
500
600
700
50c
900
130
140
150
160
170
180
100
200
210
Neighborhood Number:
Neighborhood Number: k
Figure Test results of NPPrompt for four PLMs with different neighborhood numbers.
The number of the label words impacts the performance of our method NPPrompt as well. In Figure 4, we display the performances of different models with varied neighborhood numbers. In general, NPPrompt attains símilar test accuracy across different neighborhood numbers. Regardless of the choice for neighborhood number, topic classification tasks AG News and DBPedia, and it gains over 90% accuracy in sentiment classification tasks on IMDB and Amazon. real-world applications, we can simply choose fixed neighborhood number (e.g. 8-10) achieve decent performance.
Cerrompt-RoBERTTa
achieves over 80% accuracy in
5.7
How DOES NPPROMPT PERFORM WITH DIFFERENT PLMs?
Method
AG News
DBPedia
IMDB
Amazon
Avg.
NPPrompt- BERT-base
DePrompt-BERTTbeee
79.4
30
77.8
57.7
53.5
180
67.1
NPPrompt- DePrompt-BERT--AE BERT-large
82.7
8
80.9
30
81.6
10
80.8
210
81.5
NPPrompt-Ro base
75.3
11
82.8
88.7
83.9
250
82.7
NPProompt-RoBBERTT--lllrrg
85.0
12
86.8
94.1
500
93.9
170
90.0
Table 6: The zero-shot results of different backbones. We also include the best neighborhood num- ber as the second column in each category. datasets.
performs the best in all
NPPrompt highly depends on the pre-trained language model. The label words for the categories with various PLMs are different. a result of their unique initial word embedding and vocabularies. To study the effect of applying different PLMs, we conduct extra experiments using BERT-base-cased, BERT-large-casee and RoBERTa-bann RoBERTa-large generates the best performance, which may result from the fact that RoBERTa-large has the largest number of parameters and that is pre-trained on largest corpus. In general, larger models MaoBERRTa-lrgg/BERRRTTT base/BERRTbbas) as expected, and RoBERTa shows better accuracy than BERT models on average.
models. The results are shown in Table NPPrompt with
achieve better performances than base models (RoBERTa-
5.8 WHAT LABEL WORDS DO DIFFERENT PLMS CHOOSE?
We summarize the label words of different PLMs for SCHOOL category in DBPedia in Table 7. RoBERTa-large and RoBERTa-base share similar sets of label words yet with minor discrepancy
8

---

Under review as a conference paper at ICLR 2023
RoBERTa-large
RoBERTa-base
BERT-large
BERT-base
Word
Sim
Word
Sim
Word
Sim
Word
Sim
"cchool"
1.00
school""
1.00
"school"
1.00
"school"
1.00
"School"
0.80
School"
0.75
"School"
0.69
"School"
0.70
schools"
0.77
schools"
0.71
"schools"
0.63
"schools"
0.63
"scoool"
0.74
"school"
0.70
"college"
0.55
"college"
0.54
SCHOOL"
0.69
"School"
0.70
"university"
0.50
"university"
0.51
"School"
0.68
SCHOOL"
0.56
"student"
0.42
"College"
0.40
university"
0.66
college"
0.50
"church"
0.41
"church"
0.40
college"
0.65
university"
0.50
"house"
0.38
"student"
0.37
Schools"
0.65
Schools"
0.49
"education"
0.38
"students"
0.37
schooling"
0,64
schooling"
0.45
"students"
0.37
"Schools"
0.37
preschool"
0.63
preschool"
0.44
"class"
0.37
"academy"
0.37
kindergarten"
0.63
kindergarten"
0.41
"town"
0.37
"class"
0.36
classroom"
0.60
student"
0.41
"College"
0.36
"education"
0,36
student"
0.58
students"
0.39
"Schools"
0.36
"University"
0.35
education"
0.58
classroom"
0.38
"work"
0.35
"house"
0.35
Table 7: The top 15 similar words of SCHOOL category in the DBPedia dataset. Sim: similarity scores.
between their similarity scores. RoBERTa-large usually produces larger similarities than RoBERTa- base. In contrast, the label words in RoBERTa are quite different from those in BERT.
DISCUSSION
NPPrompt achieves superior results in zero-shot text classifications. We attribute good performance two aspects. Firstly, compared fixed words or human-designed label words, using the initial word embedding from PLMs enables us to find cognates of the label words. For example, we have [" school", School", schools", SCHOOL"... for the SCHOOL category, as shown in Table 7. Secondly, effectively elicit the potential of pre-trained language models. During the pre- pre-trained process, language models are required to predict the masked token. The prediction the [MASK] token of the PLM is not fixed in the inference stage, that there is standard correct answer to fit into the context and instead, multiple words sharing similar meanings can be predicted. Our approach reformulates the zero-shot classification problem the masked token prediction problem which is well aligned with the pre-training process.
promising way deal with dynamic and open zero-shot classification prob- lems where new classes can emerge or old classes should be deleted. Efficient PLMs and category names are all we need. Together with the key words design Equation 7, NPPrompt can also work in special scenarios where label names do not have semantic meaning (c.g. category with label name ("A", "B", "C")). This technique can be widely deployed in real- wallworld applications.
NPPrompt points
Limitations and Future Directions For those label names without semantic meanings, several keywords are still required for NPPrompt to work well. In addition, we only focus on the zero- shot setting. There are more explore when considering the few-shot examples in many practical applications. Besides, we only test on text classification and NLU tasks from the GLUE benchmark. Whether NPPrompt works on other tasks such as ranking or relation extraction remains an open question.
CONCLUSION
In this paper, we propose NPPrompt, novel and effective method for fully zero-shot learning with pre-trained language models. We use initial word embedding of PLM to automatically find related words for category names, which enables construct the verbalizers without manual design or unlabeled corpus. Experimental results show that NPPrompt outperforms the previous zero-shot methods by large margins.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The sixth pascal recognizing textual entailment challenge. In TAC, 2009.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. Ms marco: human generated machine reading comprehension dataset. ArXiv, abs/ 109268, 2016.
Z. Chen, H. Zhang, X. Zhang, and L. Zhao. Quora question pairs, 2018. URL https://w//w. qww.a...........ooo.......... Com/pprofilee/RiccyyRcichee 2/First-Quorar -ewaaaae--eeeeeeeeeeeeeeee
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. Openprompt: An open-source framework for prompt-learning. arXiv preprint arXiv:2111.01998, 2021.
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In IJNLPP, 2005.
Joshua Feldman, Joe Davison, and Alexander M. Rush. Commonsense knowledge mining from pretrained models. In EMNLP, 2019.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. ArXiv, abs/2105. 11259, 2021.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juan-Zi Li, and Maosong Sun. Knowl- edgeable prompt-tuning Incorporating knowledge into prompt verbalizer for classification. In ACL, 2022.
Robert I. Logan IV, Ivana Balavzevi'c, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language mod- els. In FINDINGS, 2022.
Kamran Kowsari, K. Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura E. Barnes, and Don- ald E. Brown. Text classification algorithms: A survey. Information, 10:150, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervis learning of language representations. In ICLR, 2019.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, S. Auer, and Christian Bizer. Dbpedia a lcrge-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167-195, 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sqquence-to-sqquence pre- training for natural language generation, translation, and comprehension. In ACL, 2020.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert Y. S. Lam. Reconstructing capsule networks for zero-shot intent classification. In EMNLP, 2019a.
10

---

Under review as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692 2019b.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, A. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In RecSys, 2013.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text classification using label names only; A language model self-training approach. In EMNLP, 2020.
Jinscok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz. All-in text: Learning document, label, and word representations jointly. In AAAI, 2016.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-611 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In EMNLP, 2019.
Anthony Rios and Ramakanth Kavuluru. Few-shot and zero-shot multi-label learning for structured label spaces. In EMNLP, 2018.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACI, 2021a.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL, 2021b.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In EMNLP, 2013.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval, 2021. URL
15316.
Yi Sun, Yu Zheng, Chao Hao, and Hangping Qiu. Nsp-bert: A prompt-based zero-shot learner through an original pre-training task-next sentence prediction. ArXiv, abs/2109.03564, 2021.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In NAACI, 2019.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNI EMNLP, 2018.
11

---

Under review as a conference paper at ICLR 2023
Ben Wang. Tessh-rrassfomer-JAA Model-Parallel Implementation of Transformer Language
Model with JAX. https://gitte http://....nnnggllo/////ssssttee http:///ess--rraafffrrrrr-raaaae May 2021.
Han Wang, Canwen Xu, and Julian McAuley. Automatic multi-label prompting: Simple and inter-
pretable few-shot classification. In NAACL, 2022.
Yue Wang, Lijun Wu, Xiaobo Liang, Juntao Li, and Min Zhang. Are bert families zero-shot learners? study their potential and limitations. OpenReview, https//opeemeeiev...oooooddddddd USkf, 2021.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou.
Chain of thought prompting elicits reasoning large language models. In NeurlPS, 2022.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Chang, and Philip S. Yu. Zero-shot user intent detection via capsule neural networks. In EMNLP, 2018.
E. Zelikman, Yuhuai Wu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. ArXiv, Ahs/2203.14465 2022.
Jingqing Zhang, Piyawat Lertvittayakujjrn, and Yike Guo. Integrating semantic knowledge to tackle zero-shot text classification. NAACI, 2019.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
A APPENDIX
A.1 EXPERIMENTAL DETAILS
Table shows all the manual templates of NSP-BERT. Table 9 summarizes manual designed de- scriptions of each dataset for Semantic Retrieval, As for GPT-3, we query the OpenAl API' and test with Davinci model. The prompts for GPT-3 are shown in Table 10. We list all templates and label names for NPPrompt of all experiments in Table 11. We also list the related words result in sentiment classification (GoOD/BAD) and NLI (YEs/NO)) tasks in Table A.I.
Dataset
Template
AG News
News: label name.
DBPedia
News: label name.
IMDB
This text shows label name sentiment. The attitude of this text is label name.
Amazon
Table The templates of NSP-BERT (Sun et al., 2021) in Table
htttps://opeaai com/appp/
12

---

Under review as a conference paper at ICLR 2023
Descriptions
AG News:
The politics category is related politics, government, and law.
The sports category is related to sports, competition, and athletics.
The business category related to business, portfolio, economics, and money.
The technology category is related to technology. software, system, and science.
DBPedia:
The company category is related to company, corporation, enterprise, brand, and business.
The school category is related to school, academy, university, and college.
The artist category is related artist, art, painter, musician, singer, and creative.
The athlete category is related to athletes, sports, Olympic, and gym.
The politics category is related to politics, government, and law.
The transportation category is related to transportation, transport, vehicle, and traffic. The building category related to buildings, construction, and structure. The mountain category is related to river, lake, bay, and mountain. The village category is related to village, town, and rural.
The animal category is related to animal, wildlife, and nature.
The plant category is related to plant, shrub, tree, and forest. The album category is related to album, lyrics, cd, and song. The film category is related to film, movic, cinema, and video. The book category is related to book, novel, and publication. IMDB:
The bad category is related to negative and bad reviews.
The good category is related to positive and good reviews.
Amazon:
The bad category is related to negative and bad reviews. The good category is related to positive and good reviews.
Table 9: Descriptions for Semantic Retrieval in Table 4.
Prompts for GPT-3
AG News:
[Descriptions] Definition: In this task, you are given sentence. Your job is to classify the fol- lowing sentence into one of the four different categories. The categories are: "politics", "sports", "business", and "technology". Input: [x]. Output:
DBPedia:
[Descriptions: Definition: In this task, you are given sentence. Your job is to classify the following sentence into one of the fourteen different categories. The categories are: "company", "school", "artist", "athlete", "politics", "transportation" "building", "mountain", "village", "an- imal", "plant", "album", "film". and "book". Input: [x]. Output:
IMDB:
[Descriptions] Definition: In this task, you are given a sentence. Your job is to classify the following sentence into one of the two categories. The categories are: "bad" and "good". Input:
[x]. Output:
Amazon:
desccription: Definition: In this task, you are given sentence. Your job is to classify the following sentence into one the two categories. The categories are: "bad" and "good", Input:
[x]. Output:
Table 10: Prompts for GPT-3 with descriptions [Descriptions] from Table and input text [x].
13

---

Under review as a conference paper at ICLR 2023
Dataset
Template
Label Names
k
category 1: world, politics
category 2: sports
AG News
A [MASK] news: : æ.
12
category 3: business
category 4: technology. science
category 1: company
category 2: school
category 3: artist
category 4: sports
category 5: politics, office
bus, train category 7: building, construct, room, tower 7
category 6: transportation,
31 2/2 In this sentence, #1
DBPedia
is [MASK]
category 8: river, lake, mountain
category 9: village
category 10: animal, pet
category 11: plant
category 12: album
category 13: film
category 14: book, publication
positive: good
IMDB
All in all, it was [MASK].
500
negative: bad
positive: good
Amazon
All in all, it was [MASK]
170
negative: bad
positive: great
SST-2
I1 It was [MASK]
9
negative: terrible
entailment: yes
MNLI
31 [MASK] 372
neutral: maybe
4
contradiction: no
entailment: yes
MNLI-mm
F1 [MASK] .22
neutral: maybe
4
contradiction: no
entailment: Yes, Indeed, Overall
QNLI
31 [MASK] T2
3
not entelmmiimmt:: No, Well, However
entailment: Yes
RTE
21 [MASK] 222
10
not entailment: No
equivalent: Yes
MRPC
I1 [MASK] D2
9
not equivalent: No
equivalent: Yes
QQP
H1 [MASK] 22
9
not nquievvia:::::: No
grammatical: true
CoLA
T1 This is [MASK]
7
not got-grammatiaa::: wrong
Table 11: Templates and label names for NPPrompt. k refers to the best neighborhood number for RoBERTa-large.
14

---

Under review as a conference paper at ICLR 2023
GOOD
BAD
YES
No
Word
Sim
Word
Sim
Word
Sim
Word
Sim
goodd
1.00
"bad"
1.00
Yes"
1.00
No"
1.00
"Godd"
0.73
"Bad"
0.71
yes"
0.79
"no"
0.80
GOOD"
0.72
"terrible"
0.69
YES"
0.73
"No"
0.74
"oodd"
0.69
"ADD"
0.69
"Yes"
0.72
NN"
0.70
great"
0.66
"horrible"
0.68 Yeah"
0.72 Nope
0.62
excellent"
0.66
"bad"
0.65 Yep"
0.65
Yes"
0,62
decent"
0.66
awful"
0.64 Sure"
0.62 "no"
0.61
"oodd
0.65
"Bad"
0.64 No"
0.62
Nobody"
0.59
nice"
0.64
good"
0.63 Indeed"
0.61
Nos"
0.57
bad"
0.63
badly"
0.62 yeah"
0.60
The"
0.57
better"
0.62
crappy"
0.60 "yes"
0.59
Yeah"
0.57
wonderful"
0.58
lousy"
0.60 Wow"
0.59
Nothing"
0.56
best"
0.58
worst"
0.60
Absolutely"
0,58
Not"
0.56
terrific"
0.57
horrendous"
0.60
Nope".
0.58
Never"
0.56
fantastic"
0.57
worse"
0.59
Okay"
0.57
None"
0.55
mediacre"
0.57
nasty"
0.59
Oh"
0.57
Number"
0.55
lousy"
0.57
"shitty"
"ello"
0.57
So"
0.54
satisfactory"
0.56
dreadful"
0.59
Hey"
0.57
Any"
0.54
marvelous"
0.56
rotten""
0.58
Nevertheless"
0.57
Andd
0.54
GREAT
0.56
harmfull
0.58
"However"
0.56
"NO"
0.53
Table 12: The top 20 similar words of label names in sentiment classification (GOOD/BAD) and NLI (YEs/No) tasks.
A.2 EXPERIMENTS OF NPPROMPT WITH T5 MODEL
NPPrompt also works text-to-text pre-trained language models (e.g. T5 (Raffel al., 2020)) with minor modification. We use T5-base to generate the missing spans at the end of the prompt text. We choose the first predicted token the input to the verbalizer and follow the nonparametric aggregation steps to decide the category. The results are shown in Table A.1. NPPrompt-Te T5-base shows good performance but can not outperform NPPrompt-Ro RoBERTa--argee
Dataset
Template
k
Accuracy
AG News
x this sentence, the topic is about [MASK]
15
76.82.3
DBPedia
31 22 In this sentence, 21 is [MASK]
15
78.31.s
IMDB
æ In summary, the movie was [MASK]
15
68.52.1
Amazon
All in all, it was [MASK]
15
65.3
Table 13: Prompt template and test results of NPPrompt with T5-base.
A.3 EXPERIMENTS NPPROMPT ON MULTIPLE CHOICE QUESTION ANSWERING TASK
We test NPPrompt on the CommonsenseeQA (CQA) dataset (Talmor et al., 2019), widely used multiple-choice question answering (QA) task. In this new setting, we use the prompt template "r The answer is [MASK].", e.g. "What do animals do when enemy is approaching? The answer [MASK].", Then we search for k-nearest neighbors for each target answer with = 15. Finally, follow the process when deal with text classification tasks and obtain the prediction. The experiment results are listed Table A. (few-shot results from Zelikman et al. (2022)). NPPrompt not only achieves satisfactory results CommonsenseQA dataset but even outperforms few-shot GPT-J (Wang, 2021) as well.
15

---

Under review as a conference paper at ICLR 2023
Method
CQA Dev Set Accuracy
Few-shot Direct GPT-J
20,9
Few-shot CoT GPT-J
36.6
Few-shot CoT LaMDA 137B
55.6
- loBERTa-
34.2
Table 14: Test results on CommonsenseeQ dataset. Direct: directly output the final answer; CoT: prompted with chain-of-thouue -hought (CoT) rationales; LaMDA: method in Wei et al. (2022).
A.4 EXTENSION TO MULTI- WORD EXPRESSIONS
Here extend our method to support multi-word label names like NATURALPLACE, MEANOF- TRANSPORTATION and etc. The major part is to obtain related words to a multi-word label name. Once we obtain the related words, the rest non-parametric aggregation step remains identical. We consider two scenarios:
The label name is multi-word (i.e., phrase) and related words are still single-words To model the phrase, we use average contextualized embedding instead of word embedding for both label names and related single-words compute cosine similarity. As suggested in Su al. (2021), we whiten the contextualized output of RoBERTa by linear transformation obtained from the contextualized embedding of all words in vocabulary. To obtain the best result, we select the output of layer of RoBERTa. This extension achieves 61% accuracy on the DBpedia dataset using the original multi word label names (original label names can be found at https: //rdrr/drrr///// cr/ccann/ tex tex /ddata/aaa//////aaaaaaa/// http///annnaatttt...eeee dataset_dbpedia htmml.d...hm))............
Both the label name and related words are phrases Since the search space of a related phrase exponentially large its length, we use is "[LABEL_NAME" can also be called [MASK]=n.", where n is the length of the candidate. For example, if the label name is MEANOFTR ANSPORTATION and "Mean of transportation can also be called [MASK] [MASK].". We feed it to RoBERTa and filter top-k candidate phrases of masked prediction. Since masked prediction is conditionally independent of each mask, we further re-rank the top-k candidate phrases by either the contextualized embedding method mentioned above or another nuto-regressive LM. For the latter one, we evaluate the per- plexity of the template with [MASK] filled by candidate phrases. This generates 71% accuracy on DBpedia the length of the phrase is two and the re-ranking is performed by GPT-2 (Radford al., 2019).
another prompt filter candidate words. The template
2, the template will look like
16


Reviews:
[{"review_id": "f3K8F_mBpoG", "reviewer": "Reviewer_LMgP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to effectively transfer pretrained language models to natural language understanding (NLU) tasks in a zero-shot manner. The proposed method, NPPrompt, does not require any labeled sample or rely on humans to construct prompt label words. NPPrompt generates the label words by searching the related words from the initial word embedding of the pre-trained language model. Then, it aggregates logits from the label words and predicts the category with the largest score. Experiments show that the proposed method is effective and outperforms strong baselines with a large margin.", "summary_of_the_review": "This paper proposed NPPrompt, which aims to enable LMs' zero-shot ability to NLU tasks without requiring any labeled sample or relying on humans to construct prompt label words. Results show that NNPrompt outperforms strong baselines with a large margin. However, the overall novelty of this paper is limited. ", "strength_and_weaknesses": "Strength\n1. Proposed method is simple, intuitive and effective.\n2. Conduct extensive experiments on a wide range of NLU tasks.\n\nWeaknesses\n1. The proposed method is not very novel. Generalizing LMs to NLU tasks by checking the vocabulary distribution over the masked token is a common practice. The method which leverages related words to label category does not look very novel to me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IzAu2fWH63", "reviewer": "Reviewer_Gywf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new language model named non parametric prompting PLM for natural language understanding specially for zero-shot learning. It is an important topic because these days many word-class associations are being produced by end users and previous models heavily depend on unlabeled data and human effort.  The authors showed that the proposed method outperforms state-of-the-art in terms of text classification accuracy and GLUE benchmarks on four different datasets including AG news, DBPedia, IMDB. and Amazon.", "summary_of_the_review": "Overall, zero shot learning is an interesting topic in natural language processing as so many new categories and topics are being produced on the web. The authors proposed a simple and easy to implement method for pre trained language models to minimize human effort in terms of labeling and building training data. Overall I am satisfied with the current draft of the paper and request to move forward with discussion. ", "strength_and_weaknesses": "The authors put significant effort on proving effectiveness of their method in a variety of NLP tasks. However, I wanted to see significant test results to make sure that the improvements are not random. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BzoQuofHjas", "reviewer": "Reviewer_DKg9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel method for zero-shot inference with masked language models (MLMs) such as BERT and RoBERTa for classification problems. The key idea is to use the encoders of the MLMs themselves to find relevant words for a given label name. For example, if the target label is \"SPORTS\", then the proposed method will first find the words by searching k-nearest neighbors with embedding distance. The embeddings are induced by the MLM itself. Finally, they aggregate the logits of these k words (for filling the masked positions). The authors argue that this new method (named NPPrompt) is a fully zero-shot classification method because it does not need any other information about labels. The empirical comparisons show that the proposed method has a comparable performance with other zero-shot methods (that needs some additional info such as KB or the unlabeled data). \n\n", "summary_of_the_review": "Overall, I think the paper presents a novel zero-shot classification paper but the limitations are there and the performance is not that significant. I think if the authors can extend the paper with my suggestions in the above section, the paper will be more suitable to the ICLR community. Otherwise, it will have much less impact to the field. ", "strength_and_weaknesses": "## Strengths\n\n- The proposed NPPrompt method is simple and effective. It only requires the label strings for doing zero-shot classification. \n- The empirical results show that it is on the same level as other methods that need more information about the labels.\n\n## Weakness \n\n- The proposed method is pretty limited to the classification tasks, particularly for news classification, where the label names are informative and have many semantic relevant words in the vocabulary. It is also limited when the label names are multi-word expressions or phrases. It does not support the cases where the label names are not informative enough and additional label description is needed. \n\n- The title and narrative can be misleading and an overclaim. As mentioned before, it is limited to masked LMs and the advantage is mainly for tasks like news classification. \n\nSuggestion:\n\n- I suggest the authors extend the proposed method to *multi-label classification* settings where an example can have multiple acceptable news. \n- A major limitation is the multi-word expression and out-of-vocabluary label names.  How do you generalize the method to support that?\n- Also, I suggest the authors try some multiple-choice QA problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject